LPMM: Intuitive Pose Control for Neural Talking-Head Model via Landmark-Parameter
Myung Ki Lee
NAVER WEBTOON AI
CVPR Workshop 2023, AI4CC
While current talking head models are capable of generating photorealistic talking head videos,
they provide limited pose controllability. Most methods require specific video sequences that
should exactly contain the head pose desired, being far from user-friendly pose control.
Three-dimensional morphable models (3DMM) offer semantic pose control,
but they fail to capture certain expressions. We present a novel method that utilizes
parametric control of head orientation and facial expression over a pre-trained neural-talking head model.
To enable this, we introduce a landmark-parameter morphable model (LPMM),
which offers control over the facial landmark domain through a set of semantic parameters.
Using LPMM, it is possible to adjust specific head pose factors, without distorting other facial attributes.
The results show our approach provides intuitive rig-like control over neural talking head models,
allowing both parameter and image-based inputs.
The training of our model is divided into two stages. (Left) The LP-regressor processes the input facial image to generate
LPMM parameters, and is trained so that the reconstructed facial landmark matches the original. (Right) The LP-adaptor is used to
transform LPMM parameters into the latent space of a pretrained talking head model’s pose encoder. While training LP-adaptor,
all weights other than LP-adaptor itself are frozen.
Rig-Lkie Control without Pose Source.
Rig-Lkie Control with Pose Source.
Kwangho Lee, Patrick Kwon, Myung Ki Lee, Namhyuk Ahn, Junsoo Lee. LPMM: Intuitive Pose Control for Neural Talking-Head Model via Landmark-Parameter Morphable Model In CVPR Workshop 2023, AI4CC. [Paper]