ViGA-FLAME: Aligning Latent Visual and Geometric Representations for Robust 3D Head Expression Estimation
Organizations: Department of Computer Science, Hong Kong Baptist University, Hong Kong SAR
Abstract
Estimating 3D head expression from a single image requires mapping visual appearance to underlying facial geometry. We propose ViGA-FLAME, a two-stage framework that aligns semantic visual features with a structured geometric latent space. First, a Parametric-Geometric Variational Autoencoder (PG-VAE) learns a unified representation of canonical head surfaces and FLAME parameters, supporting both detailed surface reconstruction and interpretable parameter recovery. Supervision on geometry and parameters, together with consistency between the two decoding branches, connects surface detail with parametric structure. Second, a trainable Visual-Geometric Adapter (ViGA) maps features from a pretrained semantic visual encoder into this latent space. Asymmetric attention enables ViGA to align semantic image features from the visual encoder with the structured geometric information represented in the PG-VAE latent space, without altering the original visual stream. By grounding image evidence in this latent prior, ViGA-FLAME enables robust 3D head expression estimation across severe facial deformations and large viewing angles, efficiently improving robustness over competing feed-forward methods.
Figures & tables
| Method | All typical | Shape extreme | Expression extreme | Pose extreme | All extreme | RenderMe360 [ 9 ] | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CD-L1 | F@5 | CD-L1 | F@5 | CD-L1 | F@5 | CD-L1 | F@5 | CD-L1 | F@5 | CD-L1 | F@5 | |
| DECA [ 3 ] | 3.181 | 0.864 | 3.468 | 0.822 | 3.285 | 0.849 | 6.460 | 0.650 | 6.642 | 0.608 | 3.972 | 0.747 |
| SMIRK [ 11 ] | 3.267 | 0.848 | 3.489 | 0.819 | 3.341 | 0.835 | 6.016 | 0.653 | 6.260 | 0.618 | 4.184 | 0.723 |
| TEASER [ 7 ] | 3.443 | 0.821 | 3.662 | 0.792 | 3.517 | 0.812 | 5.620 | 0.642 | 5.817 | 0.615 | 4.492 | 0.693 |
| SHeaP [ 13 ] | 4.269 | 0.823 | 4.988 | 0.769 | 4.786 | 0.788 | 8.023 | 0.613 | 8.113 | 0.585 | 3.754 | 0.780 |
| ViGA-FLAME Recon. | 2.693 | 0.937 | 2.774 | 0.923 | 2.727 | 0.930 | 3.581 | 0.835 | 3.768 | 0.798 | 4.430 | 0.696 |
| Method | Frontal ( ) | Oblique ( ) | Profile ( ) | |||
|---|---|---|---|---|---|---|
| CD-L1 | F@5 | CD-L1 | F@5 | CD-L1 | F@5 | |
| DECA [ 3 ] | 5.263 | 0.756 | 4.387 | 0.775 | 4.499 | 0.744 |
| SMIRK [ 11 ] | 4.394 | 0.779 | 4.373 | 0.774 | 4.616 | 0.723 |
| TEASER [ 7 ] | 4.116 | 0.772 | 4.407 | 0.745 | 4.564 | 0.710 |
| SHeaP [ 13 ] | 5.642 | 0.736 | 5.408 | 0.743 | 6.860 | 0.678 |
| ViGA-FLAME Recon. | 3.019 | 0.902 | 3.011 | 0.896 | 3.251 | 0.864 |
| Method | Recon. ↓ | Param. ↓ | Recon. Param. ↓ |
|---|---|---|---|
| Base | 1.0281 | 0.9472 | 1.2707 |
| + | 0.9942 | 0.8152 | 1.1726 |
| ++ | 1.0347 | 0.7643 | 1.2344 |
| +++ | 0.8647 | 0.4975 | 0.9713 |
| +++ | 0.9831 | 0.5830 | 1.1605 |
3D Face Reconstruction
3D face reconstruction estimates facial shape and appearance from images or video to create usable models and avatars.
See all 58 papers