Facial action estimation from a single image is often formulated as predicting or fitting parameters in compact expression spaces, which lack explicit semantic interpretability. However, many practical applications, such as avatar control and human-computer interaction, require interpretable facial actions that correspond to meaningful muscle movements. In this work, we propose SemanticFace, a framework for facial action estimation in the interpretable ARKit blendshape space that reformulates coefficient prediction as structured semantic reasoning. SemanticFace adopts a two-stage semantic distillation paradigm: it first derives structured semantic supervision from ground-truth ARKit coefficients and then distills this knowledge into a multimodal large language model to predict interpretable facial action coefficients from images. Extensive experiments demonstrate that language-aligned semantic supervision improves both coefficient accuracy and perceptual consistency, while enabling strong cross-identity generalization and robustness to large domain shifts, including cartoon faces.
Figures & tables
Figure 2: Overview of the SemanticFace framework. In Stage I, a frozen LLM serves as a semantic teacher , converting ground-truth ARKit blendshape coefficients into hierarchical semantic descriptions. In Stage II, these structured language-aligned representations are distilled into a MLLM acting as the student . Through language-prior semantic distillation, the student learns to predict ARKit coefficients from images under structured semantic guidance within an interpretable facial action space.
Figure 3: Emotion Distribution and Visualization. (a) t-SNE projection of scripts colored by emotion intensities. (b) Distribution of dominant emotions across all scripts.
Method
Error
R-Precision ↑
MMD ↓
Main Cross-Comparison
MSE ↓
top-1
top-2
top-3
P-Cor
S-Cor
Acc.
MSD
Deviation
GT-test
0
24.28%
38.60%
48.48%
1.00
100%
100%
1
0
0
DeadFace [ 42 ]
0.0316
13.43%
23.06%
30.25%
2.12
68.48%
67.03%
0.9836
0.0094
0.0691
SBCA [ 32 ]
0.0190
-
-
-
-
67.44%
65.61%
0.8449
0.0298
0.1470
Ours
0.0034
22.15%
37.06%
48.06%
1.18
70.50%
63.32%
0.9967
0.0060
0.0554
Table 1: Quantitative comparison with different methods on the test set. Best values (excluding GT-test) are highlighted in red . The GT-test row is shown with a gray background for reference and is not included in the comparison.
Figure 4: Qualitative Comparison on the Test Set with Ground-Truth ARKit coefficients. From top to bottom: ground truth, DeadFace, and SemanticFace. Compared to the geometric baseline, our method produces facial actions that more closely match the ground-truth coefficients. The white numbers overlaid on each result denote the corresponding MSE values.
Method
R-Precision ↑
MMD ↓
top-1
top-2
top-3
DeadFace [ 42 ]
9.06%
19.90%
25.37%
0.90
SemanticFace
23.38%
36.98%
47.10%
0.55
Table 2: Quantitative comparison with different methods on the in-the-wild data (R-Precision & MMD). Best values are highlighted in red .
Figure 5: In-the-wild qualitative comparison of interpretable facial action estimation. SemanticFace maintains coherent and semantically consistent facial actions across diverse subjects and challenging conditions. The white numbers overlaid on each result denote the corresponding MMD values.
Figure 6: Comparison of ARKit-based and FLAME-based Methods on Hard Cases. The input images are compared with results from SemanticFace (ours), DeadFace, SMIRK, EMOCA, and pixel3DMM.
Figure 7: Qualitative comparison of larger domain shift.
Method
Mean MSE ↓
Median MSE ↓
Std ↓
P90 ↓
A0
3.66
3.44
1.64
5.74
A1
3.52
3.28
1.51
5.57
A2(Ours)
3.34
3.12
1.49
5.25
Table 3: Ablation analysis on semantic supervision evaluated on the test set. All values are scaled by 10−3 . We report overall mean MSE, median MSE, standard deviation (Std), and 90th percentile (P90). Best values are highlighted in red .
Comparison
Δ MSE ↓
t-statistic
p-value
A1 vs A0
−1.42×10−4
−5.28
1.35×10−7
A2 vs A1
−1.45×10−4
8.21
2.91×10−16
A2 vs A0
−2.87×10−4
10.61
5.43×10−26
Table 4: Paired t-test analysis on the test set using ablation settings A0, A1, and A2. Negative Δ indicates lower error for the first method in the comparison (i.e., Δ MSE is calculated as MSEfirst method−MSEsecond method ). All improvements are statistically significant ( p<0.001 ).
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Main Cross-Comparison
P-Cor
S-Cor
MSD
Deviation
HeadYaw(DeadFace [ 42 ] )
20.61%
21.03%
0.0074
0.0843
HeadYaw(Ours)
57.69%
55.60%
0.0048
0.0533
HeadPitch(DeadFace)
19.60%
21.36%
0.0350
0.0903
HeadPitch(Ours)
74.89%
72.09%
0.0045
0.0557
HeadRoll(DeadFace)
90.18%
89.60%
0.0009
0.0306
Appendix
Table 5: Quantitative comparison of head pose parameters (HeadYaw, HeadPitch, HeadRoll) on the test set. Best values are highlighted in red .
MSE
P Corr
S Corr
Deviation
MouthRight
0.0000
-
-
0.0000
TongueOut
0.0000
-
-
0.0000
LeftEyeRoll
0.0000
-
-
0.0000
RightEyeRoll
0.0000
-
-
0.0000
JawRight
0.0000
-0.0003
-0.0003
0.0025
CheekPuff
0.0000
-0.0005
-0.0005
0.0032
Appendix
Table 6: Quantitative evaluation of blendshape parameters (Part 1), sorted in ascending order of MSE.
MSE
P Corr
S Corr
Deviation
CheekSquintLeft
0.0029
0.6985
0.6919
0.0531
MouthUpperUpLeft
0.0029
0.7378
0.6936
0.0488
BrowDownRight
0.0032
0.6638
0.5664
0.0544
BrowDownLeft
0.0032
0.6636
0.5664
0.0544
EyeLookInRight
0.0033
0.4145
0.3049
0.0534
MouthUpperUpRight
0.0035
0.7353
0.6903
0.0507
Appendix
Table 7: Quantitative evaluation of blendshape parameters (Part 2), sorted in ascending order of MSE.