Accurate facial expression imitation on human-face robots is crucial for achieving natural human-robot interaction. Most existing methods have achieved photorealistic expression imitation through mapping 2D facial landmarks to a robot's actuator commands. Their imitation of landmark trajectories is susceptible to interference from facial morphology, which would lead to a performance drop. In this paper, we propose a morphology-independent expression imitation method that decouples expressions from facial morphology to eliminate morphological influence and produce more realistic expressions for human-face robots. Specifically, we construct an expression decoupling module to learn expression semantics by disentangling the expression representation from the morphology representation in a self-supervised manner. We devise an expression transfer module to map the representations to the robot's actuator commands through a learning objective of perceiving expression errors, producing accurate facial expressions based on the learned expression semantics. To support experimental validation, a custom-designed and highly expressive human-face robot, namely Pengrui, is developed to serve as an experimental platform for realistic expression imitation. Extensive experiments demonstrate that our method enables the human-face robot to reproduce a wide range of human-like expressions effectively. All code and implementation details of the robot will be released.
Figures & tables
Fig. 1: The overview of the morphology-independent facial expression imitation method. An facial image is processed by the expression decoupling module to extract the disentanglemented representations of facial expressions, morphology and poses. The expression representations are then mapped to actuator commands via the expression transfer module, actuating Pengrui, i.e., our custom-designed and highl expressive human-face robot, to reproduce the target expression.
Fig. 2: Illustration of the expression decoupling module and the expression transfer module.
Fig. 3: Pengrui, a highly articulated human-face robot platform. Pengrui robot has 32 actuators with 48 degrees of freedom.
Representation
surprise ↓
fear ↓
disgust ↓
happy ↓
sad ↓
anger ↓
neutral ↓
landmark-based
8.128
8.490
10.228
7.480
8.469
8.202
8.031
morphology-independent (ours)
4.442
6.857
4.636
4.874
8.282
5.471
7.849
TABLE I: CV comparisons of Representations across Different Face Morphologies under the Same Expression
Method
Overall Representation
Expression Representation
Morphology Representation
MSE ↓
MAE ↓
MSE ↓
MAE ↓
MSE ↓
MAE ↓
RG
3.838
1.612
4.001
1.669
3.981
1.663
EDM (ours)
1.006
0.794
1.108
0.840
1.012
0.806
TABLE II: MSE and MAE comparisons of Representations Errors
Method
MSE ↓
MAE ↓
EDM + RI
0.314
0.479
EDM + RI-10
0.089
0.283
EDM + NN
0.101
0.261
RG + ETM
0.303
0.465
EDM + ETM (ours)
0.042
0.174
TABLE III: MSE and MAE comparisons of Command Errors
Fig. 4: Representation Visualizations. t-SNE visualization comparing the morphology-independent representation (our method) and landmark-based representation.
Fig. 5: Examples of interferences from facial morphology. The 2D facial landmarks for the (a) happy expression on a narrow face, (b) happy expression on a broad face and (c) angry expression on a broad face, are mapped to low-dimensional space using the t-SNE algorithm. The distance for the same expression across different morphology shown in (a) and (b) significantly exceeds that for different expressions on the same morphology shown in (a) and (c).
Fig. 6: Robot Visualizations. The actuator commands a~ generated by our method are deployed on the human-face robot. The robot accurately reproduces facial expressions from different individuals, demonstrating the effectiveness and generalizability of our approach.