ROSE: A Shared Cross-Embodiment Facial-Motion Latent Space for Humanoid Robot Expression
Authors: Yongji Fu, Man Cao, Yi Zhou, Fei Lin, Mark F. Hansen, Guanqun Cao
Organizations: Department of Engineering Mathematics, University of Bristol, Bristol, BS16 1QY, United Kingdom. · School of Architecture and Design, China University of Mining and Technology, Xuzhou, China. · Department of Computer Science, Cardiff University, Cardiff, CF10 3AT, United Kingdom. · Intelliface, Hangzhou, China · Bristol Robotics Laboratory, University of the West of England, Bristol, BS16 1QY, United Kingdom.
Natural, coherent full-face dynamic expressions are central to making humanoid robots socially approachable, yet existing systems commonly realize only limited lip motion or scripted expressions. The limitation is both physical and algorithmic. A compact head leaves little room for densely packed, independently controllable actuation; meanwhile, learned systems often generate an avatar video, landmarks, or facial parameters and then infer actuator commands through visual retargeting. This render-and-invert path adds latency and artifacts, and it does not reliably transfer generated behavior to a physical robot. We present ROSE (RObot facial expression through Shared motion Embeddings), a framework that integrates biomimetic hardware, cross-embodiment motion representation learning, and multimodal motion generation. On the hardware side, we build a flexible tendon-sheath transmission structure that integrates 34 actuators into a 19.5 cm biomimetic head. By decoupling actuator placement from facial action points, it enables coordinated full-face expression within a compact volume. On the algorithmic side, ROSE-VAE learns an identity-independent shared facial-motion latent space from human, digital-human, and robot observations. ROSE-Flow directly maps text, audio, or their combination to trajectories in this motion space without using video as a generator input. ROSE-ACT then converts these trajectories into constrained actuator commands. Our system demonstrates how high-density biomimetic hardware and a shared executable facial-motion representation can jointly unify cross-embodiment learning, multimodal generation, and physical robot expression.
Figures & tables
Fig. 1: ROSE overview. Human, digital-human, and face-robot observations are mapped by ROSE-VAE into a shared facial-motion latent space. ROSE-Flow generates motion-latent trajectories from text and/or audio, and ROSE-Act maps them to the actuator actions of a real robot. The right side illustrates real-time interaction, human-to-robot retargeting, and text/audio to expression applications.
Fig. 2: ROSE-Flow architecture. (a)The frozen ROSE-VAE encoder is used during training. The backbone is a text–motion dual-stream MMDiT followed by single-stream DiT blocks that process the concatenated condition and motion tokens. The inset shows the mask of the frame-local audio attention: each motion-frame query attends only to the audio tokens within its local window. Generated motion latents are mapped to real robot actuator commands; decoded faces are shown for visualization only. (b) Progressive modality unification: A2M pretraining, T2M adaptation initialized from θA , merging of θA and θT , and joint training of the unified model.
Fig. 3: Robot platform. Robot’s tendon-sheath transmission and actuator layout.
Motion autoencoder
CSIM-T ↑
CSIM-D ↓
AED ↓
APD, deg ↓
FLOAT VAE [ 38 ]
0.730
0.009
0.054
1.346
ROSE-VAE
0.741
0.006
0.264
1.260
TABLE I: Cross-identity reenactment for Q1. 24 driver–target pairs: NeRSemble test sequences of 50 frames each as driving motion and unseen HDTF identities as targets; both motion autoencoders use the same decode path, i.e., target appearance plus driving motion.
Dataset
Method
FID ↓
FVD ↓
CSIM ↑
E-FID ↓
P-FID ↓
LSE-D ↓
LSE-C ↑
HDTF, 86 clips
SadTalker [ 27 ]
60.3
193.1
0.931
0.265
0.002
10.526
5.556
AniTalker [ 28 ]
31.5
109.5
0.884
0.435
0.017
10.119
6.204
Hallo [ 47 ]
22.2
134.9
0.904
0.565
0.004
10.219
7.201
EchoMimic [ 48 ]
47.5
180.6
0.905
0.214
0.004
10.265
6.453
FLOAT [ 38 ]
20.3
70.2
0.862
0.323
0.002
10.388
7.070
ROSE, ours
13.5
54.6
0.934
0.167
0.002
9.785
7.773
TABLE II: Talking-face generation comparison on the HDTF and RAVDESS test lists. Every method receives the same first frame and audio of up to 5 s at 25 FPS; outputs are standardized to 512 × 512 before scoring. Best in bold, second best underlined.
Fig. 4: Cross-identity motion reenactment. Each row fixes one driving motion from the NeRSemble test sequences and reenacts it for the same four unseen target identities, one per column.
Setting
Mouth LPIPS
BS-MSE
LSE-D
w/o component perceptual loss
0.25
0.0180
6.72
w/o local audio attention
0.24
0.0208
7.18
w/o progressive modality unification
0.23
0.0193
6.89
Full ROSE
0.21
0.0164
6.44
TABLE III: ROSE component ablations. Lower is better for all metrics.
Statistic
ROSE
RIT [ 9 ]
NN landmark
Amplitude
Random
Speech 1: “Kids are talking by the door”, 95 frames, 3.8 s
Mean
0.0158
0.0202
0.4520
0.6855
0.5985
Std.
0.0084
0.0131
0.3190
0.3242
0.4491
Min.
0.0031
0.0033
0.0137
0.1827
0.0379
Max.
0.0512
0.0759
1.8102
1.7660
3.3223
Speech 2: “Dogs are sitting by the door”, 92 frames, 3.7 s
TABLE IV: Controlled lip-articulation comparison. Mean squared error, MSE, over two spoken and one sung RAVDESS test utterance. All baselines follow Hu et al. [ 9 ] and are re-implemented on the ROSE robot, evaluated on the same utterances and protocol.
Method
Realism ↑
Resonance ↑
Lip-Sync ↑
Random selection
1.34
1.41
1.38
Amplitude BL
1.52
1.47
1.69
NN landmark BL
1.72
1.66
2.05
RIT [ 9 ]
2.98
2.73
3.40
ROSE
3.41
3.36
3.78
TABLE V: User study. Mean five-point ratings of real-robot clips for realism, emotional resonance, and lip synchronization; baselines as in Table IV , executed on the ROSE robot.
Fig. 5: Temporal expression examples on the ROSE face robot. Each row shows nine real video frames of one expression sequence, arranged into three stages with the visible action described above the frames.