ROSE: A Shared Cross-Embodiment Facial-Motion Latent Space for Humanoid Robot Expression
Authors: Yongji Fu, Man Cao, Yi Zhou, Fei Lin, Mark F. Hansen, Guanqun Cao
Organizations: Department of Engineering Mathematics, University of Bristol, Bristol, BS16 1QY, United Kingdom. · School of Architecture and Design, China University of Mining and Technology, Xuzhou, China. · Department of Computer Science, Cardiff University, Cardiff, CF10 3AT, United Kingdom. · Intelliface, Hangzhou, China · Bristol Robotics Laboratory, University of the West of England, Bristol, BS16 1QY, United Kingdom.
Natural, coherent full-face dynamic expressions are central to making humanoid robots socially approachable, yet existing systems commonly realize only limited lip motion or scripted expressions. The limitation is both physical and algorithmic. A compact head leaves little room for densely packed, independently controllable actuation; meanwhile, learned systems often generate an avatar video, landmarks, or facial parameters and then infer actuator commands through visual retargeting. This render-and-invert path adds latency and artifacts, and it does not reliably transfer generated behavior to a physical robot. We present ROSE (RObot facial expression through Shared motion Embeddings), a framework that integrates biomimetic hardware, cross-embodiment motion representation learning, and multimodal motion generation. On the hardware side, we build a flexible tendon-sheath transmission structure that integrates 34 actuators into a 19.5 cm biomimetic head. By decoupling actuator placement from facial action points, it enables coordinated full-face expression within a compact volume. On the algorithmic side, ROSE-VAE learns an identity-independent shared facial-motion latent space from human, digital-human, and robot observations. ROSE-Flow directly maps text, audio, or their combination to trajectories in this motion space without using video as a generator input. ROSE-ACT then converts these trajectories into constrained actuator commands. Our system demonstrates how high-density biomimetic hardware and a shared executable facial-motion representation can jointly unify cross-embodiment learning, multimodal generation, and physical robot expression.
Figures & tables
Fig. 1: ROSE overview. Human, digital-human, and face-robot observations are mapped by ROSE-VAE into a shared facial-motion latent space. ROSE-Flow generates motion-latent trajectories from text and/or audio, and ROSE-Act maps them to the actuator actions of a real robot. The right side illustrates real-time interaction, human-to-robot retargeting, and text/audio to expression applications.
Fig. 2: ROSE-Flow architecture. (a)The frozen ROSE-VAE encoder is used during training. The backbone is a text–motion dual-stream MMDiT followed by single-stream DiT blocks that process the concatenated condition and motion tokens. The inset shows the mask of the frame-local audio attention: each motion-frame query attends only to the audio tokens within its local window. Generated motion latents are mapped to real robot actuator commands; decoded faces are shown for visualization only. (b) Progressive modality unification: A2M pretraining, T2M adaptation initialized from θA , merging of θA and θT , and joint training of the unified model.
Fig. 3: Robot platform. Robot’s tendon-sheath transmission and actuator layout.
Motion autoencoder
CSIM-T ↑
CSIM-D ↓
AED ↓
APD, deg ↓
FLOAT VAE [ 38 ]
0.730
0.009
0.054
1.346
ROSE-VAE
0.741
0.006
0.264
1.260
TABLE I: Cross-identity reenactment for Q1. 24 driver–target pairs: NeRSemble test sequences of 50 frames each as driving motion and unseen HDTF identities as targets; both motion autoencoders use the same decode path, i.e., target appearance plus driving motion.
Dataset
Method
FID ↓
FVD ↓
CSIM ↑
E-FID ↓
P-FID ↓
LSE-D ↓
LSE-C ↑
HDTF, 86 clips
SadTalker [ 27 ]
60.3
193.1
0.931
0.265
0.002
10.526
5.556
AniTalker [ 28 ]
31.5
109.5
0.884
0.435
0.017
10.119
6.204
Hallo [ 47 ]
22.2
134.9
0.904
0.565
0.004
10.219
7.201
EchoMimic [ 48 ]
47.5
180.6
0.905
0.214
0.004
10.265
6.453
FLOAT [ 38 ]
20.3
70.2
0.862
0.323
0.002
10.388
7.070
ROSE, ours
13.5
54.6
0.934
0.167
0.002
9.785
7.773
TABLE II: Talking-face generation comparison on the HDTF and RAVDESS test lists. Every method receives the same first frame and audio of up to 5 s at 25 FPS; outputs are standardized to 512 × 512 before scoring. Best in bold, second best underlined.
Fig. 4: Cross-identity motion reenactment. Each row fixes one driving motion from the NeRSemble test sequences and reenacts it for the same four unseen target identities, one per column.
Setting
Mouth LPIPS
BS-MSE
LSE-D
w/o component perceptual loss
0.25
0.0180
6.72
w/o local audio attention
0.24
0.0208
7.18
w/o progressive modality unification
0.23
0.0193
6.89
Full ROSE
0.21
0.0164
6.44
TABLE III: ROSE component ablations. Lower is better for all metrics.
Statistic
ROSE
RIT [ 9 ]
NN landmark
Amplitude
Random
Speech 1: “Kids are talking by the door”, 95 frames, 3.8 s
Mean
0.0158
0.0202
0.4520
0.6855
0.5985
Std.
0.0084
0.0131
0.3190
0.3242
0.4491
Min.
0.0031
0.0033
0.0137
0.1827
0.0379
Max.
0.0512
0.0759
1.8102
1.7660
3.3223
Speech 2: “Dogs are sitting by the door”, 92 frames, 3.7 s
TABLE IV: Controlled lip-articulation comparison. Mean squared error, MSE, over two spoken and one sung RAVDESS test utterance. All baselines follow Hu et al. [ 9 ] and are re-implemented on the ROSE robot, evaluated on the same utterances and protocol.
Method
Realism ↑
Resonance ↑
Lip-Sync ↑
Random selection
1.34
1.41
1.38
Amplitude BL
1.52
1.47
1.69
NN landmark BL
1.72
1.66
2.05
RIT [ 9 ]
2.98
2.73
3.40
ROSE
3.41
3.36
3.78
TABLE V: User study. Mean five-point ratings of real-robot clips for realism, emotional resonance, and lip synchronization; baselines as in Table IV , executed on the ROSE robot.
Fig. 5: Temporal expression examples on the ROSE face robot. Each row shows nine real video frames of one expression sequence, arranged into three stages with the visible action described above the frames.
Accurate facial expression imitation on human-face robots is crucial for achieving natural human-robot interaction. Most existing methods have achieved photorealistic expression imitation through mapping 2D facial landmarks to a robot's actuator commands. Their imitation of landmark trajectories is susceptible to interference from facial morphology, which would lead to a performance drop. In this paper, we propose a morphology-independent expression imitation method that decouples expressions from facial morphology to eliminate morphological influence and produce more realistic expressions for human-face robots. Specifically, we construct an expression decoupling module to learn expression semantics by disentangling the expression representation from the morphology representation in a self-supervised manner. We devise an expression transfer module to map the representations to the robot's actuator commands through a learning objective of perceiving expression errors, producing accurate facial expressions based on the learned expression semantics. To support experimental validation, a custom-designed and highly expressive human-face robot, namely Pengrui, is developed to serve as an experimental platform for realistic expression imitation. Extensive experiments demonstrate that our method enables the human-face robot to reproduce a wide range of human-like expressions effectively. All code and implementation details of the robot will be released.
Humanoid robots are increasingly expected to serve as embodied social agents that communicate naturally with humans through face-to-face interaction. During such communication, humanoid robots require body behaviors that are synchronized with speech, affectively expressive, and suitable for real-time execution. However, existing co-speech methods are primarily developed for digital humans and lack joint support for affective control and low-latency continuous generation on physical embodiments. To bridge this gap, we present SocialHumanoid, a system for expressive humanoid behavior via one-step co-speech motion generation. Given response speech and a specified affective condition, SocialHumanoid generates each full-body motion window in a single forward pass and connects successive windows through motion-history conditioning. The generated human motion is further converted online into embodiment-compatible robot references and tracked by a whole-body controller for physical execution. To provide explicit supervision for affective body expression, we further introduce AffectMoCap, a 4-hour dataset captured from two professional actors, containing synchronized speech, body motion, fine-grained hand motion, and emotion annotations. On BEAT2, SocialHumanoid achieves the best FGD among the compared generation methods, competitive speech-motion synchrony, and approximately 6× faster inference than GestureLSM under the same protocol. Perceptual evaluations further show that training with AffectMoCap improves affect recognition from generated body motion, while real-robot experiments demonstrate continuous affect-conditioned behavior and stable long-horizon execution. Our project page is https://rex0191.github.io/SocialHumanoid/.
Diverse, semantically synchronized, and low-latency facial expressions are essential for human-robot interaction. While large language models (LLMs) have endowed anthropomorphic robots with fluent speech, facial expression still struggles to follow the conversation. Motivated by this gap, we present Emo-H2A (hidden-to-actuation), a paradigm that drives a facial robot directly from LLM hidden states during autoregressive decoding. Without a separate emotion label, expression stays continuous and context-sensitive, and the face moves with the unfolding reply. We instantiate this paradigm as EmoManifold, a training-free method that extracts a valence-arousal-dominance (VAD) coordinate from each hidden state, projects it, and drives the face at token granularity. The readout is checked on six language models, and the same projection drives our 29-degree-of-freedom (DoF) facial robot. When driven by this projection, the face shifts its expression in the matching direction on 87.5% of within-reply valence transitions, whereas holding a single label for the whole reply leaves directional tracking near the 50% chance level, and first moves in 0.96s rather than waiting 10-26s for the reply to finish and a turn-level label to return.
Yige Liu, Jingcheng Huang, Hao Sun +2
Huazhong University of Science and Technology, Wuhan 430074, China.