Diverse, semantically synchronized, and low-latency facial expressions are essential for human-robot interaction. While large language models (LLMs) have endowed anthropomorphic robots with fluent speech, facial expression still struggles to follow the conversation. Motivated by this gap, we present Emo-H2A (hidden-to-actuation), a paradigm that drives a facial robot directly from LLM hidden states during autoregressive decoding. Without a separate emotion label, expression stays continuous and context-sensitive, and the face moves with the unfolding reply. We instantiate this paradigm as EmoManifold, a training-free method that extracts a valence-arousal-dominance (VAD) coordinate from each hidden state, projects it, and drives the face at token granularity. The readout is checked on six language models, and the same projection drives our 29-degree-of-freedom (DoF) facial robot. When driven by this projection, the face shifts its expression in the matching direction on 87.5% of within-reply valence transitions, whereas holding a single label for the whole reply leaves directional tracking near the 50% chance level, and first moves in 0.96s rather than waiting 10-26s for the reply to finish and a turn-level label to return.
Figures & tables
Fig. 1: Emo-H2A. (a) Schematic of token-level expression. As each token is generated, continuous valence and arousal coordinates are read out (bottom), driving the facial robot to transition smoothly across nuanced expressions (shown at selected tokens). (b) Overview. Existing pipelines (above) pass the reply through a category label and an expression generator to play one preset [ 1 , 2 ] . EmoManifold (below) drives the face directly from the token stream as the reply is generated.
Fig. 2: EmoManifold. Online decoding (bottom): user input is tokenized and passed through the LLM decoder; the final-layer, last-position hidden state ht is extracted and unembedded to obtain the next token. Online expression driving (top): ht is projected onto VAD; its valence–arousal (VA) components are mapped by a Fuzzy Kohonen Clustering Network (FKCN) into seven-dimensional expression weights, fused with presets to drive the facial actuators. Offline calibration (right): NRC-VAD pole words and the unembedding matrix Wout construct the affective axes once, offline and training-free.
Fig. 3: The 29-DoF facial robot platform used for the physical evaluation. (a) Unskinned mechanical skeleton: servo groups span the eyelids, brows, and mouth. (b) Full appearance with simulated silicone skin—the configuration driven and recorded in the experiments.
Fig. 4: Token-level affective physical actuation ( n=20 replies). Condition definitions are in Section IV-B . (a) Directional agreement between commanded affective transitions and the camera readout. (b) R2 gain, in percentage points, beyond a turn-level expression. (c) EmoManifold on the representative anger trial: command (blue) and calibrated camera (amber), z -scored valence.
System
Per-token (ms)
First update (s) GPT-4o-mini
First update (s) Qwen2.5-0.5B
EmoManifold
0.7±0.2
0.96±0.19
Mishra et al. [ 1 ]
—
10.2±4.3
9.9±4.0
Xpress-Turn [ 2 ]
—
9.9±3.8
26.5±5.7
Xpress-Story [ 2 ]
—
19.5±6.1
93.5±39.6
TABLE I: On-device latency (mean ± std., n=20 ).
EmoManifold
Xpress- Story
Xpress- Turn
Mishra et al.
Source
Hidden
Script
Label
Label
Expression
Continuous
Discrete
Discrete
Discrete
Update
Token
Keyframe
Turn
Turn
Latency
0.96 s
93.5 s
26.5 s
9.9 s
TABLE II: Comparison of facial robot interaction systems.
Fig. 5: Word-level Spearman correlation as a function of the number of pole words N . Solid lines show the mean across six models; shaded bands show the min–max range. All three dimensions saturate beyond N≈50 .
Fig. 6: Cross-lingual affective geometry ( ∣cos∣ , cohesion, and the random baseline are defined in the text). (a) Schematic of cross-lingual emotion word clusters (illustrative, not measured): solid lines are the three mined affective axes; dashed lines are an idealized orthogonal basis. (b) Measured cohesion after each backbone builds its axes from English pole words ( N=50 ). Parenthesized numbers are that model’s random baseline.
Fig. 7: Contextual-triplet sarcasm stress test ( n=200 , Qwen2.5-0.5B; triplet construction in the text). (a) Sentence-level projected valence V for S (sarcastic), P (sincere-positive), and N (literal-negative). Raincloud plots combine a half-violin (kernel density), a box plot (interquartile range, IQR), and per-triplet scatter points; diamonds are bootstrap means ± 95% CI, gray lines connect matched triplets. (b) Token-level valence at the primary anchored positive keyword, S versus P ( n=194 ).