REALM: A Coarse-to-Fine Generative Framework for Embodied Reactive Listening

Peizhen Li; Longbing Cao; Yang Zhang

cs.ROSep 27, 2026

REALM: A Coarse-to-Fine Generative Framework for Embodied Reactive Listening

Authors: Peizhen Li, Longbing Cao, Yang Zhang

Organizations: Macquarie University, Australia · University of North Texas, USA

Abstract

Generating responsive listener facial motion is an important task for embodied conversational AI. Two modeling challenges are central: accounting for the timing of speaker cues while maintaining continuity with the listener's ongoing motion, and capturing locally variable facial events alongside the overall motion trajectory. Listener responses may follow preceding cues with a temporal lag, while brief expressions and blinks introduce variation that is difficult to predict deterministically. These challenges motivate a framework that combines history-aware temporal alignment with stochastic expression refinement. We propose REALM (Reactive Embodied Audio-driven Listening Model), a coarse-to-fine framework for audio-driven reactive listening. A Reactive Gated Speaker-Listener Fusion module combines listener motion history with speaker audio through a delay-centered attention prior and adaptive gating. A coarse decoder predicts a base motion trajectory, which is augmented by audio-conditioned stochastic residuals in the expression subspace while retaining the coarse pose parameters. Evaluations on ViCo and L2L show improvements over the evaluated baselines across multiple motion-quality metrics. Additional analyses examine delay sensitivity, gate behavior, and blink dynamics. Finally, deployment on an Ameca humanoid robot and a perceptual user study demonstrate the applicability of the generated behavior to physical embodiment. Code: https://github.com/lipzh5/REALM Demo: https://youtu.be/Tf5mpd5S8VQ

Figures & tables

Figure 1: Motivation for responsive listener motion generation. (a) Response timing: Listener responses can follow preceding speaker cues with a temporal lag. (b) Behavioral continuity: Speaker activity and listener motion need not vary proportionally, motivating adaptive use of speaker evidence and listener history. (c) Motion dynamics: Overall motion trajectories can coexist with brief, locally variable facial events, motivating expression refinement in addition to coarse prediction.

Figure 2: Overview of REALM. Speaker audio and listener motion history are integrated through delay-aware reactive fusion. The resulting context determines a coarse listener trajectory, which is subsequently augmented by audio-conditioned stochastic refinement restricted to the expression subspace. The generated motion is finally retargeted to the physical robot.

Method	Test set	$\mathbf{L_{1}}$ $\downarrow$		FD $\downarrow$		FID ${}_{\text{fm}}$ $\downarrow$		FID ${}_{\Delta\text{fm}}$ $\downarrow$		rPCC $\downarrow$		var $\rightarrow$ GT
Method	Test set	exp	pose	exp	pose	exp	pose	exp	pose	exp	pose	exp	pose
RLHG	$\mathcal{D}_{test}$	14.80	8.27	0.69	0.72	3.19	0.084	9.88	0.89	0.039	0.024	0.113	0.150
RLHG	$\mathcal{D}_{ood}$	19.75	7.37	3.27	1.69	5.86	0.074	8.16	0.84	0.065	0.092	0.115	0.195
DSPN	$\mathcal{D}_{test}$	15.48	7.40	0.65	0.95	3.05	0.078	8.91	0.73	0.035	0.041	.140	0.240
DSPN	$\mathcal{D}_{ood}$	24.11	8.66	2.85	1.45	5.12	0.065	6.07	0.47	0.072	0.075	0.141	0.040
L2L	$\mathcal{D}_{test}$	20.36	10.70	0.58	1.25	3.84	0.096	12.06	0.60	0.040	0.070	0.097	0.033

Table 1: Quantitative evaluation of REALM against state-of-the-art baselines on the ViCo dataset. We report point-wise accuracy (

L_{1}

), distributional realism (FD,

\text{FID}_{\text{fm}}

, and

\text{FID}_{\Delta\text{fm}}

), speaker–listener correlation (rPCC), and motion variability (var).

L_{1}

\text{FID}_{\Delta\text{fm}}

, and pose FD are scaled by 100.

\downarrow

denotes lower is better. For var, values closer to the Ground Truth (GT) are better. Bold indicates the best result.

Method	$\mathbf{L_{1}}$ $\downarrow$		FD $\downarrow$		FID ${}_{\text{fm}}$ $\downarrow$		FID ${}_{\Delta\text{fm}}$ $\downarrow$		rPCC $\downarrow$		var $\rightarrow$ GT
Method	exp	pose	exp	pose	exp	pose	exp	pose	exp	pose	exp	pose
RLHG	17.49	4.92	63.88	0.87	3.19	0.037	11.38	0.052	0.102	0.063	0.153	0.006
DSPN	23.65	5.82	72.15	1.12	3.85	0.045	12.40	0.085	0.120	0.082	0.135	0.005
L2L	10.45	2.88	14.63	0.11	1.81	0.022	11.32	0.079	0.026	0.034	0.124	0.011
ListenFormer	10.32	2.57	17.12	0.07	1.44	0.016	13.41	0.051	0.008	0.010	0.173	0.010
UniLS	26.10	4.50	95.00	0.65	7.50	0.032	11.80	0.055	0.300	0.120	0.150	0.007

Table 2: Quantitative evaluation on the L2L dataset. Evaluation metrics, scaling factors, and formatting conventions are identical to Table 1 .

Modules			$\mathbf{L_{1}}$ $\downarrow$		FD $\downarrow$		FID ${}_{\text{fm}}$ $\downarrow$		FID ${}_{\Delta\text{fm}}$ $\downarrow$		rPCC $\downarrow$		var $\rightarrow$ GT
GF	RM	SA	exp	pose	exp	pose	exp	pose	exp	pose	exp	pose	exp	pose
✗	✗	✗	12.20	6.69	0.64	1.19	2.27	0.060	12.51	0.10	0.014	0.026	0.132	0.042
✓	✗	✗	12.04	6.78	0.63	1.03	2.28	0.060	12.50	0.10	0.019	0.016	0.132	0.043
✗	✓	✗	12.25	6.69	0.62	1.19	2.18	0.060	4.11	0.10	0.012	0.026	0.131	0.042
✗	✗	✓	11.98	6.52	0.68	1.32	2.19	0.060	12.53	0.10	0.012	0.018	0.131	0.041
✗	✓	✓	12.31	6.52	0.60	1.32	2.09	0.060	4.10	0.10	0.011	0.018	0.130	0.041

Table 3: Ablation study of REALM components on the ViCo (

\mathcal{D}_{test}

) split evaluating Gated Fusion (GF), the Refinement Module (RM), and Shifted Attention (SA). Because RM refines only non-rigid expressions (

\hat{\mathbf{r}}_{t}=\tilde{\mathbf{r}}_{t}

), pose metrics are determined entirely by the coarse stage (GF and SA). Metrics follow Table 1 .

Prior Center	$\tau$ (frames)	$L_{1}\downarrow$ (exp / pose)	FD $\downarrow$ (exp / pose)	rPCC $\downarrow$ (exp / pose)
0 ms (No Shift)	0	11.88 / 6.96	0.61 / 1.43	0.019 / 0.029
$\approx$ 133 ms	4	11.71 / 6.85	0.58 / 1.31	0.018 / 0.029
$\approx$ 267 ms (Ours)	8	11.58 / 6.52	0.56 / 1.01	0.009 / 0.014
$\approx$ 400 ms	12	11.76 / 6.82	0.57 / 1.36	0.020 / 0.045

Table 4: Sensitivity to the nominal delay

\tau

in the attention prior on ViCo at 30 FPS. An 8-frame shift performs best among the tested settings. Time values indicate the center of the temporal bias, not a prescribed response latency.

Figure 3: Qualitative comparison of physical embodiment on the Ameca robot. To illustrate the performance observed across the ViCo test set, we visualize two representative video sequences (frames $t_{1},t_{2},t_{3}$ and $t_{1}^{\prime},t_{2}^{\prime},t_{3}^{\prime}$ ) comparing the reactive listener motion generated by our method (REALM) against the human Ground Truth and the ListenFormer Liu et al. (2024a) baseline. Driven by the speaker’s context (Top Row), ListenFormer frequently suffers from deterministic over-smoothing, failing to synthesize intended expressions such as a conversational smile or a natural blink (highlighted in red ). In contrast, REALM effectively anchors the listener to their natural reactive manifold, accurately recovering contextually appropriate expressions and subtle stochastic micro-dynamics (e.g., smiling and blinking, highlighted in green ) that closely match the human Ground Truth.

Method	Nat. $\uparrow$	Syn. $\uparrow$	App. $\uparrow$	Pref. $\uparrow$
RLHG	2.2	2.4	2.4	2.2
DSPN	2.4	2.6	2.8	2.4
L2L	2.8	3.0	3.2	2.8
ListenFormer	3.6	3.8	4.0	3.6
UniLS	3.2	3.6	3.8	3.4
REALM (Ours)	4.4	4.2	4.2	4.0

Table 5: User study results on Ameca, reported as mean opinion scores on a 1–5 scale (

N=25

; five sequences per participant; six methods per sequence). Higher scores indicate better perceived quality. Bold indicates the highest mean score.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Analysis Category	Evaluation Metric	Result
Temporal Alignment ( $\mathbf{g}$ Peak vs. GT Onset)	Total Reactions	515
	Mean Offset	-0.50 frames (-16.6 ms)
	Median Offset	0.00 frames
Intra-Clip Velocity (Top 10% vs. Bottom 10% $\mathbf{g}$ )	Mean $\Delta V$	0.0051
Intra-Clip Velocity (Top 10% vs. Bottom 10% $\mathbf{g}$ )	Statistical Significance	$p<0.001$ ( $3.2\times 10^{-5}$ )

Appendix

Table 6: Inference-time quantitative analysis of the gating mechanism (

\mathbf{g}

) across the ViCo test set.

Actuator Group	Channel Name	Unit	$\mathbf{q}_{\min}$	$\mathbf{q}_{\max}$	Neutral $\mathbf{q}_{0}$	Primary Driving Blendshapes
Jaw	Jaw Pitch	Normalized	$0.00$	$1.00$	$1.00$	Jaw Open ( $c_{\text{jawOpen}}$ )
Jaw	Jaw Yaw	Normalized	$0.00$	$1.00$	$0.50$	Jaw Left/Right ( $c_{\text{jawLeft}},c_{\text{jawRight}}$ )
Brows & Eyelids	Brow Inner (L/R)	Normalized	$0.00$	$1.00$	$0.50$	Brow Down / Brow Inner Up
	Brow Outer (L/R)	Normalized	$0.00$	$1.00$	$0.50$	Brow Outer Up
	Eyelid Upper (L/R)	Normalized	$-1.00$	$2.00$	$1.00$	Eye Blink / Eye Wide
	Eyelid Lower (L/R)	Normalized	$-1.00$	$2.00$	$0.00$	Eye Squint

Appendix

Table 7: Physical robot actuator channels, control modalities, and absolute hardware limits

[\mathbf{q}_{\min},\mathbf{q}_{\max}]

enforced by the final post-calibration safety clamping stage (Eq. 23 ). Linear facial actuators operate in normalized control units, while head kinematics and gaze targets operate strictly in radians.

Method	$\Delta$ Rate (BPM) $\downarrow$	Duration Delta ( $\Delta$ ms) $\downarrow$	Avg Duration (ms)
Without refinement	10.50	74.65	366.67
Unmodulated noise	10.39	341.32	633.33
REALM (Ours)	7.40	18.45	273.56

Appendix

Table 8: Structural analysis of micro-dynamics on the ViCo dataset. We evaluate the anatomical correctness of generated eye blinks against the Ground Truth (GT) duration of 292.01 ms.

Audio-Driven Facial Animation

Audio-driven facial animation maps speech to synchronized facial motion for realistic, expressive, and controllable talking faces and avatars.

See all 83 papers

ReactMotion: Generating Reactive Listener Motions from Speaker Utterance
Mar 16, 2026Cheng Luo, Bizhu Wu, Bing Li +5Human Motion Generation
King Abdullah University of Science and Technology · Computer Vision Institute, School of Artificial Intelligence, Shenzhen University · School of Computer Science, University of Nottingham Ningbo China +2
GLARE: Generating Listening Heads with Appropriate Reactions
Sep 30, 2026Zikai Liao, Yumin Suh, Yi Ouyang +3Talking Head Generation
Department of Computer Science, Stony Brook University · Atmanity Inc.
EmbodiedHead: Real-Time Listening and Speaking Avatar for Conversational Agents
Apr 19, 2026Yu Zhang, Kaiyuan Shen, Yang LiStreaming Video Generation Talking Head Generation
School of Computer Science and Technology, East China Normal University, Shanghai, China · Garabido Shanghai Technology Co., Ltd., Shanghai, China

Mar 16, 2026J/K move · Enter open · S save

ReactMotion: Generating Reactive Listener Motions from Speaker Utterance

Cheng Luo, Bizhu Wu, Bing Li, Jianfeng Ren+4

King Abdullah University of Science and Technology · Computer Vision Institute, School of Artificial Intelligence, Shenzhen University · School of Computer Science, University of Nottingham Ningbo China+2

Human Motion Generation

Open paper AskPDF arXiv

In this paper, we introduce a new task, Reactive Listener Motion Generation from Speaker Utterance, which aims to generate naturalistic listener body motions that appropriately respond to a speaker's utterance. However, modeling such nonverbal listener behaviors remains underexplored and challenging due to the inherently non-deterministic nature of human reactions. To facilitate this task, we present ReactMotionNet, a large-scale dataset that pairs speaker utterances with multiple candidate listener motions annotated with varying degrees of appropriateness. This dataset design explicitly captures the one-to-many nature of listener behavior and provides supervision beyond a single ground-truth motion. Building on this dataset design, we develop preference-oriented evaluation protocols tailored to evaluate reactive appropriateness, where conventional motion metrics focusing on input-motion alignment ignore. We further propose ReactMotion, a unified generative framework that jointly models text, audio, emotion, and motion, and is trained with preference-based objectives to encourage both appropriate and diverse listener responses. Extensive experiments show that ReactMotion outperforms retrieval baselines and cascaded LLM-based pipelines, generating more natural, diverse, and appropriate listener motions.

Abstract

Audio-Driven Facial Animation

Explore similar work

ReactMotion: Generating Reactive Listener Motions from Speaker Utterance

GLARE: Generating Listening Heads with Appropriate Reactions

EmbodiedHead: Real-Time Listening and Speaking Avatar for Conversational Agents