ReactMotion: Generating Reactive Listener Motions from Speaker Utterance

Cheng Luo; Bizhu Wu; Bing Li; Jianfeng Ren; Ruibin Bai; Rong Qu; Linlin Shen; Bernard Ghanem

cs.CVMar 16, 2026

ReactMotion: Generating Reactive Listener Motions from Speaker Utterance

Authors: Cheng Luo, Bizhu Wu, Bing Li, Jianfeng Ren, Ruibin Bai, Rong Qu, Linlin Shen, Bernard Ghanem

Organizations: King Abdullah University of Science and Technology · Computer Vision Institute, School of Artificial Intelligence, Shenzhen University · School of Computer Science, University of Nottingham Ningbo China · School of Computer Science, University of Nottingham, United Kingdom ·

Abstract

In this paper, we introduce a new task, Reactive Listener Motion Generation from Speaker Utterance, which aims to generate naturalistic listener body motions that appropriately respond to a speaker's utterance. However, modeling such nonverbal listener behaviors remains underexplored and challenging due to the inherently non-deterministic nature of human reactions. To facilitate this task, we present ReactMotionNet, a large-scale dataset that pairs speaker utterances with multiple candidate listener motions annotated with varying degrees of appropriateness. This dataset design explicitly captures the one-to-many nature of listener behavior and provides supervision beyond a single ground-truth motion. Building on this dataset design, we develop preference-oriented evaluation protocols tailored to evaluate reactive appropriateness, where conventional motion metrics focusing on input-motion alignment ignore. We further propose ReactMotion, a unified generative framework that jointly models text, audio, emotion, and motion, and is trained with preference-based objectives to encourage both appropriate and diverse listener responses. Extensive experiments show that ReactMotion outperforms retrieval baselines and cascaded LLM-based pipelines, generating more natural, diverse, and appropriate listener motions.

Figures & tables

Figure 1: Illustration of the proposed new task: Reactive Listener Motion Generation from Speech Utterance. Given a speaker’s utterance, i.e . , transcript and/or audio (optionally supplemented with emotion), a generative model such as our ReactMotion generates a corresponding responsive body-motion sequence for the listener.

Figure 2: ReactMotionNet dataset construction. We curate dyadic listener motions (Step 1), synthesize speaker conditions via inverse inference and Text-to-Speech (TTS) (Step 2), filter unreliable samples (Step 3), and rank/re-tier speaker–listener pairs into gold/silver/negative preferences (Step 4).

Split	#Pairs	Speaker Utterance			Listener Reaction	#Motion/Utter.	Labels (y)
Split	#Pairs	#Trans.	#Audio	#Emo.	#Motion	(avg.)	(# $\mathcal{G}$ /# $\mathcal{S}$ /# $\mathcal{N}$ )
Train	137,879	6,631	6,631	46	1,822	20.79	7,527 / 30,862 / 99,490
Val	6,790	841	841	40	195	8.07	903 / 1,682 / 4,205
Test	6,659	826	826	39	197	8.06	877 / 1,652 / 4,130
All	151,328	8,298	8,298	47	2,029	18.24	9,307 / 34,196 / 107,825

Table 1: Dataset statistics. #Pairs is the total number of labeled speaker–listener pairs ( i.e. candidate reactions). #Trans., #Audio, and #Emo. denote the numbers of unique transcripts, audio files, and emotion categories, respectively. #Motion is the number of unique motion sequences. #Motion/Utter. reports the average number of candidate motions per speaker utterance. Label counts report the numbers of gold/silver/negative candidates (#

\mathcal{G}

\mathcal{S}

\mathcal{N}

Figure 3: Overview of the ReactMotion framework . We use modality-specific tokenizers to convert raw data, i.e. , the speaker’s utterances (including transcript, audio, and emotion) and the listener’s reactive motions, into discrete special tokens. With these tokenizers, a Seq2Seq model is employed to integrate information across modalities and learns to generate the listener’s reactive motions from the speaker’s utterances.

Mode	Win(G $>$ N) $\uparrow$	Win(G $>$ S) $\uparrow$	Win(S $>$ N) $\uparrow$	MRR(G) $\uparrow$	nDCG@3 $\uparrow$	nDCG@5 $\uparrow$	nDCG@10 $\uparrow$
T	0.992	0.873	0.983	0.829	0.864	0.878	0.932
A	0.992	0.872	0.983	0.832	0.866	0.878	0.933
T+E	0.993	0.876	0.982	0.826	0.857	0.876	0.929
A+E	0.992	0.874	0.983	0.831	0.865	0.878	0.933
T+A	0.993	0.879	0.982	0.820	0.855	0.875	0.928
T+A+E	0.993	0.878	0.982	0.828	0.859	0.878	0.930

Table 2: Multi-modal judge network reliability under strict modality missingness (Strict-L2). We evaluate six input modes (text

T

, audio

A

, emotion

E

, and their fusions) on the test set , reporting pairwise win rates (Win(G>N), Win(G>S), Win(S>N)) and ranking metrics (MRR(G), nDCG@K) with graded relevance G>S>N.

Method	Input Mod.	Win(g $>$ N) $\uparrow$	Win(g $>$ S) $\uparrow$	Win(g $>$ G) $\uparrow$	Gen@3 $\uparrow$	FID $\downarrow$	Diversity $\uparrow$
GT	-	-	-	-	-	0.278	6.187
Random Selection	-	0.265	0.122	0.006	0.099	42.363	9.880
Retrieval	$T$	0.392	0.252	0.130	0.206	7.429	8.207
LLM $\rightarrow$ T2M-GPT	$T{+}E$	0.138	0.038	0.016	0.199	49.920	4.946
LLM $\rightarrow$ T2M-GPT *	$T{+}E$	0.171	0.027	0.017	0.350	42.589	6.102
LLM $\rightarrow$ MG-MotionLLM	$T{+}E$	0.775	0.245	0.044	0.345	23.629	5.082

Table 3: Quantitative results on the test set. Main evaluation metrics are Win(g

>

N), Win(g

>

S), Win(g

>

G), Gen@3 measuring Reactive Appropriateness. We additionally evaluate motion quality (FID) and diversity. ∗ indicates that the LLM is fine-tuned using training-set speaker utterance and listener motion caption pairs.

Figure 4: Qualitative results. We compare gold and silver listener reactions, motions generated by our ReactMotion (Ours), a cross-entropy trained variant (CE), and a cascaded LLM $\rightarrow$ T2M baseline, all conditioned on the same speaker utterance. We visualize the resulting 3D motion sequences.

Figure 5: User study on reactive appropriateness.

Method	Win(g $>$ N) $\uparrow$	Win(g $>$ S) $\uparrow$	Win(g $>$ G) $\uparrow$	Gen@3 $\uparrow$	FID $\downarrow$	Diversity $\uparrow$
CE baseline	0.990	0.741	0.262	0.938	6.555	5.448
Ours (full)	1.000	0.797	0.266	0.960	4.760	4.804
w/o Inverse-frequency reweighting	0.979	0.704	0.220	0.946	5.177	4.929
w/o $\mathcal{L}_{\text{rank}}$	0.996	0.781	0.260	0.960	5.950	5.453
w/o $\ell_{\mathcal{G}}$	0.996	0.712	0.215	0.943	6.376	4.493

Table 4: Ablation studies on the test split (all use

A{+}T{+}E

unless noted). w/o denotes training without the corresponding component. The CE baseline trains the same model using only a cross-entropy loss by pairing each speaker input with a single Gold reaction as supervision.

Setup	Value
Seq2Seq backbone model	T5-base [ 65 ]
Text tokenizer	T5-base tokenizer [ 65 ]
Audio tokenizer	MiMi neural audio codec [ 15 ]
Motion tokenizer	VQ-VAE from T2M-GPT [ 96 ]
Per-device batch size	8
Gradient accumulation steps	2

Table 5: Implementation details and hyperparameters used in training.

Metric	Value
Backbone parameters	222.9M
Total trainable parameters	235.9M
Unified vocabulary size	49,002

Table 6: Model Configuration and Parameters of ReactMotion.

Figure 6: Architecture of the multimodal judge network. Given a speaker utterance and a candidate listener motion, the judge encodes transcript, MiMi audio tokens, and the discrete emotion label with three modality-specific branches, producing modality embeddings $z_{t}$ , $z_{a}$ , and $z_{e}$ , as well as hidden summaries used to form fusion tokens $u_{t}$ , $u_{a}$ , and $u_{e}$ . Modality-type embeddings and a mode embedding are added to these tokens, which are then processed by a fusion transformer and attention pooling to obtain the unified condition embedding $z_{f}$ . In parallel, the candidate listener motion, represented by VQ-VAE motion tokens, is encoded by a motion transformer and pooled into a motion embedding $z_{m}$ . The judge computes compatibility between the condition and motion embeddings in a shared normalized scoring space. During training, a group-wise InfoNCE objective is applied to the fused embedding and auxiliary modality-specific embeddings, enabling reliable scoring under both full and partial speaker utterances. Snowflake and flame icons denote frozen and trainable modules, respectively.

Parameter	Value
Backbone encoder	T5-base
Hidden dimension $d$	768
Embedding dimension	512
Transformer heads	12
Transformer layers	6
Feedforward dimension	3072

Table 7: Hyperparameters for the multimodal judge network.

Mode	Split	Win(G $>$ N) $\uparrow$	Win(G $>$ S) $\uparrow$	Win(S $>$ N) $\uparrow$	MRR(G) $\uparrow$	nDCG@3 $\uparrow$	nDCG@5 $\uparrow$	nDCG@10 $\uparrow$
Val
T	Val	0.990	0.873	0.985	0.839	0.878	0.891	0.939
A	Val	0.990	0.873	0.985	0.842	0.881	0.893	0.940
T+A	Val	0.993	0.883	0.988	0.840	0.875	0.890	0.937
T+E	Val	0.994	0.881	0.988	0.841	0.875	0.891	0.938
A+E	Val	0.990	0.875	0.985	0.840	0.878	0.892	0.939

Table 8: We evaluate the multi-modal matching judge on validation and test set across six input modes (text

T

, audio

A

, emotion

E

, and their fusions). We report pairwise win rates based on mean score comparisons (Win(G>N), Win(G>S), Win(S>N)) and ranking metrics (MRR(G), nDCG@K with graded relevance G>S>N), where

\text{G}{=}\mathcal{G}

(Gold),

\text{S}{=}\mathcal{S}

(Silver), and

\text{N}{=}\mathcal{N}

(Negative).

$\lambda_{\mathrm{rank}}$	$\lambda_{\mathrm{gn}}$	Win(g $>$ N) $\uparrow$	Win(g $>$ S) $\uparrow$	Win(g $>$ G) $\uparrow$	Gen@3 $\uparrow$	FID $\downarrow$	Diversity $\uparrow$
0.00	0.00	0.9976	0.7809	0.2585	0.9600	5.2638	5.3005
0.00	0.25	0.9976	0.7809	0.2633	0.9600	5.2638	5.3005
0.00	0.50	0.9976	0.7809	0.2615	0.9600	5.2638	5.3005
0.00	1.00	0.9964	0.7809	0.2615	0.9613	5.2638	5.3005
0.25	0.00	0.9988	0.7809	0.2331	0.9467	5.9644	4.6993
0.25	0.25	0.9952	0.7482	0.2240	0.9467	5.2102	4.8197

Table 9: Full hyperparameter sweep results for group-wise preference training. We vary the ranking margin

m

, ranking-loss weight

\lambda_{\mathrm{rank}}

, and Gold-vs-Negative weight

\lambda_{\mathrm{gn}}

. We report pairwise preference metrics (Win(g

>

N), Win(g

>

S), Win(g

>

G)), together with Gen@3, FID, and Diversity.

Panel of Figure 7 — Figure 7: Emotion distributions over the full dataset and across the train/validation/test splits.

Panel of Figure 8 — Figure 8: Hyperparameter sensitivity heatmaps under different ranking margins. We show Gen@3, Win(g $>$ S), and FID as functions of $\lambda_{\mathrm{rank}}$ and $\lambda_{\mathrm{gn}}$ .

Config	$m$	$\lambda_{\mathrm{rank}}$	$\lambda_{\mathrm{gn}}$	Win(g $>$ N) $\uparrow$	Win(g $>$ S) $\uparrow$	Win(g $>$ G) $\uparrow$	Gen@3 $\uparrow$	FID $\downarrow$	Diversity $\uparrow$
C1	2.00	0.25	1.00	0.9939	0.7748	0.2785	0.9697	5.7083	4.1561
C2	0.50	0.50	0.00	0.9964	0.8287	0.3057	0.9625	5.8396	4.1884
C3	0.00	0.50	0.00	0.9927	0.7760	0.2548	0.9443	4.6552	4.7315
C4	1.00	0.00	0.25	0.9976	0.7809	0.2609	0.9588	5.2638	5.3005

Table 10: Representative hyperparameter configurations selected from the full sweep. We emphasize Gen@3, Win(g

>

S), and Win(g

>

G), together with FID and Diversity.

Metric	Value
Token generation speed	63.6 tokens/s
Motion generation speed	1.74 turns/s
End-to-end generation speed	1.66 turns/s
Average latency per sample	$\sim$ 0.60 s
VQ-VAE decoding speed	39.12 turns/s

Table 11: Inference Efficiency on a single NVIDIA A100 (80GB).

Human Motion Generation

Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.

See all 284 papers

REALM: A Coarse-to-Fine Generative Framework for Embodied Reactive Listening
Sep 27, 2026Peizhen Li, Longbing Cao, Yang ZhangAudio-Driven Facial Animation Facial Expression Synthesis
Macquarie University, Australia · University of North Texas, USA
GLARE: Generating Listening Heads with Appropriate Reactions
Sep 30, 2026Zikai Liao, Yumin Suh, Yi Ouyang +3Talking Head Generation
Department of Computer Science, Stony Brook University · Atmanity Inc.
MoSAT: Human Motion Generation from Spatial Audio and Textual Description
Sep 20, 2026Shuyang Xu, Zhiyang Dou, Yiduo Hao +8Text-to-Motion Generation Human Motion Generation
The University of Hong Kong · Massachusetts Institute of Technology · University of Pennsylvania +5

Sep 27, 2026J/K move · Enter open · S save

REALM: A Coarse-to-Fine Generative Framework for Embodied Reactive Listening

Peizhen Li, Longbing Cao, Yang Zhang

Macquarie University, Australia · University of North Texas, USA

Audio-Driven Facial Animation Facial Expression Synthesis Embodied AI Human Motion Generation

Open paper AskPDF arXiv

Generating responsive listener facial motion is an important task for embodied conversational AI. Two modeling challenges are central: accounting for the timing of speaker cues while maintaining continuity with the listener's ongoing motion, and capturing locally variable facial events alongside the overall motion trajectory. Listener responses may follow preceding cues with a temporal lag, while brief expressions and blinks introduce variation that is difficult to predict deterministically. These challenges motivate a framework that combines history-aware temporal alignment with stochastic expression refinement. We propose REALM (Reactive Embodied Audio-driven Listening Model), a coarse-to-fine framework for audio-driven reactive listening. A Reactive Gated Speaker-Listener Fusion module combines listener motion history with speaker audio through a delay-centered attention prior and adaptive gating. A coarse decoder predicts a base motion trajectory, which is augmented by audio-conditioned stochastic residuals in the expression subspace while retaining the coarse pose parameters. Evaluations on ViCo and L2L show improvements over the evaluated baselines across multiple motion-quality metrics. Additional analyses examine delay sensitivity, gate behavior, and blink dynamics. Finally, deployment on an Ameca humanoid robot and a perceptual user study demonstrate the applicability of the generated behavior to physical embodiment. Code: https://github.com/lipzh5/REALM Demo: https://youtu.be/Tf5mpd5S8VQ

Abstract

Human Motion Generation

Explore similar work

REALM: A Coarse-to-Fine Generative Framework for Embodied Reactive Listening

GLARE: Generating Listening Heads with Appropriate Reactions

MoSAT: Human Motion Generation from Spatial Audio and Textual Description