Listen Before You Speak: Response Planning from Listener Facial Reactions for Conversational Speech Generation

Yunji Chu

cs.AISep 18, 2026

Listen Before You Speak: Response Planning from Listener Facial Reactions for Conversational Speech Generation

Authors: Yunji Chu

Organizations: Department of Artificial Intelligence, Sogang University, Seoul, Republic of Korea

Abstract

Conversational speech depends on dialogue context and the listener's immediately preceding behavior. We propose ReACT-TTS, a two-stage framework that uses a one-second pre-response listener facial sequence to plan the next utterance's emotion and prosody before speech realization. On a strict dyadic MELD protocol, Temporal conditioning yields higher mean macro-F1 and VAD concordance than Text-only across ten seeds, while accuracy remains essentially unchanged. Ablations show that temporal modeling performs best among the visual variants and that an explicit early-to-late difference is unnecessary; correct listener reactions also outperform cyclic mismatches on average. In a contextual-appropriateness study with 20 speech researchers, 76% of judgments prefer Temporal, 9% Text-only, and 15% report no preference. We further connect the predicted response style to a Grad-TTS backbone for end-to-end speech realization. Overall, the results support pre-response listener dynamics as complementary cues for conversational response planning. The source code is available at https://github.com/CYJ1/ReACT-TTS_public.

Figures & tables

Figure 1: Overall architecture of ReACT-TTS. Dialogue context and target text are combined with the listener facial reaction from the 1-s pre-response window to predict a 256-dimensional response-style representation. During speech realization, a train-split speaker-reference waveform is encoded with Resemblyzer [ 19 ] , while the predicted style is mapped by the ResponseStyleAdapter; the two 256-dimensional vectors are concatenated to condition Grad-TTS, and HiFi-GAN converts the generated 128-bin mel-spectrogram to waveform speech.

Split	Response planning	Speech generation
Train	1,117	–
Dev	116	115
Test	261	252

Table 1: Final MELD protocol. Response-planning samples satisfy the same strict filtering criteria in all splits. Speech-generation evaluation additionally requires a target speaker with a training-split reference.

Figure 2: Listener-reaction timing and representation variants. (a) Visual evidence is restricted to the 1-s interval immediately before target onset and retained only after the strict dyadic/track-validity filter. (b) The controlled ablation changes only the visual representation supplied to the same downstream gated planner: a static representation without temporal encoding, the full temporal representation, or the temporal representation augmented with an explicit early-to-late difference.

Method	Accuracy $\uparrow$	Macro-F1 $\uparrow$	CCC $\uparrow$
Text-only	$0.4521\!\pm\!0.0316$	$0.2489\!\pm\!0.0152$	$0.2173\!\pm\!0.0463$
Temporal listener	$0.4483\!\pm\!0.0243$	$\mathbf{0.2582\!\pm\!0.0131}$	$\mathbf{0.2282\!\pm\!0.0291}$

Table 2: Stage-A response-planning results on the 261-sample test set over ten seeds (42–51), reported as mean

\pm

standard deviation.

Configuration	Macro-F1 $\uparrow$
Text-only	$0.2367\pm 0.0021$
Static face	$0.2429\pm 0.0272$
Temporal + explicit $\Delta$	$0.2407\pm 0.0191$
Temporal, no $\Delta$	$\mathbf{0.2558\pm 0.0135}$

Table 3: Five-seed fair ablation (seeds 42–46). The temporal model without an explicit difference feature performs best in Macro-F1.

Listener input	Macro-F1 $\uparrow$
Mismatched listener	$0.2526\pm 0.0117$
Correct listener	$\mathbf{0.2582\pm 0.0131}$

Table 4: Correct-versus-mismatched listener intervention over ten seeds. Mismatched reactions are produced by within-batch cyclic reassignment while text context and target response remain fixed.

Condition	WER $\downarrow$	CER $\downarrow$	Spk. Sim. $\uparrow$
Text-only	$\mathbf{1.0320}$	$\mathbf{0.8824}$	$0.5976$
Temporal listener	$1.0384$	$0.8967$	$\mathbf{0.6059}$

Table 5: Objective speech-generation evaluation on 252 test utterances at length scale 1.5, selected on the development split. Lower WER/CER is better; higher speaker similarity is better.

Controllable Speech Generation

Controllable speech generation steers synthesized speech's voice, prosody, emotion, timing, pronunciation, and conversational delivery.

See all 117 papers

REALM: A Coarse-to-Fine Generative Framework for Embodied Reactive Listening
Sep 27, 2026Peizhen Li, Longbing Cao, Yang ZhangAudio-Driven Facial Animation Facial Expression Synthesis
Macquarie University, Australia · University of North Texas, USA
GLARE: Generating Listening Heads with Appropriate Reactions
Sep 30, 2026Zikai Liao, Yumin Suh, Yi Ouyang +3Talking Head Generation
Department of Computer Science, Stony Brook University · Atmanity Inc.
ReactMotion: Generating Reactive Listener Motions from Speaker Utterance
Mar 16, 2026Cheng Luo, Bizhu Wu, Bing Li +5Human Motion Generation
King Abdullah University of Science and Technology · Computer Vision Institute, School of Artificial Intelligence, Shenzhen University · School of Computer Science, University of Nottingham Ningbo China +2

Sep 27, 2026J/K move · Enter open · S save

REALM: A Coarse-to-Fine Generative Framework for Embodied Reactive Listening

Peizhen Li, Longbing Cao, Yang Zhang

Macquarie University, Australia · University of North Texas, USA

Audio-Driven Facial Animation Facial Expression Synthesis Embodied AI Human Motion Generation

Open paper AskPDF arXiv

Generating responsive listener facial motion is an important task for embodied conversational AI. Two modeling challenges are central: accounting for the timing of speaker cues while maintaining continuity with the listener's ongoing motion, and capturing locally variable facial events alongside the overall motion trajectory. Listener responses may follow preceding cues with a temporal lag, while brief expressions and blinks introduce variation that is difficult to predict deterministically. These challenges motivate a framework that combines history-aware temporal alignment with stochastic expression refinement. We propose REALM (Reactive Embodied Audio-driven Listening Model), a coarse-to-fine framework for audio-driven reactive listening. A Reactive Gated Speaker-Listener Fusion module combines listener motion history with speaker audio through a delay-centered attention prior and adaptive gating. A coarse decoder predicts a base motion trajectory, which is augmented by audio-conditioned stochastic residuals in the expression subspace while retaining the coarse pose parameters. Evaluations on ViCo and L2L show improvements over the evaluated baselines across multiple motion-quality metrics. Additional analyses examine delay sensitivity, gate behavior, and blink dynamics. Finally, deployment on an Ameca humanoid robot and a perceptual user study demonstrate the applicability of the generated behavior to physical embodiment. Code: https://github.com/lipzh5/REALM Demo: https://youtu.be/Tf5mpd5S8VQ

Abstract

Controllable Speech Generation

Explore similar work

REALM: A Coarse-to-Fine Generative Framework for Embodied Reactive Listening

GLARE: Generating Listening Heads with Appropriate Reactions

ReactMotion: Generating Reactive Listener Motions from Speaker Utterance