PolySLGen: Online Multimodal Speaking-Listening Reaction Generation in Polyadic Interaction

Zhi-Yi Lin; Thomas Markhorst; Jouh Yeong Chew; Xucong Zhang

cs.CVApr 9, 2026

PolySLGen: Online Multimodal Speaking-Listening Reaction Generation in Polyadic Interaction

Authors: Zhi-Yi Lin, Thomas Markhorst, Jouh Yeong Chew, Xucong Zhang

Organizations: Computer Vision Lab, Delft University of Technology · Honda Research Institute Japan

Abstract

Human-like multimodal reaction generation is essential for natural group interactions between humans and embodied AI. However, existing approaches are limited to single-modality or speaking-only responses in dyadic interactions, making them unsuitable for realistic social scenarios. Many also overlook nonverbal cues and complex dynamics of polyadic interactions, both critical for engagement and conversational coherence. In this work, we present PolySLGen, an online framework for Polyadic multimodal Speaking and Listening reaction Generation. Given past conversation and motion from all participants, PolySLGen generates a future speaking or listening reaction for a target participant, including speech, body motion, and speaking state score. To model group interactions effectively, we propose a pose fusion module and a social cue encoder that jointly aggregate motion and social signals from the group. Extensive experiments, along with quantitative and qualitative evaluations, show that PolySLGen produces contextually appropriate and temporally coherent multi-modal reactions, outperforming several adapted and state-of-the-art baselines in motion quality, motion-speech alignment, speaking state prediction, and human-perceived realism.

Figures & tables

Figure 1: Overview of the online reaction generation task in polyadic interaction. Given past group interactions, including speech and body motion of all participants, PolySLGen reasons about and generates the future reaction of the target participant (dark blue). The output includes speech, body motion, and a speaking state score (“Speak?”), which serves as an indicator of whether the generated reaction is speaking or listening.

Figure 2: Overview of PolySLGen. Past multimodal group interactions are encoded into text $\mathbf{e}^{\text{t}}$ , speech style $\mathbf{e}^{\text{s}}$ , motion $\mathbf{e}^{\text{m}}$ , and social cue $\mathbf{e}^{\text{c}}$ embeddings as the inputs. The motion embeddings are aggregated by a pose fusion module, and the social cue encoder captures signals from non-target participants’ head orientations. The model generates target participant’s future reactions in text, speech style, and motion, through modality-specific decoders from embeddings $\mathbf{h}^{\text{t}}$ , $\mathbf{h}^{\text{s}}$ , and $\mathbf{h}^{\text{m}}$ , along with a speaking state score for turn-taking.

Method	Motion					Speech			State
	Root $\downarrow$	MPJPE $\downarrow$	FID $\downarrow$	Div. $\rightarrow$	BeatAlign	BERT	WER $\downarrow$	SIM $\uparrow$	AP $\uparrow$
	(mm)	(mm)	FID $\downarrow$	Div. $\rightarrow$	Diff. $\downarrow$	Score $\uparrow$	WER $\downarrow$	SIM $\uparrow$	AP $\uparrow$
Random	140.4	200.4	17.82	120.46	0.018	0.458	1.699	0.494	0.50 †
NN cond.	134.7	187.7	16.36	105.42	0.023	0.451	2.075	0.520	0.52 †
LLM + ConvoFusion [ 47 ]	125.7	170.1	18.57	72.45	-	0.388	13.318	-	0.50 †
LM-L2L Adapted [ 50 ]	185.2	187.6	17.22	116.36	-	-	-	-	-

Table 1: Comparison of PolySLGen with baselines on motion, speech, and speaking state score. SOLAMI is evaluated with LoRA fine-tuning and without pre-training. † : Speaking state inferred from generated text. * : Speech synthesized using prompts from test set audio. For the Diversity metric, the value closer to the ground-truth (117.09) is better.

Pose Fusion	Social Cue	Motion					Speech			State
		Root $\downarrow$	MPJPE $\downarrow$	FID $\downarrow$	Div. $\rightarrow$	BeatAlign	BERT	WER $\downarrow$	SIM $\uparrow$	AP $\uparrow$
		(mm)	(mm)	FID $\downarrow$	Div. $\rightarrow$	Diff. $\downarrow$	Score $\uparrow$	WER $\downarrow$	SIM $\uparrow$	AP $\uparrow$
✗	✗	126.1	153.3	14.01	121.37	0.012	0.489	1.454	0.631	0.60
✓	✗	116.7	148.2	12.97	118.34	0.007	0.511	1.447	0.646	0.66
✗	✓	124.6	152.5	13.53	123.19	0.009	0.474	1.715	0.625	0.59
✓	✓	108.7	144.9	12.18	113.32	0.007	0.508	1.436	0.642	0.67

Table 3: Ablation study on pose fusion and social cue encoder. Diversity closer to the ground truth (117.09) is better.

Figure 3: Visual comparison between PolySLGen and SOLAMI. The first column presents speaking reactions, and the second column shows listening reactions. The top row contains input observations, followed by the ground truth and outputs from SOLAMI and PolySLGen. The predicted speaking state score is denoted by $r$ , and different colors correspond to different speakers. Note that we overlay the ground truth motion (in gray) with the generated outputs.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Transcription workflow. Utterances are extracted by pyannote-audio [ 7 ] , and then transcribed using stable-ts [ 66 ] .

Figure 5: Three examples of the conversational chunks extracted from the DnD Gesture Dataset [ 47 ] . Keywords essential for generating in-context responses are underlined.

f

indicates the frame index. The output speaker in each example is the target participant.

Panel of Figure 6 — Figure 6: Multimodal instruction template for polyadic interaction. <TEXT> denotes text embeddings, <STYLE> represents speech style embeddings, <POSE> corresponds to motion embeddings, and <SCUE> refers to social cue embeddings.

Modality Order	Motion					Speech			State
	Root $\downarrow$	MPJPE $\downarrow$	FID $\downarrow$	Div. $\rightarrow$	BeatAlign	BERT	WER $\downarrow$	SIM $\uparrow$	AP $\uparrow$
	(mm)	(mm)	FID $\downarrow$	Div. $\rightarrow$	Diff. $\downarrow$	Score $\uparrow$	WER $\downarrow$	SIM $\uparrow$	AP $\uparrow$
M $\rightarrow$ C $\rightarrow$ S $\parallel$ M $\rightarrow$ S	135.1	154.9	14.23	123.94	0.018	0.502	1.403	0.638	0.59
S $\rightarrow$ M $\rightarrow$ C $\parallel$ S $\rightarrow$ M (Ours)	108.7	144.9	12.18	113.32	0.007	0.508	1.436	0.642	0.67

Appendix

Table 10: Impact of modality ordering. Diversity closer to the ground truth (117.09) indicates better performance. Root and MPJPE are reported in millimeters (mm). Motion, social cue, and speech are denoted as M , C , and S , respectively.

	Root $\downarrow$	MPJPE $\downarrow$	FID $\downarrow$	Div. $\rightarrow$
n = 1	118.8	152.3	13.56	123.84
n = 4	113.7	149.8	13.04	120.43
n = 2 (ours)	108.7	144.9	12.18	113.32

Appendix

Table 11: Impact of social cue embedding lengths

n

. Diversity closer to the ground truth (117.09) indicates better performance. Root and MPJPE are reported in millimeters (mm).

Multimodal Generation

Multimodal generation creates coordinated text, images, audio, video, 3D, and other outputs from mixed inputs while preserving meaning and consistency.

See all 243 papers

ReMoGen: Real-time Human Interaction-to-Reaction Generation via Modular Learning from Diverse Data
Apr 1, 2026Yaoqin Ye, Yiteng Xu, Qin Sun +3Human Motion Generation
ShanghaiTech University · Guangzhou Institute of Energy Conversion, CAS · University of Science and Technology of China +2
Learn2Chat: Rethinking Dyadic Talking Heads via Interaction-Modulated Monologic Priors
Jul 11, 2026Zikai Huang, Siyue Chen, Xuemiao Xu +4Disentangled Representation Learning Human Motion Generation
School of Computer Science and Engineering, South China University of Technology, Guangdong, China · School of Design, South China University of Technology, Guangdong, China · Guangdong Engineering Center for Large Model and GenAI Technology +2
U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generation
Feb 27, 2026Xiang Deng, Feng Gao, Yong Zhang +5Multimodal Agents Unified Multimodal Models
Tsinghua University · Meituan

Apr 1, 2026J/K move · Enter open · S save

ReMoGen: Real-time Human Interaction-to-Reaction Generation via Modular Learning from Diverse Data

Yaoqin Ye, Yiteng Xu, Qin Sun, Xinge Zhu+2

ShanghaiTech University · Guangzhou Institute of Energy Conversion, CAS · University of Science and Technology of China+2

Human Motion Generation

Open paper AskPDF arXiv

Human behaviors in real-world environments are inherently interactive, with an individual's motion shaped by surrounding agents and the scene. Such capabilities are essential for applications in virtual avatars, interactive animation, and human-robot collaboration. We target real-time human interaction-to-reaction generation, which generates the ego's future motion from dynamic multi-source cues, including others' actions, scene geometry, and optional high-level semantic inputs. This task is fundamentally challenging due to (i) limited and fragmented interaction data distributed across heterogeneous single-person, human-human, and human-scene domains, and (ii) the need to produce low-latency yet high-fidelity motion responses during continuous online interaction. To address these challenges, we propose ReMoGen (Reaction Motion Generation), a modular learning framework for real-time interaction-to-reaction generation. ReMoGen leverages a universal motion prior learned from large-scale single-person motion datasets and adapts it to target interaction domains through independently trained Meta-Interaction modules, enabling robust generalization under data-scarce and heterogeneous supervision. To support responsive online interaction, ReMoGen performs segment-level generation together with a lightweight Frame-wise Segment Refinement module that incorporates newly observed cues at the frame level, improving both responsiveness and temporal coherence without expensive full-sequence inference. Extensive experiments across human-human, human-scene, and mixed-modality interaction settings show that ReMoGen produces high-quality, coherent, and responsive reactions, while generalizing effectively across diverse interaction scenarios.

Abstract

Multimodal Generation

Explore similar work

ReMoGen: Real-time Human Interaction-to-Reaction Generation via Modular Learning from Diverse Data

Learn2Chat: Rethinking Dyadic Talking Heads via Interaction-Modulated Monologic Priors

U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generation