ReMoGen: Real-time Human Interaction-to-Reaction Generation via Modular Learning from Diverse Data

Yaoqin Ye; Yiteng Xu; Qin Sun; Xinge Zhu; Yujing Sun; Yuexin Ma

cs.CVApr 1, 2026

ReMoGen: Real-time Human Interaction-to-Reaction Generation via Modular Learning from Diverse Data

Authors: Yaoqin Ye, Yiteng Xu, Qin Sun, Xinge Zhu, Yujing Sun, Yuexin Ma

Organizations: ShanghaiTech University · Guangzhou Institute of Energy Conversion, CAS · University of Science and Technology of China · The Chinese University of Hong Kong ·

Abstract

Human behaviors in real-world environments are inherently interactive, with an individual's motion shaped by surrounding agents and the scene. Such capabilities are essential for applications in virtual avatars, interactive animation, and human-robot collaboration. We target real-time human interaction-to-reaction generation, which generates the ego's future motion from dynamic multi-source cues, including others' actions, scene geometry, and optional high-level semantic inputs. This task is fundamentally challenging due to (i) limited and fragmented interaction data distributed across heterogeneous single-person, human-human, and human-scene domains, and (ii) the need to produce low-latency yet high-fidelity motion responses during continuous online interaction. To address these challenges, we propose ReMoGen (Reaction Motion Generation), a modular learning framework for real-time interaction-to-reaction generation. ReMoGen leverages a universal motion prior learned from large-scale single-person motion datasets and adapts it to target interaction domains through independently trained Meta-Interaction modules, enabling robust generalization under data-scarce and heterogeneous supervision. To support responsive online interaction, ReMoGen performs segment-level generation together with a lightweight Frame-wise Segment Refinement module that incorporates newly observed cues at the frame level, improving both responsiveness and temporal coherence without expensive full-sequence inference. Extensive experiments across human-human, human-scene, and mixed-modality interaction settings show that ReMoGen produces high-quality, coherent, and responsive reactions, while generalizing effectively across diverse interaction scenarios.

Figures & tables

Figure 2: Overview of the ReMoGen Framework . Our framework is designed to address the challenges of data scarcity and real-time responsiveness in interaction-to-reaction generation.

Figure 3: Qualitative results of ReMoGen across Human–Human, Human–Scene, and mixed Human–Human–Scene scenarios. Blue meshes denote the generated ego motion, while red meshes represent the observed motions of others. The examples cover diverse interaction behaviors, including Taichi-style movements, chasing, chatting, and scene-aware interaction, demonstrating the versatility of ReMoGen across heterogeneous interaction settings.

Method	FID $\downarrow$	R-Precision (Top-3) $\uparrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	Peak Jerk $\downarrow$	Collision(%) $\downarrow$	Latency (s/frame) $\downarrow$
GT	0.000	0.472	4.051	4.084	0.343	–	–
ReGenNet [ 42 ]	11.622	0.269	6.092	3.452	2.321	0.665	0.210
FreeMotion [ 8 ]	3.383	0.284	5.438	3.394	2.207	0.545	0.221
$\text{FreeMotion}^{\text{off}}$ [ 8 ]	0.492	0.417	4.330	3.913	0.304	1.220	–
SymBridge [ 2 ]	2.569	0.355	4.955	3.598	0.212	0.646	0.040
Ours	0.181	0.464	4.076	3.911	0.222	0.989	0.042

Table 1: Human–Human Interaction results on Inter-X. Bold indicates the best and underlined indicates the second-best. Reported latency is end-to-end inference time. Green latency values meet the 0.1 s/frame real-time threshold (equivalent to 10 FPS), while red values do not.

Method	FID $\downarrow$	R-Precision (Top-3) $\uparrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	Peak Jerk $\downarrow$	Collision(%) $\downarrow$	Latency (s/frame) $\downarrow$
GT	0.000	0.609	1.954	7.678	0.161	–	–
TRUMANS [ 17 ]	4.731	0.178	10.822	6.543	0.279	10.754	0.074
LINGO [ 16 ]	3.633	0.218	9.597	5.503	0.252	10.610	0.189
Ours	1.201	0.530	3.408	7.421	0.169	7.822	0.042

Table 2: Human-Scene Interaction results on LINGO. Bold indicates best performance. Green latency values meet the 0.1 s/frame real-time threshold. Ours outperforms both baselines across all evaluation metrics.

Type	Step	FID $\downarrow$	MM Dist. $\downarrow$	Div. $\rightarrow$
GT		0.000	2.894	1.399
HHI	Zeroshot	18.377	8.665	3.140
HSI	Zeroshot	15.087	8.669	4.360
Compose	Zeroshot	14.740	8.333	4.557
From Scratch	500k	2.341	4.474	2.718
Use Prior	2k	2.757	4.479	2.908

Table 3: Mixed-modality results on EgoBody. Zero-shot composition provides complementary priors but reflects clear domain mismatch. Prior-initialized finetuning rapidly adapts to the target domain, outperforming scratch training within a few thousand steps.

Method	FID $\downarrow$	R-Prec. $\uparrow$	MM Dist. $\downarrow$	Div. $\rightarrow$
GT	0.000	0.472	4.051	4.084
Prior Only	3.735	0.231	5.736	3.235
Scratch (No Prior)	0.270	0.412	4.385	3.907
Joint-Finetune	0.298	0.439	4.188	3.862
Ours	0.181	0.464	4.076	3.911

Table 4: Ablation on different ways of using the universal prior.

Train	Infer	FID $\downarrow$	R-Prec. $\uparrow$	MM Dist. $\downarrow$	Div. $\rightarrow$	Latency $\downarrow$
GT		0.000	0.472	4.051	4.084	–
Seg.	Seg.	0.181	0.464	4.076	3.911	0.042
Slide	Slide	4.136	0.289	5.519	3.700	0.305
Seg.	Slide	1.785	0.380	4.789	3.832	0.305
Seg.	FWSR	0.166	0.462	4.076	4.003	0.047

Table 5: Ablation on Frame-wise Segment Refinement (FWSR). Seg. denotes standard segment-based autoregressive rollout, which updates once per segment. Slide regenerates a full segment at every frame and uses only the first predicted frame. The gray row shows the baseline segment rollout, and the highlighted row shows our FWSR, which performs per-frame refinement while keeping segment-level generation unchanged.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A : Architecture of Meta-Interaction Block.

Method	FID $\downarrow$	R-Precision (Top-3) $\uparrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	Contact Precision $\uparrow$	Contact Recall $\uparrow$	Collision(%) $\downarrow$
GT	0.000	0.472	4.051	4.084	–	–	–
ReGenNet [ 42 ]	11.622	0.269	6.092	3.452	0.549	0.505	0.665
FreeMotion [ 8 ]	3.383	0.284	5.438	3.394	0.497	0.321	0.545
$\text{FreeMotion}^{\text{off}}$ [ 8 ]	0.492	0.417	4.330	3.913	0.496	0.484	1.220
SymBridge [ 2 ]	2.569	0.355	4.955	3.598	0.614	0.447	0.646
Ours	0.181	0.464	4.076	3.911	0.527	0.415	0.989

Appendix

Table A : Evaluation with contact-based metrics on HHI.

Component	Time (s)
Denoising (total, 10 cond + 10 uncond)	0.29
Denoiser	0.067
Meta-Interaction Modules	0.069
per step	0.0136
Decoding	0.005
Frame-wise Segment Refinement	0.047

Appendix

Table B: Latency breakdown of ReMoGen.

Figure B : Results under no-text and shuffled-text settings. In no-text , the original intent is “Shake hands”. In shuffled-text , the original intent is “Help others up” while the input is “Standing still”.

Encoder Type	FID $\downarrow$	R-Precision $\uparrow$	MM Dist. $\downarrow$	Div. $\rightarrow$
GT	0.000	0.472	4.051	4.084
MLP	0.236	0.464	4.109	3.890
Transformer	0.223	0.457	4.135	3.816
TCN (Ours)	0.181	0.464	4.076	3.911

Appendix

Table C : Encoder sensitivity evaluation on HHI.

Figure C : Qualitative comparisons on Human–Human Interaction tasks. For typographical reasons, we have presented the optimal offline version of FreeMotion. Our method produces smoother and more coordinated reactions aligned with the intent, whereas baselines exhibit unnatural timing, misaligned contact, or unstable body dynamics.

Figure D : Qualitative comparisons on Human-Scene Interaction tasks. All methods are evaluated with goal location provided.

Figure E : Zero-shot Human–Human–Scene Interaction results.

Figure F : Few-step finetuning on EgoBody. Initializing from our universal prior enables rapid adaptation, producing natural reactions after only a small number of updates.

Figure G : Results in diverse Human–Human–Scene settings. ReMoGen generates semantically rich and varied interaction behaviors across different scenes and activity types. We present the results of finetuning 65k steps.

Figure H : Ablation on the Universal Motion Prior. We present ”One person walks behind the other person and strikes a pose by extending both hands above the head.” Training from scratch or joint finetuning leads to unnatural or unstable reactions, while our prior-guided modular learning produces natural, coherent motion.

Figure I : Ablation on Frame-wise Segment Refinement. We present ”A person runs towards someone and passes by them.” FWSR provides fine-grained updates that improve responsiveness without sacrificing stability, outperforming both segment-only and naive slide-style inference.

Human Motion Generation

Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.

See all 291 papers

Learning Reactive Human Motion Generation from Paired Interaction Data Using Transformer-Based Models
Apr 24, 2026Masato Soga, Ryuki TakebayashiHuman Motion Generation
Faculty of Systems Engineering Graduate School of Systems Engineering Wakayama University
NOUS: Video-Driven 3D Human Reaction Generation via Observation-Reaction Mutual Steering
Mar 20, 2026Yuan Zhou, Luanyuan Dai, Yongzhi Li +7Cross-Modal Representation Learning Human Motion Generation
College of Computing and Data Science, Nanyang Technological University, Singapore 639798 · School of Information Science and Technology, University of Science and Technology of China, Hefei 230026, China · School of Computer Science and Information Engineering (School of Artificial Intelligence), Hefei University of Technology, Hefei 230601, China +1
PolySLGen: Online Multimodal Speaking-Listening Reaction Generation in Polyadic Interaction
Apr 9, 2026Zhi-Yi Lin, Thomas Markhorst, Jouh Yeong Chew +1Multimodal Generation Human Motion Generation
Computer Vision Lab, Delft University of Technology · Honda Research Institute Japan

Apr 24, 2026J/K move · Enter open · S save

Learning Reactive Human Motion Generation from Paired Interaction Data Using Transformer-Based Models

Masato Soga, Ryuki Takebayashi

Faculty of Systems Engineering Graduate School of Systems Engineering Wakayama University

Human Motion Generation

Open paper AskPDF arXiv

Recent advances in deep learning have enabled the generation of videos from textual descriptions as well as the prediction of future sequences from input videos. Similarly, in human motion modeling, motions can be generated from text or predicted from a single person's motion sequence. However, these approaches primarily focus on single-agent motion generation. In contrast, this study addresses the problem of generating the motion of one person based on the motion of another in interaction scenarios, where the two motions are mutually dependent. We construct a dataset of paired action-reaction motion sequences extracted from boxing match videos and investigate the effectiveness of Transformer-based models for this task. Specifically, we implement and compare three models: a simple Transformer, iTransformer, and Crossformer. In addition, we introduce a person ID embedding to explicitly distinguish between individuals, enabling the model to maintain structural consistency and better capture interaction dynamics. Experimental results show that the simple Transformer can generate plausible interaction-aware motions without suffering from posture collapse, while iTransformer and Crossformer accumulate errors over time, leading to unstable motion generation. Furthermore, the proposed person ID embedding contributes to preventing structural collapse and improving motion consistency. These results highlight the importance of explicitly modeling individual identity in interaction-aware motion generation.

Abstract

Human Motion Generation

Explore similar work

Learning Reactive Human Motion Generation from Paired Interaction Data Using Transformer-Based Models

NOUS: Video-Driven 3D Human Reaction Generation via Observation-Reaction Mutual Steering

PolySLGen: Online Multimodal Speaking-Listening Reaction Generation in Polyadic Interaction