ReMoGen: Real-time Human Interaction-to-Reaction Generation via Modular Learning from Diverse Data
Authors: Yaoqin Ye, Yiteng Xu, Qin Sun, Xinge Zhu, Yujing Sun, Yuexin Ma
Organizations: ShanghaiTech University · Guangzhou Institute of Energy Conversion, CAS · University of Science and Technology of China · The Chinese University of Hong Kong · Nanyang Technological University
Human behaviors in real-world environments are inherently interactive, with an individual's motion shaped by surrounding agents and the scene. Such capabilities are essential for applications in virtual avatars, interactive animation, and human-robot collaboration. We target real-time human interaction-to-reaction generation, which generates the ego's future motion from dynamic multi-source cues, including others' actions, scene geometry, and optional high-level semantic inputs. This task is fundamentally challenging due to (i) limited and fragmented interaction data distributed across heterogeneous single-person, human-human, and human-scene domains, and (ii) the need to produce low-latency yet high-fidelity motion responses during continuous online interaction. To address these challenges, we propose ReMoGen (Reaction Motion Generation), a modular learning framework for real-time interaction-to-reaction generation. ReMoGen leverages a universal motion prior learned from large-scale single-person motion datasets and adapts it to target interaction domains through independently trained Meta-Interaction modules, enabling robust generalization under data-scarce and heterogeneous supervision. To support responsive online interaction, ReMoGen performs segment-level generation together with a lightweight Frame-wise Segment Refinement module that incorporates newly observed cues at the frame level, improving both responsiveness and temporal coherence without expensive full-sequence inference. Extensive experiments across human-human, human-scene, and mixed-modality interaction settings show that ReMoGen produces high-quality, coherent, and responsive reactions, while generalizing effectively across diverse interaction scenarios.
Figures & tables
Figure 2: Overview of the ReMoGen Framework . Our framework is designed to address the challenges of data scarcity and real-time responsiveness in interaction-to-reaction generation.
Figure 3: Qualitative results of ReMoGen across Human–Human, Human–Scene, and mixed Human–Human–Scene scenarios. Blue meshes denote the generated ego motion, while red meshes represent the observed motions of others. The examples cover diverse interaction behaviors, including Taichi-style movements, chasing, chatting, and scene-aware interaction, demonstrating the versatility of ReMoGen across heterogeneous interaction settings.
Method
FID ↓
R-Precision (Top-3) ↑
MM Dist. ↓
Diversity →
Peak Jerk ↓
Collision(%) ↓
Latency (s/frame) ↓
GT
0.000
0.472
4.051
4.084
0.343
–
–
ReGenNet [ 42 ]
11.622
0.269
6.092
3.452
2.321
0.665
0.210
FreeMotion [ 8 ]
3.383
0.284
5.438
3.394
2.207
0.545
0.221
FreeMotionoff [ 8 ]
0.492
0.417
4.330
3.913
0.304
1.220
–
SymBridge [ 2 ]
2.569
0.355
4.955
3.598
0.212
0.646
0.040
Ours
0.181
0.464
4.076
3.911
0.222
0.989
0.042
Table 1: Human–Human Interaction results on Inter-X. Bold indicates the best and underlined indicates the second-best. Reported latency is end-to-end inference time. Green latency values meet the 0.1 s/frame real-time threshold (equivalent to 10 FPS), while red values do not.
Method
FID ↓
R-Precision (Top-3) ↑
MM Dist. ↓
Diversity →
Peak Jerk ↓
Collision(%) ↓
Latency (s/frame) ↓
GT
0.000
0.609
1.954
7.678
0.161
–
–
TRUMANS [ 17 ]
4.731
0.178
10.822
6.543
0.279
10.754
0.074
LINGO [ 16 ]
3.633
0.218
9.597
5.503
0.252
10.610
0.189
Ours
1.201
0.530
3.408
7.421
0.169
7.822
0.042
Table 2: Human-Scene Interaction results on LINGO. Bold indicates best performance. Green latency values meet the 0.1 s/frame real-time threshold. Ours outperforms both baselines across all evaluation metrics.
Type
Step
FID ↓
MM Dist. ↓
Div. →
GT
0.000
2.894
1.399
HHI
Zeroshot
18.377
8.665
3.140
HSI
Zeroshot
15.087
8.669
4.360
Compose
Zeroshot
14.740
8.333
4.557
From Scratch
500k
2.341
4.474
2.718
Use Prior
2k
2.757
4.479
2.908
Table 3: Mixed-modality results on EgoBody. Zero-shot composition provides complementary priors but reflects clear domain mismatch. Prior-initialized finetuning rapidly adapts to the target domain, outperforming scratch training within a few thousand steps.
Method
FID ↓
R-Prec. ↑
MM Dist. ↓
Div. →
GT
0.000
0.472
4.051
4.084
Prior Only
3.735
0.231
5.736
3.235
Scratch (No Prior)
0.270
0.412
4.385
3.907
Joint-Finetune
0.298
0.439
4.188
3.862
Ours
0.181
0.464
4.076
3.911
Table 4: Ablation on different ways of using the universal prior.
Train
Infer
FID ↓
R-Prec. ↑
MM Dist. ↓
Div. →
Latency ↓
GT
0.000
0.472
4.051
4.084
–
Seg.
Seg.
0.181
0.464
4.076
3.911
0.042
Slide
Slide
4.136
0.289
5.519
3.700
0.305
Seg.
Slide
1.785
0.380
4.789
3.832
0.305
Seg.
FWSR
0.166
0.462
4.076
4.003
0.047
Table 5: Ablation on Frame-wise Segment Refinement (FWSR). Seg. denotes standard segment-based autoregressive rollout, which updates once per segment. Slide regenerates a full segment at every frame and uses only the first predicted frame. The gray row shows the baseline segment rollout, and the highlighted row shows our FWSR, which performs per-frame refinement while keeping segment-level generation unchanged.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A : Architecture of Meta-Interaction Block.
Method
FID ↓
R-Precision (Top-3) ↑
MM Dist. ↓
Diversity →
Contact Precision ↑
Contact Recall ↑
Collision(%) ↓
GT
0.000
0.472
4.051
4.084
–
–
–
ReGenNet [ 42 ]
11.622
0.269
6.092
3.452
0.549
0.505
0.665
FreeMotion [ 8 ]
3.383
0.284
5.438
3.394
0.497
0.321
0.545
FreeMotionoff [ 8 ]
0.492
0.417
4.330
3.913
0.496
0.484
1.220
SymBridge [ 2 ]
2.569
0.355
4.955
3.598
0.614
0.447
0.646
Ours
0.181
0.464
4.076
3.911
0.527
0.415
0.989
Appendix
Table A : Evaluation with contact-based metrics on HHI.
Component
Time (s)
Denoising (total, 10 cond + 10 uncond)
0.29
Denoiser
0.067
Meta-Interaction Modules
0.069
per step
0.0136
Decoding
0.005
Frame-wise Segment Refinement
0.047
Appendix
Table B: Latency breakdown of ReMoGen.
Figure B : Results under no-text and shuffled-text settings. In no-text , the original intent is “Shake hands”. In shuffled-text , the original intent is “Help others up” while the input is “Standing still”.
Encoder Type
FID ↓
R-Precision ↑
MM Dist. ↓
Div. →
GT
0.000
0.472
4.051
4.084
MLP
0.236
0.464
4.109
3.890
Transformer
0.223
0.457
4.135
3.816
TCN (Ours)
0.181
0.464
4.076
3.911
Appendix
Table C : Encoder sensitivity evaluation on HHI.
Figure C : Qualitative comparisons on Human–Human Interaction tasks. For typographical reasons, we have presented the optimal offline version of FreeMotion. Our method produces smoother and more coordinated reactions aligned with the intent, whereas baselines exhibit unnatural timing, misaligned contact, or unstable body dynamics.
Figure D : Qualitative comparisons on Human-Scene Interaction tasks. All methods are evaluated with goal location provided.
Figure E : Zero-shot Human–Human–Scene Interaction results.
Figure F : Few-step finetuning on EgoBody. Initializing from our universal prior enables rapid adaptation, producing natural reactions after only a small number of updates.
Figure G : Results in diverse Human–Human–Scene settings. ReMoGen generates semantically rich and varied interaction behaviors across different scenes and activity types. We present the results of finetuning 65k steps.
Figure H : Ablation on the Universal Motion Prior. We present ”One person walks behind the other person and strikes a pose by extending both hands above the head.” Training from scratch or joint finetuning leads to unnatural or unstable reactions, while our prior-guided modular learning produces natural, coherent motion.
Figure I : Ablation on Frame-wise Segment Refinement. We present ”A person runs towards someone and passes by them.” FWSR provides fine-grained updates that improve responsiveness without sacrificing stability, outperforming both segment-only and naive slide-style inference.
Recent advances in deep learning have enabled the generation of videos from textual descriptions as well as the prediction of future sequences from input videos. Similarly, in human motion modeling, motions can be generated from text or predicted from a single person's motion sequence. However, these approaches primarily focus on single-agent motion generation. In contrast, this study addresses the problem of generating the motion of one person based on the motion of another in interaction scenarios, where the two motions are mutually dependent. We construct a dataset of paired action-reaction motion sequences extracted from boxing match videos and investigate the effectiveness of Transformer-based models for this task. Specifically, we implement and compare three models: a simple Transformer, iTransformer, and Crossformer. In addition, we introduce a person ID embedding to explicitly distinguish between individuals, enabling the model to maintain structural consistency and better capture interaction dynamics. Experimental results show that the simple Transformer can generate plausible interaction-aware motions without suffering from posture collapse, while iTransformer and Crossformer accumulate errors over time, leading to unstable motion generation. Furthermore, the proposed person ID embedding contributes to preventing structural collapse and improving motion consistency. These results highlight the importance of explicitly modeling individual identity in interaction-aware motion generation.
Masato Soga, Ryuki Takebayashi
Faculty of Systems Engineering Graduate School of Systems Engineering Wakayama University
Video-driven 3D human reaction generation aims to synthesize 3D human motion in response to the action observed in a video, playing an important role in interactive multimedia systems and embodied agents. Yet reaction motions generated by current methods often fail to match what the observed video calls for. We observe that one factor behind this failure is relational distortion in the correspondence between visual observations and reactions: videos lying close in the visual space may correspond to entirely different motions in the reaction space, which misleads the model into generating reactions inconsistent with the conditioning video. This motivates us to propose a new observatioN-reactiOn mUtual Steering (NOUS) framework that enables mutual steering between the video and motion modalities. It first performs Motion Feedback Steering (MFS), equipping the frozen pretrained video encoder with a lightweight rectification modulator and training the modulator with a relational margin loss that pulls each video embedding toward the motion prototype of its own category and away from those of other categories. In this way, the misaligned correspondence between visual observations and reactions can be calibrated. NOUS then applies Observation-Guided Refinement (OGR), which in turn exploits the rectified observations to further refine the generated reactions and enhance their quality. The results on the ViMo dataset demonstrate that NOUS improves the quality of reaction motion while incurring negligible computational overhead at inference. Also, NOUS yields consistent gains across four pretrained video encoders, showing its good compatibility.
Yuan Zhou, Luanyuan Dai, Yongzhi Li +7
College of Computing and Data Science, Nanyang Technological University, Singapore 639798 · School of Information Science and Technology, University of Science and Technology of China, Hefei 230026, China · School of Computer Science and Information Engineering (School of Artificial Intelligence), Hefei University of Technology, Hefei 230601, China +1
Human-like multimodal reaction generation is essential for natural group interactions between humans and embodied AI. However, existing approaches are limited to single-modality or speaking-only responses in dyadic interactions, making them unsuitable for realistic social scenarios. Many also overlook nonverbal cues and complex dynamics of polyadic interactions, both critical for engagement and conversational coherence. In this work, we present PolySLGen, an online framework for Polyadic multimodal Speaking and Listening reaction Generation. Given past conversation and motion from all participants, PolySLGen generates a future speaking or listening reaction for a target participant, including speech, body motion, and speaking state score. To model group interactions effectively, we propose a pose fusion module and a social cue encoder that jointly aggregate motion and social signals from the group. Extensive experiments, along with quantitative and qualitative evaluations, show that PolySLGen produces contextually appropriate and temporally coherent multi-modal reactions, outperforming several adapted and state-of-the-art baselines in motion quality, motion-speech alignment, speaking state prediction, and human-perceived realism.
Zhi-Yi Lin, Thomas Markhorst, Jouh Yeong Chew +1
Computer Vision Lab, Delft University of Technology · Honda Research Institute Japan