Long-horizon human-scene interaction requires coordinated spatial reasoning, physical feasibility, and the ability to recover from compounding execution errors. Text-driven motion models are largely scene-agnostic and lack closed-loop correction, while geometry-conditioned approaches depend on paired scene-motion training or explicit 3D preprocessing that limits generalization to novel layouts and open-vocabulary goals. We present InteractAgent, an agentic multimodal LLM system for long-horizon human-scene interaction that plans, executes, observes, and reflects in a closed loop, using a single MLLM as both planner and evaluator and a frozen MotionLLM as the executor. InteractAgent integrates Online Observation for step-wise recovery decisions (continue/redo/replan) and Memory-Augmented Reflection for multi-round plan refinement with adaptive stopping. On UniHSI, InteractAgent improves task success and reduces penetration compared with training-free baselines. Code: https://github.com/AIGeeksGroup/InteractAgent.
Figures & tables
Figure 2: Overview of InteractAgent. Our framework couples a single multimodal LLM with a frozen MotionLLM executor. Two key components enable training-free, long-horizon interaction generation: (1) Online Observation provides step-wise visual feedback during execution, allowing the MLLM to correct local deviations; (2) Memory-Augmented Reflection aggregates planning–execution histories across rounds and refines future plans at the trajectory level. Together, these mechanisms form a dual-layer reflective loop that progressively improves motion quality and scene consistency.
Figure 3: Trajectory visualization image used as a compact reflection signal for the MLLM.
Methods
Success Rate (%) ↑
pe max↓
pe 10% ↓
Low
Mid
High
UniHSI-CoCo
85.1
47.7
17.7
18.2
9.3
Naive
25.5
18.2
4.4
528.4
113.6
Online
74.4
52.3
31.1
102.7
54.5
Ref + Offline
72.3
47.7
26.7
62.3
36.8
InteractAgent
84.3
79.5
64.4
48.6
29.7
Table 1: Result of methods on tasks of different complexity levels. Success Rate (SR, %) and penetration errors are reported.
LLM Type
SR (%) ↑
PC (%) ↑
Human
72
–
w. Qwen-VL-Max p
56
75
w. GPT-4o
64
81
Table 2: Comparison of SR and PC (Y/N decided by human) across different LLM types.
Motion Executor
SR (%)↑
pe max ↓
pe 10% ↓
MotionLLM
70
48.6
29.7
MG-MotionLLM
72
23.3
16.9
MG stateless
70
46.2
24.3
Table 3: Comparison of SR and penetration between Motion Executors.