Text- and goal-conditioned human-object interaction (HOI) provides an intuitive interface for specifying high-level intents in physics-based character animation. Mapping such intents to physically plausible, contact-rich behaviors naturally calls for a generative planner and a physics-based controller, yet existing planner-controller systems either execute plans open-loop or close the loop while keeping the planner fixed, so execution feedback never reshapes what is planned. We present InfCoiL, a closed-loop framework for physics-based HOI from coordinated planner-controller learning. InfCoiL couples a morphology-aware flow-matching planner with a unified multi-morphology controller through a shared morphology interface in an autoregressive plan-and-execute loop, and alternately fine-tunes both components with reinforcement learning from the same physical rollouts. To optimize the flow-matching planner online, we introduce Semantic Flow Policy Optimization (SFPO), which stochasticizes selected denoising transitions with channel-wise semantic covariance over heterogeneous HOI features and assigns group-relative, sequence-level rewards from complete closed-loop executions. On held-out OMOMO sequences, InfCoiL outperforms open-loop hierarchical baselines in goal-reaching task success and physical tracking accuracy. Project page: https://infcoil.github.io/
Figures & tables
Figure 1: Overview of InfCoiL. (a) Closed-loop framework: the flow-matching planner generates an Nf -frame reference window, the controller executes its first Nr frames in Isaac Gym, and the realized human, object, and contact states are fed back for receding-horizon replanning. (b) Coordinated learning: controller (PPO) and planner (SFPO) training alternate to narrow the gap between planner-generated references and what the controller can execute. (c) SFPO: for each task condition, a group of plans is sampled with semantic noise injected during denoising and executed in closed loop. Their sequence rewards are normalized within the group into relative advantages that update the planner. (d) Reward design: the controller is trained on stepwise transition and task rewards, and each plan is scored by the time-averaged sequence reward. The interaction term is used only in open-loop controller training.
Figure 2: Qualitative comparison on three OMOMO sequences. The instructions, from top to bottom, are ‘kick and lift smallbox’, ‘kick and lift suitcase’, and ‘lift and rotate monitor’. Green checks and red crosses denote recorded rollout success and failure. In the small-box example, InterMimic student lifts the box but loses its grasp; this rollout is a failure.
Table 1: Task-oriented evaluation on the re-split OMOMO test set. Each sequence is evaluated three times and metrics are averaged. C and P denote controller (PPO) and planner (SFPO) fine-tuning phases: InfCoiL (C → P) uses 10k controller iterations followed by 500 planner iterations, and InfCoiL (C → P → C) adds 4k further controller iterations.
Table 2: Planner RL ablation. All methods start from the same controller-adapted checkpoint (InfCoiL w/o FT P in Table 1 ) and update the planner with the controller frozen. SFPO with λn=0.1 corresponds to InfCoiL (C → P). BC is behavior-cloning anchor and Avg. SR averages the three thresholds, where errors are in cm.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Method
SR ↑
Eh↓
Eo↓
Rprec ↑
FID ↓
Top-1
Top-2
Top-3
subset-2 with ground-truth references
InterMimic Teacher
48.21
9.58
9.04
12.50
31.25
34.38
10.36
InfCoiL Teacher
92.86
6.03
6.13
18.75
43.75
53.12
3.29
InfCoiL Teacher w/o CSI
67.86
7.01
13.34
9.38
28.12
43.75
3.98
subset-8 with ground-truth references
Appendix
Table 3: InterMimic-style reference-tracking evaluation. Rollouts are uniformly sampled from the dataset under a fixed sampling budget and reported by the best sampled rollout. SR denotes not triggering termination under the 0.5 m deviation threshold, not task-goal success.
Table 4: Coordinated learning at a matched controller budget under the closed-loop protocol of Table 1 . C and P denote controller (PPO) and planner (SFPO) fine-tuning phases, with iteration counts in parentheses. Avg. SR averages the three thresholds, and errors are in cm.
Table 5: Planner update rules under closed- and open-loop execution. All rows use the controller after 10k closed-loop iterations (C (10k) in Table 4 ), and the planner is kept fixed, fine-tuned by SFT on executed trajectories as in SimGenHOI*, or fine-tuned online by SFPO. Open-loop rows execute references generated once by the same three planners.
Variant
FS ↓
MPJPE ↓
Troot↓
Tobj↓
Oobj↓
MPJVE ↓
V-base
0.579
11.71
7.40
17.25
1.098
1.122
X-base (ours)
0.350
11.87
7.43
16.06
1.046
0.686
Appendix
Table 6: Planner prediction and loss parameterization, evaluated by open-loop autoregressive generation on the official OMOMO test split. V-base predicts v with a velocity-space loss, and X-base predicts x0 with an x -space loss.
Figure 3: Planner references (left) and controller motions (right) for floorlamp, monitor, plasticbox, smallbox, and table. Each row shows matching frames from one episode, ordered from left to right.
Figure 4: Planner references (left) and controller motions (right) for smalltable, suitcase, trashcan, woodchair, and tripod. Each row shows matching frames from one episode, ordered from left to right.
Channel group
Dim.
κ
Body joint positions
72
0.35
Hand joints
90
0.175
Human global rotations
312
0.07
Object translation
3
0.7
Object rotation
9
0.07
Contact labels
4
0.175
Appendix
Table 7: Semantic noise scales κ used by SFPO, constant within each channel group of the 490-dimensional planner frame.
Hyperparameter
Value
Group size G
16
Denoising steps K / stochastic steps S
10 / 1 (final step, τ : 0.1→0 )
Global noise level λn
0.1
Ratio clipping ϵc
0.03
Advantage clipping Amax
5
Group skipped if reward std ≤
10−6
Appendix
Table 8: SFPO hyperparameters used in all reported planner fine-tuning phases.
Physics-based humanoid control has achieved remarkable progress in enabling realistic and high-performing single-agent behaviors, yet extending these capabilities to cooperative human-object interaction (HOI) remains challenging. We present TeamHOI, a framework that enables a single decentralized policy to handle cooperative HOIs across any number of cooperating agents. Each agent operates using local observations while attending to other teammates through a Transformer-based policy network with teammate tokens, allowing scalable coordination across variable team sizes. To enforce motion realism while addressing the scarcity of cooperative HOI data, we further introduce a masked Adversarial Motion Prior (AMP) strategy that uses single-human reference motions while masking object-interacting body parts during training. The masked regions are then guided through task rewards to produce diverse and physically plausible cooperative behaviors. We evaluate TeamHOI on a challenging cooperative carrying task involving two to eight humanoid agents and varied object geometries. Finally, to promote stable carrying, we design a team-size- and shape-agnostic formation reward. TeamHOI achieves high success rates and demonstrates coherent cooperation across diverse configurations with a single policy.
Stefan Lionar, Gim Hee Lee
Garena · Sea AI Lab · National University of Singapore
We present LYRIC, a generative flow-matching controller for language-driven physics-based contact-rich interaction control, that enables simulated characters to perform contact-rich whole-body object interactions from a free-form language instruction and a sparse terminal object goal. To obtain reliable expert trajectories from imperfect motion-capture references, a single tracking policy is trained using geometry-conditioned interaction rewards and relaxed reference tracking near hand-object contact. To guide interaction progress without prescribing a full-body kinematic reference, we factorize the controller into a task-level planner that predicts short-horizon object and humanoid-root trajectories, and an action generator that resolves whole-body motion and contacts in closed loop. After behavior cloning, we freeze the planner and post-tune the action generator on policy using the planner's predictions as stable supervision for intermediate task progression. In a controlled OMOMO evaluation, our tracker achieves 64.3% success compared with 53.2% for an InterMimic reimplementation, while a unified policy achieves 76.5% on the full OMOMO dataset. On the held-out split, LYRIC achieves 90.3% task success, compared with 74.2% for the strongest matched kinematic-planner baseline, with better semantic alignment and motion quality. Without retraining, the controller also supports test-time object-waypoint guidance. Qualitative results further demonstrate robust, natural contact-rich interactions and zero-shot transfer to novel object shapes. The webpage is available at https://neu-vi.github.io/LYRIC/
Zeyu Han, Zichong Meng, Julian Tanke +8
Northeastern University, USA · Sony AI America Inc, USA · Sony Interactive Entertainment Inc, Japan +3
Generating physically plausible dynamic motions of human-object interaction (HOI) remains challenging, mainly due to existing HOI datasets limited to static interactions, and pretrained agents capable of either dynamic full-body motions without objects or static HOI motions. Recent works such as InsActor and CLoSD generate HOI motions in planning and execution stages, are yet limited to either static or short-term contacts e.g. striking. In this work, we propose a framework that fulfills dynamic and long-term interaction motions such as running while holding a table, by combining pretrained motion priors and imitation agents in planning and execution stages. In the planning stage, we augment HOI datasets with dynamic priors from a pretrained human motion diffusion model, followed by object trajectory generation. This plans dynamic HOI sequences. In the execution stage, a composer network blends actions of pretrained imitation agents specialized either for dynamic human motions or static HOI motions, enabling spatio-temporal composition of their complementary skills. Our method over relevant prior-arts consistently improves success rates while maintaining interaction for dynamic HOI tasks. Furthermore, blending pretrained experts with our composer achieves competitive performance in significantly reduced training time. Ablation studies validate the effectiveness of our augmentation and composer blending.