Text- and goal-conditioned human-object interaction (HOI) provides an intuitive interface for specifying high-level intents in physics-based character animation. Mapping such intents to physically plausible, contact-rich behaviors naturally calls for a generative planner and a physics-based controller, yet existing planner-controller systems either execute plans open-loop or close the loop while keeping the planner fixed, so execution feedback never reshapes what is planned. We present InfCoiL, a closed-loop framework for physics-based HOI from coordinated planner-controller learning. InfCoiL couples a morphology-aware flow-matching planner with a unified multi-morphology controller through a shared morphology interface in an autoregressive plan-and-execute loop, and alternately fine-tunes both components with reinforcement learning from the same physical rollouts. To optimize the flow-matching planner online, we introduce Semantic Flow Policy Optimization (SFPO), which stochasticizes selected denoising transitions with channel-wise semantic covariance over heterogeneous HOI features and assigns group-relative, sequence-level rewards from complete closed-loop executions. On held-out OMOMO sequences, InfCoiL outperforms open-loop hierarchical baselines in goal-reaching task success and physical tracking accuracy. Project page: https://infcoil.github.io/
Figures & tables
Figure 1: Overview of InfCoiL. (a) Closed-loop framework: the flow-matching planner generates an Nf -frame reference window, the controller executes its first Nr frames in Isaac Gym, and the realized human, object, and contact states are fed back for receding-horizon replanning. (b) Coordinated learning: controller (PPO) and planner (SFPO) training alternate to narrow the gap between planner-generated references and what the controller can execute. (c) SFPO: for each task condition, a group of plans is sampled with semantic noise injected during denoising and executed in closed loop. Their sequence rewards are normalized within the group into relative advantages that update the planner. (d) Reward design: the controller is trained on stepwise transition and task rewards, and each plan is scored by the time-averaged sequence reward. The interaction term is used only in open-loop controller training.
Figure 2: Qualitative comparison on three OMOMO sequences. The instructions, from top to bottom, are ‘kick and lift smallbox’, ‘kick and lift suitcase’, and ‘lift and rotate monitor’. Green checks and red crosses denote recorded rollout success and failure. In the small-box example, InterMimic student lifts the box but loses its grasp; this rollout is a failure.
Table 1: Task-oriented evaluation on the re-split OMOMO test set. Each sequence is evaluated three times and metrics are averaged. C and P denote controller (PPO) and planner (SFPO) fine-tuning phases: InfCoiL (C → P) uses 10k controller iterations followed by 500 planner iterations, and InfCoiL (C → P → C) adds 4k further controller iterations.
Table 2: Planner RL ablation. All methods start from the same controller-adapted checkpoint (InfCoiL w/o FT P in Table 1 ) and update the planner with the controller frozen. SFPO with λn=0.1 corresponds to InfCoiL (C → P). BC is behavior-cloning anchor and Avg. SR averages the three thresholds, where errors are in cm.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Method
SR ↑
Eh↓
Eo↓
Rprec ↑
FID ↓
Top-1
Top-2
Top-3
subset-2 with ground-truth references
InterMimic Teacher
48.21
9.58
9.04
12.50
31.25
34.38
10.36
InfCoiL Teacher
92.86
6.03
6.13
18.75
43.75
53.12
3.29
InfCoiL Teacher w/o CSI
67.86
7.01
13.34
9.38
28.12
43.75
3.98
subset-8 with ground-truth references
Appendix
Table 3: InterMimic-style reference-tracking evaluation. Rollouts are uniformly sampled from the dataset under a fixed sampling budget and reported by the best sampled rollout. SR denotes not triggering termination under the 0.5 m deviation threshold, not task-goal success.
Table 4: Coordinated learning at a matched controller budget under the closed-loop protocol of Table 1 . C and P denote controller (PPO) and planner (SFPO) fine-tuning phases, with iteration counts in parentheses. Avg. SR averages the three thresholds, and errors are in cm.
Table 5: Planner update rules under closed- and open-loop execution. All rows use the controller after 10k closed-loop iterations (C (10k) in Table 4 ), and the planner is kept fixed, fine-tuned by SFT on executed trajectories as in SimGenHOI*, or fine-tuned online by SFPO. Open-loop rows execute references generated once by the same three planners.
Variant
FS ↓
MPJPE ↓
Troot↓
Tobj↓
Oobj↓
MPJVE ↓
V-base
0.579
11.71
7.40
17.25
1.098
1.122
X-base (ours)
0.350
11.87
7.43
16.06
1.046
0.686
Appendix
Table 6: Planner prediction and loss parameterization, evaluated by open-loop autoregressive generation on the official OMOMO test split. V-base predicts v with a velocity-space loss, and X-base predicts x0 with an x -space loss.
Figure 3: Planner references (left) and controller motions (right) for floorlamp, monitor, plasticbox, smallbox, and table. Each row shows matching frames from one episode, ordered from left to right.
Figure 4: Planner references (left) and controller motions (right) for smalltable, suitcase, trashcan, woodchair, and tripod. Each row shows matching frames from one episode, ordered from left to right.
Channel group
Dim.
κ
Body joint positions
72
0.35
Hand joints
90
0.175
Human global rotations
312
0.07
Object translation
3
0.7
Object rotation
9
0.07
Contact labels
4
0.175
Appendix
Table 7: Semantic noise scales κ used by SFPO, constant within each channel group of the 490-dimensional planner frame.
Hyperparameter
Value
Group size G
16
Denoising steps K / stochastic steps S
10 / 1 (final step, τ : 0.1→0 )
Global noise level λn
0.1
Ratio clipping ϵc
0.03
Advantage clipping Amax
5
Group skipped if reward std ≤
10−6
Appendix
Table 8: SFPO hyperparameters used in all reported planner fine-tuning phases.