cs.CVNov 8, 2026

InfCoiL: Coordinated Planner-Controller Learning for Closed-Loop Physics-Based Human-Object Interaction

Authors: Yude Zou, Yifei Yao, Yiwei Hao, Hanqing Wang, Xing Gao

Organizations: Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory

Abstract

Text- and goal-conditioned human-object interaction (HOI) provides an intuitive interface for specifying high-level intents in physics-based character animation. Mapping such intents to physically plausible, contact-rich behaviors naturally calls for a generative planner and a physics-based controller, yet existing planner-controller systems either execute plans open-loop or close the loop while keeping the planner fixed, so execution feedback never reshapes what is planned. We present InfCoiL, a closed-loop framework for physics-based HOI from coordinated planner-controller learning. InfCoiL couples a morphology-aware flow-matching planner with a unified multi-morphology controller through a shared morphology interface in an autoregressive plan-and-execute loop, and alternately fine-tunes both components with reinforcement learning from the same physical rollouts. To optimize the flow-matching planner online, we introduce Semantic Flow Policy Optimization (SFPO), which stochasticizes selected denoising transitions with channel-wise semantic covariance over heterogeneous HOI features and assigns group-relative, sequence-level rewards from complete closed-loop executions. On held-out OMOMO sequences, InfCoiL outperforms open-loop hierarchical baselines in goal-reaching task success and physical tracking accuracy. Project page: https://infcoil.github.io/

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. TeamHOI: Learning a Unified Policy for Cooperative Human-Object Interactions with Any Team Size

    Mar 9, 2026Stefan Lionar, Gim Hee LeeHuman-Object InteractionHOI Generation

  2. LYRIC: Language-Driven Physics-Based Character Control for Contact-Rich Whole-Body Object Interaction

    Sep 17, 2026Zeyu Han, Zichong Meng, Julian Tanke +8HOI GenerationHuman Motion Generation