cs.CVNov 19, 2026

MixReact: Physics-Based Reactive Humanoid Control from Multimodal Cues via a Discrete-Continuous Mixture of Transformers

Authors: Bin Li, Xing Zhu, Jingya Wang

Organizations: ShanghaiTech University · InstAdapt

Abstract

Humanoid agents in everyday environments are expected to react to diverse cues as humans do, with semantically appropriate and physically executable behaviors. Existing cue-driven reaction generation methods support only a narrow range of cue modalities and are limited to purely kinematic generation, leaving the resulting reactions susceptible to physical artifacts. To this end, we present MixReact, the first unified framework for physics-based reactive humanoid control from text, audio, and video cues, integrating a Discrete--Continuous Mixture of Transformers (DisCo-MoT) for state prediction with a History-aware Action Transformer (HAT) for physical execution. DisCo-MoT couples a discrete masked-modeling branch with a continuous autoregressive diffusion branch via one-way mixture attention within a MoT architecture. Guidance from discrete context mitigates collapse into averaged behaviors in one-to-many reaction generation, while direct prediction in continuous state space bypasses the decoding bottleneck of discrete representations. HAT then maps the predicted next state and observed execution history to executable actions, enabling robust closed-loop control in physics-based simulation. Experiments across all three modalities demonstrate consistent improvements over the baselines in fidelity, diversity, and physical consistency, while ablations validate the complementary contributions of the components. Our code will be released to facilitate future research. Project page: https://binlee26.github.io/MixReact_Page/

Figures & tables

Explore similar work

CardsList
  1. MIND: Multi-Scale Intent Diffusion for Text-Driven Physics-Based Humanoid Control

    May 25, 2026Bin Li, Ruichi Zhang, Han Liang +4Robotic ControlHumanoid Robot Control

  2. SCRIPT: Scalable Diffusion Policy with Multi-stage Training for Language-driven Physics-based Humanoid Control

    May 21, 2026Jingyan Zhang, Han Liang, Ruichi Zhang +6RL for RoboticsHumanoid Robot Control

  3. ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs

    Sep 9, 2026Yizhan Li, Jianxin You, Mengyang Xiong +5Physical ReasoningRobotics Simulation