cs.ROOct 10, 2026

Chunk-wise Model-based Policy Optimization

Authors: Weixi Song, Donglin Wang, Yong-Lu Li

Organizations: Zhejiang University · Shanghai Innovation Institute · Westlake University · Shanghai Jiao Tong University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Model-based reinforcement learning has shown superior sample efficiency by leveraging the learned latent dynamics either in the form of derivative-free planning/imagined rollout training or dynamics gradient-based policy updates. While these approaches achieve comparatively fast learning, they all suffer, to varying degrees, from issues of recurrent compounding errors, high inference-time latency, and instability in backward propagation through time (BPTT). In this work, we propose a model-based RL approach that uses the chunk world model as a differentiable training-time evaluator to optimize the chunk actor without recurrent model invocation. Through the collaborative chunk-wise prediction of the world model and policy, we can generate H-step hypothetical rollouts in a single forward pass and directly optimize value expansion, avoiding the costly and unstable BPTT. We evaluate our method on 19 challenging high-dimensional humanoid tasks from HumanoidBench and deploy our method to the real-world for quadruped bipedal walking. Project Website: https://song-wx.github.io/cmpo.github.io/

Figures & tables

Figure 1: Computation graph of the Collaborative Chunk-wise policy update with the value expansion.
Figure 2: Results in HumanoidBench (9 tasks shown). Mean of 3 runs with different seeds. Shaded areas are 95% confidence intervals. CMPO achieves the best average performance and sample efficiency across baselines. Refer to Appendix C.4 for additional results.
 WalkRun
Wall-timeTD-MPC2CMPOTD-MPC2CMPO
200k training steps (h) ↓\downarrow3.571.743.541.72
one inference step (ms) ↓\downarrow40.81.439.51.3
Table 1: Training and inference Wall-time.
Figure 3: Sim-to-real transfer of standing locomotion on Unitree Go2. : Left: Representative snapshots of the learned standing locomotion policy in simulation and on the real robot. Right: Training curve of the standing linear-tracking reward in simulation.
Figure 5
Figure 6
 WalkRunStairSlide
CMPO954.1629.260.9265.8
w MPC931.1864.571.9391.2
w/o CM22.29.637.136.1
Table 2: Best average performance of variants with model prediction control (w MPC) and without the chunk-wise model (w/o CM).
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Visualization of the selected tasks.
Reward termDefinitionWeight wiw_{i}
Linear velocity trackingexp⁡ ⁣(−(cx+vzb)2+(cy−vyb)2σ)1up\displaystyle\exp\!\left(-\frac{(c_{x}+v_{z}^{b})^{2}+(c_{y}-v_{y}^{b})^{2}}{\sigma}\right)\mathbf{1}_{\mathrm{up}}2.02.0
Angular velocity trackingexp⁡ ⁣(−(cyaw−ωxb)2σ)1up\displaystyle\exp\!\left(-\frac{(c_{\mathrm{yaw}}-\omega_{x}^{b})^{2}}{\sigma}\right)\mathbf{1}_{\mathrm{up}}2.02.0
DOF acceleration∑j(q˙t−1,j−q˙t,jΔt)2\displaystyle\sum_{j}\left(\frac{\dot{q}_{t-1,j}-\dot{q}_{t,j}}{\Delta t}\right)^{2}−2.5×10−7-2.5\times 10^{-7}
Joint power∑j∣q˙j∣ ∣τj∣\displaystyle\sum_{j}|\dot{q}_{j}|\,|\tau_{j}|−2.0×10−5-2.0\times 10^{-5}
Base heightexp⁡ ⁣(−(h−h⋆)20.25),h⋆=0.55 m\displaystyle\exp\!\left(-\frac{(h-h^{\star})^{2}}{0.25}\right),\qquad h^{\star}=0.55\,\mathrm{m}8.08.0
Collision∑k∈C1 ⁣[∥fk∥2>0.1]\displaystyle\sum_{k\in\mathcal{C}}\mathbf{1}\!\left[\|\mathbf{f}_{k}\|_{2}>0.1\right]−2.0-2.0
Appendix
Table 3: Reward terms for bipedal Go2 locomotion.
Figure 7: All results in Humanoid Benchmark . The reward curves of 19 tasks in Humanoid Bench. Mean of 3 runs with different seeds. Shaded areas are 95% confidence intervals.
More in this topic

Model-Based RL

Model-based reinforcement learning uses environment models to plan actions and learn policies with fewer real-world interactions.

See all 245 papers

Explore similar work

CardsList

  1. DREAM-Chunk: Reactive Action Chunking with Latent World Model

    Jun 17, 2026Wenxi Chen, Kaidi Zhang, Chi Lin +6Test-Time ScalingRobotic Manipulation

    Purdue University · Stanford University

  2. Inference Time Policy Optimization for Offline RL with Differentiable World Models

    Mar 23, 2026Rohan Deb, Stephen J. Wright, Arindam BanerjeeInference-Time OptimizationOffline RL

    Siebel School of Computing and Data Science, Department of Computer Sciences, University of Illinois Urbana-Champaign · University of Wisconsin Madison

  3. ChunkFlow: Towards Continuity-Consistent Chunked Policy Learning

    Jul 14, 2026Zhao Yang, Yinan Shi, Mingyuan Yao +3Visuomotor Policy LearningVision-Language-Action Models

    State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, China. · Jiangsu Cytoderm Intelligent Technology Co., Ltd., China.