Scaling Tasks, Not Samples: Mastering Humanoid Control through Multi-Task Model-Based Reinforcement Learning

Shaohuai Liu; Weirui Ye; Yilun Du; Le Xie

cs.AIMar 2, 2026

Scaling Tasks, Not Samples: Mastering Humanoid Control through Multi-Task Model-Based Reinforcement Learning

Authors: Shaohuai Liu, Weirui Ye, Yilun Du, Le Xie

Organizations: Texas A&M University · MIT · Harvard University

Abstract

Developing generalist robots capable of mastering diverse skills remains a central challenge in embodied AI. While recent progress emphasizes scaling model parameters and offline datasets, such approaches are limited in robotics, where learning requires active interaction. We argue that effective online learning should scale the number of tasks, rather than the number of samples per task. This regime reveals a structural advantage of model-based reinforcement learning (MBRL). Because physical dynamics are invariant across tasks, a shared world model can aggregate multi-task experience to learn robust, task-agnostic representations. In contrast, model-free methods suffer from gradient interference when tasks demand conflicting actions in similar states. Task diversity therefore acts as a regularizer for MBRL, improving dynamics learning and sample efficiency. We instantiate this idea with EfficientZero-Multitask (EZ-M), a sample-efficient multi-task MBRL algorithm for online learning. Evaluated on HumanoidBench, a challenging whole-body control benchmark, EZ-M achieves state-of-the-art performance with significantly higher sample efficiency than strong baselines, without extreme parameter scaling. These results establish task scaling as a critical axis for scalable robotic learning. The project website is available here.

Figures & tables

Figure 1: Normalized task-average scores on HumanoidBench-Hard . EZ-M matches and surpasses the strong baselines with environment interactions limited to 1 million. All runs are with 3 random seeds. EZ-M significantly outperforms all baselines.

Figure 2: Overview of EfficientZero-Multitask (EZ-M) . Green arrows and notations represent model inference and predictions. Black notations and bold arrows indicate target/grounded values and transitions. (A) Task-sharing model architecture . EZ-M uses a shared model to predict policy, value, and reward while training multiple tasks, with each component conditioned by task embeddings. (B) Balanced data collection . Rollout workers sample tasks with a balanced schedule and execute the corresponding task policy to interact with the environment. This prevents learning collapse from data imbalance and maintains diverse task coverage over time. (C) Multi-task rollout learning . For each sampled trajectory segment, the model is unrolled in latent space conditioned on the task index to produce $\hat{r}_{\tau,k}$ , $\hat{v}_{\tau,k}$ , and $\hat{p}_{\tau,k}$ along the imagined rollout. Supervision comes from grounded transitions $(a_{\tau,k},r_{\tau,k})$ and reanalyzed targets $(\pi_{\tau,k},v_{\tau,k})$ , where cross-entropy loss enables a single network to learn consistent predictions across tasks. Blue dashed lines represents temporal consistency and path consistency. (D) Distributed implementation . Data collection (Rollout) feeds trajectories into task-independent replay buffers, while Reanalyze periodically recomputes search-based targets with the latest model. The learner consumes reanalyzed batches to update parameters and broadcasts updated weights back to rollout/reanalyze workers for scalable asynchronous training.

Figure 3: Gradient similarities over training process . We compare the gradient similarities on different model modules between BRC and EZ-M. We choose two task pairs, ( h1-walk , h1-run ) and ( h1-walk , h1-crawl ), representing relevant and irrelevant tasks, respectively. dyn , rew , and vp represent the dynamics, reward, and value-policy model in EZ-M. (Left) We validate that the gradient similarities between the relevant task pair are higher than the irrelevant across the training process, indicating positive knowledge transfer. (Right) We showcase that EZ-M modules have higher gradient similarities than BRC modules in the relevant task pair. Curves are slightly smoothed for better visualization.

Figure 4: Performance scale with the number of tasks. y -axis denotes the task-average, normalized episodic return and x -axis represents number of tasks in training simultaneously. We use 3 different random seeds and report 95% confidence intervals.

Figure 5: Ablations on model components. y -axis denotes the task-average, normalized episodic return and x -axis represents different settings. $-$ represents removing a component. TE represents task embedding. Dyn , Rew , VP , Rep represent the dynamics, reward, value-policy, representation model, respectively. - Dyn - Rew - VP _ TE indicates removing the task embedding on dynamics, reward, and value-policy models. IER represents independent experience replay. PathCons is the path consistency loss. We use 3 different random seeds and report 95% confidence intervals.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Unnormalized curves on HumanoidBench-Medium benchmark. y -axis denotes the episodic return and x -axis represents environment interactions per million. All experiments use 3 different random seeds. We report 95% confidence intervals.

Figure 7: Unnormalized curves on HumanoidBench-Hard benchmark. y -axis denotes the episodic return and x -axis represents environment interactions per million. All experiments use 3 different random seeds. We report 95% confidence intervals.

Task	Random	Success	MHSAC	TD-MPC2	BRC	BRC-ST	EZ-M(ours)
h1-stand	10.545	800.0	74.904	810.688	829.358	841.246	806.034
h1-walk	2.377	700.0	29.352	814.249	844.784	858.926	802.969
h1-run	2.020	700.0	17.434	778.845	772.299	439.246	775.867
h1-stair	3.112	700.0	25.421	497.040	474.852	322.088	459.234
h1-crawl	272.658	700.0	449.088	963.360	949.326	971.295	830.971
h1-slide	3.191	700.0	29.004	334.359	831.837	578.478	577.740

Appendix

Table 1: Scores for HumanoidBench-Medium (1M) tasks. We bold the results surpass the success score or achieve the SoTA.

Task	Random	Success	DreamerV3	TD-MPC2	BRC	BRC-ST	EZ-M(ours)
h1hand-stand	11.973	800.0	40.938	196.511	840.061	315.990	891.780
h1hand-walk	2.505	700.0	19.819	234.061	515.383	771.471	919.960
h1hand-run	1.927	700.0	6.580	32.551	171.790	44.512	818.397
h1hand-stair	3.161	700.0	16.193	46.537	188.810	64.864	380.349
h1hand-crawl	278.868	800.0	261.801	897.086	735.992	849.207	644.386
h1hand-slide	3.142	700.0	20.625	89.454	313.723	148.877	535.404

Appendix

Table 2: Scores for HumanoidBench-Hard (1M) tasks. We bold the results surpass the success score or achieve the SoTA.

Hyperparameter	Value
Environment	HumanoidBench
Discount factor $\gamma$	0.99
Unroll steps	5
TD steps	5
Batch size	512
Replay buffer size	$1\times 10^{6}$ (Medium) / $2\times 10^{6}$ (Hard)

Appendix

Table 3: Key hyperparameters used in HumanoidBench experiments

Module	Architecture
Representation $f_{\theta}$	MLP(305 $\to$ 512) + LN + Tanh; 3 $\times$ ResBlock(512)
Task embedding	Embedding(14, 128), max_norm=1
Dynamics $g_{\theta}$	LN(701); MLP(701 $\to$ 512 $\to$ 512); 3 $\times$ ResBlock(512)
Reward head $r_{\theta}$	act-MLP(189 $\to$ 64)+LN+Tanh;
	dyn trunk: LN(704)+MLP(704 $\to$ 512 $\to$ 512)+3 $\times$ ResBlock(512);
	rew: [Linear(512 $\to$ 512)+BN+ReLU] $\times$ 2; Linear(512 $\to$ 51)

Appendix

Table 4: EfficientZero network architecture (for HumanoidBench-Hard).

Multi-Task RL

Multi-task reinforcement learning trains agents to solve related tasks by sharing policies, representations, or experience.

See all 84 papers

Representation Learning Enables Scalable Multitask Deep Reinforcement Learning
Jun 4, 2026Johan Obando-Ceron, Lu Li, Scott Fujimoto +3Reinforcement Learning Representation Learning for RL
Mila – Québec AI Institute · Université de Montréal · McGill University +2
A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control
Oct 8, 2026Octi Zhang, Mateo Guaman Castro, Patrick Yin +4Robotic RL Data-Efficient Robot Learning
University of Washington · NVIDIA
Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation
Jun 4, 2026Junheng Li, Liang Wu, Sergio A. Esteban +3RL Control Humanoid Robot Locomotion
California Institute of Technology, CA 91106, USA · Johns Hopkins University, MD 21218, USA

Jun 4, 2026J/K move · Enter open · S save

Representation Learning Enables Scalable Multitask Deep Reinforcement Learning

Johan Obando-Ceron, Lu Li, Scott Fujimoto, Pierre-Luc Bacon+2

Mila – Québec AI Institute · Université de Montréal · McGill University+2

Reinforcement Learning Representation Learning for RL Continuous Control Multi-Task RL Model-Free RL

Open paper AskPDF arXiv

Scaling reinforcement learning (RL) to diverse multitask settings remains a central challenge. While recent advances in model-based RL achieve strong performance, they rely on planning and complex training pipelines, making it unclear which components are essential for scalability. We revisit this question and argue that the primary driver of scalable multitask RL is not model-based control, but representation learning. In particular, we show that combining predictive, model-based representations with high-capacity value function approximation is sufficient to achieve strong performance, even without planning. We evaluate a simple model-free algorithm, MR.Q, coupled with auxiliary predictive objectives into a scalable actor-critic architecture. This approach outperforms a recent world-model-based method and a range of deep RL baselines across a diverse suite of multitask continuous control tasks, while significantly reducing computational overhead and improving wall-clock efficiency. We observe consistent improvements with increased model capacity and show through ablations that predictive representation learning is critical for performance.

Abstract

Multi-Task RL

Explore similar work

Representation Learning Enables Scalable Multitask Deep Reinforcement Learning

A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control

Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation