Authors: Michal Korniak, Marco Bagatella, Ralf Römer, Michal Nauman, Yarden As, Pranav Vignesh, Manan Agarwal, Leander Diaz-Bone, Piotr Zalewski, Mikołaj Czarnecki, Lenart Treven, Shashwat Saxena, Sreyas Venkataraman, Sarvesh Patil, Zhiyuan Zhou, Daniel Marta, Oier Mees, Nur Muhammad Mahi Shafiullah, Carmelo Sferrazza, Bhavya Sukhija, Angela P. Schoellig, Andreas Krause, Max Simchowitz, Andrew Wagenmaker
Organizations: ETH Zurich · MPI for Intelligent Systems, Tübingen · Technical University of Munich · University of Warsaw · Nomagic · Carnegie Mellon University · UC Berkeley · Microsoft · Amazon FAR · Physical Intelligence
The introduction of generalist robot policies has enabled a single, large model to learn to perform a variety of tasks from a vast pre-training corpus. While these models often exhibit powerful zero-shot generalization capabilities, they typically require an adaptation phase to achieve sufficient performance in deployment. Iterated self-improvement via reinforcement learning (RL) has demonstrated promise in enabling this adaptation, yet such approaches have been largely explored in single-task settings - where the goal is to simply improve on a narrow deployment task - which lies in stark contrast to the multi-task capabilities of the base policy. In this work we ask whether generalist robot policies are capable of multi-task, generalist improvement in deployment - targeting many diverse tasks simultaneously - and what algorithmic recipe enables this. To answer this question, we first develop a set of structured evaluations to test generalist improvement capabilities, and systematically evaluate algorithmic and practical design choices regarding value estimation, policy extraction and resource allocation. We show that not only can generalist policies be improved online across diverse tasks, but that the right recipe can enable generalization and increase performance on tasks not considered during online training. We characterize when this generalization occurs and show that both the training task distribution and the use of value-based RL are critical. To the best of our knowledge, this is among the first evidence for online generalist improvement for robot foundation models.
Figures & tables
Figure 1: Scaling the number of RL training tasks enables generalist improvement of generalist robot policies. As training task diversity increases, critic-guided policy extraction (OGPO) or refinement (BoN) prevent forgetting and improve performance on unseen tasks on MolmoSpaces , unlike filtered supervised fine-tuning (FSFT).
Figure 2: Generalist improvement and evaluation. (1) A generalist policy collects K rollouts τn,k across training tasks cn∼ρtrain . (2) Task-specific experience is pooled in a shared replay buffer and used to enact one of three families of methods : direct policy adaptation updates the policy according to success labels , critic-guided refinement learns critics for action ranking , and critic-guided policy extraction learns critics and updates the policy towards actions with high critic values. This process is iterated M times. (3) Evaluation is carried out over training tasks ( c∼ρtrain ), as well as on unseen test tasks ( c∼ρtest ) to test generalization.
Figure 3: Objects and scenes in MolmoSpaces . Left: 4 tasks covering two objects, each with two scenes. Center: unseen scenes of the training objects. Right: examples of unseen objects.
Figure 4: Value-based generalist improvement generalizes beyond the training tasks. Final success rate on training tasks (left) and held-out tasks (center and right, unseen scenes and objects) in MolmoSpaces as the number of distinct training tasks increases (x-axis). BoN and OGPO display stronger generalization. We report the mean and 95% stratified bootstrap CIs across three seeds.
Figure 5: Compute efficiency of generalist improvement. Post-training a policy on multiple tasks at once achieves the performance of post-trained single-task experts in significantly fewer gradient steps.
Figure 6: Diversification transfers better along its own axis. OGPO success rate when scaling scenes of a single object, or objects with one scene each, evaluated on eight unseen scenes ( left ) and eight unseen objects ( right ). Each axis transfers best along itself, though only scaling objects surpasses the base policy on both. Mean and 95% stratified bootstrap CIs over three seeds.
Figure 7: MolmoSpaces . Final success rate on each of the 6 unseen objects, none of which appears in any training split, as a function of the number of training tasks. Points show the mean over three seeds with 95% stratified bootstrap CIs.
Figure 8: Generalization across objects. Left: Training objects per MolmoSpaces split. Each split contains all objects of the previous one; + marks objects added. The 1-task split is trained separately on a single box or spatula task. Full task details are in Table 3 . Right: We train each method on tasks involving a single object (box or spatula) and evaluate on the unseen knife. For methods that use a critic (BoN, OGPO), training on a similar object transfers better, whereas FSFT shows no such pattern. We report the mean and 95% stratified bootstrap CIs across three seeds.
Figure 9: Recipe ablations. Left: full VLA vs. expert-only tuning for FSFT on MolmoSpaces (64 tasks). Full tuning is essential in-distribution and on unseen scenes, but degrades most on unseen objects. Center: architecture for BoN. Right: TD vs MC, classification vs regression for BoN. Center and right are evaluated on LIBERO (4 tasks; see Table 4 ) and the object-diverse MolmoSpaces (8 objects, 1 scene each; see Table 1 ).
Figure 10: Policy extraction schemes. We compare OGPO ( Patil et al., 2026 ) to AWR ( Peng et al., 2019 ) and a direct combination of FSFT and BoN. We find that natively actor-critic algorithms perform best.
Appendix figures & tables42 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11: A taxonomy of a few VLA post-training approaches. Approaches above the dashed line adopt a purely supervised objective for policy extraction. We italicize the three approaches at the center of the analysis in Sections 4.1 and 4.2 .
Figure 12: Value-based generalist improvement is limited when tasks are loosely related. Performance of FSFT and BoN on training tasks (left) and held-out tasks (right) in LIBERO as the training tasks grow in number. The single-task lead of BoN reverses in larger multi-task experiments, but FSFT exhibits substantial forgetting on held-out tasks, unlike BoN.
Figure 13: Diversify along the axis of desired generalization. Scaling scenes of a single object ( left two ) or objects with one scene each ( right two ), evaluated on unseen scenes and unseen objects. Each axis transfers best along itself, and BoN scales better than FSFT, which falls below the base policy on unseen objects. Mean over three seeds, 95% stratified bootstrap CIs.
Table 1: Task splits for the diversity ablation. Task IDs are given without the molmo_ prefix and grouped by object; Init. SR is the mean success rate of the base policy over a split. Scaling scenes varies the number of scenes of a single object; scaling objects varies the number of objects, with one scene each. Both regimes are evaluated on the same two held-out splits: eight unseen scenes of box, the one object common to every training split, and one scene of each of eight objects never seen during training. We note that the MolmoSpaces (8 tasks) split is used in Figures 9 and 16 .
Figure 14: Post-training benefits from more data and more frequent iterations. Final success rates for critic-guided refinement (BoN) on LIBERO . Each subplot changes one hyperparameter: the number of episodes collected per iteration (left), the number of gradient steps per iteration (middle) and the number of iterations (right) over which a data and compute budget is distributed. The other hyperparameters are fixed per plot. 1× marks the values used across the standard experimental protocol described in Section 3.3 . Collecting more data is beneficial with diminishing gains, while more updates may be harmful; frequent iterations beat a single update phase after data collection.
Figure 15: Pretrained critic for accelerating value learning. A critic pretrained on MolmoSpaces (64 tasks) starts higher than one trained from scratch, but the gap closes within one iteration and then reverses, on both unseen scenes ( left ) and unseen objects ( right ). Mean and 95% stratified bootstrap CIs across three seeds.
Figure 16: Learning rate ablation for FSFT. We sweep three learning rates across four task splits: LIBERO (1 task), LIBERO (16 tasks), MolmoSpaces (1 task), and MolmoSpaces (8 tasks). We evaluate performance on the training tasks (In-Distr.) and on the validation tasks (OOD). Bars show the mean over three seeds, with error bars denoting 95% stratified bootstrap confidence.
Figure 17: Impact of the number of candidates N in Best-of- N on performance. Success rate of BoN on each of the four single-task MolmoSpaces runs as N grows. On average, performance grows with N and peaks at N=32 , though the gains over N∈{8,16} are marginal. We report mean and 95% stratified bootstrap CIs over three seeds.
Table 2: BoN inference cost. Wall-clock time of one Best-of- N step on a single GH200 GPU, relative to N=1 .
Table 3: MolmoSpaces task splits. For each training split, we list the task IDs it contains, grouped by object and given without the molmo_ prefix. Scenes/obj. is the number of scenes per object; Init. SR the mean success rate of the base policy. Each training split is evaluated only on unseen scenes of the objects it was trained on ( Eval. tasks : 8 held-out scenes per training object), and every split is evaluated on the same 8 scenes of each of 6 objects never seen during training. † Four independent single-task runs (one per listed task), each evaluated on the 8 unseen scenes of its own object. ‡ Pool of unseen-scene tasks; each split uses the rows of its own training objects.
boiler
box
cup
ladle
pot
tissuepaper
332
164
18
28
309
11
33
86
292
311
227
361
242
70
48
50
265
103
369
267
185
56
325
4
126
366
24
399
277
260
383
138
62
187
127
221
Appendix
Figure 18: MolmoSpaces training tasks – visualization. Initial exterior-camera frame of every training task of the 144-task split, ordered as in Table 3 . Task IDs are shown in the lower-left corner of each frame.
boiler
box
cup
ladle
pot
tissuepaper
754
55
219
925
425
766
809
885
398
952
271
771
883
905
749
807
346
958
8
273
222
73
972
49
961
999
424
692
735
637
611
965
799
651
634
940
Appendix
Figure 19: MolmoSpaces unseen scenes – visualization. Initial exterior-camera frame of every unseen scene task of the 144-task split, ordered as in Table 3 . Task IDs are shown in the lower-left corner of each frame.
atomizer
bowl
fork
knife
remotecontrol
winebottle
209
665
281
419
465
679
397
118
181
433
152
206
192
214
372
629
349
371
99
356
345
934
353
448
140
101
167
155
390
226
57
136
290
223
357
340
Appendix
Figure 20: MolmoSpaces unseen objects – visualization. Initial exterior-camera frame of every unseen object task of the 144-task split, ordered as in Table 3 . Task IDs are shown in the lower-left corner of each frame.
Table 4: LIBERO task splits. All tasks are drawn from LIBERO-90 ; task IDs are given without the libero_90_ prefix. Init. SR is the mean success rate of the base policy over the tasks of a split. The training splits are nested, and the unseen tasks are disjoint from all of them.
Table 5: Hyperparameters. Training budget : each run performs M collection rounds of K episodes per task, with the given number of gradient steps per round (single-task / multi-task). Tuned weights : whether the full VLA or only the action expert is fine-tuned. Trained on : whether the SFT loss uses only successful episodes. Update interval : critic steps per policy update. Bootstrap : SARSA uses the next action stored in the buffer, TD uses an action sampled from the current policy. Critic-based methods use a replay buffer of 1M transitions (2M for OGPO on the 144-task split). BoN and OGPO run for 10 collection rounds, except on the 144-task split, where we extend it to 15 rounds because performance had not yet plateaued. FSFT, in contrast, plateaus early and then slightly declines, so 10 rounds was enough.
Figure 21: MolmoSpaces . Final success rate on unseen scenes of the training objects. Points show the mean over three seeds with 95% stratified bootstrap CIs.
Figure 22: MolmoSpaces (1 task): training tasks. Success rate on the training tasks as a function of normalized training progress, where 1 denotes the end of post-training. Lines show the mean over three seeds. Shaded regions denote 95% stratified bootstrap CIs.
Figure 23: MolmoSpaces (1 task): unseen scenes. Final success rate on unseen scenes. Bars show the mean over three seeds with 95% stratified bootstrap CIs.
Figure 24: MolmoSpaces (1 task): unseen objects. Final success rate on unseen objects. Bars show the mean over three seeds with 95% stratified bootstrap CIs.
Figure 25: MolmoSpaces (4 tasks): training tasks. Success rate on the training tasks as a function of normalized training progress, where 1 denotes the end of post-training. Lines show the mean over three seeds. Shaded regions denote 95% stratified bootstrap CIs.
Figure 26: MolmoSpaces (4 tasks): unseen scenes. Final success rate on unseen scenes. Bars show the mean over three seeds with 95% stratified bootstrap CIs.
Figure 27: MolmoSpaces (4 tasks): unseen objects. Final success rate on unseen objects. Bars show the mean over three seeds with 95% stratified bootstrap CIs.
Figure 28: MolmoSpaces (16 tasks): training tasks. Success rate on the training tasks as a function of normalized training progress, where 1 denotes the end of post-training. Lines show the mean over three seeds. Shaded regions denote 95% stratified bootstrap CIs.
Figure 29: MolmoSpaces (16 tasks): unseen scenes. Final success rate on unseen scenes. Bars show the mean over three seeds with 95% stratified bootstrap CIs.
Figure 30: MolmoSpaces (16 tasks): unseen objects. Final success rate on unseen objects. Bars show the mean over three seeds with 95% stratified bootstrap CIs.
Figure 31: MolmoSpaces (64 tasks): training tasks. Success rate on the training tasks as a function of normalized training progress, where 1 denotes the end of post-training. Lines show the mean over three seeds. Shaded regions denote 95% stratified bootstrap CIs.
Figure 32: MolmoSpaces (64 tasks): unseen scenes. Final success rate on unseen scenes. Bars show the mean over three seeds with 95% stratified bootstrap CIs.
Figure 33: MolmoSpaces (64 tasks): unseen objects. Final success rate on unseen objects. Bars show the mean over three seeds with 95% stratified bootstrap CIs.
Figure 34: MolmoSpaces (144 tasks, Part 1): training tasks. Success rate on the training tasks as a function of normalized training progress, where 1 denotes the end of post-training. Lines show the mean over three seeds. Shaded regions denote 95% stratified bootstrap CIs.
Figure 35: MolmoSpaces (144 tasks, Part 2): training tasks. Success rate on the training tasks as a function of normalized training progress, where 1 denotes the end of post-training. Lines show the mean over three seeds. Shaded regions denote 95% stratified bootstrap CIs.
Figure 36: MolmoSpaces (144 tasks, Part 1): unseen scenes. Final success rate on unseen scenes. Bars show the mean over three seeds with 95% stratified bootstrap CIs.
Figure 37: MolmoSpaces (144 tasks, Part 2): unseen scenes. Final success rate on unseen scenes. Bars show the mean over three seeds with 95% stratified bootstrap CIs.
Figure 38: MolmoSpaces (144 tasks): unseen objects. Final success rate on unseen objects. Bars show the mean over three seeds with 95% stratified bootstrap CIs.
Figure 39: LIBERO (1 task): training tasks. Success rate on the training tasks as a function of normalized training progress, where 1 denotes the end of post-training. Lines show the mean over three seeds. Shaded regions denote 95% stratified bootstrap CIs.
Figure 40: LIBERO (1 task): unseen tasks. Final success rate on unseen tasks. Bars show the mean over three seeds with 95% stratified bootstrap CIs.
Figure 41: LIBERO (4 tasks): training tasks. Success rate on the training tasks as a function of normalized training progress, where 1 denotes the end of post-training. Lines show the mean over three seeds. Shaded regions denote 95% stratified bootstrap CIs.
Figure 42: LIBERO (4 tasks): unseen tasks. Final success rate on unseen tasks. Bars show the mean over three seeds with 95% stratified bootstrap CIs.
Figure 43: LIBERO (16 tasks): training tasks. Success rate on the training tasks as a function of normalized training progress, where 1 denotes the end of post-training. Lines show the mean over three seeds. Shaded regions denote 95% stratified bootstrap CIs.
Figure 44: LIBERO (16 tasks): unseen tasks. Final success rate on unseen tasks. Bars show the mean over three seeds with 95% stratified bootstrap CIs.
Figure 45: LIBERO (64 tasks): training tasks. Success rate on the training tasks as a function of normalized training progress, where 1 denotes the end of post-training. Lines show the mean over three seeds. Shaded regions denote 95% stratified bootstrap CIs.
Figure 46: LIBERO (64 tasks): unseen tasks. Final success rate on unseen tasks. Bars show the mean over three seeds with 95% stratified bootstrap CIs.
Figure 47: MolmoSpaces (16 tasks): recipes for policy extraction. Success rate on the training tasks as a function of normalized training progress, where 1 denotes the end of post-training. Lines show the mean over three seeds. Shaded regions denote 95% stratified bootstrap CIs.
Generalist robot policies learn a diverse repertoire of behaviors from large-scale pretraining. In principle, this makes them excellent priors for downstream adaptation via reinforcement learning (RL). In practice, however, standard RL methods leveraging this prior optimize directly over robot actions, requiring the base policy's action distribution to be close to that of a performant policy from the start. This assumption breaks down for complex or long-horizon tasks that fall outside the pretraining distribution. Our key insight is that, for sufficiently expressive generalist policies, language prompts are an effective alternative space for learning to solve such tasks: modulating language inputs elicits skills already within the policy's repertoire, which can be composed to solve tasks beyond its zero-shot capabilities. We propose Semantic Action Reinforcement Learning (SARL), which learns to optimize this prompt space through online interaction, treating the generalist policy as a controllable skill prior. Importantly, leveraging pretrained skills rather than learning new ones from scratch yields structured, semantically meaningful exploration and highly efficient online improvement, and learning to modulate prompts through experience grounds them in induced real-world behaviors for robust task-solving. Across real-world settings and simulated benchmarks, we show SARL unlocks fundamentally new capabilities -- adapting VLA behavior to solve complex, long-horizon tasks -- and significantly outperforms existing approaches for improving robot behavior in deployment.
Jagdeep Singh Bhatia, Andrew Wagenmaker, William Chen +1
Generalist robot policies increasingly benefit from large-scale pretraining, but offline data alone is insufficient for robust real-world deployment. Deployed robots encounter distribution shifts, long-tail failures, task variations, and human correction opportunities that fixed demonstration datasets cannot fully capture. We present Learning While Deploying (LWD), a fleet-scale offline-to-online reinforcement learning framework for continual post-training of generalist Vision-Language-Action (VLA) policies. Starting from a pretrained VLA policy, LWD closes the loop between deployment, shared physical experience, policy improvement, and redeployment by using autonomous rollouts and human interventions collected across a robot fleet. To stabilize learning from heterogeneous, sparse-reward fleet data, LWD combines Distributional Implicit Value Learning (DIVL) for robust value estimation with Q-learning via Adjoint Matching (QAM) for policy extraction in flow-based VLA action generators. We validate LWD on a fleet of 16 dual-arm robots across eight real-world manipulation tasks, including semantic grocery restocking and 3--5 minute long-horizon tasks. A single generalist policy improves as fleet experience accumulates, reaching an average success rate of 95%, with the largest gains on long-horizon tasks.
Yi Wang, Xinchen Li, Pengwei Xie +13
Shanghai Innovation Institute. · AGIBOT Finch. · Columbia University.
Developing generalist robots capable of mastering diverse skills remains a central challenge in embodied AI. While recent progress emphasizes scaling model parameters and offline datasets, such approaches are limited in robotics, where learning requires active interaction. We argue that effective online learning should scale the number of tasks, rather than the number of samples per task. This regime reveals a structural advantage of model-based reinforcement learning (MBRL). Because physical dynamics are invariant across tasks, a shared world model can aggregate multi-task experience to learn robust, task-agnostic representations. In contrast, model-free methods suffer from gradient interference when tasks demand conflicting actions in similar states. Task diversity therefore acts as a regularizer for MBRL, improving dynamics learning and sample efficiency. We instantiate this idea with EfficientZero-Multitask (EZ-M), a sample-efficient multi-task MBRL algorithm for online learning. Evaluated on HumanoidBench, a challenging whole-body control benchmark, EZ-M achieves state-of-the-art performance with significantly higher sample efficiency than strong baselines, without extreme parameter scaling. These results establish task scaling as a critical axis for scalable robotic learning. The project website is available here.