Robot deployment generates experience that can support policy improvement, but continued reliance on expert corrective actions makes learning difficult to scale across growing fleets. We propose learning from fleet experience: using deployed robots' own executions as a source of supervision for further policy improvement. We call this paradigm Collective Experience Learning (CEL), in which robots pool execution experience to update a shared policy that returns to the fleet. Following expert-supervised initialization, we realize CEL through failure-guided experience retrieval fine-tuning (FERT). FERT uses failure contexts to retrieve relevant segments from successful trajectories and prioritizes their recorded actions for supervised updates, without collecting additional expert demonstrations or corrective actions. Operators provide binary episode-outcome labels and environment resets. A fault-tolerant asynchronous training system (FATS) coordinates robot execution, cloud inference and policy optimization despite local interruptions. Across five real-world manipulation tasks, the resulting policies achieve success rates of 93.3-99.2%, evaluated over 120 trials per task on robots excluded from training data collection. With collection time per robot held fixed, larger fleets provide more experience and generally improve performance across the evaluated sizes. On multi-stage phone packaging, learning with 96 robots raises success from 79.2% to 98.3%. These results support CEL as a scalable approach to improving robot policy reliability through experience generated during deployment.
Figures & tables
Figure 1: Overview of Collective Experience Learning (CEL). We propose a new training paradigm CEL, so that robots improve from fleet experience rather than continued human experience, enabling large-scale online training with up to 96 robots. We develop a training algorithm FERT under CEL paradigm, which uses failure contexts to retrieve relevant successful segments and prioritize their recorded actions for supervised updates. To support scalable online training, FATS coordinates fault-tolerant asynchronous execution, cloud inference and policy optimization. Across five real-world manipulation tasks, policies improve after FERT, and success rates generally scale with fleet size.
Figure 2: Sources of action supervision for robot policy learning: human demonstrations, expert corrections and fleet experience. CEL uses robot-generated actions for further policy improvement after expert-supervised initialization. Our implementation retains human-provided episode-outcome labels and environment resets.
Figure 3: Failure-guided experience retrieval fine-tuning. Robots asynchronously collect autonomous experience with binary task-outcome labels. Observations are encoded using visual and proprioceptive features. Failure contexts retrieve related segments from successful trajectories, and these segments receive increased sampling priority during policy fine-tuning. Updated policies return to the fleet to generate further experience.
Interaction
Experience
Unit
Observation and action chunk
Trajectory fields assembled into an episode
Size
0.45 MB of images per model input
1.58 GB per Phone Packaging episode
Timing
Each inference request
Fields stream during execution, episodes complete about every 4 min
Requirement
Reply before the robot’s next interaction
Complete record and outcome before use in training
Path
Robot observations to cloud inference, actions back to the robot
Cloud-side assembly reuses inference data and receives additional fields from the robot site
Table 1: Interaction and experience paths in FATS. Sizes describe model-input images and a complete episode record, respectively.
Figure 4: Fault-tolerant asynchronous training system (FATS). a , The SwarmLink data plane connects robot workers at the edge to shared cloud inference and assembles streamed trajectory fields for training. The HotSwarm control plane manages membership and worker identity, and recovers failed workers. b , Synchronous collection makes each policy update wait for every robot, which leaves faster robots idle. In asynchronous collection, robots continue independently. c , HotSwarm restarts the worker of a failed robot in place while the other robots continue, and the action computed for the failed worker (#1) is discarded. d , Inference data and execution records are joined in the cloud, and an episode is released to training only after its required fields and outcome arrive. e , Robot time available for collection after fault handling, measured for FATS in the 96-robot run and modeled for restarting the entire system after each fault event with restart times between 2 and 10 min ( Section D.4 ). Durations in b and c are illustrative.
Artifact
Hazard if the identity is not checked
Response
Membership record
A peer builds a connection to a dead process and waits indefinitely
Refuse and wait for a fresh record
Control message in flight
A late failure report retires the replacement
Ignore
Connection held by a peer
The replacement joins an outdated connection group
Rebuild under a new epoch
Data in the pipeline
Training pairs actions with another worker’s observations
Drop and receive again
Queued transfer
The transfer runs on a destroyed connection
Replay on the new connection
Table 2: Artifacts that can outlive a failed worker, the hazard each poses to its replacement, and the response of HotSwarm.
Figure 5: Overview of the five manipulation tasks performed with AgiBot G2 robots: Flower Arrangement, Clothes Folding, Phone Charging, Stool Assembly and Phone Packaging. A detailed step-by-step description is provided in Appendix E .
Figure 6: Main results across five tasks. Each subfigure compares the success rates of human teleoperation, SFT, the selected multi-robot HG-DAgger (labelled (M)HG-DAgger), RLT, RECAP, and FERT on one task. For FERT, we report the checkpoint trained with the largest robot fleet. Error bars indicate 95% Wilson confidence intervals.
Task name
(M)HG-DAgger (%) ↓
FERT (%) ↓
Relative failure reduction (%) ↑
Flower Arrangement
1.67
0.83
50.0
Clothes Folding
14.17
1.67
88.2
Phone Charging
40.00
2.50
93.8
Stool Assembly
18.33
6.67
63.6
Phone Packaging
20.83
1.67
92.0
Task avg
19.00
2.67
86.0
Table 3: Failure rate table. (M)HG-DAgger reports the performance of the checkpoints used to initialize FERT. FERT reports results obtained with the maximum number of training robots for each task.
Figure 7: Scaling with fleet size across five tasks. Each panel shows success rate as a function of the number of training robots. The blue and pink curves represent HG-DAgger and FERT, respectively. HG-DAgger starts from SFT, whereas FERT starts from the selected (M)HG-DAgger checkpoint; horizontal dashed lines in different colours indicate the success rates of these initial policies. Error bars indicate 95% Wilson confidence intervals.
Figure 8: Ablation studies of FERT on Phone Packaging with 16 training robots. Pink bars indicate the full FERT configuration, while gray bars indicate the corresponding ablated variants. In (a), FR denotes failure-guided retrieval. In (b), the interval is measured in optimizer steps. In (c), each value specifies both hpre and hpost , with L=90 frames held fixed. In (d), ρ is the priority retention factor; larger values retain previous priorities for longer. Error bars indicate 95% Wilson confidence intervals.
Figure 9: FATS in the 96-robot training run. a , Episodes completed per minute in 5-min bins. Marks above the curve show fault events, with height proportional to the number of robots affected, and the shaded interval is the lunch break. b , Fault incidents by initiating cause and duration. Circles mark incidents in which the robot was restarted, and squares mark incidents in which recovery was blocked. c , Episodes collected when each robot’s measured episodes and pauses are replayed under other designs, relative to the run. A full restart stops all robots for 2, 5 or 10 minutes after each fault event, and synchronous rounds make each policy update wait for one episode from every robot. The two synchronous variants replace the pauses that overlap a fault, or all pauses, with the robot’s median pause. d , Completed episodes per robot, sorted by count. Statistics are computed from the detailed traces in Section H.2 .
Figure 10: Measured communication-operation durations across observation, action and experience channels. Bars report the fraction and count of operations in each duration interval, and each panel shows the total number of events.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11: Multi-robot HG-DAgger with human interventions. Human operators take over execution when correction is needed. Successful trajectories are stored with intervention indicators, and eligible intervention chunks are used for supervised policy updates.
Figure 12: Robot-site network and cloud resources for FATS. The site supports 100 robots, with 96 used in the largest training run. Six cloud servers provide 48 NVIDIA A800 GPUs for inference and policy optimization. The robot site and cloud communicate through a dedicated 30 Gbps link.
Task
Language prompt
Flower arrangement
“Put the flowers into the vase.”
Clothes folding
“Fold the cloth.”
Phone charging
“Charge the phone.”
Stool assembly
“Assemble the stool with a screwdriver.”
Phone packaging
“Pack the phone.”
Task
Head-camera crop box
Two wrist-cameras crop box
Appendix
Table 4: Task prompts and observation crop parameters. Wrist crops apply to both the left and right cameras.
Parameter
Setting
Max learning rate
2.5×10−5
Global batch size
1024
Learning-rate schedule
Cosine
Learning-rate warmup
80 optimizer steps
Learning-rate schedule horizon
10,000 optimizer steps
Gradient norm clipping
4.0
Appendix
Table 5: Key FERT settings. Batch sizes are global.
Task
Data source
Trajectories
Intervention
Total
Successful
Failed
fraction
Flower arrangement
(M)HG-DAgger
160
160
0
74.6%
Newly collected data
48
13
35
2.4%
Clothes folding
(M)HG-DAgger
148
148
0
68.0%
Newly collected data
118
39
79
18.0%
Phone charging
(M)HG-DAgger
716
716
0
39.9%
Appendix
Table 6: RECAP training data sources and composition. Intervention fractions are measured over frames.
Parameter
Setting
Value peak learning rate
10−4
Value global batch size
2,048
Policy peak learning rate
5×10−5
Policy global batch size
256
Number of value bins
201
Value support
[−1,0]
Appendix
Table 7: Key RECAP settings. Batch sizes are global.
Parameter
Setting
Actor learning rate
3×10−4
Critic learning rate
3×10−4
Global batch size
256
Discount factor
0.99
Update-to-data ratio
5
Critic-to-actor update ratio
2:1
Appendix
Table 8: Key RLT settings shared across tasks, except for the overrides in Table 9 . Deviation budgets are in normalized action coordinates.
Task
Parameter
Setting
Phone Charging
Deviation budget
0.2 for the 7 right-arm dimensions; 0 for the left arm and grippers
Flower Arrangement
Deviation budget
0.5 for the 14 arm dimensions; 0 for the grippers
Exploration standard deviation
0.005
Clothes Folding
Deviation budget
0.5 for the 14 arm dimensions; 0 for the grippers
Exploration standard deviation
0.005
Appendix
Table 9: Task-specific RLT settings. Unlisted settings follow Table 8 . Deviation budgets are in normalized action coordinates.
Task
Intervention fraction
Complete episodes
Flower Arrangement
30%
30
Clothes Folding
20%
30
Phone Packaging
40%
40
Stool Assembly
80%
100
Phone Charging
40%
70
Appendix
Table 10: Online RLT intervention fractions and complete-episode counts by task. All values are approximate.
Table 11: Task success rates for SFT, (M)HG-DAgger, RLT, RECAP, and FERT. Results are reported for two held-out test robots, with 60 trials per robot for each evaluated checkpoint. Robot 1 and Robot 2 denote the held-out test robots; their rows report successful trials out of 60. Avg. (%) pools successes over 120 trials. Bold values indicate the highest average success rate for each task.
FERT (16 robots)
w/o FR
No extension
Retention ρ
Sync interval
0.8
0.95
100
1500
Robot 1
53/60
33/60
34/60
44/60
50/60
46/60
48/60
Robot 2
51/60
44/60
35/60
41/60
49/60
44/60
48/60
Avg. (%)
86.67
64.17
57.50
70.83
82.50
75.00
80.00
Appendix
Table 12: Success rates for the FERT ablation study on Phone Packaging with 16 training robots. FERT denotes the full configuration; w/o FR disables failure-guided retrieval; No extension sets hpre=hpost=0 . Retention ρ is the priority retention factor, and Sync interval is the policy synchronization interval in optimizer steps. Each configuration is evaluated over 60 trials on each of two held-out test robots. The robot rows report successful trials out of 60, and Avg. (%) pools successes over 120 trials. Bold indicates the highest average success rate.
ID
Flower Arrangement
Clothes Folding
Phone Charging
Stool Assembly
Phone Packaging
1
20/20
20/20
12/20
20/20
16/20
2
19/20
16/20
18/20
20/20
13/20
3
20/20
19/20
12/20
17/20
19/20
4
12/20
15/20
12/20
15/20
19/20
5
18/20
14/20
19/20
14/20
18/20
6
20/20
15/20
13/20
14/20
5/20
Appendix
Table 13: Human teleoperation success rates across five tasks. Each of the six operators completes 20 trials per task, yielding 120 trials per task in total. ID denotes the operator; each entry reports successful trials out of 20. Avg. (%) pools successes over all 120 trials per task.
Figure 13: Human teleoperation success rates across tasks and operators. Each operator is evaluated over 20 trials per task.
Figure 14: Detailed per-robot trial timelines during a 96-robot training run. Successful trials, failed trials and system errors are marked separately; blank intervals include environment resets and operator breaks.
Figure 15: Active robot count during a 96-robot training run. A robot is active while executing actions; inactive intervals include environment resets, operator breaks and robot errors. Light curves show raw measurements, and dark curves show Gaussian-smoothed measurements ( σ=60 s).
Figure 16: Error Types and Occurrence Times During a 96-Robot Training Run. The x-axis indicates time, and the y-axis lists the IDs of robots that encountered at least one error. Colors distinguish different error types.
Figure 17: Visualize the retrieval effect in Flower Arrangement. The first row shows a clip of the robot failing to insert the second flower, while the next three rows show three matched successful clips. FERT effectively matched the clip of successfully inserting the second flower.
Figure 18: Visualize the retrieval effect in Clothes Folding. The top row shows a failure segment in which the robot attempts to fold a T-shirt in half but fails to grasp its right corner with the right gripper. The following rows below show the corresponding retrieved segments. With an appropriate feature representation, FERT retrieves the same critical step across clothes of different colors.
Figure 19: Visualize the retrieval effect in Phone Charging. It illustrates the most easily mistaken step in the phone charging: inserting the charger plug into the socket, which requires high precision.
Figure 20: Visualize the retrieval effect in Stool Assembly. The figure shows a failed attempt to place a screw into a screw hole alongside the corresponding retrieved successful segments.
Figure 21: Visualize the retrieval effect in Phone Packaging. The figure illustrates the lid-placement stage: placing the lid over the box at an angle. This step requires high precision despite limited visibility. The top two rows show a failed attempt in which the left arm repeatedly adjusts the lid’s angle, but the lid still topples over. The three examples below show successful attempts.
Robot learning has largely relied on human-teleoperated demonstrations to acquire effective learnable behaviors. However, human-operated data collection processes can be unintuitive, difficult to scale, and inherently asynchronous. We explore an alternative: distilling physical behavior from general-purpose multimodal foundation models into deployable robot policies by using the foundation model itself as an autonomous demonstrator. While sufficiently capable models can generate successful zero-shot manipulation trajectories, repeatedly invoking them during physical execution is slow and expensive, limiting their utility as scalable data generators. As a solution, we introduce CAPEX, an experience-conditioned demonstration collection framework that uses execution experience from previous attempts to adapt how frequently the foundation model must observe, reason, and replan. We evaluate across RoboCasa tasks and on physical Franka and bimanual YAM-arm platforms, measuring task success, model calls, token usage, collection time, and cost. We further train Diffusion Policy and ACT on matched sets of human-teleoperated and foundation-model-generated demonstrations to evaluate the downstream learning value of autonomously collected data. We find that CAPEX increases the number of successful demonstrations by 4.3x while reducing the cost per successful demonstration by 80%. Policies trained on CAPEX-generated data approach the performance of those trained on matched human demonstrations; with longer training, this gap largely closes for policies trained from scratch. These results suggest that foundation models can serve as scalable sources of reusable robot experience. Project page: https://capex-paper.github.io/
Shivam Aarya, Zhang Xi-Jia, Chengyue Huang +6
School of Interactive Computing Georgia Institute of Technology Atlanta, GA 30332, USA
Generalist robot policies increasingly benefit from large-scale pretraining, but offline data alone is insufficient for robust real-world deployment. Deployed robots encounter distribution shifts, long-tail failures, task variations, and human correction opportunities that fixed demonstration datasets cannot fully capture. We present Learning While Deploying (LWD), a fleet-scale offline-to-online reinforcement learning framework for continual post-training of generalist Vision-Language-Action (VLA) policies. Starting from a pretrained VLA policy, LWD closes the loop between deployment, shared physical experience, policy improvement, and redeployment by using autonomous rollouts and human interventions collected across a robot fleet. To stabilize learning from heterogeneous, sparse-reward fleet data, LWD combines Distributional Implicit Value Learning (DIVL) for robust value estimation with Q-learning via Adjoint Matching (QAM) for policy extraction in flow-based VLA action generators. We validate LWD on a fleet of 16 dual-arm robots across eight real-world manipulation tasks, including semantic grocery restocking and 3--5 minute long-horizon tasks. A single generalist policy improves as fleet experience accumulates, reaching an average success rate of 95%, with the largest gains on long-horizon tasks.
Yi Wang, Xinchen Li, Pengwei Xie +13
Shanghai Innovation Institute. · AGIBOT Finch. · Columbia University.
Robot policies inevitably encounter failures when deployed in real environments. Naive retries often repeat the same mistakes, while many existing recovery methods rely on human intervention. In this paper, we propose Failure-Aware Retry (FAR), a framework that enables robots to learn from previous failures at test time, adapt their behavior accordingly, and eventually complete the task autonomously. FAR combines Failure-Contrastive Preference Adaptation, which constructs preference learning data from failures to steer the policy away from previously unsuccessful behaviors, with lightweight action perturbations during retries to encourage local exploration. We further incorporate successful recovery trajectories into a training loop for continual policy improvement. Experiments in both simulation and real-world manipulation tasks show that FAR substantially improves success rates and robustness, with average gains of 17.6% over the standard diffusion policy in simulation and 11.7% in the real world. In addition, FAR improves data efficiency under both reset and timestep budgets during continual policy improvement by exploiting informative failure cases. Videos and code are available at https://hoar012.github.io/FAR-Project.