Vision-language-action (VLA) models are trained as general-purpose foundation models for robot learning, and require fine-tuning for each downstream task and robot. This raises practical questions such as how much data and training updates are needed, how checkpoints should be selected, and whether simulation can be leveraged to alleviate the cost of real data collection. We study these questions on a task that requires both precise manipulation and spatial understanding of the scene. In this task, an SO-101 arm moves a chess king between two squares that are given only by their names, such as e4 and a5. We fine-tune π0.5 with various dataset sizes and update budgets, using teleoperated demonstrations as well as simulated demonstrations from a randomized scripted expert that imitates the variability of human demonstrations. In simulation, the success rate scales with both data and updates. Importantly, we find that a short learning-rate cool-down adds up to 31 points and is necessary for comparing checkpoints, and that validation loss and action error are unreliable for model selection. On the real arm, we fine-tune on up to 256 teleoperated demonstrations and evaluate ten policies in 25 trials each. Partial results indicate that pretraining on the simulated demonstrations helps substantially, most clearly when real data is scarce. We also present a negative result, in which training on the policy's own rollouts with filtered behavior cloning, RECAP-style advantage conditioning, corrections from an automatic coach, or residual reinforcement learning did not yield a measurable improvement over plain behavior cloning in our setting. We release the simulator, the expert and the real demonstrations.
Figures & tables
Figure 1: The five stages of a move on the real arm (top) and in simulation (bottom), labeled with the stage tokens the policy predicts.
Figure 2: Gripper paths seen from above, from the randomized scripted expert, teleoperated demonstrations, trials of the simulation policy, and the first trial of eight of the real policies of Section 3.3 . Dashed paths indicate unsuccessful trials.
Figure 3: Inputs and outputs of the policy at one frame of a real trial. The policy observes the two rectified camera images, the joint positions, the instruction and the current stage. It outputs a chunk of 50 joint targets and predicts whether to transition to the next stage.
Figure 4: Success rate of the cooled-down models over updates, one curve per dataset size, with Wilson 95% intervals. Dashed lines mark the end of each size’s first epoch.
Figure 5: Constant-rate checkpoints against the cooled-down branches started from them.
Figure 6: Validation loss (left) and open-loop action error (middle) of the constant-rate trunks, against the success rate of the cooled-down models (right). One curve per dataset size.
Figure 7: Share of the 1,000 trials that reach each milestone (blue, strict success dashed) and that topple the piece or time out (orange, timeouts dashed), at 10k demonstrations. Left: cooled-down models. Right: constant-rate checkpoints.
Figure 8: The value function of the 1K+1K round. Left: predicted value (solid) and actual return (dashed) on held-out episodes. Right: distribution of the advantage over the training frames, on a symmetric-log axis. Rollout frames right of the dashed threshold are labeled positive; demonstration frames are labeled positive regardless. One tick is one control step, 1/15 s.
Figure 9: Partial results in the real setting, 25 trials per model. Left: policies initialized from the simulation policy. Right: policies initialized from the base π0.5 model. Top: success rate (%) over demonstrations and updates, where a dash marks a model without a full evaluation. Bottom: share of trials that reach each milestone, per evaluated model.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Both camera frames as recorded (raw) and rectified to the simulator’s pinhole model, as the policy sees them.
Model
Demos
Updates
Success
π0.5 , behavior cloning
100
165k
9.9
π0.5 , behavior cloning
316
165k
22.0
π0.5 , behavior cloning
1,000
165k
31.7
π0.5 , behavior cloning
3.2k
165k
39.7
π0.5 , behavior cloning
10k
165k
60.2
π0.5 , behavior cloning
10k
237k
64.1
Appendix
Table 1: Simulation results: success rate (% of 1,000 trials). Top: behavior cloning, with models cooled down unless noted. Bottom: learning from 1,000 of the policy’s own rollouts.
Pretrained in simulation
Base
Demos
Updates
Touched
Gripped
Success
Touched
Gripped
Success
32
2k
25
12
7
6
0
0
128
45k
25
24
18
15
8
2
256
15k
25
23
14
23
13
5
256
45k
25
22
14
23
17
10
256
75k
25
23
17
24
15
10
Appendix
Table 2: Evaluation on the real arm, as the number of the 25 trials that reached each milestone.
Figure 11: Accuracy (left) and cross-entropy (right) of the predicted stage on held-out demonstrations.
Figure 12: Error of the 1K+1K value function during its single training epoch, on held-out and on training episodes. One tick is 1/15 s.
Figure 13: Learning from the policy’s own rollouts, cooled-down models, each sampled with the positive label.
Figure 14: Success of the outcome-label RECAP policy at 79k updates, sampled with different labels and guidance strengths β . Shaded: the Wilson 95% interval of behavior cloning.
Figure 15: Partial results of the pretraining ablation, for cooled-down models at 10k demonstrations. Left: π0.5 with and without robot pretraining, trained with the same recipe. Right: other model families with their stock recipes.
Dataset
Source
Episodes
Contents
king moves, real
teleoperation
261
the demonstrations of Section 3.3
Appendix
Table 3: Released dataset.
Milestone
Observed
0
none
the jaws never touched the king
1
touched
a jaw touched the king
2
gripped
the king was between both jaws, touching both
3
lifted
gripped and lifted clear of the board (about 1.5 cm)
4
over target
carried until its center was above the target square
5
success
at the end the king stands, released, its center inside the target square
Appendix
Table 4: Milestones graded by the evaluator. Success is milestone 5 or 6, strict success milestone 6.
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China. · Huazhong University of Science and Technology, Wuhan, China. · Westlake University, Hangzhou, China. +1