From 12 to 200 Demonstrations: How ACT, Diffusion Policy, SmolVLA and $π_{0.5}$ Scale on a Low-Cost Arm

Simon Cowling

cs.ROOct 27, 2026

From 12 to 200 Demonstrations: How ACT, Diffusion Policy, SmolVLA and $π_{0.5}$ Scale on a Low-Cost Arm

Authors: Simon Cowling

Organizations: fysical.ai, Denmark

Abstract

We trained four widely used imitation-learning policies, ACT, Diffusion Policy, SmolVLA and $π_{0.5}$ , on nested subsets of 12, 25, 50, 100 and 200 teleoperated demonstrations of one pick-and-place task with distractor objects, using one fixed training recipe per policy. Every trained model was then run on a physical SO-101 arm on a fixed set of object placements: 1,340 scored episodes in 126 blocks. ACT, trained from scratch, rose from 1 success in 40 episodes at 12 demonstrations to 37 in 40 at 100. $π_{0.5}$ with its action expert fine-tuned was the best policy when data was scarce (16 of 40 at 12 demonstrations, 28 of 40 at 25) and then stayed near 70%; from 100 demonstrations ACT was ahead, by a margin this study does not fully resolve. SmolVLA and Diffusion Policy peaked near 50%, and both were affected by a failure to start: SmolVLA's arm never started moving in 47% of episodes, and Diffusion Policy hardly ever started until an idle lead-in was trimmed from the demonstrations. For $π_{0.5}$ the fine-tuning recipe mattered more than the amount of data: low-rank adaptation succeeded in 4 of 110 episodes, against 130 of 200 for action-expert fine-tuning. From the joint and camera recordings we place every episode on a four-stage ladder (started, reached the ball, carried it, delivered it) and show that for ACT more data moves failures from reaching, to grasping, to none, while the failures $π_{0.5}$ retains are in grasping. We also measure how weak a ten-episode block is as a measurement (the same checkpoint tested twice differs by 2 successes in 10 on average), separate seed-to-seed variation from that noise, and audit the human scoring against the recordings. Training length was fixed within each recipe, so the amount of data and the number of times it was seen change together; we state which conclusions survive that confound and which do not.

Figures & tables

Component	Description
Follower arm	SO-101 follower ( Knight et al., 2024 ) : five arm joints and a gripper, six Feetech STS3215 bus servos. Driven through LeRobot’s so101_follower with joint angles in degrees and the gripper on a 0 to 100 scale. Silicone tape was added to the inner gripper surfaces to increase friction. The follower arm was mounted on a custom raised platform to simulate mounting on a lerobot Kiwi.
Leader arm	SO-101 leader used for teleoperation (mounted on the same type of platform as the follower arm).
Cameras	Two USB cameras read through OpenCV at 640 $\times$ 480 pixels and 30 frames per second: front , fixed and facing the workspace, and wrist , mounted on the gripper. Note that an overhead camera was specifically not used as subsequent research will be on a mobile platform where this would not be possible.
Evaluation computer	Linux workstation with an NVIDIA RTX 5070 Ti (16 GB), 64GB ram, Intel Core Ultra 7 270K CPU. All policy inference ran locally on this machine.
Training hardware	Hugging Face Jobs, lerobot-gpu container pinned by digest: A10G for ACT and Diffusion Policy, A100 for SmolVLA and $\pi_{0.5}$
Software	LeRobot at commit 8352ca62 (21 Sep 2026, development version 0.6.2), with the peft extra added for $\pi_{0.5}$

Table 1: Hardware, software and workspace.

Figure 1: Study design. Every model at a given budget is trained on the same demonstrations, and every model is tested on the same ten placements.

Recipe	Starting point	Weights trained	Steps $\times$ batch	Learning rate	Actions predicted / executed per chunk
ACT	From scratch; ResNet-18 image encoder initialised from ImageNet	All	100,000 $\times$ 8	1e-5, constant	100 / 100
Diffusion Policy, untrimmed demos	From scratch; ResNet-18 image encoder initialised from ImageNet	All	100,000 $\times$ 64	1e-4, cosine decay	48 / 24
Diffusion Policy, trimmed demos	As above, trained on the trimmed pool	All	100,000 $\times$ 64	1e-4, cosine decay	48 / 24
SmolVLA	Pretrained smolvla_base	Action expert; vision-language model frozen	20,000 $\times$ 64	1e-4, cosine decay	50 / 50
$\pi_{0.5}$ , LoRA	Pretrained pi05_base	Rank-32 low-rank adapters on the action expert: 2.7 million weights, 0.07% of the model	6,000 $\times$ 32	2.5e-5, decaying to 2.5e-6	50 / 50
$\pi_{0.5}$ , expert-only	Pretrained pi05_base	Whole action expert: 693 million of 4.14 billion weights, 16.7% of the model; vision-language model frozen	6,000 $\times$ 32	2.5e-5, decaying to 2.5e-6	50 / 50

Table 2: Training recipes. Six recipes for four policies. Steps, batch size and learning-rate schedule were the same at every data budget.

	LoRA (first grid)	Expert-only (second grid)
Base model	lerobot/pi05_base , 4,143,404,816 weights: a PaliGemma vision-language model and a smaller action expert	Same
What is trained	Rank-32 adapters (alpha 32) on the query and value attention projections in the 18 layers of the action expert, and on its four small action and time projections	Every weight of the action expert and its projections; no adapters
What stays frozen	Every original weight: the whole vision-language model, and in the action expert the feed-forward blocks, the normalisation layers and the other attention projections	The whole vision-language model, both vision encoder and language model
Trainable weights	2,705,408, or 0.065% of the model	693,422,112, or 16.7% of the model
Optimiser	AdamW, weight decay 0.01, gradient clipping at 1.0	Same
Learning rate	2.5e-5 after 1,000 warm-up steps, then cosine decay to 2.5e-6 at step 6,000	Same

Table 3: The two

\pi_{0.5}

fine-tuning recipes. Parameter counts are those printed by the training script.

Recipe	Episodes	Blocks	Models tested	Test dates (2026)	Front-camera lux, range (median)
ACT	300	20	20	23–25 Sep	223–394 (313)
Diffusion Policy, untrimmed demos	110	11	10	28 Sep–1 Oct	247–337 (278)
SmolVLA	200	20	20	29 Sep	135–304 (262)
Diffusion Policy, trimmed demos	150	15	10	1–7 Oct	207–366 (240)
$\pi_{0.5}$ , LoRA	110	11	11	5–6 Oct	214–362 (272)
$\pi_{0.5}$ , expert-only	200	20	15	7–8 Oct	117–209 (169)

Table 4: Blocks in the main comparison. Dim light, standard protocol.

Outcome	Definition	Episodes
Success	Green ball delivered to the basket within 30 seconds	463
Initiation failure	The arm never started	282
Location failure	The gripper closed one or more times, never with the ball between its jaws	259
Grip failure	The ball was between the jaws at a close and was then lost	161
Collision	The arm knocked the ball out of play (79), or collided with the basket (25), the table (19) or the camera (4)	127
Other	Wrong ball picked (18), still holding at 30 seconds (9), missed the basket (7), dropped in transit (5), started too late (5), no grasp attempt (4)	48

Table 5: Outcome classes over all 1,340 scored episodes.

Figure 2: Success rate against number of demonstrations for the four working recipes.

Recipe	12 demos	25 demos	50 demos	100 demos	200 demos	Odds ratio per doubling
ACT	1/40 (3; 0–13)	11/40 (28; 16–43)	18/40 (45; 31–60)	37/40 (93; 80–97)	35/40 (88; 74–95)	3.89 (2.70–5.62)
$\pi_{0.5}$ , expert-only	16/40 (40; 26–55)	28/40 (70; 55–82)	27/40 (68; 52–80)	30/40 (75; 60–86)	29/40 (73; 57–84)	1.37 (1.15–1.63)
SmolVLA	2/40 (5; 1–17)	4/40 (10; 4–23)	21/40 (53; 37–67)	14/40 (35; 22–50)	15/40 (38; 24–53)	1.61 (1.21–2.14)
Diffusion Policy, trimmed demos	0/20 (0; 0–16)	6/20 (30; 15–52)	25/50 (50; 37–63)	6/30 (20; 10–37)	12/30 (40; 25–58)	1.32 (0.88–1.97)
Diffusion Policy, untrimmed demos	0/20 (0; 0–16)	1/20 (5; 1–24)	0/20 (0; 0–16)	0/30 (0; 0–11)	2/20 (10; 3–30)	1.79 (0.48–6.70)
$\pi_{0.5}$ , LoRA	1/30 (3; 1–17)	2/30 (7; 2–21)	1/30 (3; 1–17)	0/10 (0; 0–28)	0/10 (0; 0–28)	0.71 (0.40–1.28)

Table 6: Success by recipe and number of demonstrations. Each cell gives successes over episodes and, in brackets, the rate with its 95% Wilson interval in percent. The last column is the odds ratio per doubling of data, with its 95% interval.

Figure 3: Outcome of every episode, by recipe and number of demonstrations.

Recipe	Episodes	Started	Reached the ball	Carried it	Delivered it
ACT	300	300	208	157	150
$\pi_{0.5}$ , expert-only	200	200	177	135	130
SmolVLA	200	106	75	58	56
Diffusion Policy, trimmed demos	150	135	80	55	49
Diffusion Policy, untrimmed demos	110	4	3	3	3
$\pi_{0.5}$ , LoRA	110	110	39	13	4

Table 7: Episodes reaching each stage of the task, all budgets pooled.

Recipe	12 demos	25 demos	50 demos	100 demos	200 demos
ACT	4/60 (74)	24/60 (51)	33/60 (45)	53/60 (18)	51/60 (27)
$\pi_{0.5}$ , expert-only	24/40 (38)	33/40 (23)	31/40 (25)	37/40 (19)	34/40 (26)
SmolVLA	3/18 (75)	7/12 (37)	21/31 (24)	17/22 (28)	17/20 (14)
Diffusion Policy, trimmed demos	3/11 (108)	12/19 (38)	36/46 (28)	3/30 (120)	12/27 (46)
$\pi_{0.5}$ , LoRA	1/30 (92)	7/28 (79)	3/30 (135)	1/10 (83)	0/9 (119)

Table 8: First grasp attempt: closes with the ball between the jaws, over episodes in which the gripper closed. In brackets, the median distance in pixels between the ball and the jaw centre at that close, in a 320-pixel-wide wrist view.

Recipe	12 demos	25 demos	50 demos	100 demos	200 demos	All
ACT	0/60	0/60	0/60	0/60	0/60	0/300
$\pi_{0.5}$ , expert-only	0/40	0/40	0/40	0/40	0/40	0/200
$\pi_{0.5}$ , LoRA	0/30	0/30	0/30	0/10	0/10	0/110
SmolVLA	22/40	28/40	8/40	17/40	19/40	94/200
Diffusion Policy, untrimmed demos	20/20	19/20	20/20	29/30	18/20	106/110
Diffusion Policy, trimmed demos	9/20	1/20	4/50	0/30	1/30	15/150

Table 9: Episodes in which the arm never started, over episodes run.

Figure 4: SmolVLA by number of demonstrations: episodes that started, and success counted over started episodes and over all episodes.

Change	Compared	With the change	Without	Difference
Trimming the idle lead-in (Diffusion Policy)	All budgets	49/150	3/110	+30 (21 to 38)
Expert-only instead of LoRA ( $\pi_{0.5}$ )	All budgets	130/200	4/110	+61 (53 to 68)
EMA weights ( $\pi_{0.5}$ expert-only)	Seed 0, five budgets, same day, same placements (paired)	33/50	34/50	$-$ 2 ( $-$ 19 to 15)
EMA enabled in training (Diffusion Policy, trimmed)	Same four cells	15/40	31/80	$-$ 1 ( $-$ 18 to 17)
Averaged instead of final weights (Diffusion Policy, same run)	Two checkpoints (paired)	6/20	8/20	$-$ 10 ( $-$ 37 to 17)
Bright light (Diffusion Policy, untrimmed)	Five checkpoints	8/50	2/50	+12 ( $-$ 0.2 to 25)

Table 10: Effect of recipe and condition changes on success. Differences are in percentage points with 95% intervals that treat episodes as independent, so they understate the uncertainty where blocks disagree. The two rows marked paired compare the same placements and use an interval for paired differences.

Variant	Trainable weights	Peak learning rate	Successes
LoRA, standard recipe (the grid model)	2,705,408	2.5e-5	0/10
LoRA, ten times the learning rate	2,705,408	2.5e-4	5/10
Wide LoRA: adapters on the attention and feed-forward layers of both the language model and the action expert	53,282,816	2.5e-5	trained, not tested
Wide LoRA, ten times the learning rate	53,282,816	2.5e-4	trained, not tested
Expert-only	693,422,112	2.5e-5	9/10

Table 11:

\pi_{0.5}

pilots on one cell: 100 demonstrations, seed 0, ten episodes. All variants use 6,000 steps at batch 32.

Figure 5: Every block of the four working recipes, by seed. Colour gives the training seed, and within a budget the models also run left to right in seed order. For

\pi_{0.5}

the three models are seed 0 without EMA (circles, tested twice), seed 0 with EMA and seed 3 with EMA (squares). Diffusion Policy has seeds 1 and 3 only.

Recipe	Models per budget	12 demos	25 demos	50 demos	100 demos	200 demos	Spread between models	Spread between re-tests
ACT	4	1.00	0.60	0.27	0.049	1.00	1.4 (p = 0.16)	not re-tested
$\pi_{0.5}$ , expert-only	3	0.91	0.27	0.45	0.80	0.72	1.3 (p = 0.25)	1.4 (p = 0.21)
SmolVLA	4	1.00	0.89	0.022	0.69	0.057	1.6 (p = 0.06)	not re-tested
Diffusion Policy, trimmed demos	2	1.00	1.00	0.042	< 0.001	0.69	3.0 (p = 0.02)	2.2 (p = 0.07)

Table 12: Do seeds differ? For each budget, the p-value of an exact test that all models at that budget share one success rate. The last two columns give the spread between blocks as a ratio to what independent draws would produce (1 means no excess), first between different models and then between repeat tests of the same checkpoint.

Recipe	Actions per chunk	Control rate (Hz)	Pause per chunk (s)	Chunk jump ( ∘ )	Time to start (s)	Length of a success (s)	Successes whose first close was on the ball
ACT	100	29.9	0.01	3.1	1.1	16.5	127/150
$\pi_{0.5}$ , expert-only	50	27.3	0.17	1.9	1.3	16.8	121/130
SmolVLA	50	28.7	0.08	1.5	3.6	22.0	50/56
Diffusion Policy, trimmed demos	24	28.7	0.04	2.6	1.1	17.1	41/49
$\pi_{0.5}$ , LoRA	50	27.0	0.19	4.5	3.0	n/a	n/a

Table 13: Control and timing, episodes in which the arm started. Medians. The pause is the stall at each new action chunk, estimated from the frames recorded in full-length episodes. The chunk jump is the step in commanded joint angles where one chunk ends and the next begins, measured during the reach.

Recipe	Samples drawn	12 demos	25 demos	50 demos	100 demos	200 demos
Diffusion Policy, trimmed demos	6.4 million	1,256	603	302	151	75
Diffusion Policy, untrimmed demos	6.4 million	1,038	498	249	125	62
SmolVLA	1.28 million	208	100	50	25	12
ACT	0.8 million	130	62	31	16	7.8
$\pi_{0.5}$ , both recipes	0.19 million	31	15	7.5	3.7	1.9

Table 14: Passes over the training data. Samples drawn are steps times batch size.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Recipe	Model	12 demos	25 demos	50 demos	100 demos	200 demos
ACT	Seed 0	0	1	2	7	9
ACT	Seed 1	0	3	6	10	9
ACT	Seed 2	1	3	6	10	8
ACT	Seed 3	0	4	4	10	9
$\pi_{0.5}$ , expert-only	Seed 0, no EMA	4 $\rightarrow$ 5	4 $\rightarrow$ 8	9 $\rightarrow$ 6	6 $\rightarrow$ 8	8 $\rightarrow$ 7
$\pi_{0.5}$ , expert-only	Seed 0, EMA	3	9	5	8	8

Appendix

Table 15: Successes out of ten for every block in the main comparison, by seed. Where a checkpoint was tested more than once, the runs are listed in order with arrows.

Change	Episodes	Basis
Success to failure	6	Ball still in the gripper at 30 seconds (video, and the timeout rule)
Success to failure	4	The arm never moved
Success to failure	4	The ball never left its start position, or was knocked aside and never carried
Success to failure	2	Grip failure seen on video
Failure to success	6	Ball carried and delivered, on video or in the camera data
Failure class changed	5	Arm movement and gripper data contradicted the label

Appendix

Table 16: Changes made by the scoring audit.

Probe	Checkpoints	Episodes	Successes	Never started
Bright light, ACT (200 demos, seed 0)	1	10	7	0
Bright light, Diffusion Policy untrimmed (one per budget)	5	50	8	34
Bright light, SmolVLA (one per budget)	5	50	10	14
Diffusion Policy trimmed, EMA run, averaged weights (50 and 100 demos, seeds 1 and 3)	4	40	15	1
Diffusion Policy trimmed, EMA run, final weights (50 and 100 demos, seed 3)	2	20	8	1
Diffusion Policy untrimmed, EMA run (50 demos seed 1, 200 demos seed 3)	2	20	5	8

Appendix

Table 17: Supplementary blocks outside the main comparison.

Recipe	N00	N01	N02	N03	N04	N05	N06	N07	N08	N09	p
ACT	8/20	9/20	15/20	11/20	8/20	9/20	12/20	11/20	9/20	10/20	0.53
$\pi_{0.5}$ , expert-only	12/20	16/20	16/20	6/20	10/20	10/20	12/20	17/20	15/20	16/20	0.002
SmolVLA	3/8	8/10	6/11	2/11	8/11	4/6	11/14	6/11	5/9	3/15	0.008
Diffusion Policy, trimmed demos	1/14	5/13	5/15	4/11	5/13	5/14	8/15	6/13	5/12	5/15	0.56

Appendix

Table 18: Successes over episodes for each of the ten placements, all budgets pooled. For SmolVLA and Diffusion Policy only episodes in which the arm started are counted.

Recipe	Settings	Hardware	Time per run
ACT	Default configuration: chunk size 100, 100 actions executed per chunk, variational objective, ResNet-18 encoder with ImageNet weights, learning rate 1e-5, batch 8, 100,000 steps	A10G (small)	4.6 h
Diffusion Policy	Two observation steps, horizon 48, 24 actions executed per chunk, ResNet-18 encoder with ImageNet weights, learning rate 1e-4 with cosine schedule and 500 warm-up steps, batch 64, 100,000 steps	A10G (large)	13.4 h
SmolVLA	lerobot/smolvla_base , action expert trained, chunk size 50, learning rate 1e-4 with cosine decay, batch 64, 20,000 steps	A100 (large)	2.7 h
$\pi_{0.5}$ , LoRA	lerobot/pi05_base , LoRA rank 32 and alpha 32 on the library’s default targets (query and value attention projections of the action expert, plus the action and time projections), 2,705,408 trainable weights; AdamW, learning rate 2.5e-5, betas 0.9 and 0.95, weight decay 0.01, gradient clipping 1.0; 1,000 warm-up steps, cosine decay to 2.5e-6 at step 6,000; batch 32; 6,000 steps; 32-bit training with gradient checkpointing; checkpoints at 3,000 and 6,000 steps; cameras mapped to the base and left-wrist slots; images 224 $\times$ 224; 50-action chunks; quantile normalisation	A100 (large)	10.6 h
$\pi_{0.5}$ , expert-only	As LoRA but with no adapters: the library’s train_expert_only and freeze_vision_encoder options are set, so the action expert and projections are trained in full and the vision-language model is frozen; 693,422,112 trainable weights	A100 (large)	10.6 h

Appendix

Table 19: Training configurations.

Imitation Learning

Imitation learning trains agents to perform tasks by learning from expert demonstrations, feedback, or observed behavior.

See all 265 papers

Does Imitation Learning Preserve Temporal Robustness in Dexterous Manipulation? An Expert-Learner Comparison Across Task Execution Speeds
Sep 1, 2026Clinton Enwerem, John S. Baras, Calin BeltaContact-Rich Robotic Manipulation Robotic Policy Evaluation
Institute for Systems Research University of Maryland College Park, MD, U.S.A.
Where Small Mistakes Matter: Quantifying State Criticality for Imitation Learning
Oct 13, 2026Yusuf Umut Ciftci, Joonwon Kang, Ajay Suresha Sathya +1Imitation Learning Robot Imitation Learning
University of Southern California · Stanford University · Georgia Institute of Technology +1
How to Better Train VLAs: Lessons Learned From the REAL-I Challenge at ICRA 2026
Sep 12, 2026Jiaming Wang, Jizhuo Chen, Diwen Liu +11Robot Policy Adaptation Robotic Policy Evaluation

Sep 1, 2026J/K move · Enter open · S save

Does Imitation Learning Preserve Temporal Robustness in Dexterous Manipulation? An Expert-Learner Comparison Across Task Execution Speeds

Clinton Enwerem, John S. Baras, Calin Belta

Institute for Systems Research University of Maryland College Park, MD, U.S.A.

Contact-Rich Robotic Manipulation Robotic Policy Evaluation Robotic Manipulation Robot Imitation Learning

Open paper AskPDF arXiv

Dexterous manipulation policies learned by imitation are typically evaluated for robustness to variation in scenes, objects, or instructions, but their performance across task execution speeds is less often examined. This leaves open how much temporal robustness a learner retains relative to the expert it imitates. We compare an expert and learner under the same task conditions, initial-condition draws, and speedup factors. We instantiate the evaluation in ParcelStow, a contact-rich task in which the robot acquires, reorients, and inserts a parcel. The demonstrations span the speedup range for the manipulation phases after parcel acquisition. A scripted expert and an Action Chunking with Transformers (ACT) policy trained from the expert's demonstrations both achieve 100 percent task success at nominal speed. Their success rates diverge within the demonstrated range: at its maximum, expert success is 84 percent and ACT success is 53 percent. Two ACT policies with different parameter initializations show similar degradation, decreasing by 34 and 48 percentage points from nominal speed to the maximum demonstrated speed, compared with 16 points for the expert. Stage-level analysis shows that 35 of ACT's 47 failures at the maximum demonstrated speed are insertion misalignments. Under the relative-motion handoff, every ACT acquisition retains the parcel through reorientation and transfer in free space, but only 64 percent complete the overall task, compared with 95 percent after expert acquisition. Across all evaluated policies and speeds, none of the 414 acquisitions without force closure completes the task. Equal nominal task success therefore does not imply preservation of expert performance across execution speeds. Code, data, and evaluation scripts are available at https://github.com/coenwerem/parcelstow.

Abstract

Imitation Learning

Explore similar work

Does Imitation Learning Preserve Temporal Robustness in Dexterous Manipulation? An Expert-Learner Comparison Across Task Execution Speeds

Where Small Mistakes Matter: Quantifying State Criticality for Imitation Learning

How to Better Train VLAs: Lessons Learned From the REAL-I Challenge at ICRA 2026