Does Validation Loss Predict Real-Robot Success? Evidence from Four Imitation-Learning Policies on a Low-Cost Arm

Simon Cowling

cs.ROOct 30, 2026

Does Validation Loss Predict Real-Robot Success? Evidence from Four Imitation-Learning Policies on a Low-Cost Arm

Authors: Simon Cowling

Organizations: fysical.ai, Denmark

Abstract

Held-out validation loss is the standard offline signal for choosing checkpoints and comparing models, but its relation to closed-loop success on a real robot is rarely measured. We measure it for ACT, Diffusion Policy, SmolVLA and $π_{0.5}$ (action expert only), trained on nested subsets of 12 to 200 demonstrations of a pick-and-place task on an SO-101 arm. We score 82 checkpoints on 25 held-out demonstrations, with each policy's training loss and with the error of the actions it outputs, and compare the scores with 930 bench rollouts. We also score all 700 intermediate checkpoints of 70 training runs, and recover the ball positions of every training, held-out and bench episode from the front camera. Within each policy, hold-out action error correlates with bench success (Spearman $ρ$ from -0.23 to -0.88), but no more strongly than the number of training episodes alone, and not at all among seeds trained on the same data. Across policies the ranking is inverted ( $ρ$ = +0.90 at 200 episodes): the policy with the lowest hold-out error succeeded in 3 of 110 rollouts. The most common failure, the arm never leaving its rest pose, ended 96% of rollouts for Diffusion Policy trained on untrimmed demonstrations and 47% for SmolVLA, and cannot occur in a hold-out set made of demonstrations. Among rollouts in which the arm moved, SmolVLA's hold-out action error predicts success well ( $ρ$ = -0.81). For the diffusion and flow-matching policies, hold-out loss rises two- to five-fold during training while hold-out action error falls by up to 29%. Held-out metrics are a sound diagnostic of learning but no substitute for evaluation on the robot, and we recommend reporting initiation failure and success among started rollouts as separate bench outcomes.

Figures & tables

Policy	Training objective	What the loss compares	Randomness at scoring time
ACT	L1 regression of an action chunk (plus a KL term in training only)	predicted actions with demonstrated actions, normalised	none: deterministic in evaluation mode
Diffusion Policy	denoising (DDPM noise prediction, MSE)	the noise the network predicts with the noise that was added to a demonstrated chunk	noise level and noise sample; averaged over 4 fixed-seed draws
SmolVLA	flow matching (MSE on velocity)	the predicted direction back to the clean chunk with the true direction	time step and noise sample; 4 draws
$\pi_{0.5}$ (action expert only)	flow matching (MSE on velocity)	as SmolVLA	as SmolVLA

Table 1: Training objective of each policy and what its loss compares.

Model line	Training steps	Checkpoints saved	Seeds per budget	Cells in the bench analysis
ACT	100,000	every 10,000	4	20
Diffusion Policy, untrimmed data	100,000	every 10,000	4 trained, 2 benched	10 (+2 EMA)
Diffusion Policy, trimmed data	100,000	every 10,000	2 (seeds 1, 3)	10 (+4 EMA)
SmolVLA	20,000	every 2,000	4	20
$\pi_{0.5}$ , action expert only	6,000	3,000 and 6,000	1 plain + 2 EMA	15 (5 plain, 10 EMA)

Table 2: Training configurations. A training cell is one combination of model line, budget and seed.

Figure 1: Bench success (top) and hold-out action error (bottom) by number of training episodes. Grey dots are single checkpoints; bars are 95% Wilson intervals on the pooled rate. Error horizons differ between columns, so compare shapes, not levels.

Model line	Checkpoints	$\rho$ , hold-out error vs success	Expected from episodes alone [95% range]	$\rho$ within budget (p)
ACT	20	$-0.88$	$-0.89$ [ $-0.95$ , $-0.82$ ]	+0.28 (0.82)
$\pi_{0.5}$ expert-only	15	$-0.55$	$-0.50$ [ $-0.80$ , $-0.13$ ]	$-0.22$ (0.85)
SmolVLA	20	$-0.68$	$-0.62$ [ $-0.78$ , $-0.42$ ]	$-0.08$ (0.69)
Diffusion Policy, trimmed	10	$-0.48$	$-0.48$ [ $-0.76$ , $-0.16$ ]	+0.11 (0.75)
Diffusion Policy, untrimmed	10	$-0.23$	$-0.34$ [ $-0.70$ , +0.29]	$-0.16$ (1.00)

Table 3: Rank correlation between hold-out action error and bench success across each model line’s checkpoints, the range the episode count alone would produce, and the correlation among checkpoints trained on the same number of episodes (with its permutation p-value).

Figure 2: Observed correlations (filled) against the range the episode count alone would produce (grey), and the correlation among checkpoints trained on the same number of episodes (hollow).

Figure 3: Hold-out action error against bench success for each model line from 12 to 200 episodes. If validation predicted success across models, the lines would run from top left to bottom right.

Figure 4: Outcome of every bench rollout by model variant (standard dim lighting, placements N00 to N09). As in [ 5 ] , late starts and rollouts without a grasp attempt count as other failures.

Figure 5: SmolVLA and

\pi_{0.5}

by budget: hold-out action error (left), success over all rollouts (middle), and success over the rollouts in which the arm started (right).

Figure 6: Each row pairs a model trained without EMA (hollow) with a model trained separately with EMA on the same data, budget and seed (filled), scored with its averaged weights.

Figure 7: The same Diffusion Policy cells trained with and without the idle lead-in.

Figure 8: Hold-out loss as a multiple of its first-checkpoint value (top, log scale) and hold-out action error as a change from its first-checkpoint value (bottom), against training progress. Shades of blue are the episode budgets.

Figure 9: ACT: hold-out (blue) and training-probe (grey) loss (top, log scale) and action error (bottom) at every budget. Thin lines are seeds, bold lines their mean; the ring marks the lowest hold-out mean.

Figure 10: Diffusion Policy, untrimmed data, in the same layout. Hold-out loss rises from the first checkpoint while hold-out action error falls.

Figure 11: SmolVLA in the same layout (20,000 training steps).

Model line	12 $\rightarrow$ 200 episodes (final step)	10 $\times$ more steps at 12 episodes	10 $\times$ more steps at 200 episodes	Hold-out loss over training, 12 / 200 episodes
ACT	$-40$ %	$-4$ %	$-2$ %	$\times$ 1.0 / $\times$ 1.0
Diffusion Policy, untrimmed	$-39$ %	$-2$ %	$-29$ %	$\times$ 3.5 / $\times$ 4.7
Diffusion Policy, trimmed	$-40$ %	+3%	$-18$ %	$\times$ 2.8 / $\times$ 5.3
SmolVLA	$-31$ %	+2%	$-12$ %	$\times$ 2.8 / $\times$ 2.0

Table 4: Change in hold-out action error from more data and from more training steps, and the change in hold-out loss over training.

Figure 12: Hold-out action error by episode count at the first, middle and final checkpoint. The vertical gap between lines is what training longer bought; the slope is what more data bought.

Figure 13: Green ball start positions in all 225 recorded episodes, coloured by the smallest budget that includes them, with the hold-out (diamonds) and the ten bench placements (stars). Dotted rings mark balls placed from a partial view.

Figure 14: Training green balls in each budget (dots, shaded by their area), with hold-out balls (diamonds) and bench placements (stars).

Figure 15: The ten bench layouts over the 200 training positions, with each placement’s success rate over all rollouts in standard (dim) lighting (106 per placement).

Figure 16: Bench success rate of each placement at each budget against the distance from its green ball to the nearest training green ball. Marker size is the number of rollouts.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Component	Description	Settings
Demonstrations ( fysical/greenball_pool_225_v1 )	225 teleoperated demonstrations; a trimmed copy has the idle lead-in removed	30 fps; front and wrist cameras, 640 $\times$ 480
Data split ( subsets.json )	25 hold-out episodes and nested training subsets S12 to S200	data seed 20260922; hold-out stratified over nine recording batches
Scorer ( val_loss.py )	loss and action error per checkpoint, overall and per episode	evaluation mode; 4 noise draws, seed 20261007; probe S12; plain or EMA weights; rejects any model whose training set contains a hold-out episode
Final-checkpoint scores	128 checkpoints, on the hold-out and the training probe	every frame ( $\pi_{0.5}$ every fifth frame, bfloat16)
Training-curve scores	700 intermediate checkpoints of ACT, Diffusion Policy and SmolVLA	every fifth frame; NVIDIA A10G
Bench record	one row per rollout with its audited outcome label, from [ 5 ]	primary set: standard (dim) lighting, placements N00 to N09

Appendix

Table A1: Data, scoring components and settings.

Robotic Policy Evaluation

Robotic policy evaluation measures whether learned controllers succeed reliably, generalize across conditions, recover from failures, and behave safely.

See all 134 papers

Oct 27, 2026

From 12 to 200 Demonstrations: How ACT, Diffusion Policy, SmolVLA and $π_{0.5}$ Scale on a Low-Cost Arm

We trained four widely used imitation-learning policies, ACT, Diffusion Policy, SmolVLA and

π_{0.5}

, on nested subsets of 12, 25, 50, 100 and 200 teleoperated demonstrations of one pick-and-place task with distractor objects, using one fixed training recipe per policy. Every trained model was then run on a physical SO-101 arm on a fixed set of object placements: 1,340 scored episodes in 126 blocks. ACT, trained from scratch, rose from 1 success in 40 episodes at 12 demonstrations to 37 in 40 at 100.

π_{0.5}

with its action expert fine-tuned was the best policy when data was scarce (16 of 40 at 12 demonstrations, 28 of 40 at 25) and then stayed near 70%; from 100 demonstrations ACT was ahead, by a margin this study does not fully resolve. SmolVLA and Diffusion Policy peaked near 50%, and both were affected by a failure to start: SmolVLA's arm never started moving in 47% of episodes, and Diffusion Policy hardly ever started until an idle lead-in was trimmed from the demonstrations. For

π_{0.5}

the fine-tuning recipe mattered more than the amount of data: low-rank adaptation succeeded in 4 of 110 episodes, against 130 of 200 for action-expert fine-tuning. From the joint and camera recordings we place every episode on a four-stage ladder (started, reached the ball, carried it, delivered it) and show that for ACT more data moves failures from reaching, to grasping, to none, while the failures

π_{0.5}

retains are in grasping. We also measure how weak a ten-episode block is as a measurement (the same checkpoint tested twice differs by 2 successes in 10 on average), separate seed-to-seed variation from that noise, and audit the human scoring against the recordings. Training length was fixed within each recipe, so the amount of data and the number of times it was seen change together; we state which conclusions survive that confound and which do not.

Simon Cowling

fysical.ai, Denmark

Imitation Learning Robot Imitation Learning Robotic Manipulation Robotic Policy Evaluation Diffusion Policy

Ask PDF

Nov 2, 2026

Rollout then Recycle: 3,718 Labelled Real-Robot Episodes from Benchmarking 7 Policies on 12 Tasks

We release all 3,718 episodes from the ArmnetBench v0.1 evaluation (arXiv:2607.24481), which contains 600 teleoperated demonstrations and 3,118 rollouts of seven imitation-learning and vision-language-action policies on 12 tabletop tasks, recorded on low-cost SO-101 arms; 73% of the rollouts are failures. Every policy for a task was trained or fine-tuned on the same 50 demonstrations and rolled out on the same cell, so the corpus contains many policies' failures on shared tasks. An operator labelled every rollout successful, suboptimal or failure. Each episode has three camera views, joint states and actions and is available on the Hugging Face Hub in LeRobot v3.0 and a reward-model format. This note documents how the episodes were collected and the limitations that could affect their use.

Praveen Selvaraj, Lorenzo Uttini, Ville Kuosmanen

Armnet · Imperial College London

Robot Manipulation Datasets Robot Manipulation Benchmarks Robotic Policy Evaluation

Ask PDF

Jun 29, 2026

Critical Interval MSE: Toward Reliable Offline Validation for Robot Manipulation Policies

Real-world evaluation is the gold standard for robot manipulation policies because it tests them against the physical conditions and deployment challenges they are ultimately designed to handle. However, real-world evaluation is also the bottleneck for iterating on robot imitation learning policies: it is costly, difficult to reproduce, and often too sparse to reliably compare nearby model variants. A straightforward proxy for performance is validation loss on expert demonstrations, but this proxy is often poorly correlated with real-world performance. In this paper, we introduce Critical Interval MSE (CI-MSE), an intuitively simple yet effective offline validation metric. CI-MSE uses a zero-shot agent to localize task-critical segments, restricts error computation to those segments, and applies simple action-alignment procedures that better match rollout-time behavior. Across simulation and real-world experiments, CI-MSE generally yields a stronger correlation between validation error and rollout performance than raw MSE. Across 27 distinct policy checkpoints, CI-MSE achieves a Spearman rank correlation of

-0.87

, closer to the ideal value of

-1

than raw MSE's

-0.61

, demonstrating a significant improvement. Sensitivity analysis shows that our metric is robust across a wide range of hyperparameter settings. We further study the effectiveness of CI-MSE under evaluation distribution shifts and identify practical boundaries for its use. In summary, this paper provides a simple and reliable offline validation tool for accelerating policy iteration. Project webpage: https://ci-mse.github.io/

Haoxu Huang, Tongsam Zheng, Yifan Chen +2

Tsinghua University · Shanghai Qi Zhi Institute · Spirit AI

Robot Policy Learning Off-Policy Evaluation Robotic Policy Evaluation Policy Evaluation Robotic Manipulation

Ask PDF

Abstract

Robotic Policy Evaluation

Explore similar work

From 12 to 200 Demonstrations: How ACT, Diffusion Policy, SmolVLA and π0.5π_{0.5}π0.5​ Scale on a Low-Cost Arm

Rollout then Recycle: 3,718 Labelled Real-Robot Episodes from Benchmarking 7 Policies on 12 Tasks

Critical Interval MSE: Toward Reliable Offline Validation for Robot Manipulation Policies

From 12 to 200 Demonstrations: How ACT, Diffusion Policy, SmolVLA and $π_{0.5}$ Scale on a Low-Cost Arm