Large-scale pretraining is reducing the amount of adaptation required to deploy embodied foundation models, but it is not guaranteed that a pretrained model already provide the level of performance required in every deployment. Practitioners therefore need to understand how well a pretrained model matches a target deployment and how much adaptation data is left to be collected. We study whether a small set of successful target-domain demonstrations can reveal whether and how much target-domain data will be required before any adaptation. We introduce FlowEval, a set of training-free metrics that use the flow-matching objective to probe an embodied foundation model with a set of demonstrations. FlowEval quantifies predictive disagreement on successful target demonstrations to assess how well a pretrained model fits a target deployment. Across four embodied foundation model families: Pi-0.5, MolmoAct2, GR00T, and DreamZero, FlowEval distinguishes seen from unseen deployments across real-world and simulated manipulation. Further, by finetuning Pi-0.5 and GR00T on 17 RoboCasa365 manipulation tasks, we showed that FlowEval correlates with how success rate improves as additional adaptation data is provided. Together, these results suggest that FlowEval can serve as a lightweight, training-free diagnostic of how much model--deployment adaptation may be required.
Figures & tables
Model
Metric
Step 1
Best
Relative
Pi05-Base
Cosine
0.868
0.885
98.0%
Velocity Error
1.097
1.104
99.4%
DreamZero-DROID
Action Cosine
0.455
0.498
91.3%
Action Velocity Error
0.244
0.267
91.3%
Video Cosine
0.799
1.170
68.3%
Video Velocity Error
0.735
1.141
64.4%
Table 1: Distinguishability score at step 1 compared with the best value observed across all steps. Relative performance is computed as 100× step 1 / Best. Evaluated on Robocasa365 target set.
In-Pretraining
DROID
Robocasa
YAM
LIBERO
Roboset
OpenArm
Franka
Pi-05
No
No
No
No
No
No
No
GR00T-N1.6
Yes
Yes
No
No
No
No
No
MolmoAct2
Yes
No
Yes
Yes
No
No
No
DreamZero
Yes
No
No
No
No
No
No
Table 2: The full set of model–dataset pairs on which FlowEval metrics are evaluated, along with whether each dataset was included in model pretraining. Information about those datasets is based on model releases [ 16 , 33 , 8 , 46 ] and, where applicable, company blogs [ 3 , 10 , 47 ] . ‘ No ’ indicates that we found no evidence that the dataset was included in pretraining, although the authors did not explicitly state that it was excluded. ‘No’ indicates that the authors explicitly stated that the dataset was excluded, or the pretraining dataset is publicly released and does not contain that dataset.
Figure 1: FlowEval results across various models and action representations.
Figure 2: Regression between success rate and metric values, with significances displayed in the plots. All interaction effects are significant. Under the GR00T-N1.5 500-demonstration condition, training instability causes 29.4% of runs to collapse; these runs are excluded from the analysis (Appendix A.5.5 ). Training collapse is negligible in other conditions.
Figure 3: Relation between performance gain and metric values. 500 - 50 demos success rate for Pi-05, 150 - 50 demos success rate for GR00T-N1.5.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Cosine similarity per layer, evaluted on Pi-05. Left: noisy actions, right: pure noise.
Task
Condition
Mean
Variance
Franka Object
Noisy actions
3.121
3.390
Pure noise
1.770
0.504
Franka Object+
Noisy actions
3.089
3.286
Pure noise
1.704
0.492
Franka Object Two
Noisy actions
3.198
3.274
Pure noise
1.775
0.513
Appendix
Table 3: Statistics under noisy-action and pure-noise conditions on 4 collected Franka tasks.
Model
Action Cosine
Action VE
Action MSE
MSE / Cosine
MSE / VE
π0.5
0.816
1.07
1.10
1.35 ×
1.03 ×
DreamZero
0.498
0.267
0.179
0.359 ×
0.670 ×
Appendix
Table 4: Distinguishability score of Action MSE compared with FlowEval Metrics
Step
0
1
2
3
4
5
6
7
Cosine (Action)
0.498
0.455
0.398
0.398
0.398
0.398
0.323
0.323
VE (Action)
0.267
0.244
0.205
0.205
0.205
0.205
0.144
0.144
Cosine (Video)
1.17
0.799
0.799
0.799
0.799
0.799
1.07
1.07
VE (Video)
0.956
0.735
0.723
0.723
0.723
0.723
1.14
1.14
Step
8
9
10
11
12
13
14
15
Cosine (Action)
0.323
0.323
0.231
0.231
0.231
0.224
0.234
0.231
Appendix
Table 5: DreamZero 16 diffusion steps
Step
0
1
2
3
4
5
6
7
8
9
Cosine
0.816
0.868
0.862
0.875
0.886
0.869
0.858
0.847
0.836
0.833
VE
1.07
1.10
1.10
1.10
1.10
1.10
1.10
1.10
1.10
1.09
Appendix
Table 6: Pi-05 10 diffusion steps
Figure 5: Regression results in Sec. 4.2 , recomputed with action MSE (left).
Figure 6: FlowEval results across various models and action representations on the reconstructed franka dataset.
Figure 7: DreamZero comparison with hold still baseline on Franka and DROID.
Checkpoint
Action Channel
DreamZero-DROID
Relative joint position
GR00T-N1.6-DROID
Relative joint position
GR00T-N1.5-DROID
Absolute joint position; joint velocity
GR00T-N1.5
Delta Cartesian
Pi-0.5-Base
Delta joint position
Pi-0.5-DROID
Joint velocity
Appendix
Table 7: Action channels the checkpoints are pretrained on [ 16 , 8 , 33 , 46 ] .
Figure 8: Additional results of FlowEval rating unseen deployment settings.
Checkpoint
Seen reference
Unseen deployment
KS statistic
p -value
Sig.
Velocity error
Pi-0.5-RoboCasa
RoboCasa-pretrain
DROID
0.940
3.88×10−114
Pi-0.5-RoboCasa
RoboCasa-pretrain
MolmoAct2-YAM
1.000
2.57×10−112
Pi-0.5-RoboCasa
RoboCasa-pretrain
OpenArm
1.000
4.05×10−162
Pi-0.5-LIBERO
LIBERO
RoboCasa-target
0.775
6.08×10−229
MolmoAct2
MolmoAct2-YAM
RoboCasa-pretrain
1.000
5.24×10−112
Appendix
Table 8: Two-sample Kolmogorov–Smirnov tests comparing FlowEval score distributions on seen-reference and unseen datasets. All tested unseen deployments are statistically distinguishable from their corresponding seen-reference distributions.
Shift
Model
Demos
Cosine similarity
Velocity error / loss
With
Pi05-base
50/150/500
+0.474 ( .0047 )
−0.0382 ( .0093 )
With
GR00T-N1.5
50/150
−5.537 ( 1.81×10−8 )
+2.363 ( 3.12×10−8 )
Without
Pi05-RoboCasa
50/150/500
+0.275 ( .3112 )
−0.0382 ( .1056 )
Without
GR00T-N1.6
50/150/500
+0.266 ( .4521 )
−0.1102 ( .4464 )
Appendix
Table 9: Initial-metric × demonstration-budget interaction coefficients, with p -values in parentheses. Bold denotes p<.05 .
Deployment shift
Model
Cosine similarity
Velocity error / loss
With
Pi05-base
+0.474 ( .0047 )
−0.0382 ( .0093 )
With
GR00T-N1.5
−0.0073 ( .9964 )
−0.0258 ( .9699 )
Without
Pi05-RoboCasa
+0.275 ( .3112 )
−0.0382 ( .1056 )
Without
GR00T-N1.6
+0.266 ( .4521 )
−0.1102 ( .4464 )
Appendix
Table 10: Initial-metric × demonstration-budget interaction coefficients, with p -values in parentheses; bold denotes p<.05 . Positive cosine coefficients associate higher initial cosine similarity with larger gains; negative loss coefficients associate lower initial loss with larger gains.
Table 11: Cross-model task consistency of success and FlowEval metrics. Strong success consistency confirms shared task effects in RoboCasa; corresponding FlowEval consistency would indicate that these effects could induce the FlowEval–performance relationship.
Model
Demos
Metric
β3
95% CI
p
Pi05-Base
50/150/500
Cosine similarity (Original)
0.4738
[0.1455,0.8022]
0.00468
Cosine similarity (Controlled)
0.2595
[−0.04683,0.5659]
0.09684
Pi05-Base
50/150/500
Velocity error (Original)
−0.03820
[−0.06699,−0.00942]
0.009283
Velocity error (Controlled)
−0.02106
[−0.04391,0.001801]
0.07099
GR00T-N1.5
50/150
Cosine similarity (Original)
−5.537
[−7.465,−3.609]
1.81×10−8
Cosine similarity (Controlled)
−5.468
[−7.563,−3.373]
3.12×10−7
Appendix
Table 12: Control for baseline success in the FlowEval scaling analysis. Reported coefficients are the FlowEval metric ×log(demos) interaction β3 . The controlled specification additionally includes 50-demo success S50 and its interaction with log(demos) . Coefficient magnitudes should be compared before and after adjustment within each metric rather than across metrics.
Model
Demonstrations
Collapsed runs
Fraction of runs
GR00T-N1.5
500
5
29.4% (5/17)
GR00T-N1.5
150
1
5.88% (1/17)
GR00T-N1.6 (Action Expert)
All
0
0.00% (0/51)
Pi-05-Base
All
1
1.96% (1/51)
Pi-05-RoboCasa
All
2
3.92% (2/51)
Appendix
Table 13: Runs exhibiting persistent or sudden performance collapse.
Figure 9: Performance curves for GR00T-N1.5 finetuning. Training collapse occurs on task 18 with 150 demonstrations and on tasks 2, 10, 12, 16, and 17 with 500 demonstrations.