Large-scale pretraining is reducing the amount of adaptation required to deploy embodied foundation models, but it is not guaranteed that a pretrained model already provide the level of performance required in every deployment. Practitioners therefore need to understand how well a pretrained model matches a target deployment and how much adaptation data is left to be collected. We study whether a small set of successful target-domain demonstrations can reveal whether and how much target-domain data will be required before any adaptation. We introduce FlowEval, a set of training-free metrics that use the flow-matching objective to probe an embodied foundation model with a set of demonstrations. FlowEval quantifies predictive disagreement on successful target demonstrations to assess how well a pretrained model fits a target deployment. Across four embodied foundation model families: Pi-0.5, MolmoAct2, GR00T, and DreamZero, FlowEval distinguishes seen from unseen deployments across real-world and simulated manipulation. Further, by finetuning Pi-0.5 and GR00T on 17 RoboCasa365 manipulation tasks, we showed that FlowEval correlates with how success rate improves as additional adaptation data is provided. Together, these results suggest that FlowEval can serve as a lightweight, training-free diagnostic of how much model--deployment adaptation may be required.
Figures & tables
Model
Metric
Step 1
Best
Relative
Pi05-Base
Cosine
0.868
0.885
98.0%
Velocity Error
1.097
1.104
99.4%
DreamZero-DROID
Action Cosine
0.455
0.498
91.3%
Action Velocity Error
0.244
0.267
91.3%
Video Cosine
0.799
1.170
68.3%
Video Velocity Error
0.735
1.141
64.4%
Table 1: Distinguishability score at step 1 compared with the best value observed across all steps. Relative performance is computed as 100× step 1 / Best. Evaluated on Robocasa365 target set.
In-Pretraining
DROID
Robocasa
YAM
LIBERO
Roboset
OpenArm
Franka
Pi-05
No
No
No
No
No
No
No
GR00T-N1.6
Yes
Yes
No
No
No
No
No
MolmoAct2
Yes
No
Yes
Yes
No
No
No
DreamZero
Yes
No
No
No
No
No
No
Table 2: The full set of model–dataset pairs on which FlowEval metrics are evaluated, along with whether each dataset was included in model pretraining. Information about those datasets is based on model releases [ 16 , 33 , 8 , 46 ] and, where applicable, company blogs [ 3 , 10 , 47 ] . ‘ No ’ indicates that we found no evidence that the dataset was included in pretraining, although the authors did not explicitly state that it was excluded. ‘No’ indicates that the authors explicitly stated that the dataset was excluded, or the pretraining dataset is publicly released and does not contain that dataset.
Figure 1: FlowEval results across various models and action representations.
Figure 2: Regression between success rate and metric values, with significances displayed in the plots. All interaction effects are significant. Under the GR00T-N1.5 500-demonstration condition, training instability causes 29.4% of runs to collapse; these runs are excluded from the analysis (Appendix A.5.5 ). Training collapse is negligible in other conditions.
Figure 3: Relation between performance gain and metric values. 500 - 50 demos success rate for Pi-05, 150 - 50 demos success rate for GR00T-N1.5.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Cosine similarity per layer, evaluted on Pi-05. Left: noisy actions, right: pure noise.
Task
Condition
Mean
Variance
Franka Object
Noisy actions
3.121
3.390
Pure noise
1.770
0.504
Franka Object+
Noisy actions
3.089
3.286
Pure noise
1.704
0.492
Franka Object Two
Noisy actions
3.198
3.274
Pure noise
1.775
0.513
Appendix
Table 3: Statistics under noisy-action and pure-noise conditions on 4 collected Franka tasks.
Model
Action Cosine
Action VE
Action MSE
MSE / Cosine
MSE / VE
π0.5
0.816
1.07
1.10
1.35 ×
1.03 ×
DreamZero
0.498
0.267
0.179
0.359 ×
0.670 ×
Appendix
Table 4: Distinguishability score of Action MSE compared with FlowEval Metrics
Step
0
1
2
3
4
5
6
7
Cosine (Action)
0.498
0.455
0.398
0.398
0.398
0.398
0.323
0.323
VE (Action)
0.267
0.244
0.205
0.205
0.205
0.205
0.144
0.144
Cosine (Video)
1.17
0.799
0.799
0.799
0.799
0.799
1.07
1.07
VE (Video)
0.956
0.735
0.723
0.723
0.723
0.723
1.14
1.14
Step
8
9
10
11
12
13
14
15
Cosine (Action)
0.323
0.323
0.231
0.231
0.231
0.224
0.234
0.231
Appendix
Table 5: DreamZero 16 diffusion steps
Step
0
1
2
3
4
5
6
7
8
9
Cosine
0.816
0.868
0.862
0.875
0.886
0.869
0.858
0.847
0.836
0.833
VE
1.07
1.10
1.10
1.10
1.10
1.10
1.10
1.10
1.10
1.09
Appendix
Table 6: Pi-05 10 diffusion steps
Figure 5: Regression results in Sec. 4.2 , recomputed with action MSE (left).
Figure 6: FlowEval results across various models and action representations on the reconstructed franka dataset.
Figure 7: DreamZero comparison with hold still baseline on Franka and DROID.
Checkpoint
Action Channel
DreamZero-DROID
Relative joint position
GR00T-N1.6-DROID
Relative joint position
GR00T-N1.5-DROID
Absolute joint position; joint velocity
GR00T-N1.5
Delta Cartesian
Pi-0.5-Base
Delta joint position
Pi-0.5-DROID
Joint velocity
Appendix
Table 7: Action channels the checkpoints are pretrained on [ 16 , 8 , 33 , 46 ] .
Figure 8: Additional results of FlowEval rating unseen deployment settings.
Checkpoint
Seen reference
Unseen deployment
KS statistic
p -value
Sig.
Velocity error
Pi-0.5-RoboCasa
RoboCasa-pretrain
DROID
0.940
3.88×10−114
Pi-0.5-RoboCasa
RoboCasa-pretrain
MolmoAct2-YAM
1.000
2.57×10−112
Pi-0.5-RoboCasa
RoboCasa-pretrain
OpenArm
1.000
4.05×10−162
Pi-0.5-LIBERO
LIBERO
RoboCasa-target
0.775
6.08×10−229
MolmoAct2
MolmoAct2-YAM
RoboCasa-pretrain
1.000
5.24×10−112
Appendix
Table 8: Two-sample Kolmogorov–Smirnov tests comparing FlowEval score distributions on seen-reference and unseen datasets. All tested unseen deployments are statistically distinguishable from their corresponding seen-reference distributions.
Shift
Model
Demos
Cosine similarity
Velocity error / loss
With
Pi05-base
50/150/500
+0.474 ( .0047 )
−0.0382 ( .0093 )
With
GR00T-N1.5
50/150
−5.537 ( 1.81×10−8 )
+2.363 ( 3.12×10−8 )
Without
Pi05-RoboCasa
50/150/500
+0.275 ( .3112 )
−0.0382 ( .1056 )
Without
GR00T-N1.6
50/150/500
+0.266 ( .4521 )
−0.1102 ( .4464 )
Appendix
Table 9: Initial-metric × demonstration-budget interaction coefficients, with p -values in parentheses. Bold denotes p<.05 .
Deployment shift
Model
Cosine similarity
Velocity error / loss
With
Pi05-base
+0.474 ( .0047 )
−0.0382 ( .0093 )
With
GR00T-N1.5
−0.0073 ( .9964 )
−0.0258 ( .9699 )
Without
Pi05-RoboCasa
+0.275 ( .3112 )
−0.0382 ( .1056 )
Without
GR00T-N1.6
+0.266 ( .4521 )
−0.1102 ( .4464 )
Appendix
Table 10: Initial-metric × demonstration-budget interaction coefficients, with p -values in parentheses; bold denotes p<.05 . Positive cosine coefficients associate higher initial cosine similarity with larger gains; negative loss coefficients associate lower initial loss with larger gains.
Table 11: Cross-model task consistency of success and FlowEval metrics. Strong success consistency confirms shared task effects in RoboCasa; corresponding FlowEval consistency would indicate that these effects could induce the FlowEval–performance relationship.
Model
Demos
Metric
β3
95% CI
p
Pi05-Base
50/150/500
Cosine similarity (Original)
0.4738
[0.1455,0.8022]
0.00468
Cosine similarity (Controlled)
0.2595
[−0.04683,0.5659]
0.09684
Pi05-Base
50/150/500
Velocity error (Original)
−0.03820
[−0.06699,−0.00942]
0.009283
Velocity error (Controlled)
−0.02106
[−0.04391,0.001801]
0.07099
GR00T-N1.5
50/150
Cosine similarity (Original)
−5.537
[−7.465,−3.609]
1.81×10−8
Cosine similarity (Controlled)
−5.468
[−7.563,−3.373]
3.12×10−7
Appendix
Table 12: Control for baseline success in the FlowEval scaling analysis. Reported coefficients are the FlowEval metric ×log(demos) interaction β3 . The controlled specification additionally includes 50-demo success S50 and its interaction with log(demos) . Coefficient magnitudes should be compared before and after adjustment within each metric rather than across metrics.
Model
Demonstrations
Collapsed runs
Fraction of runs
GR00T-N1.5
500
5
29.4% (5/17)
GR00T-N1.5
150
1
5.88% (1/17)
GR00T-N1.6 (Action Expert)
All
0
0.00% (0/51)
Pi-05-Base
All
1
1.96% (1/51)
Pi-05-RoboCasa
All
2
3.92% (2/51)
Appendix
Table 13: Runs exhibiting persistent or sudden performance collapse.
Figure 9: Performance curves for GR00T-N1.5 finetuning. Training collapse occurs on task 18 with 150 demonstrations and on tasks 2, 10, 12, 16, and 17 with 500 demonstrations.
Robot foundation models (RFMs), including vision-language-action (VLA) policies, are often discussed through a scaling view: more data, larger models, and broader benchmarks should improve generalization. In robotics, however, a model can generalize while work still remains before it can run on a robot with a particular body. The work required differs across methods and target robots, and those differences affect practical deployment. We call the gap between reusable models, representations, or data and their use in execution on the target robot the embodiment gap. This survey examines what can be reused across robot embodiments and what must still be implemented on a new robot. We place existing methods on a two-axis map that shows the type of shared structure and the stage at which adaptation is needed for execution on the target robot. We then examine recent work through three overlapping research directions: sharing semantics and perception, sharing robot data and interfaces, and learning correspondence across embodiments. We also propose a reporting framework for adaptation work that success rate alone does not reveal. The framework identifies the work that should be checked when comparing cross-embodiment learning and highlights work that remains on a new robot and questions for future study.
Yukiyasu Domae, Keisuke Shirai, Hanbit Oh +7
National Institute of Advanced Industrial Science and Technology (AIST), Japan
We present and tackle two problems associated with deploying Foundation Models (FMs) on Embodied Agents performing navigation: 1) Training bias in FMs leading to poor personalization in unseen environments, and 2) Limited FM context length hindering success, especially on long horizon tasks. Our solution for the former involves priming the FM with human-habit data mined from the scene and our solution for the latter involves active memory management via a novel `memory head' augmentation. We first present a taxonomy of existing literature on FM-based Embodied Navigation, and highlight these limitations. We then present our approaches, Transit-Aware Planning (TAP) and MemCtrl to address the limitations. With TAP, we present real-world results in a lab environment with a Turtlebot for personalized target finding that shows an average improvement of 18% over a non-TAP baseline. On MemCtrl, we report a 6% average improvement across various embodied tasks, with 20% on long instruction subsets, all while using nearly half the context used in the baseline model. Motivated by these result, we present our stance the deployability of FM-based embodied agents in real-world environments, and highlight open research directions.
Embodied foundation models are expected to benefit from data scaling like large language models, but face a much tighter data bottleneck. Teleoperated real-robot trajectories remain the dominant pretraining source due to their precise action supervision and embodiment alignment, yet their scalability is limited by high collection cost, acquisition difficulty, and low behavioral and environmental diversity. These limitations have sparked interest in egocentric human video as a scalable, substantially lower-cost, and more diverse alternative for embodied model pretraining. However, its effectiveness compared to teleoperated real-robot data remains underexplored. To address this question, we conduct a systematic study comparing egocentric human video and teleoperated real-robot trajectories as pretraining data sources for embodied foundation models, under fixed post-training and validation protocols. Surprisingly, we find that egocentric data, when processed through a carefully designed filtering and labeling pipeline, is not merely a viable substitute for model pretraining but can lead to superior performance. With the same amount of pretraining data, models pretrained on egocentric data achieve a 24% lower validation loss on real-robot action prediction, as well as 52.5% and 90% higher success rates on in-distribution and out-of-distribution real-robot task execution, respectively. This finding verifies a scalable paradigm for embodied foundation models: pretrain on egocentric human video to learn diverse world representations, then adapt with a small amount of labeled real-robot data for action-space alignment. We hope this study encourages broader exploration of egocentric data and offers guidance for data quality assessment before costly robot data collection.