cs.RONov 4, 2026

RACE: Calibrated Cumulative Scoring for Rollout-Free Robot Policy Checkpoint Selection

Authors: Changyeon Kim, Yekyung Nah, Kimin Lee

Organizations: KAIST · Config

Abstract

Assessing the deployment performance of learned robot policies during training is challenging, as directly measuring task success requires costly real-world trials for each candidate checkpoint. Averaged validation loss offers a rollout-free alternative, but its checkpoint rankings can be inconsistent with deployment performance. We analyze two limitations of this criterion: it penalizes action deviations without considering whether the policy can recover from them, and gives later reference predictions equal weight even when earlier deviations may lead execution away from the reference. We propose RACE, a novel rollout-free metric for ranking checkpoints using a fixed set of held-out successful trajectories. RACE calibrates action deviations against state-conditional thresholds derived from action differences at similar states in these trajectories. We aggregate these scores through a running product, so that a substantial deviation reduces the contribution of subsequent reference predictions. Averaging the resulting cumulative scores over time and across trajectories yields the checkpoint score. In simulation, RACE outperforms baselines across validation sources, while several baselines degrade sharply when the validation set changes. On real robots, its rankings align with deployment success across the evaluated policy backbones and tasks of varying difficulty. Perturbation analyses further show that RACE distinguishes deviations with markedly different effects on task success, even when baselines respond similarly.

Figures & tables

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Beyond Binary Success: Sample-Efficient and Statistically Rigorous Robot Policy Comparison

    Mar 13, 2026David Snyder, Apurva Badithela, Nikolai Matni +4Robotic Policy EvaluationSequential Hypothesis Testing

  2. Critical Interval MSE: Toward Reliable Offline Validation for Robot Manipulation Policies

    Jun 29, 2026Haoxu Huang, Tongsam Zheng, Yifan Chen +2Robot Policy LearningOff-Policy Evaluation