Imitation learning typically treats every demonstration state as equally important, optimizing a behavior-cloning objective uniformly across the dataset. Yet not all states are created equal: in a small number of critical states, even a slight deviation from the expert action can lead to a failure, while in many other states such errors are recoverable and inconsequential. We formalize state criticality in imitation learning as the worst-case degradation in downstream value induced by a bounded perturbation of the expert action at a given state. We then propose SCRIM, a practical State CRIticality Measure that estimates this quantity by training an adversarial disturbance policy and a value critic around a fixed imitation policy. SCRIM assigns each demonstration state a score reflecting how vulnerable the task outcome is to small, policy-relevant action errors. We show that these scores can be used to reweight behavior-cloning losses, improving imitation at consequential states and increasing overall task performance. We further show that SCRIM enables data curation by selecting compact, high-value subsets of demonstrations that preserve performance while substantially reducing the amount of training data. We evaluate SCRIM across various simulated and real-world robotic manipulation tasks, and demonstrate that SCRIM improves policy training, enables efficient data selection, and provides interpretable insights into which moments in a demonstration are most critical for task success. Project website: https://sia-lab-git.github.io/SCRIM/
Figures & tables
Figure 3: Faucet models ordered by mean SCRIM over the demonstrations, from the two easiest (top) to the two hardest (bottom).
Figure 5: SCRIM distinguishes a brittle (left) from a robust (right) cube-stacking demonstration.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
StackCube
TurnFaucet
PegInsertion
Hammer
Disturbance bound ϵ (per axis)
0.005
0.005
0.002
0.20
SCRIM adversary
200k
100k
1M
500k
Appendix
Table 1: Disturbance bounds and interaction budgets. Environment interactions used to train the SCRIM adversary behind Table 1 baselines.
StackCube
TurnFaucet
PegInsertion
Hammer
ESS% target τ
44.7
40.6
69.7
50.0
Temperature T⋆
1.97
1.72
3.75
5.62
Appendix
Table 2: Sampling target. Top: selected ESS% target τ and resulting temperature T⋆ for SCRIM. Bottom: success rate (%) of SCRIM-weighted BC across ESS% targets.
ϵ
0
0.001
0.003
0.005
0.01
0.02
0.05
Success (%)
72.0
64.0
42.0
24.0
14.0
0.0
0.0
Appendix
Table 3: Random-residual calibration (StackCube). Success rate (%) of the frozen base policy under uniform random residuals of size ϵ .
ϵ
0.003
0.005
0.01
0.02
0.05
StackCube (uniform BC 68.0)
Success (%)
83.0
91.0
73.0
77.0
73.0
TurnFaucet (uniform BC 39.0)
Success (%)
42.0
47.0
37.0
42.0
42.0
Appendix
Table 4: Sensitivity to the perturbation bound. Success rate (%) of SCRIM-weighted BC for each ϵ at a fixed ESS% target.
StackCube
TurnFaucet
PegInsertion
Hammer
Gated fraction k
5%
5%
30%
10%
Clean success (%)
69.0
29.0
13.0
66.0
Adversary at high-SCRIM states
13.0
13.0
13.0
20.0
Adversary at random states
6.0
4.0
6.0
1.0
Uniform ϵ -noise at high-SCRIM states
1.0
0.0
0.0
3.0
Appendix
Table 5: Score-gated disturbance. Drop in success rate (percentage points) relative to clean rollouts when the disturbance is applied at a fraction k of steps.
StackCube
TurnFaucet
PegInsertion
Hammer
Noise scale c=0
13.0
13.0
13.0
20.0
c=0.003
16.0 (14.3)
12.0 (4.1)
13.0 (12.2)
20.0 (18.4)
c=0.01
14.0 (4.3)
9.0 (1.2)
8.0 (3.7)
19.0 (5.5)
c=1
3.0 (0.04)
−1.0 (0.01)
5.0 (0.04)
2.0 (0.06)
Halfway checkpoint
4.0
1.0
5.0
23.0
Random states (reference)
6.0
4.0
6.0
1.0
Appendix
Table 6: Degraded critics. Drop in success rate (percentage points) from the adversary at high-SCRIM states, with the score SNR in parentheses, at the gated fractions of Table 5 .