A vision--language--action (VLA) policy can complete complex tasks while ignoring the evidence that should determine its actions. An object held near the wrist camera can displace the instructed target. Language and action show the same pattern: a familiar noun can trigger the operation it was paired with in training even after the verb changes, and a gripper that closed on nothing may lift anyway. We call these dependencies modality shortcuts: regularities in successful demonstrations make visual, lexical, or motor cues sufficient to predict expert actions without the task evidence needed for the underlying decision. More demonstrations of the same kind can raise task success while leaving these shortcuts intact. We propose Perturbot which makes task-relevant evidence easier to use and shortcuts insufficient on their own: it applies task-preserving wrist-view perturbations, enriches instructions with decision-relevant captions, and adds random and failed trajectory segments relabeled with the behavior they contain. It complements scaling by changing what is scaled, and leaves inference unchanged. Moreover, we propose GroundingFscore, an offline score that diagnoses how severely a policy relies on modality shortcuts. Task success rate shows whether a policy improves, while GroundingFscore reveals whether the policy scales healthily, relying on task evidence rather than shortcuts. Together, Perturbot and GroundingFscore provide a training-and-evaluation framework for disentangling VLA decisions from shortcut priors while preserving responsiveness to task-relevant evidence.
Figures & tables
Figure 1: Modality shortcuts and Perturbot . Top: familiar visual, lexical, and motor cues can replace task evidence (solid: hypothetical shortcut choices; dashed: required alternatives). Bottom: task-preserving wrist-view perturbations (V), behavior-supported caption enrichment (C), and screened, relabeled trajectory segments (R; Section 3.2 ). Examples are conceptual; relabeled local behavior does not by itself establish recovery.
Figure 2: GroundFscore from paired inputs. (a) For each family, a null edit must leave the action unchanged (an off-path decoy, a paraphrase, another approach to the same pose), and a causal edit must change it by a known target Δae⋆ (the target moved by Δp , another object named, a grasp that closed on nothing). (b) Rc normalizes the predicted change δ ; Sc projects it onto Δae⋆ and is clipped to [0,1] . (c) Policies with equal success on the demonstration distribution can differ sharply in GroundFscore .
Figure 3: Real-robot General PnP setup. Left: the material pool and the six objects and three containers used in the task. Middle: an example stored layout, specified by each item’s position and orientation. Right: an instruction-conditioned scene; every policy is tested on the same 50 stored cases.
Task success ↑
GroundFscore ↑
Failure ↓
Method
G1
G2
G3
G4
G5
SR
Prog.
GFV
GFL
GFA
SC
MI
π0.5 fine-tune
40 ± 30
40 ± 30
50 ± 30
40 ± 30
40 ± 30
42 ± 14
2.71 ± 0.55
0.28 ± 0.04
0.24 ± 0.03
0.19 ± 0.03
68 ± 13
82 ± 11
+ V
50 ± 30
50 ± 30
60 ± 30
50 ± 30
50 ± 30
52 ± 14
3.12 ± 0.54
0.51 ± 0.04
0.26 ± 0.03
0.21 ± 0.03
36 ± 13
80 ± 11
+ C
60 ± 30
50 ± 30
50 ± 30
50 ± 30
60 ± 30
54 ± 14
3.20 ± 0.53
0.30 ± 0.04
0.47 ± 0.04
0.22 ± 0.03
62 ± 14
78 ± 11
+ R
50 ± 30
60 ± 30
50 ± 30
60 ± 30
60 ± 30
56 ± 14
3.34 ± 0.52
0.29 ± 0.04
0.27 ± 0.03
0.44 ± 0.04
64 ± 13
44 ± 14
+ V + C
70 ± 30
60 ± 30
60 ± 30
60 ± 30
70 ± 30
64 ± 13
3.62 ± 0.50
0.55 ± 0.04
0.50 ± 0.04
0.24 ± 0.03
30 ± 13
76 ± 12
Table 1: Main results on real-robot General PnP with a π0.5 backbone. G1–G5 are success rates (%) on five blocks of ten cases, SR is success over all 50 cases, and Prog. is the mean 0–5 progress score. GFV , GFL , and GFA are GroundFscore readouts on 100 held-out observations ( Section 3.3 ); SC and MI are failure rates (%) on salience-capture and motor-inertia probes of 50 trials each, separate from the 50 cases. ± is the half-width of the 95% bootstrap interval over the evaluation units: the ten cases of a block, the 50 cases for SR and Prog., the 100 observations for GroundFscore , and the 50 probe trials for SC and MI. V, C, and R denote wrist-view perturbation, detailed captions, and trajectory expansion; the upper block covers all eight combinations and the lower block the controls. Bold marks the best value in each column; the shaded row is the full method.
Full
Clean2Random
Method
Clean
Rand.
Avg.
Clean
Rand.
Avg.
π0 ( Black et al., 2025b )
65.9
58.4
62.2
–
–
–
π0.5 ( Black et al., 2025a )
82.7
76.8
79.8
70.7
46.0
58.4
ABot-M0 ( Yang et al., 2026b )
81.2
80.4
80.8
–
–
–
X-VLA ( Zheng et al., 2026 )
–
–
–
68.0
20.9
44.5
X-WAM ( Guo et al., 2026 )
–
–
–
70.0
25.8
47.9
Table 2: RoboTwin 2.0 results (task-averaged success, %). Avg. is the mean of clean and randomized scenes. Published results are as reported by Cai et al. (2026b) , rounded to one decimal, and – marks an unreported setting; Table 7 lists every published result, including systems with higher reported scores. The lower block uses our π0.5 fine-tuning recipe. Bold marks the best value in each column; the shaded row is the full method.
Figure 4: Scaling and ablation effects with a π0.5 backbone. (a) Success (left) and mean GroundFscore (right) on a log-scaled demonstration budget; Table 4 compares their slopes. (b) Mean GroundFscore gain per family from adding V, C, or R, averaged over four contrasts with the other components fixed ( Table 1 ). (c) Change in success from the selected rate as one sampling rate varies ( Table 10 ); shading shows the loss, not uncertainty.
Table 7Table 8
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Real-robot platform. Two AgileX PiPER X arms are mounted side by side and face the tabletop workspace. A RealSense D435 head camera on the central vertical post provides the third-person view, and a RealSense wrist camera beside each gripper provides a wrist view.
Full
Clean2Random
Method
Clean
Rand.
Avg.
Clean
Rand.
Avg.
π0 ( Black et al., 2025b )
65.9
58.4
62.2
–
–
–
π0.5 ( Black et al., 2025a )
82.7
76.8
79.8
70.7
46.0
58.4
ABot-M0 ( Yang et al., 2026b )
81.2
80.4
80.8
–
–
–
Motus ( Bi et al., 2026 )
88.7
87.0
87.8
–
–
–
Fast-WAM ( Yuan et al., 2026 )
91.9
91.8
91.8
77.8
1.9
39.9
Appendix
Table 7: All published RoboTwin 2.0 results reported by Cai et al. (2026b) (task-averaged success, %), rounded to one decimal; – marks an unreported setting. They come from different architectures and training recipes; Table 2 gives the matched π0.5 comparison.
Source
Collection
Recordings
General PnP demonstrations
teleoperated task executions
1,000
Random motion
free arm motion among the objects
200
Failed execution
deliberate contact attempts and failures
200
Total
1,400
Appendix
Table 8: Real-robot training data, all collected on the General PnP setup. Main experiments use a fixed subset of 500 demonstrations ( 1× ) and all auxiliary recordings; the 2× budget uses all 1,000 demonstrations.
Setting
Value
Adaptation
full fine-tuning
Optimizer
AdamW, (β1,β2,ϵ)=(0.9,0.999,10−8)
Learning rate
2×10−5 , constant, no warmup
Weight decay
0
Gradient clipping
1.0
Batch size
256 (32 per GPU, no accumulation)
Appendix
Table 9: Fine-tuning configuration for π0.5 , shared by all conditions.
Wrist perturbation pV
Detailed captions pC
Auxiliary fraction ρ
Level
SR ↑
GF↑
SC ↓
MI ↓
SR ↑
GF↑
SC ↓
MI ↓
SR ↑
GF↑
SC ↓
MI ↓
0
68
0.43
58
38
66
0.43
34
40
64
0.43
30
76
1/2×
76
0.50
34
30
74
0.50
26
32
74
0.50
24
46
1×
84
0.57
20
24
84
0.57
20
24
84
0.57
20
24
2×
80
0.55
16
26
82
0.58
20
26
78
0.56
22
20
Appendix
Table 10: Sensitivity to the sampling rates on real-robot General PnP with a π0.5 backbone. One rate varies at a time, relative to its validation-selected value ( 1× , the full method; 2× is capped at one), with the other two held at 1× . Level 0 disables the component and matches the corresponding leave-one-out row of Table 1 . Columns follow Table 4 .
Table 14
Predictor
ρ
95% interval
GroundFscore − predictor
No-wrist below baseline
Task success
0.31
[ − 0.06, 0.60]
0.41 [0.12, 0.68]
yes
Stability ( 1−R )
0.44
[0.09, 0.69]
0.28 [0.04, 0.52]
no
Responsiveness ( S )
0.38
[0.02, 0.65]
0.34 [0.08, 0.60]
yes
GroundFscore
0.72
[0.49, 0.86]
–
yes
Appendix
Table 13: Spearman ρ with the negative online failure rate over 36 checkpoints, with 95% cluster-bootstrap intervals. The fourth column is the paired difference from GroundFscore ; the last records whether the predictor ranks 2the no-wrist policy below the fine-tuned baseline.
Shortcut learning is a prevalent issue in robot learning. The limited diversity of robot demonstration datasets can mislead policies into exploiting spurious correlations between tasks and irrelevant features, such as viewpoint or background. Collecting sufficiently diverse robot demonstrations is costly and inefficient, motivating algorithmic alternatives. We focus on vision-language-action (VLA) models and discover that different vision-language model backbones exhibit substantially different levels of susceptibility to visual shortcut learning. We find that model behavior correlates with our proposed representation-level metric, action margin, which requires no policy rollouts. Visual shortcuts consistently enter action representations in early layers, with models differing in the extent to which later layers correct them by incorporating language information. To boost models' attention to language, we introduce task scrubbing, a new domain-adversarial training method that decreases models' likelihood of using visual shortcuts and improves VLAs' generalization. Experiments in both simulation and the real world across multiple VLAs and visual cues show that task scrubbing improves out-of-distribution robustness and often eliminates visual shortcut learning.
Vision-Language-Action (VLA) models that couple pretrained Vision-Language Models (VLMs) with continuous action experts have achieved strong manipulation performance, yet generalization to out-of-distribution (OOD) language instructions remains poor. A known challenge is the structural imbalance in VLA data, where language is far less diverse than visual and action content, making policies prone to visual shortcuts. While discrete-action methods mitigate this through vision-language co-training, continuous action experts lack such protection: they start from random initialization and learn entirely from imbalanced data, producing noisy gradients that corrupt the VLM and fail to exploit its language capability. We address this from a Bayesian perspective, factorizing the policy into a language-agnostic Vision-Action (VA) prior and a language-conditioned VLA likelihood, and propose APT, a two-stage training method emphasizing Action expert PreTraining. In Stage 1, the action expert is pretrained as a VA prior on vision-action pairs from a frozen VLM, bypassing the language imbalance. In Stage 2, language tokens are injected through a gated fusion mechanism that integrates VLM features while preserving the learned visuomotor prior. APT applies to mainstream VLA architectures, including the π and GR00T-style architectures. Comprehensive experiments validate that APT achieves consistent gains on unseen instructions and compositional tasks. Project Page: https://xukechun.github.io/papers/APT/
Kechun Xu, Zhenjie Zhu, Anzhe Chen +2
Zhejiang University · Zhejiang Humanoid Robot Innovation Center
Vision-Language-Action models (VLAs) achieve remarkable performance in sequential decision-making but remain fragile to subtle environmental shifts, such as small changes in object pose. We attribute this brittleness to trajectory overfitting, where VLAs over-attend to the spurious correlation between actions and entities, then reproduce memorized action patterns. We propose Perturbation learning with Delayed Feedback (PDF), a verifier-free test-time adaptation framework that improves decision performance without fine-tuning the base model. PDF mitigates the spurious correlation through uncertainty-based data augmentation and action voting, while an adaptive scheduler allocates augmentation budgets to balance performance and efficiency. To further improve stability, PDF learns a lightweight perturbation module that retrospectively adjusts action logits guided by delayed feedback, correcting overconfidence issue. Experiments on LIBERO (+7.4% success rate) and Atari (+10.3 human normalized score) demonstrate consistent gains of PDF in task success over vanilla VLA and VLA with test-time adaptation, establishing a practical path toward reliable test-time adaptation in multimodal decision-making agents. The code is available at https://github.com/zhoujiahuan1991/CVPR2026-PDF.
Zehua Zang, Xi Wang, Fuchun Sun +4
Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Tsinghua University, Beijing, China +2