PerturBot: Breaking Shortcut Priors in Vision-Language-Action Models with Perturbative Training

Mingyu Liu; Chonghao Sima; Tianjian Feng; Hanqing Wang; Cong Chen; Hao Chen; Chunhua Shen

cs.ROOct 3, 2026

PerturBot: Breaking Shortcut Priors in Vision-Language-Action Models with Perturbative Training

Authors: Mingyu Liu, Chonghao Sima, Tianjian Feng, Hanqing Wang, Cong Chen, Hao Chen, Chunhua Shen

Organizations: Zhejiang University · Shanghai Innovation Institute · University of Hong Kong · HKUST(GZ)

Abstract

A vision--language--action (VLA) policy can complete complex tasks while ignoring the evidence that should determine its actions. An object held near the wrist camera can displace the instructed target. Language and action show the same pattern: a familiar noun can trigger the operation it was paired with in training even after the verb changes, and a gripper that closed on nothing may lift anyway. We call these dependencies modality shortcuts: regularities in successful demonstrations make visual, lexical, or motor cues sufficient to predict expert actions without the task evidence needed for the underlying decision. More demonstrations of the same kind can raise task success while leaving these shortcuts intact. We propose Perturbot which makes task-relevant evidence easier to use and shortcuts insufficient on their own: it applies task-preserving wrist-view perturbations, enriches instructions with decision-relevant captions, and adds random and failed trajectory segments relabeled with the behavior they contain. It complements scaling by changing what is scaled, and leaves inference unchanged. Moreover, we propose GroundingFscore, an offline score that diagnoses how severely a policy relies on modality shortcuts. Task success rate shows whether a policy improves, while GroundingFscore reveals whether the policy scales healthily, relying on task evidence rather than shortcuts. Together, Perturbot and GroundingFscore provide a training-and-evaluation framework for disentangling VLA decisions from shortcut priors while preserving responsiveness to task-relevant evidence.

Figures & tables

Figure 1: Modality shortcuts and Perturbot . Top: familiar visual, lexical, and motor cues can replace task evidence (solid: hypothetical shortcut choices; dashed: required alternatives). Bottom: task-preserving wrist-view perturbations (V), behavior-supported caption enrichment (C), and screened, relabeled trajectory segments (R; Section 3.2 ). Examples are conceptual; relabeled local behavior does not by itself establish recovery.

Figure 2: GroundFscore from paired inputs. (a) For each family, a null edit must leave the action unchanged (an off-path decoy, a paraphrase, another approach to the same pose), and a causal edit must change it by a known target $\Delta a^{\star}_{e}$ (the target moved by $\Delta p$ , another object named, a grasp that closed on nothing). (b) $R_{c}$ normalizes the predicted change $\delta$ ; $S_{c}$ projects it onto $\Delta a^{\star}_{e}$ and is clipped to $[0,1]$ . (c) Policies with equal success on the demonstration distribution can differ sharply in GroundFscore .

Figure 3: Real-robot General PnP setup. Left: the material pool and the six objects and three containers used in the task. Middle: an example stored layout, specified by each item’s position and orientation. Right: an instruction-conditioned scene; every policy is tested on the same 50 stored cases.

	Task success $\uparrow$							GroundFscore $\uparrow$			Failure $\downarrow$
Method	G1	G2	G3	G4	G5	SR	Prog.	$\mathrm{GF}_{V}$	$\mathrm{GF}_{L}$	$\mathrm{GF}_{A}$	SC	MI
$\pi_{0.5}$ fine-tune	40 $\pm$ 30	40 $\pm$ 30	50 $\pm$ 30	40 $\pm$ 30	40 $\pm$ 30	42 $\pm$ 14	2.71 $\pm$ 0.55	0.28 $\pm$ 0.04	0.24 $\pm$ 0.03	0.19 $\pm$ 0.03	68 $\pm$ 13	82 $\pm$ 11
+ V	50 $\pm$ 30	50 $\pm$ 30	60 $\pm$ 30	50 $\pm$ 30	50 $\pm$ 30	52 $\pm$ 14	3.12 $\pm$ 0.54	0.51 $\pm$ 0.04	0.26 $\pm$ 0.03	0.21 $\pm$ 0.03	36 $\pm$ 13	80 $\pm$ 11
+ C	60 $\pm$ 30	50 $\pm$ 30	50 $\pm$ 30	50 $\pm$ 30	60 $\pm$ 30	54 $\pm$ 14	3.20 $\pm$ 0.53	0.30 $\pm$ 0.04	0.47 $\pm$ 0.04	0.22 $\pm$ 0.03	62 $\pm$ 14	78 $\pm$ 11
+ R	50 $\pm$ 30	60 $\pm$ 30	50 $\pm$ 30	60 $\pm$ 30	60 $\pm$ 30	56 $\pm$ 14	3.34 $\pm$ 0.52	0.29 $\pm$ 0.04	0.27 $\pm$ 0.03	0.44 $\pm$ 0.04	64 $\pm$ 13	44 $\pm$ 14
+ V + C	70 $\pm$ 30	60 $\pm$ 30	60 $\pm$ 30	60 $\pm$ 30	70 $\pm$ 30	64 $\pm$ 13	3.62 $\pm$ 0.50	0.55 $\pm$ 0.04	0.50 $\pm$ 0.04	0.24 $\pm$ 0.03	30 $\pm$ 13	76 $\pm$ 12

Table 1: Main results on real-robot General PnP with a

\pi_{0.5}

backbone. G1–G5 are success rates (%) on five blocks of ten cases, SR is success over all 50 cases, and Prog. is the mean 0–5 progress score.

\mathrm{GF}_{V}

\mathrm{GF}_{L}

, and

\mathrm{GF}_{A}

are GroundFscore readouts on 100 held-out observations ( Section 3.3 ); SC and MI are failure rates (%) on salience-capture and motor-inertia probes of 50 trials each, separate from the 50 cases.

\pm

is the half-width of the 95% bootstrap interval over the evaluation units: the ten cases of a block, the 50 cases for SR and Prog., the 100 observations for GroundFscore , and the 50 probe trials for SC and MI. V, C, and R denote wrist-view perturbation, detailed captions, and trajectory expansion; the upper block covers all eight combinations and the lower block the controls. Bold marks the best value in each column; the shaded row is the full method.

	Full			Clean2Random
Method	Clean	Rand.	Avg.	Clean	Rand.	Avg.
$\pi_{0}$ ( Black et al., 2025b )	65.9	58.4	62.2	–	–	–
$\pi_{0.5}$ ( Black et al., 2025a )	82.7	76.8	79.8	70.7	46.0	58.4
ABot-M0 ( Yang et al., 2026b )	81.2	80.4	80.8	–	–	–
X-VLA ( Zheng et al., 2026 )	–	–	–	68.0	20.9	44.5
X-WAM ( Guo et al., 2026 )	–	–	–	70.0	25.8	47.9

Table 2: RoboTwin 2.0 results (task-averaged success, %). Avg. is the mean of clean and randomized scenes. Published results are as reported by Cai et al. (2026b) , rounded to one decimal, and – marks an unreported setting; Table 7 lists every published result, including systems with higher reported scores. The lower block uses our

\pi_{0.5}

fine-tuning recipe. Bold marks the best value in each column; the shaded row is the full method.

Figure 4: Scaling and ablation effects with a

\pi_{0.5}

backbone. (a) Success (left) and mean GroundFscore (right) on a log-scaled demonstration budget; Table 4 compares their slopes. (b) Mean GroundFscore gain per family from adding V, C, or R, averaged over four contrasts with the other components fixed ( Table 1 ). (c) Change in success from the selected rate as one sampling rate varies ( Table 10 ); shading shows the loss, not uncertainty.

Table 7

Table 8

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Real-robot platform. Two AgileX PiPER X arms are mounted side by side and face the tabletop workspace. A RealSense D435 head camera on the central vertical post provides the third-person view, and a RealSense wrist camera beside each gripper provides a wrist view.

	Full			Clean2Random
Method	Clean	Rand.	Avg.	Clean	Rand.	Avg.
$\pi_{0}$ ( Black et al., 2025b )	65.9	58.4	62.2	–	–	–
$\pi_{0.5}$ ( Black et al., 2025a )	82.7	76.8	79.8	70.7	46.0	58.4
ABot-M0 ( Yang et al., 2026b )	81.2	80.4	80.8	–	–	–
Motus ( Bi et al., 2026 )	88.7	87.0	87.8	–	–	–
Fast-WAM ( Yuan et al., 2026 )	91.9	91.8	91.8	77.8	1.9	39.9

Appendix

Table 7: All published RoboTwin 2.0 results reported by Cai et al. (2026b) (task-averaged success, %), rounded to one decimal; – marks an unreported setting. They come from different architectures and training recipes; Table 2 gives the matched

\pi_{0.5}

comparison.

Source	Collection	Recordings
General PnP demonstrations	teleoperated task executions	1,000
Random motion	free arm motion among the objects	200
Failed execution	deliberate contact attempts and failures	200
Total		1,400

Appendix

Table 8: Real-robot training data, all collected on the General PnP setup. Main experiments use a fixed subset of 500 demonstrations (

1\times

) and all auxiliary recordings; the

2\times

budget uses all 1,000 demonstrations.

Setting	Value
Adaptation	full fine-tuning
Optimizer	AdamW, $(\beta_{1},\beta_{2},\epsilon)=(0.9,\,0.999,\,10^{-8})$
Learning rate	$2\times 10^{-5}$ , constant, no warmup
Weight decay	0
Gradient clipping	1.0
Batch size	256 (32 per GPU, no accumulation)

Appendix

Table 9: Fine-tuning configuration for

\pi_{0.5}

, shared by all conditions.

	Wrist perturbation $p_{V}$				Detailed captions $p_{C}$				Auxiliary fraction $\rho$
Level	SR $\uparrow$	$\overline{\mathrm{GF}}$ $\uparrow$	SC $\downarrow$	MI $\downarrow$	SR $\uparrow$	$\overline{\mathrm{GF}}$ $\uparrow$	SC $\downarrow$	MI $\downarrow$	SR $\uparrow$	$\overline{\mathrm{GF}}$ $\uparrow$	SC $\downarrow$	MI $\downarrow$
$0$	68	0.43	58	38	66	0.43	34	40	64	0.43	30	76
$1/2\times$	76	0.50	34	30	74	0.50	26	32	74	0.50	24	46
$1\times$	84	0.57	20	24	84	0.57	20	24	84	0.57	20	24
$2\times$	80	0.55	16	26	82	0.58	20	26	78	0.56	22	20

Appendix

Table 10: Sensitivity to the sampling rates on real-robot General PnP with a

\pi_{0.5}

backbone. One rate varies at a time, relative to its validation-selected value (

1\times

, the full method;

2\times

is capped at one), with the other two held at

1\times

. Level

0

disables the component and matches the corresponding leave-one-out row of Table 1 . Columns follow Table 4 .

Table 14

Predictor	$\rho$	95% interval	GroundFscore $-$ predictor	No-wrist below baseline
Task success	0.31	[ $-$ 0.06, 0.60]	0.41 [0.12, 0.68]	yes
Stability ( $1-R$ )	0.44	[0.09, 0.69]	0.28 [0.04, 0.52]	no
Responsiveness ( $S$ )	0.38	[0.02, 0.65]	0.34 [0.08, 0.60]	yes
GroundFscore	0.72	[0.49, 0.86]	–	yes

Appendix

Table 13: Spearman

\rho

with the negative online failure rate over 36 checkpoints, with 95% cluster-bootstrap intervals. The fourth column is the paired difference from GroundFscore ; the last records whether the predictor ranks 2the no-wrist policy below the fine-tuned baseline.

Spatial Grounding for VLA Models

See all 24 papers

When Listening Becomes Easier: Scrubbing Visual Cues for Shortcut-Free VLAs
Oct 7, 2026Jasper Gerigk, Kenzo Aspuru-Takata, Chin-Hsuan Wu +3Vision-Language-Action Models Robustness of VLA Models
University of Toronto, Toronto, ON M5S 3H5, Canada · Vector Institute, Toronto, ON M5G 0C6, Canada
APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
Jun 10, 2026Kechun Xu, Zhenjie Zhu, Anzhe Chen +2Visuomotor Policy Learning Vision-Language-Action Models
Zhejiang University · Zhejiang Humanoid Robot Innovation Center
Test-Time Perturbation Learning with Delayed Feedback for Vision-Language-Action Models
Apr 20, 2026Zehua Zang, Xi Wang, Fuchun Sun +4Test-Time Adaptation Test-Time Adaptation of VLMs
Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Tsinghua University, Beijing, China +2

Oct 7, 2026J/K move · Enter open · S save

When Listening Becomes Easier: Scrubbing Visual Cues for Shortcut-Free VLAs

Jasper Gerigk, Kenzo Aspuru-Takata, Chin-Hsuan Wu, Mohammad Mohammadi+2

University of Toronto, Toronto, ON M5S 3H5, Canada · Vector Institute, Toronto, ON M5G 0C6, Canada

Vision-Language-Action Models Robustness of VLA Models Adversarial Training

Open paper AskPDF arXiv

Shortcut learning is a prevalent issue in robot learning. The limited diversity of robot demonstration datasets can mislead policies into exploiting spurious correlations between tasks and irrelevant features, such as viewpoint or background. Collecting sufficiently diverse robot demonstrations is costly and inefficient, motivating algorithmic alternatives. We focus on vision-language-action (VLA) models and discover that different vision-language model backbones exhibit substantially different levels of susceptibility to visual shortcut learning. We find that model behavior correlates with our proposed representation-level metric, action margin, which requires no policy rollouts. Visual shortcuts consistently enter action representations in early layers, with models differing in the extent to which later layers correct them by incorporating language information. To boost models' attention to language, we introduce task scrubbing, a new domain-adversarial training method that decreases models' likelihood of using visual shortcuts and improves VLAs' generalization. Experiments in both simulation and the real world across multiple VLAs and visual cues show that task scrubbing improves out-of-distribution robustness and often eliminates visual shortcut learning.

Abstract

Spatial Grounding for VLA Models

Explore similar work

When Listening Becomes Easier: Scrubbing Visual Cues for Shortcut-Free VLAs

APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies

Test-Time Perturbation Learning with Delayed Feedback for Vision-Language-Action Models