Paraphrase Consensus Is Not Self-Consistency: Instruction Reliance and Robustness in VLA Policies

Hanlong Li; Sihan Tan; Takeshi Ashizawa; Benjamin Yen; Kazuhiro Nakadai

cs.ROOct 14, 2026

Paraphrase Consensus Is Not Self-Consistency: Instruction Reliance and Robustness in VLA Policies

Authors: Hanlong Li, Sihan Tan, Takeshi Ashizawa, Benjamin Yen, Kazuhiro Nakadai

Organizations: Institute of Science Tokyo, Tokyo, Japan

Abstract

Vision-language-action (VLA) policies fine-tuned on LIBERO are trained and evaluated on the same ten instruction strings, and rewording the instruction changes base OpenVLA's success by -38.0 points. Test-time prompt ensembling, which improves language models, does not recover this loss in any setting we ran. Paraphrase consensus (the medoid action over K paraphrases) costs base OpenVLA 16.8 points when the training instruction is a member. Without it, the medoid of the paraphrases scores -32.4 points relative to the training instruction, and confidence-based selection scores -28.0. The same medoid over samples of one paraphrase improves on greedy decoding but stays -27.6 points below the training instruction, consistent with aggregation reducing variance without removing the bias the paraphrases share. A wrong-instruction probe measures how much of a policy's success depends on the instruction. Across three policies and four suites, paraphrases cost a policy only where this share is high, and the stronger policies lose a much smaller fraction of it than base OpenVLA. LoRA post-training on paraphrased instructions reduces the one-paraphrase effect to +2.0 points, against -32.0 for a matched control, reproduces across two seeds and on OpenVLA-OFT, transfers to withheld phrasings, and leaves the policy's reliance on the instruction unchanged.

Figures & tables

Method	$n$	Succ. (%)	$\Delta$ vs. ref. [95% CI]	$p_{\text{Holm}}$
base OpenVLA / LIBERO-Spatial, reference: Canonical (greedy)
Canonical (greedy)	250	85.6 [80.7, 89.4]	—	—
Canonical (replicate run)	250	83.6 [78.5, 87.7]	-2.0 [-5.6, 1.2]	0.888
Paraphrase medoid, $K{=}4$	250	68.8 [62.8, 74.2]	-16.8 [-22.8, -10.8]	$<$ 0.001
One paraphrase, fixed per episode	250	47.6 [41.5, 53.8]	-38.0 [-44.8, -31.2]	$<$ 0.001
Paraphrase-only medoid, $K{-}1{=}3$ (no training instr.)	250	53.2 [47.0, 59.3]	-32.4 [-39.2, -25.6]	$<$ 0.001

Table 1: Base OpenVLA on LIBERO-Spatial,

n{=}250

per arm, paired. The replicate row is a second run of the baseline under a mis-routed controller name and measures run-to-run noise. Sampling rows use stochastic decodes of the training instruction with the same medoid code. Generated by lcc/analysis/main_table.py .

Arm	Canonical	Paraphrase	Held-out par.	Wrong instr.	Paraphrase cost [95% CI]
untouched checkpoint	85.6	47.6	57.6	2.4	$-38.0$ $[-44.8,-31.2]$
augmentation, p=1	81.6	83.6	83.6	2.8	$+2.0$ $[-4.0,+8.0]$
canonical-only control	82.8	50.8	56.4	–	$-32.0$ $[-38.8,-25.2]$
augmentation, p=0.5	82.4	78.4	–	–	$-4.0$ $[-10.0,+2.0]$
augmentation, lr 5e-5	83.6	82.8	–	–	$-0.8$ $[-6.8,+5.2]$

Table 2: Instruction augmentation on base OpenVLA / LIBERO-Spatial (LoRA rank 32, learning rate

10^{-4}

2{,}000

steps). Paraphrased instructions with ground-truth actions against a matched control trained on training instructions and the untouched checkpoint. All cells

n{=}250

, paired on identical initial states; held-out paraphrases were withheld from training. Generated by lcc/distill/report_p5.py .

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Suite	Published	Ours	$\Delta$	95% Wilson CI	Gate ( $\pm 2$ )
LIBERO-Spatial	97.6	97.8	+0.2	[96.1, 98.8]	pass
LIBERO-Object	98.4	97.4	-1.0	[95.6, 98.5]	pass
LIBERO-Goal	97.9	98.4	+0.5	[96.9, 99.2]	pass
LIBERO-Long (10)	94.5	93.4	-1.1	[90.9, 95.3]	pass

Appendix

Table 3: Reproduction of the published OpenVLA-OFT LIBERO results on the corrected stack. Intervals are Wilson 95% intervals on our own rate ( Wilson, 1927 ) ; the gate is

\pm 2

points (500 episodes per suite).

Component	Version / value
openvla-oft	SHA e4287e94541f459edc4feabc4e181f537cd569a8
LIBERO	SHA 8f1084e3132a39270c3a13ebe37270a43ece2a01
python	Python 3.10.14
torch	2.2.0+cu121
MUJOCO_GL	egl
gpu	NVIDIA A100 80GB PCIe, 81920 MiB

Appendix

Table 4: Environment, checkpoint revisions and the pinned versions that the numbers in Tab. 3 were produced with.

Suite	Task	Succ.	$n$
LIBERO-Spatial	pick up the black bowl between the plate and the ramekin and place it on the plate	100%	50
LIBERO-Spatial	pick up the black bowl next to the ramekin and place it on the plate	100%	50
LIBERO-Spatial	pick up the black bowl from table center and place it on the plate	100%	50
LIBERO-Spatial	pick up the black bowl on the cookie box and place it on the plate	98%	50
LIBERO-Spatial	pick up the black bowl in the top drawer of the wooden cabinet and place it on the plate	96%	50
LIBERO-Spatial	pick up the black bowl on the ramekin and place it on the plate	96%	50

Appendix

Table 5: Per-task success rates for the reproduction runs.

Suite	Controller	$K$	$n$	Success (95% CI)	Published	transformers
LIBERO-Long (10)	canonical-1	1	100	$50.0$ $[40.4,59.6]$	$53.7\pm 1.3$	stock 4.40.1
LIBERO-Long (10)	canonical-1	1	100	$55.0$ $[45.2,64.4]$	$53.7\pm 1.3$	stock 4.40.1
LIBERO-Goal	canonical-1	1	250	$76.0$ $[70.3,80.9]$	$79.2\pm 1.0$	stock 4.40.1
LIBERO-Object	canonical-1	1	250	$87.6$ $[82.9,91.1]$	$88.4\pm 0.8$	stock 4.40.1
LIBERO-Spatial	canonical-1	1	20	$30.0$ $[14.5,51.9]$ †	$84.7\pm 0.9$	OFT fork (non-causal)
LIBERO-Spatial	canonical-1	1	250	$85.6$ $[80.7,89.4]$	$84.7\pm 0.9$	stock 4.40.1

Appendix

Table 6: Base OpenVLA through our harness, before and after restoring causal attention, against the published reference (OpenVLA’s own README; 500 trials per suite). Intervals are Wilson 95% intervals on our rate;

\dagger

: the interval does not reach the published point estimate. LIBERO-Long appears twice because two independent runs of the same configuration exist:

50.0

is the paired reference of § 4.3 ,

55.0

that of Tab. 17 .

Method	$n$	Succ. (%)	$\Delta$ vs. ref. [95% CI]	$p_{\text{Holm}}$
OpenVLA-OFT / LIBERO-Spatial with distractor, reference: Canonical-8
Canonical-8	500	88.6 [85.5, 91.1]	—	—
Paraphrase medoid, $K{=}4$	500	90.0 [87.1, 92.3]	+1.4 [-0.6, 3.4]	0.743
Paraphrase mean, $K{=}4$	500	87.4 [84.2, 90.0]	-1.2 [-3.8, 1.2]	0.859
One of $K{=}4$ instructions (canonical incl.), per episode	500	87.8 [84.6, 90.4]	-0.8 [-3.2, 1.6]	0.859

Appendix

Table 7: Shifted LIBERO-Spatial (distractor mug), OpenVLA-OFT,

n{=}500

per arm paired on identical initial states. Generated by lcc/analysis/main_table.py .

Method	$n$	Succ. (%)	$\Delta$ vs. ref. [95% CI]	$p_{\text{Holm}}$
OpenVLA-OFT / LIBERO-Spatial, reference: Canonical-8
Canonical-8	500	97.2 [95.4, 98.3]	—	—
Canonical (replicate run)	500	98.0 [96.4, 98.9]	+0.8 [0.2, 1.6]	1.000
Canonical (greedy)	408	95.6 [93.1, 97.2]	-1.0 [-3.4, 1.5]	1.000
Canonical-4	430	98.1 [96.4, 99.1]	+0.5 [-1.2, 2.1]	1.000
Gated LCC	500	97.2 [95.4, 98.3]	+0.0 [-1.4, 1.4]	1.000

Appendix

Table 8: Standard LIBERO-Spatial, all controllers. Generated by lcc/analysis/main_table.py .

$K$	mode	median (ms)	p95 (ms)	$\times$ vs. $K{=}1$	control rate (Hz, $H{=}8$ )	peak GiB
1	sequential	101	104	1.00	79.3	15.0
2	batched	190	193	1.89	42.0	15.0
2	sequential	201	202	1.99	39.8	15.0
4	batched	301	303	2.98	26.6	15.5
4	sequential	406	409	4.02	19.7	15.0
8	batched	583	586	5.78	13.7	16.7

Appendix

Table 9: Measured wall-clock per replanning step for OpenVLA-OFT on one idle A100-40 GB (no other compute on the card), median and p95 of 20 CUDA-synchronised repetitions. Sequential is

K

separate reference calls; batched computes the vision features once and issues one language-model forward per distinct prompt token length. Peak GPU allocation was 14.98 GiB at

K{=}1

and 16.66 GiB at

K{=}8

Detector	AUROC [95% CI]	AUROC@2	AUPRC	FAR@80%rec
Replan count (length only)	0.999 [0.997, 1.000]	0.999	0.981	0.002
Language (ours)	0.600 [0.512, 0.683]	0.617	0.179	0.675
Visual TTA	0.512 [0.436, 0.590]	0.522	0.102	0.641
Activation pert.	0.509 [0.428, 0.589]	0.534	0.103	0.773
Temporal consistency (ACC)	0.524 [0.430, 0.622]	0.454	0.121	0.842

Appendix

Table 10: Length-controlled failure detection on the shifted suite (OpenVLA-OFT,

500

shadow episodes,

51

failures), every detector scored on the same rollouts over a fixed prefix of five replanning steps; the first row, replan count scored on the whole trace, is the confound. AUROC@2: the same statistic over a prefix of two replans; FAR@80%rec: false-alarm rate at 80% recall; ACC: action-chunk consistency. Generated by lcc/analysis/failure.py .

Figure 1: Risk-set AUROC by replanning step on the shifted suite, one curve per ensemble source; the risk set at

t

is the episodes still running at

t

, so length cannot leak. Generated by lcc/analysis/riskset.py .

Figure 2: Per-replan disagreement

u_{t}

by ensemble source on identical canonical trajectories (OpenVLA-OFT, shifted LIBERO-Spatial). Generated by lcc/analysis/sensitivity.py .

Action-level disagreement by ensemble source
Source		median	mean	p90	$n$ replans
language ( $\times$ 1.00)		0.0191	0.0275	0.0599	7429
visual TTA ( $\times$ 4.09)		0.0782	0.0932	0.1527	7429
activation pert. ( $\times$ 0.48)		0.0091	0.0119	0.0167	7429
sampling	n/a — policy has a deterministic action head (L1 regression), so a sampling ensemble is identically zero; report the cell as n/a
language, successful episodes		0.0179	0.0243	0.0486	6001

Appendix

Table 11: Action-level disagreement by source and by episode outcome (OpenVLA-OFT, shifted LIBERO-Spatial,

500

shadow episodes). Generated by lcc/analysis/sensitivity.py .

base OpenVLA / LIBERO-Spatial, reference: Canonical (greedy)
Method	$n$	Succ. (%)	$\Delta$ vs. ref. [95% CI]	$p_{\text{Holm}}$
Canonical (greedy)	250	85.6 [80.7, 89.4]	—	—
Canonical unless outlier	250	83.2 [78.1, 87.3]	-2.4 [-7.6, 2.8]	0.903
Paraphrase medoid [K=2]	250	85.6 [80.7, 89.4]	+0.0 [-3.6, 3.6]	1.000
Paraphrase medoid [K=4]	250	68.8 [62.8, 74.2]	-16.8 [-22.8, -10.8]	$<$ 0.001
Paraphrase medoid [K=8]	100	74.0 [64.6, 81.6]	-10.0 [-21.0, 1.0]	0.331

Appendix

Table 12: Paraphrase consensus at

K\in\{2,4,8\}

and the outlier-gated variant, base OpenVLA / LIBERO-Spatial, paired against canonical greedy.

K{=}2

tie-breaks to the canonical member by construction. Generated by lcc/analysis/main_table.py .

Condition	$n$	All paraphrases $\Delta$ [95% CI]	$n$	Human-equivalent only
OpenVLA-OFT / Long (llm)	100	$-3.0$ $[-8.0,+2.0]$	56	$-5.4$ $[-12.5,+0.0]$
OpenVLA-OFT / Goal (llm)	250	$-10.4$ $[-14.4,-6.8]$	198	$-8.6$ $[-12.6,-5.1]$
OpenVLA-OFT / Object (llm)	250	$-1.6$ $[-4.0,+0.4]$	143	$-2.1$ $[-5.6,+0.7]$
OpenVLA-OFT / Spatial (llm)	500	$-1.0$ $[-3.0,+1.0]$	290	$-0.3$ $[-2.8,+1.7]$
base OpenVLA / Long (llm)	100	$-19.0$ $[-32.0,-6.0]$	56	$-19.6$ $[-37.5,-1.8]$
base OpenVLA / Goal (llm)	250	$-36.8$ $[-44.0,-29.6]$	198	$-31.8$ $[-39.9,-23.7]$

Appendix

Table 13: One-paraphrase cost on all evaluated paraphrases and on those judged equivalent by a human. Generated by lcc/analysis/human_equivalence.py .

Figure 3: Aggregation with and without exchangeable members, base OpenVLA / LIBERO-Spatial, paired

\Delta

vs. canonical greedy with

95\%

bootstrap intervals,

n{=}250

. Paraphrase rows: one paraphrase or the medoid of four. Sampling rows: one or four stochastic decodes of the training instruction, identical medoid code. Generated by lcc/analysis/main_table.py .

Feature (vs. training instruction)	base OpenVLA ( $n{=}210$ )	OpenVLA-OFT ( $n{=}120$ )	$\pi_{0.5}$ ( $n{=}120$ )
char.-trigram cosine	$+0.36$ [ $<$ 0.001]	$+0.21$ [0.022]	$+0.02$ [0.810]
word Jaccard	$+0.34$ [ $<$ 0.001]	$+0.34$ [ $<$ 0.001]	$+0.09$ [0.312]
embedding cosine	$+0.47$ [ $<$ 0.001]	$+0.21$ [0.025]	$-0.02$ [0.801]
length ratio	$-0.13$ [0.071]	$-0.13$ [0.163]	$-0.20$ [0.028]
human: same task	$-0.00$ [0.986]	$+0.03$ [0.713]	$+0.02$ [0.836]

Appendix

Table 14: Spearman correlation of per-paraphrase cost with paraphrase features (permutation

p

in brackets). Generated by lcc/analysis/paraphrase_features.py .

Comparison	$n$	$\Delta$	Task-clustered 95% CI
base OpenVLA, one paraphrase vs. canonical (Spatial)	250	$-38.0$	$[-53.2,-21.6]$
base OpenVLA, paraphrase medoid vs. canonical (Spatial)	250	$-16.8$	$[-30.4,-4.0]$
sampling medoid vs. paraphrase medoid (Spatial)	250	$+17.6$	$[+7.2,+28.4]$
base OpenVLA, paraphrase-only medoid vs. one paraphrase (Spatial)	250	$+5.6$	$[-4.0,+16.0]$
base OpenVLA, sampling medoid around a paraphrase vs. greedy paraphrase (Spatial)	250	$+10.4$	$[+3.2,+18.0]$
base OpenVLA, sampling medoid vs. one sample around a paraphrase (Spatial)	250	$+6.4$	$[-1.6,+14.0]$

Appendix

Table 15: Headline paired comparisons with task-clustered

95\%

bootstrap intervals (

10{,}000

resamples). Generated by lcc/analysis/clustered_ci.py .

Policy	Suite	Tasks	Share range	Spearman $\rho$ (share, cost) [ $p$ ]
OpenVLA-OFT	Spatial	10	-14–83%	$-0.65$ [0.04]
OpenVLA-OFT	Object	10	0–8%	$-0.57$ [0.10]
OpenVLA-OFT	Goal	10	100–100%	$--$ [nan]
OpenVLA-OFT	Long	10	-11–80%	$-0.23$ [0.59]
OpenVLA-OFT	all	40		$-0.56$ [0.000]
base OpenVLA	Spatial	10	71–100%	$-0.52$ [0.21]

Appendix

Table 16: Spearman correlation of per-task language share with per-task paraphrase cost. Generated by lcc/analysis/reliance_per_task.py .

Policy	Suite	Correct instr.	Wrong instr.	Language share
OpenVLA-OFT	Spatial	96.8 [93.8, 98.4]	74.4 [68.6, 79.4]	23%
OpenVLA-OFT	Object	98.0 [95.4, 99.1]	96.4 [93.3, 98.1]	2%
OpenVLA-OFT	Goal	100.0 [98.5, 100.0]	0.0 [0.0, 1.5]	100%
OpenVLA-OFT	Long	95.0 [88.8, 97.8]	87.0 [79.0, 92.2]	8%
base OpenVLA	Spatial	85.6 [80.7, 89.4]	2.4 [1.1, 5.1]	97%
base OpenVLA	Object	87.6 [82.9, 91.1]	1.2 [0.4, 3.5]	99%

Appendix

Table 17: Instruction-reliance probe. Success on the original task while conditioned on another task’s instruction, scene and success predicate unchanged; paired on identical initial states (

n{=}250

per cell,

n{=}100

on Long). Language share

=(\text{correct}-\text{wrong})/\text{correct}

. Correct-instruction cells are each probe’s paired reference run. Generated by lcc/analysis/probe_table.py .

Suite	Executable pairs	Missing goal objects (task $i\to i{+}5$ )
Spatial	10/10	–
Object	6/10	4 $\to$ 9: orange_juice; 5 $\to$ 0: alphabet_soup; 6 $\to$ 1: cream_cheese; 7 $\to$ 2: salad_dressing
Goal	10/10	–
Long	0/10	0 $\to$ 5: black_book; 1 $\to$ 6: chocolate_pudding, plate, porcelain_mug; 2 $\to$ 7: alphabet_soup, basket, cream_cheese; 3 $\to$ 8: flat_stove, moka_pot; 4 $\to$ 9: microwave; 5 $\to$ 0: alphabet_soup, basket, tomato_sauce; 6 $\to$ 1: basket, butter, cream_cheese; 7 $\to$ 2: flat_stove, moka_pot; 8 $\to$ 3: akita_black_bowl, white_cabinet; 9 $\to$ 4: plate

Appendix

Table 18: Executability of the wrong instruction in the probe. Generated by lcc/analysis/probe_feasibility.py .

character trigrams
Paraphrase type	Spatial	Object	Goal	Long
LLM	85.7	100.0	98.6	98.6
lexical	82.9	100.0	94.3	100.0
structural	75.7	100.0	92.9	98.6
typos/casing	97.1	100.0	100.0	100.0
held-out	88.6	100.0	98.6	98.6

Appendix

Table 19: Closed-set canonicalization: accuracy (%) of retrieving the correct task’s training instruction from the ten in the suite, by paraphrase type. Generated by lcc/analysis/canonicalize_retrieval.py .

Type	Instruction
training instruction	pick up the black bowl between the plate and the ramekin and place it on the plate
LLM	the plate is where the black bowl between it and the ramekin needs to go
lexical	retrieve the black bowl between the plate and the ramekin and settle it onto the plate
structural	between the plate and the ramekin there is a black bowl, and its new spot is on the plate
typos/casing	what i want is the black bowl between the plate and the ramekn picked up and put on the plate
held-out	the plate is where i want the black bowl that’s wedged between the plate and the ramekin, could you do that

Appendix

Table 20: One LIBERO-Spatial task: the training instruction and one paraphrase of each type (App. N ).

Task	Canonical (greedy)	Paraphrase medoid, $K{=}4$	One paraphrase, fixed per episode
0	84 (25)	92 (25)	92 (25)
1	100 (25)	52 (25)	32 (25)
2	88 (25)	56 (25)	48 (25)
3	84 (25)	84 (25)	84 (25)
4	72 (25)	76 (25)	16 (25)
5	88 (25)	72 (25)	44 (25)

Appendix

Table 21: Per-task success on LIBERO-Spatial, base OpenVLA, paired on identical initial states.

Arm	$n_{\text{para}}$	All $K$ draws	Canonical-drawn	Paraphrase-only	$\Delta$ vs. canonical [95% CI]
oft:hf:libero_spatial:llm	369	96.4	96.9	96.2	$-1.1$ $[-3.0,+0.8]$
oft:hf:libero_spatial_with_mug:llm	369	87.8	87.0	88.1	$-0.8$ $[-4.1,+2.4]$
openvla:goal-aug-p100:libero_goal:llm	188	73.6	80.6	71.3	$-4.8$ $[-11.7,+2.1]$
openvla:hf:libero_goal:llm	188	52.4	75.8	44.7	$-30.9$ $[-38.8,-22.3]$
openvla:aug-lr1e4-s2k-p100:libero_spatial:heldout	188	84.4	85.5	84.0	$+2.1$ $[-4.3,+8.5]$
openvla:aug-lr1e4-s2k-p100:libero_spatial:llm	188	83.6	82.3	84.0	$+2.1$ $[-5.3,+9.6]$

Appendix

Table 22: Legacy random-draw conditions re-scored on their paraphrase-only episodes (all arms, both suites). “All

K

draws” is the rate as originally run; “paraphrase-only” keeps the episodes whose draw was a paraphrase and pairs them with the canonical run on those episodes. Generated by lcc/analysis/paraphrase_only.py .

Arm	Canonical	Paraphrase	Held-out par.	Wrong instr.	Paraphrase cost [95% CI]
untouched checkpoint	85.6	47.6	–	2.4	$-38.0$ $[-44.8,-31.2]$
lr 1e-4, 2k steps, p=1	81.6	83.6	83.6	2.8	$+2.0$ $[-4.0,+8.0]$
lr 1e-4, 2k steps, p=0.5	82.4	78.4	–	–	$-4.0$ $[-10.0,+2.0]$
lr 5e-5, 2k steps, p=1	83.6	82.8	–	–	$-0.8$ $[-6.8,+5.2]$
lr 5e-4, 5k steps, p=1	75.6	74.8 ‡	73.6 ‡	5.2	$-0.8$ $[-6.4,+4.8]$ ‡
control, lr 1e-4, 2k steps	82.8	50.8	–	–	$-32.0$ $[-38.8,-25.2]$

Appendix

Table 23: Every post-training recipe tried, base OpenVLA / LIBERO-Spatial,

n{=}250

paired. Held-out and wrong-instruction conditions were evaluated for a subset of recipes; – marks the rest. Paraphrase cells are one paraphrase per episode, never the training instruction, except those marked ‡ : the first recipe and its control were evaluated only with the earlier one-of-

K

draw, which includes the training instruction in a quarter of episodes and so understates the cost. Generated by lcc/distill/report_p5.py .

Arm	Canonical	Paraphrase	Held-out par.	Wrong instr.	Paraphrase cost [95% CI]
untouched checkpoint	85.6	47.6	57.6	2.4	$-38.0$ $[-44.8,-31.2]$
augmentation, seed 1	81.6	83.6	83.6	2.8	$+2.0$ $[-4.0,+8.0]$
augmentation, seed 2	83.2	81.2	84.0	–	$-2.0$ $[-8.0,+4.0]$
control, seed 1	82.8	50.8	56.4	–	$-32.0$ $[-38.8,-25.2]$
control, seed 2	80.0	43.2	52.4	–	$-36.8$ $[-44.0,-29.6]$

Appendix

Table 24: Second training seed of the augmentation arm and its matched control (same recipe, seed 2), base OpenVLA / LIBERO-Spatial,

n{=}250

paired per cell, beside seed 1 and the untouched checkpoint. Paired arm against arm, seed 2’s augmentation beats its control by

+38.0

[+30.8,+44.8]

on trained and

+31.6

[+25.2,+38.4]

on held-out phrasings, and its paraphrase success differs from seed 1’s by

-2.4

[-8.0,+3.2]

. Generated by lcc/distill/report_p5.py and lcc/analysis/arm_compare.py .

Policy	Suite	Correct	Wrong instr.	Language share [95% CI]	One paraphrase: $\Delta$ [95% CI]	$p_{\text{Holm}}$	Lost
OpenVLA-OFT	Spatial	96.8	74.4	23% [17, 29]	$-1.0$ $[-3.0,+1.0]$	1.000	4% [-5, 15]
OpenVLA-OFT	Object	98.0	96.4	2% [0, 3]	$-1.6$ $[-4.0,+0.4]$	1.000	–
OpenVLA-OFT	Goal	100.0	0.0	100% [100, 100]	$-10.4$ $[-14.4,-6.8]$	$<$ 0.001	10% [7, 14]
OpenVLA-OFT	Long	95.0	87.0	8% [2, 16]	$-3.0$ $[-8.0,+2.0]$	1.000	–
base OpenVLA	Spatial	85.6	2.4	97% [95, 99]	$-38.0$ $[-44.8,-31.2]$	$<$ 0.001	46% [37, 55]
base OpenVLA	Object	87.6	1.2	99% [97, 100]	$-15.6$ $[-22.4,-9.2]$	$<$ 0.001	18% [11, 26]

Appendix

Table 25: Language share (correct minus wrong-instruction success, over correct, with a

95\%

percentile bootstrap interval over paired episodes) and the paraphrase-only penalty (one paraphrase per episode, paired with the canonical run) for each published checkpoint on all four suites (

\pi_{0.5}

: openpi’s LIBERO client,

10

episodes per task on Object and Long). Correct, wrong and share are computed on the probe’s episodes, the paraphrase effect on all episodes paired with the canonical run. Base OpenVLA / Long’s canonical run here is the probe’s reference (

55.0\%

); the consensus result of § 4.3 is paired with a separate run (

50.0\%

), the replicate gap of App. B . Lost is the paraphrase effect as a fraction of the success the instruction carries,

-\Delta/(\text{correct}-\text{wrong})

, with a

95\%

bootstrap interval, left blank where the instruction carries under

10

points. On Long no wrong instruction is executable in the given scene (App. L ), so its share measures success when the instruction cannot be followed. Generated by lcc/analysis/reliance_vs_cost.py .

Policy	Paraphrase type	$n$	Success (%)	$\Delta$ vs. canonical [95% CI]	$p_{\text{Holm}}$
augmented (p=1)	LLM paraphrases (default)	250	83.6 [78.5, 87.7]	$+2.0$ $[-4.0,+8.0]$	1.000
augmented (p=1)	lexical substitution (easy)	250	82.4 [77.2, 86.6]	$+0.8$ $[-4.8,+6.4]$	1.000
augmented (p=1)	structural rewrite (hard)	250	79.2 [73.7, 83.8]	$-2.4$ $[-8.4,+3.6]$	1.000
augmented (p=1)	typos and casing (adversarial)	250	74.8 [69.1, 79.8]	$-6.8$ $[-13.2,-0.4]$	0.249
augmented (p=1)	register shift (held-out)	250	83.6 [78.5, 87.7]	$+2.0$ $[-3.6,+7.6]$	1.000
base OpenVLA	LLM paraphrases (default)	250	47.6 [41.5, 53.8]	$-38.0$ $[-44.8,-31.2]$	$<$ 0.001

Appendix

Table 26: Paraphrase penalty by paraphrase type, base OpenVLA / LIBERO-Spatial: one paraphrase drawn per episode from the

K{-}1{=}3

paraphrases of the given type (never the canonical instruction), paired with each checkpoint’s own canonical run. The augmented checkpoint trained on the LLM, lexical and structural sets only; typos/casing and the register-shifted held-out set were never seen. Generated by lcc/analysis/variant_table.py .

Arm	Canonical	Paraphrase	Held-out par.	Wrong instr.	Paraphrase cost [95% CI]
untouched checkpoint	85.6	47.6	–	2.4	$-38.0$ $[-44.8,-31.2]$
augmentation, p=1	81.6	83.6	83.6	2.8	$+2.0$ $[-4.0,+8.0]$
augmentation, p=0.5 (matched control)	82.4	78.4	–	–	$-4.0$ $[-10.0,+2.0]$
+ consistency, lambda=1	84.4	81.6	81.2	4.8	$-2.8$ $[-8.0,+2.4]$
+ consistency, lambda=0.1	81.2	82.4	–	–	$+1.2$ $[-4.4,+7.2]$

Appendix

Table 27: Explicit paraphrase-consistency term against augmentation alone, base OpenVLA / LIBERO-Spatial,

n{=}250

paired. All arms share learning rate

10^{-4}

and

2{,}000

steps. Generated by lcc/distill/report_p5.py .

Method	$n$	Succ. (%)	$\Delta$ vs. ref. [95% CI]	$p_{\text{Holm}}$
base OpenVLA / LIBERO-Spatial, reference: Canonical (greedy)
Canonical (greedy)	250	81.6 [76.3, 85.9]	—	—
Paraphrase medoid, $K{=}4$	250	84.8 [79.8, 88.7]	+3.2 [-2.4, 8.8]	0.322

Appendix

Table 28: Paraphrase consensus on the augmented checkpoint, paired against that checkpoint’s own canonical run. Generated by lcc/analysis/main_table.py .

Arm	Canonical	Paraphrase	Held-out par.	Wrong instr.	Paraphrase cost [95% CI]
untouched checkpoint	76.0	39.2	–	0.0	$-36.8$ $[-44.0,-29.6]$
augmentation, p=1	78.4	74.4	–	0.0	$-4.0$ $[-10.0,+2.0]$
canonical-only control	78.0	34.4	–	–	$-43.6$ $[-50.4,-36.4]$

Appendix

Table 29: Instruction augmentation on LIBERO-Goal (learning rate

10^{-4}

2{,}000

steps), base OpenVLA,

n{=}250

paired. The untouched checkpoint’s canonical and wrong-instruction cells are its probe runs of § 4.4 ; its paraphrase cell was run for this table. Generated by lcc/distill/report_p5.py .

Arm	Canonical	Paraphrase	Held-out par.	Paraphrase cost [95% CI]
untouched checkpoint	100.0	89.6	92.0	$-10.4$ $[-14.4,-6.8]$
augmentation, p=1	96.8	96.8	94.8	$+0.0$ $[-2.4,+2.0]$
canonical-only control	97.2	87.2	90.0	$-10.0$ $[-14.4,-5.6]$

Appendix

Table 30: Instruction augmentation on OpenVLA-OFT / LIBERO-Goal,

n{=}250

paired, against a matched control trained on the training instructions and the untouched checkpoint. Generated by lcc/distill/report_p5.py .

Robustness of VLA Models

Robustness of vision-language-action (VLA) models concerns reliable robot behavior under changes to language, vision, sensors, tasks, and hardware.

See all 219 papers

LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models
Mar 30, 2026Chanyoung Kim, Minwoo Kim, Minseok Kang +2Vision-Language Grounding Vision-Language-Action Models
School of Computer Science and Engineering, Soongsil University, Seoul, Korea · Department of Artificial Intelligence, Chung-Ang University, Seoul, Korea
When Instructions Retrieve Trajectories: Diagnosing and Mitigating Generalization Failures in VLA Models
Sep 30, 2026Hung-Jen Chen, Yu-Hsun Hou, Yan-Hong Chen +4Robotic Manipulation Vision-Language-Action Models
National Tsing Hua University · National Taiwan University
Do VLAs Really Follow Instructions? Diagnosing How-Level Instruction Following in Vision-Language-Action Models
Oct 15, 2026Sangwu Park, Wonjoong Kim, Yeonjun In +1Vision-Language-Action Models VLA Model Training
KAIST

Mar 30, 2026J/K move · Enter open · S save

LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models

Chanyoung Kim, Minwoo Kim, Minseok Kang, Hyunwoo Kim+1

School of Computer Science and Engineering, Soongsil University, Seoul, Korea · Department of Artificial Intelligence, Chung-Ang University, Seoul, Korea

Vision-Language Grounding Vision-Language-Action Models Robot Manipulation Benchmarks Instruction Generalization in VLA Models Robustness of VLA Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models achieve strong performance in robotic manipulation by leveraging pre-trained vision-language backbones. However, in downstream robotic settings, they are typically fine-tuned with limited data, leading to overfitting to specific instruction formulations and leaving robustness to paraphrased instructions underexplored. To study this gap, we introduce LIBERO-Para, a controlled benchmark that independently varies action expressions and object references for fine-grained analysis of linguistic generalization. Across seven VLA configurations (0.6B-7.5B), we observe consistent performance degradation of 22-52 pp under paraphrasing. This degradation is primarily driven by object-level lexical variation: even simple synonym substitutions cause large drops, indicating reliance on surface-level matching rather than semantic grounding. Moreover, 80-96% of failures arise from planning-level trajectory divergence rather than execution errors, showing that paraphrasing disrupts task identification. Binary success rate treats all paraphrases equally, obscuring whether models perform consistently across difficulty levels or rely on easier cases. To address this, we propose PRIDE, a metric that quantifies paraphrase difficulty using semantic and syntactic factors. Our benchmark and corresponding code are available at: https://github.com/cau-hai-lab/LIBERO-Para

Abstract

Robustness of VLA Models

Explore similar work

LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models

When Instructions Retrieve Trajectories: Diagnosing and Mitigating Generalization Failures in VLA Models

Do VLAs Really Follow Instructions? Diagnosing How-Level Instruction Following in Vision-Language-Action Models