From Association to Decision Value: Testing Paired Instruction Dependence for VLA Corpus Selection

Nurmukhambet Serikbayev; Abdul Basit; Muhammad Shafique

cs.AIOct 21, 2026

From Association to Decision Value: Testing Paired Instruction Dependence for VLA Corpus Selection

Authors: Nurmukhambet Serikbayev, Abdul Basit, Muhammad Shafique

Organizations: eBRAIN Lab, Division of Engineering, NYUAD, Abu Dhabi, UAE

Abstract

Choosing among robot demonstration corpora is a ranking problem: an upstream diagnostic must preserve which candidates produce better downstream policies, not merely correlate with average performance. We study paired instruction dependence, the rate at which a demonstrator changes its first decision when only the goal changes. Across 136 independently seeded quadruped-navigation corpora, predictions developed on 56 corpora are evaluated prospectively on 80 held-out corpora. The audit is reliable and associated with downstream paired correctness, and lowers mean absolute prediction error when added to a correctness-aware baseline, but the primary proper-log-score increment is +0.059 nats (95% corpus-bootstrap CI [-0.027, 0.133]) and the prespecified top-20% comparison does not establish an incremental selection gain. An exploratory response decomposition suggests a structural source of this predictive-ranking mismatch: under the tested predictor specification, separating both-correct and both-wrong responses yields larger held-out gains than their unsigned switch statistic. Seed-to-seed agreement indicates that the held-out endpoint retains recoverable rank signal. These results show that counterfactual response structure can be reliable and predictive while providing limited incremental decision value in this setting when compressed into an unsigned statistic, motivating diagnostics that preserve outcome-relevant structure and are validated against the decisions they are intended to support.

Figures & tables

Figure 1: Why prediction is not yet selection; (a) A schematic third-person view illustrates the same-observation goal swap used to test whether the demonstrator changes its first response. (b) In the primary held-out test, adding $A_{d}$ lowers MAE $.110\!\to\!.096$ while Spearman falls $.589\!\to\!.490$ . (c) The prespecified top-20% rule ranks 80 corpora and selects 16, so ordering determines utility.

Figure 2: Prospective test of paired response structure; Audit states and training trajectories are disjoint, and only the goal changes within an audit pair. Baseline $M_{0}$ uses marginal per-goal correctness and corpus entropies; $M_{1}$ adds the same-state switch rate $A_{d}$ . Both models predict paired learner correctness and rank the same held-out corpora.

Figure 3: Experimental design across simulation and hardware; (a) Paired states span three semantic families with RGB fixed across goals. (b) Audit, training, and evaluation observations are disjoint. (c) Controlled scene shift balances a $2\!\times\!2$ geometry/appearance design. (d) Stationary real images evaluate 16 adapters; 48 physical runs describe two deployed adapters.

Figure 4: Association, prediction, and selection answer different questions; (a) Held-out corpora, a 15-corpus moving average, and the realized top 20%. (b) Equal-count bins with corpus-bootstrap intervals and corpus MAE. (c) The 20% operating point was prespecified before outcome reveal; its displayed confidence interval and the remaining curve are exploratory.

Figure 5: Incremental gains are heterogeneous and uncertain; Light bands distinguish negative from positive gain. Points are held-out corpora, half-densities show gain distributions, markers are equal-corpus means, and bars are corpus-bootstrap 95% intervals. Every interval crosses zero and every mean lies below the prespecified magnitude reference.

Setting	Gain [95% CI]	MAE	Spearman	Selected
Qwen2.5 ID	$+.059\ [-.027,.133]$	$.110\!\to\!.096$	$.589\!\to\!.490$	$.838\!\to\!.811$
Qwen2.5 shift	$+.029\ [-.030,.086]$	$.106\!\to\!.097$	$.460\!\to\!.391$	$.945\!\to\!.917$
Qwen3.5 ID	$+.036\ [-.005,.075]$	$.077\!\to\!.073$	$.752\!\to\!.745$	$.965\!\to\!.971$
Qwen3.5 shift	$+.010\ [-.010,.030]$	$.080\!\to\!.078$	$.729\!\to\!.723$	$.969\!\to\!.964$

Table 1: Cross-setting summary; gain is

M_{1}-M_{0}

log score. Arrows show

M_{0}\!\to\!M_{1}

; lower MAE and higher Spearman/selected correctness are better.

Selector	Gain [95% CI]	Spearman	Selected	Regret	Overlap
$M_{0}$	—	.589	.838	.078	.312
$M_{1}=M_{0}+A_{d}$	$+.059\ [-.027,.133]$	.490	.811	.105	.250
$M_{0}+a_{d}$	$+.104\ [.049,.153]$	.560	.846	.070	.312
$M_{0}+b_{d}$	$+.110\ [.022,.188]$	.531	.842	.074	.250
$M_{0}+$ training first-correct	$+.156\ [.078,.225]$	.584	.831	.085	.312
Demonstrator rollout success	—	.281	.838	.078	.312

Table 2: Exploratory selection baselines on Qwen2.5-ID; all methods rank the same 80 corpora;

K=16

. Gain is proper log score relative to the prespecified

M_{0}

when defined. The seed-to-seed row is an empirical resolution benchmark rather than a deployable selector.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Feature	Definition	Same-state pairs?	Correctness labels?	Range
Per-goal correctness	Fraction of goal-conditioned queries with the correct demonstrator branch	No	Yes	.438–.792
Instruction entropy	Entropy of training instruction labels	No	No	1.564–1.585
Action entropy	Entropy of coarse training actions	No	No	.902–1.524
$A_{d}$	Fraction of paired states whose first choice changes	Yes	No	0–.750

Appendix

Table 3: Measurements entering

M_{0}

and

M_{1}

. Marginal correctness and

A_{d}

reuse the same 48 demonstrator responses but summarize them differently.

Feature	Range	Pearson $r$	Spearman $\rho$	$M_{0}$ coef.	$M_{1}$ coef.
Per-goal correctness	.438–.792	.718	.462	.842	.541
Instruction entropy	1.564–1.585	$-.470$	$-.454$	$-.001$	.113
Action entropy	.902–1.524	.815	.564	.966	.549
$A_{d}$	0–.750	1.000	1.000	—	1.089
$\log_{10}$ checkpoint episodes	1.000–4.699	.570	.566	—	—

Appendix

Table 4: Baseline feature ranges, correlation with

A_{d}

over 136 corpora, and standardized coefficients. Checkpoint maturity is included only in the exploratory sensitivity.

Factor	Levels or sampling	Status
Corpus population	independently trained RL demonstrators	varying
Split	56 development / 80 test	disjoint
Demonstrator checkpoint	10, 50, 250, 1,000, 50,000 episodes	test: 16 each; dev.: 12/11/11/11/11
Demonstrator and scene seeds	unique training, layout, physics, rendering seeds	varying
Goal families	three semantic object pairs, balanced sides	fixed design
Audit states	24 paired states / 48 counterfactual queries	fixed quota

Appendix

Table 5: Target corpus population. All varying seeds are disjoint across development and test sets; nested states and goals do not increase the independent-corpus count.

Check	Result	Interpretation
Audit reliability, all 136	$.802\ [.688,.859]$	stable corpus property
Audit reliability, test 80	$.766\ [.523,.853]$	stability persists on test corpora
Primary gain, pooled seeds	$+.059\ [-.027,.133]$	positive, uncertain
Bootstrap replicates $>0$	92.3%	positive direction
Bootstrap replicates $>.105$	12.5%	12.5% exceed benchmark
Learner seed 20262001	$+.067\ [-.051,.164]$	positive, uncertain

Appendix

Table 6: Sensitivity of the primary Qwen2.5-ID conclusion. The prespecified row remains primary; all other rows are exploratory checks. Intervals resample whole corpora unless explicitly described as audit-measurement resampling.

Episodes	Corpora	$a$	$b$	$c$	Halt/query
10	28	.156	.016	.827	.015
50	27	.389	.074	.537	.098

Appendix

Table 7: Joint audit correctness at the two least mature checkpoints.

a

b

, and

c

denote both correct, both wrong, and exactly one correct.

Method	Gain [95% CI]	Spearman	Selected	Regret	Overlap
$M_{0}+a_{d}$	$+.104\ [.049,.153]$	.560	.846	.070	.312
$M_{0}+b_{d}$	$+.110\ [.022,.188]$	.531	.842	.074	.250
$M_{0}+$ train first-correct	$+.156\ [.078,.225]$	.584	.831	.085	.312
$M_{0}+$ acceptance	$+.000\ [.000,.000]$	.589	.838	.078	.312
Demonstrator rollout success	—	.281	.838	.078	.312
Qwen3.5 pooled outcomes	—	.715	.866	.050	.438

Appendix

Table 8: Exploratory Qwen2.5-ID diagnostics. Selected correctness, regret, and overlap use

K=16

; gain is relative to the prespecified

M_{0}

where available.

Model	Mid-PIT mean	Mid-PIT variance	50% cov.	80% cov.	90% cov.
$M_{0}$	.486	.061	.725	.925	.956
$M_{1}$	.487	.054	.763	.938	.963

Appendix

Table 9: Held-out Qwen2.5-ID predictive calibration.

Figure 6: Exploratory diagnostics of the remaining paired signal; Light bands label the interpretation of each region. (a) Marginal per-goal correctness contributes substantially beyond corpus entropies, whereas the paired increment is smaller and uncertain. (b)

A_{d}

has a weak relationship with the signed residual left by

M_{0}

. (c) The four regions distinguish improvements in MAE, proper log score, both, or neither; confident misses can lower MAE while worsening proper score.

Backbone	Setting	Correct	Gain	95% CI
Qwen2.5	ID	.708	+.059	$[-.027,.133]$
Qwen2.5	Shift	.801	+.029	$[-.030,.086]$
Qwen3.5	ID	.761	+.036	$[-.005,.075]$
Qwen3.5	Shift	.776	+.010	$[-.010,.030]$

Appendix

Table 10: Test-corpus predictive results. Correct is mean paired correctness across test corpora and learner seeds. Gain is

M_{1}-M_{0}

in nats; intervals resample whole corpora.

		Abs. error		Pearson		Spearman
Backbone	Setting	$M_{0}$	$M_{1}$	$M_{0}$	$M_{1}$	$M_{0}$	$M_{1}$
Qwen2.5	ID	.110	.096	.887	.900	.589	.490
Qwen2.5	Shift	.106	.097	.908	.921	.460	.391
Qwen3.5	ID	.077	.073	.927	.934	.752	.745
Qwen3.5	Shift	.080	.078	.923	.925	.729	.723

Appendix

Table 11: Absolute prediction error and rank diagnostics. Lower absolute error is better; higher correlation is better.

		Selected		Regret		Oracle overlap
Backbone	Setting	$M_{0}$	$M_{1}$	$M_{0}$	$M_{1}$	$M_{0}$	$M_{1}$
Qwen2.5	ID	.838	.811	.078	.105	.312	.250
Qwen2.5	Shift	.945	.917	.055	.083	.250	.188
Qwen3.5	ID	.965	.971	.031	.026	.562	.625
Qwen3.5	Shift	.969	.964	.031	.036	.438	.438

Appendix

Table 12: Top-20% corpus-selection diagnostics. Selected is realized mean correctness; regret is relative to the oracle top set. Regret and overlap are descriptive point estimates.

Figure 7: Supporting analyses; Controlled-shift performance remains substantial in every geometry/appearance cell; audit-size prefixes do not improve selection monotonically; and only models with an adequate instruction-conditioned action interface enter the test-corpus comparison.

Cell	Qwen2.5	Qwen3.5
Geometry A / appearance A	.797	.756
Geometry A / appearance B	.794	.793
Geometry B / appearance A	.822	.785
Geometry B / appearance B	.790	.770

Appendix

Table 13: Paired correctness in the balanced geometry

\times

appearance shift.

Figure 8: Local response and embodied completion diverge; (a) Lines pair first-command correctness and target success within each of 16 rank-stratified corpora. (b) Exact episode flows expose 134 later failures after a correct first command. (c) First-command and task-success intervals resample corpus groups; zero collision/fall and recovery counts are descriptive.

Figure 9: A correct local decision can still fail downstream; Representative simulated trajectories show success after a correct first command, failure after a correct first command, and failure after a wrong first command.

Figure 10: Physical deployment transfers the local decision but does not establish strict terminal success; (a) Representative first/last policy views. (b) Executed actions; red outlines mark seven known strict failures. (c) Recorder output, operator judgment, and terminal verification distinguish three evidential levels.

Endpoint	Qwen2.5	Qwen3.5	Evidence
Correct first choice	24/24	23/24	recorder log
Operator-reported completion	24/24	23/24	operator judgment
Strict success	not verifiable		terminal evidence unavailable
Known strict failure	1/24	6/24	objective violation/report
Executed forward actions	72	78	controller log
Safety FWD $\rightarrow$ HALT vetoes	20	29	depth-mask log

Appendix

Table 14: Physical Go2 results for one fixed adapter per backbone. All counts are descriptive; the study was not designed to compare backbones.

Figure 11: The physical logs expose motion, latency, and safety mediation; (a) All odometric paths after translating and rotating their starting poses to a common frame. (b) Empirical inference-latency distributions over 267 decisions; dashed lines mark model medians. (c) The depth mask changes 49 proposed forward actions to halt , affecting 35/48 runs. The study therefore evaluates a VLA–controller–safety stack rather than unconstrained VLA stopping.

Element	Recorded study	Consequence
Decision horizon	six rather than eight	terminal rule differs from plan
Trial status	40 pilot; 8 unspecified	all 48 marked nonconforming
Instruction string	17/48 non-exact	wording is not fully standardized
Terminal evidence	no external video, target coordinates, distance, or adjudication	strict success is not estimable
Configuration metadata	required pretrial files absent	geometric conditions cannot be fully reconstructed
Run order	grouped rather than counterbalanced	order effects are uncontrolled

Appendix

Table 15: Departures from the planned physical evaluation protocol. They restrict the deployment study to descriptive embodied-transfer evidence.

Figure 12: Stationary physical images provide breadth across adapters; (a) Six of 36 physical-Go2 states across three layouts and surrogate object pairs. (b) Matched first-command accuracy for the same 16 Qwen2.5 and Qwen3.5 corpus adapters in simulation and on physical images. The aggregate right-goal weakness does not imply that every selected adapter has the same asymmetry.

Operation	Median	Mean	Output
Paired audit rendering	4.18 min	4.78 min	24 states / 48 queries
Training-trajectory rendering	10.61 min	11.35 min	48 trajectories
Qwen2.5 adapter fit	—	10.68 min	one learner seed
Qwen2.5 evaluation generation	—	0.78 min	48 goal-conditioned queries
Qwen3.5 adapter fit	13.08 min	13.41 min	one learner seed (121 measured fits)

Appendix

Table 16: Measured per-corpus cost. Rendering used one simulator GPU; adapter time is per fit.

Item	Configuration
Base artifacts	Qwen2.5-VL-3B-Instruct, Qwen3.5-VL-0.8B.
Input processor	One current RGB frame per decision; Qwen pixel cap 102,400; architecture-native chat template; action history serialized oldest first. No explicit token truncation beyond the base processor/model limit.
Quantization	NF4 4-bit weights, double quantization, bfloat16 compute; vision tower held fixed.
LoRA	Rank 8, alpha 16, dropout .05, no bias; language modules q/k/v/o_proj , gate/up/down_proj ; all visual modules excluded and asserted absent.
Optimization	AdamW (PyTorch), learning rate $2\times 10^{-4}$ , cosine schedule, zero warmup, weight decay 0, batch size 1, gradient accumulation 8, three epochs (105–135 optimizer updates as corpus steps vary from 273–358).
Decoding	Greedy (no sampling), at most 16 new tokens; strict five-class grammar {forward, left, right, halt, done}; invalid generations count as errors.

Appendix

Table 17: Learner and demonstrator configuration held fixed within each backbone. Qwen3.5 is a supporting replication under the same learner recipe.

Robot Learning

Robot learning develops methods for robots to acquire skills and adapt from demonstrations, interaction, data, and physical models.

See all 321 papers

Oct 14, 2026

Paraphrase Consensus Is Not Self-Consistency: Instruction Reliance and Robustness in VLA Policies

Vision-language-action (VLA) policies fine-tuned on LIBERO are trained and evaluated on the same ten instruction strings, and rewording the instruction changes base OpenVLA's success by -38.0 points. Test-time prompt ensembling, which improves language models, does not recover this loss in any setting we ran. Paraphrase consensus (the medoid action over K paraphrases) costs base OpenVLA 16.8 points when the training instruction is a member. Without it, the medoid of the paraphrases scores -32.4 points relative to the training instruction, and confidence-based selection scores -28.0. The same medoid over samples of one paraphrase improves on greedy decoding but stays -27.6 points below the training instruction, consistent with aggregation reducing variance without removing the bias the paraphrases share. A wrong-instruction probe measures how much of a policy's success depends on the instruction. Across three policies and four suites, paraphrases cost a policy only where this share is high, and the stronger policies lose a much smaller fraction of it than base OpenVLA. LoRA post-training on paraphrased instructions reduces the one-paraphrase effect to +2.0 points, against -32.0 for a matched control, reproduces across two seeds and on OpenVLA-OFT, transfers to withheld phrasings, and leaves the policy's reliance on the instruction unchanged.

Hanlong Li, Sihan Tan, Takeshi Ashizawa +2

Institute of Science Tokyo, Tokyo, Japan

Robustness of VLA Models Instruction Generalization in VLA Models

Ask PDF

Aug 24, 2026

InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation

Vision-language-action (VLA) models have made general-purpose robot manipulation increasingly plausible by conditioning robot actions on natural-language instructions. A key test of such generality is whether policies actually follow language instructions. Yet many manipulation benchmarks leave this ability underdetermined: the intended object or destination is often visually salient or uniquely feasible, allowing policies to succeed without grounding the instruction. We argue that instruction-following evaluation should be text-indispensable: multiple actions should be visually and physically plausible, while only one should be consistent with the language instruction. We introduce InstructMove, a text-indispensable benchmark for instruction-following manipulation. InstructMove instantiates this principle in pick-and-place scenes with semantic distractors, decomposing instruction following into category identification, attribute discrimination, spatial reasoning, and compositional pick-and-place. InstructMove supports a train-eval protocol with InstructMove training data and held-out evaluation tasks, with additional diagnostics for language dependence. Experiments with representative VLA policies show that InstructMove provides a controlled testbed for diagnosing visual shortcuts and that InstructMove simulation data can improve real-world instruction-following manipulation performance. Code: https://github.com/HorizonRobotics/RoboOrchardSim

Mengao Zhao, Ziang Li, Chaodong Huang +15

Horizon Robotics · Southeast University · Huazhong University of Science and Technology +1

Robot Manipulation Benchmarks Language-Conditioned Robot Manipulation Robotic Manipulation Vision-Language-Action Models

Ask PDF

Oct 23, 2026

TTT in VLA Models: Exploring Training with Human Demonstrations

Long-horizon robot tasks require retaining and using past information. We integrate Test-Time Training (TTT), which stores context in fast weights, into InternVLA-A1 and Pi0.5 to study action generation conditioned on human demonstrations through training on paired demonstrations and robot trajectories. We implement memory transfer between the two phases and truncated backpropagation, and examine training objectives, gradient coverage, and contrastive regularization across video, short-video, and single-frame cue tasks. In our experiments, adding TTT and training on paired data at the current scale did not lead to consistent target selection based on demonstrations. Under the tested settings, shorter sequences, full trajectory gradient connectivity, and contrastive regularization did not clearly improve responsiveness to demonstrations. When demonstrations were reduced to single-frame cues, real-robot execution still showed a preference for cups at fixed positions. We document the implementation, training configurations, and observations to support further inspection and replication.

Qiangong Zhou, Jiafeng Liang, Yimin Dong +2

Standard Robots Co., Ltd.

VLA Model Training Vision-Language-Action Models Learning from Demonstration Robot Imitation Learning Temporal Modeling for VLA Models

Ask PDF

Abstract

Robot Learning

Explore similar work

Paraphrase Consensus Is Not Self-Consistency: Instruction Reliance and Robustness in VLA Policies

InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation

TTT in VLA Models: Exploring Training with Human Demonstrations