Correcting WHERE, Preserving HOW: Compositional Generalization for Vision-Language-Action Models via Referential Guidance

Yanyan Zhang; Disheng Liu; Xinpeng Li; Chaoda Song; Mohsen Hariri; Debargha Ganguly; Wang Yang; Kai Ye

cs.ROSep 29, 2026

Correcting WHERE, Preserving HOW: Compositional Generalization for Vision-Language-Action Models via Referential Guidance

Authors: Yanyan Zhang, Disheng Liu, Xinpeng Li, Chaoda Song, Mohsen Hariri, Debargha Ganguly, Wang Yang, Kai Ye, +3 more

Organizations: Case Western Reserve University Cleveland, OH, USA

Abstract

While Vision-Language-Action (VLA) models enable flexible action generation, their generalization across diverse environmental elements, including manipulated objects, destinations, and backgrounds, is limited by the lack of diversity in robotic training data. Trained end-to-end on such data, VLAs tend to exploit visual shortcuts, associating actions with task-irrelevant visual features rather than the intended task semantics. These shortcuts block recomposition of elements already seen by the policy, that is, compositional generalization. Existing approaches mitigate such entanglement through task-relevant perception or targeted data diversification, but offer no explicit mechanism for unseen recomposition and require backbone-specific modifications with retraining. We observe that under such recomposition, VLAs often fail at global grounding while retaining local manipulation skills that recover near the correct target in familiar configurations. Therefore, we propose Referential Guidance (ReGuide), a training-free wrapper that, given object poses from a grounding module, combines semantic and geometric rebinding to guide the end-effector into demonstration-supported configurations of the instructed referent, where the frozen policy can resume execution. Experiments in simulation across multiple VLA backbones as well as on a real robot show that ReGuide improves success rates under compositional shifts by up to 56.8 and 75.0 percentage points, respectively, while preserving standard-task performance.

Figures & tables

Figure 1: Approaches to compositional failures. Foundation VLAs over-rely on visual features. Previous methods require coverage-bound retraining or use test-time steering and external oracles, while ReGuide corrects the referent and preserves the frozen policy’s manipulation competence. The second and third panels from the left correspond to strands 1–2 and strand 3 respectively.

Figure 2: Overview of ReGuide. Semantic rebinding monitors referent alignment and interaction progress. Geometric rebinding transfers demonstration-derived configurations and coordinates guidance, frozen-policy execution and recovery.

Figure 3: Compositional Tasks on LIBERO . Each setting changes exactly one factor in a training task while keeping the others fixed. Together, they isolate where compositional generalization fails and test whether the policy follows instructions beyond memorized task configurations.

Figure 4: Real-robot evaluation on the xArm6 . The setup includes five trained pairings, eight recombined pairings, and two unseen object cases across pick and place and pushing . Bare $\pi_{0.5}$ and ReGuide are evaluated on the same paired layouts per cell.

	Method	Year	Orig.	Recombination Axes				Average
	Method	Year	Orig.	Target Region	Target Object	Unseen Object	Task Scene	All	Comp. Avg.
Found.	OpenVLA-OFT	2025	$\mathbf{97.6}_{\pm 1.4}$	$13.5_{\pm 4.7}$	$1.5_{\pm 1.9}$	$34.5_{\pm 6.5}$	$6.5_{\pm 3.5}$	$46.2_{\pm 2.7}$	$14.0_{\pm 2.4}$
	$\pi_{0}$	2025	$86.0_{\pm 3.0}$	$5.0_{\pm 3.1}$	$1.5_{\pm 1.9}$	$27.0_{\pm 6.1}$	$0.5_{\pm 1.3}$	$38.3_{\pm 2.6}$	$8.5_{\pm 1.9}$
	$\pi_{0.5}$	2025	$94.0_{\pm 2.1}$	$40.5_{\pm 6.7}$	$25.0_{\pm 6.0}$	$49.5_{\pm 6.9}$	$41.0_{\pm 6.8}$	$60.2_{\pm 2.7}$	$39.0_{\pm 3.4}$
	X-VLA	2026	$\mathbf{97.6}_{\pm 1.4}$	$39.0_{\pm 6.7}$	$9.5_{\pm 4.1}$	$34.0_{\pm 6.5}$	$0.0_{\pm 0.9}$	$50.2_{\pm 2.7}$	$20.6_{\pm 2.8}$
	GR00T N1.7	2026	$94.6_{\pm 2.0}$	$14.5_{\pm 4.9}$	$9.5_{\pm 4.1}$	$28.0_{\pm 6.2}$	$0.0_{\pm 0.9}$	$44.4_{\pm 2.7}$	$13.0_{\pm 2.3}$

Table 1: Success rate (%) on the single-factor composition evaluation of LIBERO. Each axis recombines one factor of a training task and pools four cells

\times

50 paired trials. Orig. reports the ten unmodified anchor tasks under the same protocol; All pools composition and Orig. trials, while Comp. Avg. pools the 16 composition cells only. Subscripts are 95% Wilson confidence half-widths.

Method	Pick-and-Place			Push		Average (%)
Method	Orig.	Comp.	Unseen	Orig.	Comp.	Orig.	Comp.	All
Bare $\pi_{0.5}$	26/30	7/60	8/20	17/20	0/20	$86.0_{\pm 9.6}$	$15.0_{\pm 7.0}$	$38.7_{\pm 7.7}$
+ ReGuide	28/30	53/60	17/20	18/20	20/20	$92.0_{\pm 7.8}$	$90.0_{\pm 6.0}$	$90.7_{\pm 4.7}$

Table 2: Real-robot evaluation on the xArm6. Entries are successes/trials for trained ( Orig. ), recombined ( Comp. ), and unseen-object ( Unseen ) tasks. Average success rates (%) pool 50 original, 100 recombined and unseen-object, or all 150 trials per method. Subscripts give 95% Wilson confidence half-widths. Both methods use the same paired layouts.

Figure 5: Real-robot success per object and destination pairing. Off the trained diagonal the bare policy succeeds on one pairing (7/10); ReGuide reaches at least 7/10 on every pairing.

Figure 6: Paired rollouts under recomposition. The bare policy acts on the habitual referent, while ReGuide transfers the interaction configuration to the instructed referent and guides execution there.

Variant	Recombination Axes				Comp. Avg.	$\bm{\Delta}$ vs Full
Variant	Target Region	Target Object	Unseen Object	Task Scene	Comp. Avg.	$\bm{\Delta}$ vs Full
Bare $\pi_{0.5}$	$40.5_{\pm 6.7}$	$25.0_{\pm 6.0}$	$49.5_{\pm 6.9}$	$41.0_{\pm 6.8}$	$39.0_{\pm 3.4}$	$-55.5^{***}$
Naive transport	$43.5_{\pm 6.8}$	$70.5_{\pm 6.3}$	$75.5_{\pm 5.9}$	$84.5_{\pm 5.0}$	$68.5_{\pm 3.2}$	$-26.0^{***}$
w/o competing referents	$86.5_{\pm 4.7}$	$77.0_{\pm 5.8}$	$\underline{91.5}_{\pm 3.9}$	$88.5_{\pm 4.4}$	$85.9_{\pm 2.4}$	$-8.6^{***}$
w/o chunk commitment	$89.0_{\pm 4.4}$	$83.5_{\pm 5.1}$	$81.5_{\pm 5.4}$	$\underline{91.5}_{\pm 3.9}$	$86.4_{\pm 2.4}$	$-8.1^{***}$
w/o pre-contact sets	$87.0_{\pm 4.7}$	$70.5_{\pm 6.3}$	$75.5_{\pm 5.9}$	$70.5_{\pm 6.3}$	$75.9_{\pm 3.0}$	$-18.6^{***}$

Table 3: Component-group ablations and hand-back assistance (success, %). The first 4 removals test semantic and geometric rebinding; the final removal isolates post-hand-back assistance.

Figure 7: When, how often and at what cost ReGuide intervenes. Classes are the verdicts of ReGuide, excluding one-step transports cancelled at a clause switch.

Figure 8: Failure stages and parameter sensitivity. (a) Failure stage by removed component. (b) No tested parameter variant differs significantly from the default under paired McNemar tests.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$\pi_{\theta}$ , $A_{t}$ , $H$ , $h$	frozen policy, action chunk, chunk length, executed actions per replan
$a=(u,w,g)$ , $x_{t}=(p_{t},R_{t})$	translation, rotation and gripper command, end-effector pose
$\sigma$ , $\kappa$	command scales of the controller, tracking gains fitted on demonstrations
$e$ , $c_{e}$ , $R_{e}$ , $b_{e}$	scene entity, its position, orientation and bounding-box half-extents
$e_{k}^{\star}$ , $q_{t}$	instructed referent of stage $k$ , moving point (end-effector or held object)
$\mathcal{R}_{t}$ , $\hat{q}_{t}$ , $\hat{R}_{t}$	unexecuted chunk remainder, predicted endpoint and posture

Appendix

Table 4: Notation used in the method and its implementation details.

Figure 9: Geometric identity. Left, height and largest width of the 16 training types (1 black bowl, 2 alphabet soup, 3 bbq sauce, 4 black book, 5 butter, 6 chocolate pudding, 7 cream cheese, 8 ketchup, 9 milk, 10 moka pot, 11 orange juice, 12 porcelain mug, 13 salad dressing, 14 tomato sauce, 15 white and yellow mug, 16 wine bottle) and of the three unseen objects of the benchmark, with an arrow to the type whose statistics they take. Right, the distance $\|\log\phi(e)-\log\phi(e^{\prime})\|$ over the four components of the descriptor between each unseen object and every type. The macaroni and cheese box and the yellow book take the black book, at 0.33 and 0.19, and the white bowl takes the black bowl, at 0.58.

Figure 10: Height profiles of the ten destination classes. The height of the bottom of the held object above the destination plane is plotted against the horizontal distance to the destination, as the median

m_{f}

(solid) and the lower envelope

\underline{m}_{f}

, the

Q_{.05}

(dashed), in bins of 2 cm with at least 20 samples. Numbers in parentheses are the contributing demonstrations. In the three masked classes the envelope stays below the destination plane over the final approach, and a transport onto them keeps the current height.

Figure 11: Per-type demonstration statistics of the LIBERO training suites. The commitment radius

\rho_{e}

is the

Q_{.25}

of the distance to the box surface at the entry event, the entry altitude

\bar{z}

the

Q_{.5}

of the height of the end-effector above the object top at the entry event, and the gripper band the

Q_{.95}

of the gripper width at the start of a carry. Numbers in parentheses are the demonstrations of the type. The entry altitude is close to zero for the wine bottle and the ketchup.

Statistic	Measured on	Level and role
tracking gains $\kappa_{p}$ , $\kappa_{R}$	realized against commanded motion, per demonstration	$Q_{.5}$ , unit conversion
step bounds $s$ , $s_{R}$	per-step displacement and rotation, approach and carry	$Q_{.95}$ , upper bound
commitment radius $\rho_{e}$	distance to the box at the entry event, per object type	$Q_{.25}$ , trigger
carry radius	distance to the destination during demonstrated carries, per class	$Q_{.25}$ , trigger
entry altitude $\bar{z}$	height above the object top at the entry event, per type	$Q_{.5}$ , target
grasp offset $\bar{o}$ , rotation $\bar{R}$	end-effector pose at engagement in the canonical frame, per slice	median and medoid, target

Appendix

Table 5: Demonstration statistics read by ReGuide. A slice is a pair of object type and task.

Figure 12: Composition cells of the Target Region and Target Object axes. For every cell the left image is the initial state of the unmodified anchor task and the right image the initial state of the cell, rendered from the first initial state of its evaluation bank. The changed factor of each cell is listed in Table 6 .

Figure 13: Composition cells of the Unseen Object and Task Scene axes, in the layout of Fig. 12 . For the Task Scene cells the left image shows the source task in its own scene and the right image the same task on the host surface.

Cell	Host scene and anchor task	Change	Instruction
A1	Spatial, bowl at table centre to plate	destination is the stove	pick up the black bowl from table center and place it on the stove
A2	Goal, cream cheese to bowl	destination is the open top drawer	put the cream cheese in the top drawer of the wooden cabinet
A3	Goal, bowl to plate	destination is the open top drawer	put the black bowl in the top drawer of the wooden cabinet
A4	Goal, wine bottle to rack	destination is the stove	put the wine bottle on the stove
B1	Living room 6, white mug to plate	the mug is replaced in place by a moka pot	put the moka pot on the plate
B2	Living room 6, white mug to plate	the target is the chocolate pudding of the scene	put the chocolate pudding on the plate

Appendix

Table 6: The 16 composition cells. Axes A to D are Target Region, Target Object, Unseen Object and Task Scene.

Method	Checkpoint	Execution
$\pi_{0}$ , $\pi_{0.5}$	official LIBERO checkpoints of openpi	official client, 5 actions executed per replan, chunks of 50 actions for $\pi_{0}$ and 10 for $\pi_{0.5}$
OpenVLA-OFT	official checkpoint trained on the four suites	chunk of 8 executed open loop, two images and proprioception
X-VLA	official LIBERO checkpoint	official client with absolute end-effector targets, the full chunk of 30 actions executed open loop
GR00T N1.7	fine-tuned by us on the four suites, see below	official horizon, 16 actions predicted and 8 executed
OTTER	trained by us with the official code, as no checkpoint is released, see below	official evaluation loop, one action per step with a context of 12 frames
GuidedVLA	released checkpoint, built on $\pi_{0}$ , see below	official server with corrected depth-adapter checkpoint loading

Appendix

Table 7: Checkpoints and execution settings.

Figure 14: Pre-contact initialization diagnostic. Default and pre-contact starts for frozen

\pi_{0.5}

. (a) Pooled grasp success with 95% confidence intervals. (b) Per-cell grasp success; O5 and O6 are multi-clause tasks excluded from the pooled original-task results. (c) Cumulative grasp success from the first decision, including initialization time.

Metric	Task type	Default start	Pre-contact start
Grasp	Original	400/400	362/400
	Recomposed	258/400	383/400
Task success	Original	397/400	343/400
	Recomposed	136/400	266/400

Appendix

Table 8: Diagnostic outcomes. Grasp requires lifting the instructed object more than 3 cm with attachment maintained. Task success uses the original benchmark criterion. All trials in the pooled cells are retained.

Figure 15: The ten original tasks O1 to O10 of Table 10 , each shown at the first initial state of its evaluation bank.

Method	A1	A2	A3	A4	B1	B2	B3	B4	C1	C2	C3	C4	D1	D2	D3	D4	$\Sigma$
OpenVLA-OFT	0	0	27	0	3	0	0	0	3	0	31	35	0	13	0	0	112
$\pi_{0}$	0	0	10	0	0	0	3	0	0	7	0	47	0	0	1	0	68
$\pi_{0.5}$	1	5	42	33	4	13	9	24	3	19	27	50	0	42	37	3	312
X-VLA	0	47	31	0	1	6	12	0	0	2	16	50	0	0	0	0	165
GR00T N1.7	0	0	8	21	2	1	10	6	0	1	9	46	0	0	0	0	104
OTTER	0	0	0	0	0	0	0	0	0	0	15	11	0	0	1	0	27

Appendix

Table 9: Successes out of 50 per composition cell.

Method	O1	O2	O3	O4	O5	O6	O7	O8	O9	O10	$\Sigma$
OpenVLA-OFT	49	49	50	50	46	46	50	50	48	50	488
$\pi_{0}$	49	49	40	50	5	40	49	48	50	50	430
$\pi_{0.5}$	50	50	47	50	31	44	50	50	48	50	470
X-VLA	49	49	46	50	48	47	50	50	50	49	488
GR00T N1.7	50	49	49	49	34	44	50	50	48	50	473
OTTER	46	42	31	50	21	30	41	50	50	50	411

Appendix

Table 10: Successes out of 50 per original task. O1 black bowl from table centre to plate, O2 cream cheese to bowl, O3 wine bottle to rack, O4 bowl to plate, O5 both moka pots to stove, O6 white mug to plate and chocolate pudding to the right of the plate, O7 alphabet soup, O8 ketchup, O9 salad dressing, O10 milk to basket.

Figure 16: Every ReGuide episode in simulation as one row, composition cells above and original tasks below, sorted within each cell by the start of the first transport. Light blue marks the steps of an approach transport and dark blue those of a carry transport. Orange marks a transport start without a landing, that is a transport that was dissolved, hijacked or cancelled, and the squares on the right mark failed episodes.

Figure 17: Causes of the transports issued on the 800 composition episodes (left) and the step of the first transport of an episode (right, boxes are quartiles and whiskers the range without outliers). Wrong binding denotes a predicted commitment to an incorrect object during approach or a habitual destination during carrying. Posture covers the posture mismatch and the unseen-object rule, and the remaining causes are the failure to bind and the completion certificates.

Figure 18: Change of the episode length under ReGuide per composition cell, in steps, over the pairs of trials solved by both the bare backbone and ReGuide. The number of pairs is given above each box, and cells with fewer than three pairs are left empty.

Variant	A1	A2	A3	A4	B1	B2	B3	B4	C1	C2	C3	C4	D1	D2	D3	D4	$\Sigma$
Naive transport	35	2	0	50	7	50	50	34	16	38	48	49	39	34	46	50	548
w/o competing referents	34	49	45	45	49	49	14	42	44	48	44	47	38	43	48	48	687
w/o chunk commitment	49	41	43	45	45	49	50	23	25	47	49	42	44	44	49	46	691
w/o pre-contact sets	50	48	43	33	28	49	49	15	23	29	50	49	47	40	29	25	607
w/o bounded transport	50	31	41	46	45	41	49	40	49	50	38	43	41	44	47	46	701
ReGuide (full)	50	48	46	48	49	49	49	39	48	46	46	49	49	42	48	50	756

Appendix

Table 13: Ablation on

\pi_{0.5}

, successes out of 50 per cell.

Quantity	Default	Variant	$k/800$	Success (%)	Default only	Variant only	$p$
Default	–	–	756	94.5 ±1.6	–	–	–
$n$	3	1	760	95.0 ±1.5	27	31	0.694
$n$	3	5	759	94.9 ±1.5	25	28	0.784
$\varepsilon$ (mm)	4	2	755	94.4 ±1.6	34	33	1.000
$\varepsilon$ (mm)	4	8	752	94.0 ±1.7	36	32	0.716
trigger level	$Q_{.25}$	$Q_{.50}$	766	95.8 ±1.4	25	35	0.245

Appendix

Table 15: Sensitivity to the declared constants and to two design forms, on the 800 composition trials with

\pi_{0.5}

. The last columns count trials solved only by the default or only by the variant.

Cell	Full	No assistance
A1	50	48
A2	48	49
A3	46	43
A4	48	48
B1	49	35
B2	49	48

Appendix

Table 16: Post-hand-back assistance by composition cell. Successes out of 50. No assistance disables post-hand-back constraints and completion assistance.

Grounding	Target Region	Target Object	Unseen Object	Task Scene	Comp. Avg.
Oracle	$96.0_{\pm 2.8}$	$93.0_{\pm 3.6}$	$94.5_{\pm 3.2}$	$94.5_{\pm 3.2}$	$94.5_{\pm 1.6}$
Perception	$86.0_{\pm 4.8}$	$86.5_{\pm 4.7}$	$91.5_{\pm 3.9}$	$86.0_{\pm 4.8}$	$87.5_{\pm 2.3}$

Appendix

Table 18: ReGuide with oracle and perception-based grounding on

\pi_{0.5}

(compositional success rate, %). Each composition axis contains 200 trials. Subscripts are 95% Wilson confidence half-widths. Registration failures are included.

Action	No hand-back	Before grasp	After grasp
VLA (full)	120/139 (86.3)	510/531 (96.0)	126/130 (96.9)
Default	126/147 (85.7)	52/513 (10.1)	127/140 (90.7)
Hold	121/141 (85.8)	0/529 (0.0)	103/130 (79.2)

Appendix

Table 19: Outcomes by first hand-back stage. Successes/trials with percentages in parentheses. Groups are defined within each rollout, so their membership can differ across conditions.

Cell	Full	Default	Hold	Cell	Full	Default	Hold
A1	50	50	49	C1	48	0	0
A2	48	2	1	C2	46	1	0
A3	46	9	9	C3	46	0	0
A4	48	47	42	C4	49	0	0
B1	49	0	0	D1	49	34	0
B2	49	15	13	D2	42	42	37

Appendix

Table 21: Hand-back action replacement by cell. Successes out of 50.

Method	A	B	C	D	Comp.	Orig.	All
Bare $\pi_{0.5}$	1/4	0/4	2/4	1/4	4/16	10/10	14/26
Harness VLA	3/4	1/4	2/4	4/4	10/16	8/10	18/26
ReGuide	4/4	4/4	3/4	4/4	15/16	8/10	23/26

Appendix

Table 22: Exploratory comparison with Harness VLA. Successes/trials on one paired initial state per cell. A–D denote the four composition axes. These counts are separate from the 50-trial-per-cell evaluation.

Measurement	Bare $\pi_{0.5}$	ReGuide
Policy query	127.2 / 128.1	126.7 / 127.7
Replanning latency	129.0 / 129.9	129.2 / 130.3
Client computation per step	1.60 / 1.81	2.14 / 2.61
Client computation (99th percentile)	2.04	3.49
Simulator step (median)	10.3	10.1

Appendix

Table 23: Runtime of oracle-grounding ReGuide and the bare policy. Timings in milliseconds after the first decision. Each entry reports median / 90th percentile unless otherwise stated.

Figure 19: One evaluation layout per pick-and-place cell with trained objects, as seen by the base camera (left) and the wrist camera (right) at the start of the episode. Rows group the original and the recombined pairings.

Figure 20: One evaluation layout per unseen-object and push cell, same views as Fig. 19 . In the unseen-object cells one training object is replaced by the unseen one.

Figure 21: Displacement of the object along the push line after contact in the 20 recombined push episodes (left) and its value when the push is declared complete (right). The dashed lines are the demonstrated quantiles. In 19 episodes the completion line is reached, and one episode is stopped by the operator at 45 mm with the object already past the lower quantile.

Figure 22: Outcome of the 150 paired layouts of the real-robot study. Each square is one layout of one cell, coloured by whether both methods, only ReGuide, only the bare policy or neither succeeded.

Figure 23: Left, the update of a transport target by the refined perception, in motion and on arrival, 536 updates over the 110 pick-and-place episodes of ReGuide with a median of 19 mm. Right, the displacement of the target measured by the arrival refinement, 51 measurements.

Figure 24: Share of real-robot trials solved within a given time. One successful bare trial on an original cell has no recorded duration and is not drawn.

Explore similar work

Aug 3, 2026cs.RO

Grounded Semantic Re-Binding for Robust Instruction Generalization in Vision-Language-Action Models

Vision-Language-Action (VLA) models excel in robotic manipulation but suffer catastrophic performance drops when canonical instructions are simply paraphrased. Although this brittleness is typically addressed through costly data scaling, our probing reveals that the root cause is architectural rather than a lack of semantic understanding. Specifically, we demonstrate that current VLAs successfully retain the correct task identity internally. The failure actually stems from the joint encoding of dynamic visual observations and text, which introduces systematic feature shifts. Because the downstream action policy is highly vulnerable to these variations, it fails to translate the preserved semantics into correct control commands. To resolve this structural bottleneck, we propose Grounded Semantic Re-binding (GSR), an elegant intervention that bypasses unstable joint routing by explicitly fusing independently extracted task semantics with native visual features to train a completely re-initialized action expert from scratch. This targeted intervention dramatically restores paraphrastic invariance using only canonical demonstrations. On the LIBERO-Para benchmark, GSR improves success rates by up to 44.6 percent. It enables lightweight models to rival massively scaled baselines and pushes state-of-the-art models to a new record PRIDE score of 70.4, outperforming the recently introduced large-scale pretrained model Xiaomi-Robotics-0 in instruction generation capabilities. Building on these insights, we also introduce ParaVLA, a natively decoupled 0.33B-parameter model exhibiting near-perfect robustness to instruction rewording. Ultimately, our work proves that robust semantic grounding can be achieved through elegant structural design, bypassing the inefficient brute-force data scaling paradigm.

Zhaokai Yin, Zhipeng Zhang

1AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University · 2Research Lab, Anyverse Dynamics

Robotic Manipulation Action Expert

Ask PDF

May 12, 2026cs.RO

GuidedVLA: Specifying Task-Relevant Factors via Plug-and-Play Action Attention Specialization

Vision-Language-Action (VLA) models aim for general robot learning by aligning action as a modality within powerful Vision-Language Models (VLMs). Existing VLAs rely on end-to-end supervision to implicitly enable the action decoding process to learn task-relevant features. However, without explicit guidance, these models often overfit to spurious correlations, such as visual shortcuts or environmental noise, limiting their generalization. In this paper, we introduce GuidedVLA, a framework designed to manually guide the action generation to focus on task-relevant factors. Our core insight is to treat the action decoder not as a monolithic learner, but as an assembly of functional components. Individual attention heads are supervised by manually defined auxiliary signals to capture distinct factors. As an initial study, we instantiate this paradigm with three specialized heads: object grounding, spatial geometry, and temporal skill logic. Across simulation and real-robot experiments, GuidedVLA improves success rates in both in-domain and out-of-domain settings compared to strong VLA baselines. Finally, we show that the quality of these specialized factors correlates positively with task performance and that our mechanism yields decoupled, high-quality features. Our results suggest that explicitly guiding action-decoder learning is a promising direction for building more robust and general VLA models.

Xiaosong Jia, Bowen Yang, Zuhao Ge +17

Institute of Trustworthy Embodied AI (TEAI), Fudan University · 2Shanghai Key Laboratory of Multimodal Embodied AI · Shanghai Jiao Tong University +1

Recent Vision-Language Models Guidance Task-Relevant Information Plug-And-Play

Ask PDF

Jul 17, 2026cs.RO

AC-VLA: Robust Out-of-Distribution Action Execution via Compositional Learning

Vision-Language-Action (VLA) models excel at end-to-end robotic manipulation but struggle with out-of-distribution (OOD) generalization when familiar sub-tasks are recombined in unseen configurations. We identify two mutually reinforcing failure modes: \emph{trajectory overfitting}, where models overfit to holistic trajectory patterns rather than compositional sub-skill semantics; and \emph{perceptual shortcut}, where action tokens over-rely on wrist-view textures at the expense of global spatial grounding. To address both, we introduce \textbf{AC-VLA}, a plug-and-play Action Compositional learning framework comprising two architecture-agnostic components: \textbf{(i)} a compositional learning module that uses an LLM-driven instruction decomposer and a proprioceptive trajectory aligner to generate dense sub-task supervision, followed by mixed training on complete demonstrations and decomposed data to endow the model with compositional generalization; and \textbf{(ii)} a state-conditioned asymmetric masking strategy that suppresses wrist-view inputs during closed-gripper phases, enforcing global semantic grounding. All components are architectural modification-free and directly integrable into any VLA backbone. Instantiated on

π_{0.5}

and evaluated on LIBERO and LIBERO-OOD benchmarks, AC-VLA achieves a ~28% absolute improvement on compositional OOD tasks while maintaining near-perfect in-distribution performance.

Xiaojiang Peng, Kai Peng, Jie Lu +3

Shenzhen Technology University · Tongji University · Great Bay University +1

Robotic Manipulation Spatial Grounding Out-Of-Distribution

Ask PDF

Cell	Full	Default	Hold	Cell	Full	Default	Hold
A1	50	50	49	C1	48	0	0
A2	48	2	1	C2	46	1	0
A3	46	9	9	C3	46	0	0
A4	48	47	42	C4	49	0	0
B1	49	0	0	D1	49	34	0
B2	49	15	13	D2	42	42	37

Cell	Full	Default	Hold	Cell	Full	Default	Hold
A1	50	50	49	C1	48	0	0
A2	48	2	1	C2	46	1	0
A3	46	9	9	C3	46	0	0
A4	48	47	42	C4	49	0	0
B1	49	0	0	D1	49	34	0
B2	49	15	13	D2	42	42	37

Cell	Full	Default	Hold	Cell	Full	Default	Hold
A1	50	50	49	C1	48	0	0
A2	48	2	1	C2	46	1	0
A3	46	9	9	C3	46	0	0
A4	48	47	42	C4	49	0	0
B1	49	0	0	D1	49	34	0
B2	49	15	13	D2	42	42	37