ProGAL-VLA: Grounded Alignment through Prospective Reasoning in Vision-Language-Action Models

Nastaran Darabi; Amit Ranjan Trivedi

cs.ROApr 10, 2026

ProGAL-VLA: Grounded Alignment through Prospective Reasoning in Vision-Language-Action Models

Authors: Nastaran Darabi, Amit Ranjan Trivedi

Organizations: University of Illinois Chicago, IL, USA

Abstract

Vision language action (VLA) models enable generalist robotic agents but often exhibit language ignorance, relying on visual shortcuts and remaining insensitive to instruction changes. We present Prospective Grounding and Alignment VLA (ProGAL-VLA), which constructs a 3D entity-centric graph (GSM), uses a slow planner to produce symbolic sub-goals, and aligns them with grounded entities via a Grounding Alignment Contrastive (GAC) loss. All actions are conditioned on a verified goal embedding $g_t$ , whose attention entropy provides an intrinsic ambiguity signal. On LIBERO-Plus, ProGAL-VLA increases robustness under robot perturbations from 30.3 to 71.5 percent, reduces language ignorance by 3x-4x, and improves entity retrieval from 0.41 to 0.71 Recall@1. On the Custom Ambiguity Benchmark, it reaches AUROC 0.81 (vs. 0.52), AUPR 0.79, and raises clarification on ambiguous inputs from 0.09 to 0.81 without harming unambiguous success. The verification bottleneck increases mutual information of language-actions, the GAC loss imposes an entity-level InfoNCE bound, and attention entropy yields calibrated selective prediction, indicating that explicit verified grounding is an effective path toward instruction-sensitive, ambiguity-aware agents.

Figures & tables

Figure 1: Overview of ProGAL-VLA . Language instruction $L$ and observation $O_{t}$ are processed by the Prospective Planner and the Grounded State Module (GSM). The State Alignment Cross Attention (SACA) module verifies alignment between the symbolic sub-goal $s_{t}$ and 3D entities $E_{t}$ , producing a verified goal embedding $g_{t}$ for the Action Policy ( $\pi_{\text{fast}}$ ). The Grounding Alignment Contrastive (GAC) objective enforces correct symbolic to entity binding during training.

Quantity	Value
Scenes (train/val/test)	$32/8/8$
Objects per scene	$3/4.8/6$
(min / mean / max)
Object categories	blocks, mugs,
	bottles, fruit
Attributes	colors: 4 $\approx 25\%$ each;

Table 1: Summary of Custom Ambiguity Benchmark.

Model	Camera	Robot	Language	Light	Background	Noise	Layout	Total
OpenVLA [ 16 ]	0.8	3.5	23.0	8.1	34.8	15.2	28.5	17.3
OpenVLA-OFT [ 15 ]	56.4	31.9	79.5	88.7	93.3	75.8	74.2	70.0
OpenVLA-OFT_w [ 15 ]	10.4	38.7	70.5	76.8	93.6	49.9	69.9	56.4
NORA [ 13 ]	2.2	37.0	65.1	45.7	58.6	12.8	62.1	39.8
WorldVLA [ 6 ]	0.1	27.9	41.6	43.7	17.1	10.9	38.0	25.3
UniVLA [ 5 ]	1.8	46.2	69.6	69.0	81.0	21.2	31.9	43.9

Table 2: Robustness across perturbation dimensions on LIBERO-Plus [ 9 ] . Bold values mark the best score in each column. The bottom row reports ProGAL-VLA along with absolute gains over the base OpenVLA model.

Figure 2: Success rate and language-ignorance error on LIBERO-Plus [ 9 ] across different input configurations ( $L$ , $s_{t}$ , and $g_{t}$ ). Conditioning on the verified grounding $g_{t}$ yields both higher task success and lower language-ignorance error, consistent with the analysis in Sec. 4 .

Figure 3: Selective prediction performance on CAB. Risk-Coverage curves from entropy-based selective policies show that ProGAL-VLA maintains strictly lower failure risk across all coverage levels, indicating better calibrated uncertainty and more reliable abstention on ambiguous inputs.

Model	AUROC $\uparrow$	AUPR $\uparrow$	ECE $\downarrow$	Cov@95 $\uparrow$	FPR@95 $\downarrow$	Clar@Ambig $\uparrow$	Unambig SR $\uparrow$	Total $\uparrow$
OpenVLA [ 16 ]	0.52	0.49	14.3	0.31	0.72	0.09	0.74	0.47
ProGAL (w/o $\mathcal{L}_{\text{GAC}}$ )	0.66	0.63	9.8	0.57	0.41	0.42	0.82	0.64
ProGAL (entropy-only)	0.73	0.70	7.1	0.63	0.33	0.58	0.84	0.69
ProGAL-VLA (Ours)	0.81	0.79	4.6	0.78	0.18	0.81	0.89	0.79

Table 3: Ambiguity detection and selective prediction on CAB. Bold entries mark the best performance. AUROC, AUPR, Cov@95, Clar@Ambig, and Unambig SR are higher is better; ECE and FPR@95 are lower is better. Total is the macro average of all normalized ambiguity metrics.

	Retrieval (SACA logits)				Language-ignorance
Model	N=8 R@1 $\uparrow$	N=8 R@5 $\uparrow$	N=16 R@1 $\uparrow$	N=32 R@1 $\uparrow$	Simple $\downarrow$	Spatial $\downarrow$	Relational $\downarrow$
OpenVLA [ 16 ]	0.41	0.72	0.28	0.15	0.36	0.49	0.57
ProGAL (w/o $\mathcal{L}_{\text{GAC}}$ )	0.57	0.86	0.43	0.29	0.22	0.31	0.45
ProGAL ( $L\!\rightarrow\!\pi_{\text{fast}}$ )	0.50	0.83	0.35	0.24	0.27	0.33	0.42
ProGAL (Ours)	0.71	0.93	0.58	0.41	0.08	0.14	0.19

Table 4: Summary of entity retrieval from SACA logits and language-ignorance (lower is better). For retrieval, we report Recall@1 with varying candidate set sizes.

Detector	GSM	SACA	$\pi_{\text{fast}}$	Total
43.0 $\pm$ 26.4	15.8 $\pm$ 9.7	10.7 $\pm$ 6.6	26.9 $\pm$ 16.5	96.4 $\pm$ 59.2

Table 5: Per step latency of ProGAL-VLA on LIBERO-Plus [ 9 ] over 100 episodes. Values are mean and standard deviation in milliseconds. Throughput: 10.31 FPS.

Prospective Planner ( $\pi_{slow}$ )
Component	Specification
Model Backbone	Qwen-2.5-VL-Instruct-7B
Input Resolution	$448\times 448$
Output Format	Symbolic template (e.g., grasp_green_mug )
Invocation Frequency	Once per episode (or when grounding fails)
Grounded State Module (GSM)

Table 6: Hyperparameters for ProGAL-VLA Components.

	Robot Perturbation		Camera Perturbation
Model	Joint Drift	Joint Jitter	Viewpoint Shift	FOV Change
OpenVLA [ 16 ]	2.1	4.9	1.2	0.4
OpenVLA-OFT+ [ 15 ]	28.5	35.1	68.4	78.0
ProGAL-VLA (Ours)	68.2	74.8	85.1	91.7

Table 7: Robustness Breakdown (Success Rate %). “Drift” refers to systematic joint offsets; “Jitter” refers to high-frequency noise.

Model	Grounding Fail	Grasp Fail	Planning Fail	Success Rate
OpenVLA [ 16 ]	41.2%	12.5%	5.1%	41.2%
ProGAL (w/o $\mathcal{L}_{GAC}$ )	22.4%	14.1%	4.8%	58.7%
ProGAL-VLA	6.3%	7.1%	4.5%	82.1%

Table 8: Failure Mode Distribution (Lower is better). Values are percentages of total episodes.

Panel of Figure 4 — Figure 4: Fine-grained Robustness Analysis on LIBERO-Plus. We visualize the success rates across seven perturbation dimensions (x-axis) for four task suites (Spatial, Object, Goal, Long). While baselines such as OpenVLA and $\pi_{0}$ exhibit significant performance degradation under Camera and Robot perturbations, ProGAL-VLA maintains high robustness across most dimensions.

Explore similar work

Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration
Mar 6, 2026Ninghao Zhang, Bin Zhu, Shijie Zhou +1Vision-Language-Action Models Robustness of VLA Models
Tsinghua University · Singapore Management University · Institute of Trustworthy Embodied AI, Fudan University +1
Grounded Semantic Re-Binding for Robust Instruction Generalization in Vision-Language-Action Models
Aug 3, 2026Zhaokai Yin, Zhipeng ZhangVision-Language Grounding Vision-Language-Action Models
1AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University · 2Research Lab, Anyverse Dynamics
In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use
Aug 6, 2026Jiarui Yang, Wen Huang, Jiale Zhang +2ICL for Robotics Continual Learning for VLA Models
Nankai University · Tsinghua University · Swiss Federal Institute of Technology in Lausanne (EPFL)

Mar 6, 2026J/K move · Enter open · S save

Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration

Ninghao Zhang, Bin Zhu, Shijie Zhou, Jingjing Chen

Tsinghua University · Singapore Management University · Institute of Trustworthy Embodied AI, Fudan University+1

Vision-Language-Action Models Robustness of VLA Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models enable robots to perform manipulation tasks directly from natural language instructions and are increasingly viewed as a foundation for generalist robotic policies. However, their reliability under Out-of-Distribution (OOD) instructions remains underexplored. In this paper, we reveal a critical failure mode in which VLA policies continue executing visually plausible actions even when the language instruction contradicts the scene. We refer to this phenomenon as linguistic blindness, where VLA policies prioritize visual priors over instruction semantics during action generation. To systematically analyze this issue, we introduce ICBench, a diagnostic benchmark constructed from the LIBERO dataset that probes language-action coupling by injecting controlled OOD instruction contradictions while keeping the visual environment unchanged. Evaluations on three representative VLA architectures, including Pi0, Pi0.5 and OpenVLA OFT, show that these models frequently succeed at tasks despite logically impossible instructions, revealing a strong visual bias in action generation. To mitigate this issue, we propose Instruction-Guided Attention Recalibration (IGAR), a train-free inference-time mechanism that rebalances attention distributions to restore the influence of language instructions. IGAR operates without retraining or architectural modification and can be directly applied to existing VLA models. Experiments across 30 LIBERO tasks demonstrate that IGAR substantially reduces erroneous execution under OOD contradictory instructions while preserving baseline task performance. We additionally validate the approach on a real Franka robotic arm, where IGAR effectively prevents manipulation triggered by inconsistent instructions.

Abstract

Explore similar work

Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration

Grounded Semantic Re-Binding for Robust Instruction Generalization in Vision-Language-Action Models

In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use