TAG: Target-Agnostic Guidance for Stable Object-Centric Inference in Vision-Language-Action Models

Jiaying Zhou; Zhihao Zhan; Ruifeng Zhai; Qinhan Lyu; Hao Liu; Keze Wang; Liang Lin; Guangrun Wang

cs.CVMar 25, 2026

TAG: Target-Agnostic Guidance for Stable Object-Centric Inference in Vision-Language-Action Models

Authors: Jiaying Zhou, Zhihao Zhan, Ruifeng Zhai, Qinhan Lyu, Hao Liu, Keze Wang, Liang Lin, Guangrun Wang

Organizations: Sun Yat-sen University · Guangdong Key Lab of Big Data Analysis & Processing · X-Era AI Lab

Abstract

Vision--Language--Action (VLA) policies have shown strong progress in mapping language instructions and visual observations to robotic actions, yet their reliability degrades in cluttered scenes with distractors. By analyzing failure cases, we find that many errors do not arise from infeasible motions, but from instance-level grounding failures: the policy often produces a plausible grasp trajectory that lands slightly off-target or even on the wrong object instance. To address this issue, we propose TAG (Target-Agnostic Guidance), a simple inference-time guidance mechanism that explicitly reduces distractor- and appearance-induced bias in VLA policies. Inspired by classifier-free guidance (CFG), TAG contrasts policy predictions under the original observation and an object-erased observation, and uses their difference as a residual steering signal that strengthens the influence of object evidence in the decision process. TAG does not require modifying the policy architecture and can be integrated with existing VLA policies with minimal training and inference changes. We evaluate TAG on standard manipulation benchmarks, including LIBERO, LIBERO-Plus, and VLABench, where it consistently improves robustness under clutter and reduces near-miss and wrong-object executions.

Figures & tables

Figure 1: Qualitative comparison of original and erased visual observations. The top row displays original robotic manipulation trajectories from the Libero and VLABench datasets. The bottom row presents the corresponding sequences where the target objects (e.g., the bowl and apple) have been digitally erased.

Figure 2: Overview of the Target-Agnostic Guidance (TAG) framework. Our method consists of two core components. The top shows the Counterfactual Synthesis pipeline, which automatically erases the manipulated target from raw videos using target parsing, tracking, and inpainting to generate unconditional data. The bottom illustrates the Policy Architecture, where a dual-branch network processes both the original conditional input and the erased unconditional input. By extracting the residual guidance ( $\Delta v$ ) between them, TAG effectively isolates background distractors, forcing the model to focus precisely on the target object and intended manipulation to generate accurate action sequences.

Figure 3: Attention map comparison between TAG and $\pi_{0.5}$ . TAG (left) precisely grounds the attention on the target and ignores distractors, resulting in successful manipulation. $\pi_{0.5}$ (right) struggles with distractor competition, showing diffused attention that leads to task failure.

Model	Spatial	Object	Goal	Long	Average
Model	SR (%) $\uparrow$	SR (%) $\uparrow$	SR (%) $\uparrow$	SR (%) $\uparrow$	SR (%) $\uparrow$
Diffusion Policy [ 9 ]	78.3	92.5	68.3	50.5	72.4
TraceVLA [ 39 ]	84.6	85.2	75.1	54.1	74.8
Octo [ 22 ]	78.9	85.7	84.6	51.1	75.1
MDT [ 27 ]	78.5	87.5	73.5	64.8	76.1
OpenVLA [ 17 ]	84.7	88.4	79.2	53.7	76.5

Table 1: Task success rates in LIBERO task. Values in parentheses indicate the absolute improvement in success rate (SR) over the corresponding baseline under identical settings. All metrics are reported as percentages (%), with the best results highlighted in bold .

Model	Camera	Robot	Language	Light	Background	Noise	Layout	Average
Model	SR (%) $\uparrow$	SR (%) $\uparrow$	SR (%) $\uparrow$	SR (%) $\uparrow$	SR (%) $\uparrow$	SR (%) $\uparrow$	SR (%) $\uparrow$	SR (%) $\uparrow$
OpenVLA [ 17 ]	0.8	3.5	23.0	8.1	34.8	15.2	28.5	15.6
WorldVLA [ 5 ]	0.1	27.9	41.6	43.7	17.1	10.9	38.0	25.0
NORA [ 14 ]	2.2	37.0	65.1	45.7	58.6	12.8	62.1	39.0
UniVLA [ 4 ]	1.8	46.2	69.6	69.0	81.0	21.2	31.9	43.9
$\pi_{0}$ [ 2 ]	13.8	6.0	58.8	85.0	81.4	79.0	68.9	53.6

Table 2: Robustness evaluation on the LIBERO-Plus benchmark. Values in parentheses indicate the absolute improvement in success rate (SR) over the corresponding baseline under identical settings. All metrics are reported as percentages (%), with the best results highlighted in bold .

$\pi_{0}$ [ 2 ]
Model	Select Toy	Select Fruit	Select Painting	Select Poker	Select Mahjong	Average
Model	SR (%) $\uparrow$	SR (%) $\uparrow$	SR (%) $\uparrow$	SR (%) $\uparrow$	SR (%) $\uparrow$	SR (%) $\uparrow$
base	54.00	48.00	16.00	6.00	6.98	26.20
+ TAG-bg (ours)	48.00	50.00	36.00	48.00	15.22	39.44 (+13.24)
+ TAG-black (ours)	52.00	50.00	40.00	34.00	11.63	37.53 (+11.33)
$\pi_{0.5}$ [ 15 ]

Table 3: Success Rate (SR) evaluation on the VLABench benchmark. We compare the baseline models(

\pi_{0}

and

\pi_{0.5}

) with our proposed TAG variants across five fine-grained selection tasks. Values in parentheses denote absolute SR improvements over the respective baselines. Best results are highlighted in bold.

Figure 4: Comparison on the VLABench benchmark. For tasks involving highly similar distractors (e.g., “Please pick the poker 3 of diamonds”), baseline policies ( $\pi_{0}$ and $\pi_{0.5}$ ) are misled by visually similar distractors and grasp the wrong cards. By leveraging the TAG strategy, our method effectively filters out these distractions, shifting attention precisely to the correct target object to prevent misjudgments and ensure an accurate grasp.

$\pi_{0}$ [ 2 ]
Model	Select Toy	Select Fruit	Select Painting	Select Poker	Select Mahjong	Average
Model	PS $\uparrow$	PS $\uparrow$	PS $\uparrow$	PS $\uparrow$	PS $\uparrow$	PS $\uparrow$
base	0.76	0.72	0.16	0.1000	0.0814	0.3643
+ TAG-bg (ours)	0.72	0.73	0.36	0.5267	0.2283	0.5130 (+0.1487)
+ TAG-black (ours)	0.73	0.73	0.40	0.3933	0.2093	0.4925 (+0.1282)
$\pi_{0.5}$ [ 15 ]

Table 4: Quantitative results on the VLABench benchmark (Process Score, PS). We evaluate the performance of baseline models (

\pi_{0}

and

\pi_{0.5}

) and our proposed TAG variants across five selection tasks. Integrating the TAG strategy consistently yields substantial performance gains for both architectures. Values in parentheses denote absolute PS improvements. Best results are highlighted in bold .

Figure 5: Visual comparison of different erase methods during inference in Libero. In the task “put both moka pots on the stove”, we evaluate various strategies for constructing the unconditional observation to apply TAG. The sequence demonstrates that alternative methods, such as simple masking (Mask-gray/black), inpainting (Erase), and full blackout (Black), disrupt essential spatial priors or introduce visual artifacts, leading to manipulation failures (e.g., colliding with the distractor cup). In contrast, our Background strategy successfully provides a stable and effective unconditional input, enabling the policy to accurately place the moka pot on the stove.

Training	Eval	Select Toy	Select Fruit	Select Painting	Select Poker	Select Mahjong	Average
Method	Method	SR (%) $\uparrow$	SR (%) $\uparrow$	SR (%) $\uparrow$	SR (%) $\uparrow$	SR (%) $\uparrow$	SR (%) $\uparrow$
B	BG	52.00	26.00	32.00	72.00	53.06	47.01
BG	BG	50.00	38.00	20.00	62.00	26.67	39.33
E	BG	60.00	48.00	38.00	78.00	53.06	55.41
B	B	42.00	30.00	36.00	70.00	57.14	47.03
BG	B	52.00	38.00	34.69	68.00	34.69	45.48

Table 5: Ablation study on unconditioning strategies during training and evaluation. We evaluate the impact of different image processing methods used for TAG: Black (B), Background (BG), and Erase (E). The results indicate that applying the Erase method during training and the Background method during evaluation yields the highest average success rate across all fine-grained selection tasks. Best results are highlighted in bold .

Real-time	Method	Spatial	Object	Goal	Long	Average
Real-time	Method	SR (%) $\uparrow$	SR (%) $\uparrow$	SR (%) $\uparrow$	SR (%) $\uparrow$	SR (%) $\uparrow$
$\times$	background	97.8	99.2	97.4	97.0	97.85
$\times$	black	97.6	99.8	97.4	95.8	97.65
$\checkmark$	gray-masked	87.8	76.6	77.6	12.6	63.65
$\checkmark$	black-masked	87.4	77.8	75.8	13.6	63.65
$\checkmark$	erase	87.6	79.4	77.2	13.8	64.50

Table 6: Ablation study on inference-time unconditional methods for

\pi_{0.5}

. We compare real-time and non-real-time strategies for generating unconditional inputs. Non-real-time static methods (background and black) significantly outperform real-time masking and erasing, successfully avoiding the severe performance drops observed in long-horizon tasks (Libero-Long). Best results are in bold .

guidance	Spatial	Object	Goal	Long	Average
guidance	SR (%) $\uparrow$	SR (%) $\uparrow$	SR (%) $\uparrow$	SR (%) $\uparrow$	SR (%) $\uparrow$
1	98.2	98.6	98.6	92.8	97.05
1.25	98.2	99.2	97.8	94.6	97.45
1.3	96.4	99.8	97.0	95.2	97.10
1.5	97.4	99.4	97.0	95.0	97.20
1.8	98.0	99.6	96.0	95.0	97.15

Table 7: Ablation study on TAG guidance scale for

\pi_{0.5}

. We evaluate the impact of different guidance scales on the Libero benchmark with a fixed 10 denoising steps. Best results are highlighted in bold .

Figure S1 : Automated Counterfactual Synthesis Pipeline for Target-Erased Video Synthesis. The framework systematically extracts target-erased videos from raw demonstration videos through a four-stage process: (1) Target Parsing : Qwen3-VL translates abstract task instructions into purely visual descriptions. (2) Detection & Masking : Grounding DINO localizes the targets. SAM 2 then generates spatio-temporal tracking masks. Here, $S$ denotes the bounding box area relative to the entire frame; (3) Inpainting : Minimax-Remover erases the target object and hallucinates the occluded background. (4) Verification : Qwen3-VL conducts a closed-loop quality assessment to discard sequences with incomplete erasures or destructive artifacts before adding them to the training dataset.

Figure S2 : Automated Counterfactual Synthesis Pipeline for Background Reference Synthesis. Adapted from the training pipeline, this process exhaustively parses all foreground objects (including the robot embodiment) from the initial frame ( $t=0$ ). The extracted targets are then processed through the same grounding, inpainting, and verification workflow described above. Here, $S$ denotes the bounding box area relative to the entire frame.

Figure S3 : Comparison of keyframes in real-world pick-block tasks under camera pose perturbations. Despite a shift in camera perspective and height, TAG exhibits cross-view robustness , completing the pick-block tasks. In contrast, the baseline $\pi_{0.5}$ is highly sensitive to viewpoint variation , suffering from spatial misalignment that leads to action stagnation above the target.

Explore similar work

Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration
Mar 6, 2026Ninghao Zhang, Bin Zhu, Shijie Zhou +1Vision-Language-Action Models Robustness of VLA Models
Tsinghua University · Singapore Management University · Institute of Trustworthy Embodied AI, Fudan University +1
Overcoming Visual Clutter in Vision Language Action Models via Concept-Gated Visual Distillation
Mar 11, 2026Sangmim Song, Sarath Kodagoda, Marc Carmichael +1Vision-Language-Action Models Robustness of VLA Models
University of Technology Sydney, NSW, Australia · Western Sydney University, NSW, Australia
ActGaze: Learning Action-Grounded Gaze through Counterfactual Visual Interventions for High-Precision Manipulation
Sep 24, 2026Jinxuan Zhu, Jiaheng Wang, Chao Tang +7Visuomotor Policy Learning Robotic Manipulation
Samsung Robotics eXperience · National University of Singapore

Mar 6, 2026J/K move · Enter open · S save

Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration

Ninghao Zhang, Bin Zhu, Shijie Zhou, Jingjing Chen

Tsinghua University · Singapore Management University · Institute of Trustworthy Embodied AI, Fudan University+1

Vision-Language-Action Models Robustness of VLA Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models enable robots to perform manipulation tasks directly from natural language instructions and are increasingly viewed as a foundation for generalist robotic policies. However, their reliability under Out-of-Distribution (OOD) instructions remains underexplored. In this paper, we reveal a critical failure mode in which VLA policies continue executing visually plausible actions even when the language instruction contradicts the scene. We refer to this phenomenon as linguistic blindness, where VLA policies prioritize visual priors over instruction semantics during action generation. To systematically analyze this issue, we introduce ICBench, a diagnostic benchmark constructed from the LIBERO dataset that probes language-action coupling by injecting controlled OOD instruction contradictions while keeping the visual environment unchanged. Evaluations on three representative VLA architectures, including Pi0, Pi0.5 and OpenVLA OFT, show that these models frequently succeed at tasks despite logically impossible instructions, revealing a strong visual bias in action generation. To mitigate this issue, we propose Instruction-Guided Attention Recalibration (IGAR), a train-free inference-time mechanism that rebalances attention distributions to restore the influence of language instructions. IGAR operates without retraining or architectural modification and can be directly applied to existing VLA models. Experiments across 30 LIBERO tasks demonstrate that IGAR substantially reduces erroneous execution under OOD contradictory instructions while preserving baseline task performance. We additionally validate the approach on a real Franka robotic arm, where IGAR effectively prevents manipulation triggered by inconsistent instructions.

Abstract

Explore similar work

Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration

Overcoming Visual Clutter in Vision Language Action Models via Concept-Gated Visual Distillation

ActGaze: Learning Action-Grounded Gaze through Counterfactual Visual Interventions for High-Precision Manipulation