RefGuard: Identity-Aware Language-Guided Robot Manipulation via Joint Target-Anchor-Frame Grounding

Lan Wei; Kangyi Lu; Yongchen Wang; Chenmeng Bi; Qi Chen; Hanlin Niu; Yip Fun Yeung; Dandan Zhang

cs.ROSep 5, 2026

RefGuard: Identity-Aware Language-Guided Robot Manipulation via Joint Target-Anchor-Frame Grounding

Authors: Lan Wei, Kangyi Lu, Yongchen Wang, Chenmeng Bi, Qi Chen, Hanlin Niu, Yip Fun Yeung, Dandan Zhang

Organizations: Imperial College London, UK · RACE, United Kingdom Atomic Energy Authority, UK · Oxford Robotics Institute, University of Oxford, UK · Avant Intelligence Ltd, London, UK

Abstract

Vision-language-action (VLA) models have substantially advanced language-guided robot manipulation, yet reliable execution still hinges on identifying which physical object an instruction refers to. In cluttered scenes containing repeated objects, ambiguous anchors, or frame-dependent spatial terms, a robot can execute a geometrically valid action on a semantically compatible but unintended instance; we call this failure an identity switch. The referent is jointly determined by three coupled latent variables: the target, the anchor, and the reference frame, so committing to any one of them before execution turns residual ambiguity into a silent and irreversible error. We propose RefGuard, an identity-aware grounding framework that delays commitment by maintaining a joint posterior over all three variables. RefGuard builds a frame-conditioned object-centric scene graph from RGB-D observations, separating frame-independent geometry from directional relations, and routes the posterior through a decision policy that executes, clarifies, reobserves, or aborts. On a real UF850 arm, RefGuard records no identity switch on any ambiguity-stress trial and executes correctly on 90.0% of them, whereas fine-tuned VLA and LLM (Large Language Model)-based baselines switch identity in 33-46% of the same trials, while retaining 93.3% success on unambiguous scenes and recovering from post-grounding scene changes in 86.7% of trials. On a 3200-episode procedural suite, it raises correct execution on solvable instructions from 56.6% to 80.5% over the ablation that commits to the anchor and frame before the target, while deferring less often (19.5% vs. 43.4%).

Figures & tables

Fig. 1: In cluttered tabletop manipulation, identity switches arise when ambiguity over the target, anchor, or reference frame is collapsed into hard choices before execution. RefGuard models these variables jointly, preserves their uncertainty, and routes the posterior through a decision module that chooses to execute, clarify, reobserve, or abort, reducing confident wrong-object execution.

Fig. 2: Overview of RefGuard. A language model decomposes the instruction into symbolic primitive queries over target, anchor, and relation, while SAM 3 perception builds a frame-conditioned scene graph from RGB-D observations. A joint hypothesis table scores candidate target–anchor–frame interpretations, and execution is committed only when the target belief is sufficiently concentrated for a safe action; otherwise the system clarifies, reobserves, or aborts. The scene graph is rebuilt after each primitive to support subsequent ones.

Fig. 3: Identity-switch stress suite. S1 tests ordinary unambiguous execution. S2 tests duplicate targets with a unique anchor. S3 contains duplicate targets and duplicate anchors, requiring joint target–anchor reasoning; geometrically near-tied assignments trigger clarify . S4 tests explicit frame-conditioned grounding. S5 tests robustness under execution-time perturbation. S6 is an ambiguous hard negative, where the correct behaviour is to defer rather than execute an arbitrary action.

Method	S1 Clean TS $\uparrow$	S2 Dup-T ISR $\downarrow$	S3 Dup-T/A ISR $\downarrow$	S4 Frame-Cond. ISR $\downarrow$	S5 Perturb Recovery $\uparrow$	S6 Hard-Neg. CDR $\uparrow$
GR00T	25/30 (83.3%)	13/25 (52.0%)	10/25 (40.0%)	7/25 (28.0%)	12/30 (40.0%)	N/A
$\pi_{0.5}$	27/30 (90.0%)	11/26 (42.3%)	9/25 (36.0%)	16/27 (59.3%)	14/30 (46.7%)	N/A
LLM-based	27/30 (90.0%)	12/29 (41.4%)	12/27 (44.4%)	17/30 (56.7%)	0/30 (0.0%)	N/A
RefGuard (Ours)	28/30 (93.3%)	0/28 (0.0%)	0/27 (0.0%)	0/26 (0.0%)	26/30 (86.7%)	30/30 (100.0%)

TABLE I: Main real-robot results across six UF850 settings, with 30 trials per method and setting. Each cell gives the count with the percentage in parentheses. S1 reports clean-scene task success (TS,

C/N

); S2–S4 report the execution-conditional identity-switch rate

\mathrm{ISR}=I/(C+I)

, whose denominator is the number of executed trials and therefore differs across cells (lower is better); S5 reports post-perturbation recovery; and S6 reports the correct defer rate (CDR; higher safe deferral is better). Best column results are bolded .

Fig. 4: Outcome decomposition over S2–S4 only ( $N=90$ trials per method; no macro-averaging). Episodes are categorised as correct execution/identity switch/low-level failure/defer ( $C/I/F/D$ ): GR00T $45/30/15/0$ , $\pi_{0.5}$ $42/36/12/0$ , LLM-based $45/41/4/0$ , and RefGuard $81/0/3/6$ . Percentages in the figure are episode-level outcome fractions.

Method	S2	S3	S4	Cov. $\uparrow$	CE $\uparrow$	ISR $\downarrow$
Hard-A/F	$8/1/1/0$	$4/2/1/3$	$7/1/1/1$	$26/30$	$19/30$	$4/23$
RefGuard	$9/0/1/0$	$8/0/1/1$	$9/0/0/1$	$28/30$	$26/30$	$0/26$
S5: same post-change observation and graph update
Method	Swap-T	Move-A	Insert-D	Recovery $\uparrow$	ISR $\downarrow$
LLM+Update	$8/1/1/0$	$7/2/1/0$	$7/2/1/0$	$22/30$	$5/27$
Hard-A/F+Update	$8/0/1/1$	$7/2/0/1$	$8/1/1/0$	$23/30$	$3/26$

TABLE II: Matched real-robot ablation, 10 trials per method and setting (30 per block). Entries are

C/I/F/D

: correct execution / identity switch / low-level failure / defer.

\mathrm{Cov}=(C+I+F)/N

\mathrm{CE}=C/N

, and

\mathrm{ISR}=I/(C+I)

Fig. 5: Sequential episode for “Pick the red pepper near the cup, and place it onto the plate.” Two red peppers and two cups are present, so only one joint target-anchor assignment satisfies near . Left to right: initial scene; RefGuard grasps the pepper adjacent to the right-hand cup; the pepper is held as an in-gripper memory node while the scene graph is rebuilt; the PLACE primitive is executed.

Condition	Ep.	Prim.	Prim. Comp. $\uparrow$	Seq. Succ. $\uparrow$	Wrong- ID $\downarrow$	Defer $\downarrow$
$L=2$	10	20	20/20 (100.0%)	10/10 (100.0%)	0/20	0/20
$L=4$	10	40	38/40 (95.0%)	8/10 (80.0%)	0/40	1/40 (2.5%)
$L=6$	10	60	56/60 (93.3%)	7/10 (70.0%)	0/60	2/60 (3.3%)
All	30	120	114/120 (95.0%)	25/30 (83.3%)	0/120	3/120 (2.5%)
Held-out categories (2)	18	72	68/72 (94.4%)	15/18 (83.3%)	0/72	1/72 (1.4%)

TABLE III: Sequential composition, held-out-category generalisation, free-form parsing, and post-action checkpoint results. Wrong-ID counts identity-switch executions over all primitives.

Fig. 6: Example scenes from the procedural LIBERO-style suite as clutter increases from 5 to 20 objects. These scenes are procedurally generated and are not the official LIBERO benchmark.

Method	CE@5	CE@10	CE@15	CE@20	Total solvable CE $\uparrow$
Hard-A/F	$375/600$	$354/600$	$323/600$	$306/600$	$1358/2400$ (56.6%)
RefGuard	$524/600$	$495/600$	$470/600$	$442/600$	$1931/2400$ (80.5%)

TABLE IV: Procedural LIBERO-style evaluation over 2,400 solvable and 800 hard-negative episodes. CE is correct execution; solvable deferral is lower-better, whereas safe deferral on hard negatives is higher-better.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Notation	Shape/Value	Description
$N_{c}$	2	number of RGB camera views
$X$	$224\times 224\times 3$	resized RGB observation size
$R_{\mathrm{raw}}$	$640\times 480$	raw camera resolution
$f_{\mathrm{cam}}$	30 Hz	camera frame rate
$\mathbf{s}$	$\mathbb{R}^{7}$	robot state: 6 joint angles and gripper position
$\mathbf{a}$	$\mathbb{R}^{7}$	action: 6 joint commands and gripper command

Appendix

TABLE V: Inference hyperparameters and notation used during

\pi_{0.5}

real-robot deployment.

Language-Conditioned Robot Manipulation

Language-conditioned robot manipulation connects natural-language instructions to robot perception, planning, and physical actions.

See all 230 papers

RoboIRGBench: Benchmarking Implicit Referential Grounding in Vision-Language-Action Models
Sep 28, 2026Aernaer Akelijiang, Jiannan Li, Zhineng Chen +2Language-Conditioned Robot Manipulation Vision-Language-Action Models
Singapore Management University · Fudan University
Correcting WHERE, Preserving HOW: Compositional Generalization for Vision-Language-Action Models via Referential Guidance
Sep 29, 2026Yanyan Zhang, Disheng Liu, Xinpeng Li +8Vision-Language-Action Models Test-Time Adaptation of VLA Models
Case Western Reserve University Cleveland, OH, USA
ProGAL-VLA: Grounded Alignment through Prospective Reasoning in Vision-Language-Action Models
Apr 10, 2026Nastaran Darabi, Amit Ranjan TrivediVision-Language-Action Models Vision-Language Grounding
University of Illinois Chicago, IL, USA

Sep 28, 2026J/K move · Enter open · S save

RoboIRGBench: Benchmarking Implicit Referential Grounding in Vision-Language-Action Models

Aernaer Akelijiang, Jiannan Li, Zhineng Chen, Jingjing Chen+1

Singapore Management University · Fudan University

Language-Conditioned Robot Manipulation Vision-Language-Action Models Robot Manipulation Benchmarks Robustness of VLA Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have shown strong capabilities in robotic manipulation, yet existing benchmarks typically assume that task-relevant information is explicitly specified in the instruction. In practice, however, humans frequently refer to objects, quantities, and relations implicitly, requiring robots to recover the intended target from linguistic and perceptual context. We study this capability as Implicit Referential Grounding (IRG) and introduce RoboIRG-Bench, a manipulation benchmark designed to systematically evaluate it. Built upon RoboMME, RoboIRG-Bench contains 40 variants derived from 11 tasks and covers four challenges, including direct, reasoning-mediated, spatial, and contextual referential grounding. As IRG often requires retaining and retrieving previously established context, we evaluate representative VLAs spanning different memory mechanisms. Our evaluation reveals a noticeable referential robustness gap. Models that perform well under explicit instructions can degrade sharply when the same task-relevant information must be recovered from context. Reasoning-mediated and spatial references are particularly challenging, while models using external VLMs show greater robustness but still exhibit significant failures. Moreover, replacing the external VLM with a stronger model does not eliminate these gaps. We further validate these findings on a Franka Research 3 robot arm, where the gap persists under real-world manipulation and manifests as both incorrect referent grounding and downstream execution failures. These results establish IRG as a distinct and underexplored capability for reliable robotic instruction following and highlight the need for VLAs that can robustly integrate language, perception, reasoning, and action.

Abstract

Language-Conditioned Robot Manipulation

Explore similar work

RoboIRGBench: Benchmarking Implicit Referential Grounding in Vision-Language-Action Models

Correcting WHERE, Preserving HOW: Compositional Generalization for Vision-Language-Action Models via Referential Guidance

ProGAL-VLA: Grounded Alignment through Prospective Reasoning in Vision-Language-Action Models