STAGE: Diagnosing Semantic Transfer at Grounded Execution in Embodied Agents

Baosheng Jin; Yushen Liang; Hua Shen

cs.ROSep 11, 2026

STAGE: Diagnosing Semantic Transfer at Grounded Execution in Embodied Agents

Authors: Baosheng Jin, Yushen Liang, Hua Shen

Organizations: Center for Data Science, NYU Shanghai Shanghai, China

Abstract

Embodied language grounding requires more than identifying the referent of an instruction: recovered semantics must also control the action an agent exposes. We study this missing link as a semantic-action gap, where instruction semantics are recoverable but weakly expressed in native continuous actions. We introduce SAT-Bench, a fixed-observation counterfactual benchmark that holds the visual scene and agent state fixed while changing only instruction semantics. On LIBERO target-name and pixel-grounded relation swaps, target recovery reaches 100.0% and 95.8%, whereas OpenVLA action sensitivity remains only 6.8% and 7.7%. The gap persists across 1,000 additional compositional and temporal/procedural counterfactuals, with overall action sensitivity of 6.1%. Hidden-state, threshold-free, cross-policy, and rollout diagnostics further support this semantic-action transfer failure. We introduce VISA, a lightweight execution-time interface that converts recovered semantics into ALLOW, DEFER, target-consistency, and verified-selection decisions. VISA reduces invalid-instruction blind execution from 92.7% to 2.8% while preserving 94.0% of normal commands, and verified selection further improves target-consistent action exposure without updating the underlying policy. Overall, embodied language evaluation should measure semantic-action transfer, not semantic parsing alone.

Figures & tables

Figure 1: Fixed-observation counterfactual protocol. The observation and agent state are fixed while only the instruction semantics changes.

Policy	Domain	Intervention	$\boldsymbol{N}$	Sem.	Act.	Gap	ActCheck
OpenVLA base	LIBERO	target-name	600	100.0	6.8	93.2	–
OpenVLA-LIBERO90	LIBERO	target-name	600	100.0	5.7	94.3	–
Octo-small-1.5	LIBERO	target-name	600	100.0	42.7	57.3	–
OpenVLA base	LIBERO	pixel relation	600	95.8	7.7	88.1	45.5
Octo-small-1.5	LIBERO	pixel relation	600	95.8	37.5	58.3	42.8
Octo-small-1.5	BridgeData V2	target/task	165	100.0	66.1	33.9	limited

Table 1: Semantic-to-action transfer across policies and domains. Sem. denotes semantic recovery, Act. denotes native action sensitivity, and Gap is Sem. minus Act.; numeric values except

N

are percentages.

Figure 2: Main semantic–action gap on OpenVLA counterfactuals: semantic target signals remain high while native action sensitivity is low.

Alternative explanation	Diagnostic
Semantics absent from VLA representations	Hidden-state probe
Threshold artifact	Threshold-free AUC
Grounding-only failure	Schema–action attribution
Overlapping target directions	Spatial subsets
First-step irrelevance	Short-horizon rollouts
Simple-semantics only	Richer semantic splits

Table 2: Alternative explanations and the diagnostics used to test them.

Schema/action case	Count	Rate (%)
Correct target + aligned action	311	51.8
Correct target + wrong/ambig. action	264	44.0
Wrong target	25	4.2

Table 3: Schema–action attribution on 600 pixel-grounded relation examples.

Split	$\boldsymbol{N}$	Act. sens. (%)	SAG (%)
Compositional	600	7.0	93.0
Temporal/procedural	400	4.8	95.3
Overall	1,000	6.1	93.9

Table 4: Semantic–action transfer on richer SAT-Bench interventions.

Figure 3: Non-executable language stress test on prohibited commands.

Field	Agreement	Fleiss’ $\boldsymbol{\kappa}$	Maj. acc.
Executability	0.964	0.825	1.000
Intended target	0.987	0.975	0.997
First required subgoal	0.691	0.524	0.770
Robot decision	0.978	0.916	1.000

Table 5: Blind external validation with three non-author annotators. The lower first-subgoal agreement reflects ambiguity in procedural decomposition.

Figure 4: VISA semantic–action interface: schema extraction, admissibility checking, and action exposure.

Task family	Additional semantic fields	Effort	Inst. labels
Target-name	target / existence	low	no
Relation	anchor / relation	low–med.	no
Invalid / prohibition	blocked actions / decision	low	no
Compositional	attributes / constraints	low–med.	no
Temporal / procedural	subgoal / ordering	medium	no

Table 6: Task-family schema design requirements. Schemas are specified once per family rather than annotated per instance.

Budget	$\Delta$ Recall	95% CI	McNemar $p$
2	+0.087	[0.040, 0.137]	0.0005
4	+0.080	[0.030, 0.130]	0.0027

Table 7: Target-field ablation. Positive

\Delta

Recall denotes the recall advantage of the complete schema over removing the target field.

Figure 5: Deferral–utility balance: the minimum of invalid safe deferral and normal pass-through.

Method	Safe def. $\uparrow$	Normal pass $\uparrow$	Invalid allow $\downarrow$
Template heuristic	0.123	–	0.877
Direct Qwen	0.870	0.982	0.130
VISA gate	0.780	0.940	0.220
Hybrid	1.000	0.922	0.000

Table 8: Generalization to 300 non-template invalid instructions.

Split	Native aligned	Allow	Aligned / allow
Target swap	0.682	0.938	1.000
Compositional	0.513	0.890	1.000
Temporal/procedural	0.590	0.907	1.000

Table 9: Verified action exposure across semantic interventions. Allow denotes non-deferred coverage; Aligned / allow denotes target-consistent exposure among allowed actions.

Method	Final pref. $\uparrow$	Integrated pref. $\uparrow$	Wrong-object $\downarrow$
Native	0.0002	0.0178	0.720
VISA-Rerank	0.0083	0.0216	0.600

Table 10: Independent

K=100

rollout evaluation using metrics outside the verifier.

Method	Contact	Lift	Wrong contact/lift
Native	0.90	0.70	0 / 0
Generic selective	0.90	0.90	0 / 0
VISA selective	0.90	0.80	0 / 0
Oracle selective	1.00	0.90	0 / 0

Table 11: Preliminary true environment-step manipulation pilot (

N=10

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Diagnostic pipeline for semantic-to-action transfer under fixed-observation prompt transformations.

Figure 7: Qualitative ActCheck cases on pixel-grounded relation counterfactuals. In all cases, the schema probe recovers the counterfactual target, but the native policy action is classified as wrong-target or ambiguous. The examples illustrate that the semantic target can be available while the exposed action remains weakly aligned with the recovered target.

Method	Allow	Align.	Wrong
Native	–	0.682	0.318
VISA-Rerank	0.938	1.000	0.000

Appendix

Table 57: Target-swap exposure quality. Allow denotes non-deferred coverage. Alignment and wrong/ambiguous rates are measured among exposed actions. Because target consistency is used during verification, this table is interpreted as verifier-based exposure quality rather than independent downstream success.

Split	Native	Allow	Align.
Compositional	0.513	0.890	1.000
Temporal/procedural	0.590	0.907	1.000

Appendix

Table 58: VISA-Rerank exposure quality on richer semantic interventions. Native denotes the native target-alignment rate, Allow denotes VISA-Rerank coverage, and Align. denotes target consistency among allowed actions.

Method	Final	Int.	Wrong obj.
Native	0.0002	0.0178	0.720
VISA-Rerank	0.0083	0.0216	0.600

Appendix

Table 59: Independent

K=100

behavioral evaluation. Final and Int. denote final and integrated target preference; Wrong obj. denotes wrong-object approach. These trajectory metrics are not used by the reranking verifier.

Method	Contact	Lift	Wrong
Native	0.90	0.70	0 / 0
Generic selective residual	0.90	0.90	0 / 0
VISA selective residual	0.90	0.80	0 / 0
Oracle selective residual	1.00	0.90	0 / 0

Appendix

Table 60: Preliminary environment-step manipulation pilot (

N=10

). Wrong reports wrong contact / wrong lift. VISA corrects 2/3 native lift failures, but the generic selective baseline is stronger; we therefore do not claim VISA-specific controller superiority.

Language-Conditioned Robot Manipulation

Language-conditioned robot manipulation connects natural-language instructions to robot perception, planning, and physical actions.

See all 239 papers

RoboFollow: Unveiling the Instruction Following Mirage in Embodied Agents
Sep 22, 2026Chang Guo, Yukun Xie, Bohan Tan +6Language-Conditioned Robot Manipulation Robotic Policy Evaluation
AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University · Research Lab, Anyverse Dynamics
Encoded but Not in Control: Revealing the Grounding Gap in Vision-Language Robot Policies
Oct 5, 2026Shaohan Jiang, Jiahang Cao, Qiduo He +9Vision-Language Grounding Vision-Language-Action Models
HKU · CUHK(SZ) · Shanghai AI Lab +3
RoboSemanticBench: Diagnosing Semantic Grounding in Action Prediction for VLA Models
Jun 1, 2026Bin Yu, Yao Zhang, Haishan Liu +9Language-Conditioned Robot Manipulation Vision-Language Grounding
1HIT · 2ZGCA · 3ZGCI +6

Sep 22, 2026J/K move · Enter open · S save

RoboFollow: Unveiling the Instruction Following Mirage in Embodied Agents

Chang Guo, Yukun Xie, Bohan Tan, Zheng Chang+5

AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University · Research Lab, Anyverse Dynamics

Language-Conditioned Robot Manipulation Robotic Policy Evaluation Vision-Language-Action Models Robot Manipulation Benchmarks Embodied AI Embodied Agent Benchmarks

Open paper AskPDF arXiv

Modern embodied agents achieve impressive success rates, yet their actual instruction-following ability is far weaker than these numbers suggest. We trace this illusion to a structural property we term low scene entropy: when a visual scene admits only one valid task, language becomes redundant and a policy can score highly while barely using it. We introduce RoboFollow, a diagnostic benchmark with three principles: (1) High Scene Entropy: each training scene supports multiple kinematically distinct task branches, making vision alone insufficient and forcing reliance on language. (2) Hierarchical Diagnostic Protocol: a four-level protocol (L0--L3) progressively perturbs visual layout and semantics, probing whether equivalent instructions yield consistent behavior and distinct ones yield discriminable behavior across spatial relations, attributes, trajectory constraints, and logic. (3) Confound-Controlled Diagnosis: we simplify interaction objects, restrict actions to the trained repertoire and report stage-wise Intent and Execution scores, isolating comprehension from motor execution. Evaluation of nine VLA and WAM policies shows that strong L0 performance, where attained, does not reliably transfer to L1--L3 under our fine-tuning setup. Representative mitigations, including stronger VLM backbones, QA co-training, LangForce, and Classifier-Free Guidance, all fail to close this gap. RoboFollow exposes genuine instruction following as a critical, overlooked bottleneck. Code and dataset are available at https://github.com/AutoLab-SAI-SJTU/RoboFollow and https://huggingface.co/datasets/AutoLab-SJTU/robofollow-data.

Abstract

Language-Conditioned Robot Manipulation

Explore similar work

RoboFollow: Unveiling the Instruction Following Mirage in Embodied Agents

Encoded but Not in Control: Revealing the Grounding Gap in Vision-Language Robot Policies

RoboSemanticBench: Diagnosing Semantic Grounding in Action Prediction for VLA Models