Organizations: SKLCCSE, Beihang University, China. · School of Automation Science and Electrical Engineering, Beihang University, China. · School of Artificial Intelligence, Beihang University, China. · Zhongguancun Laboratory, China. · Institute of Dataspace, China.
Embodied agents are increasingly deployed in open-world environments, yet current robustness evaluations remain biased toward hazardous instructions and jailbreak vulnerabilities. A more fundamental issue is largely overlooked: agents are rarely tested under semantically sufficient but contextually minimal instructions (i.e., common in real-world human communication), where auxiliary contextual cues are absent. For the first, we identify a novel phenomenon termed Ambient Context Illusion, where agents fail to execute tasks correctly when non-essential contextual details are omitted, despite unchanged task semantics. To systematically study this, we establish VagueBench, a benchmark built upon the AI2-THOR interactive simulation environment for the evaluation of context dependency in embodied agents. We systematically consider key semantic components of instructions (entities, procedures, constraints, and prior knowledge) and derive a unified taxonomy with four categories and nine subtypes to reduce auxiliary information. VagueBench contains 1,303 instruction pairs across diverse indoor environments, constructed via a human-agent co-generation framework that integrates semantic redundancy estimation, taxonomy-guided transformation, and rigorous semantic-invariance verification. Through extensive evaluation on 6 state-of-the-art embodied agents, we observe significant degradation on VagueBench in both task performance and executability (relatively -19.65% in task success rate and relatively +54.10% in empty plan rate) revealing a strong reliance on auxiliary linguistic signals rather than core task semantics. Furthermore, we demonstrate that this phenomenon persists in real-world robotic systems, where it can lead to incorrect decisions and potential safety risks.
Figures & tables
Figure 1: Illustration of the Ambient Context Illusion. (a) the normal case, where the agent understand the instructions and finish the task. (b) the ambient context illusion case, where agents fail to execute tasks correctly when non-essential contextual details are omitted.
Figure 2: Overview of VagueBench and the Ambient Context Illusion paradigm, which illustrates how embodied agents rely on auxiliary contextual cues in original instructions and presents the VagueBench construction pipeline and a taxonomy of instruction transformation.
Figure 3: Taxonomy of instruction in VagueBench , consisting of four high-level categories and nine fine-grained subtypes.
Figure 4: Test example of VagueBench , consisting of an instruction pair and simulation scenario.
Figure 5: Distribution across contextual cues categories.
Metric
Kitchen
Living Room
Bedroom
Bathroom
# Samples
638
239
193
233
Percentage
49.0%
18.3%
14.8%
17.9%
Table 1: Distribution across scene types.
Model
TSR ↑ (%)
SCR ↑ (%)
PE ↓
IAR ↓ (%)
EPR ↓ (%)
Qwen2.5-VL-72B-Instruct
43.51 (-15.11)
49.88 (-14.31)
22.41 (+8.52)
22.35 (+2.29)
10.67 (+7.37)
Qwen2-VL-72B-Instruct
35.38 (-5.53)
41.92 (-6.89)
13.43 (+2.30)
26.13 (-0.21)
15.35 (+6.29)
InternVL-3-78B
38.53 (-6.91)
42.01 (-6.87)
17.57 (+1.00)
17.43 (+1.95)
33.46 (+5.83)
GPT-4o
49.12 (-9.60)
56.18 (-9.68)
0.15 (-0.00)
15.76 (+0.43)
10.28 (+8.60)
ProgPrompt
40.83 (-5.60)
49.52 (-5.83)
13.35 (-2.22)
24.09 (-1.12)
4.53 (+5.14)
ReAct
31.01 (-5.07)
38.16 (-4.82)
140.75 (+50.35)
38.67 (+10.78)
0.00 (-0.00)
Table 2: Performance comparison under original and vague instructions. Each entry shows the base value followed by the absolute change. Green indicates increase and red indicates decrease.
VagueBench Taxonomy
TSR ↑ (%)
SCR ↑ (%)
PE ↓
IAR ↓ (%)
EPR ↓ (%)
Referential Indirection
33.33 (-20.29)
41.67 (-19.08)
28.99 (+7.25)
27.68 (+6.97)
7.25 (+5.80)
Attribute Elision
35.63 (-16.34)
43.03 (-16.04)
20.47 (+0.20)
25.48 (-0.90)
10.63 (+12.80)
Quantity Fuzziness
40.00 (-24.00)
48.00 (-25.00)
0.00 (+32.00)
20.07 (+6.81)
8.00 (+20.00)
Action Replacement
40.93 (-15.04)
48.01 (-14.53)
28.98 (+13.27)
22.91 (+7.47)
9.51 (+5.75)
Action Generalization
36.76 (-22.06)
46.20 (-22.55)
7.35 (+42.65)
21.82 (+12.46)
10.29 (+7.35)
Source Omission
61.87 (-14.20)
65.22 (-13.00)
26.07 (+5.06)
16.30 (+2.39)
12.26 (+7.59)
Table 3: Performance comparison of different VagueBench taxonomy. Each entry shows the base value followed by the absolute change. Green indicates increase and red indicates decrease.
Figure 6: Diverging heatmap of relative performance changes across four categories and evaluation metrics. Positive values denote degradation under vague instructions, while negative values indicate improvement. Values are symmetrically clipped to [−1.0,1.0] for better visual comparability.
Figure 7: Real-world manipulation under original and vague instructions.
Figure 8: The mitigation effect of Context Suppression Constrain.
Figure 9: Failure cases under original and vague instructions. The agent correctly executes tasks with explicit contextual cues but fails when such information is removed.
Figure 10: Attribution maps that highlight the critical image regions driving model generation under both original and vague instruction.
Figure 11: Task performance across different scales of Qwen2.5-VL under original and vague instructions. Bars correspond to absolute TSR/SCR scores (left y-axis, black), whereas the red line indicates the relative performance drop (%) (right y-axis, red).
Figure 12: Performance degradation under increasing numbers of vague instruction types.
Modern embodied agents achieve impressive success rates, yet their actual instruction-following ability is far weaker than these numbers suggest. We trace this illusion to a structural property we term low scene entropy: when a visual scene admits only one valid task, language becomes redundant and a policy can score highly while barely using it. We introduce RoboFollow, a diagnostic benchmark with three principles: (1) High Scene Entropy: each training scene supports multiple kinematically distinct task branches, making vision alone insufficient and forcing reliance on language. (2) Hierarchical Diagnostic Protocol: a four-level protocol (L0--L3) progressively perturbs visual layout and semantics, probing whether equivalent instructions yield consistent behavior and distinct ones yield discriminable behavior across spatial relations, attributes, trajectory constraints, and logic. (3) Confound-Controlled Diagnosis: we simplify interaction objects, restrict actions to the trained repertoire and report stage-wise Intent and Execution scores, isolating comprehension from motor execution. Evaluation of nine VLA and WAM policies shows that strong L0 performance, where attained, does not reliably transfer to L1--L3 under our fine-tuning setup. Representative mitigations, including stronger VLM backbones, QA co-training, LangForce, and Classifier-Free Guidance, all fail to close this gap. RoboFollow exposes genuine instruction following as a critical, overlooked bottleneck. Code and dataset are available at https://github.com/AutoLab-SAI-SJTU/RoboFollow and https://huggingface.co/datasets/AutoLab-SJTU/robofollow-data.
Chang Guo, Yukun Xie, Bohan Tan +6
AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University · Research Lab, Anyverse Dynamics
Embodied language grounding requires more than identifying the referent of an instruction: recovered semantics must also control the action an agent exposes. We study this missing link as a semantic-action gap, where instruction semantics are recoverable but weakly expressed in native continuous actions. We introduce SAT-Bench, a fixed-observation counterfactual benchmark that holds the visual scene and agent state fixed while changing only instruction semantics. On LIBERO target-name and pixel-grounded relation swaps, target recovery reaches 100.0% and 95.8%, whereas OpenVLA action sensitivity remains only 6.8% and 7.7%. The gap persists across 1,000 additional compositional and temporal/procedural counterfactuals, with overall action sensitivity of 6.1%. Hidden-state, threshold-free, cross-policy, and rollout diagnostics further support this semantic-action transfer failure. We introduce VISA, a lightweight execution-time interface that converts recovered semantics into ALLOW, DEFER, target-consistency, and verified-selection decisions. VISA reduces invalid-instruction blind execution from 92.7% to 2.8% while preserving 94.0% of normal commands, and verified selection further improves target-consistent action exposure without updating the underlying policy. Overall, embodied language evaluation should measure semantic-action transfer, not semantic parsing alone.
Baosheng Jin, Yushen Liang, Hua Shen
Center for Data Science, NYU Shanghai Shanghai, China
Real-world requests to AI agents are fundamentally underspecified. Natural human communication relies on shared context and unstated constraints that speakers expect listeners to infer. Current agentic benchmarks test explicit instruction-following but fail to evaluate whether agents can reason about implicit requirements spanning accessibility needs, privacy boundaries, catastrophic risks, and contextual constraints. We present Implicit Intelligence, an evaluation framework testing whether AI agents can move beyond prompt-following to become genuine goal-fulfillers, paired with Agent-as-a-World (AaW), a harness where interactive worlds are defined in human-readable YAML files and simulated by language models. Our scenarios feature apparent simplicity in user requests, hidden complexity in correct solutions, and discoverability of constraints through environmental exploration. Evaluating 16 frontier and open-weight models across 205 scenarios, we find that even the best-performing model achieves only 48.3% scenario pass rate, revealing substantial room for improvement in bridging the gap between literal instruction-following and human-like contextual reasoning.