If, Then, Otherwise: Diagnosing Conditional Branching in Vision-Language Navigation
Authors: Seoyoung Lee, Neel P. Bhatt, Pranay Samineni, Cong Liu, S P Sharan, Timothy Barclay, Gregory M. Wagner, Daniel Milan, Sandeep Chinchali, Ufuk Topcu, Atlas Wang
Organizations: The University of Texas at Austin · Collins Aerospace
Vision-language navigation agents are often evaluated on their ability to follow route-like instructions toward a fixed goal. Yet, real navigation instructions often depend on observed states of the environment: if a condition holds, then follow one path, otherwise take another. Such instructions require an agent to evaluate scene evidence, select the correct logical branch, and execute the corresponding navigation behavior. Existing evaluations provide limited control over conditional branch execution, making it difficult to determine whether agents fail because of perception, grounding, navigation, or logical decision-making. We introduce CondVLN, a scene-graph-grounded benchmark for diagnosing conditional branching in vision-language navigation. CondVLN programmatically generates instructions whose branch conditions are grounded in verifiable 3D scene-graph predicates, with controlled variation in branch depth, dependency chain length, spatial composition, evidence observability, and instruction horizon. CondVLN contains over 11,500 generated conditional instructions across AI2-THOR, Matterport3D, Gibson, and ReplicaCAD, and evaluates agents using standard VLN metrics and branch-specific diagnostics: Branch Selection Accuracy and Conditional Success Rate. Evaluating four state-of-the-art VLN agents (VLN-Zero, NaVid, NaVILA, and Open-Nav) shows that conditional branching exposes failures that are not captured by standard success rate or path length alone: agents can navigate plausibly while committing to a branch inconsistent with the observed scene condition. We also present a lightweight neurosymbolic branch-selection model that separates condition grounding from navigation execution, improving performance by 2x. CondVLN provides a reusable testbed for measuring whether embodied agents can not only follow instructions, but follow the right instruction under the right condition.
Figures & tables
Figure 1: Overview of approach.
Category
Depth (nesting)
Chain Length (branches)
Description
Simple
1
1
Single IF-ELSE
Nested
2
1
IF within IF
Deep Nested
3
1
Multi-level nesting
Chain
1
2
IF / ELSE IF
Long Chain
1
3
Multi-branch chain
Nested Chain
2
2
Nested + chained
Table 1: Taxonomy of conditional navigation instruction complexity.
Model
ReplicaCAD
AI2-THOR
Gibson
Matterport3D
SR
SPL
BSA
CSR
SR
SPL
BSA
CSR
SR
SPL
BSA
CSR
SR
SPL
BSA
CSR
NaVid
16.3
16.0
15.6
9.20
18.4
16.3
12.8
11.2
1.0
0.9
1.5
0.0
5.1
2.58
2.4
0
NaVILA
0.0
0.0
17.3
0.0
0.0
0.0
16.5
0.0
0.0
0.0
3.0
0.0
2.0
1.88
7.0
0.0
Open-Nav
29.0
11.5
39.2
33.3
54.0
22.5
31.8
29.9
12.0
7.0
1.6
1.1
18.0
8.0
11.4
7.1
VLN-Zero
28.0
26.6
20.3
14.0
47.0
37.6
31.5
21.0
8.0
6.1
6.8
2.1
19.0
19.0
14.1
9.9
Oracle
22.0
22.0
15.3
12.0
46.0
37.7
34.5
29.0
7.0
4.9
6.3
2.1
16.0
14.7
10.1
5.8
Table 2: Main results on CondVLN averaged across all instruction categories. We report branch selection accuracy (BSA) and conditional success rate (CSR).
Model
Base
Logical Depth Sweep
Chain Length Sweep
d=1,ℓ=1
d=2,ℓ=1
d=3,ℓ=1
d=1,ℓ=2
d=1,ℓ=3
BSA
CSR
BSA
CSR
BSA
CSR
BSA
CSR
BSA
CSR
NaVid
5.67
1.25
10.88
5.88
10.29
7.50
16.68
10.14
7.48
2.27
NaVILA
5.35
0.00
7.22
0.00
4.28
0.00
12.06
0.00
10.42
0.00
Open-Nav
22.94
17.21
17.70
14.06
14.59
12.27
21.88
11.51
13.37
11.29
VLN-Zero
17.20
8.97
23.30
11.78
13.87
10.08
25.76
18.49
13.36
9.91
Table 3: Scaling analysis on CondVLN . The base case with depth d=1 and chain length ℓ=1 is shared by both the logical-depth and chain-length sweeps and is therefore reported only once.
Conventional Vision-and-Language Navigation (VLN) benchmarks assume instructions are feasible and the referenced target exists, leaving agents ill-equipped to handle false-premise goals. We introduce VLN-NF, a benchmark with false-premise instructions where the target is absent from the specified room and agents must navigate, gather evidence through in-room exploration, and explicitly output NOT-FOUND. VLN-NF is constructed via a scalable pipeline that rewrites VLN instructions using an LLM and verifies target absence with a VLM, producing plausible yet factually incorrect goals. We further propose REV-SPL to jointly evaluate room reaching, exploration coverage, and decision correctness. To address this challenge, we present ROAM, a two-stage hybrid that combines supervised room-level navigation with LLM/VLM-driven in-room exploration guided by a free-space clearance prior. ROAM achieves the best REV-SPL among compared methods, while baselines often under-explore and terminate prematurely under unreliable instructions. VLN-NF project page can be found at https://vln-nf.github.io/.
Hung-Ting Su, Ting-Jun Wang, Jia-Fong Yeh +2
National Taiwan University · DRIC · National Tsing Hua University
Goal-conditioned visual navigation has been a long-standing testbed for embodied AI. We study a natural language-conditioned variant, language-conditioned visual navigation (LCVN), in which an embodied agent must follow a natural language instruction given only an initial egocentric observation. Without access to goal images, the agent must rely on language to shape its perception and continuous control. We introduce the LCVN Dataset, a benchmark of 39,016 trajectories and 117,048 human-verified instructions spanning diverse environments and instruction styles. Building on this benchmark, we study two complementary paradigms: (i) latent-imagination policy learning, in which a diffusion-based world model (LCVN-WM) imagines future observations and an actor-critic agent (LCVN-AC) learns its policy entirely within the imagined latent space; and (ii) unified autoregressive prediction, in which a single multimodal backbone (LCVN-Uni) jointly predicts actions and observations in one forward pass over a shared token sequence. Experiments show that two paradigms offer complementary strengths: latent imagination produces more temporally coherent rollouts, whereas unified prediction generalizes better to unseen environments. Targeted ablations further isolate the contributions of language guidance, conditioning signals, and instruction style, clarifying when language grounding versus dynamics modeling is the performance bottleneck. Together, these findings position LCVN as a testbed for studying how language, imagination, and decision-making interact in embodied agents.
Yifei Dong, Fengyi Wu, Yilong Dai +8
University of Washington · National University of Singapore · Clemson University +2
Vision-Language Navigation (VLN) requires embodied agents to generate actions based on instructions and observations. General-purpose multimodal agents offer a promising basis for this task, but selecting plausible local actions does not ensure that execution remains consistent with the intended route, particularly in long-horizon tasks. Moreover, the accumulated interaction history increases the input required for subsequent decisions, resulting in a significant inference overhead. To this end, we introduce \method, an Agentic VLN framework that includes a Goal Agent that sets adaptive goals for local actions, a Verify Agent that dynamically verifies whether a goal has been completed, a Memory Agent for multimodal context compression, and a Visuomotor Agent to execute adaptive goals. Specifically, the Goal Agent formulates adaptive goals based on the instruction, current observation, and execution history. Then the Visuomotor Agent executes navigation actions to achieve each goal, while the Verify Agent uses a goal-specific verification question to dynamically assess whether the observed outcomes satisfy the intended completion condition. Verified goal completion then marks a boundary for the Memory Agent to compress the corresponding multimodal interaction history while preserving information needed for subsequent navigation. We evaluate navigation on R2R-CE and RxR-CE, examine framework variants across three model backbones, and study context evolution during execution. For Real-World evaluation, \method achieves 83.3% success and 1.51,m navigation error across eight challenging routes evaluated three times each.
Haoxiang Shi, Zaijing Li, Muhe Ding +3
Harbin Institute of Technology (Shenzhen) · Pengcheng Laboratory