TraversalBench: Challenging Paths to Follow for Vision Language Models
Authors: Clara Petrova, Zhuo Chen, Marin Soljačić
Organizations: Massachusetts Institute of Technology, Department of Physics · NSF AI Institute for Artificial Intelligence and Fundamental Interactions · Massachusetts Institute of Technology, Institute for Data, Systems, and Society
Vision-language models (VLMs) perform strongly on multimodal benchmarks, but their ability to follow complex visual paths remains under-tested. We introduce TraversalBench, a controlled benchmark for exact visual path traversal. Each instance contains a continuous polyline with a unique start marker and labeled vertices; models must recover the ordered sequence encountered from start to finish. The benchmark balances self-intersection count, tortuosity, vertex count, and nearby confounding lines while limiting reliance on OCR, world knowledge, or open-ended planning. We find that self-intersections are the dominant source of difficulty. A first-crossing analysis localizes failures to crossing points: performance is stable before the first crossing, then drops sharply when the model must resolve the correct continuation. Nearby confounders have weaker but compounding effects, and an auxiliary reading-order benchmark reveals a consistent left-to-right bias. Together, these results characterize how VLMs perceive and fail on visual paths. Finally, we position TraversalBench as a new contribution to the growing line of sustained and precise visual grounding benchmarks for VLMs. Code, benchmark data, and rendered examples are available at https://github.com/clarapetrova/traversalbench.
Figures & tables
Figure 1: Examples from TraversalBench . Left: A non-tortuous, non-self-intersecting path that the model traces correctly. Right: A highly tortuous, heavily self-intersecting path for which the model produces an incorrect sequence. Glyphs are rendered slightly larger than in examples in the dataset.
Figure 2: Exact-match accuracy across the joint grid of tortuosity and self-intersection bins for all evaluated models (non-confound). Performance is highest in the low-tortuosity, low-crossing regime and declines sharply as self-intersections increase, with tortuosity adding a further gradient of difficulty.
Figure 3: Benchmark curves across the main controlled factors for the non-reasoning model set. Exact-match accuracy declines as paths become more complex, with self-intersections producing the strongest degradation. Dashed curves indicate matched confound conditions, showing an additional penalty from nearby non-target structure.
Figure 4: Coverage of the new-model benchmark across tortuosity and self-intersection bins, pooled over point counts. Example backbone curves placed to the left and below the heatmap illustrate the progression from lower to higher tortuosity and self-intersection without the visual clutter of glyph rendering. Covered cells are in yellow, uncovered cells are in black.
Figure 5: Local token accuracy around the first through fourth crossings. Accuracy is relatively stable immediately before the crossing, then drops sharply at the crossing itself, consistent with a localized continuation-selection bottleneck. Dashed curves show position-matched no-cross controls.
Figure 6: Effects of nearby confounding lines on traversal performance. Left: local token accuracy around the first confound, with solid curves showing matched non-confound controls. Right: token-level accuracy as a function of the number of previously encountered confounds. Confounds produce a primarily cumulative degradation across generated tokens.
Figure 7: Representative examples from the four reading-order regimes used in the auxiliary analysis. The panels illustrate the different regimes for 9 vertices and 2 lanes; glyphs are rendered slightly larger than actual dataset entries.
Figure 8: Regime effects on the auxiliary reading-order benchmark by model. Each cell shows the change in performance, in percentage points, relative to that model’s own average across regimes. Positive values indicate easier-than-average regimes for that model, and negative values indicate harder-than-average regimes. Across models, ltr_tb →↓ and ttb_lr ↓→ are generally easier, while rtl_tb ←↓ and especially ttb_rl ↓← are harder.
Model
TokAcc
Δ TokAcc (lanes)
Δ TokAcc (vertices)
GPT-5.4
98.7%
+0.3 pts
-0.9 pts
Claude Opus 4.6
63.9%
-21.0 pts
-7.5 pts
Gemini 3.1 Flash-Lite
90.1%
-7.8 pts
-4.9 pts
Qwen3-VL 32B
67.8%
-21.3 pts
-6.3 pts
Table 1: Auxiliary reading-order benchmark results for the representative models. TokAcc = token-level accuracy. The final two columns report the estimated change in token accuracy, in percentage points, from the minimum to maximum observed lane count and vertex count, respectively, based on model-specific regressions controlling for regime.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Representative benchmark instances illustrating the main controlled axes of path complexity. The panels show variation in tortuosity, self-intersection structure, vertex count, and the presence of confounding lines. Tortuosity increases as the path becomes more winding, self-intersection increases as the path crosses itself more often, lower versus higher vertex counts corresponds to fewer versus more vertices, and confounding lines add nearby distractor structure without changing the target path.
Model
Intersecting prefix exact
Matched no cross prefix exact
Delta vs matched no cross
Mean pre len
Qwen3-VL 32B
72.9%
82.7%
-9.8 pts
3.14
GPT-5.4
96.6%
99.5%
-2.9 pts
3.14
Gemini 3.1 Flash Lite
90.4%
94.0%
-3.6 pts
3.14
Claude Opus 4.6
74.9%
87.1%
-12.1 pts
3.14
Gemini 3.1 Pro (auto)
96.1%
98.8%
-2.8 pts
3.14
Gemini 3.1 Pro (low)
98.5%
98.8%
-0.3 pts
3.14
Appendix
Table 3: Matched-prefix control analysis at the first crossing. For each intersecting example, we compare exact correctness on the prefix up to the first crossing against a no-cross baseline matched by prefix length. Negative deltas indicate that prefixes on paths that eventually self-intersect are already harder than equally long prefixes on non-intersecting paths. N = 4992.
Figure 10: GPT-5.4 versus GPT-5.4 Pro on the exact overlap subset covered by the GPT-5.4 Pro run. The panels show GPT-5.4 exact match, GPT-5.4 Pro exact match, GPT-5.4 Pro answer rate, and GPT-5.4 Pro exact match conditioned on answering, across the hard-example cells. GPT-5.4 Pro’s degradation on the hardest cells is driven in large part by failures to emit a final answer; conditional on answering, accuracy remains high.
Figure 11: Token accuracy across the joint grid of tortuosity and self-intersection bins for each model on the base benchmark. Performance is highest in the low-tortuosity, low-crossing regime and declines sharply as self-intersections increase, with tortuosity adding a further gradient of difficulty.
Outcome / effect family
Before LR control
After LR control
Change
TokAcc, self-intersection ( s=1 – 6 )
-34.0 to -15.3
-33.4 to -14.4
+0.6 to +1.3
EM, self-intersection ( s=1 – 6 )
-49.4 to -28.9
-49.0 to -28.3
+0.38 to +0.83
TokAcc, tortuosity ( t=1 – 5 )
-13.8 to -2.2
-11.4 to -0.9
+1.2 to +2.7
TokAcc, point count ( mp=11 – 17 )
-6.0 to -1.7
-5.7 to -1.8
-0.06 to +0.39
Appendix
Table 4: Stability of the main benchmark difficulty effects after controlling for coarse LR/RL orientation. The table summarizes coefficient ranges from pooled regressions over the base benchmark. Adding an LR/RL indicator changes the main coefficients only modestly, especially for self-intersection, indicating that the central difficulty trends are not explained by directional bias.
Figure 12: Appendix self-test examples for interested readers. The panels span representative high-tortuosity, high-self-intersection cases at mp∈{15,17} . Readers may try to recover the target glyph sequence before consulting Table 5 . These examples are included for qualitative intuition only and do not constitute a formal human study.
Panel
t
s
mp
Answer
A
4
4
15
red square yellow square cyan tri green square cyan plus brown square yellow plus orange plus yellow star blue tri cyan circle purple square green plus cyan star red plus
B
4
4
17
red square blue circle purple square green square orange square brown star red plus green plus purple plus yellow star yellow square brown tri purple tri blue tri orange circle green tri cyan tri
C
4
5
15
red square yellow star yellow square orange circle brown square blue plus green square brown tri purple square blue square purple tri cyan plus cyan square green tri orange star
D
4
5
17
red square cyan circle orange plus purple square cyan star cyan tri orange star yellow tri yellow plus orange circle green plus orange square red circle purple tri purple star brown square brown tri
E
5
4
15
red square red star red plus orange square cyan plus green circle cyan square yellow tri brown square orange star brown plus purple square blue square cyan star blue star
F
5
4
17
red square brown star purple square yellow circle brown plus orange tri cyan square brown square orange square yellow plus purple plus yellow star yellow tri brown circle green tri red tri orange star
Appendix
Table 5: Answer key for the appendix self-test examples.
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
Vision-language models (VLMs) perform strongly on multimodal benchmarks, yet their ability to reliably follow continuous visual structure remains unclear. We focus on path following, a basic operation that requires following a selected path through successive visual continuations. Before considering complex factors such as crossings or overlaps, we first test whether current VLMs can robustly follow a simple continuous line under controlled settings that remove such confounds. Across frontier models, we find that tracing is already fragile, with this vulnerability manifesting as unstable path preservation in the presence of other nearby paths. Analysis of internal representations shows that information needed to determine the correct path continuation becomes weaker, with this degradation already emerging in the visual encoding stage. Common remedies do not resolve this weakness: model scaling provides limited improvement, additional reasoning provides uneven gains at substantial cost, and explicit tracing instructions fail to stabilize performance. The same path-preservation failures and sensitivity to nearby paths persist in visually richer and more realistic settings, suggesting that current VLMs lack a robust mechanism for following a selected path.
Benchmark accuracy is often implicitly assumed to reflect grounded visual understanding in vision-language models (VLMs), yet it remains unclear to what extent such scores truly reflect reliance on visual evidence. Motivated by a surprising observation that removing a substantial fraction of image tokens only degrades model performance very slightly on a widely used hallucination benchmark, we systematically investigate this mismatch in a set of open-source VLMs. Our analysis spans multiple levels of granularity, spanning global visual degradation, localized occlusion, question reformulation, answer-space expansion, and decision-level analyses beyond standard accuracy. We further complement these behavioral results with a layer-wise analysis of vision-token geometry. Throughout the experiments, we find that although VLMs do incorporate visual input, their predictions are less sensitive to the loss of fine-grained visual evidence that standard accuracy should have suggested. Even when the final prediction remains unchanged, the model's internal support for the correct answer may already be weakened. We further complement a representation-level analysis, which shows increasing similarity among visual tokens in deeper layers, providing a possible explanation for our findings. Together, these results suggest that current benchmarks are not sufficient to reliably evaluate fine-grained visual grounding in VLMs.
Zixuan Lan, Luzhe Sun, Matthew R. Walter +1
University of Chicago · Toyota Technological Institute at Chicago · Stony Brook University
We push the frontier of large-scale spatial intelligence in Vision-Language Models (VLMs) and introduce the first benchmark that probes geographical layout understanding from real-world videos, spanning up to 1km distances. Inspired by the cognitive science literature, we evaluate models against the hierarchical stages of human spatial awareness: anchoring via landmarks, connecting them through routes, and integrating these into global mental maps. Extensive experiments reveal a fundamental divergence in how current AI models process spatial information. Instead of utilising true path integration or forming geometric survey knowledge, we find that VLMs rely almost entirely on 2D visual recognition and text-matching to bypass complex spatial reasoning. The benchmark is publicly available at https://perception-test-challenge.github.io/kilometervision.html.
Aravindh Mahendran, Michael King, Matthew Koichi Grimes +12
Google DeepMind, Berlin, Germany · Google DeepMind, London, UK · Princeton University, Princeton, USA +3