cs.CVApr 13, 2026
SaveTraversalBench: Challenging Paths to Follow for Vision Language Models
Organizations: Massachusetts Institute of Technology, Department of Physics · NSF AI Institute for Artificial Intelligence and Fundamental Interactions · Massachusetts Institute of Technology, Institute for Data, Systems, and Society
Abstract
Vision-language models (VLMs) perform strongly on multimodal benchmarks, but their ability to follow complex visual paths remains under-tested. We introduce TraversalBench, a controlled benchmark for exact visual path traversal. Each instance contains a continuous polyline with a unique start marker and labeled vertices; models must recover the ordered sequence encountered from start to finish. The benchmark balances self-intersection count, tortuosity, vertex count, and nearby confounding lines while limiting reliance on OCR, world knowledge, or open-ended planning. We find that self-intersections are the dominant source of difficulty. A first-crossing analysis localizes failures to crossing points: performance is stable before the first crossing, then drops sharply when the model must resolve the correct continuation. Nearby confounders have weaker but compounding effects, and an auxiliary reading-order benchmark reveals a consistent left-to-right bias. Together, these results characterize how VLMs perceive and fail on visual paths. Finally, we position TraversalBench as a new contribution to the growing line of sustained and precise visual grounding benchmarks for VLMs. Code, benchmark data, and rendered examples are available at https://github.com/clarapetrova/traversalbench.
Figures & tables
Figure 1: Examples from TraversalBench . Left: A non-tortuous, non-self-intersecting path that the model traces correctly. Right: A highly tortuous, heavily self-intersecting path for which the model produces an incorrect sequence. Glyphs are rendered slightly larger than in examples in the dataset.
Figure 2: Exact-match accuracy across the joint grid of tortuosity and self-intersection bins for all evaluated models (non-confound). Performance is highest in the low-tortuosity, low-crossing regime and declines sharply as self-intersections increase, with tortuosity adding a further gradient of difficulty.
Figure 3: Benchmark curves across the main controlled factors for the non-reasoning model set. Exact-match accuracy declines as paths become more complex, with self-intersections producing the strongest degradation. Dashed curves indicate matched confound conditions, showing an additional penalty from nearby non-target structure.
Figure 4: Coverage of the new-model benchmark across tortuosity and self-intersection bins, pooled over point counts. Example backbone curves placed to the left and below the heatmap illustrate the progression from lower to higher tortuosity and self-intersection without the visual clutter of glyph rendering. Covered cells are in yellow, uncovered cells are in black.
Figure 5: Local token accuracy around the first through fourth crossings. Accuracy is relatively stable immediately before the crossing, then drops sharply at the crossing itself, consistent with a localized continuation-selection bottleneck. Dashed curves show position-matched no-cross controls.
Figure 6: Effects of nearby confounding lines on traversal performance. Left: local token accuracy around the first confound, with solid curves showing matched non-confound controls. Right: token-level accuracy as a function of the number of previously encountered confounds. Confounds produce a primarily cumulative degradation across generated tokens.
Figure 7: Representative examples from the four reading-order regimes used in the auxiliary analysis. The panels illustrate the different regimes for 9 vertices and 2 lanes; glyphs are rendered slightly larger than actual dataset entries.
Figure 8: Regime effects on the auxiliary reading-order benchmark by model. Each cell shows the change in performance, in percentage points, relative to that model’s own average across regimes. Positive values indicate easier-than-average regimes for that model, and negative values indicate harder-than-average regimes. Across models, ltr_tb and ttb_lr are generally easier, while rtl_tb and especially ttb_rl are harder.
| Model | TokAcc | TokAcc (lanes) | TokAcc (vertices) |
| GPT-5.4 | 98.7% | +0.3 pts | -0.9 pts |
| Claude Opus 4.6 | 63.9% | -21.0 pts | -7.5 pts |
| Gemini 3.1 Flash-Lite | 90.1% | -7.8 pts | -4.9 pts |
| Qwen3-VL 32B | 67.8% | -21.3 pts | -6.3 pts |
Table 1: Auxiliary reading-order benchmark results for the representative models. TokAcc = token-level accuracy. The final two columns report the estimated change in token accuracy, in percentage points, from the minimum to maximum observed lane count and vertex count, respectively, based on model-specific regressions controlling for regime.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Representative benchmark instances illustrating the main controlled axes of path complexity. The panels show variation in tortuosity, self-intersection structure, vertex count, and the presence of confounding lines. Tortuosity increases as the path becomes more winding, self-intersection increases as the path crosses itself more often, lower versus higher vertex counts corresponds to fewer versus more vertices, and confounding lines add nearby distractor structure without changing the target path.
| Model | Intersecting prefix exact | Matched no cross prefix exact | Delta vs matched no cross | Mean pre len |
| Qwen3-VL 32B | 72.9% | 82.7% | -9.8 pts | 3.14 |
| GPT-5.4 | 96.6% | 99.5% | -2.9 pts | 3.14 |
| Gemini 3.1 Flash Lite | 90.4% | 94.0% | -3.6 pts | 3.14 |
| Claude Opus 4.6 | 74.9% | 87.1% | -12.1 pts | 3.14 |
| Gemini 3.1 Pro (auto) | 96.1% | 98.8% | -2.8 pts | 3.14 |
| Gemini 3.1 Pro (low) | 98.5% | 98.8% | -0.3 pts | 3.14 |
Table 3: Matched-prefix control analysis at the first crossing. For each intersecting example, we compare exact correctness on the prefix up to the first crossing against a no-cross baseline matched by prefix length. Negative deltas indicate that prefixes on paths that eventually self-intersect are already harder than equally long prefixes on non-intersecting paths. N = 4992.
Figure 10: GPT-5.4 versus GPT-5.4 Pro on the exact overlap subset covered by the GPT-5.4 Pro run. The panels show GPT-5.4 exact match, GPT-5.4 Pro exact match, GPT-5.4 Pro answer rate, and GPT-5.4 Pro exact match conditioned on answering, across the hard-example cells. GPT-5.4 Pro’s degradation on the hardest cells is driven in large part by failures to emit a final answer; conditional on answering, accuracy remains high.
Figure 11: Token accuracy across the joint grid of tortuosity and self-intersection bins for each model on the base benchmark. Performance is highest in the low-tortuosity, low-crossing regime and declines sharply as self-intersections increase, with tortuosity adding a further gradient of difficulty.
| Outcome / effect family | Before LR control | After LR control | Change |
| TokAcc, self-intersection ( – ) | -34.0 to -15.3 | -33.4 to -14.4 | +0.6 to +1.3 |
| EM, self-intersection ( – ) | -49.4 to -28.9 | -49.0 to -28.3 | +0.38 to +0.83 |
| TokAcc, tortuosity ( – ) | -13.8 to -2.2 | -11.4 to -0.9 | +1.2 to +2.7 |
| TokAcc, point count ( – ) | -6.0 to -1.7 | -5.7 to -1.8 | -0.06 to +0.39 |
Table 4: Stability of the main benchmark difficulty effects after controlling for coarse LR/RL orientation. The table summarizes coefficient ranges from pooled regressions over the base benchmark. Adding an LR/RL indicator changes the main coefficients only modestly, especially for self-intersection, indicating that the central difficulty trends are not explained by directional bias.
Figure 12: Appendix self-test examples for interested readers. The panels span representative high-tortuosity, high-self-intersection cases at . Readers may try to recover the target glyph sequence before consulting Table 5 . These examples are included for qualitative intuition only and do not constitute a formal human study.
| Panel | Answer | |||
| A | 4 | 4 | 15 | red square yellow square cyan tri green square cyan plus brown square yellow plus orange plus yellow star blue tri cyan circle purple square green plus cyan star red plus |
| B | 4 | 4 | 17 | red square blue circle purple square green square orange square brown star red plus green plus purple plus yellow star yellow square brown tri purple tri blue tri orange circle green tri cyan tri |
| C | 4 | 5 | 15 | red square yellow star yellow square orange circle brown square blue plus green square brown tri purple square blue square purple tri cyan plus cyan square green tri orange star |
| D | 4 | 5 | 17 | red square cyan circle orange plus purple square cyan star cyan tri orange star yellow tri yellow plus orange circle green plus orange square red circle purple tri purple star brown square brown tri |
| E | 5 | 4 | 15 | red square red star red plus orange square cyan plus green circle cyan square yellow tri brown square orange star brown plus purple square blue square cyan star blue star |
| F | 5 | 4 | 17 | red square brown star purple square yellow circle brown plus orange tri cyan square brown square orange square yellow plus purple plus yellow star yellow tri brown circle green tri red tri orange star |
Table 5: Answer key for the appendix self-test examples.
More in this topic
VLM Evaluation
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
See all 989 papers