Organizations: Nanyang Technological University · DAMO Academy, Alibaba Group US · Department of Artificial Intelligence, University of Macau · Institute of Artificial Intelligence and Brain Sciences, University of Macau
Multimodal large language models can recognize visual content yet struggle to reason reliably over it across multiple steps. In counting, a model may see all objects but lose track of which ones it has counted. We argue that such failures, commonly attributed to perception, reveal a reference gap: the failure to bind language to the relevant visual entities, regions, or locations. Motivated by the theory of visual routines, we test whether external references such as numbering can narrow this gap. We introduce ReGap-Bench, a paired benchmark of five visual reasoning tasks in which the scene, question, and answer are fixed and only the reference varies. Results show that task-matched references improve accuracy by up to 25%, while mismatched references provide little benefit or even hurt performance. These findings identify the reference gap as a bottleneck distinct from visual perception and show that constructing task-appropriate visual references can serve as a training-free visual skill for improving multimodal reasoning.
Figures & tables
Figure 1: The same question to GPT-5.4 mini about the same scene, without and with numbering reference. In the plain setting (left), the model refers to shapes by vague locations and double counts. With numbering (right), it lists the ellipses by label and answers correctly.
Figure 2: Example ReGap-Bench scenes. Each row shows a crop of one scene under the plain and each reference condition. The scene content, question, and answer are identical across a row. Numbering is not applied to angle ranking, size ranking, or transformation prediction.
Figure 3: Numbering improves performance across all models on shape counting and clock ranking.
Figure 4: Shading improves across all evaluated models on both angle ranking and size ranking.
Figure 5: Grid lines improve all evaluated models on size ranking and transformation prediction.
Figure 6: Operation-matched numbering produces consistently larger gains than shading or grid lines. Shading provides only small or inconsistent gains, while grid lines are often neutral or detrimental, suggesting that visual references help when they support the routines required by the task.
Figure 7: Performance improves as dial numbering becomes more complete.
Figure 8: Scaling model size help both plain and numbering, but does not eliminate the reference gap.
Overlay
Tasks
No skill
With skill
Δ
None
43
86.0
86.0
0.0
Numbering
41
63.7
70.1
+6.4
Grid lines
13
92.5
96.7
+4.2
Shading
3
33.3
66.7
+33.3
Table 2: Visual skill performance (accuracy, %) on CartoMapQA, broken down by overlay type.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
SHAPES IN IMAGE
1–10
11–20
21–30
31–40
41–50
All
Correct
38 / 38
92 / 92
44 / 45
18 / 22
1 / 3
193 / 200
Appendix
Table 3: Counting all shapes, regardless of class (Claude-Opus-4.7, plain images).
Task
Layout
Items
Answer
Image size (px)
Images
Shape counting
3–7 × 3–7
5–47 (19.0)
0–22 (3.2)
490–1130 × 580–1340
800
Angle ranking
1–3 × 3–9
3–23 (10.4)
Ranking
490–1450 × 170–490
600
Size ranking
1–3 × 5–7
4–13 (7.7)
Ranking
1280–1790 × 260–770
600
Clock ranking
1–3 × 3–6
3–18 (9.3)
Ranking
490–970 × 170–490
1,200
Transformation
1 + 4 panels
1 shape
Option (A–D)
660 × 1000
600
Appendix
Table 4: ReGap-Bench statistics. Each task has 200 scenes. Layout is the number of rows × columns of cells. Means are given in parentheses.
ANGLE RANKING
SIZE RANKING
MODEL
Plain
Shading
Plain
Shading
Grid lines
Claude-Opus-4.7
88.5
90.9
96.8
97.4
97.6
Gemini-3.1-Flash-Lite
76.4
85.2
95.3
96.2
95.8
GPT-5.4-mini
80.1
80.5
96.2
96.7
96.6
Qwen3-VL-30B-A3B
59.7
68.0
92.4
93.2
93.1
InternVL3.5-38B
69.2
76.1
91.1
93.6
91.4
Appendix
Table 5: Pairwise accuracy (%) on angle and size ranking, averaged over valid rankings.
Wrong-side rate (%)
Model
Scenes
Angles
Plain
Shading
Random
Claude-Opus-4.7
200
1,030
3.3
1.7
47.8
Gemini-3.1-Flash-Lite
186
925
18.4
8.5
47.8
GPT-5.4-mini
181
914
13.3
13.8
47.8
Qwen3-VL-30B-A3B
95
377
31.6
27.9
47.0
InternVL3.5-38B
161
716
23.9
13.1
47.7
Appendix
Table 6: Wrong-side readings in angle ranking: the share (%) of angles below 90∘ or above 270∘ that are placed closer to their wrong-side position than to their true position (Appendix C ). Scenes are those with valid rankings for both the plain and the shaded image. Random is the rate for random rankings of the same scenes.