Multimodal large language models (MLLMs) have made rapid progress on visual reasoning, yet many tasks require evidence that is difficult to obtain through internal reasoning alone: our controlled study on 22 benchmarks shows that simply increasing the inference budget of the base model yields only modest gains. Such tasks demand grounded perception, external knowledge, and numerical measurements obtained through heterogeneous tools, whose outputs must be made interpretable to the model and retained across reasoning steps. We introduce OpenVLHarness, a multimodal reasoning harness organized around a capability layer and persistent multimodal memory. The capability layer combines task-level tool interfaces with evidence rendering, providing consistent access to heterogeneous backends and converting their outputs into model-readable observations. Persistent multimodal memory retains the corresponding images, structured data, and web references as addressable artifacts throughout each query. A coding interface executes generated programs over stored data, enabling computations that combine results from multiple tools. Together, these mechanisms support an iterative process of acquiring, inspecting, and composing multimodal evidence. Evaluated on 22 benchmarks with eight open and proprietary backbones, OpenVLHarness improves average scores by 9.7-12.6 points over matched no-tool baselines, including +11.2 with Qwen3-VL-8B, +9.7 with Kimi K3, and +9.9 with GPT-6 Astra. With GPT-6 Luna at medium reasoning effort, OpenVLHarness outperforms the base model, Codex, and Visual Sketchpad at maximum effort by 7.7-8.5 points while incurring at most 37% of their model-token cost.
Figures & tables
Figure 1: Performance of OpenVLHarness across models and settings. (a) Average score over 22 challenging multimodal benchmarks with and without OpenVLHarness, across the Qwen, Kimi, and GPT model families. (b) Benchmark scores for GPT-6 Luna and GPT-6 Astra at medium reasoning effort, each with and without OpenVLHarness; radii are rescaled per benchmark, and numbers give the raw scores of GPT-6 Astra (medium) with OpenVLHarness. (c) Average score versus model-token cost (average for 1,000 query rollouts) on 22 benchmarks for GPT-6 Luna alone and with Codex, Visual Sketchpad, or OpenVLHarness, across all five reasoning-effort settings.
Figure 2: Overview of OpenVLHarness. The MLLM orchestrator invokes capabilities through typed interfaces. The capability layer connects these interfaces to underlying models and services and renders their outputs into model-facing evidence: here, a labeled mask overlay accompanied by a textual description and normalized bounding boxes. Persistent multimodal memory retains image artifacts, structured data, and web references under reusable handles. An Environment Update advertises the available artifacts to the orchestrator, allowing subsequent calls to reuse earlier evidence, including through memory-grounded computation.
Figure 3: Qualitative examples of OpenVLHarness. (a) Camera poses, depth, and grounding are combined in code to measure a 3D distance (Example F ). (b) Grounded crops are passed to OCR and image search, and the answer is verified on the web (Example F ). (c) Masks from two grounding queries are reconciled in code to reach the exact count (Example F ).
Qwen3-VL-8B
Qwen3-VL-32B
GPT-6 Luna (medium)
GPT-6 Astra (medium)
Domain
Dataset
Base
OT
VS
Ours
Base
OT
VS
Ours
Base
VS
CX
Ours
Base
Ours
CountQA
32.00
29.06
19.83
48.56
44.70
41.75
32.85
64.14
55.17
52.68
58.25
66.95
82.00
86.40
FSC-147
15.32
11.26
9.41
37.71
18.97
12.44
12.35
39.42
28.23
30.25
28.62
41.52
44.01
46.11
OdinW-13-All
39.80
24.60
31.77
50.40
42.39
44.33
36.10
59.13
36.74
43.84
36.14
53.56
50.46
60.45
FSCD-147
1.79
0.93
1.41
23.52
2.88
2.55
2.30
28.45
4.24
6.11
2.86
25.74
15.49
30.33
Grounded Perception
FSCD-LVIS
1.47
1.36
2.49
27.37
1.79
3.91
4.57
31.63
2.44
8.65
2.23
30.55
10.55
36.02
Table 1: Main results across four task domains and 22 datasets. Results are grouped by MLLM backbone. Base denotes the backbone without tool use; for harness, OT , VS , and CX denote OctoTools, Visual Sketchpad, and Codex, respectively; Ours denotes OpenVLHarness. GPT-6 Sol, GPT-5, and Kimi K3 results are in Table 7 . Bold indicates the best result within each backbone group, and underlining indicates the best result across all shown backbones. Qwen3-VL results are means of three runs (standard deviations in Table 26 ); GPT-6 results are single runs. OdinW-13-All asks for all categories of an image’s source dataset in one query.
Reasoning effort
medium
high
xhigh
max
Base
CX
VS
Ours
Base
CX
VS
Ours
Base
CX
VS
Ours
Base
CX
VS
Ours
Grounding
25.4
25.6
28.3
43.7
26.5
26.3
29.3
44.6
27.0
27.1
29.3
45.2
27.2
30.0
29.6
45.6
VS & DR
35.2
33.5
32.4
47.5
35.3
35.4
34.6
50.3
37.4
36.7
35.6
50.6
37.8
40.3
36.9
52.7
General VQA
63.4
60.3
64.3
69.8
64.4
61.8
65.1
71.5
64.7
63.0
66.6
71.5
64.5
63.5
66.5
72.6
Spatial
57.7
54.4
56.2
64.2
59.3
57.0
57.1
65.3
58.7
58.4
58.5
65.9
61.5
59.1
61.4
68.5
Average
45.8
43.8
45.5
56.5
46.7
45.5
46.9
58.2
47.3
46.6
47.8
58.6
48.0
48.6
48.8
60.1
Table 2: Reasoning effort versus harness, by domain. GPT-6 Luna alone (Base) and with Codex (CX), Visual Sketchpad (VS), or OpenVLHarness (Ours) on the full evaluation splits of our 22 benchmarks. Rows average each domain’s benchmarks: Grounding (grounded perception, 5 ), VS & DR (visual search and deep research, 6 ), General VQA (general VQA and hallucination, 6 ), and Spatial (spatial reasoning, 5 ). /1kquestionsisthemodel−tokencostatStandard−tierprices,includingreasoningtokensandexcludingtoolexecutionandjudging,averagedoverthe22$ benchmarks. At each effort, bold marks the best score, and in the cost row the cheapest method, with the second cheapest underlined. Low effort is in Appendix Table 13 .
Variant
Grounding
VS & DR
General VQA
Spatial
Overall
8B
32B
8B
32B
8B
32B
8B
32B
8B
32B
Baseline (no tool)
27.48
31.72
18.40
23.34
57.97
62.16
44.38
47.93
37.06
41.29
+ Tools
41.70
43.30
21.44
24.89
52.41
61.41
47.83
52.84
40.84
45.61
+ Capability layer
46.73
46.20
25.97
32.74
56.05
63.63
48.16
52.63
44.23
48.80
+ Textual memory
47.27
45.74
26.92
34.43
57.54
64.84
47.98
51.46
44.93
49.12
OpenVLHarness
45.46
51.74
33.10
37.96
59.81
66.12
50.82
54.59
47.14
52.51
Table 3: Ablation study of OpenVLHarness across model scales. We progressively add raw tool access, capability layer, textual memory, and multimodal memory to the no-tool baseline. Results are reported across four domains and overall for Qwen3-VL 8B and 32B models, averaged over three runs; standard deviations are provided in Table 27 . Bold marks the best result in each column.
Figure 4: Failure diagnosis and harness specialization with Qwen3-VL-8B. We critique failure trajectories using Kimi K3. (a) Distribution of failure causes across four domains. (b) Proposed remedies: grouped into creating task-specific tools (A–C), updating harness text (Y), and no proposed remedy (Z). (c) Four datasets selected for contrasting remedy profiles: Y (orange) dominates ERQA and RealWorldQA, while A–C (blue) dominates SparBench and FREAK-VQA. (d) Gains from harness search and tool generation over the unoptimized harness. Harness search is more effective on datasets dominated by text-level changes, while tool generation yields larger gains when task-specific tools are preferred.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Tool-call volume and engagement by domain. (a) Tool calls per question, averaged over all questions in the domain. (b) Fraction of questions answered with no tool call. Engagement varies with model scale within a family and not across families, and call volume does not order the models by accuracy.
Model
Calls /
Calls / engaged
No tool
Messages /
Repeat
Failed
question
question
(%)
question
calls (%)
calls (%)
Qwen3-VL-8B
1.43
2.00
28.9
9.7
1.92
0.59
Qwen3-VL-32B
2.59
2.76
6.2
14.3
0.54
0.59
GPT-6 Luna
1.25
1.67
25.3
7.7
0.38
0.30
GPT-6 Sol
2.07
2.18
4.8
10.2
1.95
0.59
GPT-6 Astra
1.80
1.84
2.6
9.4
0.01
0.17
Appendix
Table 4: Tool-use profile of OpenVLHarness across five base models. Calls per question averages over all questions, including those answered without tools; calls per engaged question conditions on questions with at least one call. Messages count transcript turns, including the Environment Update emitted after each call. Repeat calls are byte-identical to an earlier call in the same rollout; failed calls are those the tool server returned as errors.
Figure 6: Distribution of tool calls across capabilities. All calls issued by each model, normalized to 100%. Visual Grounding is the largest consumer for every model; the models differ mainly in the second-ranked capability.
Figure 7: Distribution of tool calls by domain. Calls issued in each domain as a share of that domain’s calls, with the call total below each panel. Colors are consistent across panels. Grounded perception is dominated by Visual Grounding for every model except GPT-6 Sol and GPT-6 Astra; visual search and deep research is the only domain dominated by the web capabilities; spatial reasoning is the only domain in which four capabilities carry comparable load.
Model
Rows
Base
OpenVLHarness
Δ
Rescued
Broken
Calls per
net gain
Qwen3-VL-8B
24,410
50.02
60.09
+ 10.07
4,649
1,967
12.9
Qwen3-VL-32B
24,393
53.73
62.17
+ 8.44
4,381
2,146
28.1
GPT-6 Luna
26,226
57.88
67.67
+ 9.79
4,191
1,382
11.7
GPT-6 Sol
23,700
64.26
72.67
+ 8.41
3,153
1,025
23.1
GPT-6 Astra
23,700
70.74
76.98
+ 6.24
2,336
700
26.0
Appendix
Table 5: OpenVLHarness against the base model, paired per question. Rescued counts questions answered incorrectly by the base model and correctly by OpenVLHarness; broken counts the converse. Calls per net gain divides the total number of tool calls issued on these questions by the net number of questions whose outcome changed.
Figure 8: Per-capability paired comparison against the base model. For each capability and model, the accuracy of OpenVLHarness minus that of the base model on the questions whose rollout invoked that capability. The corresponding difference on the questions that did not invoke it, which serves as the control, is given in the last column of Table 6 together with the share of questions each capability was invoked on.
Capability
Model
Questions
Share
Calls
Base
OpenVLHarness
Δ
Δ elsewhere
Visual Grounding
Qwen-8B
13,103
54%
21,286
47.7
62.0
+ 14.3
− 20.1
Qwen-32B
17,284
71%
40,057
56.1
64.5
+ 8.3
+ 0.2
Luna
13,216
50%
17,670
56.5
67.4
+ 10.9
+ 2.7
Sol
14,369
61%
22,313
67.5
74.3
+ 6.7
+ 2.2
Astra
15,488
65%
22,385
73.6
77.9
+ 4.3
+ 5.1
Zoom-in
Qwen-8B
1,703
7%
2,714
40.3
38.8
− 1.6
+ 4.8
Appendix
Table 6: Per-capability paired comparison, corresponding to Figure 8 . For each capability and model: the questions whose rollout invoked it at least once, the calls issued on them, the accuracy of the base model and of OpenVLHarness on exactly those questions, their difference, and the same difference computed on the questions that did not invoke the capability. Share is the fraction of paired questions that invoked it, which determines how closely the final column approximates the model’s overall gain. A question that invoked three capabilities is counted once under each.
GPT-5
GPT-6 Sol (medium)
Kimi K3 (max)
Domain
Dataset
Base
Ours
Base
Ours
Base
Ours
CountQA
38.48
58.70
73.00
76.00
71.27
73.10
FSC-147
22.16
42.92
38.72
44.40
36.78
42.07
OdinW-13-All
46.27
63.12
40.86
57.88
44.35
56.91
FSCD-147
0.16
28.09
8.36
29.42
7.88
29.11
Grounded Perception
FSCD-LVIS
2.04
33.22
5.17
35.22
7.15
33.60
Appendix
Table 7: Results of OpenVLHarness with GPT-5 ( OpenAI, 2025b ) , GPT-6 Sol ( OpenAI, 2026 ) , and Kimi K3 ( Team et al., 2026 ) as the backbone, on the same 22 datasets and four task domains as Table 1 . Base: the backbone prompted directly; Ours: the backbone with OpenVLHarness. Bold marks the better of Base and Ours for each backbone.
Variant
Tool schema
Post-proc.
Env update
Bindings
URL refs
Image refs
Baseline
—
—
—
—
—
—
+ Tool
minimal
raw
off
off
off
off
+ Capability layer
full
custom
off
off
off
off
+ Textual memory
full
custom
on
on
on
off
OpenVLHarness
full
custom
on
on
on
on
Appendix
Table 8: Ablation knob settings for each variant in Table 3 . “Post-proc.” refers to per-tool output processors (custom vs. raw dumps). “Env update” is the per-turn Environment Update message. “Bindings” are out-of-context typed data bindings. “URL refs” and “Image refs” are symbolic identifiers for web pages and tool-generated images, respectively.
Training-based
Training-free
Domain
Dataset
AdaReasoner
PixelReasoner
DeepEyesV2
OpenVLHarness
CountQA
36.98
21.66
21.73
51.37
FSC-147
1.34
9.75
21.26
31.60
OdinW-13-All
0.00
10.24
8.97
7.80
FSCD-147
0.00
0.04
0.06
10.30
FSCD-LVIS
0.00
0.00
0.05
14.20
Appendix
Table 9: Training-based baselines on the Qwen2.5-VL-7B-Instruct backbone. AdaReasoner, PixelReasoner, and DeepEyesV2 fine-tune the backbone to adapt it to their tool-use harness; OpenVLHarness is training-free and uses the same weights out of the box. Training-based columns are gray; shaded rows are per-domain averages. Bold marks the best result in each row.
Figure 9: Object-count distribution across counting benchmarks. Each bar shows the percentage of samples falling into binned object-count buckets, with the per-dataset maximum count annotated above. Widely used benchmarks such as TallyQA, CountBenchQA, and PixmoCount are dominated by low counts: nearly all of their samples contain ten or fewer objects (and a majority of TallyQA fall in the 0 – 1 bucket), capping at 10 – 15 . Only FSC-147 and FSCD-LVIS contain substantial mass in the high-count regime ( >50 objects), reaching into the hundreds or thousands. This skew means accuracy on standard counting benchmarks largely reflects performance on small object counts and reveals little about high-count visual measurement.
Figure 10: Object-count distribution across grounding benchmarks. Each bar shows the percentage of samples falling into binned object-count buckets, with the per-dataset maximum count annotated above. Referring-expression benchmarks such as RefCOCO/+/g and Ref-L4/GroundingME ground a single target per sample ( 100% in the 0 – 1 bucket, max=1 ), and even detection-style sets like OdinW-13 ( 97% ) and LVIS ( 86% ) are overwhelmingly concentrated at one object per image. Only the FSCD variants exercise dense, many-object grounding. As with counting, this distribution implies that progress on common grounding benchmarks is measured almost entirely in the single- or few-object regime.
Qwen3-VL-8B
Qwen3-VL-32B
Objects
# Samples
Base
OpenVLHarness
Base
OpenVLHarness
FSC-147: exact-match accuracy (%)
≤10
149
42.3
73.8
64.4
81.9
11–20
368
28.8
59.2
33.4
63.3
21–50
383
5.0
32.9
5.2
34.5
51–100
185
1.1
9.7
2.2
9.7
Appendix
Table 10: Grounded perception performance by number of ground-truth objects. FSCD-LVIS is scored with instance-level F1 (predicted boxes matched to ground truth at IoU ≥0.5 ) because AP is not defined per sample.
FSC-147
CountQA
LVIS
Model
Method
MAE ↓
RA@5 ↑
RA@10 ↑
AP ↑
Qwen3-VL-8B
Base
29.70
33.64
43.13
18.12
OpenVLHarness
23.17
50.00
57.59
45.19
Qwen3-VL-32B
Base
60.70
46.79
58.90
22.87
OpenVLHarness
31.24
65.51
72.71
45.46
Appendix
Table 11: Complementary counting metrics and LVIS grounding. MAE is the mean absolute counting error; RA@ k is the share of predictions within k% of the ground truth.
Model
MMS
MSP
RXB
FVQA
HLE-VL
BC-VL
MM-BC
VDR
VBB
InfoSeek
Enc-VQA
Avg
Direct Inference
Qwen3-VL-8B
14.04
1.29
22.16
28.39
6.36
21.05
4.94
5.40
7.69
29.20
10.61
13.74
Qwen3-VL-8B-T
12.87
3.22
22.16
25.67
10.30
33.83
7.41
8.20
11.24
27.25
18.06
16.38
Qwen3-VL-32B
20.47
2.57
24.74
32.67
10.00
28.82
6.58
9.40
10.06
33.20
32.76
19.21
GPT-5
36.84
10.93
34.54
54.67
20.00
42.11
8.64
14.20
28.40
53.67
43.33
31.58
Agent Workflow
Appendix
Table 12: Extended visual search and deep research results on 11 information-seeking benchmarks, five of which (MM-BC, VDR, VBB, InfoSeek, Enc-VQA) are not in Table 1 . Qwen3-VL-8B refers to Qwen3-VL-8B-Instruct and Qwen3-VL-8B-T denotes Qwen3-VL-8B-Thinking. Abbreviations: MM-BC = MM-BrowseComp; MMS = MMSearch; MSP = MMSearch-Plus; BC-VL = BrowseComp-VL; RXB = RealX-Bench; VDR = VDR-Bench; VBB = VisBrowse-Bench; Enc-VQA = EncyclopedicVQA. Avg is the unweighted mean across the 11 benchmarks; per-benchmark sample counts are in Appendix D.2 .
Domain average
Method
Effort
Grounding
VS & DR
General VQA
Spatial
Average
$ / 1k
(5)
(6)
(6)
(5)
(22)
Base
low
24.28
33.49
61.63
56.54
44.31
0.36
medium
25.36
35.18
63.38
57.69
45.76
0.58
high
26.54
35.27
64.37
59.27
46.68
0.85
xhigh
27.01
37.36
64.72
58.73
47.33
1.18
Appendix
Table 13: GPT-6 Luna across five reasoning-effort settings for each method, evaluated on the full evaluation splits of all 22 benchmarks of Table 1 ( 26,229 questions per setting). Domain averages are unweighted means over the benchmarks of each domain (number of benchmarks in parentheses); per-benchmark scores are in Table 16 . /1kisthemodel−tokencostper1{,}000questionsatStandard−tierprices,includingreasoningtokensandexcludingtoolexecutionandjudging,averagedoverthe22$ benchmarks. Bold marks the best score in each score column. Each cell is a single run.
Input tokens
Output tokens
Method
Effort
Score
$/1k
Uncached
Cache read
Cache write
Total
Reasoning
Total
Base
low
44.31
0.36
1,815
30
0
1,845
301
348
medium
45.76
0.58
1,793
52
0
1,845
750
804
high
46.68
0.85
1,795
50
0
1,845
1,283
1,345
xhigh
47.33
1.18
1,796
49
0
1,845
1,938
2,006
max
48.04
3.17
216
69
1,557
1,842
5,852
5,902
Appendix
Table 14: Model tokens per question behind Table 13 , summed over every model call made for a question and averaged over the 22 benchmarks. Output totals include the reasoning tokens. For OpenVLHarness, the calls of the Python Coding Agent are logged per setting rather than per question and are apportioned evenly across questions; they account for 0.4% of its tokens and 2.5% of its model-token cost at low effort, rising to 3.6% and 14.7% at max effort.
Executes code
Executes code, by domain (%)
Effort
Questions
n
%
Grounding
VS & DR
General VQA
Spatial
Commands
Per exec.
Score
low
26,229
43
0.16
0.00
1.34
0.00
0.00
126
2.9
36.92
medium
26,229
106
0.40
0.30
2.31
0.02
0.00
679
6.4
43.78
high
26,229
221
0.84
1.11
3.21
0.09
0.00
2,051
9.3
45.46
xhigh
26,229
457
1.74
2.88
4.77
0.16
0.01
4,953
10.8
46.61
max
26,229
1,619
6.17
12.33
9.61
0.89
0.00
18,753
11.6
48.58
Appendix
Table 15: How often Codex executes code with GPT-6 Luna in the effort study. A question counts as executing code if its Codex session contains at least one command execution. Commands is the total number of executed commands, and Per exec. the mean number of commands per question that executes code. Score is the 22 -benchmark average from Table 13 .
Base
Codex
Dataset
low
med
high
xhigh
max
low
med
high
xhigh
max
CountQA
52.03
55.17
58.05
59.03
62.04
44.90
58.25
59.42
59.69
62.24
FSC-147
28.62
28.23
29.32
29.78
29.94
22.63
28.62
29.55
31.88
38.34
OdinW-13-All
35.67
36.74
38.18
38.80
36.55
30.66
36.14
36.78
37.50
38.03
FSCD-147
3.06
4.24
4.57
4.79
4.82
1.88
2.86
3.41
4.04
7.27
FSCD-LVIS
2.05
2.44
2.56
2.66
2.84
1.40
2.23
2.31
2.43
4.05
Appendix
Table 16: Per-benchmark scores behind Table 13 : GPT-6 Luna at five reasoning-effort settings for each method, on the full evaluation split of every benchmark. Scoring follows Table 1 . Bold marks the best score in each benchmark row across all 20 settings. † Requests for some questions failed on every retry ( 134 questions in 55 cells; at most 13 in a single cell). These count as incorrect. /1kquestionsisthemodel−tokencostatStandard−tierprices,averagedoverthe22$ benchmarks. Each cell is a single run.
Qwen3-VL-8B
Qwen3-VL-32B
Domain
Dataset
Base
Codex
OpenVLHarness
Base
Codex
OpenVLHarness
Grounding
CountQA
32.00
29.97
48.56
44.70
39.73
64.14
FSC-147
15.32
13.53
37.71
18.97
16.41
39.42
OdinW-13-All
39.80
33.70
50.40
42.39
34.66
59.13
FSCD-147
1.79
0.50
23.52
2.88
0.50
28.45
FSCD-LVIS
1.47
0.78
27.37
1.79
2.24
31.63
Appendix
Table 17: Codex with Qwen3-VL backbones on the 22 benchmarks of Table 1 . Base and OpenVLHarness are copied from that table. Codex runs the Codex CLI agent in code mode at medium reasoning effort with web search disabled. Bold marks the best result within each model group. Averages are taken over all 22 rows. Each Codex cell is a single run.
Capability
Interface
Backends
Search
Text Search
text query → summarized results with cited URLs
Serper ; Gemini with Google Search
Image Search
image → visual matches (entities, URLs)
Serper (Google Lens) ; Gemini with Google Search
Webpage Visit
URLs and goal → goal-relevant evidence
Serper with an LLM extractor
Perception
Zoom-in
image and box → cropped image
built-in
Appendix
Table 18: The nine capabilities, their interfaces, and their interchangeable backends. Bold marks the backend used in all experiments.
Artifact
Source tool
Stored representation in memory
Reusable handle
Image artifacts (returned to the model as a native image attachment + an ID)
Input image
(question)
Registered verbatim
input_image_ n
Cropped region
Zoom-in
Cropped image, deduplicated by content
tool_generated_image_ n
Grounded / mask overlay
Visual Grounding
Annotated image with numbered regions
tool_generated_image_ n
Colorized depth map
Depth Estimation
Rendered depth visualization
tool_generated_image_ n
Camera-trajectory plot
Camera Trajectory
Bird’s-eye-view render of the estimated camera path
tool_generated_image_ n
Appendix
Table 19: Tool-generated artifacts retained in persistent multimodal memory, grouped by how they are represented and re-addressed. Heavy payloads (image bytes, numeric arrays) are stored once and replaced in the transcript by a compact handle; the orchestrator reuses an artifact on a later turn by citing its handle, which the harness resolves back to the underlying data.
Model
Role
Access
Agent backbones
Qwen2.5-VL-7B-Instruct
Backbone (vs. training-based baselines)
vLLM
Qwen3-VL-8B-Instruct
Backbone
vLLM
Qwen3-VL-32B-Instruct
Backbone
vLLM
Qwen3-VL-8B-Thinking
Direct-inference reference (search)
vLLM
GPT-5
Backbone
OpenAI API
Appendix
Table 20: Models used in our experiments.
Domain
Benchmark
Split
# Samples
Format
Scoring
Grounded Perception
CountQA
test
1,528
integer
exact match
FSC-147
val
1,286
integer
exact match
OdinW-13-All
val
4,938
labeled boxes
box AP
FSCD-147
val
1,286
boxes
box AP
FSCD-LVIS
val
1,181
boxes
box AP
Visual Search and Deep Research
MMSearch
end-to-end
171
open-ended
GPT-4o judge
Appendix
Table 21: The 22 benchmarks of Table 1 . “LLM judge” is Qwen3-VL-8B-Instruct; MRA is mean relative accuracy for numeric answers.
Label
Meaning
Root cause (exactly one)
Tool not called
The needed evidence was never gathered, although a tool in the agent’s toolbox could have supplied it.
Tool output wrong
A tool was called but returned wrong or missing information (missed or hallucinated boxes, a bad crop, a wrong transcription).
Tool output unusable
A tool returned correct information in a form the model could not use (raw arrays, uninterpreted coordinates, an overload of detections).
Tool output ignored
A tool returned correct, plainly usable information that the model did not incorporate or contradicted.
Reasoning error
The evidence was adequate and used; the error is in reasoning, arithmetic, comparison, or answer format.
Appendix
Table 22: Critique taxonomy. Each failure receives one primary cause and one proposed fix type. Unusable and ignored outputs remain separate during critique and are combined for reporting in Fig. 4 and Table 23 .
Root cause (%)
Remedy channel (%)
Domain
Dataset
#Fail.
NC
Wrong
Unus./Ign.
Reas.
Label
A
B
C
Y
Z
Tool
Text
NC → build
Grounding
CountQA
751
32
53
4
10
1
12
54
2
31
1
68
31
8
FSC-147
803
9
89
1
2
0
2
89
2
7
0
93
7
27
OdinW-13-All
1,016
4
56
2
15
23
12
56
0
9
23
68
9
7
FSCD-147
90
0
79
6
0
16
3
81
0
0
16
84
0
0
FSCD-LVIS
370
0
98
1
0
0
1
98
0
0
0
100
0
0
Appendix
Table 23: Per-dataset failure analysis of OpenVLHarness on Qwen3-VL-8B. #Fail.: critiqued failures. Root causes: NC = tool not called, Wrong = tool output wrong, Unus./Ign. = tool output unusable or ignored, Reas. = reasoning error, Label = label or question issue. Remedies: A = evidence interpretation, B = output verification/refinement, C = tool composition, Y = harness text, Z = no proposed remedy. Tool = A+B+C and Text = Y; the larger share is bold. Percentages are rounded independently. NC → build: share of tool-not-called failures the critic marks as needing a new tool. On the box-AP grounding benchmarks, a failure is an image with F1 below 0.5 at IoU 0.5 .
Tool
What it does
Gate
Held-out
Generated from SparBench failures
image_points_to_metric_3d
Lifts localized image points or box centers into camera coordinates using depth and inverse intrinsics, and returns pairwise distances, distances from the observer, and nearest-point rankings in meters, with a warning when a point lies on a depth edge.
30/50
13/25
camera_center_image_projector
Estimates camera poses across ordered views and projects another view’s camera center into a reference image, returning its image coordinates and depth together with an annotated image.
40/50
18/25
Generated from FREAK-MCQ failures
target_visibility_absence_verifier
After grounding or zooming, checks whether a small object, part, or marking (a handle, wheel, blade, knob, …) is actually visible, and returns an explicit visible or not-visible status instead of a raw detection.
12/30
7/15
object_layout_relation_resolver
Converts grounded boxes of objects or parts into explicit layout relations: which end or side holds a part, immediate neighbors, row order, left-versus-right counts, and symmetry.
4/30
3/15
Appendix
Table 24: Accepted task-specific tools and recovery on their supporting examples. Gate : recovered failures / all examples for that direction. Held-out : recovered failures / examples withheld from the tool-writing agent. The held-out subset is included in the gate total. These counts measure recovery on selected failures, not accuracy on the full target benchmark.
Method
Backbone
Tools
Search
Training-free
OctoTools
Qwen3-VL-8B/32B
Generalist solution generator
×
Visual Sketchpad
Qwen3-VL-8B/32B, GPT-6 Luna
Detection, SoM, depth, zoom, overlay
×
Codex
Qwen3-VL-8B/32B, GPT-6 Luna
Code execution (GPT-6 Luna only; see text)
×
Training-based (released checkpoints)
AdaReasoner
AdaReasoner-7B
Point, OCR, crop, drawing
×
Appendix
Table 25: Configurations of the reproduced baselines. Search marks access to a web-search tool. All baselines except Codex decode greedily; Codex uses its own request settings.
Qwen3-VL-8B
Qwen3-VL-32B
Domain
Dataset
Base
Ours
Base
Ours
CountQA
0.40
1.08
0.40
0.53
FSC-147
0.26
0.17
0.26
0.59
OdinW-13-All
0.98
0.94
0.89
1.94
FSCD-147
0.14
0.06
0.14
0.10
Grounded Perception
FSCD-LVIS
0.15
0.14
0.15
0.22
Appendix
Table 26: Standard deviations across three runs for the Qwen3-VL Base and OpenVLHarness results in Table 1 . Average is the standard deviation of the per-run 22 -dataset average.
Variant
Grounding
VS & DR
General VQA
Spatial
Overall
8B
32B
8B
32B
8B
32B
8B
32B
8B
32B
Baseline (no tool)
0.64
0.63
0.46
0.46
0.36
0.36
0.24
0.24
0.22
0.22
+ Tool
0.23
0.19
1.11
0.75
0.40
0.43
0.59
0.73
0.34
0.29
+ Capability layer
0.27
0.31
0.71
0.64
0.36
0.42
0.55
1.01
0.25
0.32
+ Textual memory
0.22
0.20
0.74
0.56
0.56
0.46
0.31
0.39
0.25
0.21
OpenVLHarness
0.27
0.24
1.05
0.65
0.62
0.44
0.56
0.75
0.34
0.28
Appendix
Table 27: Standard deviations across three runs for the ablation in Table 3 . Domain and overall columns are standard deviations of the per-run averages.
Tool use in vision-language models (VLMs) studies how models call visual, computational, and retrieval tools to improve grounded perception and reasoning.
Recent large vision-language models (LVLMs) have demonstrated impressive reasoning ability by generating long chain-of-thought (CoT) responses. However, CoT reasoning in multimodal contexts is highly vulnerable to visual hallucination propagation: once an intermediate reasoning step becomes inconsistent with the visual evidence, subsequent steps-even if logically valid-can still lead to incorrect final answers. Existing solutions attempt to mitigate this issue by training models to "think with images" via reinforcement learning (RL). While effective, these methods are costly, model-specific, and difficult to generalize across architectures. Differently, we present a lightweight method that bypasses RL training and provides an iterative, training-free, plug-and-play framework for visually-grounded multimodal reasoning. Our key idea is to supervise each reasoning step at test time with visual evidence, ensuring that every decoded token is justified by corresponding visual cues. Concretely, we construct a textual visual-evidence pool that guides the model's reasoning generation. When existing evidence is insufficient, a visual decider module dynamically extracts additional relevant evidence from the image based on the ongoing reasoning context, expanding the pool until the model achieves sufficient visual certainty to terminate reasoning and produce the final answer. Extensive experiments on multiple LVLM backbones and benchmarks demonstrate the effectiveness of our approach. Our method achieves 16.5%-29.5% improvements on TreeBench and 13.7% RH-AUC gains on RH-Bench, substantially reducing hallucination rates while improving reasoning accuracy without additional training.
Yongchang Zhang, Oliver Ma, Tianyi Liu +2
Southeast University · AIIA, Ministry of Education, China · University of Oxford
We show that multimodal models possess strong reasoning abilities and that an appropriate harness can unlock their potential to solve tasks across diverse interactive environments. We introduce VISTA, a visual harness that gives a general-purpose multimodal model long-horizon vision. VISTA allows the model to directly perceive the environment through visual observations and maintains a lossless visual memory that preserves past observations in their original form. The model can actively retrieve these observations and reorganize its visual input as it reasons. On ARC-AGI-3, VISTA improves Claude Opus 5.0's Relative Human Action Efficiency score from 40.68 to a perfect 100.00, with the model completing all 25 public games using 57.4% fewer actions than first-time human participants. VISTA's simple design also allows it to extend naturally to diverse visual environments with minimal adaptation. Across three additional benchmarks covering a diverse range of visual games and puzzles, it substantially outperforms baselines using the same underlying model with minimal harnesses. Our results highlight VISTA's potential as a general-purpose visual harness for advancing multimodal agents in complex visual environments.
Despite rapid progress, multimodal large language models (MLLMs) still fail on tasks that humans solve effortlessly, such as navigating a grid maze from a screenshot or selecting the correct puzzle piece. Rather than retraining the model, we ask a complementary question: how much capability can be elicited from a frozen MLLM purely by improving the execution scaffold around it? We introduce MUSE, a multimodal unified structured execution harness that wraps any off-the-shelf MLLM with composable modules for task representation, visual processing, perception tool use, structured parsing, deterministic verification, and verifier-guided repair, without any model retraining. We evaluate MUSE across diverse benchmarks spanning visual spatial planning, visual perception, multimodal reasoning, and fine-grained visual discrimination, using multiple state-of-the-art MLLMs. MUSE delivers consistent gains over the bare model in all settings, with the largest jumps on challenging instances. Further analysis reveals that many MLLM failures arise from harness-level shortcomings rather than fundamental model deficits, and can be addressed through verifier-guided repair without touching the model. These findings highlight the agentic multimodal harness as a critical yet underexplored design dimension, offering an orthogonal avenue for improving MLLMs beyond model-centric optimization.