OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory

Yu Zhou; Cheng-Fu Yang; Rui Sun; Di Wu; Sibo Peng; Mingyang Zhang; Zhen Yang; Ruiyi Zhang

cs.CLOct 19, 2026

OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory

Authors: Yu Zhou, Cheng-Fu Yang, Rui Sun, Di Wu, Sibo Peng, Mingyang Zhang, Zhen Yang, Ruiyi Zhang,

Organizations: University of California, Los Angeles · Apple

Abstract

Multimodal large language models (MLLMs) have made rapid progress on visual reasoning, yet many tasks require evidence that is difficult to obtain through internal reasoning alone: our controlled study on 22 benchmarks shows that simply increasing the inference budget of the base model yields only modest gains. Such tasks demand grounded perception, external knowledge, and numerical measurements obtained through heterogeneous tools, whose outputs must be made interpretable to the model and retained across reasoning steps. We introduce OpenVLHarness, a multimodal reasoning harness organized around a capability layer and persistent multimodal memory. The capability layer combines task-level tool interfaces with evidence rendering, providing consistent access to heterogeneous backends and converting their outputs into model-readable observations. Persistent multimodal memory retains the corresponding images, structured data, and web references as addressable artifacts throughout each query. A coding interface executes generated programs over stored data, enabling computations that combine results from multiple tools. Together, these mechanisms support an iterative process of acquiring, inspecting, and composing multimodal evidence. Evaluated on 22 benchmarks with eight open and proprietary backbones, OpenVLHarness improves average scores by 9.7-12.6 points over matched no-tool baselines, including +11.2 with Qwen3-VL-8B, +9.7 with Kimi K3, and +9.9 with GPT-6 Astra. With GPT-6 Luna at medium reasoning effort, OpenVLHarness outperforms the base model, Codex, and Visual Sketchpad at maximum effort by 7.7-8.5 points while incurring at most 37% of their model-token cost.

Figures & tables

Figure 1: Performance of OpenVLHarness across models and settings. (a) Average score over 22 challenging multimodal benchmarks with and without OpenVLHarness, across the Qwen, Kimi, and GPT model families. (b) Benchmark scores for GPT-6 Luna and GPT-6 Astra at medium reasoning effort, each with and without OpenVLHarness; radii are rescaled per benchmark, and numbers give the raw scores of GPT-6 Astra (medium) with OpenVLHarness. (c) Average score versus model-token cost (average for 1,000 query rollouts) on 22 benchmarks for GPT-6 Luna alone and with Codex, Visual Sketchpad, or OpenVLHarness, across all five reasoning-effort settings.

Figure 2: Overview of OpenVLHarness. The MLLM orchestrator invokes capabilities through typed interfaces. The capability layer connects these interfaces to underlying models and services and renders their outputs into model-facing evidence: here, a labeled mask overlay accompanied by a textual description and normalized bounding boxes. Persistent multimodal memory retains image artifacts, structured data, and web references under reusable handles. An Environment Update advertises the available artifacts to the orchestrator, allowing subsequent calls to reuse earlier evidence, including through memory-grounded computation.

Figure 3: Qualitative examples of OpenVLHarness. (a) Camera poses, depth, and grounding are combined in code to measure a 3D distance (Example F ). (b) Grounded crops are passed to OCR and image search, and the answer is verified on the web (Example F ). (c) Masks from two grounding queries are reconciled in code to reach the exact count (Example F ).

		Qwen3-VL-8B				Qwen3-VL-32B				GPT-6 Luna (medium)				GPT-6 Astra (medium)
Domain	Dataset	Base	OT	VS	Ours	Base	OT	VS	Ours	Base	VS	CX	Ours	Base	Ours
	CountQA	32.00	29.06	19.83	48.56	44.70	41.75	32.85	64.14	55.17	52.68	58.25	66.95	82.00	86.40
	FSC-147	15.32	11.26	9.41	37.71	18.97	12.44	12.35	39.42	28.23	30.25	28.62	41.52	44.01	46.11
	OdinW-13-All	39.80	24.60	31.77	50.40	42.39	44.33	36.10	59.13	36.74	43.84	36.14	53.56	50.46	60.45
	FSCD-147	1.79	0.93	1.41	23.52	2.88	2.55	2.30	28.45	4.24	6.11	2.86	25.74	15.49	30.33
Grounded Perception	FSCD-LVIS	1.47	1.36	2.49	27.37	1.79	3.91	4.57	31.63	2.44	8.65	2.23	30.55	10.55	36.02

Table 1: Main results across four task domains and 22 datasets. Results are grouped by MLLM backbone. Base denotes the backbone without tool use; for harness, OT , VS , and CX denote OctoTools, Visual Sketchpad, and Codex, respectively; Ours denotes OpenVLHarness. GPT-6 Sol, GPT-5, and Kimi K3 results are in Table 7 . Bold indicates the best result within each backbone group, and underlining indicates the best result across all shown backbones. Qwen3-VL results are means of three runs (standard deviations in Table 26 ); GPT-6 results are single runs. OdinW-13-All asks for all categories of an image’s source dataset in one query.

Reasoning effort	medium				high				xhigh				max
Reasoning effort	Base	CX	VS	Ours	Base	CX	VS	Ours	Base	CX	VS	Ours	Base	CX	VS	Ours
Grounding	25.4	25.6	28.3	43.7	26.5	26.3	29.3	44.6	27.0	27.1	29.3	45.2	27.2	30.0	29.6	45.6
VS & DR	35.2	33.5	32.4	47.5	35.3	35.4	34.6	50.3	37.4	36.7	35.6	50.6	37.8	40.3	36.9	52.7
General VQA	63.4	60.3	64.3	69.8	64.4	61.8	65.1	71.5	64.7	63.0	66.6	71.5	64.5	63.5	66.5	72.6
Spatial	57.7	54.4	56.2	64.2	59.3	57.0	57.1	65.3	58.7	58.4	58.5	65.9	61.5	59.1	61.4	68.5
Average	45.8	43.8	45.5	56.5	46.7	45.5	46.9	58.2	47.3	46.6	47.8	58.6	48.0	48.6	48.8	60.1

Table 2: Reasoning effort versus harness, by domain. GPT-6 Luna alone (Base) and with Codex (CX), Visual Sketchpad (VS), or OpenVLHarness (Ours) on the full evaluation splits of our

22

benchmarks. Rows average each domain’s benchmarks: Grounding (grounded perception,

5

), VS & DR (visual search and deep research,

6

), General VQA (general VQA and hallucination,

6

), and Spatial (spatial reasoning,

5

/ 1k questions is the model-token cost at Standard-tier prices, including reasoning tokens and excluding tool execution and judging, averaged over the

22$ benchmarks. At each effort, bold marks the best score, and in the cost row the cheapest method, with the second cheapest underlined. Low effort is in Appendix Table 13 .

Variant	Grounding		VS & DR		General VQA		Spatial		Overall
Variant	8B	32B	8B	32B	8B	32B	8B	32B	8B	32B
Baseline (no tool)	27.48	31.72	18.40	23.34	57.97	62.16	44.38	47.93	37.06	41.29
+ Tools	41.70	43.30	21.44	24.89	52.41	61.41	47.83	52.84	40.84	45.61
+ Capability layer	46.73	46.20	25.97	32.74	56.05	63.63	48.16	52.63	44.23	48.80
+ Textual memory	47.27	45.74	26.92	34.43	57.54	64.84	47.98	51.46	44.93	49.12
OpenVLHarness	45.46	51.74	33.10	37.96	59.81	66.12	50.82	54.59	47.14	52.51

Table 3: Ablation study of OpenVLHarness across model scales. We progressively add raw tool access, capability layer, textual memory, and multimodal memory to the no-tool baseline. Results are reported across four domains and overall for Qwen3-VL 8B and 32B models, averaged over three runs; standard deviations are provided in Table 27 . Bold marks the best result in each column.

Figure 4: Failure diagnosis and harness specialization with Qwen3-VL-8B. We critique failure trajectories using Kimi K3. (a) Distribution of failure causes across four domains. (b) Proposed remedies: grouped into creating task-specific tools (A–C), updating harness text (Y), and no proposed remedy (Z). (c) Four datasets selected for contrasting remedy profiles: Y (orange) dominates ERQA and RealWorldQA, while A–C (blue) dominates SparBench and FREAK-VQA. (d) Gains from harness search and tool generation over the unoptimized harness. Harness search is more effective on datasets dominated by text-level changes, while tool generation yields larger gains when task-specific tools are preferred.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Tool-call volume and engagement by domain. (a) Tool calls per question, averaged over all questions in the domain. (b) Fraction of questions answered with no tool call. Engagement varies with model scale within a family and not across families, and call volume does not order the models by accuracy.

Model	Calls /	Calls / engaged	No tool	Messages /	Repeat	Failed
	question	question	(%)	question	calls (%)	calls (%)
Qwen3-VL-8B	1.43	2.00	28.9	9.7	1.92	0.59
Qwen3-VL-32B	2.59	2.76	6.2	14.3	0.54	0.59
GPT-6 Luna	1.25	1.67	25.3	7.7	0.38	0.30
GPT-6 Sol	2.07	2.18	4.8	10.2	1.95	0.59
GPT-6 Astra	1.80	1.84	2.6	9.4	0.01	0.17

Appendix

Table 4: Tool-use profile of OpenVLHarness across five base models. Calls per question averages over all questions, including those answered without tools; calls per engaged question conditions on questions with at least one call. Messages count transcript turns, including the Environment Update emitted after each call. Repeat calls are byte-identical to an earlier call in the same rollout; failed calls are those the tool server returned as errors.

Figure 6: Distribution of tool calls across capabilities. All calls issued by each model, normalized to 100%. Visual Grounding is the largest consumer for every model; the models differ mainly in the second-ranked capability.

Figure 7: Distribution of tool calls by domain. Calls issued in each domain as a share of that domain’s calls, with the call total below each panel. Colors are consistent across panels. Grounded perception is dominated by Visual Grounding for every model except GPT-6 Sol and GPT-6 Astra; visual search and deep research is the only domain dominated by the web capabilities; spatial reasoning is the only domain in which four capabilities carry comparable load.

Model	Rows	Base	OpenVLHarness	$\Delta$	Rescued	Broken	Calls per
							net gain
Qwen3-VL-8B	24,410	50.02	60.09	$+$ 10.07	4,649	1,967	12.9
Qwen3-VL-32B	24,393	53.73	62.17	$+$ 8.44	4,381	2,146	28.1
GPT-6 Luna	26,226	57.88	67.67	$+$ 9.79	4,191	1,382	11.7
GPT-6 Sol	23,700	64.26	72.67	$+$ 8.41	3,153	1,025	23.1
GPT-6 Astra	23,700	70.74	76.98	$+$ 6.24	2,336	700	26.0

Appendix

Table 5: OpenVLHarness against the base model, paired per question. Rescued counts questions answered incorrectly by the base model and correctly by OpenVLHarness; broken counts the converse. Calls per net gain divides the total number of tool calls issued on these questions by the net number of questions whose outcome changed.

Figure 8: Per-capability paired comparison against the base model. For each capability and model, the accuracy of OpenVLHarness minus that of the base model on the questions whose rollout invoked that capability. The corresponding difference on the questions that did not invoke it, which serves as the control, is given in the last column of Table 6 together with the share of questions each capability was invoked on.

Capability	Model	Questions	Share	Calls	Base	OpenVLHarness	$\Delta$	$\Delta$ elsewhere
Visual Grounding	Qwen-8B	13,103	54%	21,286	47.7	62.0	$+$ 14.3	$-$ 20.1
	Qwen-32B	17,284	71%	40,057	56.1	64.5	$+$ 8.3	$+$ 0.2
	Luna	13,216	50%	17,670	56.5	67.4	$+$ 10.9	$+$ 2.7
	Sol	14,369	61%	22,313	67.5	74.3	$+$ 6.7	$+$ 2.2
	Astra	15,488	65%	22,385	73.6	77.9	$+$ 4.3	$+$ 5.1
Zoom-in	Qwen-8B	1,703	7%	2,714	40.3	38.8	$-$ 1.6	$+$ 4.8

Appendix

Table 6: Per-capability paired comparison, corresponding to Figure 8 . For each capability and model: the questions whose rollout invoked it at least once, the calls issued on them, the accuracy of the base model and of OpenVLHarness on exactly those questions, their difference, and the same difference computed on the questions that did not invoke the capability. Share is the fraction of paired questions that invoked it, which determines how closely the final column approximates the model’s overall gain. A question that invoked three capabilities is counted once under each.

		GPT-5		GPT-6 Sol (medium)		Kimi K3 (max)
Domain	Dataset	Base	Ours	Base	Ours	Base	Ours
	CountQA	38.48	58.70	73.00	76.00	71.27	73.10
	FSC-147	22.16	42.92	38.72	44.40	36.78	42.07
	OdinW-13-All	46.27	63.12	40.86	57.88	44.35	56.91
	FSCD-147	0.16	28.09	8.36	29.42	7.88	29.11
Grounded Perception	FSCD-LVIS	2.04	33.22	5.17	35.22	7.15	33.60

Appendix

Table 7: Results of OpenVLHarness with GPT-5 ( OpenAI, 2025b ) , GPT-6 Sol ( OpenAI, 2026 ) , and Kimi K3 ( Team et al., 2026 ) as the backbone, on the same

22

datasets and four task domains as Table 1 . Base: the backbone prompted directly; Ours: the backbone with OpenVLHarness. Bold marks the better of Base and Ours for each backbone.

Variant	Tool schema	Post-proc.	Env update	Bindings	URL refs	Image refs
Baseline	—	—	—	—	—	—
+ Tool	minimal	raw	off	off	off	off
+ Capability layer	full	custom	off	off	off	off
+ Textual memory	full	custom	on	on	on	off
OpenVLHarness	full	custom	on	on	on	on

Appendix

Table 8: Ablation knob settings for each variant in Table 3 . “Post-proc.” refers to per-tool output processors (custom vs. raw dumps). “Env update” is the per-turn Environment Update message. “Bindings” are out-of-context typed data bindings. “URL refs” and “Image refs” are symbolic identifiers for web pages and tool-generated images, respectively.

		Training-based			Training-free
Domain	Dataset	AdaReasoner	PixelReasoner	DeepEyesV2	OpenVLHarness
	CountQA	36.98	21.66	21.73	51.37
	FSC-147	1.34	9.75	21.26	31.60
	OdinW-13-All	0.00	10.24	8.97	7.80
	FSCD-147	0.00	0.04	0.06	10.30
	FSCD-LVIS	0.00	0.00	0.05	14.20

Appendix

Table 9: Training-based baselines on the Qwen2.5-VL-7B-Instruct backbone. AdaReasoner, PixelReasoner, and DeepEyesV2 fine-tune the backbone to adapt it to their tool-use harness; OpenVLHarness is training-free and uses the same weights out of the box. Training-based columns are gray; shaded rows are per-domain averages. Bold marks the best result in each row.

Figure 9: Object-count distribution across counting benchmarks. Each bar shows the percentage of samples falling into binned object-count buckets, with the per-dataset maximum count annotated above. Widely used benchmarks such as TallyQA, CountBenchQA, and PixmoCount are dominated by low counts: nearly all of their samples contain ten or fewer objects (and a majority of TallyQA fall in the

0

–

1

bucket), capping at

10

–

15

. Only FSC-147 and FSCD-LVIS contain substantial mass in the high-count regime (

>50

objects), reaching into the hundreds or thousands. This skew means accuracy on standard counting benchmarks largely reflects performance on small object counts and reveals little about high-count visual measurement.

Figure 10: Object-count distribution across grounding benchmarks. Each bar shows the percentage of samples falling into binned object-count buckets, with the per-dataset maximum count annotated above. Referring-expression benchmarks such as RefCOCO/+/g and Ref-L4/GroundingME ground a single target per sample (

100\%

in the

0

–

1

bucket,

\text{max}=1

), and even detection-style sets like OdinW-13 (

97\%

) and LVIS (

86\%

) are overwhelmingly concentrated at one object per image. Only the FSCD variants exercise dense, many-object grounding. As with counting, this distribution implies that progress on common grounding benchmarks is measured almost entirely in the single- or few-object regime.

		Qwen3-VL-8B		Qwen3-VL-32B
Objects	# Samples	Base	OpenVLHarness	Base	OpenVLHarness
FSC-147: exact-match accuracy (%)
$\leq 10$	149	42.3	73.8	64.4	81.9
11–20	368	28.8	59.2	33.4	63.3
21–50	383	5.0	32.9	5.2	34.5
51–100	185	1.1	9.7	2.2	9.7

Appendix

Table 10: Grounded perception performance by number of ground-truth objects. FSCD-LVIS is scored with instance-level F1 (predicted boxes matched to ground truth at IoU

\geq 0.5

) because AP is not defined per sample.

		FSC-147	CountQA		LVIS
Model	Method	MAE $\downarrow$	RA@5 $\uparrow$	RA@10 $\uparrow$	AP $\uparrow$
Qwen3-VL-8B	Base	29.70	33.64	43.13	18.12
Qwen3-VL-8B	OpenVLHarness	23.17	50.00	57.59	45.19
Qwen3-VL-32B	Base	60.70	46.79	58.90	22.87
Qwen3-VL-32B	OpenVLHarness	31.24	65.51	72.71	45.46

Appendix

Table 11: Complementary counting metrics and LVIS grounding. MAE is the mean absolute counting error; RA@

k

is the share of predictions within

k\%

of the ground truth.

Model	MMS	MSP	RXB	FVQA	HLE-VL	BC-VL	MM-BC	VDR	VBB	InfoSeek	Enc-VQA	Avg
Direct Inference
Qwen3-VL-8B	14.04	1.29	22.16	28.39	6.36	21.05	4.94	5.40	7.69	29.20	10.61	13.74
Qwen3-VL-8B-T	12.87	3.22	22.16	25.67	10.30	33.83	7.41	8.20	11.24	27.25	18.06	16.38
Qwen3-VL-32B	20.47	2.57	24.74	32.67	10.00	28.82	6.58	9.40	10.06	33.20	32.76	19.21
GPT-5	36.84	10.93	34.54	54.67	20.00	42.11	8.64	14.20	28.40	53.67	43.33	31.58
Agent Workflow

Appendix

Table 12: Extended visual search and deep research results on

11

information-seeking benchmarks, five of which (MM-BC, VDR, VBB, InfoSeek, Enc-VQA) are not in Table 1 . Qwen3-VL-8B refers to Qwen3-VL-8B-Instruct and Qwen3-VL-8B-T denotes Qwen3-VL-8B-Thinking. Abbreviations: MM-BC = MM-BrowseComp; MMS = MMSearch; MSP = MMSearch-Plus; BC-VL = BrowseComp-VL; RXB = RealX-Bench; VDR = VDR-Bench; VBB = VisBrowse-Bench; Enc-VQA = EncyclopedicVQA. Avg is the unweighted mean across the

11

benchmarks; per-benchmark sample counts are in Appendix D.2 .

		Domain average
Method	Effort	Grounding	VS & DR	General VQA	Spatial	Average	$ / 1k
		(5)	(6)	(6)	(5)	(22)
Base	low	24.28	33.49	61.63	56.54	44.31	0.36
	medium	25.36	35.18	63.38	57.69	45.76	0.58
	high	26.54	35.27	64.37	59.27	46.68	0.85
	xhigh	27.01	37.36	64.72	58.73	47.33	1.18

Appendix

Table 13: GPT-6 Luna across five reasoning-effort settings for each method, evaluated on the full evaluation splits of all

22

benchmarks of Table 1 (

26{,}229

questions per setting). Domain averages are unweighted means over the benchmarks of each domain (number of benchmarks in parentheses); per-benchmark scores are in Table 16 .

/ 1k is the model-token cost per

1{,}000

questions at Standard-tier prices, including reasoning tokens and excluding tool execution and judging, averaged over the

22$ benchmarks. Bold marks the best score in each score column. Each cell is a single run.

				Input tokens				Output tokens
Method	Effort	Score	$/1k	Uncached	Cache read	Cache write	Total	Reasoning	Total
Base	low	44.31	0.36	1,815	30	0	1,845	301	348
	medium	45.76	0.58	1,793	52	0	1,845	750	804
	high	46.68	0.85	1,795	50	0	1,845	1,283	1,345
	xhigh	47.33	1.18	1,796	49	0	1,845	1,938	2,006
	max	48.04	3.17	216	69	1,557	1,842	5,852	5,902

Appendix

Table 14: Model tokens per question behind Table 13 , summed over every model call made for a question and averaged over the

22

benchmarks. Output totals include the reasoning tokens. For OpenVLHarness, the calls of the Python Coding Agent are logged per setting rather than per question and are apportioned evenly across questions; they account for

0.4\%

of its tokens and

2.5\%

of its model-token cost at low effort, rising to

3.6\%

and

14.7\%

at max effort.

		Executes code		Executes code, by domain (%)
Effort	Questions	$n$	%	Grounding	VS & DR	General VQA	Spatial	Commands	Per exec.	Score
low	26,229	43	0.16	0.00	1.34	0.00	0.00	126	2.9	36.92
medium	26,229	106	0.40	0.30	2.31	0.02	0.00	679	6.4	43.78
high	26,229	221	0.84	1.11	3.21	0.09	0.00	2,051	9.3	45.46
xhigh	26,229	457	1.74	2.88	4.77	0.16	0.01	4,953	10.8	46.61
max	26,229	1,619	6.17	12.33	9.61	0.89	0.00	18,753	11.6	48.58

Appendix

Table 15: How often Codex executes code with GPT-6 Luna in the effort study. A question counts as executing code if its Codex session contains at least one command execution. Commands is the total number of executed commands, and Per exec. the mean number of commands per question that executes code. Score is the

22

-benchmark average from Table 13 .

	Base					Codex
Dataset	low	med	high	xhigh	max	low	med	high	xhigh	max
CountQA	52.03	55.17	58.05	59.03	62.04	44.90	58.25	59.42	59.69	62.24
FSC-147	28.62	28.23	29.32	29.78	29.94	22.63	28.62	29.55	31.88	38.34
OdinW-13-All	35.67	36.74	38.18	38.80	36.55	30.66	36.14	36.78	37.50	38.03
FSCD-147	3.06	4.24	4.57	4.79	4.82	1.88	2.86	3.41	4.04	7.27
FSCD-LVIS	2.05	2.44	2.56	2.66	2.84	1.40	2.23	2.31	2.43	4.05

Appendix

Table 16: Per-benchmark scores behind Table 13 : GPT-6 Luna at five reasoning-effort settings for each method, on the full evaluation split of every benchmark. Scoring follows Table 1 . Bold marks the best score in each benchmark row across all

20

settings. † Requests for some questions failed on every retry (

134

questions in

55

cells; at most

13

in a single cell). These count as incorrect.

/ 1k questions is the model-token cost at Standard-tier prices, averaged over the

22$ benchmarks. Each cell is a single run.

		Qwen3-VL-8B			Qwen3-VL-32B
Domain	Dataset	Base	Codex	OpenVLHarness	Base	Codex	OpenVLHarness
Grounding	CountQA	32.00	29.97	48.56	44.70	39.73	64.14
	FSC-147	15.32	13.53	37.71	18.97	16.41	39.42
	OdinW-13-All	39.80	33.70	50.40	42.39	34.66	59.13
	FSCD-147	1.79	0.50	23.52	2.88	0.50	28.45
	FSCD-LVIS	1.47	0.78	27.37	1.79	2.24	31.63

Appendix

Table 17: Codex with Qwen3-VL backbones on the 22 benchmarks of Table 1 . Base and OpenVLHarness are copied from that table. Codex runs the Codex CLI agent in code mode at medium reasoning effort with web search disabled. Bold marks the best result within each model group. Averages are taken over all 22 rows. Each Codex cell is a single run.

Capability	Interface	Backends
Search
Text Search	text query $\rightarrow$ summarized results with cited URLs	Serper ; Gemini with Google Search
Image Search	image $\rightarrow$ visual matches (entities, URLs)	Serper (Google Lens) ; Gemini with Google Search
Webpage Visit	URLs and goal $\rightarrow$ goal-relevant evidence	Serper with an LLM extractor
Perception
Zoom-in	image and box $\rightarrow$ cropped image	built-in

Appendix

Table 18: The nine capabilities, their interfaces, and their interchangeable backends. Bold marks the backend used in all experiments.

Artifact	Source tool	Stored representation in memory	Reusable handle
Image artifacts (returned to the model as a native image attachment + an ID)
Input image	(question)	Registered verbatim	input_image_ n
Cropped region	Zoom-in	Cropped image, deduplicated by content	tool_generated_image_ n
Grounded / mask overlay	Visual Grounding	Annotated image with numbered regions	tool_generated_image_ n
Colorized depth map	Depth Estimation	Rendered depth visualization	tool_generated_image_ n
Camera-trajectory plot	Camera Trajectory	Bird’s-eye-view render of the estimated camera path	tool_generated_image_ n

Appendix

Table 19: Tool-generated artifacts retained in persistent multimodal memory, grouped by how they are represented and re-addressed. Heavy payloads (image bytes, numeric arrays) are stored once and replaced in the transcript by a compact handle; the orchestrator reuses an artifact on a later turn by citing its handle, which the harness resolves back to the underlying data.

Model	Role	Access
Agent backbones
Qwen2.5-VL-7B-Instruct	Backbone (vs. training-based baselines)	vLLM
Qwen3-VL-8B-Instruct	Backbone	vLLM
Qwen3-VL-32B-Instruct	Backbone	vLLM
Qwen3-VL-8B-Thinking	Direct-inference reference (search)	vLLM
GPT-5	Backbone	OpenAI API

Appendix

Table 20: Models used in our experiments.

Domain	Benchmark	Split	# Samples	Format	Scoring
Grounded Perception	CountQA	test	1,528	integer	exact match
	FSC-147	val	1,286	integer	exact match
	OdinW-13-All	val	4,938	labeled boxes	box AP
	FSCD-147	val	1,286	boxes	box AP
	FSCD-LVIS	val	1,181	boxes	box AP
Visual Search and Deep Research	MMSearch	end-to-end	171	open-ended	GPT-4o judge

Appendix

Table 21: The

22

benchmarks of Table 1 . “LLM judge” is Qwen3-VL-8B-Instruct; MRA is mean relative accuracy for numeric answers.

Label	Meaning
Root cause (exactly one)
Tool not called	The needed evidence was never gathered, although a tool in the agent’s toolbox could have supplied it.
Tool output wrong	A tool was called but returned wrong or missing information (missed or hallucinated boxes, a bad crop, a wrong transcription).
Tool output unusable	A tool returned correct information in a form the model could not use (raw arrays, uninterpreted coordinates, an overload of detections).
Tool output ignored	A tool returned correct, plainly usable information that the model did not incorporate or contradicted.
Reasoning error	The evidence was adequate and used; the error is in reasoning, arithmetic, comparison, or answer format.

Appendix

Table 22: Critique taxonomy. Each failure receives one primary cause and one proposed fix type. Unusable and ignored outputs remain separate during critique and are combined for reporting in Fig. 4 and Table 23 .

			Root cause (%)					Remedy channel (%)
Domain	Dataset	#Fail.	NC	Wrong	Unus./Ign.	Reas.	Label	A	B	C	Y	Z	Tool	Text	NC $\to$ build
Grounding	CountQA	751	32	53	4	10	1	12	54	2	31	1	68	31	8
	FSC-147	803	9	89	1	2	0	2	89	2	7	0	93	7	27
	OdinW-13-All	1,016	4	56	2	15	23	12	56	0	9	23	68	9	7
	FSCD-147	90	0	79	6	0	16	3	81	0	0	16	84	0	0
	FSCD-LVIS	370	0	98	1	0	0	1	98	0	0	0	100	0	0

Appendix

Table 23: Per-dataset failure analysis of OpenVLHarness on Qwen3-VL-8B. #Fail.: critiqued failures. Root causes: NC = tool not called, Wrong = tool output wrong, Unus./Ign. = tool output unusable or ignored, Reas. = reasoning error, Label = label or question issue. Remedies: A = evidence interpretation, B = output verification/refinement, C = tool composition, Y = harness text, Z = no proposed remedy. Tool = A+B+C and Text = Y; the larger share is bold. Percentages are rounded independently. NC

\to

build: share of tool-not-called failures the critic marks as needing a new tool. On the box-AP grounding benchmarks, a failure is an image with F1 below

0.5

at IoU

0.5

Tool	What it does	Gate	Held-out
Generated from SparBench failures
image_points_to_metric_3d	Lifts localized image points or box centers into camera coordinates using depth and inverse intrinsics, and returns pairwise distances, distances from the observer, and nearest-point rankings in meters, with a warning when a point lies on a depth edge.	30/50	13/25
camera_center_image_projector	Estimates camera poses across ordered views and projects another view’s camera center into a reference image, returning its image coordinates and depth together with an annotated image.	40/50	18/25
Generated from FREAK-MCQ failures
target_visibility_absence_verifier	After grounding or zooming, checks whether a small object, part, or marking (a handle, wheel, blade, knob, …) is actually visible, and returns an explicit visible or not-visible status instead of a raw detection.	12/30	7/15
object_layout_relation_resolver	Converts grounded boxes of objects or parts into explicit layout relations: which end or side holds a part, immediate neighbors, row order, left-versus-right counts, and symmetry.	4/30	3/15

Appendix

Table 24: Accepted task-specific tools and recovery on their supporting examples. Gate : recovered failures / all examples for that direction. Held-out : recovered failures / examples withheld from the tool-writing agent. The held-out subset is included in the gate total. These counts measure recovery on selected failures, not accuracy on the full target benchmark.

Method	Backbone	Tools	Search
Training-free
OctoTools	Qwen3-VL-8B/32B	Generalist solution generator	×
Visual Sketchpad	Qwen3-VL-8B/32B, GPT-6 Luna	Detection, SoM, depth, zoom, overlay	×
Codex	Qwen3-VL-8B/32B, GPT-6 Luna	Code execution (GPT-6 Luna only; see text)	×
Training-based (released checkpoints)
AdaReasoner	AdaReasoner-7B	Point, OCR, crop, drawing	×

Appendix

Table 25: Configurations of the reproduced baselines. Search marks access to a web-search tool. All baselines except Codex decode greedily; Codex uses its own request settings.

		Qwen3-VL-8B		Qwen3-VL-32B
Domain	Dataset	Base	Ours	Base	Ours
	CountQA	0.40	1.08	0.40	0.53
	FSC-147	0.26	0.17	0.26	0.59
	OdinW-13-All	0.98	0.94	0.89	1.94
	FSCD-147	0.14	0.06	0.14	0.10
Grounded Perception	FSCD-LVIS	0.15	0.14	0.15	0.22

Appendix

Table 26: Standard deviations across three runs for the Qwen3-VL Base and OpenVLHarness results in Table 1 . Average is the standard deviation of the per-run

22

-dataset average.

Variant	Grounding		VS & DR		General VQA		Spatial		Overall
Variant	8B	32B	8B	32B	8B	32B	8B	32B	8B	32B
Baseline (no tool)	0.64	0.63	0.46	0.46	0.36	0.36	0.24	0.24	0.22	0.22
+ Tool	0.23	0.19	1.11	0.75	0.40	0.43	0.59	0.73	0.34	0.29
+ Capability layer	0.27	0.31	0.71	0.64	0.36	0.42	0.55	1.01	0.25	0.32
+ Textual memory	0.22	0.20	0.74	0.56	0.56	0.46	0.31	0.39	0.25	0.21
OpenVLHarness	0.27	0.24	1.05	0.65	0.62	0.44	0.56	0.75	0.34	0.28

Appendix

Table 27: Standard deviations across three runs for the ablation in Table 3 . Domain and overall columns are standard deviations of the per-run averages.

Tool Use in VLMs

Tool use in vision-language models (VLMs) studies how models call visual, computational, and retrieval tools to improve grounded perception and reasoning.

See all 113 papers

See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs
Feb 25, 2026Yongchang Zhang, Oliver Ma, Tianyi Liu +2Multimodal Reasoning VLM Reasoning
Southeast University · AIIA, Ministry of Education, China · University of Oxford
VISTA: A Visual Harness for Reasoning in an Interactive World
Oct 1, 2026Qiushi Han, Keya Hu, Linlu Qiu +2Memory-Augmented VLMs Visual Reasoning
Massachusetts Institute of Technology
MUSE: A Unified Agentic Harness for MLLMs
Jun 2, 2026Jianglin Lu, Hailing Wang, Xu Ma +4Multimodal Reasoning Multimodal Agents
Northeastern University

Feb 25, 2026J/K move · Enter open · S save

See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs

Yongchang Zhang, Oliver Ma, Tianyi Liu, Guangquan Zhou+1

Southeast University · AIIA, Ministry of Education, China · University of Oxford

Multimodal Reasoning VLM Reasoning VLM Hallucination Mitigation Object Hallucination in VLMs

Open paper AskPDF arXiv

Recent large vision-language models (LVLMs) have demonstrated impressive reasoning ability by generating long chain-of-thought (CoT) responses. However, CoT reasoning in multimodal contexts is highly vulnerable to visual hallucination propagation: once an intermediate reasoning step becomes inconsistent with the visual evidence, subsequent steps-even if logically valid-can still lead to incorrect final answers. Existing solutions attempt to mitigate this issue by training models to "think with images" via reinforcement learning (RL). While effective, these methods are costly, model-specific, and difficult to generalize across architectures. Differently, we present a lightweight method that bypasses RL training and provides an iterative, training-free, plug-and-play framework for visually-grounded multimodal reasoning. Our key idea is to supervise each reasoning step at test time with visual evidence, ensuring that every decoded token is justified by corresponding visual cues. Concretely, we construct a textual visual-evidence pool that guides the model's reasoning generation. When existing evidence is insufficient, a visual decider module dynamically extracts additional relevant evidence from the image based on the ongoing reasoning context, expanding the pool until the model achieves sufficient visual certainty to terminate reasoning and produce the final answer. Extensive experiments on multiple LVLM backbones and benchmarks demonstrate the effectiveness of our approach. Our method achieves 16.5%-29.5% improvements on TreeBench and 13.7% RH-AUC gains on RH-Bench, substantially reducing hallucination rates while improving reasoning accuracy without additional training.

Abstract

Tool Use in VLMs

Explore similar work

See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs

VISTA: A Visual Harness for Reasoning in an Interactive World

MUSE: A Unified Agentic Harness for MLLMs