Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs

Muhammad Kamran Janjua; Hugo Silva; Di Niu; Bahador Rashidi

cs.CVApr 14, 2026

Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs

Authors: Muhammad Kamran Janjua, Hugo Silva, Di Niu, Bahador Rashidi

Organizations: Huawei Technologies, Canada · University of Alberta, Canada

Abstract

Multimodal language models (MLLMs) are increasingly paired with vision tools (e.g., depth, flow, correspondence) to enhance visual reasoning. However, despite access to these tool-generated visual cues, MLLMs often fail to benefit from them. Existing approaches typically feed raw tool outputs into the model, but these dense, pixel-level representations are misaligned with the language-native reasoning strengths of LLMs, leading to weak perception and reliance on language priors. We argue that, in problems where vision tools can provide the necessary visual cues, the bottleneck is not more tool calls or larger MLLMs, it is how tool outputs are represented. We introduce Perception Programs (P $^2$ ), a training-free, model-agnostic method that rewrites tool outputs into compact, structured, language-native summaries that MLLMs can directly parse and reason over. Across six perception-centric tasks in BLINK, P $^2$ consistently yields large improvements over base models and raw tool-augmented baselines. With GPT-5 Mini as the base model, P $^2$ raises its accuracy from 41.35% to 86.47% on multi-view reasoning, from 52.42% to 81.45% on relative depth, and achieves a 22% average gain across tasks, setting new state-of-the-art results. Even on smaller MLLMs, e.g., InternVL3.5-4B and Qwen3VL-4B, we observe 15-40% absolute gains from P $^2$ , surpassing prior agentic, supervised, and RL-based tool-use methods-without any training or model modifications.

Figures & tables

Figure 2: Under-utilization of Visual Information. Given several ICL examples along with depth map, GPT-5 Mini fails to recover near-to-far ordering from it (see Sec. 5.1 ), indicating limited utilization of the modality.

Figure 3: Perception Program Instantiations. Top: Tool outputs. Bottom: P 2 instantiations of those respective tools.


	Multi-View Reasoning	HardBLINK Depth	Visual Correspondence	Jigsaw	Object Localization	Semantic Correspondence
Human [ 8 ]	$92.48$	N/A	$99.42$	$99.00$	$98.00$	$96.07$
Thyme [ 33 ]	$51.88$	$35.21$	$63.95$	$58.00$	$48.36$	$39.57$
LATTE [ 14 ]	$48.87$	N/A	$29.65$	$75.33$	$47.54$	$33.09$
Visual Sketchpad (GPT-4o) [ 10 ]	$45.60$	N/A	$80.80$	$70.70$	$65.40$	$58.30$
Visual Sketchpad (GPT-5 Mini) ⋆ [ 10 ]	$47.37$	$63.71$	$91.86$	$79.33$	$67.21$	$60.43$

Table 1: Results on BLINK [ 8 ] . Accuracy (

\%

) on six perception-centric sub-tasks. We report three settings per model, Standard (image+question only), Raw Tool (tool output as auxiliary input), and P 2 (our training-free Perception Program instantiated from same tool output). Further, we also list representative prior tool-use or perception-oriented methods, plus previous state-of-the-art for each sub-task. Bold indicates best score within each model block. Even with the much smaller InternVL3.5-

4

B and Qwen3VL-

4

B MLLMs than prior work, P 2 yields large gains setting new state-of-the-art results. ⋆ indicates we ran with GPT-5 Mini as LLM and official codebase. denotes training/fine-tuning, and indicates no parameter updates necessary.


Methods	Visual Correspondence	Jigsaw	Semantic Correspondence
Mirage (Qwen2.5VL- $7$ B) [ 29 ]	N/A	$88.00$	N/A
Mirage (Qwen2.5VL- $3$ B) [ 29 ]	N/A	$85.00$	N/A
Zhang et al. [32] (Qwen2.5VL- $7$ B)	$69.77$	$74.67$	N/A
ViGoRL [ 17 ]	N/A	$56.00$	N/A
GPT-4 Turbo	$48.80$	$64.70$	$30.90$

Table 2: Additional BLINK Results. Comparison of task-specific methods, i.e., methods that do not report on entire BLINK benchmark, on three sub-tasks. denotes training/fine-tuning, and indicates no parameter updates necessary.

Figure 4: Mean

\Delta

vs. prior SOTA across BLINK. Bars show average accuracy improvement (percentage points) of each method over task-wise (except HardBLINK) prior state-of-the-art (at point zero; see Tab. 1 ). Positive values indicate gains over prior SOTA; negative values indicate regressions. Numeric

\Delta

are written inside/beyond the bars along with their method names. VS denotes Visual Sketchpad.

Figure 5: GPT-5 Depth Modality Analysis. Left: Kendall’s Tau (y-axis) between ground-truth and GPT-5 Mini reconstructed P 2 decreases as the grid is refined (x-axis). Right: HardBLINK-

5

accuracy (y-axis) using GPT-5 Mini’s reconstructions (GPT Recon.) across grids (x-axis).

Figure 6: GPT-5 Visual Correspondence Modality Analysis. Left: Plot of LoFTR [ 18 ] correspondences, with true displacement between views on the x-axis and GPT 5’s reconstruction error on the y-axis (both as % of image diagonal). Color indicates bin count. The strong diagonal indicates a failure mode where GPT often copies the left-image coordinate into the right-image field. Right: Accuracy (%) across $100$ matches. Naive refers to Euclidean distance baseline, while Oracle uses the tool directly.


Methods	HardBLINK			Object Localization
Methods	$\mathbf{3}$	$\mathbf{4}$	$\mathbf{5}$	Object Localization
VS [ 10 ]	$71.77$	$62.90$	$56.45$	$60.43$
VS + P 2	$\mathbf{81.45}$	$\mathbf{83.06}$	$\mathbf{79.84}$	$\mathbf{80.33}$

Table 3: Plug-and-Play. Visual Sketchpad (VS) results with P 2 .


Methods	HardBLINK
Methods	$\mathbf{3}$	$\mathbf{4}$	$\mathbf{5}$	Avg.
Aurora [ 1 ]	$66.90$	$60.50$	$54.80$	$60.73$
ReVPT- $7$ B [ 35 ]	$68.55$	$55.65$	$60.48$	$61.56$
Visual Sketchpad [ 10 ] (GPT-5 Mini)	$71.77$	$62.90$	$56.45$	$63.71$
GPT-5 Mini	$62.10$	$53.23$	$41.49$	$52.42$

Table 4: HardBLINK Breakdown. We report accuracy (%) on different sub-tasks in HardBLINK [ 1 ] : 3 point, 4 point and 5 point. Each setting is increasingly difficult from the prior, wherein more candidate points are presented to the MLLM.

Figure 7: Average Tokens/Sample. Comparison of Visual Sketchpad (with GPT-5 Mini as LLM) and GPT-5 Mini with P 2 on average token per sample across all six sub-tasks. P 2 incurs significantly lower token cost.

Figure 8: Open-Source Prompt with P 2 ICL. We present a sample prompt for open-source MLLMs (e.g., Qwen3VL and InternVL3.5). We include a single in-context example describing the use of P 2 . Both Qwen3VL and InternVL3.5 reason with the given P 2 to compute the correct answer (A) to the question.

Figure 9: Open-Source Prompt with Tool ICL. We present a sample prompt for open-source MLLMs (e.g., Qwen3VL and InternVL3.5). We include a single in-context example describing the use of optical flow as tool output. Note how the example clearly illustrates that blue hues indicate left while warm hues indicate right motion, the MLLM (Qwen3VL in this example) concludes the same that flow is dominated by blue hues, yet gives the wrong answer (B) . Note that MLLM also uses a lot more tokens than its P 2 counterpart (exhausting almost the entire $8192$ token budget), we use …for brevity purposes in this illustrative figure.

Figure 10: Correspondence Distribution. Illustration of distribution of correspondence markers in the visual correspondence task from BLINK validation set.

Explore similar work

May 25, 2026

Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning

Recent multimodal large language models (MLLMs) achieve strong performance on visual reasoning benchmarks, yet it remains unclear to what extent such performance reflects reasoning directly grounded in visual evidence. We introduce VisReason, a benchmark for vision-centric reasoning in everyday scenarios where perception and inference are tightly coupled. VisReason contains 1,505 questions across 10 categories spanning perceptual, structural, and conceptual reasoning. Our evaluation shows that VisReason poses a qualitatively different challenge from existing benchmarks, exposing substantial gaps between humans and current MLLMs and revealing limited benefits from test-time reasoning strategies. VisReason offers a focused diagnostic for evaluating vision-centric reasoning beyond language.

Longteng Guo, Yifan Wang, Pengkang Huo +4

Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences

Visual Reasoning VLM Reasoning Visually Grounded Reasoning Multimodal Model Evaluation Multimodal Understanding

Ask PDF

Jul 27, 2026

PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models

We introduce PerceptionBench, a benchmark specifically designed to evaluate the atomic visual perception capabilities of Multimodal Large Language Models (MLLMs). Existing benchmarks often fail to isolate perception: holistic evaluations conflate perceptual errors with failures in reasoning or domain knowledge, while application-driven benchmarks only cover narrow, fragmented domains shaped by heuristic designs. To address these limitations, PerceptionBench adopts a bottom-up approach: by diagnosing the earliest failure points in the responses of frontier MLLMs across 42 existing benchmarks, we construct an error taxonomy whose perception branch defines ten atomic perceptual capabilities. Guided by this taxonomy, we construct 3,000 verified questions with short, unambiguous answers, each isolating a single capability, with difficulty stemming from perception rather than reasoning or knowledge. Benchmark results across sixteen frontier MLLMs reveal that atomic perception remains largely unsolved---no model reaches 60% accuracy, perception-related hallucination is the weakest capability on average, and similar overall scores conceal sharply divergent capability profiles. PerceptionBench thus provides a capability-level standard for measuring and diagnosing the visual perception boundaries of MLLMs.

Zichao Lin, Yifeng Xie, Bowen Qu +30

Moonshot AI

Object Hallucination in VLMs Multimodal Large Language Models Multimodal Model Evaluation Fine-Grained Visual Perception

Ask PDF

Jul 21, 2025

Position: Reasoning After Perception Means Reasoning Without Vision

A common belief in multimodal research is that the perceptual weaknesses of vision--language models can be compensated by stronger language reasoning (e.g., chain-of-thought, in-context learning, or external tools). We challenge this assumption. We argue that for a broad class of visual tasks hard to specify in language, failures stem from a structural fatality where the temporal decision of when to reason strictly dictates the spatial constraint of where reasoning takes place. When visual reasoning is deferred to language generation, current architectures do not merely delay computation; they displace it from the continuous visual representation to a discrete textual space. Consequently, the sequential

Perception-then-Reasoning'' paradigm degenerates perception into a passive, one-off feature encoding process, rendering it functionally equivalent to

Reasoning-in-Text-Space'', where task-critical spatial signals are collapsed before reasoning begins. We substantiate this claim with the Turing Eye Test (TET): tasks that must be resolved in visual space and are hard to verbalize; results show text-only reasoning cannot remedy these perceptual failures. Our findings suggest rethinking the architectural divide: shifting from reasoning about perception to reasoning within perception. This facilitates actively reasoning-driven perception that operates directly on pixel-level visual representations, rather than within a collapsed textual space.

Hongcheng Gao, Zihao Huang, Jingyi Tang +12

Tsinghua University · BIT · Peking University +6

Visual Reasoning VLM Reasoning Visually Grounded Reasoning Multimodal Foundation Models Active Visual Perception Multimodal Understanding

Ask PDF