What Sketches Tell Us about LVLMs: Conventions, Grounding, and Localisation

Chaitat Utintu; Ahmed Bourouis; Arkaprabha Basu; Yi-Zhe Song

cs.CVOct 25, 2026

What Sketches Tell Us about LVLMs: Conventions, Grounding, and Localisation

Authors: Chaitat Utintu, Ahmed Bourouis, Arkaprabha Basu, Yi-Zhe Song

Organizations: SketchX, Surrey Institute for People-Centred Artificial Intelligence (PAI), University of Surrey · UKRI AI: CDT in AI for Digital Media Inclusion

Abstract

LVLM benchmarks tell us whether a model answers correctly, but not which visual conventions it has retained, nor where those conventions live. Sketches expose this hidden structure. A sketch removes texture, lighting, and photographic context while preserving the strokes humans judged sufficient for recognition. If an LVLM can operate on that abstraction, the surviving signal is structural rather than merely photometric. We introduce a training-free sketch probe with three readouts: temporal part-construction order, spatial attention-based grounding, and the mechanistic layer/head locus of grounding. We evaluate nine LVLMs on part order and five on the attention readouts, over two part-annotated sketch datasets. Three findings emerge. First, LVLM part order tracks human convention: models reproduce canonical orderings where human drawings converge, and become correspondingly variable where human orderings vary. Controls that blank, withhold, and mismatch the sketch locate this convention in the language model. Second, part-level grounding survives the abstraction: attention heads selected from the frozen model with an existing recipe recover part and object regions on object and scene sketches, with no grounding supervision. Third, the grounding signal concentrates in a narrow mid-network band whose location follows the base language model rather than parameter count in the two families we compare, and remains stable across object and scene sketches. The result is a practical diagnostic for sketch-driven LVLMs: no fine-tuning, no auxiliary segmenter, and a layer band that doubles as a head-selection prior, matching or improving grounding [email protected] in every model and dataset we tested.

Figures & tables

Figure 1: Sketches as a probe for LVLMs. Sketches keep only the strokes humans considered essential, exposing what an LVLM treats as essential. We probe this from three complementary perspectives. Left : temporal part-construction order on two object sketches (early-to-late gradient, predicted ordering beneath) against human consensus. Right : spatial attention-based grounding on one object and one scene sketch, with per-part / per-object overlays beside the human sketch.

Figure 2: Pilot: part-construction ordering and grounding across abstraction. Left : human (GT) and LVLM part-drawing trajectories on three SPG Li et al. (2018) categories (early $\rightarrow$ late coloured), with mean per-sketch part-order similarity ( $\pm$ SD, $N=100$ ) across ten categories. Right : LVLM part-localisation on Pascal-Part Chen et al. (2014) across ControlSketch Arar et al. (2025) abstraction levels, top-3 grounding-head selection (peak shifts to earlier layers), and degrading pointing accuracy and mAcc@0.1 with stable spatial entropy. See Sec. 3.1 . — Figure 2: Pilot: part-construction ordering and grounding across abstraction. Left : human (GT) and LVLM part-drawing trajectories on three SPG Li et al. (2018) categories (early $\rightarrow$ late coloured), with mean per-sketch part-order similarity ( $\pm$ SD, $N=100$ ) across ten categories. Right : LVLM part-localisation on Pascal-Part Chen et al. (2014) across ControlSketch Arar et al. (2025) abstraction levels, top-3 grounding-head selection (peak shifts to earlier layers), and degrading pointing accuracy and [email protected] with stable spatial entropy. See Sec. 3.1 .

Figure 3: Three-perspective probing framework. The same frozen LVLM is queried twice on a single sketch under two protocols. Left (temporal probing): the model receives the sketch with a shuffled set of parts and is asked to return them in drawing order. Right (spatial and mechanistic probing): the model is asked to locate a named part, and attention is filtered to the top- $k$ grounding heads and max-pooled into a grounding mask.

Figure 4: Human consensus statistics across all SPG Li et al. (2018) sketch categories.

Closed-source LVLMs
Model	SPG Li et al. (2018)
Model	Edit $\uparrow$	LCS $\uparrow$	Pair $\uparrow$
GPT-5 (nano) OpenAI (2025)	0.72 ( $\pm$ 0.26)	0.85 ( $\pm$ 0.14)	0.81 ( $\pm$ 0.19)
Claude Haiku 4.5 Anthropic (2025)	0.80 ( $\pm$ 0.20)	0.89 ( $\pm$ 0.11)	0.88 ( $\pm$ 0.13)
Gemini 2.5 Flash-Lite Comanici et al. (2025)	0.73 ( $\pm$ 0.25)	0.85 ( $\pm$ 0.14)	0.83 ( $\pm$ 0.17)
Open-source LVLMs

Table 1: Part-construction order on object sketches. LVLM trajectories

\pi(p)

compared against human trajectories

\pi^{\text{human}}

on SPG Li et al. (2018) under the class-consensus protocol, evaluated with edit distance Levenshtein (1966) , longest common subsequence Hirschberg (1975) , and pairwise-order agreement (Kendall’s

\tau

) Kendall (1938) . Best per column is highlighted within closed-source ( purple ) and open-source ( blue ).

Trained grounders
Model	Object, SPG Li et al. (2018)			Scene, FS-COCO Chowdhury et al. (2022)
Model	Point $\uparrow$	[email protected] $\uparrow$	SE $\downarrow$	Point $\uparrow$	[email protected] $\uparrow$	SE $\downarrow$
CLIPSeg Lüddecke and Ecker (2022)	0.56	0.27	0.96	0.67	0.40	0.96
GroupViT Xu et al. (2022)	0.34	0.59	0.98	0.39	0.60	0.97
Training-free LVLM probes
ShareGPT4V-7B Chen et al. (2024a)	0.42	0.72	0.87	0.45	0.72	0.86

Table 2: Attention-based grounding on object and scene sketches. Cross-attention grounding accuracy of LVLM grounding heads on object (SPG) and scene (FS-COCO) sketches, under pointing accuracy Zhang et al. (2016) , [email protected] Bilen and Vedaldi (2016) , and spatial entropy Riche et al. (2013) . Best per column is highlighted within trained grounders ( purple , shown for reference) and LVLMs ( blue ).

Table 7

Figure 5: Summary of ordering and grounding behaviour. Left : per-class human-LVLM ordering agreement under LCS Hirschberg (1975) , with viridis colour encoding LVLM agreement. Right : LVLM grounding accuracy as a function of input complexity, on object (SPG, parts) and scene (FS-COCO, objects).

Figure 6: Mechanistic profile of LVLM grounding. Top : layer distribution of the top-3 grounding heads across the five models, SPG (object) vs FS-COCO (scene). Bottom-left : distribution of combined top-3 spatial entropy, pooled across models and datasets, with low/high SE tails marked. Bottom-right : LLaVA-1.5-13B attention overlays on samples drawn from each SE tail.

Model	SPG Li et al. (2018)	FS-COCO Chowdhury et al. (2022)
Model	$\Delta$ (full $\rightarrow$ band)	$\Delta$ (full $\rightarrow$ band)
LLaVA-1.5-7B	+0.08 (0.65 $\rightarrow$ 0.73 )	+0.09 (0.69 $\rightarrow$ 0.78 )
LLaVA-1.5-13B	+0.01 (0.72 $\rightarrow$ 0.73 )	+0.03 (0.78 $\rightarrow$ 0.81 )
ShareGPT4V-7B	+0.04 (0.73 $\rightarrow$ 0.76 )	+0.05 (0.73 $\rightarrow$ 0.78 )
DeepSeek-VL-1.3B	+0.04 (0.56 $\rightarrow$ 0.60 )	+0.04 (0.61 $\rightarrow$ 0.65 )
DeepSeek-VL-7B	+0.01 (0.48 $\rightarrow$ 0.49 )	+0.04 (0.65 $\rightarrow$ 0.69 )

Table 10

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Pearson $r$	$p$	Spearman $\rho$
Gemini 2.5 Flash-Lite Comanici et al. (2025)	$+0.82$	$<0.001$	$+0.77$
Claude Haiku 4.5 Anthropic (2025)	$+0.77$	$<0.001$	$+0.80$
Qwen2.5-VL-7B Bai et al. (2025b)	$+0.71$	$<0.001$	$+0.66$
Molmo-7B-D Deitke et al. (2025)	$+0.70$	$<0.001$	$+0.64$
GPT-5 (nano) OpenAI (2025)	$+0.61$	$0.001$	$+0.64$
InternVL2-8B Chen et al. (2024b)	$+0.60$	$0.002$	$+0.59$

Appendix

Table 5: Human consensus against LVLM agreement. Correlation between per-category human consensus (Fig. 4 ) and per-category LCS with the human trajectory, over the 25 SPG categories.

	SPG Li et al. (2018)			FS-COCO Chowdhury et al. (2022)
Model	@0.1 $\uparrow$	@0.3 $\uparrow$	@0.5 $\uparrow$	@0.1 $\uparrow$	@0.3 $\uparrow$	@0.5 $\uparrow$
CLIPSeg Lüddecke and Ecker (2022)	0.27	0.17	0.12	0.40	0.31	0.20
GroupViT Xu et al. (2022)	0.58	0.28	0.11	0.60	0.21	0.08
LLaVA-1.5-13B Liu et al. (2024a)	0.72	0.29	0.06	0.78	0.35	0.11

Appendix

Table 6: Grounding at three overlap thresholds. All methods re-scored from one run.

Model	Depth, SPG	Depth, FS-COCO	Shared top-10	Frequency $r$
ShareGPT4V-7B Chen et al. (2024a)	0.355	0.355	6/10	$+0.72$
LLaVA-1.5-13B Liu et al. (2024a)	0.410	0.410	5/10	$+0.74$
LLaVA-1.5-7B Liu et al. (2024a)	0.452	0.452	5/10	$+0.66$
DeepSeek-VL-7B Lu et al. (2024)	0.552	0.552	4/10	$+0.70$
DeepSeek-VL-1.3B Lu et al. (2024)	0.609	0.609	7/10	$+0.87$
Mean			5.4/10	$+0.74$

Appendix

Table 7: Grounding-head overlap between object and scene sketches. Depth is the modal fractional depth of top-3 selections.

Ablated heads	Side accuracy $\uparrow$	Margin $\uparrow$	Change in margin
None	0.701	0.854
Top-3 grounding heads	0.713	0.736	$\mathbf{-0.118}$ ( $p=0.0001$ )
Random 3 heads	0.701	0.854	$-0.001$
3 heads, same layers	0.695	0.844	$-0.010$

Appendix

Table 8: Head ablation in LLaVA-1.5-13B. Side accuracy and logit margin towards the correct side.

Model	Edit $\uparrow$	LCS $\uparrow$	Pair $\uparrow$
Qwen2.5-VL-3B (Tab. 1 )	0.57	0.77	0.64
Qwen3-VL-4B	0.73	0.85	0.84
Qwen2.5-VL-7B (Tab. 1 )	0.69	0.84	0.80
Qwen3-VL-8B	0.72	0.85	0.83
Claude Haiku 4.5 (Tab. 1 )	0.80	0.89	0.88

Appendix

Table 9: Part-construction order for Qwen3-VL on SPG, protocol as in Tab. 1 .

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 734 papers

SketchVLM: Vision language models can annotate images to explain thoughts and guide users
Apr 23, 2026Brandon Collins, Logan Bolton, Hung Huy Nguyen +3Vision-Language Models LLM-Assisted Annotation
Auburn University · Adobe Research
When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs
Aug 25, 2026Zhengxiang Wang, Owen RambowVLM Evaluation Large Vision-Language Models
Department of Linguistics & IACS Stony Brook University
LASA: A Weak Supervision Method for Open-Vocabulary Scene Sketch Semantic Segmentation
Jun 10, 2026Liwen Yi, Xianlin Zhang, Yue Zhang +2Weakly Supervised Image Segmentation Cross-Layer Attention
Beijing University of Posts and Telecommunications Beijing, China

Apr 23, 2026J/K move · Enter open · S save

SketchVLM: Vision language models can annotate images to explain thoughts and guide users

Brandon Collins, Logan Bolton, Hung Huy Nguyen, Mohammad Reza Taesiri+2

Auburn University · Adobe Research

Vision-Language Models LLM-Assisted Annotation VLM Reasoning Visually Grounded Reasoning Multimodal Understanding

Open paper AskPDF arXiv

When answering questions about images, humans naturally point, label, and draw to explain their reasoning. In contrast, modern vision-language models (VLMs) such as Gemini-3-Pro and GPT-5 only respond with text, which can be difficult for users to verify. We present SketchVLM, a training-free, model-agnostic framework that enables VLMs to produce non-destructive, editable SVG overlays on the input image to visually explain their answers. Across seven benchmarks spanning visual reasoning (maze navigation, ball-drop trajectory prediction, and object counting) and drawing (part labeling, connecting-the-dots, and drawing shapes around objects), SketchVLM improves visual reasoning task accuracy by up to +28.5 percentage points and annotation quality by up to 1.48x relative to image-editing and fine-tuned sketching baselines, while also producing annotations that are more faithful to the model's stated answer. We find that single-turn generation already achieves strong accuracy and annotation quality, and multi-turn generation opens up further opportunities for human-AI collaboration. An interactive demo and code are at https://sketchvlm.github.io/.

Abstract

Vision-Language Models

Explore similar work

SketchVLM: Vision language models can annotate images to explain thoughts and guide users

When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs

LASA: A Weak Supervision Method for Open-Vocabulary Scene Sketch Semantic Segmentation