What Sketches Tell Us about LVLMs: Conventions, Grounding, and Localisation
Authors: Chaitat Utintu, Ahmed Bourouis, Arkaprabha Basu, Yi-Zhe Song
Organizations: SketchX, Surrey Institute for People-Centred Artificial Intelligence (PAI), University of Surrey · UKRI AI: CDT in AI for Digital Media Inclusion
LVLM benchmarks tell us whether a model answers correctly, but not which visual conventions it has retained, nor where those conventions live. Sketches expose this hidden structure. A sketch removes texture, lighting, and photographic context while preserving the strokes humans judged sufficient for recognition. If an LVLM can operate on that abstraction, the surviving signal is structural rather than merely photometric. We introduce a training-free sketch probe with three readouts: temporal part-construction order, spatial attention-based grounding, and the mechanistic layer/head locus of grounding. We evaluate nine LVLMs on part order and five on the attention readouts, over two part-annotated sketch datasets. Three findings emerge. First, LVLM part order tracks human convention: models reproduce canonical orderings where human drawings converge, and become correspondingly variable where human orderings vary. Controls that blank, withhold, and mismatch the sketch locate this convention in the language model. Second, part-level grounding survives the abstraction: attention heads selected from the frozen model with an existing recipe recover part and object regions on object and scene sketches, with no grounding supervision. Third, the grounding signal concentrates in a narrow mid-network band whose location follows the base language model rather than parameter count in the two families we compare, and remains stable across object and scene sketches. The result is a practical diagnostic for sketch-driven LVLMs: no fine-tuning, no auxiliary segmenter, and a layer band that doubles as a head-selection prior, matching or improving grounding [email protected] in every model and dataset we tested.
Figures & tables
Figure 1: Sketches as a probe for LVLMs. Sketches keep only the strokes humans considered essential, exposing what an LVLM treats as essential. We probe this from three complementary perspectives. Left : temporal part-construction order on two object sketches (early-to-late gradient, predicted ordering beneath) against human consensus. Right : spatial attention-based grounding on one object and one scene sketch, with per-part / per-object overlays beside the human sketch.
Figure 2: Pilot: part-construction ordering and grounding across abstraction. Left : human (GT) and LVLM part-drawing trajectories on three SPG Li et al. (2018) categories (early → late coloured), with mean per-sketch part-order similarity ( ± SD, N=100 ) across ten categories. Right : LVLM part-localisation on Pascal-Part Chen et al. (2014) across ControlSketch Arar et al. (2025) abstraction levels, top-3 grounding-head selection (peak shifts to earlier layers), and degrading pointing accuracy and [email protected] with stable spatial entropy. See Sec. 3.1 .
Figure 3: Three-perspective probing framework. The same frozen LVLM is queried twice on a single sketch under two protocols. Left (temporal probing): the model receives the sketch with a shuffled set of parts and is asked to return them in drawing order. Right (spatial and mechanistic probing): the model is asked to locate a named part, and attention is filtered to the top- k grounding heads and max-pooled into a grounding mask.
Figure 4: Human consensus statistics across all SPG Li et al. (2018) sketch categories.
Model
SPG Li et al. (2018)
Edit ↑
LCS ↑
Pair ↑
Closed-source LVLMs
GPT-5 (nano) OpenAI (2025)
0.72 ( ± 0.26)
0.85 ( ± 0.14)
0.81 ( ± 0.19)
Claude Haiku 4.5 Anthropic (2025)
0.80 ( ± 0.20)
0.89 ( ± 0.11)
0.88 ( ± 0.13)
Gemini 2.5 Flash-Lite Comanici et al. (2025)
0.73 ( ± 0.25)
0.85 ( ± 0.14)
0.83 ( ± 0.17)
Open-source LVLMs
Table 1: Part-construction order on object sketches. LVLM trajectories π(p) compared against human trajectories πhuman on SPG Li et al. (2018) under the class-consensus protocol, evaluated with edit distance Levenshtein (1966) , longest common subsequence Hirschberg (1975) , and pairwise-order agreement (Kendall’s τ ) Kendall (1938) . Best per column is highlighted within closed-source ( purple ) and open-source ( blue ).
Table 2: Attention-based grounding on object and scene sketches. Cross-attention grounding accuracy of LVLM grounding heads on object (SPG) and scene (FS-COCO) sketches, under pointing accuracy Zhang et al. (2016) , [email protected] Bilen and Vedaldi (2016) , and spatial entropy Riche et al. (2013) . Best per column is highlighted within trained grounders ( purple , shown for reference) and LVLMs ( blue ).
Table 7
Figure 5: Summary of ordering and grounding behaviour. Left : per-class human-LVLM ordering agreement under LCS Hirschberg (1975) , with viridis colour encoding LVLM agreement. Right : LVLM grounding accuracy as a function of input complexity, on object (SPG, parts) and scene (FS-COCO, objects).
Figure 6: Mechanistic profile of LVLM grounding. Top : layer distribution of the top-3 grounding heads across the five models, SPG (object) vs FS-COCO (scene). Bottom-left : distribution of combined top-3 spatial entropy, pooled across models and datasets, with low/high SE tails marked. Bottom-right : LLaVA-1.5-13B attention overlays on samples drawn from each SE tail.
Model
SPG Li et al. (2018)
FS-COCO Chowdhury et al. (2022)
Δ (full → band)
Δ (full → band)
LLaVA-1.5-7B
+0.08 (0.65 → 0.73 )
+0.09 (0.69 → 0.78 )
LLaVA-1.5-13B
+0.01 (0.72 → 0.73 )
+0.03 (0.78 → 0.81 )
ShareGPT4V-7B
+0.04 (0.73 → 0.76 )
+0.05 (0.73 → 0.78 )
DeepSeek-VL-1.3B
+0.04 (0.56 → 0.60 )
+0.04 (0.61 → 0.65 )
DeepSeek-VL-7B
+0.01 (0.48 → 0.49 )
+0.04 (0.65 → 0.69 )
Table 10
Figure 11
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Pearson r
p
Spearman ρ
Gemini 2.5 Flash-Lite Comanici et al. (2025)
+0.82
<0.001
+0.77
Claude Haiku 4.5 Anthropic (2025)
+0.77
<0.001
+0.80
Qwen2.5-VL-7B Bai et al. (2025b)
+0.71
<0.001
+0.66
Molmo-7B-D Deitke et al. (2025)
+0.70
<0.001
+0.64
GPT-5 (nano) OpenAI (2025)
+0.61
0.001
+0.64
InternVL2-8B Chen et al. (2024b)
+0.60
0.002
+0.59
Appendix
Table 5: Human consensus against LVLM agreement. Correlation between per-category human consensus (Fig. 4 ) and per-category LCS with the human trajectory, over the 25 SPG categories.
SPG Li et al. (2018)
FS-COCO Chowdhury et al. (2022)
Model
@0.1 ↑
@0.3 ↑
@0.5 ↑
@0.1 ↑
@0.3 ↑
@0.5 ↑
CLIPSeg Lüddecke and Ecker (2022)
0.27
0.17
0.12
0.40
0.31
0.20
GroupViT Xu et al. (2022)
0.58
0.28
0.11
0.60
0.21
0.08
LLaVA-1.5-13B Liu et al. (2024a)
0.72
0.29
0.06
0.78
0.35
0.11
Appendix
Table 6: Grounding at three overlap thresholds. All methods re-scored from one run.
Model
Depth, SPG
Depth, FS-COCO
Shared top-10
Frequency r
ShareGPT4V-7B Chen et al. (2024a)
0.355
0.355
6/10
+0.72
LLaVA-1.5-13B Liu et al. (2024a)
0.410
0.410
5/10
+0.74
LLaVA-1.5-7B Liu et al. (2024a)
0.452
0.452
5/10
+0.66
DeepSeek-VL-7B Lu et al. (2024)
0.552
0.552
4/10
+0.70
DeepSeek-VL-1.3B Lu et al. (2024)
0.609
0.609
7/10
+0.87
Mean
5.4/10
+0.74
Appendix
Table 7: Grounding-head overlap between object and scene sketches. Depth is the modal fractional depth of top-3 selections.
Ablated heads
Side accuracy ↑
Margin ↑
Change in margin
None
0.701
0.854
Top-3 grounding heads
0.713
0.736
−0.118 ( p=0.0001 )
Random 3 heads
0.701
0.854
−0.001
3 heads, same layers
0.695
0.844
−0.010
Appendix
Table 8: Head ablation in LLaVA-1.5-13B. Side accuracy and logit margin towards the correct side.
Model
Edit ↑
LCS ↑
Pair ↑
Qwen2.5-VL-3B (Tab. 1 )
0.57
0.77
0.64
Qwen3-VL-4B
0.73
0.85
0.84
Qwen2.5-VL-7B (Tab. 1 )
0.69
0.84
0.80
Qwen3-VL-8B
0.72
0.85
0.83
Claude Haiku 4.5 (Tab. 1 )
0.80
0.89
0.88
Appendix
Table 9: Part-construction order for Qwen3-VL on SPG, protocol as in Tab. 1 .
When answering questions about images, humans naturally point, label, and draw to explain their reasoning. In contrast, modern vision-language models (VLMs) such as Gemini-3-Pro and GPT-5 only respond with text, which can be difficult for users to verify. We present SketchVLM, a training-free, model-agnostic framework that enables VLMs to produce non-destructive, editable SVG overlays on the input image to visually explain their answers. Across seven benchmarks spanning visual reasoning (maze navigation, ball-drop trajectory prediction, and object counting) and drawing (part labeling, connecting-the-dots, and drawing shapes around objects), SketchVLM improves visual reasoning task accuracy by up to +28.5 percentage points and annotation quality by up to 1.48x relative to image-editing and fine-tuned sketching baselines, while also producing annotations that are more faithful to the model's stated answer. We find that single-turn generation already achieves strong accuracy and annotation quality, and multi-turn generation opens up further opportunities for human-AI collaboration. An interactive demo and code are at https://sketchvlm.github.io/.
Visual grounding is typically evaluated as a one-shot mapping from an informative referring expression to a visual target. This formulation misses a central property of real-world reference: initial referring expressions are often incomplete or ambiguous, requiring participants to establish shared understanding through interaction. We introduce a controlled evaluation framework for interactive visual grounding in large vision-language models (LVLMs), varying how much target information is provided upfront and how much must be acquired through dialogue. Across four human-grounded visual contexts and four interaction protocols, current LVLMs perform significantly below task-level human baselines. Interaction can help when follow-up questions refine or repair an initial target description. Performance is lowest when no initial description is provided and target information must be acquired through questions, indicating that proactive question-driven grounding remains difficult. LVLMs are also poorly calibrated, often reporting confidence that exceeds their empirical accuracy. Follow-up studies confirm these patterns across varied description sources (human versus AI), reasoning efforts, repeated interactions, description providers, and visual contexts. Overall, interactive visual grounding remains challenging, requiring visual matching, information seeking and synthesis.
Zhengxiang Wang, Owen Rambow
Department of Linguistics & IACS Stony Brook University
Open-vocabulary scene sketch semantic segmentation aims to assign dense semantic labels to sparse line drawings based on flexible category vocabularies specified at inference time, without relying on pixel-level annotations during training. Unlike natural images, sketches lack texture and color cues, making semantic understanding heavily dependent on stroke layout and spatial configuration, a challenge that renders single-layer vision-language features inherently unstable. Our key observation is that attention maps from different Vision Transformer layers encode complementary spatial cues: shallow layers capture global structural layouts, while deeper layers focus on local stroke intersections and object parts. This suggests that cross-layer aggregation provides a more robust structural prior than any individual layer alone. Leveraging this insight, we propose a structure-aware framework built upon Layer-wise Accumulated Structural Attention (LASA), which aggregates multi-layer attention to guide hierarchical semantic alignment under weak supervision and refine predictions during inference. Experiments on FS-COCO, SFSD, and FrISS show that LASA improves mIoU by +3.43, +8.01, and +15.74 over the prior weakly supervised baselines, demonstrating consistent gains in both segmentation accuracy and spatial coherence. Our source code will be made publicly available.
Liwen Yi, Xianlin Zhang, Yue Zhang +2
Beijing University of Posts and Telecommunications Beijing, China