SEA: Evaluating Sketch Abstraction Efficiency via Element-level Commonsense Visual Question Answering

Jiho Park; Sieun Choi; Jaeyoon Seo; Minho Sohn; Yeana Kim; Jihie Kim

Figure 1: Overview of SEA and CommonSketch. Left: SEA quantifies abstraction efficiency by balancing recognizability and detail. High scores (top-left) favor simple yet identifiable sketches, while low scores (bottom-right) denote ambiguity or over-detail. Right: CommonSketch includes element-level annotations and captions, enabling element-aware evaluation of sketch abstraction.

Dataset	# Classes	# Sketches/ Class	Total # Sketches	Common Sense	Caption	QA
TU-Berlin [ 10 ]	250	80	20K	✗	✗	✗
Sketchy [ 43 ]	125	avg. 600	75K	✗	✗	✗
QuickDraw [ 16 ]	345	avg. 144K	$\sim$ 50M	✗	✗	✗
SEVA [ 38 ]	128	avg. 703	90K	✗	✗	✗
CommonSketch (ours)	300	avg. 77	23K	✓	✓	✓

Table 1: Comparison of prior sketch datasets and ours. This table summarizes key attributes of several sketch datasets: TU-Berlin, Sketchy, QuickDraw, SEVA, and ours.

Figure 2: CommonSketch Overview. 23,100 human-drawn sketches with paired captions and element-level commonsense across 300 classes in 14 categories; (a) construction/annotation pipeline, (b) category-wise element statistics, (c) class distribution with an example.

Figure 3: Computation pipeline and case-based interpretation of the SEA metric. Given a sketch and its class label, SEA combines class recognizability

P

from a classifier with the commonsense element space

E

extracted by an LLM and the number of visually grounded elements

V

identified by a VLM. It then computes a reward–penalty balance and maps it to a bounded score

SEA\in(-1,1)

, where higher scores indicate sketches that preserve recognizability with minimal yet sufficient visual detail. Illustrative cases show abstraction failure due to low recognizability (left), incomplete abstraction caused by excessive detail (middle), and abstraction-efficient sketching that achieves high recognizability with fewer expressed elements (right).

Figure 4: Qualitative comparison of SEA scores across abstraction levels (4, 8, 16, 32) on four classes shared by SEVA and CommonSketch — Baseball (top left), Hat (bottom left), Giraffe (top right), and Guitar (bottom right). Each example reports the SEA score with visual ratio

v

, and prediction probability

P

, showing how abstraction can improve efficiency when recognizability is preserved.

Metric	Level 4	Level 8	Level 16	Level 32
SEA	-0.56 $\pm$ 0.32	-0.23 $\pm$ 0.30	0.29 $\pm$ 0.24	0.43 $\pm$ 0.29
reward( $P$ , $v$ )	0.16 $\pm$ 0.20	0.30 $\pm$ 0.20	0.52 $\pm$ 0.20	0.57 $\pm$ 0.26
penalty( $P$ , $v$ )	0.53 $\pm$ 0.11	0.41 $\pm$ 0.09	0.24 $\pm$ 0.09	0.18 $\pm$ 0.11
visual ratio ( $v$ )	0.22 $\pm$ 0.06	0.31 $\pm$ 0.08	0.40 $\pm$ 0.08	0.45 $\pm$ 0.10
prediction ( $P$ )	0.17 $\pm$ 0.17	0.37 $\pm$ 0.15	0.64 $\pm$ 0.11	0.75 $\pm$ 0.11

Table 2: Quantitative SEA results on SEVA across abstraction levels. Mean

\pm

standard deviation of SEA, reward, penalty, visual ratio

v

, and prediction probability

P

.

Figure 5: Distribution of SEA scores across abstraction levels in SEVA. Sketches at lower abstraction levels (4, 8) are concentrated near low SEA scores, whereas those at higher abstraction levels (16, 32) exhibit a noticeable shift toward higher SEA scores.

Model	Precision $\uparrow$	Recall $\uparrow$	F1 Score $\uparrow$	Accuracy $\uparrow$
Open-source VLMs
LLaVA [ 35 ]	0.749	0.819	0.782	0.706
BLIP [ 32 ]	0.731	0.760	0.746	0.666
Molmo [ 8 ]	0.798	0.949	0.867	0.812
Qwen2.5-VL [ 4 ]	0.898	0.782	0.836	0.802
mPLUG-Owl3 [ 53 ]	0.883	0.686	0.772	0.739

Table 3: Comparison of open-source VLMs and a proprietary VLM on element-level commonsense VQA. We report Precision, Recall, F1, and Accuracy. Best values are shown in bold, and second-best values are highlighted in gray.

Model	Soft F1 $\uparrow$	CLIP Score $\uparrow$	BERT Score $\uparrow$
GPT-OSS 20B [ 1 ]	0.850	0.941	0.813
Qwen2.5 32B [ 50 ]	0.833	0.930	0.797
Llama 3 8B [ 15 ]	0.828	0.935	0.802
Mistral 7B [ 25 ]	0.834	0.933	0.804

Table 4: Quantitative results of commonsense element extraction quality. Open-source models are compared to GPT-4o.

Figure 6: Consistency and Human Alignment of SEA Scores. The heatmaps show Spearman (left) and Pearson (right) correlations across different model configurations.

Metric	LLM	VLM	Agreement (%)
SEA	GPT-4o	GPT-4o	87.8
OpenSEA	GPT-OSS	Qwen2.5-VL	88.0

Table 5: Human agreement with the rank-ordering of SEA scores. Here, SEA denotes the closed-source pipeline, while OpenSEA denotes the open-source pipeline. We report participant agreement with the score ordering from each pipeline (

N=27

).

Figure 7: Comparison of sketch quality across datasets. We compare CommonSketch with QuickDraw and TU-Berlin using the distributions of SEA scores (left) and predicted probabilities for the ground-truth class (right). In the probability distribution, CommonSketch has

\mu

=0.86,

\sigma

=0.24, and mode=0.97, while QuickDraw has

\mu

=0.29,

\sigma

=0.37, and mode=0.02, and TU-Berlin has

\mu

=0.62,

\sigma

=0.41, and mode=0.96.

Figure 8: Visualization of abstraction score distributions on CommonSketch VQA , comparing human judgments with SEA (GPT-4o, Qwen2.5-VL); bars show the proportion of samples in four equal-width score bins (Q1–Q4).

Figure 9: Comparison of SEA and human scores for SEVA-only sketches. Top row shows mosquito sketches and bottom row tank sketches.

Category	#Classes	Classes
animal	61	alpaca, ant, bat, bee, bird, boar, butterfly, camel, cat, caterpillar, chameleon, cow, crab, crocodile, deer, dog, dolphin, dragonfly, duck, elephant, feather, fish, flamingo, frog, giraffe, goose, hedgehog, hippopotamus, horse, jellyfish, kangaroo, koala, lion, lobster, mole, monkey, moose, mouse, octopus, owl, panda, parrot, peacock, penguin, rabbit, rooster, scorpion, seahorse, shark, sheep, sloth, snail, snake, spider, squid, squirrel, swan, tiger, turtle, whale, zebra
body part	7	ear, eye, foot, hand, mouth, nose, tooth
clothing	8	belt, bowtie, crown, flip flops, hat, shoe, sock, t shirt
container	9	backpack, basket, bucket, envelope, mailbox, present, purse, suitcase, wine bottle
electronic device	22	alarm clock, calculator, camera, cell phone, charger, computer, fan, headphones, ipod, keyboard, laptop, megaphone, microphone, microwave, oven, radio, robot, satellite, telephone, television, toaster, walkie talkie
food	28	apple, asparagus, banana, bread, broccoli, cake, carrot, cookie, cupcake, donut, garlic, grapes, hamburger, hot dog, ice cream cone, lollipop, mushroom, noodle, onion, peanut, pear, pineapple, pizza, pretzel, pumpkin, sandwich, strawberry, watermelon

Table 6: Classes by category. For each category, we provide the class count and the full set of class labels used in the dataset.

Category	Class	4o/OSS	Elements
animal	alpaca	13/11	body, ears, eyes, head, legs, tail, feet , fleece , hooves , muzzle , neck , nostrils , smile , fur_lines , motion_lines , mouth , nose , whisker_lines
	ant	11/9	abdomen, antennae, head, legs, mandibles, thorax, compound eyes , jointed legs , mouth , petiole , stinger , body , eyes , segment_lines
	bat	14/12	body, ears, eyes, head, mouth, nose, tail, wings, feet , fingers , fur , legs , teeth , wing membrane , claws , fur_lines , motion_lines , wing_vein_lines

Table 7: Per-class element list (sample). Sample rows from Tab. 11 are shown to illustrate our per-class commonsense annotations. Elements are listed as extracted: black denotes overlap between GPT-4o and GPT-OSS, red indicates GPT-4o-only elements, and blue indicates GPT-OSS-only elements. The 4o/OSS column reports the number of elements extracted by each model.

Figure 10: Additional sketch examples from CommonSketch. The first three rows are dedicated to the animal category, showing a broad slice of its classes. The fourth row groups representative sketches from body_part , clothing , and container . Each remaining row corresponds to one category ( electronic_device , food , furniture , icon , musical_instrument , nature , sports_equipment , structure , tool , vehicle ) in order.

Category	Rank	Element	$n_{\text{in\_cat}}$	$p_{\text{cat}}$	Lift
animal	1	abdomen	4	0.07	4.92
animal	2	antennae	6	0.10	4.92
animal	3	beak	11	0.18	4.92
container	1	handle	5	0.56	4.76
container	2	body	6	0.67	1.74
electronic device	1	display	4	0.18	13.64

Table 8: Category-wise frequent elements. For each category, we list the three elements with the highest lift. Columns report the per-category class count

n_{\text{in\_cat}}

, within-category coverage

p_{\text{cat}}

, and lift. Only elements with

n_{\text{in\_cat}}\geq 3

are included.

Element	$n_{\text{classes\_global}}$	Rank
body	115	1
head	67	2
legs	59	3
eyes	57	4
tail	52	5
mouth	44	6

Table 9: Global element frequency ranking. We report the 20 elements that appear in the largest number of classes across all 300 classes.

Figure 11: Effect of the gate function in SEA scoring. Two representative cases illustrate how the gate term changes the final SEA score. The gated formulation assigns substantially lower scores in both cases, showing that it penalizes overly detailed or weakly recognizable sketches more strongly.

Figure 12: SEA vs. normalized visual representation. For fixed

E=10

, the SEA score decreases with

v

at low

P

, shows a mild plateau then decline at moderate

P

, and peaks at moderate

v

when

P

is high, illustrating SEA’s preference for efficient abstraction.

Figure 13: Effect of SEA hyperparameters on the score surface: each row varies a single parameter (left: decreased, center: default, right: increased), illustrating how

\alpha

,

\beta

, visual representation efficiency penalties

(\lambda,\eta,k)

, base penalties

(\tau,r)

, and recognizability guidance

\gamma

reshape the

(v,P)

score landscape.

Model	Top-1 Acc	Spearman’s $\rho$	Kendall’s $\tau$	Pearson’s $r$
Human	0.952	–	–	–
CLIP	0.794	0.369	0.284	0.496
OpenCLIP	0.912	0.534	0.435	0.785
CoCa	0.941	0.518	0.426	0.650

Table 10: Comparison of performance on Top-1 accuracy and correlation between Human assessment.

Figure 14: Comparison of SEA scores obtained using GPT-4o and GPT-OSS for commonsense extraction. The top and bottom panels show results with GPT-4o and Qwen as VQA models, respectively. The red dashed line indicates the identity line (

y=x

).

Figure 15: Cross-dataset sketch quality comparison. KDE plots show the Probability of the Ground Truth Class for 14 classes shared across CommonSketch, QuickDraw, and TU-Berlin, with one class chosen from each of the 14 categories. For each sketch, class probability is computed separately with CLIP, OpenCLIP, and CoCa, and the three values are then averaged to obtain a model-agnostic recognizability score. The distributions highlight differences in typical recognizability and variance across datasets, providing a fine-grained view beyond single-number quality metrics. Representative sketches at the mode of each KDE are displayed to link the quantitative trends to visually typical samples.

Figure 16: Qualitative examples by SEA score on SEVA. Six SEVA classes are shown ( baseball , butterfly , giraffe , guitar , hat , snail ), with eight sketches per class selected to span a low-to-high SEA range for visual inspection. All sketches display four SEA variants computed under OpenCLIP as a classifier: GPT4o (GPT-4o database + GPT-4o annotations), Qwen (GPT-4o database + Qwen annotations).

Figure 17: Qualitative examples by SEA score on CommonSketch (set 1). Seven CommonSketch classes are shown ( airplane , basket , bed , bridge , cactus , camera , cat ). All sketches display four SEA variants computed under OpenCLIP as a classifier: GPT4o (GPT-4o database + GPT-4o annotations), Qwen (GPT-4o database + Qwen annotations).

Figure 18: Qualitative examples by SEA score on CommonSketch (set 2). Seven additional CommonSketch classes are shown ( dragon , ear , guitar , helmet , pizza , scissors , sock ). All sketches display four SEA variants computed under OpenCLIP as a classifier: GPT4o (GPT-4o database + GPT-4o annotations), Qwen (GPT-4o database + Qwen annotations).

Figure 19: Comprehensive performance evaluation of nine VLMs. The heatmap visualizes model performance across five key metrics. Higher scores are in blue, and lower scores in red.

Figure 20: Per-category performance comparison of nine VLMs across the 14 categories of CommonSketch. The heatmap visualizes the accuracy scores, where blue indicates higher performance and red indicates lower performance.

Figure 21: Human survey UI snapshots. (a) Landing page with study overview and consent. (b) Classification interface where participants choose among four candidates, with an optional free-response field for alternative labels. (c) Abstraction rating interface, where participants score each sketch on a continuous 0–4 slider with level-specific guidelines.

Figure 22: Comparison of human abstraction score distributions and SEA metric scores across four images per class. Density curves represent individual images, with point markers showing their medians and vertical dashed lines indicating SEA scores. Four images in each class are annotated with their SEA scores and mean human abstraction scores.

Figure 23: Comparison of human scores and SEA values across pictograms to assess generalization ability.

Figure 24: Comparison between classification model performance and human accuracy.

Category	Class	4o/OSS	Elements
animal	alpaca	13/11	body, ears, eyes, head, legs, tail, feet , fleece , hooves , muzzle , neck , nostrils , smile , fur_lines , motion_lines , mouth , nose , whisker_lines
	ant	11/9	abdomen, antennae, head, legs, mandibles, thorax, compound eyes , jointed legs , mouth , petiole , stinger , body , eyes , segment_lines
	bat	14/12	body, ears, eyes, head, mouth, nose, tail, wings, feet , fingers , fur , legs , teeth , wing membrane , claws , fur_lines , motion_lines , wing_vein_lines
	bee	14/10	abdomen, antennae, head, legs, mouth, stinger, thorax, wings, compound eyes , flower context , flying motion , hairs , mouthparts , stripes , body_stripe_lines , eyes
	bird	15/10	beak, body, feet, head, legs, tail, wings, branch , crest , eyes , feathers , flying pose , standing pose , tail feathers , wing feathers , eye , feather_lines , motion_lines
	boar	13/9	body, ears, head, legs, snout, tail, tusks, brush , eyes , hair , hooves , hump , mane , eye , fur_lines

Table 11: Per-class element list. For each class, we list the commonsense elements extracted by GPT-4o and GPT-OSS. Black indicates elements shared by both models; red denotes elements unique to GPT-4o; blue denotes elements unique to GPT-OSS. The 4o/OSS column reports the number of elements extracted by GPT-4o and GPT-OSS 20B, respectively.

SEA: Evaluating Sketch Abstraction Efficiency via Element-level Commonsense Visual Question Answering

Abstract

Figures & tables

Automated Evaluation

What Sketches Tell Us about LVLMs: Conventions, Grounding, and Localisation

SketchGraphNet: A Memory-Efficient Hybrid Graph Transformer for Large-Scale Sketch Corpora Recognition

LASA: A Weak Supervision Method for Open-Vocabulary Scene Sketch Semantic Segmentation

Abstract

Automated Evaluation

Explore similar work

What Sketches Tell Us about LVLMs: Conventions, Grounding, and Localisation

SketchGraphNet: A Memory-Efficient Hybrid Graph Transformer for Large-Scale Sketch Corpora Recognition

LASA: A Weak Supervision Method for Open-Vocabulary Scene Sketch Semantic Segmentation