MYOPIA: Understanding the Limits of Tree-Based Reasoning in Vision-Language Models

Islam Mesabah; Sary Elmansoury; Gerrit Großmann; Peter Neigel; Sebastian J. Vollmer

cs.CVOct 15, 2026

MYOPIA: Understanding the Limits of Tree-Based Reasoning in Vision-Language Models

Authors: Islam Mesabah, Sary Elmansoury, Gerrit Großmann, Peter Neigel, Sebastian J. Vollmer

Organizations: German Research Centre for Artificial Intelligence (DFKI), Kaiserslautern, Germany

Abstract

Decision trees can make vision-language model (VLM) classification interpretable by replacing a single prediction with a sequence of root-to-leaf questions. However, the accuracy cost of this greedy, non-backtracking procedure remains unclear. We introduce MYOPIA, a benchmark comprising four LLM-generated, human-refined decision trees for GTSRB, MNIST, CIFAR-10, and MSCOCO, together with matched prompts, diagnostic protocols, and an evaluation harness. Across GPT-4o, Qwen-VL Max, and LLaMA-3.2 11B Vision, tree traversal underperforms zero-shot prompting in all 12 dataset--model pairs; paired bootstrap analyses consistently favor zero-shot. Our diagnostics characterize interacting failure modes associated with this gap. Models often recall tree-encoded class attributes more accurately than they ground the same attributes in images: the mean label--image accuracy gaps are 9.5, 13.8, and 12.6 points for GPT-4o, Qwen-VL Max, and LLaMA, respectively. Incorrect local decisions then irreversibly exclude the correct class, while answer extraction introduces additional model-dependent loss. A node-level audit of the four trees finds that split entropy is weakly but consistently associated with lower node accuracy, whereas depth, branching factor, and within-node grounding differences are less stable. MYOPIA therefore provides a framework for diagnosing hierarchical visual classification and motivates testing tree designs that prioritize perceptual answerability and recovery from early errors. Code is available at https://github.com/islammesabah/VLM_Classification.

Figures & tables

Figure 1: Overview of the experimental framework, showing how decision trees are constructed and how the same image is classified through direct zero-shot prompting or decision-tree traversal, with one question asked per node.

Dataset	Model	ZS	ZS+D	CoT	Tree	Tree+D
GTSRB	GPT-4o	71.14	63.60	63.00	57.71	65.04
	LLaMA	37.07	23.53	12.00	33.19	14.87
	Qwen-VL	47.06	48.95	36.00	24.64	52.28
MNIST	GPT-4o	91.50	86.40	94.00	34.60	28.60
	LLaMA	59.30	40.20	71.00	7.70	2.60
	Qwen-VL	94.40	89.20	86.00	29.30	27.10

Table 1: Mean accuracy (%) at

T=0.7

. ZS: zero-shot; D: class and image descriptions; CoT: chain-of-thought. Bold and underlined values denote the best and second-best results within each dataset–model row. CoT uses

n=100

images per dataset–model pair.

Dataset	Both+	Both-	Tree+	ZS+
GTSRB	864	1127	177	535
MNIST	673	505	43	1779
CIFAR-10	1333	821	157	689
MSCOCO	677	1005	88	780

Table 2: Instance-level Tree–ZS agreement at

T=0.7

, pooled across models. Both+/Both– denote both correct/incorrect; Tree+/ZS+ denote only Tree/ZS correct.

Dataset	CLIP ZS	Description	CHiLS
GTSRB	18.9	18.5	19.8
MNIST	21.6	32.7	10.8
CIFAR-10	89.0	86.5	79.0
MSCOCO	69.6	66.7	66.9

Table 3: Accuracy (%) of CLIP ViT-B/32 baselines.

	GPT-4o		LLaMA		Qwen-VL
Dataset	L	I	L	I	L	I
GTSRB	79.59	72.81	50.46	49.26	77.49	75.18
MNIST	80.00	63.23	35.83	40.65	51.61	24.00
CIFAR-10	97.50	89.71	100.00	57.71	100.00	82.29
MSCOCO	95.79	89.33	87.74	76.09	94.73	87.14
Mean	88.22	78.77	68.51	55.93	80.96	67.15

Table 4: Label-conditioned accuracy (L) and image-conditioned accuracy (I), with all classes represented.

Depth	GPT-4o	LLaMA	Qwen-VL
0	97.7%	55.3%	93.0%
1	83.5%	47.5%	70.8%
2	57.4%	51.8%	73.5%
3	59.3%	53.3%	88.9%
4	65.9%	42.4%	78.4%
Overall	72.81%	49.26%	75.18%

Table 5: GTSRB image-conditioned accuracy by depth. Rows show depths 0–4; Overall includes all depths 0–14. GPT-4o and LLaMA use 215 images each, and Qwen-VL uses 129.

Dataset	Mean calls	Max calls	Unresolved	Oracle
GTSRB	3.61	15	14.7%	11.0%
MNIST	2.45	4	18.6%	11.1%
CIFAR-10	3.37	4	10.8%	3.7%
MSCOCO	6.32	11	22.0%	16.2%

Table 6: Mean and maximum API calls per tree-classified image and unresolved-path rates before and after oracle re-parsing, pooled across models.

Figure 2: GPT-4o accuracy versus mean VLM calls per image at

T=0.7

; CLIP zero-shot requires no VLM calls. Tree traversal is dominated by zero-shot and CoT on every dataset.

Dataset	Model	$T$	ZS	ZS+D	CoT	Tree	Tree+H	Tree+D
GTSRB	GPT-4o	$0$	70.92	63.82	—	58.05	55.83	66.81
		$0.7$	71.14	63.60	63.00	57.71	56.71	65.04
	LLaMA	$0$	23.86	18.42	—	6.33	16.09	15.65
		$0.7$	37.07	23.53	12.00	33.19	12.43	14.87
	Qwen-VL	$0$	47.95	49.72	—	40.51	36.96	51.83
		$0.7$	47.06	48.95	36.00	24.64	36.96	52.28

Table S1 : Accuracy (%) across temperatures and prompting conditions. ZS: zero-shot; +D: class and image descriptions; +H: conversation history across tree nodes; CoT: chain-of-thought. Superscripts give

n

for incomplete cells, and an em dash indicates no data. CoT uses

n{=}100

class-stratified images per dataset–model pair at

T{=}0.7

. Boldface and underlining identify the best and second-best entries among full-pass conditions and the planned CoT probes; superscripted cells are excluded. Full-pass sizes are 901 (GTSRB), 1,000 (MNIST and CIFAR-10), and 850 (MSCOCO).

Dataset	Model	$n$	ZS	Tree	$\Delta$	95% CI	$p_{\text{Holm}}$
GTSRB	GPT-4o	901	71.14	57.71	$-13.43$	$[-16.43,-10.54]$	$<$ 0.0024
GTSRB	LLaMA	901	37.07	33.19	$-3.88$	$[-7.33,-0.44]$	0.0278
GTSRB	Qwen-VL	901	47.06	24.64	$-22.42$	$[-25.64,-19.20]$	$<$ 0.0024
MNIST	GPT-4o	1000	91.50	34.60	$-56.90$	$[-60.00,-53.70]$	$<$ 0.0024
MNIST	LLaMA	1000	59.30	7.70	$-51.60$	$[-55.00,-48.10]$	$<$ 0.0024
MNIST	Qwen-VL	1000	94.40	29.30	$-65.10$	$[-68.10,-62.10]$	$<$ 0.0024

Table S2 : Paired bootstrap over images for the tree-minus-zero-shot accuracy gap at

T{=}0.7

(no descriptions, no history, run 0). Images are resampled with replacement (

10{,}000

resamples; seed 42), and both accuracies are recomputed on each resample.

\Delta

is Tree

-

ZS in percentage points; the interval is the 2.5th–97.5th percentile range of the resampled

\Delta

p_{\text{Holm}}

is the two-sided bootstrap

p

after Holm–Bonferroni correction across the twelve dataset–model comparisons. Values preceded by

<

are upper bounds imposed by bootstrap resolution.

Dataset	Model	P0	P1	P2	P3	P4	P5	P6	P7	P8	P9	Mean	SD	$\boldsymbol{\Delta}$
GTSRB	GPT-4o	71.14	63.71	—	66.04	45.84	56.16	67.70	49.94	—	56.83	59.67	8.94	25.31
	LLaMA	37.07	23.20	15.54	20.75	11.10	17.09	18.87	8.10	5.99	7.99	16.57	9.28	31.08
	Qwen-VL	47.06	46.95	46.06	45.84	5.55	45.84	46.06	11.88	17.31	45.84	35.84	16.97	41.51
CIFAR-10	GPT-4o	87.90	86.20	88.70	86.50	89.70	89.50	86.70	87.20	79.90	88.50	87.08	2.81	9.80
	LLaMA	36.70	51.00	43.10	36.10	41.20	41.60	40.90	45.10	40.10	45.50	42.13	4.38	14.90
	Qwen-VL	77.60	78.40	76.50	50.10	77.70	57.00	78.50	56.90	78.60	78.80	71.01	11.45	28.70

Table S3 : Zero-shot accuracy (%) for each of the ten prompt paraphrases at

T{=}0.7

, on the six dataset–model pairs swept at full scale. P0, underlined, is the prompt reported in main-paper Table 1. Em dashes mark runs that did not reach full coverage and are excluded from the summary statistics. Mean, SD, and

\Delta

are computed over the available prompts, with

\Delta=\max-\min

Figure S1 : Prompt sensitivity of zero-shot accuracy at

T{=}0.7

. Dots show full-coverage prompt results, diamonds mark prompt 0 (reported in main-paper Table 1), black ticks show sweep means, and horizontal lines span the observed range. Red denotes GTSRB and blue denotes CIFAR-10. On GTSRB, prompt 0 is the highest full-coverage prompt for all three models; the GPT-4o/GTSRB row spans eight prompts rather than ten.

Dataset	Model	$n$	Prompts	Mean	SD	$\boldsymbol{\Delta}$
MNIST	GPT-4o	20	4	93.75	2.50	5.00
	Qwen-VL	20	4	97.50	2.89	5.00
MSCOCO	GPT-4o	20	4	68.75	4.79	10.00
	Qwen-VL	20	4	68.75	2.50	5.00

Table S4 : Small-scale zero-shot prompt checks on the datasets without a full-scale sweep. Each row compares four prompts on the same matched image subset, so the spread is not confounded by which images each prompt saw. Mean, SD, and

\Delta

are percentages, with

\Delta=\max-\min

		Node acc.		Unresolved		Image acc.
Dataset	$n_{\text{img}}$	Local	Global	Local	Global	Local	Global
GTSRB	50	89.1	90.1	2.1	3.1	58.0	62.0
MNIST	20	71.1	65.8	0.0	0.0	35.0	35.0
CIFAR-10	20	88.9	87.5	1.6	7.8	65.0	60.0
MSCOCO	50	91.2	91.3	0.3	3.8	46.0	50.0

Table S5 : Global-label probe with GPT-4o on images shared by both conditions. Local uses only the current node’s answer options; Global also appends the dataset’s full class list. Node accuracy and unresolved rate are computed over visited node decisions, and image accuracy is end-to-end traversal accuracy. All values are percentages.

		Node acc.		Unresolved		Image acc.
Dataset	$n_{\text{img}}$	Free	Num.	Free	Num.	Free	Num.
GTSRB	20	89.0	90.7	1.4	0.0	60.0	65.0
MNIST	20	68.3	71.4	2.4	0.0	35.0	30.0
CIFAR-10	20	92.3	92.2	1.5	0.0	75.0	75.0
MSCOCO	20	88.2	86.7	0.0	0.0	35.0	30.0

Table S6 : Numbered-choice probe with GPT-4o (

n{=}20

images per dataset). Free uses strict extraction from a free-form response; Num. requests only an option number. Node accuracy and unresolved rate use each condition’s visited nodes; image accuracy is end-to-end. All values are percentages.

Dataset	Paired nodes	Free	Num.	$\boldsymbol{\Delta}$
GTSRB	73	89.0	90.4	+1.4
MNIST	36	69.4	77.8	+8.3
CIFAR-10	63	92.1	92.1	+0.0
MSCOCO	105	90.5	86.7	-3.8

Table S7 : Numbered-choice probe on nodes reached at the same image and depth in both conditions.

\Delta

is numbered minus free-form node accuracy, in percentage points.

Dataset	Int.	Leaves	Depth	Branching	$\boldsymbol{\bar{H}}$	$\boldsymbol{H}$ range	Unreached
GTSRB	22	43	14	16 $\times$ 2, 2 $\times$ 3, 2 $\times$ 5, 2 $\times$ 8	1.02	0.35–3.00	0
MNIST	8	10	3	7 $\times$ 2, 1 $\times$ 3	1.01	0.92–1.30	0
CIFAR-10	9	10	3	9 $\times$ 2	0.96	0.81–1.00	0
MSCOCO	89	90	10	89 $\times$ 2	0.93	0.54–1.00	5
Total	128	153					5

Table S8 : Specifications of the four evaluated trees. Int. is the number of internal nodes; Depth is the maximum root-to-leaf depth. Branching gives internal-node outdegrees and their multiplicities. Split entropy is reported in unnormalized bits. Unreached counts internal nodes that no primary-condition traversal visited.

By dataset
Dataset	Nodes	Node–model pairs
GTSRB	22	61
MNIST	8	22
CIFAR-10	9	27
MSCOCO	84	237
By model

Table S9 : Tree-quality audit coverage. Dataset rows report distinct retained nodes and node–model pairs; model rows report retained pairs. Each pair requires both conditional in-traversal accuracy and a grounding gap.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 846 papers

Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs
Sep 10, 2025Sary Elmansoury, Islam Mesabah, Gerrit Großmann +3VLM Reasoning Image Classification
Department of Data Science and its Applications, German Research Centre for Artificial Intelligence (DFKI) · Department of Computer Science, University of Darmstadt · Department of Computer Science, University of Kaiserslautern–Landau (RPTU) +1
Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?
May 21, 2026Zixuan Lan, Luzhe Sun, Matthew R. Walter +1VLM Evaluation VLM Interpretability
University of Chicago · Toyota Technological Institute at Chicago · Stony Brook University
VRIQ: Benchmarking and Diagnosing the Visual-Reasoning IQ of Vision-Language Models
Feb 5, 2026Tina Khezresmaeilzadeh, Jike Zhong, Konstantinos PsounisVisual Reasoning VLM Reasoning
University of Southern California, Los Angeles, USA

Sep 10, 2025J/K move · Enter open · S save

Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs

Sary Elmansoury, Islam Mesabah, Gerrit Großmann, Peter Neigel+2

Department of Data Science and its Applications, German Research Centre for Artificial Intelligence (DFKI) · Department of Computer Science, University of Darmstadt · Department of Computer Science, University of Kaiserslautern–Landau (RPTU)+1

VLM Reasoning Image Classification VLM Evaluation Fine-Grained Image Classification

Open paper AskPDF arXiv

Vision language models (VLMs) excel at zero-shot visual classification, but their performance on fine-grained tasks and large hierarchical label spaces is understudied. This paper investigates whether structured, tree-based reasoning can enhance VLM performance. We introduce a framework that decomposes classification into interpretable decisions using decision trees and evaluates it on fine-grained (GTSRB) and coarse-grained (CIFAR-10) datasets. Although the model achieves 98.2% accuracy in understanding the tree knowledge, tree-based reasoning consistently underperforms standard zero-shot prompting. We also explore enhancing the tree prompts with LLM-generated classes and image descriptions to improve alignment. The added description enhances the performance of the tree-based and zero-shot methods. Our findings highlight limitations of structured reasoning in visual classification and offer insights for designing more interpretable VLM systems.

Abstract

VLM Reasoning

Explore similar work

Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs

Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?

VRIQ: Benchmarking and Diagnosing the Visual-Reasoning IQ of Vision-Language Models