Semantic Richness or Geometric Reasoning? The Fragility of VLM's Visual Invariance

Jason Qiu; Zachary Meurer; Xavier Thomas; Deepti Ghadiyaram

cs.CVApr 2, 2026

Semantic Richness or Geometric Reasoning? The Fragility of VLM's Visual Invariance

Authors: Jason Qiu, Zachary Meurer, Xavier Thomas, Deepti Ghadiyaram

Organizations: Boston University

Abstract

This work investigates the fundamental fragility of state-of-the-art Vision-Language Models (VLMs) under basic geometric transformations. While modern VLMs excel at semantic tasks such as recognizing objects in canonical orientations and describing complex scenes, they exhibit systematic failures at a more fundamental level: lack of robust spatial invariance and equivariance required to reliably determine object identity under simple rotations, scaling, and identity transformations. We demonstrate this limitation through a systematic evaluation across diverse visual domains, including symbolic sketches, natural photographs, and abstract art. Performance drops sharply as semantic content becomes sparse, and this behavior is observed across architectures, model capacities, and prompting strategies. Overall, our results reveal a systematic gap between semantic understanding and spatial reasoning in current VLMs, highlighting the need for stronger geometric grounding in future multimodal systems.

Figures & tables

Figure 1: Failure of visual transformation reasoning across visual domains. Given a pair of images, models are asked to determine whether they depict the same object under transformations of rotation, scale, or identity. While performance remains near-perfect on natural images (Art, Photo), accuracy drops sharply on abstract and symbolic images (Symbolic and Semantic Sketches), particularly for rotation. Results shown are for Gemini-2.5-Pro ( Comanici et al., 2025 ) , with similar trends across evaluated MLLMs.

Figure 2: Cosine similarity between features extracted from different vision encoders on pairs of images under rotation. Select Omniglot scripts are shown in orange, while Times New Roman and Handwritten English are shown in blue and purple respectively. Across all encoders, similarity decreases with increasing rotation angle, with DINOv2 showing the steepest drop and SigLIP and Qwen2.5-VL-7B maintaining relatively higher similarity.

	Qwen2.5-VL-7B			Qwen2.5-VL-32B			Qwen3-VL-8B			Qwen3-VL-30B			GPT-5.2			Gemini-2.5-Pro
Dataset	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR
Times New Roman	51.07	100.00	02.14	52.67	100.00	05.34	50.11	100.00	00.21	65.81	100.00	31.62	74.25	95.09	53.42	89.32	100.00	78.63
Handwritten English	50.85	98.08	03.63	62.50	98.08	26.92	50.00	100.00	00.00	55.98	100.00	11.97	67.84	96.58	39.10	68.27	99.57	36.97
Omniglot	50.72	99.46	01.98	54.17	94.71	13.62	51.01	99.96	02.06	56.64	99.75	13.53	75.55	80.91	70.19	76.90	98.69	55.10
Random guess	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00

Table 1: Rotation recognition performance across character datasets. Best and worst accuracy per model across datasets are highlighted in teal and red , respectively. Performance is aggregated over rotation angles

10^{\circ}

–

90^{\circ}

. While TNR remains near-perfect across all models, TPR is consistently low, indicating a failure to recognize rotated variants. Closed-source models perform better than open-source models, but the failure persists across all models.

	Qwen2.5-VL-7B			Qwen2.5-VL-32B			Qwen3-VL-8B			Qwen3-VL-30B			GPT-5.2			Gemini-2.5-Pro
Domain	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR
Photo	59.42	100.00	18.83	77.67	100.00	55.33	50.25	100.00	00.50	64.58	100.00	29.17	99.50	100.00	99.00	92.67	100.00	85.33
Art Painting	50.25	100.00	00.50	63.50	100.00	27.00	50.08	100.00	00.17	55.08	100.00	10.17	99.50	100.00	99.00	94.17	100.00	88.33
Cartoon	51.42	100.00	02.83	70.08	100.00	40.17	50.25	100.00	00.50	60.67	100.00	21.33	98.17	100.00	96.33	90.33	100.00	80.67
Sketch	52.25	100.00	04.50	57.25	100.00	14.50	50.42	100.00	00.83	52.83	100.00	05.67	92.25	99.67	84.83	86.50	99.83	73.17
Random guess	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00

Table 2: Rotation recognition performance across PACS domains. Performance aggregated over rotation angles

90^{\circ}

180^{\circ}

, and

270^{\circ}

. While TNR remains near-perfect across models, TPR varies significantly across domains, with strong performance on photos and substantial degradation on sketches, indicating reliance on semantic cues rather than true geometric reasoning.

	Qwen2.5-VL-7B			Qwen2.5-VL-32B			Qwen3-VL-8B			Qwen3-VL-30B			GPT-5.2			Gemini-2.5-Pro
Domain	OR	RR	OI	OR	RR	OI	OR	RR	OI	OR	RR	OI	OR	RR	OI	OR	RR	OI
Photo	100.00	59.42	98.75	100.00	77.67	99.75	100.00	50.25	99.25	100.00	64.58	99.00	100.00	99.50	99.25	100.00	92.67	98.25
Art Painting	98.00	50.25	96.75	99.50	63.50	97.50	99.50	50.08	98.00	98.50	55.08	98.50	98.50	99.50	97.50	100.00	94.17	97.00
Cartoon	99.50	51.42	97.00	99.00	70.08	98.00	100.00	50.25	97.75	100.00	60.67	97.75	100.00	98.17	97.75	100.00	90.33	98.25
Sketch	94.50	52.25	92.75	96.50	57.25	86.25	96.50	50.42	87.00	96.00	52.83	91.00	94.50	92.25	95.50	97.50	86.50	96.00
Random guess	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00

Table 3: Performance across task formulations on PACS. Accuracy is reported for object recognition (OR), object identification (OI), and rotation recognition (RR) tasks (Sec. 3.2.5 ) across PACS domains. For each model and domain, the best and worst task performances are highlighted in teal and red , respectively. While models perform near-perfectly on OR and OI, performance drops significantly on RR, highlighting a gap between semantic recognition and geometric reasoning.

Figure 3: Failure cases on the identity task (Sec. 3.3.1 ) for Qwen2.5-VL-7B. We show four randomly selected examples from Omniglot dataset where the model incorrectly predicts that two identical inputs correspond to different characters.

	Qwen2.5-VL-7B			Qwen2.5-VL-32B			Qwen3-VL-8B			Qwen3-VL-30B			GPT-5.2			Gemini-2.5-Pro
Dataset	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR
Times New Roman	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00
Handwritten English	99.04	98.08	100.00	99.04	98.08	100.00	100.00	100.00	100.00	99.04	98.08	100.00	99.04	98.08	100.00	99.04	98.08	100.00
Omniglot	81.33	99.88	62.78	96.95	94.52	99.38	99.72	99.45	100.00	99.57	99.88	99.26	98.77	97.54	100.00	96.67	93.35	100.00
Random guess	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00

Table 4: Model performance for the identity transformation on printed and handwritten scripts. All models achieve perfect performance on the Times New Roman dataset. The models achieve near-perfect performance on the Handwritten English dataset with a few mistakes in the negative case. The worst performances across all models occur with the Omniglot dataset.

	Qwen2.5-VL-7B			Qwen2.5-VL-32B			Qwen3-VL-8B			Qwen3-VL-30B			GPT-5.2			Gemini-2.5-Pro
Dataset	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR
Times New Roman	98.80	97.60	100.00	99.76	99.52	100.00	100.00	100.00	100.00	98.79	97.59	100.00	98.79	99.03	98.55	99.51	100.00	99.03
Handwritten English	96.77	95.56	97.98	95.36	92.34	98.39	97.59	95.19	100.00	97.59	99.03	96.15	98.07	98.07	98.07	96.63	98.07	95.19
Omniglot	77.40	92.99	61.81	74.21	67.87	80.55	76.05	97.11	54.99	77.04	95.44	58.64	79.72	93.20	66.23	82.56	90.11	75.01
Random guess	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00

Table 5: Model performance on the scale-invariance task aggregated across all scales. All models achieve near-perfect performance on Times New Roman and Handwritten English characters, indicating robustness to scale changes. In contrast, performance on Omniglot is substantially lower and exhibits greater variation in recall (TPR) and specificity (TNR) across models.

		Qwen2.5-VL-7B		Qwen2.5-VL-32B		Qwen3-VL-8B		Qwen3-VL-30B		GPT-5.2		Gemini-2.5-Pro
Script	ICL Setting	TNR	TPR $\uparrow$	TNR	TPR $\uparrow$	TNR	TPR $\uparrow$	TNR	TPR $\uparrow$	TNR	TPR $\uparrow$	TNR	TPR $\uparrow$
Malayalam (top-tier)	None	100.00	00.00	97.87	06.38	100.00	00.00	100.00	02.13	93.62	34.04	100.00	38.30
	Few-shot	100.00	00.00	65.96	51.06 $\uparrow$	100.00	34.04 $\uparrow$	78.72	72.34 $\uparrow$	91.49	85.11 $\uparrow$	97.87	34.04 $\downarrow$
	Rotational Grid	100.00	00.00	91.49	12.77 $\uparrow$	65.96	65.96 $\uparrow$	23.40	87.23 $\uparrow$	72.34	97.87 $\uparrow$	93.62	48.94 $\uparrow$
Tengwar (medium-tier)	None	100.00	00.00	88.00	16.00	100.00	00.00	100.00	00.00	100.00	20.00	96.00	32.00
	Few-shot	100.00	00.00	68.00	76.00 $\uparrow$	100.00	16.00 $\uparrow$	68.00	72.00 $\uparrow$	80.00	68.00 $\uparrow$	92.00	32.00

Table 6: Model performance with in-context learning and structured visual prompting across scripts. Arrows indicate change in TPR relative to the None setting:

\uparrow

denotes improvement and

\downarrow

denotes degradation. Both few-shot prompting and rotational grid inputs increase TPR across models, but often reduce TNRs. Improvements are larger for higher-capacity models.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Datasets used in our evaluation. Omniglot ( Lake et al., 2015 ) contains handwritten binary characters from $50$ diverse scripts. Times New Roman ( Morison and Lardent, 1932 ) provides standardized English characters rendered in a fixed typeface. Handwritten English Mann (2024) includes handwritten characters from the English alphabet. PACS ( Yu et al., 2022 ) contains images of common object categories (e.g., guitar, dog, elephant) across four visual domains: P hotograph, A rt, C artoon, and S ketch. Together, these datasets allow us to evaluate transformation invariance in MLLMs across scripts, visual styles, and images with varying levels of semantic richness.

Dataset	# Samples	Description
Omniglot	1,623	Handwritten characters across 50 scripts
Times New Roman	52	Printed English alphabet (A–Z, a–z)
Handwritten English	52	Handwritten English alphabet
PACS	800	4 domains $\times$ 200 images each

Appendix

Table 7: Dataset overview. Number of samples used for evaluation in each dataset.

Omniglot scripts
Alphabet of the Magi	Armenian	Gujarati	Mongolian
Angelic	Asomtavruli (Georgian)	Gurmukhi	N’Ko
Anglo-Saxon Futhorc	Atemayar Qelisayer	Hebrew	Ojibwe (Canadian Aboriginal)
Arcadian	Atlantean	Inuktitut (Canadian Aboriginal)	Old Church Slavonic (Cyrillic)
Aurek-Besh	Avesta	Japanese (Hiragana)	Oriya
Balinese	Bengali	Japanese (Katakana)	Sanskrit

Appendix

Table 8: Omniglot scripts. List of 50 writing systems used in the Omniglot dataset.

Categories	Dog, Elephant, Giraffe, Horse, Person, Guitar, House
Domains	Photograph, Art Painting, Cartoon, Sketch

Appendix

Table 9: PACS dataset. Object categories and visual domains used in evaluation.

Figure 5: Examples from the Omniglot dataset

Figure 6: Examples from the PACS dataset

Figure 7: Examples from the Times New Roman dataset

Figure 8: Examples from the Handwritten English dataset

Figure 9: Padding artifacts under rotation and scaling. Top row shows rotation and bottom row shows scaling. Non- $90^{\circ}$ rotations introduce padding regions corresponding to pixels not covered by the original image. Scaling similarly introduces padding for resized images. For character datasets (left), padding matches the white background and is not visually salient. In contrast, for PACS (right), padding introduces visible artifacts under both rotation and scaling. Top row: Images rotated by $45^{\circ}$ . Bottom row: Images scaled by $0.5$ .

Model	Feature Extraction	Dimension
CLIP (ViT-L/14) ( Radford et al., 2021 )	CLS token	768
DINOv2 (ViT-L/14) ( Oquab et al., 2024 )	CLS token	1024
SigLIP (ViT-SO400M) ( Zhai et al., 2023 )	MAP (multi-head attention pooling)	1152
Qwen2.5-VL-7B ( Bai et al., 2025b )	Mean over vision tokens	3584
Stable Diffusion v2.1 ( Rombach et al., 2022 )	Mean over layer 14 features	1280

Appendix

Table 10: Vision feature extraction details. For CLIP and DINOv2, we use the CLS token. For SigLIP, we use the multi-head attention pooling (MAP) output. For Qwen2.5-VL, we average vision token embeddings to obtain a global representation. For Stable Diffusion v2.1 we obtain features from layer

14

in the UNet.

Figure 10: Cosine similarity between features extracted from different vision encoders on pairs of images under rotation. Select Omniglot scripts are shown in orange, while Times New Roman and Handwritten English are shown in blue and purple respectively. Across all encoders, similarity decreases with increasing rotation angle, with DINOv2 showing the steepest drop and SigLIP and Qwen2.5-VL-7B maintaining relatively higher similarity.

Metric	Times New Roman	Handwritten English	Braille	Balinese	Anglo-Saxon_Futhorc	Tengwar	Malayalam	Grantha
Acc.	57.87	54.54	66.43	50.18	53.76	49.63	50.87	50.42
TPR	77.27	95.28	46.85	99.62	97.17	99.27	96.71	99.78
TNR	38.46	13.81	86.01	0.75	10.34	0.00	5.03	1.05

Appendix

Table 11: Idefics2 accuracy performance on the rotation task across scripts. Times New Roman and Handwritten English are shown in blue and purple respectively, while Omniglot scripts are shown in orange. Accuracy remains near chance for most scripts, despite consistently high TPR and substantially lower TNR, indicating a strong “YES” prediction bias.

Figure 11: Scale invariance performance across datasets for Qwen2.5-VL. aggregated over all scales (Sec. 3.3.2 ). Performance is high on natural image domains (Art Painting, Cartoon, Photo, Sketch) and familiar scripts (Times New Roman, Handwritten English), but drops on Omniglot, indicating reduced robustness under lower semantic familiarity.

Figure 12: Scale invariance performance across datasets for Gemini-2.5-Pro. aggregated over all scales (Sec. 3.3.2 ). Performance is near-perfect on natural image domains (Art Painting, Cartoon, Photo, Sketch) and familiar scripts (Times New Roman, Handwritten English), but drops on Omniglot, indicating reduced robustness under lower semantic familiarity.

Figure 13: Perimetric complexity vs performance. Perimetric complexity shows a weak correlation (

r=-0.18

) with accuracy of Qwen2.5-VL-7B on the scale invariance task. Times New Roman and Handwritten English are shown in blue and purple respectively, while Omniglot scripts are highlighted in orange.

Panel of Figure 14 — Figure 14: Recall and Specificity at scale $0.3\times$ for representative scripts for the scale-invariance task. English characters rendered in Times New Roman , Handwritten English characters , and Omniglot scripts are shown in blue, purple, and orange respectively, and are selected to represent high-, medium-, and low-performing groups. Across both models, familiar scripts such as Greek and Latin consistently outperform less familiar scripts like Braille. While Qwen2.5-VL-32B achieves higher recall than Qwen2.5-VL-7B on low-performing Omniglot scripts, it exhibits lower specificity.

Figure 15: Model accuracy on the scale-invariance task across scale factors. Both Qwen2.5-VL-7B and Qwen2.5-VL-32B models maintain near-perfect accuracy for both Times New Roman and Handwritten English characters across all scales, while performance on Omniglot scripts is substantially and consistently lower for both models.

Panel of Figure 16 — Figure 16: In-context learning (ICL) prompting setup. The system prompt includes two labeled examples: a positive pair (top) where one image is a rotated version of the other, and a negative pair (bottom) where the images are not related by rotation.

Panel of Figure 17 — Figure 17: Rotational grid prompting setup. The model is first given a structured system prompt describing the layout of rotational grids. Two reference grids (Character A and Character B) are then provided to illustrate rotations across different characters.

	Qwen2.5-VL-7B			Qwen2.5-VL-32B			Qwen2.5-VL-72B			Qwen3-VL-4B
Dataset	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR
Times New Roman	50.96	100.00	01.92	54.13	100.00	08.27	52.40	100.00	04.81	50.29	100.00	00.58
Handwritten English	50.96	98.08	03.85	62.50	98.08	26.92	57.21	100.00	14.42	50.00	100.00	00.00
Omniglot	50.70	99.45	01.95	54.20	94.73	13.68	57.48	96.51	18.46	50.24	99.98	00.51
	Qwen3-VL-8B			Qwen3-VL-30B			GPT-5.2			Gemini-2.5-Pro
Dataset	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR

Appendix

Table 12: Rotation recognition performance across character datasets. Best and worst accuracy per model across datasets are highlighted in teal and red , respectively. Performance aggregated over rotation angles

10^{\circ}

–

90^{\circ}

Encoder	10°	20°	30°	40°	50°	60°	70°	80°	90°	Overall
SigLIP-SO400M-384	86.5	81.8	71.5	57.3	64.0	74.6	84.4	84.9	100.0	78.3
Qwen2.5-VL-7B	94.3	86.1	78.4	68.3	64.1	77.2	87.7	93.5	99.8	83.3
DINOv2 ViT-L/14	98.0	96.3	94.7	83.2	82.4	92.6	95.6	97.3	99.7	93.3
CLIP ViT-L/14-336	91.3	86.2	79.0	69.8	68.1	79.8	85.4	89.4	99.1	83.1

Appendix

Table 13: Angle classification accuracy across vision encoders on Omniglot characters. All encoders substantially exceed chance performance (

11.1

%), suggesting that rotation information is linearly recoverable from vision encoder features.

	Qwen2.5-VL-7B			Qwen2.5-VL-32B			Qwen3-VL-8B			Qwen3-VL-30B			GPT-5.2			Gemini-2.5-Pro
Domain	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR
Photo	57.41	100.00	14.82	83.73	100.00	67.45	52.09	100.00	4.18	82.14	100.00	64.27	98.91	99.95	97.86	94.70	100.00	89.41
Art Painting	54.70	100.00	9.41	79.55	100.00	59.09	52.91	100.00	5.82	77.23	100.00	54.45	99.02	100.00	98.05	95.80	100.00	91.59
Cartoon	55.23	100.00	10.45	81.89	100.00	63.77	52.80	100.00	5.59	82.50	100.00	65.00	97.95	100.00	95.91	93.23	100.00	86.45
Sketch	53.36	100.00	6.73	73.07	100.00	46.14	52.66	100.00	5.32	76.00	100.00	52.00	95.61	99.64	91.59	90.30	99.86	80.73
Random guess	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00

Appendix

Table 14: Rotation recognition performance across PACS domains (

10^{\circ}

–

90^{\circ}

). Best and worst accuracy per model across domains are highlighted in teal and red , respectively. Performance aggregated over rotations from

10^{\circ}

90^{\circ}

in increments of

10^{\circ}

. While TNR remains near-perfect across all models, TPR varies significantly across domains, with strong performance on photographs and substantial degradation on sketches, indicating limited geometric robustness.

	Qwen2.5-VL-7B			Qwen2.5-VL-32B			Qwen3-VL-8B			Qwen3-VL-30B			GPT-5.2			Gemini-2.5-Pro
Domain	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR	Acc.	TNR	TPR
Photo	53.50	100.00	7.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00
Art Painting	52.75	100.00	5.50	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00
Cartoon	76.75	100.00	53.50	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	99.75	100.00	99.50	100.00	100.00	100.00
Sketch	92.25	100.00	84.50	100.00	100.00	100.00	100.00	100.00	100.00	99.75	100.00	99.50	100.00	100.00	100.00	100.00	100.00	100.00
Random guess	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00

Appendix

Table 15: Model performance for the identity transformation on PACS. Best accuracy values per model across domains are highlighted in teal , including ties, while worst values are shown in red . All models achieve near-perfect performance across domains, except Qwen2.5-VL-7B , which shows lower TPR on Photo and Art despite perfect TNR.

Table 16: Model performance with In-Context Learning

Dataset	Prompt Variant	Accuracy	TPR	TNR
Times New Roman	Original	52.56	5.98	99.15
	Balanced	51.71	3.42	100.0
	Forced Choice	90.49	80.98	100.0
Handwritten English	Original	61.22	32.05	90.38
	Balanced	61.97	23.93	100.0
	Forced Choice	76.60	99.36	53.85

Appendix

Table 17: Performance metrics across datasets and prompt variants for Qwen2.5-VL-32B. Forced-choice MCQ dramatically improves TPR (+ 65-75 %) on all datasets, demonstrating strong response bias. However, TNR collapse on Handwritten English (53.85%) and Omniglot (39.11%) indicates the model lacks stable geometric reasoning – it trades false negatives for false positives rather than achieving robust rotation understanding.

Dataset	Prompt Variant	Accuracy	TPR	TNR
Times New Roman	Original	89.32	78.6	100.0
	Balanced	88.1	76.3	100.0
	Forced Choice	87.9	75.9	100.0
Handwritten English	Original	68.3	37.0	99.6
	Balanced	69.6	39.5	99.6
	Forced Choice	67.2	34.4	100.0

Appendix

Table 18: Performance metrics across datasets and prompt variants for Gemini-2.5-Pro. Different prompt formats shows consistent performance across all datasets, demonstrating that the drop in performance in Handwritten English and Omniglot is not due to prompt sensitivity.

Category	Accuracy	Avg Time
Armenian	100%	2.78s $\pm$ 1.13s
Greek	100%	2.70s $\pm$ 1.12s
Hand-English	100%	2.46s $\pm$ 1.08s
Malayalam	100%	2.91s $\pm$ 1.29s
Times New Roman	100%	2.28s $\pm$ 1.05s

Appendix

Table 19: Human baseline performance on the rotation task by script category. Humans maintain perfect geometric reasoning regardless of script familiarity, highlighting a fundamental vulnerability in current VLMs.

Dataset	Qwen2.5-VL-7B	Qwen2.5-VL-32B	Qwen3-VL-8B	Qwen3-VL-30B	GPT-5.2	Gemini-2.5-Pro
Times New Roman	51.07 [47.75, 54.17]	52.67 [49.36, 55.67]	50.11 [46.79, 53.21]	65.81 [62.50, 68.80]	74.25 [71.37, 76.92]	89.32 [87.18, 91.24]
Handwritten English	50.85 [47.86, 54.17]	62.50 [59.40, 65.60]	50.00 [47.01, 53.42]	55.98 [52.99, 59.19]	67.84 [64.53, 70.94]	68.27 [65.49, 71.26]
Omniglot	50.72 [50.14, 51.25]	54.17 [53.61, 54.74]	51.01 [50.48, 51.56]	56.64 [56.10, 57.20]	75.55 [75.04, 76.02]	76.90 [76.40, 77.36]

Appendix

Table 20: Rotation task: Accuracy with 95% bootstrap confidence intervals (10,000 resamples). Narrow intervals confirm that the reported results are statistically robust.

Dataset	Qwen2.5-VL-7B	Qwen2.5-VL-32B	Qwen3-VL-8B	Qwen3-VL-30B	GPT-5.2	Gemini-2.5-Pro
Times New Roman	98.80 [97.60, 99.76]	99.76 [99.28, 100.00]	100.00 [100.00, 100.00]	98.79 [97.60, 99.76]	98.79 [97.60, 99.76]	99.51 [98.56, 100.00]
Handwritten English	96.77 [95.16, 98.19]	95.36 [93.54, 97.18]	97.59 [95.19, 99.52]	97.59 [96.15, 98.80]	98.07 [96.63, 99.28]	96.63 [94.23, 99.04]
Omniglot	77.40 [76.68, 78.16]	74.21 [73.48, 74.92]	76.05 [75.34, 76.79]	77.04 [76.32, 77.80]	79.72 [79.00, 80.42]	82.56 [81.91, 83.21]

Appendix

Table 21: Scale task: Accuracy with 95% bootstrap confidence intervals (10,000 resamples). Narrow intervals confirm that the reported results are statistically robust.

Dataset	Molmo2-8B Acc.	Molmo2-8B TNR	Molmo2-8B TPR	InternVL3.5-8B Acc.	InternVL3.5-8B TNR	InternVL3.5-8B TPR
Times New Roman	75.48	100.00	50.96	98.07	100.00	96.15
Handwritten English	73.79	100.00	47.59	82.69	100.00	65.38
Omniglot	56.07	99.61	12.52	64.28	99.35	29.22
Random guess	50.00	50.00	50.00	50.00	50.00	50.00

Appendix

Table 22: Model performance on the scale-invariance task aggregated across all scales for open-sourced models Molmo2-8B and InternVL3.5-8B.

Dataset	Molmo2-8B Acc.	Molmo2-8B TNR	Molmo2-8B TPR	InternVL3.5-8B Acc.	InternVL3.5-8B TNR	InternVL3.5-8B TPR
Times New Roman	51.60	98.08	5.13	58.01	16.03	100.00
Handwritten English	70.41	96.15	44.66	62.82	31.41	94.23
Omniglot	69.09	96.92	41.27	64.83	41.17	88.48
Random guess	50.00	50.00	50.00	50.00	50.00	50.00

Appendix

Table 23: Model performance on the rotation task aggregated across all angles for open-sourced models Molmo2-8B and InternVL3.5-8B.

Model	Task	Acc.	TPR	TNR
Baseline	Rotation	56.36	12.72	100.00
Fine-tuned	Rotation	90.75	81.50	100.00
Baseline	Scale	92.94	86.62	99.25
Fine-tuned	Scale	96.56	93.75	99.38

Appendix

Table 24: LoRA fine-tuning Qwen2.5-VL-7B on semantic VQA pairs (no geometric supervision) of sketch images from PACS improves rotation recognition. Rotation TPR improves by +68.78% while TNR is preserved.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

Vision Language Models Cannot Reason About Physical Transformation
Mar 7, 2026Dezhi Luo, Yijiang Li, Maijunxian Wang +7Physical Reasoning Vision-Language Models
University of Michigan · University of California San Diego · University of California Berkeley +6
Same Answer, Different Representations: Hidden instability in VLMs
Feb 6, 2026Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena +6VLM Evaluation VLM Robustness
Sapienza University of Rome · University of Edinburgh · University of Oxford +5
Semantic Robustness Certification for Vision-Language Models
Jun 17, 2026Peiyu Yang, Paul Montague, Feng Liu +4VLM Robustness Distribution Shift
School of Computing & Information Systems, University of Melbourne, Australia · Defence Science and Technology Group, Australia.

Mar 7, 2026J/K move · Enter open · S save

Vision Language Models Cannot Reason About Physical Transformation

Dezhi Luo, Yijiang Li, Maijunxian Wang, Tianwei Zhao+6

University of Michigan · University of California San Diego · University of California Berkeley+6

Physical Reasoning Vision-Language Models VLM Reasoning VLM Evaluation

Open paper AskPDF arXiv

Understanding physical transformations is fundamental for reasoning in dynamic environments. While Vision Language Models (VLMs) show promise in embodied applications, whether they genuinely understand physical transformations remains unclear. We introduce ConservationBench evaluating conservation -- whether physical quantities remain invariant under transformations. Spanning four properties with paired conserving/non-conserving scenarios, we generate and evaluate 23,040 questions across 112 VLMs. Results reveal systematic failure: performance remains near chance with improvements on conservation tasks accompanied by drops on controls. Control experiments show strong textual priors favoring invariance, yet models perform worse with actual visual content when performance is balanced across conserving and non-conserving scenarios. Neither temporal resolution, prompting, nor curated sampling helps. These findings show that current VLMs fail to maintain transformation-invariant representations of physical properties across dynamic scenes.

Abstract

VLM Reasoning

Explore similar work

Vision Language Models Cannot Reason About Physical Transformation

Same Answer, Different Representations: Hidden instability in VLMs

Semantic Robustness Certification for Vision-Language Models