This work investigates the fundamental fragility of state-of-the-art Vision-Language Models (VLMs) under basic geometric transformations. While modern VLMs excel at semantic tasks such as recognizing objects in canonical orientations and describing complex scenes, they exhibit systematic failures at a more fundamental level: lack of robust spatial invariance and equivariance required to reliably determine object identity under simple rotations, scaling, and identity transformations. We demonstrate this limitation through a systematic evaluation across diverse visual domains, including symbolic sketches, natural photographs, and abstract art. Performance drops sharply as semantic content becomes sparse, and this behavior is observed across architectures, model capacities, and prompting strategies. Overall, our results reveal a systematic gap between semantic understanding and spatial reasoning in current VLMs, highlighting the need for stronger geometric grounding in future multimodal systems.
Figures & tables
Figure 1: Failure of visual transformation reasoning across visual domains. Given a pair of images, models are asked to determine whether they depict the same object under transformations of rotation, scale, or identity. While performance remains near-perfect on natural images (Art, Photo), accuracy drops sharply on abstract and symbolic images (Symbolic and Semantic Sketches), particularly for rotation. Results shown are for Gemini-2.5-Pro ( Comanici et al., 2025 ) , with similar trends across evaluated MLLMs.
Figure 2: Cosine similarity between features extracted from different vision encoders on pairs of images under rotation. Select Omniglot scripts are shown in orange, while Times New Roman and Handwritten English are shown in blue and purple respectively. Across all encoders, similarity decreases with increasing rotation angle, with DINOv2 showing the steepest drop and SigLIP and Qwen2.5-VL-7B maintaining relatively higher similarity.
Qwen2.5-VL-7B
Qwen2.5-VL-32B
Qwen3-VL-8B
Qwen3-VL-30B
GPT-5.2
Gemini-2.5-Pro
Dataset
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Times New Roman
51.07
100.00
02.14
52.67
100.00
05.34
50.11
100.00
00.21
65.81
100.00
31.62
74.25
95.09
53.42
89.32
100.00
78.63
Handwritten English
50.85
98.08
03.63
62.50
98.08
26.92
50.00
100.00
00.00
55.98
100.00
11.97
67.84
96.58
39.10
68.27
99.57
36.97
Omniglot
50.72
99.46
01.98
54.17
94.71
13.62
51.01
99.96
02.06
56.64
99.75
13.53
75.55
80.91
70.19
76.90
98.69
55.10
Random guess
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
Table 1: Rotation recognition performance across character datasets. Best and worst accuracy per model across datasets are highlighted in teal and red , respectively. Performance is aggregated over rotation angles 10∘ – 90∘ . While TNR remains near-perfect across all models, TPR is consistently low, indicating a failure to recognize rotated variants. Closed-source models perform better than open-source models, but the failure persists across all models.
Qwen2.5-VL-7B
Qwen2.5-VL-32B
Qwen3-VL-8B
Qwen3-VL-30B
GPT-5.2
Gemini-2.5-Pro
Domain
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Photo
59.42
100.00
18.83
77.67
100.00
55.33
50.25
100.00
00.50
64.58
100.00
29.17
99.50
100.00
99.00
92.67
100.00
85.33
Art Painting
50.25
100.00
00.50
63.50
100.00
27.00
50.08
100.00
00.17
55.08
100.00
10.17
99.50
100.00
99.00
94.17
100.00
88.33
Cartoon
51.42
100.00
02.83
70.08
100.00
40.17
50.25
100.00
00.50
60.67
100.00
21.33
98.17
100.00
96.33
90.33
100.00
80.67
Sketch
52.25
100.00
04.50
57.25
100.00
14.50
50.42
100.00
00.83
52.83
100.00
05.67
92.25
99.67
84.83
86.50
99.83
73.17
Random guess
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
Table 2: Rotation recognition performance across PACS domains. Performance aggregated over rotation angles 90∘ , 180∘ , and 270∘ . While TNR remains near-perfect across models, TPR varies significantly across domains, with strong performance on photos and substantial degradation on sketches, indicating reliance on semantic cues rather than true geometric reasoning.
Qwen2.5-VL-7B
Qwen2.5-VL-32B
Qwen3-VL-8B
Qwen3-VL-30B
GPT-5.2
Gemini-2.5-Pro
Domain
OR
RR
OI
OR
RR
OI
OR
RR
OI
OR
RR
OI
OR
RR
OI
OR
RR
OI
Photo
100.00
59.42
98.75
100.00
77.67
99.75
100.00
50.25
99.25
100.00
64.58
99.00
100.00
99.50
99.25
100.00
92.67
98.25
Art Painting
98.00
50.25
96.75
99.50
63.50
97.50
99.50
50.08
98.00
98.50
55.08
98.50
98.50
99.50
97.50
100.00
94.17
97.00
Cartoon
99.50
51.42
97.00
99.00
70.08
98.00
100.00
50.25
97.75
100.00
60.67
97.75
100.00
98.17
97.75
100.00
90.33
98.25
Sketch
94.50
52.25
92.75
96.50
57.25
86.25
96.50
50.42
87.00
96.00
52.83
91.00
94.50
92.25
95.50
97.50
86.50
96.00
Random guess
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
Table 3: Performance across task formulations on PACS. Accuracy is reported for object recognition (OR), object identification (OI), and rotation recognition (RR) tasks (Sec. 3.2.5 ) across PACS domains. For each model and domain, the best and worst task performances are highlighted in teal and red , respectively. While models perform near-perfectly on OR and OI, performance drops significantly on RR, highlighting a gap between semantic recognition and geometric reasoning.
Figure 6
Figure 3: Failure cases on the identity task (Sec. 3.3.1 ) for Qwen2.5-VL-7B. We show four randomly selected examples from Omniglot dataset where the model incorrectly predicts that two identical inputs correspond to different characters.
Qwen2.5-VL-7B
Qwen2.5-VL-32B
Qwen3-VL-8B
Qwen3-VL-30B
GPT-5.2
Gemini-2.5-Pro
Dataset
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Times New Roman
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
Handwritten English
99.04
98.08
100.00
99.04
98.08
100.00
100.00
100.00
100.00
99.04
98.08
100.00
99.04
98.08
100.00
99.04
98.08
100.00
Omniglot
81.33
99.88
62.78
96.95
94.52
99.38
99.72
99.45
100.00
99.57
99.88
99.26
98.77
97.54
100.00
96.67
93.35
100.00
Random guess
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
Table 4: Model performance for the identity transformation on printed and handwritten scripts. All models achieve perfect performance on the Times New Roman dataset. The models achieve near-perfect performance on the Handwritten English dataset with a few mistakes in the negative case. The worst performances across all models occur with the Omniglot dataset.
Qwen2.5-VL-7B
Qwen2.5-VL-32B
Qwen3-VL-8B
Qwen3-VL-30B
GPT-5.2
Gemini-2.5-Pro
Dataset
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Times New Roman
98.80
97.60
100.00
99.76
99.52
100.00
100.00
100.00
100.00
98.79
97.59
100.00
98.79
99.03
98.55
99.51
100.00
99.03
Handwritten English
96.77
95.56
97.98
95.36
92.34
98.39
97.59
95.19
100.00
97.59
99.03
96.15
98.07
98.07
98.07
96.63
98.07
95.19
Omniglot
77.40
92.99
61.81
74.21
67.87
80.55
76.05
97.11
54.99
77.04
95.44
58.64
79.72
93.20
66.23
82.56
90.11
75.01
Random guess
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
Table 5: Model performance on the scale-invariance task aggregated across all scales. All models achieve near-perfect performance on Times New Roman and Handwritten English characters, indicating robustness to scale changes. In contrast, performance on Omniglot is substantially lower and exhibits greater variation in recall (TPR) and specificity (TNR) across models.
Qwen2.5-VL-7B
Qwen2.5-VL-32B
Qwen3-VL-8B
Qwen3-VL-30B
GPT-5.2
Gemini-2.5-Pro
Script
ICL Setting
TNR
TPR ↑
TNR
TPR ↑
TNR
TPR ↑
TNR
TPR ↑
TNR
TPR ↑
TNR
TPR ↑
Malayalam (top-tier)
None
100.00
00.00
97.87
06.38
100.00
00.00
100.00
02.13
93.62
34.04
100.00
38.30
Few-shot
100.00
00.00
65.96
51.06 ↑
100.00
34.04 ↑
78.72
72.34 ↑
91.49
85.11 ↑
97.87
34.04 ↓
Rotational Grid
100.00
00.00
91.49
12.77 ↑
65.96
65.96 ↑
23.40
87.23 ↑
72.34
97.87 ↑
93.62
48.94 ↑
Tengwar (medium-tier)
None
100.00
00.00
88.00
16.00
100.00
00.00
100.00
00.00
100.00
20.00
96.00
32.00
Few-shot
100.00
00.00
68.00
76.00 ↑
100.00
16.00 ↑
68.00
72.00 ↑
80.00
68.00 ↑
92.00
32.00
Table 6: Model performance with in-context learning and structured visual prompting across scripts. Arrows indicate change in TPR relative to the None setting: ↑ denotes improvement and ↓ denotes degradation. Both few-shot prompting and rotational grid inputs increase TPR across models, but often reduce TNRs. Improvements are larger for higher-capacity models.
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Datasets used in our evaluation. Omniglot ( Lake et al., 2015 ) contains handwritten binary characters from 50 diverse scripts. Times New Roman ( Morison and Lardent, 1932 ) provides standardized English characters rendered in a fixed typeface. Handwritten English Mann (2024) includes handwritten characters from the English alphabet. PACS ( Yu et al., 2022 ) contains images of common object categories (e.g., guitar, dog, elephant) across four visual domains: P hotograph, A rt, C artoon, and S ketch. Together, these datasets allow us to evaluate transformation invariance in MLLMs across scripts, visual styles, and images with varying levels of semantic richness.
Dataset
# Samples
Description
Omniglot
1,623
Handwritten characters across 50 scripts
Times New Roman
52
Printed English alphabet (A–Z, a–z)
Handwritten English
52
Handwritten English alphabet
PACS
800
4 domains × 200 images each
Appendix
Table 7: Dataset overview. Number of samples used for evaluation in each dataset.
Omniglot scripts
Alphabet of the Magi
Armenian
Gujarati
Mongolian
Angelic
Asomtavruli (Georgian)
Gurmukhi
N’Ko
Anglo-Saxon Futhorc
Atemayar Qelisayer
Hebrew
Ojibwe (Canadian Aboriginal)
Arcadian
Atlantean
Inuktitut (Canadian Aboriginal)
Old Church Slavonic (Cyrillic)
Aurek-Besh
Avesta
Japanese (Hiragana)
Oriya
Balinese
Bengali
Japanese (Katakana)
Sanskrit
Appendix
Table 8: Omniglot scripts. List of 50 writing systems used in the Omniglot dataset.
Categories
Dog, Elephant, Giraffe, Horse, Person, Guitar, House
Domains
Photograph, Art Painting, Cartoon, Sketch
Appendix
Table 9: PACS dataset. Object categories and visual domains used in evaluation.
Figure 5: Examples from the Omniglot dataset
Figure 6: Examples from the PACS dataset
Figure 7: Examples from the Times New Roman dataset
Figure 8: Examples from the Handwritten English dataset
Figure 9: Padding artifacts under rotation and scaling. Top row shows rotation and bottom row shows scaling. Non- 90∘ rotations introduce padding regions corresponding to pixels not covered by the original image. Scaling similarly introduces padding for resized images. For character datasets (left), padding matches the white background and is not visually salient. In contrast, for PACS (right), padding introduces visible artifacts under both rotation and scaling. Top row: Images rotated by 45∘ . Bottom row: Images scaled by 0.5 .
Model
Feature Extraction
Dimension
CLIP (ViT-L/14) ( Radford et al., 2021 )
CLS token
768
DINOv2 (ViT-L/14) ( Oquab et al., 2024 )
CLS token
1024
SigLIP (ViT-SO400M) ( Zhai et al., 2023 )
MAP (multi-head attention pooling)
1152
Qwen2.5-VL-7B ( Bai et al., 2025b )
Mean over vision tokens
3584
Stable Diffusion v2.1 ( Rombach et al., 2022 )
Mean over layer 14 features
1280
Appendix
Table 10: Vision feature extraction details. For CLIP and DINOv2, we use the CLS token. For SigLIP, we use the multi-head attention pooling (MAP) output. For Qwen2.5-VL, we average vision token embeddings to obtain a global representation. For Stable Diffusion v2.1 we obtain features from layer 14 in the UNet.
Figure 10: Cosine similarity between features extracted from different vision encoders on pairs of images under rotation. Select Omniglot scripts are shown in orange, while Times New Roman and Handwritten English are shown in blue and purple respectively. Across all encoders, similarity decreases with increasing rotation angle, with DINOv2 showing the steepest drop and SigLIP and Qwen2.5-VL-7B maintaining relatively higher similarity.
Metric
Times New Roman
Handwritten English
Braille
Balinese
Anglo-Saxon_Futhorc
Tengwar
Malayalam
Grantha
Acc.
57.87
54.54
66.43
50.18
53.76
49.63
50.87
50.42
TPR
77.27
95.28
46.85
99.62
97.17
99.27
96.71
99.78
TNR
38.46
13.81
86.01
0.75
10.34
0.00
5.03
1.05
Appendix
Table 11: Idefics2 accuracy performance on the rotation task across scripts. Times New Roman and Handwritten English are shown in blue and purple respectively, while Omniglot scripts are shown in orange. Accuracy remains near chance for most scripts, despite consistently high TPR and substantially lower TNR, indicating a strong “YES” prediction bias.
Figure 11: Scale invariance performance across datasets for Qwen2.5-VL. aggregated over all scales (Sec. 3.3.2 ). Performance is high on natural image domains (Art Painting, Cartoon, Photo, Sketch) and familiar scripts (Times New Roman, Handwritten English), but drops on Omniglot, indicating reduced robustness under lower semantic familiarity.
Figure 12: Scale invariance performance across datasets for Gemini-2.5-Pro. aggregated over all scales (Sec. 3.3.2 ). Performance is near-perfect on natural image domains (Art Painting, Cartoon, Photo, Sketch) and familiar scripts (Times New Roman, Handwritten English), but drops on Omniglot, indicating reduced robustness under lower semantic familiarity.
Figure 13: Perimetric complexity vs performance. Perimetric complexity shows a weak correlation ( r=−0.18 ) with accuracy of Qwen2.5-VL-7B on the scale invariance task. Times New Roman and Handwritten English are shown in blue and purple respectively, while Omniglot scripts are highlighted in orange.
Figure 14: Recall and Specificity at scale 0.3× for representative scripts for the scale-invariance task. English characters rendered in Times New Roman , Handwritten English characters , and Omniglot scripts are shown in blue, purple, and orange respectively, and are selected to represent high-, medium-, and low-performing groups. Across both models, familiar scripts such as Greek and Latin consistently outperform less familiar scripts like Braille. While Qwen2.5-VL-32B achieves higher recall than Qwen2.5-VL-7B on low-performing Omniglot scripts, it exhibits lower specificity.
Figure 15: Model accuracy on the scale-invariance task across scale factors. Both Qwen2.5-VL-7B and Qwen2.5-VL-32B models maintain near-perfect accuracy for both Times New Roman and Handwritten English characters across all scales, while performance on Omniglot scripts is substantially and consistently lower for both models.
Figure 16: In-context learning (ICL) prompting setup. The system prompt includes two labeled examples: a positive pair (top) where one image is a rotated version of the other, and a negative pair (bottom) where the images are not related by rotation.
Figure 17: Rotational grid prompting setup. The model is first given a structured system prompt describing the layout of rotational grids. Two reference grids (Character A and Character B) are then provided to illustrate rotations across different characters.
Qwen2.5-VL-7B
Qwen2.5-VL-32B
Qwen2.5-VL-72B
Qwen3-VL-4B
Dataset
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Times New Roman
50.96
100.00
01.92
54.13
100.00
08.27
52.40
100.00
04.81
50.29
100.00
00.58
Handwritten English
50.96
98.08
03.85
62.50
98.08
26.92
57.21
100.00
14.42
50.00
100.00
00.00
Omniglot
50.70
99.45
01.95
54.20
94.73
13.68
57.48
96.51
18.46
50.24
99.98
00.51
Qwen3-VL-8B
Qwen3-VL-30B
GPT-5.2
Gemini-2.5-Pro
Dataset
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Appendix
Table 12: Rotation recognition performance across character datasets. Best and worst accuracy per model across datasets are highlighted in teal and red , respectively. Performance aggregated over rotation angles 10∘ – 90∘ . While TNR remains near-perfect across all models, TPR is consistently low, indicating a failure to recognize rotated variants. Closed-source models perform better than open-source models, but the failure persists across all models. We additionally include results for Qwen2.5-VL-72B and Qwen3-VL-4B.
Encoder
10°
20°
30°
40°
50°
60°
70°
80°
90°
Overall
SigLIP-SO400M-384
86.5
81.8
71.5
57.3
64.0
74.6
84.4
84.9
100.0
78.3
Qwen2.5-VL-7B
94.3
86.1
78.4
68.3
64.1
77.2
87.7
93.5
99.8
83.3
DINOv2 ViT-L/14
98.0
96.3
94.7
83.2
82.4
92.6
95.6
97.3
99.7
93.3
CLIP ViT-L/14-336
91.3
86.2
79.0
69.8
68.1
79.8
85.4
89.4
99.1
83.1
Appendix
Table 13: Angle classification accuracy across vision encoders on Omniglot characters. All encoders substantially exceed chance performance ( 11.1 %), suggesting that rotation information is linearly recoverable from vision encoder features.
Qwen2.5-VL-7B
Qwen2.5-VL-32B
Qwen3-VL-8B
Qwen3-VL-30B
GPT-5.2
Gemini-2.5-Pro
Domain
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Photo
57.41
100.00
14.82
83.73
100.00
67.45
52.09
100.00
4.18
82.14
100.00
64.27
98.91
99.95
97.86
94.70
100.00
89.41
Art Painting
54.70
100.00
9.41
79.55
100.00
59.09
52.91
100.00
5.82
77.23
100.00
54.45
99.02
100.00
98.05
95.80
100.00
91.59
Cartoon
55.23
100.00
10.45
81.89
100.00
63.77
52.80
100.00
5.59
82.50
100.00
65.00
97.95
100.00
95.91
93.23
100.00
86.45
Sketch
53.36
100.00
6.73
73.07
100.00
46.14
52.66
100.00
5.32
76.00
100.00
52.00
95.61
99.64
91.59
90.30
99.86
80.73
Random guess
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
Appendix
Table 14: Rotation recognition performance across PACS domains ( 10∘ – 90∘ ). Best and worst accuracy per model across domains are highlighted in teal and red , respectively. Performance aggregated over rotations from 10∘ to 90∘ in increments of 10∘ . While TNR remains near-perfect across all models, TPR varies significantly across domains, with strong performance on photographs and substantial degradation on sketches, indicating limited geometric robustness.
Qwen2.5-VL-7B
Qwen2.5-VL-32B
Qwen3-VL-8B
Qwen3-VL-30B
GPT-5.2
Gemini-2.5-Pro
Domain
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Acc.
TNR
TPR
Photo
53.50
100.00
7.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
Art Painting
52.75
100.00
5.50
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
Cartoon
76.75
100.00
53.50
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
99.75
100.00
99.50
100.00
100.00
100.00
Sketch
92.25
100.00
84.50
100.00
100.00
100.00
100.00
100.00
100.00
99.75
100.00
99.50
100.00
100.00
100.00
100.00
100.00
100.00
Random guess
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
50.00
Appendix
Table 15: Model performance for the identity transformation on PACS. Best accuracy values per model across domains are highlighted in teal , including ties, while worst values are shown in red . All models achieve near-perfect performance across domains, except Qwen2.5-VL-7B , which shows lower TPR on Photo and Art despite perfect TNR.
Table 16: Model performance with In-Context Learning
Dataset
Prompt Variant
Accuracy
TPR
TNR
Times New Roman
Original
52.56
5.98
99.15
Balanced
51.71
3.42
100.0
Forced Choice
90.49
80.98
100.0
Handwritten English
Original
61.22
32.05
90.38
Balanced
61.97
23.93
100.0
Forced Choice
76.60
99.36
53.85
Appendix
Table 17: Performance metrics across datasets and prompt variants for Qwen2.5-VL-32B. Forced-choice MCQ dramatically improves TPR (+ 65-75 %) on all datasets, demonstrating strong response bias. However, TNR collapse on Handwritten English (53.85%) and Omniglot (39.11%) indicates the model lacks stable geometric reasoning – it trades false negatives for false positives rather than achieving robust rotation understanding.
Dataset
Prompt Variant
Accuracy
TPR
TNR
Times New Roman
Original
89.32
78.6
100.0
Balanced
88.1
76.3
100.0
Forced Choice
87.9
75.9
100.0
Handwritten English
Original
68.3
37.0
99.6
Balanced
69.6
39.5
99.6
Forced Choice
67.2
34.4
100.0
Appendix
Table 18: Performance metrics across datasets and prompt variants for Gemini-2.5-Pro. Different prompt formats shows consistent performance across all datasets, demonstrating that the drop in performance in Handwritten English and Omniglot is not due to prompt sensitivity.
Category
Accuracy
Avg Time
Armenian
100%
2.78s ± 1.13s
Greek
100%
2.70s ± 1.12s
Hand-English
100%
2.46s ± 1.08s
Malayalam
100%
2.91s ± 1.29s
Times New Roman
100%
2.28s ± 1.05s
Appendix
Table 19: Human baseline performance on the rotation task by script category. Humans maintain perfect geometric reasoning regardless of script familiarity, highlighting a fundamental vulnerability in current VLMs.
Dataset
Qwen2.5-VL-7B
Qwen2.5-VL-32B
Qwen3-VL-8B
Qwen3-VL-30B
GPT-5.2
Gemini-2.5-Pro
Times New Roman
51.07 [47.75, 54.17]
52.67 [49.36, 55.67]
50.11 [46.79, 53.21]
65.81 [62.50, 68.80]
74.25 [71.37, 76.92]
89.32 [87.18, 91.24]
Handwritten English
50.85 [47.86, 54.17]
62.50 [59.40, 65.60]
50.00 [47.01, 53.42]
55.98 [52.99, 59.19]
67.84 [64.53, 70.94]
68.27 [65.49, 71.26]
Omniglot
50.72 [50.14, 51.25]
54.17 [53.61, 54.74]
51.01 [50.48, 51.56]
56.64 [56.10, 57.20]
75.55 [75.04, 76.02]
76.90 [76.40, 77.36]
Appendix
Table 20: Rotation task: Accuracy with 95% bootstrap confidence intervals (10,000 resamples). Narrow intervals confirm that the reported results are statistically robust.
Dataset
Qwen2.5-VL-7B
Qwen2.5-VL-32B
Qwen3-VL-8B
Qwen3-VL-30B
GPT-5.2
Gemini-2.5-Pro
Times New Roman
98.80 [97.60, 99.76]
99.76 [99.28, 100.00]
100.00 [100.00, 100.00]
98.79 [97.60, 99.76]
98.79 [97.60, 99.76]
99.51 [98.56, 100.00]
Handwritten English
96.77 [95.16, 98.19]
95.36 [93.54, 97.18]
97.59 [95.19, 99.52]
97.59 [96.15, 98.80]
98.07 [96.63, 99.28]
96.63 [94.23, 99.04]
Omniglot
77.40 [76.68, 78.16]
74.21 [73.48, 74.92]
76.05 [75.34, 76.79]
77.04 [76.32, 77.80]
79.72 [79.00, 80.42]
82.56 [81.91, 83.21]
Appendix
Table 21: Scale task: Accuracy with 95% bootstrap confidence intervals (10,000 resamples). Narrow intervals confirm that the reported results are statistically robust.
Dataset
Molmo2-8B Acc.
Molmo2-8B TNR
Molmo2-8B TPR
InternVL3.5-8B Acc.
InternVL3.5-8B TNR
InternVL3.5-8B TPR
Times New Roman
75.48
100.00
50.96
98.07
100.00
96.15
Handwritten English
73.79
100.00
47.59
82.69
100.00
65.38
Omniglot
56.07
99.61
12.52
64.28
99.35
29.22
Random guess
50.00
50.00
50.00
50.00
50.00
50.00
Appendix
Table 22: Model performance on the scale-invariance task aggregated across all scales for open-sourced models Molmo2-8B and InternVL3.5-8B.
Dataset
Molmo2-8B Acc.
Molmo2-8B TNR
Molmo2-8B TPR
InternVL3.5-8B Acc.
InternVL3.5-8B TNR
InternVL3.5-8B TPR
Times New Roman
51.60
98.08
5.13
58.01
16.03
100.00
Handwritten English
70.41
96.15
44.66
62.82
31.41
94.23
Omniglot
69.09
96.92
41.27
64.83
41.17
88.48
Random guess
50.00
50.00
50.00
50.00
50.00
50.00
Appendix
Table 23: Model performance on the rotation task aggregated across all angles for open-sourced models Molmo2-8B and InternVL3.5-8B.
Model
Task
Acc.
TPR
TNR
Baseline
Rotation
56.36
12.72
100.00
Fine-tuned
Rotation
90.75
81.50
100.00
Baseline
Scale
92.94
86.62
99.25
Fine-tuned
Scale
96.56
93.75
99.38
Appendix
Table 24: LoRA fine-tuning Qwen2.5-VL-7B on semantic VQA pairs (no geometric supervision) of sketch images from PACS improves rotation recognition. Rotation TPR improves by +68.78% while TNR is preserved.
Understanding physical transformations is fundamental for reasoning in dynamic environments. While Vision Language Models (VLMs) show promise in embodied applications, whether they genuinely understand physical transformations remains unclear. We introduce ConservationBench evaluating conservation -- whether physical quantities remain invariant under transformations. Spanning four properties with paired conserving/non-conserving scenarios, we generate and evaluate 23,040 questions across 112 VLMs. Results reveal systematic failure: performance remains near chance with improvements on conservation tasks accompanied by drops on controls. Control experiments show strong textual priors favoring invariance, yet models perform worse with actual visual content when performance is balanced across conserving and non-conserving scenarios. Neither temporal resolution, prompting, nor curated sampling helps. These findings show that current VLMs fail to maintain transformation-invariant representations of physical properties across dynamic scenes.
Dezhi Luo, Yijiang Li, Maijunxian Wang +7
University of Michigan · University of California San Diego · University of California Berkeley +6
The robustness of Vision Language Models (VLMs) is commonly assessed through output-level invariance, implicitly assuming that stable predictions reflect stable multimodal processing. In this work, we argue that this assumption is insufficient. We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness (spatial consistency of vision tokens), alongside standard label-based metrics. Applying this framework to modern VLMs across the SEEDBench, MMMU, and POPE datasets reveals three distinct failure modes. First, models frequently preserve predicted answers while undergoing substantial internal representation drift; for perturbations such as text overlays, this drift approaches the magnitude of inter-image variability, indicating that representations move to regions typically occupied by unrelated inputs despite unchanged outputs. Second, robustness does not improve with scale; larger models achieve higher accuracy but exhibit equal or greater sensitivity, consistent with sharper yet more fragile decision boundaries. Third, we find that perturbations affect tasks differently: they harm reasoning when they disrupt how models combine coarse and fine visual cues, but on the hallucination benchmarks, they can reduce false positives by making models generate more conservative answers.
Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena +6
Sapienza University of Rome · University of Edinburgh · University of Oxford +5
Vision-language models (VLMs) are now widely used in downstream tasks. However, real-world applications often expose VLMs to distribution shifts induced by semantic variation (e.g., shape, size, and style). Robustness certification determines if a model's prediction changes when transformations are applied to its input. While most certification frameworks study geometric or pixel-level transformations over inputs, this work proposes a novel framework that enables certifying VLM robustness under semantic-level transformations. Leveraging the open-vocabulary capability of VLMs, we use text prompts as semantic proxies to construct transformations parameterized by an extent that controls the degree of semantic variation. By characterizing the VLM decision boundary in closed form, our framework quantitatively certifies extent intervals for which the predicted class remains unchanged under the semantic transformation. Our framework is the first to certify VLM robustness under semantic-level variations without requiring additional data for each variation, making it practical to apply. Experiments on both synthetic and real-world data show that our framework enables certifying robustness under diverse semantic variations across scenarios.
Peiyu Yang, Paul Montague, Feng Liu +4
School of Computing & Information Systems, University of Melbourne, Australia · Defence Science and Technology Group, Australia.