Multimodal Models Can See, But Can They Think? A Study of Turkish Visual Reasoning
Authors: Berke Bunyamin Sule, Yavuz Selim Aygan, Ahmed Zeer, Elif Ince, Ali Mert Temizsoy, Ozgur Agar, Himmet Toprak Kesgin, Muzaffer Kaan Yuce, Mehmet Fatih Amasyali
Organizations: Department of Computer Engineering, Yildiz Technical University, Istanbul, Turkiye
Vision-Language Models have had massive leaps in recent years. Yet, when tested on high-level cognitive tasks like pattern completion, abstract logic, and relational reasoning, they still fall short. This limitation stems from the models' lack of ability to find out underlying rules and maintain constant intermediate steps, rather than merely deciding on the correct option. To research these shortcomings, this study analyzes visual reasoning performance throughout a multilingual evaluation framework encompassing the BİLSEM, MM-IQ, Visual Puzzles, and PuzzleVQA datasets. We evaluate various data mixing strategies and supervised fine-tuning (SFT) methods. We observed that a correct answer does not always mean the reasoning chain is valid. Because of this, we adopt an LLM-as-a-Judge approach to assess the quality of reasoning beyond final answer accuracy. Our experimental results reveal that baseline models often exhibit more stable reasoning compared to fine-tuned counterparts. Most importantly, we observe that SFT can lead to model collapse, manifesting as infinite text loops or empty outputs. The study concludes that current VLMs still struggle with abstract visual reasoning.
Figures & tables
Figure 1: An overview of the proposed evaluation and training framework.
Figure 2: Representative samples from different datasets used in this study.
Figure 3: A sample from the PuzzleVQA dataset with three-stage reasoning annotation (caption, explanation, deduction).
Data Source
Features
Train
Test
Total
MM-IQ
Question and Answer
2,168
100
2,268
Visual Puzzles
Question and Answer
366
100
466
PuzzleVQA
Question, Answer, and Solution
1,600
100
1,700
BİLSEM Questions
Question, Answer, and Solution
706
100
806
Total
4,840
400
5,240
Table 1: Summary of dataset sources and data splitting
Item
Token
Amount (USD)
Input
4,929,410
0.2462
Output
14,968,788
5.9875
Total
19,898,198
6.2337
Table 2: Token usage and cost obtained from billing records for Qwen3-VL-Flash (Alibaba Cloud).
Scenario
Input (M)
Output (M)
Total (M)
Cost (USD)
Low
2.628
2.000
4.628
14.628
Medium
4.028
3.400
7.428
24.428
High
5.428
4.800
10.228
34.228
Table 3: Approximate token usage and cost for Gemini 3 Pro Preview (Batch).
Parameter
Value
LoRA Rank
16
LoRA Alpha
32
Effective Batch Size
16
Learning Rate
5×10−5
Scheduler
Cosine
Epoch
2
Table 4: Training Hyperparameters
Criterion
QwenFlash
Gemini 3 Pro
Diff.
Accuracy (%)
89.0
100.0
+11.0
Logical Consistency
50.05
89.91
+39.86
Conciseness and Clarity
35.75
89.90
+54.15
Reference Alignment
42.05
85.96
+43.91
Table 5: Rationale quality evaluation results (LLM-as-a-Judge) on PuzzleVQA: Qwen3-VL-Flash vs. Gemini 3 Pro.
Model
Dataset Configuration (SFT)
General Success (%)
PuzzleVQA (%)
BİLSEM (%)
Visual Puzzle (%)
Qwen3VL-8B
Visual (3pro)
32.0
34.0
42.0
20.0
Qwen3VL-8B
MMIQ (3pro)
28.0
36.0
24.0
24.0
Qwen3VL-8B
PuzzleVQA (3pro)
36.7
52.0
24.0
34.0
Qwen3VL-8B
Visual + MMIQ (3pro)
27.3
30.0
24.0
28.0
Qwen3VL-8B
Visual + PuzzleVQA (3pro)
37.3
52.0
28.0
32.0
Qwen3VL-8B
PuzzleVQA + MMIQ (3pro)
48.8
72.0
26.7
22.0
Table 6: Initial Baseline Selection: Model and Data Mixture Comparison on the Test Set
Model
Training Data
Preprocessing
General Success (%)
PuzzleVQA (%)
BİLSEM (%)
Visual Puzzle (%)
Qwen3VL-8B
SFT (3pro)
None
48.8
72.0
26.7
22.0
Qwen3VL-8B
SFT (3pro + QwenFlash)
None
32.3
41.0
23.4
24.0
Qwen3VL-8B
SFT (QwenFlash)
None
30.5
38.5
23.3
20.0
Qwen3VL-8B
SFT (3pro + QwenFlash)
Regex Filter
27.0
36.0
20.0
12.0
Qwen3VL-8B
SFT (QwenFlash)
Regex Filter
30.0
28.0
34.0
18.0
Qwen3VL-8B
SFT (3pro + QwenFlash)
MultiSFT
40.5
54.0
26.0
30.0
Table 7: Baseline Extension and Data-Quality Refinement on the Test Set
Question
Generated Output (Model Collapse)
Correct Answer: B) 8
“Görüntüde örtüşen üç numaralı çember vardır. Birinci çember ’?’ numaralı, ikinci çember 10 numaralı ve üçüncü çember 3 numaralı… Dolayısıyla desen, örtüşen kısımlardaki sayıların toplamı 10 sayısını verir. Örtüşen kısımlardaki sayıların toplamı 10 olduğuna dayanan desene göre, örtüşen kısımlardaki sayılar toplamı 10 olduğuna göre örtüşen…” [Output truncated due to infinite repetition loop]
Table 8: A clear instance of model collapse observed after SFT of the Qwen3VL-8B model using the PuzzleVQA dataset. The model fails to converge and falls into an infinite repetition loop.
Model
Training Data
General Success (%)
Empty Output (%)
Repetition (%)
Avg. Output Words
Qwen3VL-8B
SFT (3pro)
48.8
1.00
9.00
215.3
Qwen3VL-8B
SFT (3pro + QwenFlash)
32.3
42.25
52.00
552.7
Qwen3VL-8B
SFT (QwenFlash)
30.5
42.25
51.25
553.9
Qwen3VL-8B
MultiSFT (3pro + QwenFlash)
40.5
6.25
13.25
226.8
Table 9: Output-level diagnostics of SFT-induced collapse on the 400-sample test set.
Reasoning in vision-language models (VLMs) has recently attracted significant attention due to its broad applicability across diverse downstream tasks. However, it remains unclear whether the superior performance of VLMs stems from genuine vision-grounded reasoning or relies predominantly on the reasoning capabilities of their textual backbones. To systematically measure this, we introduce CrossMath, a novel multimodal reasoning benchmark designed for controlled cross-modal comparisons. Specifically, we construct each problem in text-only, image-only, and image+text formats guaranteeing identical task-relevant information, verified by human annotators. This rigorous alignment effectively isolates modality-specific reasoning differences while eliminating confounding factors such as information mismatch. Extensive evaluation of state-of-the-art VLMs reveals a consistent phenomenon: a substantial performance gap between textual and visual reasoning. Notably, VLMs excel with text-only inputs, whereas incorporating visual data (image+text) frequently degrades performance compared to the text-only baseline. These findings indicate that current VLMs conduct reasoning primarily in the textual space, with limited genuine reliance on visual evidence. To mitigate this limitation, we curate a CrossMath training set for VLM fine-tuning. Empirical evaluations demonstrate that fine-tuning on this training set significantly boosts reasoning performance across all individual and joint modalities, while yielding robust gains on two general visual reasoning tasks. Source code is available at https://github.com/xuyige/CrossMath.
Yige Xu, Yongjie Wang, Zizhuo Wu +3
College of Computing and Data Science, Nanyang Technological University, Singapore · Alibaba-NTU Global e-Sustainability CorpLab (ANGEL) · Tongyi Lab, Alibaba Group, China
Despite the impressive performance of vision-language models (VLMs) on downstream tasks, their ability to understand and reason about causal relationships in visual inputs remains unclear. Robust causal reasoning is fundamental to solving complex high-level reasoning tasks, yet existing benchmarks often include a mixture of reasoning questions, and VLMs can frequently exploit object recognition and activity identification as shortcuts to arrive at the correct answers, making it challenging to truly assess their causal reasoning abilities. To bridge this gap, we introduce VQA-Causal and VCR-Causal, two new benchmarks specifically designed to isolate and rigorously evaluate VLMs' causal reasoning abilities. Our findings reveal that while VLMs excel in object and activity recognition, they perform poorly on causal reasoning tasks, often only marginally surpassing random guessing. Further analysis suggests that this limitation stems from a severe lack of causal expressions in widely used training datasets, where causal relationships are rarely explicitly conveyed. We additionally explore fine-tuning strategies with hard negative cases, showing that targeted fine-tuning can improve model's causal reasoning while maintaining generalization and downstream performance. Our study highlights a key gap in current VLMs and lays the groundwork for future work on causal understanding.
Zhaotian Weng, Haoxuan Li, Xin Eric Wang +2
University of Southern California · University of California, Santa Barbara · Texas A&M University
Vision-Language Models (VLMs) achieve strong performance in visual reasoning tasks, but it remains unclear whether they understand visual relations, or simply employ shortcuts such as language cues or priors. To investigate this, we use the Qwen3-VL-4B (Bai et al., 2025), a modern VLM, to decode how visual information is encoded across depths. For this, we propose a synthetic dataset of simple geometric shapes for controlled analysis, along with queries crafted to precisely test language cues. Furthermore, the dataset is modified to test causal reliance on visual evidence. Our results show that current VLMs combine genuine visual reasoning with shortcut strategies primarily rooted in language cues.