Multimodal Models Can See, But Can They Think? A Study of Turkish Visual Reasoning
Organizations: Department of Computer Engineering, Yildiz Technical University, Istanbul, Turkiye
Abstract
Vision-Language Models have had massive leaps in recent years. Yet, when tested on high-level cognitive tasks like pattern completion, abstract logic, and relational reasoning, they still fall short. This limitation stems from the models' lack of ability to find out underlying rules and maintain constant intermediate steps, rather than merely deciding on the correct option. To research these shortcomings, this study analyzes visual reasoning performance throughout a multilingual evaluation framework encompassing the BİLSEM, MM-IQ, Visual Puzzles, and PuzzleVQA datasets. We evaluate various data mixing strategies and supervised fine-tuning (SFT) methods. We observed that a correct answer does not always mean the reasoning chain is valid. Because of this, we adopt an LLM-as-a-Judge approach to assess the quality of reasoning beyond final answer accuracy. Our experimental results reveal that baseline models often exhibit more stable reasoning compared to fine-tuned counterparts. Most importantly, we observe that SFT can lead to model collapse, manifesting as infinite text loops or empty outputs. The study concludes that current VLMs still struggle with abstract visual reasoning.
Figures & tables
| Data Source | Features | Train | Test | Total |
|---|---|---|---|---|
| MM-IQ | Question and Answer | 2,168 | 100 | 2,268 |
| Visual Puzzles | Question and Answer | 366 | 100 | 466 |
| PuzzleVQA | Question, Answer, and Solution | 1,600 | 100 | 1,700 |
| BİLSEM Questions | Question, Answer, and Solution | 706 | 100 | 806 |
| Total | 4,840 | 400 | 5,240 |
| Item | Token | Amount (USD) |
|---|---|---|
| Input | 4,929,410 | 0.2462 |
| Output | 14,968,788 | 5.9875 |
| Total | 19,898,198 | 6.2337 |
| Scenario | Input (M) | Output (M) | Total (M) | Cost (USD) |
|---|---|---|---|---|
| Low | 2.628 | 2.000 | 4.628 | 14.628 |
| Medium | 4.028 | 3.400 | 7.428 | 24.428 |
| High | 5.428 | 4.800 | 10.228 | 34.228 |
| Parameter | Value |
|---|---|
| LoRA Rank | 16 |
| LoRA Alpha | 32 |
| Effective Batch Size | 16 |
| Learning Rate | |
| Scheduler | Cosine |
| Epoch | 2 |
| Criterion | QwenFlash | Gemini 3 Pro | Diff. |
|---|---|---|---|
| Accuracy (%) | 89.0 | 100.0 | +11.0 |
| Logical Consistency | 50.05 | 89.91 | +39.86 |
| Conciseness and Clarity | 35.75 | 89.90 | +54.15 |
| Reference Alignment | 42.05 | 85.96 | +43.91 |
| Model | Dataset Configuration (SFT) | General Success (%) | PuzzleVQA (%) | BİLSEM (%) | Visual Puzzle (%) |
|---|---|---|---|---|---|
| Qwen3VL-8B | Visual (3pro) | 32.0 | 34.0 | 42.0 | 20.0 |
| Qwen3VL-8B | MMIQ (3pro) | 28.0 | 36.0 | 24.0 | 24.0 |
| Qwen3VL-8B | PuzzleVQA (3pro) | 36.7 | 52.0 | 24.0 | 34.0 |
| Qwen3VL-8B | Visual + MMIQ (3pro) | 27.3 | 30.0 | 24.0 | 28.0 |
| Qwen3VL-8B | Visual + PuzzleVQA (3pro) | 37.3 | 52.0 | 28.0 | 32.0 |
| Qwen3VL-8B | PuzzleVQA + MMIQ (3pro) | 48.8 | 72.0 | 26.7 | 22.0 |
| Model | Training Data | Preprocessing | General Success (%) | PuzzleVQA (%) | BİLSEM (%) | Visual Puzzle (%) |
|---|---|---|---|---|---|---|
| Qwen3VL-8B | SFT (3pro) | None | 48.8 | 72.0 | 26.7 | 22.0 |
| Qwen3VL-8B | SFT (3pro + QwenFlash) | None | 32.3 | 41.0 | 23.4 | 24.0 |
| Qwen3VL-8B | SFT (QwenFlash) | None | 30.5 | 38.5 | 23.3 | 20.0 |
| Qwen3VL-8B | SFT (3pro + QwenFlash) | Regex Filter | 27.0 | 36.0 | 20.0 | 12.0 |
| Qwen3VL-8B | SFT (QwenFlash) | Regex Filter | 30.0 | 28.0 | 34.0 | 18.0 |
| Qwen3VL-8B | SFT (3pro + QwenFlash) | MultiSFT | 40.5 | 54.0 | 26.0 | 30.0 |
| Question | Generated Output (Model Collapse) |
| Correct Answer: B) 8 | “Görüntüde örtüşen üç numaralı çember vardır. Birinci çember ’?’ numaralı, ikinci çember 10 numaralı ve üçüncü çember 3 numaralı… Dolayısıyla desen, örtüşen kısımlardaki sayıların toplamı 10 sayısını verir. Örtüşen kısımlardaki sayıların toplamı 10 olduğuna dayanan desene göre, örtüşen kısımlardaki sayılar toplamı 10 olduğuna göre örtüşen…” [Output truncated due to infinite repetition loop] |
| Model | Training Data | General Success (%) | Empty Output (%) | Repetition (%) | Avg. Output Words |
|---|---|---|---|---|---|
| Qwen3VL-8B | SFT (3pro) | 48.8 | 1.00 | 9.00 | 215.3 |
| Qwen3VL-8B | SFT (3pro + QwenFlash) | 32.3 | 42.25 | 52.00 | 552.7 |
| Qwen3VL-8B | SFT (QwenFlash) | 30.5 | 42.25 | 51.25 | 553.9 |
| Qwen3VL-8B | MultiSFT (3pro + QwenFlash) | 40.5 | 6.25 | 13.25 | 226.8 |