ROLECUE: Diagnosing Cross-Lingual Role-Binding Failures in Vision-Language Models
Organizations: Advanced AI Lab Limited
Abstract
Vision-language models (VLMs) can recognize events in images and parse sentences, but it remains unclear when they bind visual Agent-Patient structure to linguistic semantic roles across languages whose role cues depend on morphology, word order or construction. We introduce ROLECUE, a controlled diagnostic that pairs 192 validated two-person action images with German, Turkish, Japanese, English, Korean and Bangla sentences in canonical and patient-first orders. The evaluation separates image-only role recognition, text-only role parsing, direct image-sentence matching, response-compliance behavior and an external modular composition of the model's own component predictions. Across eight recent open-weight VLMs, ROLECUE reveals a structured spectrum: some models fail visual or linguistic component tests, some largely solve the diagnostic, some collapse to constant semantic assent and several show component-joint gaps in which reliable image-only and text-only role predictions do not survive direct joint inference. The largest non-degenerate gaps reach 44.9 percentage points, with final bootstrap intervals confirming that the effect is not a point-estimate artifact. These results reveal multilingual component-joint performance gaps and acceptance biases hidden by aggregate VLM benchmarks and show why component diagnostics are necessary before interpreting joint multimodal errors.
Figures & tables
| Model | Primary set | Images | Primary acc. | Full acc. | Chasing | Gate |
| Qwen3-VL-8B | full | 192 | 99.0 | 99.0 | 87.5 | ✓ |
| Gemma-4-12B-it | full | 192 | 100.0 | 100.0 | 100.0 | ✓ |
| InternVL3.5-8B | no-chasing | 176 | 99.4 | 91.7 | 6.2 | no-chasing |
| LLaVA-OneVision-7B | no-chasing | 176 | 100.0 | 96.4 | 56.2 | no-chasing |
| MiniCPM-V-4.5 | no-chasing | 176 | 100.0 | 96.4 | 56.2 | no-chasing |
| Qwen2.5-VL-32B | no-chasing | 176 | 99.4 | 97.9 | 81.2 | no-chasing |
| Model | German | Japanese | Turkish | English | Korean | Bangla |
| Qwen3-VL-8B | 87.5 | 100.0 | 100.0 | 16.7 | 100.0 | 45.8 |
| Gemma-4-12B-it | 87.5 | 70.8 | 100.0 | 87.5 | 100.0 | 100.0 |
| InternVL3.5-8B | 59.1 | 100.0 | 54.5 | 36.4 | 95.5 | 45.5 |
| LLaVA-OneVision-7B | 45.5 | 95.5 | 36.4 | 77.3 | 90.9 | 54.5 |
| MiniCPM-V-4.5 | 15.6 | 41.8 | 1.4 | 11.4 | 41.5 | 12.5 |
| Qwen2.5-VL-32B | 36.4 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 |
| Model | Language | Order | Image | Text | Joint | Pred. match | Gap |
| InternVL3.5-8B | Japanese | Patient-first | 99.4 | 100.0 | 54.5 | 94.9 | 44.9 |
| InternVL3.5-8B | Turkish | Agent-first | 99.4 | 100.0 | 56.2 | 90.9 | 43.2 |
| InternVL3.5-8B | Bangla | Agent-first | 99.4 | 100.0 | 56.5 | 93.5 | 42.9 |
| Qwen3-VL-8B | Japanese | Agent-first | 99.0 | 100.0 | 63.3 | 84.1 | 35.7 |
| LLaVA-OneVision-7B | German | Agent-first | 100.0 | 100.0 | 64.8 | 85.2 | 35.2 |
| Qwen3-VL-8B | Turkish | Patient-first | 99.0 | 100.0 | 65.1 | 73.4 | 33.9 |
Multilingual VLM Evaluation
Multilingual vision-language model evaluation measures how well models interpret images, documents, video, and other visual inputs across languages and cultures.
See all 61 papers