Vision-language models (VLMs) can recognize events in images and parse sentences, but it remains unclear when they bind visual Agent-Patient structure to linguistic semantic roles across languages whose role cues depend on morphology, word order or construction. We introduce ROLECUE, a controlled diagnostic that pairs 192 validated two-person action images with German, Turkish, Japanese, English, Korean and Bangla sentences in canonical and patient-first orders. The evaluation separates image-only role recognition, text-only role parsing, direct image-sentence matching, response-compliance behavior and an external modular composition of the model's own component predictions. Across eight recent open-weight VLMs, ROLECUE reveals a structured spectrum: some models fail visual or linguistic component tests, some largely solve the diagnostic, some collapse to constant semantic assent and several show component-joint gaps in which reliable image-only and text-only role predictions do not survive direct joint inference. The largest non-degenerate gaps reach 44.9 percentage points, with final bootstrap intervals confirming that the effect is not a point-estimate artifact. These results reveal multilingual component-joint performance gaps and acceptance biases hidden by aggregate VLM benchmarks and show why component diagnostics are necessary before interpreting joint multimodal errors.
Figures & tables
Figure 1: ROLECUE independently varies visual role, grammatical role, and noun order. The figure shows the English condition for readability; the same controlled manipulation is instantiated with validated German, Turkish, Japanese, English, Korean, and Bangla sentences. Patient-first order makes first-noun heuristics fail while grammatical role cues still identify the sentence agent.
Model
Primary set
Images
Primary acc.
Full acc.
Chasing
Gate
Qwen3-VL-8B
full
192
99.0
99.0
87.5
✓
Gemma-4-12B-it
full
192
100.0
100.0
100.0
✓
InternVL3.5-8B
no-chasing
176
99.4
91.7
6.2
no-chasing
LLaVA-OneVision-7B
no-chasing
176
100.0
96.4
56.2
no-chasing
MiniCPM-V-4.5
no-chasing
176
100.0
96.4
56.2
no-chasing
Qwen2.5-VL-32B
no-chasing
176
99.4
97.9
81.2
no-chasing
Table 1: Image-only visual-role gate on ROLECUE -192. Primary acc. is computed on the set used for integration analysis: all 192 images for full-set models and the 176-image no-chasing subset for restricted models. Full acc. is shown for transparency, and Chasing gives the action-level score that determines the subset.
Figure 2: ROLECUE -192 diagnostic overview. (a) Patient-first direct-joint accuracy across eight open-weight models and six language–construction conditions; the full or no-chasing evaluation set follows the independently determined image-only gate. (b) Direct joint inference versus external modular composition in non-degenerate component-competent settings, where image-only and text-only accuracy are at least 85% and the direct joint output is not a constant semantic response. We retain each model’s strongest qualifying recovery and show the six largest model-level gains. The dashed line is the 50% random baseline; gains are percentage points. Complete agent-first, patient-first, text-only, response-compliance, and modular results appear in the supplement.
Model
German
Japanese
Turkish
English
Korean
Bangla
Qwen3-VL-8B
87.5
100.0
100.0
16.7
100.0
45.8
Gemma-4-12B-it
87.5
70.8
100.0
87.5
100.0
100.0
InternVL3.5-8B
59.1
100.0
54.5
36.4
95.5
45.5
LLaVA-OneVision-7B
45.5
95.5
36.4
77.3
90.9
54.5
MiniCPM-V-4.5
15.6
41.8
1.4
11.4
41.5
12.5
Qwen2.5-VL-32B
36.4
100.0
100.0
100.0
100.0
100.0
Table 2: Patient-first text-only role-parsing accuracy. Values are computed over unique sentence templates per language/order condition (24 for full-set models, 22 for no-chasing models). Comparing this table with Figure 2 (a) separates sentence-role parsing failures from joint matching failures.
Model
Language
Order
Image
Text
Joint
Pred. match
Gap
InternVL3.5-8B
Japanese
Patient-first
99.4
100.0
54.5
94.9
44.9
InternVL3.5-8B
Turkish
Agent-first
99.4
100.0
56.2
90.9
43.2
InternVL3.5-8B
Bangla
Agent-first
99.4
100.0
56.5
93.5
42.9
Qwen3-VL-8B
Japanese
Agent-first
99.0
100.0
63.3
84.1
35.7
LLaVA-OneVision-7B
German
Agent-first
100.0
100.0
64.8
85.2
35.2
Qwen3-VL-8B
Turkish
Patient-first
99.0
100.0
65.1
73.4
33.9
Table 3: Largest non-degenerate integration gaps among settings with strong component performance. Gap is modular accuracy minus direct joint accuracy. Pred. match is the percentage of direct-joint trials where the model predicts that the sentence matches the image; near-constant match cells are reported as response-compliance failures in the supplement.
Multilingual vision-language model evaluation measures how well models interpret images, documents, video, and other visual inputs across languages and cultures.
Vision-Language-Action (VLA) models have shown strong performance in language-conditioned robotic manipulation, yet their robustness to linguistic variation remains poorly understood. In this work, we present the first systematic multilingual evaluation of VLA models by translating the LIBERO benchmark into ten languages, revealing severe performance degradation under non-English instructions, with success rates dropping by 30-50%. Through fine-grained analysis of task executions, we find that language influence is highly non-uniform across steps: certain steps exhibit strong language dependence and dominate overall task failure, while others are largely language-agnostic. Based on this insight, we propose a step-wise inference-time intervention that aligns representations according to step language sensitivity, substantially improving performance under linguistic variation. Our results indicate that language robustness in VLA models is fundamentally a step-wise control problem, highlighting the importance of temporally structured analysis for reliable embodied agents.
When a Vision-Language Model (VLM) sees a blue banana and answers "yellow", is the problem of perception or arbitration? We explore the question in ten VLMs with various sizes and reveal an Encoding-Grounding Dissociation: models that fail to report what they see (and thus provide a wrong answer) still encode the visual evidence as strongly as models that provide the correct answer. Using Multimodal Arbitration Crossover (MAC) analysis with layer-by-layer Logit Lens probing, we track the competition between visual and prior signals across every layer of each model. We show that visual attributes can be linearly decodable from early layers (AUC > 0.86). The accuracy remains nearly identical for both successful and failed samples. However, the gap in the final-layer logit - not the strength of encoding - better predicts grounding outcomes with a correlation of ρ= 0.847. After having studied when VLMs base their answers on image clues rather than prior knowledge, we want to understand the causal relationships. We establish causality through full-sequence activation patching. The standard last-token interventions in LLM interpretability do not affect VLMs. In contrast, replacing the full token sequence at layers identified by MAC alters 60 to 84% of outputs. Partial-token decomposition shows that image tokens carry almost all of the causal impact, while text tokens have none. Scaling addresses the remaining architectural differences to achieve perfect retention. Moving from diagnosis to intervention, we show that training-free activation steering - both linear and sparse autoencoder-guided - in early layers can improve visual grounding by up to +3.8% with degrading performance in some setups. Overall, these findings lead to a clear conclusion: VLMs already see well, but the challenge is acting on what they see. Targeted interventions can help to bridge this gap.
Farhad Nooralahzadeh, Omid Rohanian, Yi Zhang +2
Institute of Computer Science, Zurich University of Applied Sciences, Switzerland · University of Oxford, UK.
Vision-language models (VLMs) rely on storing and transferring appropriate information across their sub-components. Verifying that the VLMs exhibit desired behaviours, while avoiding harmful ones, is central to their reliable deployment. Yet, methods that identify VLM behaviours remain scarce. We present DiaVLo, a diagnostic framework that leverages human curation and VLMs' generation capabilities to construct specifications of desired and observed VLM behaviours, surfacing potential misalignments. Beyond this, DiaVLo also provides causal estimates to identify the most influential concepts steering VLM behaviours. We evaluate DiaVLo on several open-source VLMs under both classification and generation conditions. Our experiments show that DiaVLo produces behaviour labels that correlate with model performance and provide context for measured performance. DiaVLo surfaced behaviours that are clearly aligned and misaligned, alongside patterns in how VLMs perceive, organise, and prioritise concepts.