cs.CVNov 7, 2026

ROLECUE: Diagnosing Cross-Lingual Role-Binding Failures in Vision-Language Models

Authors: Niaz Mohaiman Abtahi, Evan Maksud

Organizations: Advanced AI Lab Limited

Abstract

Vision-language models (VLMs) can recognize events in images and parse sentences, but it remains unclear when they bind visual Agent-Patient structure to linguistic semantic roles across languages whose role cues depend on morphology, word order or construction. We introduce ROLECUE, a controlled diagnostic that pairs 192 validated two-person action images with German, Turkish, Japanese, English, Korean and Bangla sentences in canonical and patient-first orders. The evaluation separates image-only role recognition, text-only role parsing, direct image-sentence matching, response-compliance behavior and an external modular composition of the model's own component predictions. Across eight recent open-weight VLMs, ROLECUE reveals a structured spectrum: some models fail visual or linguistic component tests, some largely solve the diagnostic, some collapse to constant semantic assent and several show component-joint gaps in which reliable image-only and text-only role predictions do not survive direct joint inference. The largest non-degenerate gaps reach 44.9 percentage points, with final bootstrap intervals confirming that the effect is not a point-estimate artifact. These results reveal multilingual component-joint performance gaps and acceptance biases hidden by aggregate VLM benchmarks and show why component diagnostics are necessary before interpreting joint multimodal errors.

Figures & tables

Explore similar work

CardsList
  1. Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts

    Apr 10, 2026Farhad Nooralahzadeh, Omid Rohanian, Yi Zhang +2Vision-Language GroundingVisual Evidence Sensitivity in VLMs

  2. DiaVLo: Diagnosing Behaviours of Vision-Language Models

    Sep 18, 2026Lorenzo Corti, Jie YangVLM EvaluationVLM Interpretability