cs.CVNov 28, 2025

Breaking the Visual Shortcuts in Multimodal Knowledge-Based Visual Question Answering

Authors: Dosung Lee, Sangwon Jung, Boyoung Kim, Minyoung Kim, Sungyeon Kim, Junyoung Sung, Wonpyo Park, Paul Hongsuck Seo

Organizations: Korea University · KAIST · Amazon

Abstract

Existing Multimodal Knowledge-Based Visual Question Answering (MKB-VQA) benchmarks suffer from "visual shortcuts", as the query image typically matches the primary subject entity of the target document. We demonstrate that models can exploit these shortcuts, achieving comparable results using visual cues alone. To address this, we introduce Relational Entity Text-Image kNowledge Augmented (RETINA) benchmark, automatically constructed using an LLM-driven pipeline, consisting of 120k training and 2k human-curated test set. RETINA contains queries referencing secondary subjects (i.e. related entities) and pairs them with images of these related entities, removing the visual shortcut. When evaluated on RETINA existing models show significantly degraded performance, confirming their reliance on the shortcut. Furthermore, we propose Multi-Image MultImodal Retriever (MIMIR), which enriches document embeddings by augmenting images of multiple related entities, effectively handling RETINA, unlike prior work that uses only a single image per document. Our experiments validate the limitations of existing benchmarks and demonstrate the effectiveness of RETINA and MIMIR. Our project is available at: Project Page.

Figures & tables

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. M3^3-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering

    Apr 28, 2026Jiatong Ma, Longteng Guo, Yuchen Liu +4Multimodal UnderstandingVisual Question Answering

  2. Look Twice: Training-Free Evidence Highlighting for Knowledge-based Visual Question Answering

    Apr 1, 2026Marco Morini, Sara Sarto, Marcella Cornia +2Knowledge-Based Visual Question AnsweringVLM Inference

  3. Beyond Visual Similarity: Entity-Aligned Retrieval for Knowledge-Based Visual Question Answering

    Aug 19, 2026Hangrui Xu, Zhengxian Wu, Yunyao Yu +6Knowledge-Based Visual Question AnsweringVision-Language Retrieval