Hidden Meanings in Plain Sight: RebusBench for Evaluating Cognitive Visual Reasoning

Vision-language model (VLM) errors on visual reasoning may come from missing visual structure, from mistakes after that structure is extracted, or from both. We introduce VRIQ, a benchmark of 1,391 human-verified IQ-style visual reasoning puzzles spanning five psychometric categories, each instantiated in category-matched abstract and natural-image domains. To study perception and reasoning more directly, we also introduce DiagVRIQ, a 209-item diagnostic subset paired with human-verified, rule-neutral textual descriptions. Across 21 VLMs, the 17 non-frontier models reach a 26.8% unweighted category mean on abstract puzzles, while three 2026 frontier models improve substantially but unevenly, with unweighted category means of 41.3--70.9% on abstract puzzles and at most 78.2% on natural-image puzzles; all remain below human performance. On DiagVRIQ, even when their descriptions are judged correct, the strongest models still answer approximately 9--12% of those items incorrectly. Two interventions, ground-truth description augmentation and tool-augmented inference, improve accuracy, with the largest tool gains on spatially demanding categories, but neither eliminates errors. VRIQ and DiagVRIQ provide a framework for studying visual reasoning through complementary end-to-end, description-based, and intervention-based analyses rather than treating it as a single monolithic capability.

Large Vision Language Models have integrated reasoning capabilities, elevating cognitive performance to new levels. However, existing evaluations either focus solely on perception or rely on specific domains such as maths or coding. Evaluation for reasoning capabilities that align with an open-world environment is still required, especially one that considers perception and reasoning jointly. To bridge this gap, we propose to evaluate LVLMs by exploiting visual illusions as a diagnostic tool. Visual illusions are phenomena in which the human visual system misinterprets objective signals, resulting in an understanding that deviates from reality. We constructed IllusionReasoning, a benchmark of illusion images collected from the real world, incorporating diverse annotated question-answer pairs. Based on IllusionReasoning, we show that the reasoning capabilities of a wide range of LVLMs are not as advanced as claimed. Our work provides new insights into LVLMs and offers future direction for optimisation.

Recent multimodal large language models (MLLMs) achieve strong performance on visual reasoning benchmarks, yet it remains unclear to what extent such performance reflects reasoning directly grounded in visual evidence. We introduce VisReason, a benchmark for vision-centric reasoning in everyday scenarios where perception and inference are tightly coupled. VisReason contains 1,505 questions across 10 categories spanning perceptual, structural, and conceptual reasoning. Our evaluation shows that VisReason poses a qualitatively different challenge from existing benchmarks, exposing substantial gaps between humans and current MLLMs and revealing limited benefits from test-time reasoning strategies. VisReason offers a focused diagnostic for evaluating vision-centric reasoning beyond language.