The Prestige: Benchmarking Cognitive Visual Reasoning using Magic Tricks

Shuo Wen; Beixi Du; Edwin Meriaux; Junming Shi; Chloe Si; David Meger; Doina Precup; Gregory Dudek

cs.CVOct 28, 2026

The Prestige: Benchmarking Cognitive Visual Reasoning using Magic Tricks

Authors: Shuo Wen, Beixi Du, Edwin Meriaux, Junming Shi, Chloe Si, David Meger, Doina Precup, Gregory Dudek

Organizations: McGill University · Mila – Quebec AI Institute · Cornell University · Université Paris-Saclay, CentraleSupélec

Abstract

We introduce The Prestige, a video understanding benchmark that probes cognitive visual reasoning through professional magic performances, where deliberate misdirection challenges models to track events and explain unexpected outcomes. The benchmark comprises open-ended questions ranging from temporal and causal understanding to robustness against adversarial questions. Evaluation of leading open-source and proprietary VLMs reveals a substantial gap between even the strongest model and human performance. We also find systematic discrepancies between human and automated grading and identify recurring model failure modes. Ablations further show that models can select correct multiple-choice answers even when they fail to generate correct explanations independently, motivating further investigations into question formats. Together, these findings highlight limitations in both VLM reasoning and its evaluation, underscoring the need for more reliable video understanding models and more rigorous benchmarks.

Figures & tables

Figure 1: An example prompt to challenge representative VLMs to explain the elements of surprise in a video sequence. Results highlight current limitations in VLM visual reasoning. All models are evaluated using their latest web versions at the time of testing. Responses are excerpted, and images are cropped for brevity. Project page

Dataset	# videos	# QA	Format	Commonsense	Cognitive Flow	Multi-step	Long-context	Annotation
FunQA [ 31 ]	4,365	311,950	QA	$\checkmark$	$\triangle$	$\times$	$\times$	Manual+LLM
Video-MME-v2 [ 11 ]	800	3,200	8-way MC	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	Manual+LLM
Causal-VidQA [ 15 ]	26,900	107,600	MC	$\checkmark$	$\times$	$\times$	$\times$	Manual
VRBench [ 34 ]	960	8,243	MC&QA	$\times$	$\times$	$\checkmark$	$\checkmark$	Manual+LLM
Video-Holmes [ 8 ]	270	1,837	MC	$\checkmark$	$\times$	$\checkmark$	$\times$	Manual+LLM
CG-Bench [ 4 ]	1,219	12,129	MC&QA	$\times$	$\times$	$\checkmark$	$\checkmark$	Manual

Table 1: Comparison of different video understanding benchmarks. MC stands for multiple choice questions, QA stands for open Question-Answering. Cognitive Flow denotes whether the benchmark centrally evaluates reasoning over videos designed to examine the cognitive visual reasoning of the models.

\triangle

indicates partial coverage.

Figure 2: Examples of the six distinct categories of video reasoning tasks evaluated: Instant Reasoning (IR), Contextual Reasoning (CR), Temporal Summarization (TS), Adversarial Questions—False Implication (AQ.FI) and False Statement (AQ.FS), and Paradoxical Reasoning (PR). Each panel displays a sample question, the ground-truth human answer, and a film strip of the relevant video clip. Images are zoomed in for clarity. Representative frames are only selected for demonstration purposes. In actual evaluations, models receive frames sampled uniformly at 1 FPS over the available temporal interval, up to a maximum duration of four minutes.

Figure 3: Question Type Distribution

Model Size	IR	CR	PR	AQ.FI	AQ.FS	TS	Overall
Open-source models
Qwen3VL-Thinking [ 2 ] 8B	40.00	34.69	9.76	24.46	15.22	17.86	23.84
Qwen3VL-Instruct [ 2 ] 8B	46.34	39.90	17.25	18.88	20.65	29.46	29.12
InternVideo2.5-Chat [ 29 ] 8B	30.24	27.57	3.74	3.86	5.43	8.93	15.20
InternVL3.5 [ 28 ] 8B	25.37	26.81	4.14	19.31	9.78	12.50	16.63
MiMo-VL-SFT-2508 [ 36 ] 7B	22.44	22.11	1.87	21.89	15.22	8.04	14.15

Table 2: Model accuracy by question type across 14 models over the full dataset. All model results are scored by human evaluation.

Open-source models
Model	IR	CR	PR	AQ.FI	AQ.FS	TS	Overall
Qwen3VL-Thinking-8B	7.80	3.81	6.42	3.86	10.87	8.04	5.60
Qwen3VL-Instruct-8B	8.78	5.47	8.96	4.29	9.78	13.39	7.44
InternVideo2.5-Chat-8B	6.83	4.32	2.54	0.86	4.34	7.14	3.72
InternVL3.5-8B	7.81	3.18	2.94	3.86	3.26	10.71	4.00
MiMo-VL-SFT-2508-7B	8.29	2.80	1.07	7.30	8.70	3.58	3.49

Table 3: Signed percentage-point difference between human and machine grading scores, computed as human score minus machine score. The final row reports the mean for each column, with the corresponding standard deviation.

Figure 5: Error Type Distribution

Figure 7: Ablation study on the effects of multiple-choice questions for (a) 50 failure-selected and (b) 200 randomly sampled questions. Stacked bars show the question-type distributions.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Sample question generation with generation tool.

Figure 9: Custom interface used by human evaluators to grade VLM responses.

Model Size	IR	CR	PR	AQ.FI	AQ.FS	TS	Overall
Open-source models
Qwen3VL-Thinking [ 2 ] 8B	32.20	30.88	3.34	20.60	4.35	9.82	18.24
Qwen3VL-Instruct [ 2 ] 8B	37.56	34.43	8.29	14.59	10.87	16.07	21.68
InternVideo2.5-Chat [ 29 ] 8B	23.41	23.25	1.20	3.00	1.09	1.79	11.48
InternVL3.5 [ 28 ] 8B	17.56	23.63	1.20	15.45	6.52	1.79	12.63
MiMo-VL-SFT-2508 [ 36 ] 7B	14.15	19.31	0.80	14.59	6.52	4.46	10.66

Appendix

Table 4: Model accuracy by question type across 14 models over the full dataset. All model results are scored by machine evaluation.

Visual Reasoning

Visual reasoning studies how AI systems interpret and combine visual evidence to solve spatial, relational, compositional, and physical problems.

See all 165 papers

Can LVLMs Uncover the Truth Behind Visual Illusions? An Analysis of Perceptual and Reasoning Capabilities
Jul 30, 2026Liangjie Zhao, Jiaqing Lyu, Kexin Tang +5VLM Evaluation Visual Reasoning
Adelaide University · Tsinghua University · Amap, Alibaba Group +3
Not Another Text Benchmark: Putting the "Visual" Back in Visual Question Answering for Large Video Models
Sep 15, 2026Rwiddhi Chakraborty, Yinong Oliver Wang, Cheng Zhang +6Video-Language Model Evaluation Video QA
Oliver · University of Copenhagen · Carnegie Mellon University +4
Accuracy Without Grounding: Diagnosing Visual Dependency Dissociation in Video LLM Benchmarks
Jul 14, 2026Jae Joong LeeVLM Evaluation Video-Language Model Evaluation
Department of Computer Science Purdue University West Lafayette, Indiana, USA

Jul 30, 2026J/K move · Enter open · S save

Can LVLMs Uncover the Truth Behind Visual Illusions? An Analysis of Perceptual and Reasoning Capabilities

Liangjie Zhao, Jiaqing Lyu, Kexin Tang, Zecheng Fang+4

Adelaide University · Tsinghua University · Amap, Alibaba Group+3

VLM Evaluation Visual Reasoning VLM Reasoning

Open paper AskPDF arXiv

Large Vision Language Models have integrated reasoning capabilities, elevating cognitive performance to new levels. However, existing evaluations either focus solely on perception or rely on specific domains such as maths or coding. Evaluation for reasoning capabilities that align with an open-world environment is still required, especially one that considers perception and reasoning jointly. To bridge this gap, we propose to evaluate LVLMs by exploiting visual illusions as a diagnostic tool. Visual illusions are phenomena in which the human visual system misinterprets objective signals, resulting in an understanding that deviates from reality. We constructed IllusionReasoning, a benchmark of illusion images collected from the real world, incorporating diverse annotated question-answer pairs. Based on IllusionReasoning, we show that the reasoning capabilities of a wide range of LVLMs are not as advanced as claimed. Our work provides new insights into LVLMs and offers future direction for optimisation.

Abstract

Visual Reasoning

Explore similar work

Can LVLMs Uncover the Truth Behind Visual Illusions? An Analysis of Perceptual and Reasoning Capabilities

Not Another Text Benchmark: Putting the "Visual" Back in Visual Question Answering for Large Video Models

Accuracy Without Grounding: Diagnosing Visual Dependency Dissociation in Video LLM Benchmarks