cs.CVMar 27, 2026

PerceptionComp: A Video Benchmark for Complex Perception-Centric Reasoning

Authors: Shaoxuan Li, Zhixuan Zhao, Hanze Deng, Zirun Ma, Shulin Tian, Zuyan Liu, Yushi Hu, Haoning Wu, 

Organizations: Tsinghua University · Nanyang Technological University · University of Washington

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We introduce PerceptionComp, a manually annotated benchmark for complex, long-horizon, perception-centric video reasoning. PerceptionComp is designed so that no single moment is sufficient: answering each question requires multiple temporally separated pieces of visual evidence and compositional constraints under conjunctive and sequential logic, spanning perceptual subtasks such as objects, attributes, relations, locations, actions, and events, and requiring skills including semantic recognition, visual correspondence, temporal reasoning, and spatial reasoning. The benchmark contains 1,114 highly complex questions on 279 videos from diverse domains including city walk tours, indoor villa tours, video games, and extreme outdoor sports, with 100% manual annotation. Human studies show that PerceptionComp requires substantial test-time thinking and repeated perception steps: participants take much longer than on prior benchmarks, and accuracy drops to near chance (18.97%) when rewatching is disallowed. State-of-the-art MLLMs also perform substantially worse on PerceptionComp than on existing benchmarks: the best model in our evaluation, Gemini-3-Flash, reaches only 45.96% accuracy in the five-choice setting, while open-source models remain below 40%. These results suggest that perception-centric long-horizon video reasoning remains a major bottleneck, and we hope PerceptionComp will help drive progress in perceptual reasoning.

Figures & tables

Figure 1: Overview of the PerceptionComp benchmark. (a) An example from PerceptionComp, where models are required to perform complex, perception-centric reasoning with various types of subconditions to arrive at the final answer. (b) Results from a human study measuring question-answering time, showing that PerceptionComp is more challenging for humans than previous perception and reasoning video benchmarks, largely due to its emphasis on perception-centric reasoning.
BenchmarkProperties
Video Domain# QAPer.ReaAnnotation
MMVU Zhao et al. (2025)Educational videos3,000✗Manual
VideoMME Fu et al. (2024)YouTube videos2,700✗Manual
VCR-Bench Qi et al. (2025)Short films1,034✗Manual
MINERVA Nagrani et al. (2025)Mix1,515✗Manual
VideoMMMU Hu et al. (2025)Lectures900✗Manual
Table 1: Comparison of PerceptionComp with other benchmarks. PerceptionComp distinguishes itself from previous benchmarks by emphasizing perception-centric reasoning, assessing how models integrate visual evidence with reasoning processes.
Figure 2: Data construction and statistics of PerceptionComp. (a) Annotation pipeline, which integrates diverse subconditions and supports two types of compositional questions. (b) Benchmark statistics: higher difficulty levels contain more subconditions, increasing the demand for perception-centric reasoning.
Figure 3: Examples from PerceptionComp. , , and denote difficulty levels 1, 2, and 3, respectively. PerceptionComp spans diverse video sources and uses subconditions to construct conjunctive and sequential questions that require perception-centric reasoning.
Figure 4: Analysis on PerceptionComp. Accuracy as a function of perception budget and reasoning budget . Left/middle: accuracy vs. the number of uniformly sampled input frames for GPT-o3 and Qwen3-VL-8B. Right: accuracy vs. the thinking-token budget for Gemini-2.5-Flash.
Figure 5: Example of model reasoning on PerceptionComp. We show responses and judgments of frontier models. Even state-of-the-art models exhibit limitations in capturing perceptual information and often fail to maintain coherent reasoning chains leading to the correct answer.
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Illustration of the model’s successful multi-step visual reasoning. The system first anchors on the iconic glass-cube Apple Store, then tracks two yellow food trucks as spatial reference points. At the moment the camera passes the nearer truck, a yellow SUV appears on the right, enabling the model to localize the bicyclist adjacent to it and accurately identify the yellow-green vest color. This showcases robust landmark grounding, spatio-temporal alignment, and fine-grained visual detail recognition across a dynamic street scene.
Figure 7: Failure mode illustrating the model’s breakdown in multi-step visual reasoning. Instead of following the prescribed sequence—identifying the dominant bench style (Style A), tracking the third used bench to determine Direction B, and grounding Type C using the nearest person when the camera first faces that direction—the model prematurely abandons the correct logic path. It misclassifies the bench styles, hallucinates an alternative meaning for “Type C,” and ultimately selects the wrong individual as D, leading to the incorrect prediction that D is holding a phone rather than the actual red coffee cup.
Figure 8: Failure mode illustrating Gemini-3.0-Pro’s breakdown in multi-step visual reasoning. Instead of following the prescribed sequence—locating the initial man to determine the shop sign’s dominant color (Color A: Yellow), tracking this variable to spot the man holding the corresponding yellow plastic bag later in the video, and grounding the nearest person (the accompanying woman) to identify her shirt color—the model suffers from critical logical hallucinations. It prematurely breaks the reasoning chain by failing to connect the yellow shop sign to “Color A” and hallucinates an alternative path by fixating on a random blue plastic bag. Consequently, it misjudges the spatial and social context, selecting a passing pedestrian rather than the walking companion, leading to the incorrect prediction that the closest person is wearing a white shirt rather than the actual black shirt.
Figure 9: Failure mode illustrating Qwen3-VL’s breakdown in multi-step visual reasoning. Instead of following the prescribed sequence—tracking all four parkour runners continuously, registering their initial dismount flip directions, and visually comparing them to their secondary flips at the new location—the model prematurely abandons precise visual verification. Constrained by temporal sampling limitations, it falls into a narrative trap: it exhibits a “protagonist bias” by overly fixating on the prompt’s initial temporal anchor (the grey-clothed runner) while superficially dismissing the other athletes. Ultimately, the model substitutes rigorous 3D spatial orientation analysis with a hallucinated “behavioral consistency” heuristic, incorrectly assuming the main subject would naturally repeat his movement pattern, which leads to the erroneous selection of the grey-clothed runner.
Figure 10: Failure mode illustrating the model’s breakdown in multi-step visual reasoning. Instead of following the prescribed sequence—identifying the key natural landscapes, locating the orange teppanyaki sign, and directly deducing the relative direction between them from the observer’s perspective—the model becomes bogged down by excessive noise and spatial confusion. It first over-focuses on irrelevant minor details (such as the pedestrian’s attire and alternative signs), and then attempts to construct an unnecessarily complex absolute coordinate system involving arbitrary cardinal directions (North, West, East). This convoluted reasoning chain distorts the original egocentric spatial layout, leading to the incorrect prediction that the small landscape is located directly behind the observer rather than the actual direction of directly to the right.
Figure 11: Error analysis of Gemini-3-Pro Google AI for Developers [2026] failures on PerceptionComp. (a) Distribution of incorrect reasoning steps across the reasoning chain. (b) Distribution of error causes, including spatial understanding, static feature perception, incomplete reasoning, counting errors, and dynamic feature perception.
More in this topic

Video Reasoning

Video reasoning studies how AI systems use video evidence to infer events, actions, temporal relations, causes, and future states.

See all 199 papers

Explore similar work

CardsList
May 31, 2026

Perception First: A Frontier Native-Video Model with Self-Consistency for Implicit Video Question Answering

We describe our submission to the VRR Challenge @ CVPR 2026, built on the ImplicitQA / VRR-QA benchmark~\cite{implicitqa}: multiple-choice video question answering in which answers are deliberately not observable in any single frame and must be inferred from spatial layout, motion, depth, viewpoint, causality, and social context across discontinuous frames of creative video. We conduct a systematic, training-free study spanning open-source Video-LMMs (Qwen2.5-VL~\cite{qwen25vl}, Qwen3-VL~\cite{qwen3vl}, InternVL3, Gemma-3, and the RL-tuned video reasoners Video-R1~\cite{videor1} and VideoChat-R1.5~\cite{videochatr15}) and a battery of inference-time strategies (chain-of-thought, question decomposition, describe-then-reason cascades, audio transcripts, spatial state prompting, self-consistency~\cite{selfconsistency}, multi-model ensembling, and category routing). Our central finding is that this benchmark is perception-bound rather than reasoning-bound: reasoning-side augmentations are neutral-to-harmful, whereas base-model perceptual capability and lightweight test-time denoising are the only reliable levers. A per-category error analysis localizes the difficulty to low-level perception -- relative depth, viewpoint, and counting are the hardest categories, while causal and social reasoning are nearly solved -- and a prompt that explicitly injects monocular depth cues to attack the weakest category lowers test accuracy by 5.85.8 points, confirming that the model needs a better percept, not a better procedure.

Ali Alavi

The Ohio State University

Visual Spatial ReasoningVideo QAVisually Grounded ReasoningPhysical Reasoning in Video
AskPDF
May 25, 2026

Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning

Recent multimodal large language models (MLLMs) achieve strong performance on visual reasoning benchmarks, yet it remains unclear to what extent such performance reflects reasoning directly grounded in visual evidence. We introduce VisReason, a benchmark for vision-centric reasoning in everyday scenarios where perception and inference are tightly coupled. VisReason contains 1,505 questions across 10 categories spanning perceptual, structural, and conceptual reasoning. Our evaluation shows that VisReason poses a qualitatively different challenge from existing benchmarks, exposing substantial gaps between humans and current MLLMs and revealing limited benefits from test-time reasoning strategies. VisReason offers a focused diagnostic for evaluating vision-centric reasoning beyond language.

Longteng Guo, Yifan Wang, Pengkang Huo +4

Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences

Visual ReasoningVLM ReasoningVisually Grounded ReasoningMultimodal Model EvaluationMultimodal UnderstandingReasoning for VLMs
AskPDF
Aug 4, 2026

Perception Before Reasoning: Dynamic Latent Reasoning for Video Understanding and Question Answering

Video question answering requires models to ground language queries in visual evidence and, when necessary, reason over that evidence across time. Existing methods typically rely on long textual chain-of-thought rationales, even though many questions can be answered as soon as the relevant object, action, or frame is localized. We propose Dynamic Latent Reasoning (DyLaR), which first grounds a question in a short block of perception latents (continuous hidden states that encode query-relevant visual evidence), and then adaptively decides whether to append reasoning latents (continuous thoughts that reason over this evidence in latent space) before answering. DyLaR learns this behavior by grounding perception latents in verified visual evidence and distilling verified rationales into reasoning latents, followed by reinforcement learning that further refines when to reason. Across nine video benchmarks and four multimodal language model backbones, DyLaR improves average accuracy over same-backbone baselines while generating fewer than 20 tokens per query. On Qwen3-VL-4B, for example, DyLaR improves average accuracy over Qwen3-VL-4B-Thinking from 54.0 to 58.2 while reducing response length from 1,220.7 to 18.5 tokens per query. Ablations further show that grounded perception latents, rationale-supervised reasoning latents, and adaptive routing each improve accuracy.

Haotian Xia, Zilin Xiao, Junbo Zou +2

Department of Computer Science, Rice University · College of Sciences, Georgia Institute of Technology

Video QALatent Visual ReasoningVideo Reasoning
AskPDF