cs.CVNov 9, 2026

HumanSightBench: A Benchmark for Human Visual Perception in MLLMs

Authors: Zixin Su, Yangguang Lin, Zegang Peng, Yuntong Sun, Xuanyi Wang, Jiahao Wang, Qiankai Xu, Bing Yuan, 

Organizations: ByteDance Seed · Tsinghua University · Beijing University of Posts and Telecommunications · Renmin University of China · 

Abstract

Multimodal large language models (MLLMs) have demonstrated strong performance on knowledge-intensive and reasoning-based vision-language benchmark datasets, but still struggle to solve the visual questions that people can solve easily. Evaluating the differences between MLLMs and human-level visual perception is challenging due to end-to-end evaluation which combines various skills, or because we are uncertain if individual items are valid. Specifically, observed errors may reflect visual perception, language priors, background knowledge, question ambiguity, and underspecified answers. Our guideline for HumanSightBench is that we construct items that are human-easy by construction and model-hard by evaluation. HumanSightBench contains 2,876 visual questions with short answers across 7 top-level perceptual categories and 21 subcategories. We only include items that pass five inclusion gates: question clarity, answer uniqueness, visual grounding, human solvability, and text-only shortcut filtering. According to the protocol, errors are defined as incorrect answers on human-validated visual tasks with the evaluated interface. The inclusion gates are intended to reduce confounds from ambiguity and external knowledge. The best pass@3 score is 65.6% across eight evaluated MLLMs. No evaluated model exceeds 27.5% on Counting. These results identify capability gaps that are hidden by overall scores and can help future training experiments and held-out evaluation.

Explore similar work

CardsList
  1. Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning

    May 25, 2026Longteng Guo, Yifan Wang, Pengkang Huo +4Visually Grounded ReasoningVisual Reasoning

  2. Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs

    Feb 23, 2025Jen-Tse Huang, Dasen Dai, Jen-Yuan Huang +7VLM EvaluationVision-Language Models