Large vision-language models (LVLMs) have demonstrated remarkable performance on multimodal reasoning benchmarks, yet their perceptual reliability under physically constrained imaging conditions remains poorly understood. Existing evaluations predominantly assume ideal visual inputs and therefore fail to characterize how camera distance, illumination, viewpoint, and pixel density fundamentally affect semantic recoverability. We introduce SynDORBench, the first physically grounded benchmark for evaluating LVLM perceptual robustness under DORI-calibrated conditions aligned with human visual capability standards. SynDORBench comprises over 54k question--answer pairs generated through a controllable synthetic pipeline that systematically varies viewing distance, lighting, camera geometry, and action pose according to physically interpretable pixel-density regimes. To support scalable low-visibility supervision, we further propose a discernibility annotation framework that propagates human perceptual labels using mask-conditioned statistical features and ensemble learning. We evaluate 16 open-source LVLMs, a commercial LVLM baseline, and YOLO11x across human-presence classification and action recognition tasks under progressively degraded visibility conditions. Our results reveal that perceptual failure in LVLMs is strongly governed by pixel density and physical imaging constraints rather than model scale alone. Surprisingly, several compact open-source LVLMs outperform larger commercial baselines and substantially exceed YOLO11x robustness under long-range and low-light conditions. SynDORBench establishes a new benchmark paradigm for physically grounded multimodal evaluation, enabling systematic analysis of LVLM reliability under real-world perceptual constraints and direct comparison against human visibility thresholds.
Figures & tables
Figure 1: Minimum pixel density (px/m) required for human perceptual tasks under the BS EN 62676-4:2015 DORI standard [ 6 ] . The DORI framework defines physically grounded visibility thresholds for Detection, Observation, Recognition, and Identification.
Benchmark
Modality
Answer Type
# Models
# QA Pairs
DORI-Calibrated Dist.
SEED-BENCH [ 27 ]
Image, Video
Multiple-choice
18
19,242
✗
LV-BENCH [ 57 ]
Video
Multiple-choice
27
1,549
✗
MMMU [ 62 ]
Image
Multiple-choice, Open-ended
24
11,550
✗
CinePile [ 47 ]
Video
Multiple-choice
24
303,828
✗
LOKI [ 60 ]
Video, Image, 3D, Audio
Multiple-choice, Open-ended
28
18,000
✗
SynDORBench-54k(Ours)
Image
Multiple-choice, Open-ended
18
54,864
✓
Table 1: Comparison of multimodal LVLM benchmarks.
Figure 2: Conceptual overview of SynDORBench: The benchmark systematically varies physical visibility factors (lighting, scene type, camera azimuth/elevation, action pose) and DORI-calibrated viewing distance, evaluating models on two tasks: 1) binary human-presence classification and 2) open-ended action recognition.
Model Type
Models
# Params
Vision Encoder (VE)
VE Type
LLM Backbone
Detection-only (non-LVLM)
YOLO11x [ 25 ]
56.9M
–
–
–
Small LVLMs (SVLMs) [ 32 ]
SmolVLM2-256M-Video-Instruct [ 35 , 21 ]
256M
SigLIP-B/16-512
SigLIP
SmolLM2–135M
SmolVLM2-500M-Video-Instruct [ 35 , 22 ]
500M
SigLIP-B/16-512
SigLIP
SmolLM2–360M
InternVL3-1B-hf [ 64 ]
1B
InternViT-300M-448px-V2.5
InternViT
Qwen2.5–0.5B
Perception-LM-1B [ 10 ]
1B
PE L/14-448
PE
Llama-3.2-1B
SmolVLM2-2.2B-Instruct [ 35 , 52 ]
2.2B
SigLIP-So400M-patch14-384
SigLIP
SmolLM2-1.7B
Table 2: Overview of evaluated models by scale and modality (Custom encoders denoted by * with limited publicly disclosed details).
Model
5.4 m
10.8m
23.0 m
Acc
F1
Acc
F1
Acc
F1
YOLO11x
73.92
50.17
65.38
50.36
50.29
42.98
SmolVLM2-256M-Video- Instruct
10.02
9.63
13.47
12.68
22.70
18.69
SmolVLM2-500M-Video- Instruct
23.20
20.23
26.51
24.04
27.19
22.08
InternVL3-1B-hf
87.77
59.63
85.69
64.58
83.97
68.13
Perception-LM-1B
69.77
48.21
62.64
49.11
45.94
39.89
Table 3: Performance degradation across DORI-calibrated distances for presence classification, with accuracy (Acc) and macro-F1 (F1) reported. Best scores are in bold.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Manufacturer
Model
Resolution
Sensor
Lens Type
Focal Length (mm)
Aperture
Axis
Q1615-LE Mk III
1920×1080
1/2.8”
Varifocal
2.8–8.5
f/1.2
Axis
Q1715
1920×1080
1/2.8”
Varifocal
4–84.6
f/1.6–4.5
Axis
P1385
1920×1080
1/2.8”
Varifocal
2.8–13
f/1.4
Axis
P1385-B
1920×1080
1/2.8”
–
–
–
Axis
P1385-E
1920×1080
1/2.8”
Varifocal
2.8–13
f/1.4
Axis
P1385-BE
1920×1080
1/2.8”
–
–
–
Appendix
Table 4: Specifications of representative commercial camera models commonly used in fixed-camera systems.
Model
Macro F1 (%)
Accuracy (%)
Decision Tree
82.41
90.25
RF + XGBoost Ensemble
83.55
90.50
Appendix
Table 5: Performance of discernibility annotation models on validation set (5-Fold cross-validation).
Model
5.4 m
10.8 m
23.0 m
Non-Expert Human
0.94
0.91
0.83
SmolVLM2-256M-Video-Instruct
0.78 (-1.45)
0.77 (-1.39)
0.77 (-1.15)
SmolVLM2-500M-Video-Instruct
0.80 (-0.76)
0.79 (-0.75)
0.78 (-0.69)
Internvl3-1B-hf
0.81 (-0.23)
0.80 (-0.43)
0.78 (-0.72)
Perception-LM-1B
0.77 (-1.87)
0.76 (-1.78)
0.75 (-1.72)
SmolVLM2-2.2B-Instruct
0.84 (0.93)
0.83 (0.98)
0.83 (1.09)
Appendix
Table 6: Action recognition performance across DORI distances. Mean E5Score and corresponding z-score per distance. Best z-scores are in bold.
Figure 3: HPC Robustness: Accuracy across DORI-calibrated distances under different lighting conditions. The left, middle, and right subplots correspond to day, night-vision (NV), and night settings, respectively. With each subplot, distance increases from left to right, corresponding to progressively more difficult evaluation conditions.
Vision-language models (VLMs) achieve strong performance on standard, high-quality datasets, but we still do not fully understand how they perform under real-world image distortions. We present VLM-RobustBench, a benchmark spanning 49 augmentation types across noise, blur, weather, digital, and geometric perturbations, evaluated under graded severities (low/mid/high) and binary transforms, yielding 133 corrupted settings. We evaluate VLMs from four families (Qwen, InternVL, Molmo, Gemma) on two complementary benchmarks: MMBench (visually grounded) and MMMU-Pro (reasoning-oriented). Our results reveal that visual severity is a weak predictor of difficulty: low-severity spatial perturbations often degrade performance more than visually severe photometric corruptions. In particular, low-severity glass_blur reduces MMBench accuracy by about 8 pp on average across models, while the largest drops arise from resampling and geometric distortions (e.g., upsample, elastic_transform), reaching up to 34 pp. Overall, our findings suggest current VLMs are semantically strong but spatially fragile, motivating the definition of novel robustness evaluation protocols and training regimes that emphasize resampling and geometric invariances.
Large Vision Language Models have integrated reasoning capabilities, elevating cognitive performance to new levels. However, existing evaluations either focus solely on perception or rely on specific domains such as maths or coding. Evaluation for reasoning capabilities that align with an open-world environment is still required, especially one that considers perception and reasoning jointly. To bridge this gap, we propose to evaluate LVLMs by exploiting visual illusions as a diagnostic tool. Visual illusions are phenomena in which the human visual system misinterprets objective signals, resulting in an understanding that deviates from reality. We constructed IllusionReasoning, a benchmark of illusion images collected from the real world, incorporating diverse annotated question-answer pairs. Based on IllusionReasoning, we show that the reasoning capabilities of a wide range of LVLMs are not as advanced as claimed. Our work provides new insights into LVLMs and offers future direction for optimisation.
Liangjie Zhao, Jiaqing Lyu, Kexin Tang +5
Adelaide University · Tsinghua University · Amap, Alibaba Group +3
We introduce PerceptionBench, a benchmark specifically designed to evaluate the atomic visual perception capabilities of Multimodal Large Language Models (MLLMs). Existing benchmarks often fail to isolate perception: holistic evaluations conflate perceptual errors with failures in reasoning or domain knowledge, while application-driven benchmarks only cover narrow, fragmented domains shaped by heuristic designs. To address these limitations, PerceptionBench adopts a bottom-up approach: by diagnosing the earliest failure points in the responses of frontier MLLMs across 42 existing benchmarks, we construct an error taxonomy whose perception branch defines ten atomic perceptual capabilities. Guided by this taxonomy, we construct 3,000 verified questions with short, unambiguous answers, each isolating a single capability, with difficulty stemming from perception rather than reasoning or knowledge. Benchmark results across sixteen frontier MLLMs reveal that atomic perception remains largely unsolved---no model reaches 60% accuracy, perception-related hallucination is the weakest capability on average, and similar overall scores conceal sharply divergent capability profiles. PerceptionBench thus provides a capability-level standard for measuring and diagnosing the visual perception boundaries of MLLMs.