PhysCount: Do MLLMs Count Visual Appearances or Physical Entities?

Lidong Lu; Jiaxin Lv; Liangrui Dong; Xinyi Mao; Lidong Zou; Zhenquan Li; Tong Lu; Guo Chen

cs.CVOct 29, 2026

PhysCount: Do MLLMs Count Visual Appearances or Physical Entities?

Authors: Lidong Lu, Jiaxin Lv, Liangrui Dong, Xinyi Mao, Lidong Zou, Zhenquan Li, Tong Lu, Guo Chen

Organizations: Nanjing University · NVIDIA

Abstract

Images reveal appearances, but counting often asks about physical objects. A single entity may be visible directly, in several reflections, or through glass, so detecting every visible occurrence does not determine how many entities are present. Existing visual counting benchmarks rarely separate these two operations. We introduce PhysCount, a benchmark for evaluating whether multimodal large language models (MLLMs) can distinguish what appears in an image from what physically exists in the scene. PhysCount contains 546 real-world images and 4,595 human-written questions organized into 651 target-specific groups. For each group, annotators locate the relevant appearances, label how each is observed, and identify appearances of the same entity. Four complementary question families test the enumeration of appearances, the enumeration of physical entities, image-formation-specific counting, and appearance--entity correspondence. We measure both accuracy on each family and the proportion of groups answered entirely correctly. Across 40 MLLMs, counting visible appearances is substantially easier than recovering the identities that connect them. Even the strongest model answers every question correctly in only 13.36% of groups. Performance declines when one entity produces many appearances or when its appearances cross observation modes. Higher image resolution improves access to visual evidence but brings limited gains in complete-group accuracy, whereas answering related questions jointly produces a marked improvement. These results expose a gap between recognizing what is visible and correctly recovering the physical entities behind those observations.

Figures & tables

Figure 1: One scene, different counting units. Five cat appearances correspond to four entities because D1 and R1 depict the same cat. With the image and target fixed, the questions count all appearances (5), distinct entities (4), reflections (2), or entities seen both directly and in reflection (1).

Figure 2: PhysCount construction pipeline. Candidate mirror and glass scenes undergo automatic filtering and human selection. An annotator labels direct, reflected, and through-glass appearances, groups appearances of the same entity, and writes questions whose answers are derived from these groups. Here, five cat appearances form four entities, and the correspondence question selects the one entity seen both directly and in reflection. Annotation review and automated QA checking send consistent QAs to PhysCount and flagged QAs to final human correction.

Figure 3: Benchmark statistics. (a) Visual-appearance and physical-entity counts across 651 annotation groups; bubble area encodes the number of groups at each coordinate, and the adjacent chart shows the appearance-type composition of the 229 groups on $y=x$ . (b) Distribution of 4,595 questions across the four question families. (c) Diversity and frequency of counting targets.

	Appearance		Entity		Image Formation		Correspondence
Model	MAE $\downarrow$	NAE $\downarrow$	MAE $\downarrow$	NAE $\downarrow$	MAE $\downarrow$	NAE $\downarrow$	MAE $\downarrow$	NAE $\downarrow$	Score $\uparrow$	Group-Acc $\uparrow$	Phys-Gnd $\uparrow$
Human (5-person avg.)	0.037	0.59	0.023	0.83	0.035	1.05	0.033	2.21	97.10	95.98	99.49
Gemini 3.7 Flash Google DeepMind (2026)	2.724	20.08	2.046	27.36	2.467	30.62	1.396	33.88	36.45	13.36	73.00
GPT-5.6 Sol OpenAI (2026a)	3.028	22.82	2.808	36.58	2.544	31.94	2.113	52.11	21.73	3.84	55.65
GPT-6 Astra OpenAI (2026b)	3.049	24.87	2.673	37.72	2.517	30.99	1.905	47.83	20.77	3.38	51.89
GPT-5.6 Terra OpenAI (2026a)	3.395	26.06	2.726	36.39	3.227	37.30	1.863	50.99	20.42	4.30	53.43

Table 1: Results on PhysCount for all 40 evaluated models and five human participants. Each participant independently answers all 4,595 questions under the same image–question protocol as the models; the human row reports their mean. Best and second-best model results are bolded and underlined.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Example 1: reflection and transmission of the same target category. Three cat appearances form two physical identities: one identity is observed directly and by reflection, while the other is seen through glass. The table reports every question derived from these annotations.

Figure 8: Example 2: one person with two reflected appearances. The direct appearance and both mirror appearances share one identity. The questions contrast appearance enumeration with physical occupancy, identity, and reflection multiplicity.

Figure 9: Example 3: one person distributed across three mirror regions. Four appearances map to one physical identity, enabling questions about total appearances, reflected subsets, physical–reflection overlap, and maximum multiplicity.

Figure 10: Example 4: high reflection multiplicity from a single object. One physical die produces six reflected appearances. The resulting questions separate the seven visible appearances from one physical identity and its reflection multiplicity.

Figure 11: Example 5: two physical objects paired with their mirror appearances. Four shoe appearances form two identity groups. The questions require two independent direct–reflection correspondences rather than a single global count.

Figure 12: Example 6: a direct camera repeated across two mirrors. Three camera appearances belong to one physical identity. The questions distinguish the direct camera, the reflected subset, their shared identity, and reflection multiplicity.

Model	People	Containers	Lighting	Art	Furniture	Architecture	Personal	Household	Electronics	Vehicles	Sports	Toys	Plants	Food	Animals	Other
Gemini 3.7 Flash	39.43	27.53	30.85	41.06	20.09	21.91	17.33	15.54	45.31	25.51	13.78	38.13	7.51	47.51	47.56	48.89
Muse-Glimmer-30B	30.16	23.31	26.72	34.90	27.08	26.96	17.12	20.59	56.25	22.04	23.84	39.23	20.65	41.39	49.34	29.19
Qwen3.8-27B	29.47	22.60	21.48	42.57	12.88	16.19	23.97	10.10	56.25	26.78	18.68	47.78	21.11	71.31	39.61	38.68
Qwen3.5-397B-A17B	28.57	20.44	22.68	40.11	25.75	29.82	29.77	10.56	27.08	19.15	15.68	34.57	24.26	48.07	48.26	25.35
Gemma 4-31B	27.88	19.67	16.46	35.00	21.57	11.89	8.63	2.62	75.00	19.98	16.85	29.48	10.90	42.15	31.87	45.24
Qwen3.5-27B	27.11	19.04	21.06	42.99	29.98	18.25	16.47	14.72	20.31	25.10	20.87	34.59	10.23	28.75	27.54	35.51

Appendix

Table 4: Performance across counting-target domains. Each entry is the group-macro Score (%) for one counting-target domain.

	Entity NAE $\downarrow$				Score $\uparrow$
Model	1	2	3	$\geq 4$	1	2	3	$\geq 4$
Gemini 3.7 Flash	27.55	26.58	26.29	32.54	40.54	35.27	31.19	30.07
Muse-Glimmer-30B	35.32	31.36	27.56	26.38	28.72	29.68	29.51	33.27
Qwen3.8-27B	35.75	30.19	27.70	36.07	29.98	28.19	30.26	26.61
Qwen3.5-397B	33.20	32.50	25.23	45.92	27.71	27.89	31.36	26.48
Gemma 4-31B	33.47	37.28	34.66	51.63	29.73	24.68	25.27	22.34

Appendix

Table 5: Performance across appearance multiplicity. Each entry is a group-macro percentage within the indicated maximum number of appearances per entity. The final bin produces the clearest rise in Entity NAE and the broadest decline in Score.

	Correspondence NAE $\downarrow$			Score $\uparrow$
Model	Within	Cross	$\Delta$	Within	Cross	$\Delta$
Gemini 3.7 Flash	30.08	35.65	+5.57	29.73	34.51	+4.78
Muse-Glimmer-30B	32.81	42.67	+9.86	23.87	30.54	+6.67
Qwen3.8-27B	24.17	45.03	+20.86	27.14	28.54	+1.40
Qwen3.5-397B	39.34	45.02	+5.68	24.03	28.49	+4.46
Gemma 4-31B	41.23	42.66	+1.43	16.34	25.06	+8.72

Appendix

Table 6: Identity correspondence within and across image-formation modes. Both conditions are reweighted to the same appearance-multiplicity distribution. Positive

\Delta

values indicate higher error or Score in the cross-mode condition.

Figure 13: An appearance inventory that is not pairwise balanced. The crop contains two directly visible plants but three plant reflections. The unmatched reflection remains an observed appearance even though its physical source is outside the direct view.

Figure 14: Four shoe appearances connected into two identities. The mirror boundary does not remove visual evidence; it determines which observations should be linked before the entities are counted.

Figure 15: A nonempty physical set but an empty physical-only set. The direct dancer and the two mirror appearances belong to one identity; the exclusion condition removes that identity from the answer.

Object Counting

Object counting estimates the number and locations of target instances in images or video, from dense crowds to fine-grained categories.

See all 62 papers

Oct 28, 2026

What do MLLMs know about 'Waldo'? Evaluating Visual Reasoning under Multi-Instance Multi-Object Challenges

Recent advances in Multimodal Large Language Models (MLLMs) have led to significant progress in visual reasoning capabilities. However, existing benchmarks curate image-question pairs pertaining to object categories with few instances, enabling trivial object referencing and limited visual ambiguity. In this work, we introduce MIMO-Bench, a benchmark for evaluating MLLMs in a Multi-Instance per Multi-Object categories (MIMO) setting. Each image-question pair in our benchmark pertains to instances belonging to object categories with multiple instances. The presence of such similar feature distractors causes hindrance in basic visual reasoning tasks, necessitating the referring expression to be composed of multiple visual subtasks to uniquely and unambiguously identify the target object. MIMO-Bench comprises 3,802 image-question pairs across attribute binding and numerical counting tasks. Despite impressive performances on existing benchmarks, both SOTA open-source (InternVL, QwenVL) and proprietary (Gemini, GPT-5) models exhibit substantially lower performance on MIMO-Bench, highlighting their limitations in the MIMO setting. The benchmark will be made publicly available.

Saumya Mundra, Ajoy Mondal, Vineeth N Balasubramanian +1

IIIT Hyderabad · Microsoft Research India · IIT Hyderabad

MLLM Evaluation Compositional Reasoning in VLMs

Ask PDF

Jul 7, 2026

HoloCount: A Holistic Visual Counting Benchmark for MLLMs

Visual counting is a fundamental pillar of multimodal intelligence, requiring a seamless integration of fine-grained grounding and spatial reasoning. While Multimodal Large Language Models (MLLMs) have achieved remarkable success in qualitative scene understanding, their quantitative precision remains a significant bottleneck, often characterized by persistent numerical hallucinations. Existing counting benchmarks primarily focus on basic perception in simplified contexts, failing to capture the complex failure modes that emerge under logical constraints or adversarial conditions. To address these limitations, we introduce HoloCount, a holistic and diagnostically rich benchmark structured around a three-level hierarchical taxonomy. HoloCount evaluates MLLMs across: (1) Semantic Counting, focusing on atomic and property-based enumeration; (2) Analytical Counting, assessing logical composition through spatial and set-based reasoning; and (3) Robustness Testing, probing model integrity against adverse scenarios and grounded counter-priors, such as high-density scenes and linguistic biases. Through an exhaustive evaluation of over 20 state-of-the-art MLLMs, we reveal a critical performance gap: even top-tier models degrade significantly as tasks transition from perception to complex analytical reasoning and adverse scenarios. Our findings provide a systematic landscape of current MLLM counting capabilities and offer a roadmap for developing more grounded and reliable multimodal systems. The dataset is available at https://mm-mvr.github.io/HoloCount/.

Jinhong Deng, Limeng Qiao, Guanglu Wan

Meituan

Multimodal Robustness Object Counting Multimodal Reasoning Multimodal Model Evaluation

Ask PDF

Oct 10, 2026

HumanPhys: A Human-Centric Physical-World Perception Benchmark for Real-World Visual Reasoning

We introduce HumanPhys, a human-centric benchmark for real-world physical perception, and a training corpus emitted by the same pipeline. Each question targets a physically grounded quantity related to people, including depth, inter-person distance, gaze angle, displacement, velocity, heading-step trajectory, blink count, and contact-deformation strength. The rendered probes additionally test illumination inference from body and face cues and camera ego-motion, both with and without a person in the frame. All labels are derived programmatically from sensor calibration, source metadata, or renderer state, rather than written by annotators or produced by estimators on the benchmark frames. HumanPhys systematically probes six dimensions of human-centric physical perception: spatial state, kinematics, blink events, deformation, illumination, and camera pose. To the best of our knowledge, no existing MLLM benchmark-and-training corpus jointly supervises this set of human-centered physical quantities under direct metadata- or renderer-derived labels. The six dimensions appear only partially dissociable across the tested MLLMs. We benchmark a range of open- and closed-source MLLMs on HumanPhys and find that all six capability dimensions remain challenging for current models. We hope HumanPhys will support future research on multimodal systems that perceive and reason about people in the physical world.

Yiming Zhang, Enzhi Zhou, Junan Lin +4

Nanyang Technological University, Singapore · Ant Group, Hangzhou, China

Human Motion Understanding VLM Reasoning Camera Pose Estimation Depth Estimation

Ask PDF

Abstract

Object Counting

Explore similar work

What do MLLMs know about 'Waldo'? Evaluating Visual Reasoning under Multi-Instance Multi-Object Challenges

HoloCount: A Holistic Visual Counting Benchmark for MLLMs

HumanPhys: A Human-Centric Physical-World Perception Benchmark for Real-World Visual Reasoning