PhysCount: Do MLLMs Count Visual Appearances or Physical Entities?

Lidong Lu; Jiaxin Lv; Liangrui Dong; Xinyi Mao; Lidong Zou; Zhenquan Li; Tong Lu; Guo Chen

cs.CVOct 29, 2026

PhysCount: Do MLLMs Count Visual Appearances or Physical Entities?

Authors: Lidong Lu, Jiaxin Lv, Liangrui Dong, Xinyi Mao, Lidong Zou, Zhenquan Li, Tong Lu, Guo Chen

Organizations: Nanjing University · NVIDIA

Abstract

Images reveal appearances, but counting often asks about physical objects. A single entity may be visible directly, in several reflections, or through glass, so detecting every visible occurrence does not determine how many entities are present. Existing visual counting benchmarks rarely separate these two operations. We introduce PhysCount, a benchmark for evaluating whether multimodal large language models (MLLMs) can distinguish what appears in an image from what physically exists in the scene. PhysCount contains 546 real-world images and 4,595 human-written questions organized into 651 target-specific groups. For each group, annotators locate the relevant appearances, label how each is observed, and identify appearances of the same entity. Four complementary question families test the enumeration of appearances, the enumeration of physical entities, image-formation-specific counting, and appearance--entity correspondence. We measure both accuracy on each family and the proportion of groups answered entirely correctly. Across 40 MLLMs, counting visible appearances is substantially easier than recovering the identities that connect them. Even the strongest model answers every question correctly in only 13.36% of groups. Performance declines when one entity produces many appearances or when its appearances cross observation modes. Higher image resolution improves access to visual evidence but brings limited gains in complete-group accuracy, whereas answering related questions jointly produces a marked improvement. These results expose a gap between recognizing what is visible and correctly recovering the physical entities behind those observations.

Figures & tables

Figure 1: One scene, different counting units. Five cat appearances correspond to four entities because D1 and R1 depict the same cat. With the image and target fixed, the questions count all appearances (5), distinct entities (4), reflections (2), or entities seen both directly and in reflection (1).

Figure 2: PhysCount construction pipeline. Candidate mirror and glass scenes undergo automatic filtering and human selection. An annotator labels direct, reflected, and through-glass appearances, groups appearances of the same entity, and writes questions whose answers are derived from these groups. Here, five cat appearances form four entities, and the correspondence question selects the one entity seen both directly and in reflection. Annotation review and automated QA checking send consistent QAs to PhysCount and flagged QAs to final human correction.

Figure 3: Benchmark statistics. (a) Visual-appearance and physical-entity counts across 651 annotation groups; bubble area encodes the number of groups at each coordinate, and the adjacent chart shows the appearance-type composition of the 229 groups on $y=x$ . (b) Distribution of 4,595 questions across the four question families. (c) Diversity and frequency of counting targets.

	Appearance		Entity		Image Formation		Correspondence
Model	MAE $\downarrow$	NAE $\downarrow$	MAE $\downarrow$	NAE $\downarrow$	MAE $\downarrow$	NAE $\downarrow$	MAE $\downarrow$	NAE $\downarrow$	Score $\uparrow$	Group-Acc $\uparrow$	Phys-Gnd $\uparrow$
Human (5-person avg.)	0.037	0.59	0.023	0.83	0.035	1.05	0.033	2.21	97.10	95.98	99.49
Gemini 3.7 Flash Google DeepMind (2026)	2.724	20.08	2.046	27.36	2.467	30.62	1.396	33.88	36.45	13.36	73.00
GPT-5.6 Sol OpenAI (2026a)	3.028	22.82	2.808	36.58	2.544	31.94	2.113	52.11	21.73	3.84	55.65
GPT-6 Astra OpenAI (2026b)	3.049	24.87	2.673	37.72	2.517	30.99	1.905	47.83	20.77	3.38	51.89
GPT-5.6 Terra OpenAI (2026a)	3.395	26.06	2.726	36.39	3.227	37.30	1.863	50.99	20.42	4.30	53.43

Table 1: Results on PhysCount for all 40 evaluated models and five human participants. Each participant independently answers all 4,595 questions under the same image–question protocol as the models; the human row reports their mean. Best and second-best model results are bolded and underlined.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Example 1: reflection and transmission of the same target category. Three cat appearances form two physical identities: one identity is observed directly and by reflection, while the other is seen through glass. The table reports every question derived from these annotations.

Figure 8: Example 2: one person with two reflected appearances. The direct appearance and both mirror appearances share one identity. The questions contrast appearance enumeration with physical occupancy, identity, and reflection multiplicity.

Figure 9: Example 3: one person distributed across three mirror regions. Four appearances map to one physical identity, enabling questions about total appearances, reflected subsets, physical–reflection overlap, and maximum multiplicity.

Figure 10: Example 4: high reflection multiplicity from a single object. One physical die produces six reflected appearances. The resulting questions separate the seven visible appearances from one physical identity and its reflection multiplicity.

Figure 11: Example 5: two physical objects paired with their mirror appearances. Four shoe appearances form two identity groups. The questions require two independent direct–reflection correspondences rather than a single global count.

Figure 12: Example 6: a direct camera repeated across two mirrors. Three camera appearances belong to one physical identity. The questions distinguish the direct camera, the reflected subset, their shared identity, and reflection multiplicity.

Model	People	Containers	Lighting	Art	Furniture	Architecture	Personal	Household	Electronics	Vehicles	Sports	Toys	Plants	Food	Animals	Other
Gemini 3.7 Flash	39.43	27.53	30.85	41.06	20.09	21.91	17.33	15.54	45.31	25.51	13.78	38.13	7.51	47.51	47.56	48.89
Muse-Glimmer-30B	30.16	23.31	26.72	34.90	27.08	26.96	17.12	20.59	56.25	22.04	23.84	39.23	20.65	41.39	49.34	29.19
Qwen3.8-27B	29.47	22.60	21.48	42.57	12.88	16.19	23.97	10.10	56.25	26.78	18.68	47.78	21.11	71.31	39.61	38.68
Qwen3.5-397B-A17B	28.57	20.44	22.68	40.11	25.75	29.82	29.77	10.56	27.08	19.15	15.68	34.57	24.26	48.07	48.26	25.35
Gemma 4-31B	27.88	19.67	16.46	35.00	21.57	11.89	8.63	2.62	75.00	19.98	16.85	29.48	10.90	42.15	31.87	45.24
Qwen3.5-27B	27.11	19.04	21.06	42.99	29.98	18.25	16.47	14.72	20.31	25.10	20.87	34.59	10.23	28.75	27.54	35.51

Appendix

Table 4: Performance across counting-target domains. Each entry is the group-macro Score (%) for one counting-target domain.

	Entity NAE $\downarrow$				Score $\uparrow$
Model	1	2	3	$\geq 4$	1	2	3	$\geq 4$
Gemini 3.7 Flash	27.55	26.58	26.29	32.54	40.54	35.27	31.19	30.07
Muse-Glimmer-30B	35.32	31.36	27.56	26.38	28.72	29.68	29.51	33.27
Qwen3.8-27B	35.75	30.19	27.70	36.07	29.98	28.19	30.26	26.61
Qwen3.5-397B	33.20	32.50	25.23	45.92	27.71	27.89	31.36	26.48
Gemma 4-31B	33.47	37.28	34.66	51.63	29.73	24.68	25.27	22.34

Appendix

Table 5: Performance across appearance multiplicity. Each entry is a group-macro percentage within the indicated maximum number of appearances per entity. The final bin produces the clearest rise in Entity NAE and the broadest decline in Score.

	Correspondence NAE $\downarrow$			Score $\uparrow$
Model	Within	Cross	$\Delta$	Within	Cross	$\Delta$
Gemini 3.7 Flash	30.08	35.65	+5.57	29.73	34.51	+4.78
Muse-Glimmer-30B	32.81	42.67	+9.86	23.87	30.54	+6.67
Qwen3.8-27B	24.17	45.03	+20.86	27.14	28.54	+1.40
Qwen3.5-397B	39.34	45.02	+5.68	24.03	28.49	+4.46
Gemma 4-31B	41.23	42.66	+1.43	16.34	25.06	+8.72

Appendix

Table 6: Identity correspondence within and across image-formation modes. Both conditions are reweighted to the same appearance-multiplicity distribution. Positive

\Delta

values indicate higher error or Score in the cross-mode condition.

Figure 13: An appearance inventory that is not pairwise balanced. The crop contains two directly visible plants but three plant reflections. The unmatched reflection remains an observed appearance even though its physical source is outside the direct view.

Figure 14: Four shoe appearances connected into two identities. The mirror boundary does not remove visual evidence; it determines which observations should be linked before the entities are counted.

Figure 15: A nonempty physical set but an empty physical-only set. The direct dancer and the two mirror appearances belong to one identity; the exclusion condition removes that identity from the answer.

Object Counting

Object counting estimates the number and locations of target instances in images or video, from dense crowds to fine-grained categories.

See all 62 papers

What do MLLMs know about 'Waldo'? Evaluating Visual Reasoning under Multi-Instance Multi-Object Challenges
Oct 28, 2026Saumya Mundra, Ajoy Mondal, Vineeth N Balasubramanian +1MLLM Evaluation Compositional Reasoning in VLMs
IIIT Hyderabad · Microsoft Research India · IIT Hyderabad
HoloCount: A Holistic Visual Counting Benchmark for MLLMs
Jul 7, 2026Jinhong Deng, Limeng Qiao, Guanglu WanMultimodal Robustness Object Counting
Meituan
HumanPhys: A Human-Centric Physical-World Perception Benchmark for Real-World Visual Reasoning
Oct 10, 2026Yiming Zhang, Enzhi Zhou, Junan Lin +4Human Motion Understanding VLM Reasoning
Nanyang Technological University, Singapore · Ant Group, Hangzhou, China

Oct 28, 2026J/K move · Enter open · S save

What do MLLMs know about 'Waldo'? Evaluating Visual Reasoning under Multi-Instance Multi-Object Challenges

Saumya Mundra, Ajoy Mondal, Vineeth N Balasubramanian, C. V. Jawahar

IIIT Hyderabad · Microsoft Research India · IIT Hyderabad

MLLM Evaluation Compositional Reasoning in VLMs

Open paper AskPDF arXiv

Recent advances in Multimodal Large Language Models (MLLMs) have led to significant progress in visual reasoning capabilities. However, existing benchmarks curate image-question pairs pertaining to object categories with few instances, enabling trivial object referencing and limited visual ambiguity. In this work, we introduce MIMO-Bench, a benchmark for evaluating MLLMs in a Multi-Instance per Multi-Object categories (MIMO) setting. Each image-question pair in our benchmark pertains to instances belonging to object categories with multiple instances. The presence of such similar feature distractors causes hindrance in basic visual reasoning tasks, necessitating the referring expression to be composed of multiple visual subtasks to uniquely and unambiguously identify the target object. MIMO-Bench comprises 3,802 image-question pairs across attribute binding and numerical counting tasks. Despite impressive performances on existing benchmarks, both SOTA open-source (InternVL, QwenVL) and proprietary (Gemini, GPT-5) models exhibit substantially lower performance on MIMO-Bench, highlighting their limitations in the MIMO setting. The benchmark will be made publicly available.

Abstract

Object Counting

Explore similar work

What do MLLMs know about 'Waldo'? Evaluating Visual Reasoning under Multi-Instance Multi-Object Challenges

HoloCount: A Holistic Visual Counting Benchmark for MLLMs

HumanPhys: A Human-Centric Physical-World Perception Benchmark for Real-World Visual Reasoning