cs.CVFeb 28, 2026
SaveUNICBench: UNIfied Counting Benchmark for MLLM
Organizations: Northwestern Polytechnical University · Hong Kong University of Science and Technology · Institute of Artificial Intelligence (TeleAI), China Telecom · Sun Yat-sen University ·
Abstract
Counting is a core capability for multimodal large language models (MLLMs), yet there is no unified counting dataset to rigorously evaluate this ability across image, text, and audio. We present UNICBench, a unified multimodal, multi level counting benchmark and evaluation toolkit with accurate ground truth, deterministic numeric parsing, and stratified reporting. The corpus comprises 5,300 images (5,508 QA), 872 documents (5,888 QA), and 2,069 audio clips (2,905 QA), annotated with a three level capability taxonomy and difficulty tags. Under a standardized protocol with fixed splits/prompts/seeds and modality specific matching rules, we evaluate 45 state-of-the-art MLLMs across modalities. Results show strong performance on some basic counting tasks but significant gaps on reasoning and the hardest partitions, highlighting long-tail errors and substantial headroom for improving general counting. UNICBench offers a rigorous and comparable basis for measurement and a public toolkit to accelerate progress.
Figures & tables
| Category | Level | Image | Text | Audio |
|---|---|---|---|---|
| Capability level | Pattern (L1) | Q: “How many people are visible?” | Q: “How many times does ‘Figure’ appear?” | Q: “How many drum hits in this clip?” |
| Semantic (L2) | Q: “How many people are wearing red shirts ?” | Q: “How many non-repeated citations ?” | Q: “How many bird calls of species X ?” | |
| Reasoning (L3) | Q: “How many folders in the specified path shown in the screenshot have a modification date in the year 2022 ?” | Q: “Count references from 2010 2020 , excluding appendices .” | Q: “In this audio, how many questions are asked in total?” | |
| Difficulty level | Easy: 1–10 | low density, small occlusion | short span, low repetition | sparse, low overlap |
| Medium: 11–100 | moderate density, some occlusion | moderate repetition, some deduplication | moderate overlap, brief bursts | |
| Hard: 100 | high density, severe occlusion | long documents, heavy deduplication | overlapping events, long audio |
Table 1: Representative counting examples organized by category (capability vs. difficulty), level, and modality. The leftmost column groups three rows under “Capability level” (L1–L3) and a compact summary row for “Difficulty level” (Easy/Medium/Hard).
Figure 2: a) Sample visualization in three modalities. b) Overview of dataset composition. Sample and question distributions across modalities and the capability/difficulty breakdown. c) Smoothed ground‑truth count distributions for three modalities, which are skewed and long‑tailed and thus motivate our stratified difficulty thresholds and evaluation protocol. d) Category word cloud based on question counts. e) f) g) Distribution of resolution/text length/ audio duration in three modalities, respectively.
Figure 3: Overview of the UNICBench pipeline. We standardize multi-modal datasets and define a unified QA-evidence schema with difficulty and capability labels. The end-to-end framework enables assessment of MLLM counting across image, text, and audio.
| Model | Success (%) | Hit rate (%) | Overall | Easy | Medium | Hard | Pattern | Reasoning | Semantic | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @100% | @90% | @80% | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | ||
| Claude-Sonnet-4-20250514 | 100.0 | 14.9 | 27.8 | 42.4 | 78.1 | 1.1e6 | 5.4 | 6.1e3 | 16.8 | 4.5e3 | 444.6 | 7.0e6 | 68.8 | 1.2e6 | 4.4 | 5.7e1 | 191.6 | 4.1e5 |
| Gemini-2.5-Flash | 100.0 | 15.6 | 25.1 | 38.0 | 140.5 | 1.0e6 | 12.0 | 5.7e3 | 55.8 | 2.9e4 | 694.2 | 6.5e6 | 131.4 | 1.0e6 | 6.7 | 2.1e3 | 280.8 | 1.1e6 |
| Gemini-2.5-Pro-Thinking | 100.0 | 20.3 | 37.5 | 51.3 | 90.0 | 8.1e5 | 4.3 | 5.8e2 | 22.1 | 2.9e4 | 504.9 | 5.3e6 | 71.1 | 6.5e5 | 4.6 | 8.4e1 | 291.1 | 2.5e6 |
| GPT-4o-mini | 100.0 | 12.5 | 20.7 | 33.0 | 73.3 | 1.5e6 | 2.3 | 3.8e1 | 15.0 | 5.0e2 | 424.6 | 9.7e6 | 72.7 | 1.7e6 | 5.3 | 7.4e1 | 109.8 | 5.7e4 |
| GPT-5 | 100.0 | 16.8 | 32.3 | 48.3 | 54.1 | 2.0e6 | 2.5 | 2.1e2 | 11.0 | 3.1e2 | 312.4 | 1.4e7 | 55.1 | 2.4e6 | 5.9 | 1.0e2 | 67.4 | 3.3e4 |
Table 2: Benchmark results on the Image-modality counting track . Metrics are: SuccessRate (%), Hit rates (@100%/@90%@80%), MAE/MSE for Overall, per-difficulty, and per-capability. MSE values are shown in scientific notation with one decimal.
Figure 4: Distribution of prediction error on image modality . Whiskers and outliers indicate extreme failures—long whiskers or many outliers show a model makes severe errors on some samples (e.g., rare classes, label noise, or collapse cases). Such frequent extreme errors can substantially increase MAE/MSE even when the median error looks small.The model ordering is consistent with that in Table 2 .
| Model | Success (%) | Hit rate (%) | Overall | Easy | Medium | Hard | Pattern | Reasoning | Semantic | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @100% | @90% | @80% | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | ||
| Gemini-2.5-Flash-Nothinking | 99.0 | 37.3 | 43.7 | 50.5 | 67.0 | 3.4e5 | 7.4 | 2.6e4 | 24.1 | 7.1e4 | 746.3 | 4.1e6 | 33.3 | 1.7e4 | 120.1 | 7.2e5 | 19.8 | 7.3e4 |
| Gemini-2.5-Pro-Thinking | 99.2 | 63.3 | 72.8 | 76.1 | 30.5 | 9.0e4 | 3.6 | 4.5e2 | 11.3 | 1.0e4 | 337.4 | 1.2e6 | 10.5 | 3.2e3 | 56.4 | 2.0e5 | 10.6 | 1.1e4 |
| GPT-5-mini | 97.6 | 35.3 | 40.6 | 48.6 | 1665.2 | 1.4e10 | 2347.8 | 2.1e10 | 17.9 | 1.1e4 | 1321.4 | 2.9e7 | 20.8 | 4.4e3 | 3804.8 | 3.2e10 | 8.1 | 2.2e3 |
| GPT-o3 | 96.8 | 61.1 | 68.2 | 72.0 | 112.9 | 2.3e6 | 3.5 | 4.7e2 | 16.4 | 2.5e4 | 1510.8 | 3.2e7 | 15.5 | 1.0e4 | 245.6 | 5.2e6 | 5.6 | 1.5e3 |
| GPT-5 | 97.8 | 36.2 | 39.7 | 47.2 | 106.9 | 1.2e6 | 6.0 | 5.7e2 | 16.6 | 2.5e3 | 1393.6 | 1.7e7 | 23.6 | 4.2e3 | 225.8 | 2.7e6 | 7.0 | 1.8e3 |
Table 3: Benchmark results on the Text-modality counting track . Metrics are: SuccessRate (%), Hit rates (@100%/@90%@80%), MAE/MSE for Overall, per-difficulty, and per-capability. MSE values are shown in scientific notation with one decimal.
Figure 5: Prediction vs. ground truth scatter. Each point is one sample. The upper‑right green box highlights a small set of extreme high‑count samples where the thinking mode defeated non‑thinking mode, which drives most of the overall MAE gap.
| Model | Success (%) | Hit rate (%) | Overall | Easy | Medium | Hard | Pattern | Reasoning | Semantic | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @100% | @90% | @80% | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | ||
| Gemini-2.5-Pro-Thinking | 64.0 | 31.0 | 31.0 | 31.0 | 1.1 | 2.3e0 | 1.1 | 2.3e0 | - | - | - | - | 1.1 | 2.3e0 | - | - | - | - |
| Gemini-2.5-Flash-Nothinking | 64.0 | 48.0 | 48.0 | 48.0 | 0.7 | 1.0e0 | 0.7 | 1.0e0 | - | - | - | - | 0.7 | 1.0e0 | - | - | - | - |
| GPT-4o-mini-Audio-Preview | 64.0 | 11.3 | 11.3 | 11.3 | 1.7 | 4.1e0 | 1.7 | 4.1e0 | - | - | - | - | 1.7 | 4.1e0 | - | - | - | - |
| GPT-4o-Audio-Preview | 63.9 | 32.4 | 32.4 | 32.4 | 0.9 | 1.3e0 | 0.9 | 1.3e0 | - | - | - | - | 0.9 | 1.3e0 | - | - | - | - |
| GPT-Audio | 64.0 | 33.9 | 33.9 | 33.9 | 0.9 | 1.5e0 | 0.9 | 1.5e0 | - | - | - | - | 0.9 | 1.5e0 | - | - | - | - |
Table 4: Benchmark results on the audio‑modality counting track . Metrics are: SuccessRate (%), Hit rates (@100%/@90%@80%), MAE/MSE for Overall, per‑difficulty, and per‑capability. MSE values are shown in scientific notation with one decimal. A “-” indicates that no valid values were obtained for this statistical dimension.
Figure 6: Error-type analysis on the audio modality . Models that refuse or fail to produce numeric outputs more often avoid counting those errors in MAE (but are penalized in SuccessRate).
| Modality | Samples | Questions | Categories | Avg Count | Count Range |
|---|---|---|---|---|---|
| Image | 5,300 | 5,508 | 49 | 63.53 | [0, 10,294] |
| Text | 872 | 5,888 | 12 | 120.40 | [0, 71,176] |
| Audio | 2,069 | 2,905 | 2 | 30.32 | [0, 635] |
| Total | 8,241 | 14,301 | 63 | 71.42 | [0, 71,176] |
Table 5: Modality-Specific Construction Statistics
Figure 7: Modality comprehensive comparison. The figure illustrates cross-modal distributions among image, text, and audio, comparing sample and question counts, average values, difficulty types, and count distributions.
Figure 8: Count-range distribution across the three modalities. Image, text, and audio samples exhibit very different count scales, where text spans the widest range while image and audio concentrate in lower ranges—reflecting distinct counting difficulty patterns across modalities.
Figure 9: Distribution of question difficulty levels across modalities. Image questions are predominantly Pattern-level, whereas Text contains substantially more Reasoning and Semantic questions. Audio shows a more balanced mix, highlighting modality-dependent complexity differences.
Figure 10: Question length distribution across modalities. The left histogram shows the overall length distribution of all counting questions, while the right boxplot compares question lengths across image, text, and audio modalities. Text questions are significantly longer and more variable, indicating higher linguistic complexity and reflecting the greater reasoning demand in text-based counting tasks.
Figure 11: Category-level statistics heatmap (top 30 by sample count). This heatmap summarizes the top 30 categories by sample count, showing clear modality-dependent differences: audio–environment contributes large volumes with low counts, text categories exhibit high-density and complex count structures, while image categories remain more uniform with moderate difficulty.
| Level | Cognitive Demand | Example Tasks |
|---|---|---|
| Pattern (L1) | Direct perception | Count visible objects, discrete events |
| Semantic (L2) | Attribute filtering + deduplication | Unique libraries, distinct speakers |
| Reasoning (L3) | Multi-step inference | Rule-based counting, logical constraints |
Table 6: Sample Difficulty Characteristics
Figure 12: Example of a pattern-level sample from the image modality and the answer from different models.
Figure 13: Example of a Semantic-level sample from the image modality,and the answer from different models.
Figure 14: Example of a reasoning-level sample from the image modality, and the answer from different models.
Figure 15: Example of a pattern-level sample from the text modality, and responses from different models.
Figure 16: Example of a semantic-level sample from the text modality, and responses from different models.
Figure 17: Example of a reasoning-level sample from the text modality, and responses from different models.
Figure 18: Example of a pattern-level sample from the audio modality and the answer from different models.
Figure 19: Example of a semantic-level sample from the audio modality and the answer from different models.
Figure 20: Example of a reasoning-level sample from the audio modality and the answer from different models.
Figure 21: Image modality accuracy comparison. Exact-match accuracy remains low across models, while allowing small error tolerance (10% and 20%) yields significantly higher performance, revealing strong approximate-counting ability but limited precise counting.
Figure 22: Performance across difficulty levels in the image modality. Models perform best on Pattern-level (L1) samples, degrade on Semantic-level (L2), and show the largest errors on Reasoning-level (L3) samples, demonstrating the expected progression of counting difficulty.
Figure 23: Text modality accuracy comparison. Models achieve higher accuracy on text counting than on other modalities under relaxed error thresholds, but Exact Match remains challenging, especially for complex or long documents.
Figure 24: Performance across difficulty levels in the text modality. Performance degrades from Pattern-level (L1) to Semantic-level (L2) and Reasoning-level (L3), reflecting the increased complexity of attribute filtering, reference resolution, and multi-step reasoning required by higher levels.
Figure 25: Audio modality accuracy comparison. Overall accuracy of audio-capable models under Exact Match, 10% error, and 20% error thresholds. Audio counting shows lower precision due to temporal ambiguity and variable acoustic patterns.
Figure 26: Audio performance across difficulty levels. MAE and MSE (log 2 scale) across L1–L3 tasks show increasing error with complexity, highlighting the challenge of temporal reasoning in audio-based counting.
| Modality | None | Out of Context | Out of Thinking | Incorrect Format |
|---|---|---|---|---|
| Image | 114,534 | 0 | 1,052 | 82 |
| Text | 125,919 | 7,019 | 0 | 2,386 |
| Audio | 28,850 | 6,271 | 769 | 830 |
| Total | 269,303 | 13,290 | 1,821 | 3,298 |
Table 7: Distribution of error types across modalities.
Figure 27: MAE and MSE comparison for the image modality (log 2 scale). Numerical labels indicate original (non-log) error values.
Figure 28: Category-level MAE heatmap for image-based counting . Rows denote categories and columns correspond to models. Darker colors represent larger numerical deviations.
Figure 29: MAE and MSE comparison for the text modality (log 2 scale). Numerical labels denote original (non-log) error values.
Figure 30: Category-level MAE heatmap for text-based counting . Categories differ widely in structural complexity, causing substantial variation in numerical deviation.
Figure 31: MAE and MSE comparison for the audio modality (log 2 scale). Despite small ground-truth counts, numerical deviations remain large across many models.
Figure 32: Category-level MAE heatmap for audio counting . Environmental sounds are relatively easier, while conversational speech produces disproportionately large errors.
More in this topic
Multimodal Large Language Models
Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.
See all 1,091 papers