What Factors Matter for Collection-Level Counting with Human-AI Collaboration?
Organizations: University of Massachusetts Amherst · University of Wyoming
Abstract
Many scientific and policy decisions require estimating the total number of target objects across a large image collection. Exhaustive human labeling is often infeasible, while raw model predictions can be biased, especially under domain shift. Recent human--AI estimation methods combine imperfect model predictions with limited human verification to obtain reliable collection-level estimates. However, existing work provides little guidance on how practitioners should choose among models, estimators, and adaptation strategies for a new deployment. We study these choices within a common human-in-the-loop counting framework on CLC-12, a benchmark of 12 tasks spanning biomedical microscopy, ground-level and aerial wildlife surveys, and overhead imagery of urban structures. Our results show that modern multimodal models provide a strong, training-free starting point for many tasks, while fine-tuned detectors become more advantageous at larger labeling budgets and across repeated deployments. Estimation efficiency improves further when model predictions are used to prioritize which units humans label, rather than sampling uniformly. We also find that each model's ranking quality can be estimated with a small pilot set, yielding practical guidance for selecting strategies on new counting tasks. In a field deployment on aerial pronghorn surveys, a task-specific adaptation of these methods meets the survey's 5% tolerance after reviewing a median of only 3.2% of collected images.
Figures & tables
| Estimator | Sampler | Model |
|---|---|---|
| Monte Carlo | uniform | — |
| {model-only, diff, ratio, reg} | uniform | {fine-tuned, frozen} |
| AIS, spotAIS | fine-tuned | |
| DIS, spotDIS | frozen |
| Tier (% ) | Dataset | Class | % | ||
|---|---|---|---|---|---|
| full ( ) | Malaria | trophozoite | 0.0 | 2.42 | 655 |
| Malaria | difficult | 0.0 | 1.29 | 347 | |
| Malaria | red blood cell | 2.1 | 49.19 | 1,688 | |
| partial ( – ) | Birds | bird | 24.4 | 14.36 | 1,302 |
| DOTA | tennis court | 37.5 | 4.63 | 689 | |
| DOTA | harbor | 52.1 | 3.90 | 2,084 |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Coverage | Half-width / | |||||||
|---|---|---|---|---|---|---|---|---|
| Estimator | full | part | spar | rare | full | part | spar | rare |
| Monte Carlo | 0.94 | 0.94 | 0.93 | 0.86 | 0.07 | 0.15 | 0.20 | 0.49 |
| difference | 0.89 | 0.93 | 0.88 | 0.88 | 0.09 | 0.12 | 0.18 | 0.47 |
| ratio | 0.87 | 0.92 | 0.88 | 0.86 | 0.12 | 0.12 | 0.16 | 0.43 |
| regression | 0.90 | 0.92 | 0.88 | 0.86 | 0.07 | 0.11 | 0.16 | 0.41 |
| DIS | 0.92 | 0.92 | 0.93 | 0.86 | 0.08 | 0.11 | 0.09 | 0.19 |
| Coverage | Half-width / | |||||||
|---|---|---|---|---|---|---|---|---|
| Estimator | ||||||||
| Monte Carlo | 0.86 | 0.90 | 0.92 | 0.94 | 0.38 | 0.28 | 0.19 | 0.10 |
| difference | 0.82 | 0.87 | 0.90 | 0.92 | 0.37 | 0.27 | 0.18 | 0.09 |
| ratio | 0.81 | 0.86 | 0.89 | 0.91 | 0.41 | 0.31 | 0.19 | 0.10 |
| regression | 0.81 | 0.86 | 0.89 | 0.92 | 0.36 | 0.26 | 0.17 | 0.08 |
| DIS | 0.87 | 0.89 | 0.91 | 0.92 | 0.31 | 0.21 | 0.14 | 0.07 |
| Tier | Dataset | Class | % | CV | img. pool | img. total | tile px | |||
| full ( empty) | Malaria | red blood cell | 2.1 | 49.19 | 0.77 | 1,688 | 83,034 | 1,328 | 1,328 | 1,168 |
| Malaria | trophozoite | 0.0 | 2.42 | 0.72 | 655 | 1,584 | 655 | 655 | 1,488 | |
| Malaria | difficult | 0.0 | 1.29 | 0.44 | 347 | 446 | 347 | 347 | 1,424 | |
| Malaria | schizont | 0.0 | 1.14 | 0.35 | 166 | 190 | 166 | 166 | 1,536 | |
| Malaria | gametocyte | 0.0 | 1.06 | 0.23 | 147 | 156 | 147 | 147 | 1,408 | |
| Malaria | leukocyte | 0.0 | 1.10 | 0.27 | 94 | 103 | 94 | 94 | 1,248 |
| MobileNetV3 | ResNet-50 | |||
|---|---|---|---|---|
| Estimator | Cov. | Shift | Cov. | Shift |
| faithful replay, in expectation | .90 | 0 | .90 | 0 |
| model-only | .61 | .64 | ||
| Monte Carlo (exact, measured) | .91 | .91 | ||
| difference | .86 | .89 | ||
| ratio | .86 | .88 | ||
| MLLM + OVD ( ) | MLLM ( ) | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| labeled | labeled | labeled | labeled | |||||||||||||
| Proxy | beats | win | ratio | beats | win | ratio | beats | win | ratio | beats | win | ratio | ||||
| 0.85 | 0.84 | 0.53 | 1.00 | 0.81 | 0.84 | 0.59 | 1.00 | 0.83 | 0.82 | 0.48 | 1.00 | 0.76 | 0.82 | 0.54 | 1.00 | |
| 0.66 | 0.79 | 0.47 | 1.02 | 0.62 | 0.78 | 0.46 | 1.06 | 0.65 | 0.77 | 0.42 | 1.02 | 0.56 | 0.75 | 0.41 | 1.06 | |
| RMSE | 0.44 | 0.72 | 0.39 | 1.15 | 0.40 | 0.72 | 0.39 | 1.18 | 0.53 | 0.76 | 0.36 | 1.11 | 0.47 | 0.74 | 0.35 | 1.12 |
| MAE | 0.41 | 0.70 | 0.37 | 1.15 | 0.36 | 0.70 | 0.37 | 1.22 | 0.52 | 0.75 | 0.33 | 1.13 | 0.44 | 0.72 | 0.32 | 1.18 |
| Model | $/1,000 units | Units/$1 | Normalized |
|---|---|---|---|
| Faster R-CNN | |||
| MobileNetV3 | $0.017 | 59,507 | 581.3 |
| ResNet-50 | $0.017 | 57,444 | 561.1 |
| Open-vocabulary detectors | |||
| Grounding DINO | $0.126 | 7,913 | 77.3 |
| Florence-2 | $0.410 | 2,440 | 23.8 |
| Model | Origin | Size | GPU-h/unit | API cost/unit | Extraction method |
| Qwen3-VL-8B | HuggingFace | 8B | — | Two-pass in-model count choice | |
| InternVL3.5-8B | HuggingFace | 8B | — | Two-pass in-model count choice | |
| Molmo-7B-D-0924 | HuggingFace | 7B | — | Two-pass in-model count choice | |
| Aya Vision-8B | HuggingFace | 8B | — | Two-pass in-model count choice | |
| Molmo2-8B | HuggingFace | 8B | — | Two-pass in-model count choice | |
| GPT-5.2 | OpenRouter | — | — | $0.004645 | “Final answer:” line + GPT-5.2 extractor pass |
| Prompt | Placeholders and details | Template |
|---|---|---|
| Open-source count (stage 1) | The placeholder {object_name} is replaced with the task-specific object name (e.g. “birds”, “storage tanks”). | How many {object_name} are in this image? Only output the number and no other text. |
| Open-source count extraction (stage 2, in-model second pass) | The placeholder {nlg} is the free-form text generated in stage 1, and {max_choice} is the upper bound of the structured choice list (default 300). The vLLM batched variant omits the “from 0 to {max_choice}” clause. | What number was this response trying to say? Response: {nlg} Answer only with one integer from 0 to {max_choice}. |
| Proprietary count (stage 1) | The placeholder {object_name} is replaced with the task-specific object name. | How many {object_name} are in this image? Think briefly from visible evidence, then finish with a line exactly like Final answer: <number>. |
| Proprietary count extraction (stage 2, GPT-5.2 extractor pass) | The placeholder {response} is the full reasoning text produced in stage 1. | Extract the count from the response below. Response: {response} Use the number on the final answer line when present. If the final answer is a range or alternative like ’2 or 3’, output the first number. Answer only with one number. If the final answer has no digit, answer unknown. |
| Verification (optional stage 3, GPT-5.2 verifier pass) | The placeholder {answer_text} is the extracted count (or “unknown”) and {response} is the stage-1 reasoning text. | Does the response explicitly contain this extracted count as its answer? Extracted count: {answer_text} Response: {response} Answer yes if the exact count is plainly present in the response. Answer no if the response does not plainly contain that count. |
Object Counting
Object counting estimates the number and locations of target instances in images or video, from dense crowds to fine-grained categories.
See all 64 papers