CrowdCue: Specialist-Cue Conditioning for Vision-Language Crowd Counting

Moshiur Farazi; Bekir Ciftler; Abdulhalim Dandoush; Reda Bendraou

cs.CVSep 19, 2026

CrowdCue: Specialist-Cue Conditioning for Vision-Language Crowd Counting

Authors: Moshiur Farazi, Bekir Ciftler, Abdulhalim Dandoush, Reda Bendraou

Organizations: University of Doha for Science and Technology, Doha, Qatar

Abstract

Generative vision-language models (VLMs) offer a counting paradigm in which one model produces both a count and a natural-language account of the scene, yet their raw counting accuracy sits in the range of sub-million-parameter specialist regressors. The open question is whether auxiliary guidance from a pretrained specialist can lift them into useful territory, and through which channel that guidance is best routed. We evaluate Qwen2.5-VL-7B on four widely used crowd counting benchmarks (ShanghaiTech A and B, UCF-QNRF, NWPU-Crowd). Zero-shot prompting rarely produces a parseable count, so LoRA supervised fine-tuning establishes the baseline at overall MAE 81.64. Conditioning on a P2PNet-derived density heatmap as an auxiliary visual signal fails in every encoding we tested, and an adversarial-swap protocol shows the model reads the heatmap but applies it counterproductively. We propose CrowdCue, a family that supplies the same specialist's already-integrated integer count to the VLM as a discrete symbol. The text-channel variant reaches MAE 72.04. The visual-channel variant, which renders the integer as printed digits and supplies it as a second image, reaches MAE 62.65, the strongest result in this paper and well ahead of the cue-supplying specialist alone (84.45 on the same split). In the late-fusion VLM we study, the binding constraint is not the channel but the abstraction level at which the specialist signal is delivered.

Figures & tables

Method	Year	Paradigm	Key contribution
ZIP ( Ma et al., 2025b )	2025	Density	Zero-Inflated Poisson loss; SHA-A 47.8 (105M), 71.1 (0.81M)
DSGC-Net ( Wu et al., 2025 )	2025	Density	Dual-stream graph CNN; SHA-A 48.9, SHB 5.9
RCCFormer ( Chen et al., 2025 )	2025	Density	Multi-feature Transformer; SHA 48.3, QNRF 77.6, NWPU 74.3
TCFormer ( Liu et al., 2025 )	2025	Density	5M-parameter weakly-supervised Transformer
CrowdDiff ( Ranasinghe et al., 2024 )	2024	Diffusion	Multi-hypothesis density; SHA-A 47.4
STEERER ( Han et al., 2023 )	2023	Density	Selective inheritance for scale variation

Table 1: Summary of related work. The three families are separated by horizontal rules.

Figure 1: Overview of the evaluation framework. A crowd image is optionally paired with a P2PNet-derived density heatmap and fed to Qwen2.5-VL-7B under one of two training regimes. We evaluate three heatmap conditioning strategies against the single-image baseline. The CrowdCue variants (Section 3.7 ) replace the heatmap with a specialist integer estimate.

Figure 2: Examples of P2PNet-derived density heatmaps across four density levels, with the corresponding printed-digit cue used by CrowdCue-V. Warm colours (red, yellow) indicate high crowd density; cool colours (blue, green) indicate low density.

Dataset	Train	Test	Count Range	Scene Type
ShanghaiTech A	300	182	33–3,138	Dense gatherings
ShanghaiTech B	400	316	9–576	Street scenes
UCF-QNRF	1,201	334	49–12,865	Web-crawled crowds
NWPU-Crowd	3,109	500	0–20,033	Diverse scenes
Total	5,010	1,332

Table 2: Dataset summary. NWPU-Crowd test labels are not public; we use the validation split as test.

Configuration	Valid Format	Rate
Zero-shot, single image	179 / 1,332	13.4%
Zero-shot, dual image	11 / 1,332	0.8%
SFT, single image	1,332 / 1,332	100%
SFT, dual image	1,332 / 1,332	100%

Table 3: Format compliance: fraction of test images producing a valid parseable count.

	SHA-A		SHA-B	QNRF	NWPU	Overall
Config	MAE	RMSE	MAE	MAE	MAE	MAE
SFT single	70.17	116.01	15.47	119.19	102.55	81.64
SFT dual	71.16	114.45	13.93	153.61	119.13	96.26

Table 4: SFT results on four benchmarks. Best MAE per dataset in bold . The single-image baseline outperforms the dual-image model with a valid P2PNet-derived heatmap on three of four datasets and overall.

Configuration	SHA-A	SHB	QNRF	NWPU	Overall	RMSE
SFT-single (no heatmap, baseline)	70.17	15.47	119.19	102.55	81.64	352.66
SFT-dual (valid P2PNet heatmap)	71.16	13.93	153.61	119.13	96.26	443.13
SFT-side-by-side (valid heatmap)	75.80	16.84	156.40	136.63	104.86	497.93
SFT-overlay $\alpha=0.4$ (valid heatmap)	73.37	17.66	170.30	560.86	267.45	4586.92

Table 5: Three visual encodings of the (valid) P2PNet density heatmap as SFT auxiliary signal. All degrade MAE relative to the no-heatmap baseline. Trained separately, evaluated on the combined 1,332-image test split. The final column is overall RMSE across all test images.

Heatmap condition	MAE	$\Delta$ from correct
Correct (P2PNet heatmap)	96.26	—
Inverted (hot $\leftrightarrow$ cold, same spatial pattern)	97.85	$+$ 1.6
Gray (uniform, no spatial pattern)	101.80	$+$ 5.5
Random noise	103.12	$+$ 6.9
Shuffled (different valid heatmap)	119.65	$+$ 23.4

Table 6: Adversarial heatmap-swap MAE on the dual-image SFT with valid P2PNet heatmaps. Spread is 23.39 , with the shuffled condition (a valid heatmap from a different image) the worst. The model is causally sensitive to the heatmap content but applies it counterproductively.

Configuration	Channel	Form	SHA-A	SHB	QNRF	NWPU	Overall	RMSE
SFT-single (baseline)	—	none	70.17	15.47	119.19	102.55	81.64	352.66
P2PNet specialist alone	—	integer source	61.26	21.93	121.88	107.41	84.45	221.93
SFT-dual (valid heatmap)	visual	dense field	71.16	13.93	153.61	119.13	96.26	443.13
CrowdCue-T (Sec. 4.7 )	text	integer	66.08	11.74	113.44	84.65	72.04	249.83
Per-region (text, $2\times 2$ )	text	4 sub-counts + total	71.74	12.80	115.20	78.36	71.19	191.47
Grid (text, $4\times 4$ )	text	16 sub-counts + total	68.68	13.15	120.06	79.19	72.33	213.12

Table 7: Two mechanism controls separating channel from abstraction level, with the standalone specialist added for reference. CrowdCue-V (visual channel, integer symbol) yields the lowest MAE and RMSE of any configuration in the paper. The per-region text injection adds only marginal improvement over the single-integer text. The final column is overall RMSE across all 1,332 test images.

Inject override	MAE	$\Delta$ from correct
Correct $\hat{n}_{P}$	72.04	—
$0$	96.90	$+$ 24.9
Random $\in\{0,\ldots,1000\}$	133.08	$+$ 61.0
Half $\hat{n}_{P}$	138.86	$+$ 66.8
$10\cdot\hat{n}_{P}$	376.22	$+$ 304.2

Table 8: Adversarial overrides on CrowdCue-T, evaluated on the combined 1,332-image test split.

\Delta

is the change in overall MAE from the correct injection.

Figure 3: Left: overall MAE for single-image, dual-image, and CrowdCue-T SFT models; the dashed line marks the single-image baseline. Right: MAE under the four adversarial text-channel overrides on a log scale. The 304-point spread under the

10\times

override contrasts with the 23.4-point spread across the five visual adversarial conditions on valid heatmaps (Section 4.5 ).

Override	Sparse	Medium	Dense	Very dense	Extreme	Overall
Correct	16.78	13.71	50.67	122.77	379.70	72.04
Zero	16.85	19.30	56.62	205.50	498.37	96.90
Half	11.68	20.48	75.56	250.84	832.50	138.86
Random	10.19	20.73	90.55	206.67	783.17	133.08
$10\times$	16.77	16.47	68.94	249.12	3393.13	376.22

Table 9: Per-density-bucket MAE for CrowdCue-T under correct and adversarial overrides. The override effect is concentrated in the very-dense and extreme buckets; the sparse bucket is nearly insensitive.

Figure 4: Per-image anchoring coefficient

\alpha=\partial\hat{y}/\partial\hat{n}_{P}

for CrowdCue-T, estimated by linear regression through the five injection values evaluated per image. Left:

\alpha

against ground-truth count, coloured by density bucket (darker = denser). Right: per-bucket distribution, with diamonds marking bucket means. Mean

\alpha

rises monotonically from near zero in sparse scenes (the model ignores the cue) to 0.106 in extreme scenes (the model anchors on it), an input-conditional trust policy learned from the gradient signal alone. The

\alpha

axis is clipped at 0.32 for readability; nine of 1,332 images exceed the clip (maximum 1.30) and are drawn as triangles at the top edge.

Config	SHA-A	SHB	QNRF	NWPU (held out)	Overall
SFT single (excl-NWPU)	80.03	22.70	172.96	133.99	109.99
CrowdCue-T (excl-NWPU)	70.84	13.78	116.34	106.18	81.97

Table 10: Cross-dataset transfer: SFT on SHA + SHB + QNRF only (1,901 samples), evaluated on the full test set. NWPU is held out from training. The CrowdCue-T advantage transfers: it leads on the held-out NWPU and on all three in-distribution datasets.

Assistant target	SHA-A	SHB	QNRF	NWPU	Overall
Verify-and-adjust template	74.15	13.46	110.08	91.23	75.17
Flat scene descriptor (CrowdCue-T)	66.08	11.74	113.44	84.65	72.04

Table 11: Training-target ablation. Both variants use the same image input and the same text-injection prompt; only the assistant SFT target differs. The flat target reaches lower MAE on every dataset.

Predictor	MAE	RMSE
P2PNet alone	84.45	221.93
SFT-single alone	81.64	352.66
Mean of the two	74.13	256.37
Fitted linear blend (5-fold CV)	87.80	259.59
Per-bucket gate (P2PNet-predicted bucket)	75.42	217.03
Per-bucket gate (ground-truth bucket, oracle)	70.00	195.19

Table 12: CrowdCue against fixed-rule fusion baselines computed from stored per-image predictions of the same two estimators (SFT-single and P2PNet). The gate baselines pick the better predictor per density bucket, using either the P2PNet-predicted bucket (realistic) or the ground-truth bucket (oracle). CrowdCue-T is ahead of every realistic rule; CrowdCue-V is ahead of the oracle bound.

Density Bucket	Count Range	$n$ (test)	MAE
Sparse	0–50	171	8.99
Medium	50–200	498	16.20
Dense	200–500	378	52.89
Very dense	500–1,000	160	144.20
Extreme	$>$ 1,000	125	448.56

Table 13: MAE by density bucket for SFT single-image model. Error scales superlinearly with crowd count.

Method	Params	SHA-A	SHA-B	QNRF	NWPU	Free-form output
ZIP-B ( Ma et al., 2025b )	105.6M	47.8	5.5	69.4	60.1	no
DSGC-Net ( Wu et al., 2025 )	$\sim$ 30M	48.9	5.9	—	—	no
RCCFormer ( Chen et al., 2025 )	—	48.3	6.6	77.6	74.3	no
CrowdDiff ( Ranasinghe et al., 2024 )	—	47.4	—	—	—	no
CLIP-EBC ( Ma et al., 2025a )	—	55.0	6.3	—	—	partial
ZIP-P ( Ma et al., 2025b )	0.81M	71.1	—	—	—	no

Table 14: VLM counting against the 2025 specialist front. Numbers are MAE; — denotes a result the original paper did not report. The last column flags methods that emit free-form reasoning alongside the count. ZIP-P matches our 7B VLM on SHA-A with 0.81M parameters; CrowdCue-V closes most of the baseline-to-ZIP-B gap on NWPU and roughly a quarter of it on the other datasets. Our NWPU column uses the validation split (test labels are not public), while published NWPU results are test-server numbers where available, so NWPU cells are not split-matched across rows. The P2PNet row reports our measurement with the SHA-A checkpoint applied to all four datasets (Section 4.7 ).

Figure 5: Qualitative comparison of SFT-single, CrowdCue-T, and CrowdCue-V predictions across four density levels, with ground truth and per-model absolute error for each example.

Crowd Counting

Crowd counting estimates how many people appear in images or video, supporting occupancy monitoring, safety planning, and crowd management.

See all 23 papers

Counting to Four is still a Chore for VLMs
Apr 11, 2026Duy Le Dinh Anh, Patrick Amadeus Irawan, Tuan Van VoAttention Intervention in VLMs Vision-Language Models
MBZUAI
CounterCount: Diagnosing Knowledge Bias in How Vision-Language Models Count Attributes
May 18, 2026Reem Alzahrani, Bushra Bin Hemid, Hassan Alshanqiti +4VLM Evaluation Vision-Language Models
King Abdullah University of Science and Technology, Thuwal, Saudi Arabia. · University of Edinburgh
Counting Circuits: Mechanistic Interpretability of Visual Reasoning in Large Vision-Language Models
Mar 19, 2026Liwei Che, Zhiyu Xue, Yihao Quan +7VLM Reasoning Large Vision-Language Models
Rutgers University · UC Santa Barbara · University of Washington +1

Apr 11, 2026J/K move · Enter open · S save

Counting to Four is still a Chore for VLMs

Duy Le Dinh Anh, Patrick Amadeus Irawan, Tuan Van Vo

MBZUAI

Attention Intervention in VLMs Vision-Language Models VLM Evaluation VLM Interpretability

Open paper AskPDF arXiv

Vision--language models (VLMs) have achieved impressive performance on complex multimodal reasoning tasks, yet they still fail on simple grounding skills such as object counting. Existing evaluations mostly assess only final outputs, offering limited insight into where these failures arise inside the model. In this work, we present an empirical study of VLM counting behavior through both behavioral and mechanistic analysis. We introduce COUNTINGTRICKS, a controlled evaluation suite of simple shape-based counting cases designed to expose vulnerabilities under different patchification layouts and adversarial prompting conditions. Using attention analysis and component-wise probing, we show that count-relevant visual evidence is strongest in the modality projection stage but degrades substantially in later language layers, where models become more susceptible to text priors. Motivated by this finding, we further evaluate Modality Attention Share (MAS), a lightweight intervention that encourages a minimum budget of visual attention during answer generation. Our results suggest that counting failures in VLMs stem not only from visual perception limits, but also from the underuse of visual evidence during language-stage reasoning. Code and dataset will be released at https://github.com/leduy99/-CVPRW26-Modality-Attention-Share.

Abstract

Crowd Counting

Explore similar work

Counting to Four is still a Chore for VLMs

CounterCount: Diagnosing Knowledge Bias in How Vision-Language Models Count Attributes

Counting Circuits: Mechanistic Interpretability of Visual Reasoning in Large Vision-Language Models