Dynamic Query as Budgeting for Tiny Object Detection

Yi Zhang; Di Xiong; Ge Gao; Xiangyue Zhang; Yihang Qiu; Yuxuan Zhou; Shuo Chen

cs.CVOct 24, 2026

Dynamic Query as Budgeting for Tiny Object Detection

Authors: Yi Zhang, Di Xiong, Ge Gao, Xiangyue Zhang, Yihang Qiu, Yuxuan Zhou, Shuo Chen

Organizations: School of Intelligence Science and Technology, Nanjing University, China · Center for Advanced Intelligence Project, RIKEN, Japan

Abstract

Recent DETR-based tiny object detectors adopt dynamic query mechanisms to handle severe density imbalance. However, existing designs entangle two effects: how many and which queries are decoded. This entanglement obscures what drives the gains and complicates dynamic-query design; additional learned predictors and heuristic filtering also introduce overhead and potential selection errors. We disentangle these effects and show that dynamic query primarily acts as budgeting---allocating decoder query capacity according to object density---rather than intricate query selection.Motivated by this view, we formulate a monotonic and conservative budgeting principle and propose a budgeting-based DETR (BUTR), a streamlined framework that uses a lightweight, non-learned budgeter to estimate an input-dependent query budget $K(x)$ from encoder confidences. BUTR retains standard Top- $K(x)$ query initialization without additional learned modules or heuristic filtering in the budgeter.At $800\times800$ resolution, BUTR-L achieves 35.7 AP at 47.8 FPS on AI-TOD-v2 and 29.8 AP at 42.1 FPS on VisDrone on an NVIDIA T4 GPU. Inference-time integration on COCO further demonstrates its applicability to general scenario detection. Code is available at https://github.com/zhangyi-magic/BUTR.

Figures & tables

Panel of Figure 1 — Figure 1: Qualitative comparison on AI-TOD-v2. Detection outcomes are color-coded as true positives (TP, green), false positives (FP, blue), and false negatives (FN, red). Replacing Dome-DETR’s dynamic-query initialization in (a) with a fixed Top- $600$ initializer in (c) yields highly similar detection results, whereas the smaller fixed budget in (b) (Top- $300$ ) is clearly insufficient for this extreme-density scene.

Init.	Configuration			AI-TOD-v2		VisDrone
Init.	Split	Mask	NMS	\|Q\| ( $\mu\scriptstyle\,\pm\,\sigma$ )	AP ( $\Delta$ )	\|Q\| ( $\mu\scriptstyle\,\pm\,\sigma$ )	AP ( $\Delta$ )
Full PAQI	✓	✓	✓	$630{\scriptstyle\,\pm\,229}$	35.1	$388{\scriptstyle\,\pm\,21}$	29.3
Operator Variants	✓	✓	$\times$	$875{\scriptstyle\,\pm\,395}$	35.1 ( $\mathbf{\pm 0.0}$ )	$494{\scriptstyle\,\pm\,14}$	29.3 ( $\mathbf{\pm 0.0}$ )
	✓	$\times$	✓	$1083{\scriptstyle\,\pm\,116}$	35.1 ( $\mathbf{\pm 0.0}$ )	$449{\scriptstyle\,\pm\,14}$	29.3 ( $\mathbf{\pm 0.0}$ )
	$\times$	✓	$\times$	$759{\scriptstyle\,\pm\,484}$	32.8 ( $-\mathbf{2.3}$ )	$491{\scriptstyle\,\pm\,23}$	29.2 ( $-\mathbf{0.1}$ )
	$\times$	✓	✓	$461{\scriptstyle\,\pm\,299}$	32.7 ( $-\mathbf{2.4}$ )	$299{\scriptstyle\,\pm\,40}$	28.9 ( $-\mathbf{0.4}$ )

Table 1: Inference-time ablation on AI-TOD-v2 and VisDrone. A Dome-DETR-L is trained on the respective trainval split and evaluated on the test set. For each configuration, we report query count

|Q|

(mean

\mu

\pm

std

\sigma

) and overall AP.

\Delta

is relative to the full PAQI pipeline (Row 1).

Init.	AI-TOD-v2				VisDrone
	Non-extreme		Extreme		Non-dense		Dense
	$\|Q\|$ ( $\mu\scriptstyle\,\pm\,\sigma$ )	AP ( $\Delta$ )	$\|Q\|$ ( $\mu\scriptstyle\,\pm\,\sigma$ )	AP ( $\Delta$ )	$\|Q\|$ ( $\mu\scriptstyle\,\pm\,\sigma$ )	AP ( $\Delta$ )	$\|Q\|$ ( $\mu\scriptstyle\,\pm\,\sigma$ )	AP ( $\Delta$ )
PAQI	$629{\scriptstyle\,\pm\,227}$	35.1	$1326{\scriptstyle\,\pm\,100}$	29.9	$388{\scriptstyle\,\pm\,22}$	29.1	$386{\scriptstyle\,\pm\,14}$	34.6
Fixed Top- $K$	600	35.1 ( $\mathbf{\pm 0.0}$ )	600	23.4 ( $-\mathbf{6.5}$ )	400	29.1 ( $\mathbf{\pm 0.0}$ )	400	34.5 ( $-\mathbf{0.1}$ )
	1500	35.1 ( $\mathbf{\pm 0.0}$ )	1500	29.9 ( $\mathbf{\pm 0.0}$ )

Table 2: Coarse density-stratified evaluation on AI-TOD-v2 and VisDrone.

Panel of Figure 2 — Figure 2: Fine density-stratified AP–budget response. Curves show AP versus fixed decoder budget $K$ within each density bucket: Minimal ( $\#\mathrm{GT}\leq 50$ ), Sparse ( $50<\#\mathrm{GT}\leq 100$ ), Moderate ( $100<\#\mathrm{GT}\leq 300$ ), and Dense ( $300<\#\mathrm{GT}\leq 600$ ). Diamonds mark PAQI’s bucket-wise mean query count $\mu$ with horizontal bars for $\pm\sigma$ ; squares denote budget-matched fixed Top- $K$ baselines at $K\in\{\mu-\sigma,\mu,\mu+\sigma\}$ .

Figure 3: Overview of query initialization: (1) fixed-query DETRs (e.g., Deformable DETR Zhu et al. (2020) , DINO Zhang et al. (2022) , and RT-DETR Zhao et al. (2024) ); dynamic-query DETRs include (2) DQ-DETR Huang et al. (2024) , (3) Dome-DETR Hu et al. (2025) , and (4) BUTR.

Method	#Params	GFLOPs	$\bm{\|Q\|}$	Tr. (h)	Lat. (ms)	AP	AP 50	AP 75	AP vt	AP t	AP s	AP m
Fixed-query detectors
D-FINE-S Peng et al. (2024)	11.6M	115.1	600	33.7	11.27	32.1	65.5	27.4	17.5	32.3	38.1	45.7
D-FINE-M Peng et al. (2024)	22.3M	248.2	600	40.4	16.86	34.3	68.0	30.3	19.7	34.8	39.1	47.7
D-FINE-L Peng et al. (2024)	31.9M	329.6	600	43.0	20.34	35.2	69.3	31.1	20.4	36.0	40.0	47.9
DEIM-S Huang et al. (2025)	11.6M	115.1	600	36.6	11.26	29.0	61.0	23.9	16.8	28.6	35.0	41.7
DEIM-M Huang et al. (2025)	22.3M	248.2	600	43.4	16.92	33.7	67.3	30.1	20.3	34.3	38.4	46.1

Table 3: Comparison on AI-TOD-v2. DQ-DETR results use the official checkpoint; others are our re-implementations. Dynamic-query GFLOPs and

|Q|

are averages. Tr. (h): training time (hours); Lat. (ms): T4 TensorRT latency (milliseconds). Ratios are computed relative to Dome-DETR at the same model scale.

Method	#Params	GFLOPs	$\bm{\|Q\|}$	Tr. (h)	Lat. (ms)	AP	AP 50	AP 75	AP s	AP m	AP l
Fixed-query detectors
D-FINE-S Peng et al. (2024)	11.6M	113.3	400	6.7	11.39	26.6	45.1	27.0	16.8	37.0	53.6
D-FINE-M Peng et al. (2024)	22.3M	245.9	400	9.2	17.35	29.2	48.9	30.0	18.9	40.5	53.8
D-FINE-L Peng et al. (2024)	34.5M	329.5	400	11.9	22.06	29.6	49.6	30.2	19.0	40.8	55.7
DEIM-S Huang et al. (2025)	11.6M	113.3	400	8.5	11.23	21.9	41.1	22.4	17.3	27.5	31.7
DEIM-M Huang et al. (2025)	22.3M	245.9	400	10.7	17.40	24.5	44.6	24.9	18.4	31.8	36.8

Table 4: Comparison on VisDrone. Results are from our re-implementations. Dynamic-query GFLOPs and

|Q|

are averages. Tr. (h): training time (hours); Lat. (ms): T4 TensorRT latency (milliseconds).

Panel of Figure 4 — Figure 4: Query usage versus image density. Each point corresponds to one test image; solid lines and shaded regions denote the binned mean and standard deviation, respectively. The dashed gray line is an ideal reference proportional to #GT. BUTR shows the clearest monotonic increase with image density.

Panel of Figure 5 — Figure 5: Density-stratified tail performance across model families and scales. (a) AP ext on the extreme subset of AI-TOD-v2. (b) AP den on the dense subset of VisDrone.

Category	Variant	$\bm{\|Q\|}$	Tr. (h)	AP	AP 50	AP 75	AP vt	AP t	AP s	AP m	AP ext
Estimator $e$	Max	302	37.8	35.6	69.6	31.5	20.4	36.4	40.4	49.0	27.7
	Max-sigmoid	303	37.8	35.5	69.3	32.2	21.1	36.1	40.5	49.1	28.9
	Sum-prob	304	41.5	35.3	69.4	31.1	20.3	36.0	39.6	47.0	31.0
	Union-prob	304	40.7	35.2	68.9	31.4	18.9	36.0	40.2	48.5	28.3
Mapping $g$	$\gamma=1.2$	305	/	35.6	69.4	32.2	21.2	36.2	40.6	49.1	29.5
	$\gamma=1.5$	308	/	35.6	69.5	32.2	21.2	36.2	40.6	49.2	29.6

Table 5: Ablations of budget estimation and mapping methods for BUTR-L on AI-TOD-v2. Estimator

e

variants are trained and evaluated without an additional mapping (i.e.,

\gamma=1.0

), while the mapping

g

study is conducted by inference-time ablation using the Max-sigmoid estimator. Tr. (h): training wall-clock time.

Method	#Params	GFLOPs	$\bm{\|Q\|}$	Lat. (ms)	AP val	AP ${}^{val}_{50}$	AP ${}^{val}_{75}$	AP ${}^{val}_{s}$	AP ${}^{val}_{m}$	AP ${}^{val}_{l}$
D-FINE-S Peng et al. (2024)	10.2M	25.2	300	3.63	48.5	65.4	52.6	29.3	52.2	65.4
D-FINE-S+BUTR	10.2M	24.6	232	4.24	48.6 ( $+$ 0.1 )	65.5	52.8	29.3	52.1	65.2
D-FINE-M Peng et al. (2024)	19.3M	56.7	300	5.66	52.3	69.9	56.4	33.2	56.5	70.2
D-FINE-M+BUTR	19.3M	56.6	289	6.44	52.4 ( $+$ 0.1 )	70.0	56.5	33.2	56.6	70.1
D-FINE-L Peng et al. (2024)	30.8M	91.0	300	7.94	54.0	71.6	58.4	36.5	58.0	71.9
D-FINE-L+BUTR	30.8M	90.2	253	8.81	54.1 ( $+$ 0.1 )	71.8	58.4	36.6	58.0	72.0

Table 6: Generality of BUTR on COCO under an inference-time plug-in setting without retraining. We report average GFLOPs and

|Q|

for the BUTR-augmented variants. Lat. (ms): T4 TensorRT latency (App. H ).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Initialization	AI-TOD-v2		VisDrone
Initialization	$\|Q\|$	AP	$\|Q\|$	AP
Original Top- $K$	600	35.2	400	29.6
DDQ-style	600	35.2	400	29.6

Appendix

Table 7: DDQ-style initialization within D-FINE-L. Each variant is separately trained with its initializer, which is also used at inference. Original Top-

K

results are those reported in Tabs. 3 and 4 .

Panel of Figure 6 — Figure 6: Qualitative examples of window-mask filtering. Green and red regions indicate mask-positive (preserved) and mask-negative (filtered) areas, respectively. Cyan stars denote GT centers; green/red points are preserved/filtered high-confidence proposals. Without Split, high-confidence proposals aligned with GT can be erroneously filtered.

Panel of Figure 7 — Figure 7: Qualitative examples of encoder proposal redundancy using different budgets. Cyan stars denote GT centers, and proposal points are colored from yellow to red by confidence. Even with Top-300, most GT objects are already covered by nearby proposals; increasing the budget to Top-600 mainly increases proposal density and redundancy rather than introducing new object coverage. This redundancy helps explain why budget-matched Top- $K$ can closely match PAQI in overall AP.

Panel of Figure 8 — Figure 8: Full density-stratified results on AI-TOD-v2, including the main-text extreme subset.

Panel of Figure 9 — Figure 9: Full density-stratified results on VisDrone, including the main-text dense subset.

Method	Run 1	Run 2	Run 3	AP ext
Dome-DETR-L	29.9	29.1	28.8	$29.27\pm 0.57$
BUTR-L	29.8	29.7	29.4	$\bm{29.63\pm 0.21}$

Appendix

Table 8: Fixed-seed reproducibility of AP ext on AI-TOD-v2: three independent seed-0 training runs. Run 1 is the originally reported result. Mean and standard deviation describe variation across training runs.

Method	Seed 0	Seed 1	Seed 2	AP ext
Dome-DETR-L	29.9	27.2	27.5	$28.20\pm 1.48$
BUTR-L	29.8	29.3	29.3	$\bm{29.47\pm 0.29}$

Appendix

Table 9: Multi-seed robustness of AP ext on AI-TOD-v2 using matched seeds 0, 1, and 2. Seed 0 is the originally reported result. Mean and sample standard deviation describe variation across seeds.

Split	Patches	Annotations	Objects/patch	Median	Min–Max	P90	P95	P99
Trainval	1,859	178,238	$95.88\pm 89.32$	70.0	1–429	229.0	272.0	350.4
Test	433	45,341	$104.71\pm 94.31$	75.0	1–523	236.8	277.4	361.3

Appendix

Table 10: Statistics of the cropped Sorghum-Grain-Count splits. Objects/patch reports the mean and standard deviation across retained patches. P90, P95, and P99 denote percentiles of the patch-level object-count distribution.

Method	Tr. (h)	Init.	$\|Q\|$ ( $\mu\pm\sigma$ )	AP	AP 50	AP 75	AP t	AP s	AP m
D-FINE-L	1.5	Fixed	400	57.0	92.6	64.9	10.6	50.1	63.6
D-FINE-L	1.7	Fixed	1100	57.7	93.9	65.6	13.4	50.9	64.2
DEIM-L	1.6	Fixed	400	57.2	92.9	65.1	10.9	50.3	63.8
DEIM-L	1.8	Fixed	1100	58.0	94.0	66.3	14.3	51.3	64.3
Dome-DETR-L	3.3	Dynamic	$1103\pm 195$	57.6	94.0	65.4	13.1	50.8	64.1
BUTR-L	1.5	Dynamic	$378\pm 119$	58.0	93.8	66.1	14.5	51.5	64.1

Appendix

Table 11: Comparison on Sorghum-Grain-Count using the cropped trainval/test protocol. Query-count mean and standard deviation describe variation across evaluation patches, not repeated training runs. Tr. (h): training time. T4 deployment latency was not measured for this supplementary experiment.

Small Object Detection

Small object detection develops methods for finding and localizing objects that occupy few pixels, often amid clutter, blur, or low resolution.

See all 87 papers

PaQ-DETR: Learning Pattern and Quality-Aware Dynamic Queries for Object Detection
Mar 6, 2026Zhengjian Kang, Jun Zhuang, Kangtong Mo +3Object Detection Detection Transformer
New York University · Boise State University · University of Illinois at Urbana-Champaign +3
TinyFormer: Preserving Tiny Objects in YOLO-DETR Hybrid Real-time Detectors
May 24, 2026Jun-Wei Hsieh, Meng-Yu Kao, Ghufron Wahyu Kurniawan +1Hybrid CNN-Transformer Architectures Small Object Detection
College of Artificial Intelligence, National Yang Ming Chiao Tung University, Taiwan · 2Mitsubishi Electric Research Laboratories
Order Matters: Competition-Guided Query Ordering for RNN-Based Object Detection
Oct 4, 2026Shengjian Wu, Li Sun, Yu Shangguan +1Detection Transformer Object Detection
Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University · FinVolution Group · Key Laboratory of Advanced Theory and Application in Statistics and Data Science, East China Normal University

Mar 6, 2026J/K move · Enter open · S save

PaQ-DETR: Learning Pattern and Quality-Aware Dynamic Queries for Object Detection

Zhengjian Kang, Jun Zhuang, Kangtong Mo, Qi Chen+2

New York University · Boise State University · University of Illinois at Urbana-Champaign+3

Object Detection Detection Transformer

Open paper AskPDF arXiv

Detection Transformer (DETR) has redefined object detection by casting it as a set prediction task within an end-to-end framework. Despite its elegance, DETR and its variants still rely on fixed learnable queries and suffer from severe query utilization imbalance, which limits adaptability and leaves the model capacity underused. We propose PaQ-DETR (Pattern and Quality-Aware DETR), a unified framework that enhances both query adaptivity and supervision balance. It learns a compact set of shared latent patterns capturing global semantics and dynamically generates image-specific queries through content-conditioned weighting. In parallel, a quality-aware one-to-many assignment strategy adaptively selects positive samples based on localizatio-classification consistency, enriching supervision and promoting balanced query optimization. Experiments on COCO, CityScapes, and other benchmarks show consistent gains of 1.5%-4.2% mAP across DETR backbones, including ResNet and Swin-Transformer. Beyond accuracy improvement, our method provides interpretable insights into how dynamic patterns cluster semantically across object categories.

Abstract

Small Object Detection

Explore similar work

PaQ-DETR: Learning Pattern and Quality-Aware Dynamic Queries for Object Detection

TinyFormer: Preserving Tiny Objects in YOLO-DETR Hybrid Real-time Detectors

Order Matters: Competition-Guided Query Ordering for RNN-Based Object Detection