Order Matters: Competition-Guided Query Ordering for RNN-Based Object Detection

Shengjian Wu; Li Sun; Yu Shangguan; Qingli Li

cs.CVOct 4, 2026

Order Matters: Competition-Guided Query Ordering for RNN-Based Object Detection

Authors: Shengjian Wu, Li Sun, Yu Shangguan, Qingli Li

Organizations: Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University · FinVolution Group · Key Laboratory of Advanced Theory and Application in Statistics and Data Science, East China Normal University

Abstract

DETR-style detectors use one-to-one bipartite matching during training to assign object queries to ground-truth objects, enabling end-to-end set prediction without non-maximum suppression (NMS). However, without an explicit de-duplication procedure, multiple queries can still produce highly similar hypotheses for the same object, making training unstable and predictions less decisive. Inspired by the sequential ordering of NMS, we propose DETRNN, a plug-and-play module that turns unordered object queries into a competition-aware sequence for recurrent refinement. DETRNN builds an explicit confidence-and-similarity based order from prior predictions, then refines queries with an RNN along this order to model competition inside the decoder. This ordered recurrent refinement reduces redundant predictions, stabilizes optimization, and improves final detection accuracy. Experiments on multiple DETR-style detectors show consistent gains with comparable efficiency.

Figures & tables

Panel of Figure 1 — Figure 1 : A comparison of de-duplication mechanisms. Classical NMS de-duplicates by removing highly overlapping proposals with a fixed IoU threshold (a). DETR predicts end-to-end, but its order-agnostic decoder can still yield duplicated results (b). DETRNN internalizes de-duplication into the decoder: CGQO builds a competition-aware query order, and a Bi-RNN refines queries sequentially to softly suppress redundant predictions (dashed boxes) without discarding outputs (c).

Figure 2 : Overview of DETRNN. Given prior predictions from encoder proposals or preceding decoder outputs, Competition-Guided Query Ordering (CGQO) first constructs an explicit competition order among queries. Specifically, queries are ranked by confidence; the top-scoring query is selected as a pivot, and other queries are grouped and reordered by their similarity to the pivot (highly similar competitors are placed nearby). This pivot-and-grouping process is repeated until all queries are ordered, yielding a competition-aware sequence. Bidirectional RNN then replaces decoder self-attention and refines queries sequentially along this order, enabling learnable competition resolution inside the decoder. Unlike NMS, CGQO does not discard queries; it only provides ordering cues that the decoder can internalize for de-duplication.

Method	Backbone	Epochs	Params	FLOPs $\downarrow$	FPS $\uparrow$	AP	AP 50	AP 75	AP S	AP M	AP L
Def-DETR++ Zhu et al. (2020)	R50	12	47.5M	300G	21	47.6	65.6	51.7	30.0	51.0	61.5
Def-DETR++ Zhu et al. (2020)	R50	24	47.5M	300G	21	49.3	67.9	53.7	32.0	52.6	63.7
Def-DETRNN++ (ours)	R50	12	47.6M	301G	20	48.7 (+1.1)	67.0	53.2	32.3	52.4	63.2
Def-DETRNN++ (ours)	R50	24	47.6M	301G	20	50.1 (+0.8)	68.7	54.8	33.7	53.2	64.7
DINO Zhang et al. (2022)	R50	12	47.5M	300G	21	49.0	66.6	53.5	32.0	52.3	63.0
DINO Zhang et al. (2022)	R50	24	47.5M	300G	21	50.4	68.3	54.8	33.3	53.7	64.8

Table 1 : Results on COCO val2017 with ResNet-50 and 900 queries. Under unified

1\times

2\times

schedules, DETRNN consistently improves AP across baselines with comparable FLOPs / FPS.

Method	Backbone	Epochs	Queries	AP	AP 50	AP 75	AP S	AP M	AP L
Def-DETR++ Zhu et al. (2020)	Swin-L	12	300	54.5	74.0	59.3	37.0	58.6	71.0
Def-DETRNN++ (ours)	Swin-L	12	300	55.6 (+1.1)	75.1	60.7	38.5	59.8	71.4
LoRA-DETR Zhang et al. (2026)	Swin-L	12	900	58.2	76.2	63.4	41.8	63.0	74.4
LoRA-DETRNN (ours)	Swin-L	12	900	58.5 (+0.3)	76.5	64.0	42.5	63.1	74.9

Table 2 : Results with Swin-L backbone.

Def-DETR++ (1-decoder)
	AP	+NMS	$\Delta$ AP $\downarrow$
Base	40.6	43.0	2.4
+Ours	46.3 (+5.7)	46.6	0.3 (-2.1)

Table 3 : 1 -decoder de-duplication stress test (900 queries,

1\times

schedule). We report AP and +NMS AP (AP after NMS@0.8);

\Delta

AP denotes the NMS gain, where a larger value indicates more residual duplicates. DETRNN improves AP while sharply reducing

\Delta

AP, suggesting stronger decoder-internal de-duplication.

Panel of Figure 3 — Figure 3 : Training dynamics and competition indicators on COCO val ( 6 -decoder). (a) Validation AP across epochs for Def-DETR++ and ours. (b) AP gain from evaluation-time NMS@0.8; larger values indicate more residual duplicates. (c) Top1–Top2 score gap, where Top1/Top2 are the best and second-best predictions matched to the same GT; a larger gap indicates clearer separation between the correct detection and its closest competitor.

DINO-
Method	BR	CGQO	AP	AP 50	AP 75
1			49.0	66.6	53.5
2	✓		49.2 (+0.2)	66.9	53.8
3		✓	49.0	66.6	53.5
4	✓	✓	50.0 (+1.0)	68.1	54.0

Table 4 : Ablations on baselines MS- and DINO-. BR denotes replacing self-attention with Bi-RNN, and CGQO denotes Competition-Guided Query Ordering. Using BR or CGQO alone yields negligible gains, while combining them leads to clear improvements, confirming that Bi-RNN can effectively model inter-query dependencies under an explicit competition order.

Detector	Unordered	Random	Conf-rank	Group-IoU	CGQO
DINO-	49.2	48.8 ( -0.4 )	49.3 ( +0.1 )	49.8 ( +0.6 )	50.0 ( +0.8 )
MS-	50.0	49.8 ( -0.2 )	50.0	50.2 ( +0.2 )	50.5 ( +0.5 )

Table 5 : Ordering variants for Bi-RNN refinement measured by AP . Unordered: Method 2 in Tab. 4 . Conf-rank: confidence-only ranking. Group-IoU: pivot-and-group with IoU-only similarity. CGQO: Method 4 in Tab. 4 . Parentheses report AP changes relative to the Unordered variant.

Panel of Figure 4 — Figure 4 : RNN variants and efficiency scaling on Def-DETRNN++ detector ( $1\times$ schedule). (a) Different RNN-style modules yield stable gains over baseline. (b) The Mamba variant has controllable FPS overhead, with the gap to Attn narrowing as the number of queries increases.

Detector	w/o NMS	+NMS@0.7	+NMS@0.8	+NMS@0.9	+Ours
Def-DETR++	46.5	46.2 ( -0.3 )	46.7 ( +0.2 )	46.6 ( +0.1 )	47.6 ( +1.1 )
MS-DETR	48.4	48.3 ( -0.1 )	48.6 ( +0.2 )	48.6 ( +0.2 )	49.2 ( +0.8 )

Table 6 : Threshold sensitivity on CrowdHuman dataset. Each entry reports AP under a specific setting. NMS is sensitive to the IoU threshold: a lower

\tau

(0.7) can reduce AP, while higher

\tau

brings only marginal gains. In contrast, our method consistently improves accuracy without threshold tuning.

Detector	w/o NMS	+NMS@0.7	+NMS@0.8	+NMS@0.9	+Ours
Def-DETR++	47.6	47.6	47.7 ( +0.1 )	47.7 ( +0.1 )	48.7 ( +1.1 )
MS-DETR	50.0	50.1 ( +0.1 )	50.2 ( +0.2 )	50.2 ( +0.2 )	50.5 ( +0.5 )

Table 7 : COCO consistency check (AP). NMS brings only marginal gains (+0.1–0.2) for the baselines.

EASE-DETR
	FLOPs $\downarrow$	FPS $\uparrow$	AP
Base	301G	18	49.6
+Ours	301G	19	49.9 (+0.3)

Table 8 : Replacement on competition-aware decoders. DETRNN improves pairwise-interaction methods (EASE-/Relation-) and a query-selection method (DDQ) with higher throughput.

Def-DETRNN++
Stage	FPS $\uparrow$	AP
All	18	48.4
Last 2	20	48.7
Last 1	21	48.4

Table 9 : Stage placement study across different baselines. “Stage” indicates the decoder stages where our module is inserted.

Panel of Figure 5 — Figure 5 : Mechanistic analysis of CGQO grouping. (a) Pivot-match evolution during the first training epoch. (b) Composition of the Hungarian-matched query’s CGQO group. (c) Summary statistics of object-centered grouping.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6 : Structures of the three representative RNN variants used in our Bi-RNN module. TTT is also evaluated but omitted from the architecture illustration for space.

Panel of Figure 7 — Figure 7 : Sensitivity of CGQO hyperparameters on COCO val (DINO-RNN). (a) Performance varies little across $\alpha\in[0,0.4]$ . (b) Varying the saturation constant $K$ around the default $K{=}18$ also leads to minor changes.

Panel of Figure 8 — Figure 8 : Robustness of CGQO thresholding on COCO val . (a,b) Sweeping $\tau_{\min}$ / $\tau_{\max}$ (Eq. 3 ) changes AP only slightly. (c) The density-adaptive rule performs best compared to fixed thresholds.

Detector	Variant	AP	AP 50	AP 75
Def-DETR++	Attn baseline	47.6	65.6	51.7
	w/o IOF-PE	48.6 (+1.0)	66.9	53.1
	Full	48.7 (+1.1)	67.0	53.2
MS-DETR	Attn baseline	50.0	67.3	54.4
	w/o IOF-PE	50.3 (+0.3)	67.7	55.0
	Full	50.5 (+0.5)	68.2	55.4

Appendix

Table 10 : Effect of IOF positional encoding in the default Mamba instantiation on COCO val (R50, 1

\times

Variant	CGQO + Bi-RNN	Selected stage	IOF-PE	AP
Baseline (self-attn)	$\times$	$\times$	$\times$	47.6
Ordered recurrent refinement	✓	$\times$	$\times$	48.2 (+0.6)
+ IOF-PE	✓	$\times$	✓	48.4 (+0.8)
+ Selected stage	✓	✓	$\times$	48.6 (+1.0)
Full DETRNN	✓	✓	✓	48.7 (+1.1)

Appendix

Table 11 : Component analysis of DETRNN on Def-DETRNN++/R50/

1\times

Module	$N{=}300$		$N{=}900$		$N{=}1500$
Module	Time (ms) $\downarrow$	Mem (MB) $\downarrow$	Time (ms) $\downarrow$	Mem (MB) $\downarrow$	Time (ms) $\downarrow$	Mem (MB) $\downarrow$
Self-attn	0.2	6.7	0.2	53.0	0.4	143.2
Mamba	0.4	3.7	0.4	11.3	0.4	19.0
CGQO	0.3	0.5	0.5	3.8	0.7	9.7
CGQO+Mamba	0.7 (+0.5)	4.2 (-2.5)	0.9 (+0.7)	15.1 (-37.9)	1.1 (+0.7)	28.7 (-114.5)

Appendix

Table 12 : Profiling of query-interaction modules under different query counts

N

. Runtime is reported in ms/image and rounded to 0.1 ms; extra peak memory is reported in MB. CGQO+Mamba adds small runtime overhead but uses much less memory as

N

grows, and the overhead is minor relative to the full detector cost (

\sim

50 ms,

\sim

700 MB at 1024

\times

1333).

Module	$N{=}2000$		$N{=}4000$		$N{=}5000$
Module	Time (ms) $\downarrow$	Mem (MB) $\downarrow$	Time (ms) $\downarrow$	Mem (MB) $\downarrow$	Time (ms) $\downarrow$	Mem (MB) $\downarrow$
Self-attn	0.8	252.0	3.2	992.2	5.1	1545.4
Mamba	0.4	25.0	0.4	49.9	0.4	62.7
CGQO	0.9	16.9	1.9	65.6	2.3	101.4
CGQO+Mamba	1.3 (+0.5)	41.9 (-210.1)	2.3 (-0.9)	115.5 (-876.7)	2.7 (-2.4)	164.1 (-1381.3)

Appendix

Table 13 : Module-level profiling under larger query counts

N

. Runtime is reported in ms/image and rounded to 0.1 ms; extra peak memory is reported in MB. CGQO+Mamba greatly reduces memory compared with self-attention and has lower isolated module runtime from

N{=}4000

onward.

Method	Backbone	Epochs	Queries	AP	AP 50	AP 75	AP S	AP M	AP L
Def-DETR++ [ 7 ]	R50	12	900	47.6	65.6	51.7	30.0	51.0	61.5
DINO [ 8 ]	R50	12	900	49.0	66.6	53.5	32.0	52.3	63.0
Salience-DETR [ 27 ]	R50	12	900	49.2	67.1	53.8	32.7	53.0	63.1
DAC-DETR [ 2 ]	R50	12	900	50.0	67.6	54.7	32.9	53.1	64.2
MS-DETR [ 9 ]	R50	12	900	50.0	67.3	54.4	31.6	53.2	64.0
Rank-DETR [ 23 ]	R50	12	900	50.4	67.9	55.2	33.6	53.8	64.2

Appendix

Table 14 : Full-recipe comparison with recent DETR-style SOTA on COCO val under comparable ResNet-50 / Swin-L, 1

\times

, and 900-query settings when available. The DETRNN entries use our strongest LoRA-DETRNN full recipes. This is a leaderboard-style context rather than a controlled module comparison, since methods use different recipes.

Baseline	Metric	Stage 4	Stage 5	Stage 6
Def-DETR++	AP	47.1	47.4	47.6
Def-DETR++	AR100	67.4	67.7	67.6
MS-DETR	AP	49.1	49.5	50.0
MS-DETR	AR100	72.7	73.0	72.6

Appendix

Table 15 : Stage-wise AP and AR100 of the original Def-DETR++ and MS-DETR baselines on COCO val2017 . AR100 saturates earlier than AP, indicating that later decoder stages mainly refine already covered objects.

Detector	w/o NMS	+NMS@0.7	+NMS@0.8	+NMS@0.9	+Ours
Def-DETR++	46.2	46.3 ( +0.1 )	46.3 ( +0.1 )	46.3 ( +0.1 )	47.2 ( +1.0 )

Appendix

Table 16 : WiderPerson consistency check (AP). NMS brings only marginal gains (+0.1) for the baseline.

Panel of Figure 9 — Figure 9 : (a), (b), and (c) show the synthetic data example, convergence comparison on the synthetic val dataset, and experiment results comparing Def-DETR* [ 7 ] and DETRNN with and without NMS, respectively. The dataset and model are simplified for rapid validation.

Compared queries	Mean	25th percentile	75th percentile
tail-1 vs. tail	0.734	0.610	0.906
tail vs. next pivot	0.330	0.133	0.486

Appendix

Table 17 : Cosine similarity around CGQO group boundaries.

Detector	Baseline	Seed 1	Seed 2	Seed 3	Mean	Std.
Def-DETRNN++	47.6	48.7	48.9	48.6	48.73	0.15
DINO-RNN	49.0	50.0	50.0	49.8	49.93	0.12
MS-DETRNN	50.0	50.5	50.3	50.5	50.43	0.12
LoRA-DETRNN	52.5	53.0	53.0	52.9	52.97	0.06

Appendix

Table 18 : Multi-seed results on COCO. Mean and standard deviation are computed over three runs.

Method	Params.	AP	AP 50	AP 75
Self-attention baseline	47.5M	49.0	66.6	53.5
CGQO + Uni-RNN	47.6M	49.5	67.7	53.9
CGQO + rank-bias attn	47.7M	49.7	67.8	54.1
CGQO + causal attn	47.5M	49.5	67.6	54.0
CGQO + 1D Conv	47.7M	49.6	67.6	54.1
CGQO + Bi-RNN	47.6M	50.0	68.1	54.0

Appendix

Table 19 : Comparison of order-aware interaction modules with CGQO on DINO/R50/

1\times

Figure 10 : Visual comparison of predictions for Def-DETR++ and Def-DETRNN++, with and without NMS. The results highlight Def-DETRNN++’s minimal redundancy and effective end-to-end predictions, while Def-DETR++ shows significant redundancy without NMS.

Explore similar work

Enhanced Knowledge Distillation for Detection Transformer via Teacher Prediction Refinement
Sep 17, 2026Yitong Xing, Yuhao Cheng, Yanping Li +1Rt-Detr Detection Transformer
School of Computer Science, Shanghai Jiao Tong University, Shanghai 200240, China
AnyDepth-DETR/-YOLO: Any-depth object detection with a single network
May 10, 2026Woochul Kang, Hyungseop Lee, Jiho LeeYolo26 Unsupervised Detection
Incheon Nat’l Univ.
MDS-DETR: DETR with Masked Duplicate Suppressor
May 22, 2026Chanho Lee, Seunghee Koh, Yunho Jeon +1Rt-Detr Detection Transformer
Samsung Research, Seoul 06765, South Korea · Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, South Korea · Department of Artificial Intelligence Software, Hanbat National University, Daejeon 34158, South Korea

Sep 17, 2026, cs.CVJ/K move · Enter open · S save

Enhanced Knowledge Distillation for Detection Transformer via Teacher Prediction Refinement

Yitong Xing, Yuhao Cheng, Yanping Li, Yichao Yan

School of Computer Science, Shanghai Jiao Tong University, Shanghai 200240, China

Rt-Detr Detection Transformer Object Detection Knowledge Distillation Teacher Suppression

Open paper AskPDF arXiv

Detection Transformers (DETRs) achieve strong performance in object detection but remain challenging to deploy on edge devices due to their high computational cost. Existing DETR distillation methods mainly focus on aligning distillation points, while largely overlooking the quality of the teacher's supervision itself. We observe that due to stage-wise non-monotonic prediction behavior in DETRs, well-localized or correctly classified predictions from earlier stages may degrade in later ones, and some negative predictions become increasingly overconfident. As a result, relying solely on the current stage's predictions yields inaccurate and inconsistent supervision. To address this issue, we propose Teacher Prediction Refinement Distillation (TPRD), a plug-and-play module that refines teacher predictions before distillation by exploiting stage-wise prediction information. TPRD improves supervision quality through Positive Prediction Correction (PPC), which corrects degraded positive predictions by restoring more accurate ones from earlier stages, ensuring reliable localization and classification signals, and Negative Prediction Suppression (NPS) suppresses the influence of overconfident negatives, preventing them from providing misleading supervision to the student. To preserve informative dark knowledge, we further introduce Maximum Dark Knowledge Preservation (MDKP), which selectively refines target-class logits while retaining non-target relations. Extensive experiments on MS COCO and PASCAL VOC demonstrate the effectiveness and robustness of the proposed method. Our code is available at https://github.com/xingyitong1/TPRD.

Abstract

Explore similar work

Enhanced Knowledge Distillation for Detection Transformer via Teacher Prediction Refinement

AnyDepth-DETR/-YOLO: Any-depth object detection with a single network

MDS-DETR: DETR with Masked Duplicate Suppressor