Don't let the information slip away

Taozhe Li; Guansu Wang; Bo Yu; Yiming Liu; Wei Sun

cs.CVFeb 26, 2026

Don't let the information slip away

Authors: Taozhe Li, Guansu Wang, Bo Yu, Yiming Liu, Wei Sun

Organizations: University of Oklahoma · University of Melbourne · University of Utah

Abstract

Real-time object detection has advanced rapidly in recent years. The YOLO series of detectors is among the most well-known CNN-based object detection models and cannot be overlooked. The latest version, YOLOv26, was recently released, while YOLOv12 achieved state-of-the-art (SOTA) performance with 55.2 mAP on the COCO val2017 dataset. Meanwhile, transformer-based object detection models, also known as DEtection TRansformer (DETR), have demonstrated impressive performance. RT-DETR is an outstanding model that outperformed the YOLO series in both speed and accuracy when it was released. Its successor, RT-DETRv2, achieved 53.4 mAP on the COCO val2017 dataset. However, despite their remarkable performance, all these models let information to slip away. They primarily focus on the features of foreground objects while neglecting the contextual information provided by the background. We believe that background information can significantly aid object detection tasks. For example, cars are more likely to appear on roads rather than in offices, while wild animals are more likely to be found in forests or remote areas rather than on busy streets. To address this gap, we propose an object detection model called Association DETR, which achieves state-of-the-art results compared to other object detection models on the COCO val2017 dataset.

Figures & tables

Figure 1: Association DETR Overview. The input image is first fed into the backbone network. The shallowest image feature is denoted as $S_{1}$ , the second shallowest as $S_{2}$ , and the deepest as $S_{3}$ . we feed the shallowest feature $S_{1}$ into the Background Attention Module, which is designed to capture background information. Also in Figure 4 . We visualize some sample for better understanding the function of the Background Attention Module. And The Features $S_{1}$ , $S_{2}$ , and $S_{3}$ are fed into the Hybrid Encoder for both intra-feature and inter-feature enhancement. The output $F_{b}$ of the Background Attention Module is then fed into the Association Module, which performs feature enhancement related to background information. After that, the output of the Association Module, $F_{a}$ performs an addition operation with $F_{b}$ . Additionally, Feature $F_{b}$ is added to $F_{3}$ as $F_{\hat{3}}$ , the $F_{3}$ refers to the output of the Hybrid Encoder corresponding to the input $S_{3}$ . Finally, the features $F_{1}$ , $F_{2}$ , and $F_{\hat{3}}$ undergo query selection and are passed into the Decoder and Detection Head to predict object bounding boxes and classes.

Model	Backbone	#Epochs	#Params (M)	#FPS bs=1	AP val	AP ${}^{val}_{50}$
YOLOv10-M	-	300	15.4	210	51.1	68.1
YOLOv10-L	-	300	24.4	137	53.2	70.1
YOLOv10-X	-	300	29.5	93	54.4	71.6
YOLOv11-M	-	300	20.1	212	51.5	67.9
YOLOv11-L	-	300	25.3	161	53.4	70.1
YOLOv12-M	-	300	20.2	206	52.5	69.5

Table 1: Comparison with SOTA . All models are trained on the COCO Dataset. And the FPS is reported on T4 GPU with TensorRT. For Evaluation, all input size are fixed on 640

\times

640, Our Association DETR-R34 achieves 54.60 in AP val , 71.6 in AP

{}^{val}_{50}

and 153 FPS.Meanwhile, our Association DETR-R50 achieves 55.7 in AP val , 74.0 in AP

{}^{val}_{50}

and 104 FPS. Both of them outperform other YOLO detectors and DETR models with similar scale.

Model	Backbone	#Epochs	#Params (M)	#FPS bs=1	AP val	AP ${}^{val}_{50}$
RT-DETR-R34 (base)	R34	72	31	161	48.9	66.8
AE + DETR-R34 (Ours)	R34	72	34.1	153	54.6( $\uparrow$ 5.7)	71.6 ( $\uparrow$ 4.8 )
RT-DETR-R50 (base)	R50	72	42	108	53.1	71.3
AE + DETR-R50 (Ours)	R50	72	45.1	104	55.7 ( $\uparrow$ 2.6 )	74.0 ( $\uparrow$ 2.7 )
RT-DETRv2-M (base)	R34	72	31	161	49.9	67.5
AE + RT-DETRv2-M	R34	72	34.1	153	52.4 ( $\uparrow$ 2.5 )	70.2 ( $\uparrow$ 2.7 )

Table 2: Effectiveness . We integrate our Association Encoder(AE) into various models, including RT-DETRv34, RT-DETR-R50, RT-DETRv2-R50, DETRv2-R34, DETR and Deformable DETR.Specifically, for the RT-DETR-R34 model, it increases AP val by 5.7 and AP

{}^{val}_{50}

by 4.8, while reducing FPS by less than 5.7%. Similarly, it improves AP val by 2.6 and AP

{}^{val}_{50}

by 2.7 for RT-DETR-R50. Moreover, 2.5 in AP val and 2.7 in AP

{}^{val}_{50}

for RT-DETRv2-M; 0.3 in AP val and 0.6 in AP

{}^{val}_{50}

for RT-DETRv2-L with R50 as backbone; DETR-R50, it increases 2.7 in AP val and 3.6 in AP

{}^{val}_{50}

. We need to noticed that it even outperforms than DETR-R101(base). Moreover, 2.6 in AP val and 3.2 in AP

{}^{val}_{50}

for Deformable DETR. All these models outperform their respective baselines after integrating our Association Encoder (AE). Notably, the Association-DETR-R34 and Association-DETR-R50 models achieve state-of-the-art performance compared to other models.

Figure 2: Background Attention Module & Single RFCBAMConv Block.On the left side is the structure of the Background Attention Module, and on the right side are the details of a single RFCBAMConv Block, which is located within the Background Association Module.

Figure 3: Association Module. We incorporate ConvFFN and Window Attention for trade off between performance and speed.

Panel of Figure 4 — Figure 4: Visualization of Background Attention. BG refers to background. The figure is plotted by pytorch-grad-cam. The BAM (Background Attention Module) effectively captures background information across various scenarios. In the first image featuring bears, it successfully identifies the grass behind them. In the second image, it accurately detects grass and even fencing, despite fencing is not included in our training categories. Furthermore, in the third and fourth images, the module correctly identifies the road, sky, and grass.

Model	Backbone	#Epochs	#Params (M)	#FPS bs=1	AP val	AP ${}^{val}_{50}$
RT-DETR-R34 (base)	R34	72	31	161	48.9	66.8
AE + DETR-R34 (Ours)	R34	72	34.1	153	54.6( $\uparrow$ 5.7)	71.6 ( $\uparrow$ 4.8 )
RT-DETR-R50 (base)	R50	72	42	108	53.1	71.3
AE + DETR-R50 (Ours)	R50	72	45.1	104	55.7 ( $\uparrow$ 2.6 )	74.0 ( $\uparrow$ 2.7 )
RT-DETRv2-M (base)	R34	72	31	161	49.9	67.5
AE + RT-DETRv2-M	R34	72	34.1	153	52.4 ( $\uparrow$ 2.5 )	70.2 ( $\uparrow$ 2.7 )

Table 3: Effectiveness . We integrate our Association Encoder(AE) into various models, including RT-DETRv34, RT-DETR-R50, RT-DETRv2-R50, DETRv2-R34, DETR and Deformable DETR.Specifically, for the RT-DETR-R34 model, it increases AP val by 5.7 and AP

{}^{val}_{50}

by 4.8, while reducing FPS by less than 5.7%. Similarly, it improves AP val by 2.6 and AP

{}^{val}_{50}

by 2.7 for RT-DETR-R50. Moreover, 2.5 in AP val and 2.7 in AP

{}^{val}_{50}

for RT-DETRv2-M; 0.3 in AP val and 0.6 in AP

{}^{val}_{50}

for RT-DETRv2-L with R50 as backbone; DETR-R50, it increases 2.7 in AP val and 3.6 in AP

{}^{val}_{50}

. We need to noticed that it even outperforms than DETR-R101(base). Moreover, 2.6 in AP val and 3.2 in AP

{}^{val}_{50}

Model	BAM	AM	EL	#Params (M)	AP val	AP ${}^{val}_{50}$
RT-DETR-R34				31	48.9	66.8
RT-DETR-R34	$\checkmark$			33.4	52.1( $\uparrow$ 3.2 )	70.4( $\uparrow$ 3.6 )
RT-DETR-R34		$\checkmark$		31.7	50.2 ( $\uparrow$ 1.3 )	68.7( $\uparrow$ 1.9 )
RT-DETR-R34			$\checkmark$	32.1	49.9( $\uparrow$ 1.0)	68.6( $\uparrow$ 1.8)
RT-DETR-R34	$\checkmark$		$\checkmark$	34.5	53.5( $\uparrow$ 4.6)	70.6 ( $\uparrow$ 3.8)
RT-DETR-R34	$\checkmark$	$\checkmark$		34.1	54.6 ( $\uparrow$ 5.7 )	71.6 ( $\uparrow$ 4.8 )

Table 4: Ablation . All models are trained on the COCO 2017 dataset. For training, the number of epochs is fixed at 72 to ensure a fair comparison. For evaluation, all input sizes are fixed at 640×640. BAM, AM, and EL represent the Background Attention Module, Association Module, and Basic ViT Encoder Layer, respectively.

Item	Value
optimizer	AdamW
base learning rate	1e-4
learning rate of backbone	1e-5
freezing BN	True
linear warm-up start factor	0.001
linear warm-up steps	2000

Table 5: hyperparameters of Association DETR. Most of hyper-parameters are as same as our baseline RT-DETR

Object Detection

Object detection locates and classifies objects in images or video, balancing localization accuracy, robustness, and inference speed.

See all 624 papers

Jun 2, 2026

Ultralytics YOLO26: Unified Real-Time End-to-End Vision Models

Real-time vision demands models that are accurate, efficient, and simple to deploy across diverse hardware. The YOLO family has become widely deployed for this reason, yet most YOLO detectors still rely on non-maximum suppression at inference, carry heavy detection heads due to Distribution Focal Loss, require long training schedules, and can leave the smallest objects without positive label assignments. We present Ultralytics YOLO26, a unified real-time vision model family that addresses these limitations through coordinated architecture and training advances. YOLO26 uses a dual-head design for native NMS-free end-to-end inference and removes DFL entirely, yielding a lighter head with unconstrained regression range. Its training pipeline combines MuSGD, a hybrid Muon-SGD optimizer adapted from large language model training; Progressive Loss, which shifts supervision toward the inference-time head; and STAL, a label assignment strategy that guarantees positive coverage for small objects. Beyond detection, YOLO26 introduces task-specific head and loss designs for instance segmentation, pose estimation, and oriented detection, producing consistent gains across tasks and scales. The family spans five scales (n/s/m/l/x) and supports detection, instance segmentation, pose estimation, classification, and oriented detection in a single pipeline, with an open-vocabulary extension, YOLOE-26, for text-, visual-, and prompt-free inference. Across all scales, YOLO26 achieves 40.9-57.5 mAP on COCO at 1.7-11.8 ms T4 TensorRT latency, advancing the accuracy-latency Pareto front over prior real-time detectors, while YOLOE-26x reaches 40.6 AP on LVIS minival under text prompting. Code and models are available at https://github.com/ultralytics/ultralytics.

Glenn Jocher, Jing Qiu, Mengyu Liu +3

Ultralytics

Open-Vocabulary Object Detection Instance Segmentation Efficient Object Detection Object Detection Real-Time Object Detection

Ask PDF

Oct 29, 2025

RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models

Real-time object detection has achieved substantial progress through meticulously designed architectures and optimization strategies. However, the pursuit of high-speed inference via lightweight network designs often leads to degraded feature representation, which hinders further performance improvements and practical on-device deployment. In this paper, we propose a cost-effective and highly adaptable distillation framework that harnesses the rapidly evolving capabilities of Vision Foundation Models (VFMs) to enhance lightweight object detectors. Given the significant architectural and learning objective disparities between VFMs and resource-constrained detectors, achieving stable and task-aligned semantic transfer is challenging. To address this, on one hand, we introduce a Deep Semantic Injector (DSI) module that facilitates the integration of high-level representations from VFMs into the deep layers of the detector. On the other hand, we devise a Gradient-guided Adaptive Modulation (GAM) strategy, which dynamically adjusts the intensity of semantic transfer based on gradient norm ratios. Without increasing deployment and inference overhead, our approach painlessly delivers striking and consistent performance gains across diverse DETR-based models, underscoring its practical utility for real-time detection. Our new model family, RT-DETRv4, achieves state-of-the-art results on COCO, attaining AP scores of

49.8/53.7/55.4/57.0

at corresponding speeds of

273/169/124/78

FPS. Code is publicly available at https://github.com/RT-DETRs/RT-DETRv4.

Zijun Liao, Yian Zhao, Xin Shan +5

School of Electronic and Computer Engineering, Peking University, Shenzhen, China · Department of Automation and BNRist, Tsinghua University, Beijing, China

Real-Time Object Detection Efficient Object Detection Detection Transformer Knowledge Distillation

Ask PDF

Feb 24, 2026

Le-DETR: Revisiting Real-Time Detection Transformer with Efficient Encoder Design

Real-time object detection is crucial for real-world applications as it requires high accuracy with low latency. While Detection Transformers (DETR) have demonstrated significant performance improvements, current real-time DETR models are challenging to reproduce from scratch due to excessive pre-training overheads on the backbone, constraining research advancements by hindering the exploration of novel backbone architectures. In this paper, we want to show that by using general good design, it is possible to have high performance with low pre-training cost. After a thorough study of the backbone architecture, we propose EfficientNAT at various scales, which incorporates modern efficient convolution and local attention mechanisms. Moreover, we re-design the hybrid encoder with local attention, significantly enhancing both performance and inference speed. Based on these advancements, we present Le-DETR (Low-cost and Efficient DEtection TRansformer), which achieves a new SOTA in real-time detection using only ImageNet1K and COCO2017 training datasets, saving about 80% images in pre-training stage compared with previous methods. We demonstrate that with well-designed, real-time DETR models can achieve strong performance without the need for complex and computationally expensive pretraining. Extensive experiments show that Le-DETR-M/L/X achieves 52.9/54.3/55.1 mAP on COCO Val2017 with 4.45/5.01/6.68 ms on an RTX4090. It surpasses YOLOv12-L/X by +0.6/-0.1 mAP while achieving similar speed and +20% speedup. Compared with DEIM-D-FINE, Le-DETR-M achieves +0.2 mAP with slightly faster inference, and surpasses DEIM-D-FINE-L by +0.4 mAP with only 0.4 ms additional latency. Code and weights will be open-sourced.

Jiannan Huang, Aditya Kane, Fengzhe Zhou +2

Beijing Jiaotong University

Real-Time Object Detection Efficient Object Detection Detection Transformer Efficient ViTs Hybrid CNN-Transformer Architectures

Ask PDF

Abstract

Object Detection

Explore similar work

Ultralytics YOLO26: Unified Real-Time End-to-End Vision Models

RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models

Le-DETR: Revisiting Real-Time Detection Transformer with Efficient Encoder Design