Le-DETR: Revisiting Real-Time Detection Transformer with Efficient Encoder Design

Jiannan Huang; Aditya Kane; Fengzhe Zhou; Yunchao Wei; Humphrey Shi

cs.CVFeb 24, 2026

Le-DETR: Revisiting Real-Time Detection Transformer with Efficient Encoder Design

Authors: Jiannan Huang, Aditya Kane, Fengzhe Zhou, Yunchao Wei, Humphrey Shi

Organizations: Beijing Jiaotong University

Abstract

Real-time object detection is crucial for real-world applications as it requires high accuracy with low latency. While Detection Transformers (DETR) have demonstrated significant performance improvements, current real-time DETR models are challenging to reproduce from scratch due to excessive pre-training overheads on the backbone, constraining research advancements by hindering the exploration of novel backbone architectures. In this paper, we want to show that by using general good design, it is possible to have high performance with low pre-training cost. After a thorough study of the backbone architecture, we propose EfficientNAT at various scales, which incorporates modern efficient convolution and local attention mechanisms. Moreover, we re-design the hybrid encoder with local attention, significantly enhancing both performance and inference speed. Based on these advancements, we present Le-DETR (Low-cost and Efficient DEtection TRansformer), which achieves a new SOTA in real-time detection using only ImageNet1K and COCO2017 training datasets, saving about 80% images in pre-training stage compared with previous methods. We demonstrate that with well-designed, real-time DETR models can achieve strong performance without the need for complex and computationally expensive pretraining. Extensive experiments show that Le-DETR-M/L/X achieves 52.9/54.3/55.1 mAP on COCO Val2017 with 4.45/5.01/6.68 ms on an RTX4090. It surpasses YOLOv12-L/X by +0.6/-0.1 mAP while achieving similar speed and +20% speedup. Compared with DEIM-D-FINE, Le-DETR-M achieves +0.2 mAP with slightly faster inference, and surpasses DEIM-D-FINE-L by +0.4 mAP with only 0.4 ms additional latency. Code and weights will be open-sourced.

Figures & tables

Figure 1: The performance of each model. The results show that our model outperforms the existing models and achieves a new SOTA for real-time detection models. These results are tested using vanilla PyTorch profiler on 1

\times

RTX4090.

Figure 2: Left: Comparison of Le-DETR and Real-Time DETR series in training overheads, While saving lots of training overheads offered by efficient encoder design, our model outperforms the previous SOTA model, D-FINE and DEIM-D-FINE. Right: Unlike previous works such as DEIM and D-FINE mainly focus on decoder architecture and training objects, we focus on efficient encoder design.

Model	KD & ATI	Variant	Pretraining Image Nums	Latency(ms)	AP val
RT-DETRv2-L	✓	✓	5M	5.46	53.4
	✗	✓	1M	5.46	51.6 ( $\downarrow$ 1.8 )
	✗	✗	1M	4.91	51.6 ( $\downarrow$ 1.8 )
Le-DETR-L	✗	✗	1M	5.01	54.3

Table 1: This table presents the performance of the RT-DETRv2 models without the use of knowledge distillation (KD) techniques and additional training images (ATI) for backbone pretraining. Variant means whether to use ResNet50-D [ 16 ] . When compared to RT-DETRv2-L, our model achieves a +0.9 mAP improvement. Under identical pre-training settings, the performance gap further widens, with our model surpassing RT-DETRv2-L by +2.7 mAP.

Figure 3: The proposed Le-DETR is structured in three distinct stages: backbone, encoder, and decoder. Each first block of stages in this backbone serves as the downsampling block. In this figure, we also illustrate the encoder, which incorporates both the NAIFI and Feature Fusion components.

Figure 4: Overview of EfficientNATBlock and Neighborhood Attention-based Improved Feature Inference(NAIFI).

YOLO Series Models
Models	Params (M)	GFLOPs	Latency(ms)	AP val	AP ${}^{val}_{50}$	AP ${}^{val}_{75}$	AP ${}^{val}_{S}$	AP ${}^{val}_{M}$	AP ${}^{val}_{L}$
YOLOv5-M [ 36 ]	25.1	64.2	3.57	49.1	66.0	53.8	31.2	54.2	65.4
YOLOv5-L [ 36 ]	46.5	109.1	5.57	49.0	67.3	-	-	-	-
YOLOv5-X [ 36 ]	86.7	205.7	8.55	50.7	68.9	-	-	-	-
YOLOv8-M [ 37 ]	25.9	78.9	4.06	50.3	67.3	54.8	32.3	55.9	66.5
YOLOv8-L [ 37 ]	43.7	165.2	6.52	52.9	69.8	57.7	35.5	58.5	69.8

Table 2: Comparison of our proposed Le-DETR and previous SOTA object detection models on COCO Val 2017. Experiments show that our model Le-DETR-X outperforms the previous SOTA YOLOs and DEIM.

Models	Params (M)	GFLOPs	Latency(ms)	AP val	AP ${}^{val}_{50}$	AP ${}^{val}_{75}$	AP ${}^{val}_{S}$	AP ${}^{val}_{M}$	AP ${}^{val}_{L}$
Le-DETR w/ ResNet50_vd_ssld	43.8	128.3	5.80	53.6	71.4	58.0	35.5	58.3	70.9
Le-DETR w/ EfficientViT	64.7	135.7	6.60	53.8	71.5	58.6	36.8	58.1	71.3
Le-DETR w/o NAIFI	41.5	124.3	5.18	54.1	71.2	58.9	36.3	58.6	70.4
Le-DETR (Ours Full Method)	41.5	124.3	5.01	54.3	71.7	58.9	37.1	58.7	71.4

Table 3: Ablation experiment of components in our proposed Le-DETR. These experiments show the effectiveness of each module by offering both model performance improvement or speedup model inference while keeping the same performance.

Model Name	Params(M)	GFLOPs	Latency(ms)	AP val	AP ${}^{val}_{50}$
Le-DETR-M 4	31.4	114.1	4.19	52.5	69.3
Le-DETR-M 5	32.7	115.0	4.45	52.9	70.0
Le-DETR-M 6	34.0	115.6	4.73	52.9	70.0
Le-DETR-L 4	38.8	122.6	4.53	53.7	70.9
Le-DETR-L 5	40.1	123.5	4.79	54.2	71.6
Le-DETR-L 6	41.5	124.3	5.01	54.3	71.7

Table 4: Results of the ablation experiments using different number of layers in the model decoder, we use six decoder layers for training, with corner labels corresponding to different inference layers.

Model Name		Block Number	Top1 Acc	Top5 Acc	Throughput
L	P ${}_{\mathrm{A}}\text{-}1$ ✓	(1, 1, 4, 4)	81.994	95.432	1798.75
	P ${}_{\mathrm{A}}\text{-}2$	(1, 1, 6, 6)	82.135	95.302	1607.68
	P B	(1, 1, 4, 8)	82.046	94.972	1524.13
	P C	(1, 1, 10, 2)	81.946	95.494	1278.94
X	P ${}_{\mathrm{A}}\text{-}1$	(2, 2, 8, 8)	82.300	95.224	1247.82
	P ${}_{\mathrm{A}}\text{-}2$	(2, 2, 10, 10)	82.238	95.086	1166.25

Table 5: We conducted experiments to explore different patterns on the backbone. As described in Section 3.4 , we designed our backbone scale study using three distinct patterns. These are referred to as P A , P B , and P C , corresponding to balanced distribution, late-stage heavy distribution and an early-stage heavy distribution patterns, respectively. The results demonstrate that, for the L-scale, P A yields the best performance, whereas P C for the X-scale. The images clearly show the preference of different patterns at different scales.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

COCO Training Settings	Le-DETR-M	Le-DETR-L	Le-DETR-X
Backbone	EfficientNAT-M	EfficientNAT-L	EfficientNAT-X
Backbone Freezing Layer	(0, 1)	(0, 1)	(0, 1)
Embedding Dimension	256	256	256
Feedforward Dimension	1024	1024	1024
Encoder Layer Number	1	1	1
NAAIFI Kernel Size	63	63	63

Appendix

Table 6: Implementation details and training hyperparameters of our proposed method.

Figure 5: Visualization of applying Le-DETR-L into hard cases of object detection.

Figure 6: Visualization of applying Le-DETR-X into hard cases of object detection.

Explore similar work

RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models
Oct 29, 2025Zijun Liao, Yian Zhao, Xin Shan +5Real-Time Object Detection Efficient Object Detection
School of Electronic and Computer Engineering, Peking University, Shenzhen, China · Department of Automation and BNRist, Tsinghua University, Beijing, China
RiO-DETR: DETR for Real-time Oriented Object Detection
Mar 10, 2026Zhangchi Hu, Yifan Zhao, Yansong Peng +8Object Detection Detection Transformer
University of Science and Technology of China · Tsinghua University · Huawei Technologies Co., Ltd. +1
Ultralytics YOLO26: Unified Real-Time End-to-End Vision Models
Jun 2, 2026Glenn Jocher, Jing Qiu, Mengyu Liu +3Open-Vocabulary Object Detection Instance Segmentation
Ultralytics

Oct 29, 2025J/K move · Enter open · S save

RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models

Zijun Liao, Yian Zhao, Xin Shan, Yu Yan+4

School of Electronic and Computer Engineering, Peking University, Shenzhen, China · Department of Automation and BNRist, Tsinghua University, Beijing, China

Real-Time Object Detection Efficient Object Detection Detection Transformer Knowledge Distillation

Open paper AskPDF arXiv

Real-time object detection has achieved substantial progress through meticulously designed architectures and optimization strategies. However, the pursuit of high-speed inference via lightweight network designs often leads to degraded feature representation, which hinders further performance improvements and practical on-device deployment. In this paper, we propose a cost-effective and highly adaptable distillation framework that harnesses the rapidly evolving capabilities of Vision Foundation Models (VFMs) to enhance lightweight object detectors. Given the significant architectural and learning objective disparities between VFMs and resource-constrained detectors, achieving stable and task-aligned semantic transfer is challenging. To address this, on one hand, we introduce a Deep Semantic Injector (DSI) module that facilitates the integration of high-level representations from VFMs into the deep layers of the detector. On the other hand, we devise a Gradient-guided Adaptive Modulation (GAM) strategy, which dynamically adjusts the intensity of semantic transfer based on gradient norm ratios. Without increasing deployment and inference overhead, our approach painlessly delivers striking and consistent performance gains across diverse DETR-based models, underscoring its practical utility for real-time detection. Our new model family, RT-DETRv4, achieves state-of-the-art results on COCO, attaining AP scores of

49.8/53.7/55.4/57.0

at corresponding speeds of

273/169/124/78

FPS. Code is publicly available at https://github.com/RT-DETRs/RT-DETRv4.

Abstract

Explore similar work

RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models

RiO-DETR: DETR for Real-time Oriented Object Detection

Ultralytics YOLO26: Unified Real-Time End-to-End Vision Models