Does YOLO Really Need to See Every Training Image in Every Epoch?

Xingxing Xie; Jiahua Dong; Junwei Han; Gong Cheng

cs.CVMar 18, 2026

Does YOLO Really Need to See Every Training Image in Every Epoch?

Authors: Xingxing Xie, Jiahua Dong, Junwei Han, Gong Cheng

Organizations: School of Automation, Northwestern Polytechnical University · School of Artificial Intelligence, Chongqing University of Posts and Telecommunications

Abstract

YOLO detectors are known for their fast inference speed, yet training them remains unexpectedly time-consuming due to their exhaustive pipeline that processes every training image in every epoch, even when many images have already been sufficiently learned. This stands in clear contrast to the efficiency suggested by the ``You Only Look Once'' philosophy. This naturally raises an important question: Does YOLO really need to see every training image in every epoch? To explore this, we propose an Anti-Forgetting Sampling Strategy (AFSS) that dynamically determines which images should be used and which can be skipped during each epoch, allowing the detector to learn more effectively and efficiently. Specifically, AFSS measures the learning sufficiency of each training image as the minimum of its detection recall and precision, and dynamically categorizes training images into easy, medium, or hard levels accordingly. Easy training images are sparsely resampled during training in a continuous review manner, with priority given to those that have not been used for a long time to reduce redundancy and prevent forgetting. Moderate training images are partially selected, prioritizing recently unused ones and randomly choosing the rest from unselected images to ensure coverage and prevent forgetting. Hard training images are fully sampled in every epoch to ensure sufficient learning. The learning sufficiency of each training image is periodically updated, enabling detectors to adaptively shift its focus toward the informative training images over time while progressively discarding redundant ones. On widely used natural image detection benchmarks (MS COCO 2017 and PASCAL VOC 2007) and remote sensing detection datasets (DOTA-v1.0 and DIOR-R), AFSS achieves more than $1.43\times$ training speedup for YOLO-series detectors while also improving accuracy.

Figures & tables

Figure 2: Overview of the proposed Anti-Forgetting Sampling Strategy (AFSS) at the $t$ -th epoch of training.

Method	MS COCO 2017			PASCAL VOC 2007
Method	AP $\uparrow$	Training time (h) $\downarrow$	Speedup ( $\times$ ) $\uparrow$	mAP $\uparrow$	Training time (h) $\downarrow$	Speedup ( $\times$ ) $\uparrow$
YOLOv8n	37.3	30.4	–	75.9	4.5	–
YOLOv8n + AFSS	37.4	21.2	1.43 $\times$	76.0	2.8	1.60 $\times$
YOLOv10n	38.5	39.9	–	76.1	5.6	–
YOLOv10n + AFSS	38.6	27.4	1.45 $\times$	76.2	3.5	1.60 $\times$
YOLO11n	39.5	32.7	–	76.3	5.2	–

Table 1: Accuracy and training efficiency of different models with and without AFSS on MS COCO 2017 and PASCAL VOC 2007 datasets, evaluated on two RTX 4090 GPUs (24 GB each). All subsequent results are obtained under the same hardware configuration.

Method	DOTA-v1.0			DIOR-R
Method	mAP $\uparrow$	Training time (h) $\downarrow$	Speedup ( $\times$ ) $\uparrow$	mAP $\uparrow$	Training time (h) $\downarrow$	Speedup ( $\times$ ) $\uparrow$
YOLOv8n-OBB	78.0	94.1	–	79.9	5.1	–
YOLOv8n-OBB + AFSS	78.1	57.5	1.63 $\times$	79.9	3.1	1.64 $\times$
YOLO11n-OBB	78.4	89.3	–	80.2	5.0	–
YOLO11n-OBB + AFSS	78.4	53.8	1.65 $\times$	80.3	3.0	1.66 $\times$
YOLOv8s-OBB	79.5	106.2	–	80.5	8.8	–

Table 2: Accuracy and training efficiency of different models with and without AFSS on DOTA-v1.0 and DIOR-R datasets.

Method	AP $\uparrow$	Training time (h) $\downarrow$	Speedup $\uparrow$
YOLO11s (Baseline)	47.0	43.9	–
+ Curriculum Learning	43.7	32.5	1.35 $\times$
+ Self-Paced Learning	44.5	33.6	1.30 $\times$
+ Data Pruning	40.5	31.8	1.38 $\times$
+ Dataset Distillation	35.6	29.2	1.50 $\times$
+ AFSS (Ours)	47.2	28.4	1.54 $\times$

Table 3: Comparison of different training methods.

LSM	CR	STC	SU	AP $\uparrow$	Speedup $\uparrow$
				47.0	–
✓				44.8	1.45 $\times$
✓	✓			45.5	1.34 $\times$
✓		✓		46.6	1.31 $\times$
✓	✓	✓		47.2	1.26 $\times$
✓	✓	✓	✓	47.2	1.54 $\times$

Table 4: Ablation of AFSS core components using YOLOv11s as the baseline. LSM: Learning Sufficiency Metric; CR: Continuous Review; STC: Short-Term Coverage; SU: State Update.

Metrics	AP $\uparrow$	Speedup $\uparrow$
Loss-based	46.0	1.52 $\times$
Gradient-based	46.9	1.45 $\times$
F1 score-based	46.6	1.51 $\times$
Our metric	47.2	1.54 $\times$

Table 5: Ablation studies on learning sufficiency metric, continuous review interval, short-term coverage interval, and state update interval.

Figure 3: Changes in the number of samples at easy, moderate, and hard levels during the training of YOLO11s with AFSS.

Figure 4: An example illustrating the learning performance of YOLO11s and YOLO11s+AFSS at the 100th, 300th, and 600th training epochs on the hard image.

Efficient Object Detection

Efficient object detection reduces the computation, latency, memory, and energy needed to find and locate objects in images and video.

See all 95 papers

Aug 7, 2026

YOLO-PEFT: Parameter-Efficient Fine-Tuning on YOLO Family

Generic parameter-efficient fine-tuning (PEFT) methods transferred from language models can fail silently on real-time detectors, whose heterogeneous operators and detection-specific components impose placement constraints absent from regular Transformer stacks. We propose YOLO-PEFT, a structure-aware framework that formulates adapter placement as an auditable constraint-planning problem. Given a detector graph, a PEFT request, and a resource budget, YOLO-PEFT assigns operator and semantic roles, evaluates explicit operator-validity, detector-semantic, graph-interface, and deployment predicates, records a reason code for each excluded module, and either emits a budgeted target-module plan or returns Refuse before training. Under the official VOC07+12 trainval-to-VOC07 test protocol, planner-selected RS-LoRA reaches 0.7138 and 0.7307 mAP50-95 on YOLO11s and YOLO12s, respectively, compared with 0.6428 and 0.6662 for Full-SFT. On RT-DETR-L, all seven evaluated LoRA-family configurations cross the predefined catastrophic threshold, supporting a calibrated Refuse-to-Full-SFT decision within the evaluated coverage. A controlled YOLO11 audit further shows that LoRA reduces peak training memory by 43.9 percent, although training takes 1.72 times longer. Within the evaluated detector families, placement policies, and calibration coverage, YOLO-PEFT replaces manual target-module trial and error with explicit, inspectable planning while preserving verified train-save-merge-export paths; refusal on unseen detector architectures remains an open validation problem. Project Page: github.com/Tencent/YOLO-Master

Xu Lin, WenJie Nie, Jinlong Peng +4

Tencent · Xiamen University

Object Detection Real-Time Object Detection Parameter-Efficient Fine-Tuning

Ask PDF

Jun 2, 2026

Ultralytics YOLO26: Unified Real-Time End-to-End Vision Models

Real-time vision demands models that are accurate, efficient, and simple to deploy across diverse hardware. The YOLO family has become widely deployed for this reason, yet most YOLO detectors still rely on non-maximum suppression at inference, carry heavy detection heads due to Distribution Focal Loss, require long training schedules, and can leave the smallest objects without positive label assignments. We present Ultralytics YOLO26, a unified real-time vision model family that addresses these limitations through coordinated architecture and training advances. YOLO26 uses a dual-head design for native NMS-free end-to-end inference and removes DFL entirely, yielding a lighter head with unconstrained regression range. Its training pipeline combines MuSGD, a hybrid Muon-SGD optimizer adapted from large language model training; Progressive Loss, which shifts supervision toward the inference-time head; and STAL, a label assignment strategy that guarantees positive coverage for small objects. Beyond detection, YOLO26 introduces task-specific head and loss designs for instance segmentation, pose estimation, and oriented detection, producing consistent gains across tasks and scales. The family spans five scales (n/s/m/l/x) and supports detection, instance segmentation, pose estimation, classification, and oriented detection in a single pipeline, with an open-vocabulary extension, YOLOE-26, for text-, visual-, and prompt-free inference. Across all scales, YOLO26 achieves 40.9-57.5 mAP on COCO at 1.7-11.8 ms T4 TensorRT latency, advancing the accuracy-latency Pareto front over prior real-time detectors, while YOLOE-26x reaches 40.6 AP on LVIS minival under text prompting. Code and models are available at https://github.com/ultralytics/ultralytics.

Glenn Jocher, Jing Qiu, Mengyu Liu +3

Ultralytics

Open-Vocabulary Object Detection Instance Segmentation Efficient Object Detection Object Detection Real-Time Object Detection

Ask PDF

May 10, 2026

AnyDepth-DETR/-YOLO: Any-depth object detection with a single network

Modern object detectors are static, fixed-depth networks optimized for a single operating point, requiring separate models for different deployment scenarios. We present an any-depth detection framework that enables a single network to span a continuous range of accuracy--efficiency trade-offs by controlling depth at inference time without retraining. Each backbone and neck stage is divided into an essential path, which always executes, and a skippable refinement path; this decomposition preserves the full multi-scale feature hierarchy at every depth configuration, unlike conventional early exiting that discards entire stages. To train such a network, jointly optimizing many sub-networks of varying depth introduces conflicting gradient signals. We address this via self-distillation between only the two extremes, with prediction-level and feature-level alignment losses that enforce stage-wise modularity, ensuring the outputs of each stage remain compatible regardless of the paths taken. Instantiated on RT-DETR and YOLOv12, our full-depth configurations match or surpass their respective SOTA baselines with negligible parameter overhead, while the most efficient configurations achieve up to

1.82\times

speedup at a cost of only 2.0 AP, all from a single set of weights.

Woochul Kang, Hyungseop Lee, Jiho Lee

Incheon Nat’l Univ.

Efficient Neural Network Inference Dynamic Neural Networks Efficient Object Detection Object Detection Real-Time Object Detection

Ask PDF

Abstract

Efficient Object Detection

Explore similar work

YOLO-PEFT: Parameter-Efficient Fine-Tuning on YOLO Family

Ultralytics YOLO26: Unified Real-Time End-to-End Vision Models

AnyDepth-DETR/-YOLO: Any-depth object detection with a single network