Dual-Stage Invariant Continual Learning under Extreme Visual Sparsity

Rangya Zhang; Jiaping Xiao; Lu Bai; Yuhang Zhang; Mir Feroskhan

cs.CVMar 27, 2026

Dual-Stage Invariant Continual Learning under Extreme Visual Sparsity

Authors: Rangya Zhang, Jiaping Xiao, Lu Bai, Yuhang Zhang, Mir Feroskhan

Organizations: School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore 639798, Singapore

Abstract

Continual learning seeks to maintain stable adaptation under non-stationary environments, yet this problem becomes particularly challenging in object detection, where most existing methods implicitly assume relatively balanced visual conditions. In extreme-sparsity regimes, such as those observed in space-based resident space object (RSO) detection scenarios, foreground signals are overwhelmingly dominated by background observations. Under such conditions, we analytically demonstrate that background-driven gradients destabilize the feature backbone during sequential domain shifts, causing progressive representation drift. This exposes a structural limitation of continual learning approaches relying solely on output-level distillation, as they fail to preserve intermediate representation stability. To address this, we propose a dual-stage invariant continual learning framework via joint distillation, enforcing structural and semantic consistency on both backbone representations and detection predictions, respectively, thereby suppressing error propagation at its source while maintaining adaptability. Furthermore, to regulate gradient statistics under severe imbalance, we introduce a sparsity-aware data conditioning strategy combining patch-based sampling and distribution-aware augmentation. Experiments on a high-resolution space-based RSO detection dataset show consistent improvement over established continual object detection methods, achieving an absolute gain of +4.0 mAP under sequential domain shifts.

Figures & tables

Fig. 1 : Overall framework of our method. The top left panel illustrates the space-based observation configuration. The top right shows our sparsity-aware data conditioning pipeline. The bottom panel depicts the dual-stage invariance in COD.

Fig. 2 : Overview of dataset characteristics. (a) Raw $4418\times 4418$ 16-bit grayscale image with sparse targets barely visible. (b) Reference image after contrast enhancement for visualization. (c) Zoomed regions highlighting LEO (blue), MEO (green), and GEO (red) RSOs. (d) Per-class target count (top) and target size distribution (bottom) for the shown example.

Fig. 3 : Temporal target distributions across all four camera datasets. Each subplot shows the per-frame target count over time, with LEO/MEO/GEO class breakdown. Background regions labeled D 1 to D 4 indicate domain divisions.

Fig. 4 : Spatial distributions of target centers across three consecutive subsets of 500 images each (Domain 1 to Domain 3). Each heatmap reflects the local density of targets per grid cell, clipped at a maximum count of 5.

Panel of Figure 5 — Fig. 5 : Pairwise t-SNE visualizations with density contours overlaid for each domain. Contour maps highlight structural differences in feature distributions across domains.

Domain Pair	MMD	Wasserstein	DA (%)
Domain 1 vs 2	0.088	0.018	98.0
Domain 1 vs 3	0.259	0.032	97.3
Domain 2 vs 3	0.050	0.011	97.3

TABLE I : Pairwise domain discrepancy metrics. DA denotes domain classification accuracy.

Method	Domain 1				Domain 2				Total Metric
Method	Pre.	Rec.	F1	mAP	Pre.	Rec.	F1	mAP	Pre.	Rec.	F1	mAP
Single Domain 1	49.38	47.00	48.16	47.86	41.85	41.00	41.42	31.52	45.61	44.00	44.79	39.69
Single Domain 2	31.69	33.00	32.33	28.98	48.98	44.00	46.36	36.32	40.34	38.50	39.35	32.65
Sequential w/o CL	37.92	37.00	37.46	34.28	44.25	47.00	45.59	36.18	41.09	42.00	41.52	35.23
Joint Training	48.42	48.00	48.21	47.05	52.06	45.00	48.27	38.06	50.24	46.50	48.24	42.56
YOLOv10m [ 36 ]	40.39	38.00	39.15	37.71	48.54	46.00	47.23	36.58	44.47	42.00	43.19	37.14

TABLE II : Comparison with existing detectors and CL methods under a domain-incremental setting. “Domain 1” and “Domain 2” are the two sequential domains. “Total Metric” is the macro-average over the two domains. mAP reports mAP@50. Pre., Rec., and F1 denote precision, recall, and F1-score respectively.

Patch	Aug	Feat	Head	Task 1				Task 2				Total Metric
Patch	Aug	Feat	Head	Pre.	Rec.	F1	mAP	Pre.	Rec.	F1	mAP	Pre.	Rec.	F1	mAP
$\times$	$\times$	$\times$	$\times$	0.02	0.04	0.03	0.00	0.02	0.04	0.03	0.00	0.02	0.04	0.03	0.00
✓	$\times$	$\times$	$\times$	28.32	26.66	26.38	18.38	20.93	47.41	27.94	37.03	24.63	37.03	27.16	27.70
✓	✓	$\times$	$\times$	37.92	37.00	37.46	34.28	44.25	47.00	45.59	36.18	41.09	42.00	41.52	35.23
✓	✓	$\times$	✓	42.57	45.00	43.75	37.84	52.54	44.00	47.89	36.78	47.56	44.50	45.82	37.31
✓	✓	✓	$\times$	42.82	36.29	39.28	30.74	45.74	42.70	44.17	42.47	44.28	39.50	41.73	36.61

TABLE III : Ablation study on different components of the proposed framework. “Patch” denotes patch selection, “Aug” denotes domain-aware augmentation, “Feat” and “Head” represent feature-level and head-level distillation, respectively.

Method	Domain 1				Domain 2				Domain 3				Total Metric
Method	Pre.	Rec.	F1	mAP	Pre.	Rec.	F1	mAP	Pre.	Rec.	F1	mAP	Pre.	Rec.	F1	mAP
Single Domain 1	49.38	47.00	48.16	47.86	41.85	41.00	41.42	31.52	25.93	31.00	28.24	18.35	39.05	39.67	39.27	32.58
Single Domain 2	31.69	33.00	32.33	28.98	48.98	44.00	46.36	36.32	38.93	38.00	38.46	21.30	39.87	38.33	39.05	28.87
Single Domain 3	25.12	14.00	17.98	14.18	21.23	21.00	21.12	8.40	59.93	55.00	57.36	38.05	35.43	30.00	32.15	20.21
Sequential w/o CL	32.54	22.00	26.25	25.25	28.14	23.00	25.31	15.56	56.10	50.00	52.87	38.88	39.92	31.67	34.81	26.56
Joint Training	47.61	48.00	47.81	46.44	47.83	47.00	47.41	35.18	54.15	44.00	48.55	32.63	49.86	46.33	47.92	38.08

TABLE IV : Comparison across three sequential domains. Each task corresponds to a distinct domain shift.

Method	Domain 1				Domain 2				Total Metric
Method	Pre.	Rec.	F1	mAP	Pre.	Rec.	F1	mAP	Pre.	Rec.	F1	mAP
Single Domain 1	49.38	47.00	48.16	47.86	41.85	41.00	41.42	31.52	45.61	44.00	44.79	39.69
Single Domain 2	31.69	33.00	32.33	28.98	48.98	44.00	46.36	36.32	40.34	38.50	39.35	32.65
Sequential w/o CL	43.10	47.00	44.97	44.90	50.70	40.00	44.72	34.81	46.90	43.50	44.84	39.85
Joint Training	48.42	48.00	48.21	47.05	52.06	45.00	48.27	38.06	50.24	46.50	48.24	42.56
Ours	49.70	43.00	46.11	48.25	52.82	40.00	45.52	36.16	51.26	41.50	45.82	42.20

TABLE V : Evaluation under reversed domain sequence (

\mathcal{D}^{(2)}

\rightarrow

\mathcal{D}^{(1)}

Fig. 6 : Qualitative comparison of detection results under the domain-incremental setting. (a) Ground truth. (b) Shmelkov et al. [ 23 ] . (c) Backbone-only distillation. (d) EWC [ 11 ] . (e) OGD [ 38 ] . (f) CWD [ 39 ] . (g) CL-DETR [ 25 ] . (h) Ours.

Continual Object Detection

See all 13 papers

Focus, Align, and Sustain: Counteracting Gradient Dilution in Incremental Object Detection
Jun 13, 2026Aoting Zhang, Dongbao Yang, Chang Liu +2Class-Incremental Learning Detection Transformer
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Nankai University +2
STAR-IOD: Scale-decoupled Topology Alignment with Pseudo-label Refinement for Remote Sensing Incremental Object Detection
May 20, 2026Yaoteng Zhang, Qing Zhou, Junyu Gao +1Pseudo-Labeling Object Detection
School of Computer Science, Northwestern Polytechnical University, Xi’an 710072, China · School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University, Xi’an 710072, China · Northwestern Polytechnical University, Xi’an 710072, China
Continual Segmentation under Joint Nonstationarity
May 19, 2026Prashant Pandey, Himanshu Kumar, Devineni Sri Venkatraya Chowdary +1Class-Incremental Learning Continual Learning
Bharti School of Telecommunications Technology & Management, Indian Institute of Technology, Delhi, India · Yardi School of Artificial Intelligence, Indian Institute of Technology, Delhi, India · Department of Computer Science and Engineering, Indian Institute of Technology (Indian School of Mines), Dhanbad, India

Jun 13, 2026J/K move · Enter open · S save

Focus, Align, and Sustain: Counteracting Gradient Dilution in Incremental Object Detection

Aoting Zhang, Dongbao Yang, Chang Liu, Xiaopeng Hong+1

Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Nankai University+2

Class-Incremental Learning Detection Transformer Experience Replay Object Detection Incremental Learning

Open paper AskPDF arXiv

Adapting Detection Transformers to Incremental Object Detection (IOD) poses a systemic challenge, as set-based optimization is inherently destabilized by sequential learning. In this work, we identify Gradient Dilution as the root cause of performance degradation, wherein optimization signals required to preserve old knowledge are progressively weakened. This phenomenon manifests as a cascading erosion of preservation gradients in magnitude, direction, and support coverage, driven by three tightly coupled factors: Signal Dispersion, where foreground gradients are overwhelmed by background noise; Assignment Drift, where stochastic query-target matching induces inconsistent gradient trajectories; and Support Attrition, where gradients from retained samples insufficiently cover the old-class feature space, weakening decision boundaries under interference from new classes. To counteract this, we propose FAS, a unified framework that Focuses, Aligns, and Sustains gradient flow throughout incremental learning. Specifically, we introduce prior-injected queries to focus discriminative signals by filtering background interference at the source. We further propose deterministic anchor distillation to align query-target assignments and enforce semantic consistency across stages under unstable matching. Finally, we devise manifold-support replay to sustain distributional support of old classes, counteracting representational erosion induced by continual updates. Extensive experiments show that FAS restores robust optimization dynamics and outperforms state-of-the-art methods, achieving over 5.0 AP improvement in the challenging 40+10x4 incremental setting.

Abstract

Continual Object Detection

Explore similar work

Focus, Align, and Sustain: Counteracting Gradient Dilution in Incremental Object Detection

STAR-IOD: Scale-decoupled Topology Alignment with Pseudo-label Refinement for Remote Sensing Incremental Object Detection

Continual Segmentation under Joint Nonstationarity