Rethinking Infrared Small Target Detection: A Foundation-Driven Efficient Paradigm
Organizations: Key Laboratory of Opto-Electronic Information Processing, Chinese Academy of Sciences · Shenyang Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · MMLab, The Chinese University of Hong Kong ·
Abstract
While large-scale visual foundation models (VFMs) exhibit strong generalization across diverse visual domains, their potential for single-frame infrared small target (SIRST) detection remains largely unexplored. To fill this gap, we first systematically introduce frozen VFM representations into the SIRST task and propose a Foundation-Driven Efficient Paradigm (FDEP), a general and efficient framework compatible with diverse VFMs and SIRST networks, which improves detection accuracy without additional VFM-related inference overhead. Specifically, a Semantic Alignment and Modulated Fusion (SAMF) module is designed to achieve dynamic alignment and deep fusion of the global semantic priors from VFMs with task-specific features. Meanwhile, to avoid the inference-time overhead introduced by VFMs, we propose a Collaborative Optimization-based Implicit Self-Distillation (CO-ISD) strategy, which enables implicit semantic transfer between the main and lightweight branches through parameter sharing and synchronized backpropagation. In addition, to unify the fragmented evaluation system, we construct a Holistic SIRST Evaluation (HSE) metric that performs multi-threshold integral evaluation at both pixel-level confidence and target-level robustness, providing a stable and comprehensive basis for fair model comparison. Extensive experiments demonstrate that the SIRST detection networks equipped with our FDEP framework achieve state-of-the-art (SOTA) performance on multiple public datasets. Our code is available at https://github.com/YuChuang1205/FDEP-Framework
Figures & tables
| Conventional Accuracy Metrics | HSE Series Accuracy Metrics | Computational Cost | ||||||||
| Methods | Description | IoU ↑ | nIoU ↑ | ↑ | ↓ | HSE-P ↑ | HSE-T ↑ | HSE ↑ | Param. (M) | Time (s) |
| Original | 62.76 | 61.25 | 94.62 | 18.78 | 82.41 | 82.58 | 68.05 | 0.40 | 0.022 | |
| + FDEP w/o CO-ISD | 70.43 | 69.73 | 96.48 | 19.70 | 90.02 ( 7.61) | 86.66 ( 4.08) | 78.01 ( 9.96) | 0.85 | 0.082 | |
| ACM [ 5 ] | + FDEP Framework | 68.56 | 68.01 | 95.42 | 13.92 | 89.17 ( 6.76) | 85.02 ( 2.44) | 75.81 ( 7.76) | 0.85 | 0.022 |
| Original | 68.30 | 69.55 | 95.28 | 34.07 | 87.56 | 84.74 | 74.20 | 0.43 | 0.021 | |
| + FDEP w/o CO-ISD | 73.17 | 73.23 | 97.14 | 24.95 | 91.73 ( 4.17) | 88.67 ( 3.93) | 81.34 ( 7.14) | 0.88 | 0.082 | |
| NUAA-SIRST (213:214) | NUDT-SIRST (663:664) | IRSTD-1K (800: 201) | ||||||||||||||
| Methods | Description | IoU | HSE-P | HSE-T | HSE | IoU | HSE-P | HSE-T | HSE | IoU | HSE-P | HSE-T | HSE | |||
| Original | 67.07 | 91.63 | 87.25 | 89.23 | 77.86 | 67.31 | 97.88 | 86.31 | 85.22 | 73.55 | 63.26 | 90.24 | 83.86 | 86.70 | 72.71 | |
| + FDEP w/o CO-ISD | 70.94 | 94.68 | 90.95 | 90.16 | 82.01 | 72.16 | 98.41 | 90.85 | 85.78 | 77.93 | 67.71 | 89.23 | 88.57 | 83.76 | 74.19 | |
| ACM | + FDEP Framework | 70.34 | 94.68 | 89.60 | 89.39 | 80.09 | 69.82 | 97.67 | 89.56 | 85.24 | 76.34 | 65.96 | 89.90 | 87.44 | 84.34 | 73.75 |
| Original | 70.64 | 92.02 | 90.56 | 87.09 | 78.87 | 72.73 | 96.19 | 91.20 | 88.86 | 81.04 | 67.15 | 90.91 | 88.82 | 86.97 | 77.24 | |
| + FDEP w/o CO-ISD | 74.15 | 95.82 | 93.15 | 91.66 | 85.38 | 77.01 | 98.20 | 93.95 | 89.42 | 84.01 | 68.73 | 95.29 | 89.52 | 88.45 | 79.18 | |
| Schemes | NUAA-SIRST | NUDT-SIRST | IRSTD-1K | |||
| NUDT-SIRST | IRSTD-1K | NUAA-SIRST | IRSTD-1K | NUAA-SIRST | NUDT-SIRST | |
| Original | 48.70 | 45.32 | 63.12 | 31.10 | 75.39 | 38.97 |
| +FDEP w/o CO-ISD | 61.35 (+12.65) | 62.61 (+17.29) | 68.21 (+5.09) | 38.17 (+7.07) | 80.46 (+5.07) | 53.32 (+14.35) |
| +FDEP Framework | 52.26 (+3.56) | 49.50 (+4.18) | 67.23 (+4.11) | 31.41 (+0.31) | 83.34 (+7.95) | 41.98 (+3.01) |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| MSDA-Net + FDEP | MSDA-Net + FDEP | ||||||
| Levels | HSE-P ↑ | HSE-T ↑ | HSE ↑ | Levels | HSE-P ↑ | HSE-T ↑ | HSE ↑ |
| 1 | 96.17 | 93.69 | 90.10 | 12/24 | 96.70 | 94.31 | 91.21 |
| 8 | 96.91 | 93.95 | 91.04 | 8/16/24 | 96.91 | 94.19 | 91.28 |
| 16 | 96.68 | 94.20 | 91.07 | 6/12/18/24 | 96.92 | 94.21 | 91.31 |
| 24 | 96.92 | 93.88 | 90.98 | 4/9/14/19/24 | 96.41 | 94.59 | 91.19 |
| MSDA-Net + FDEP | MSDA-Net + FDEP | |||||||
| Scales | SAMF | HSE-P ↑ | HSE-T ↑ | HSE ↑ | SAMF | HSE-P ↑ | HSE-T ↑ | HSE ↑ |
| ViT-S/14 | ✗ | 96.52 | 93.45 | 90.20 | ✓ | 96.66 (+0.14) | 93.63 (+0.18) | 90.51 (+0.31) |
| ViT-B/14 | ✗ | 96.49 | 93.70 | 90.41 | ✓ | 96.91 (+0.42) | 93.87 (+0.17) | 90.98 (+0.57) |
| ViT-L/14 | ✗ | 96.76 | 93.73 | 90.92 | ✓ | 96.92 (+0.16) | 94.21 (+0.48) | 91.31 (+0.39) |
| MSDA-Net + FDEP w/o CO-ISD | MSDA-Net + FDEP w/o CO-ISD | |||||||
| Scales | SAMF | HSE-P ↑ | HSE-T ↑ | HSE ↑ | SAMF | HSE-P ↑ | HSE-T ↑ | HSE ↑ |
| ViT-S/14 | ✗ | 97.08 | 93.30 | 90.58 | ✓ | 97.21 (+0.13) | 93.61 (+0.31) | 91.00 (+0.42) |
| ViT-B/14 | ✗ | 96.76 | 94.08 | 91.03 | ✓ | 97.34 (+0.58) | 93.83 (-0.25) | 91.34 (+0.31) |
| ViT-L/14 | ✗ | 96.93 | 94.29 | 91.40 | ✓ | 97.25 (+0.32) | 94.32 (+0.03) | 91.73 (+0.33) |
| Conventional Metrics | HSE Series Metrics | ||||||
| Scheme | mIoU ↑ | nIoU ↑ | P d ↑ | F a ↓ | HSE-P ↑ | HSE-T ↑ | HSE ↑ |
| Baseline (MSDA-Net) | 82.86 | 85.46 | 97.01 | 12.53 | 96.43 | 93.04 | 89.72 |
| Encoder (Each stage) | 82.71 | 85.13 | 97.81 | 15.20 | 96.31 (-0.12) | 93.47 (+0.43) | 90.02 (+0.30) |
| Decoder (Each stage) | 83.86 | 85.55 | 96.54 | 11.49 | 96.80 (+0.37) | 93.27 (+0.23) | 90.28 (+0.56) |
| CL Connection (Each stage) | 83.23 | 85.81 | 97.28 | 21.71 | 96.58 (+0.15) | 93.52 (+0.48) | 90.33 (+0.61) |
| Final FDEP (Ours) | 84.05 | 86.33 | 97.54 | 14.24 | 96.92 (+0.49) | 94.21 (+1.17) | 91.31 (+1.59) |
| MSDA-Net + FDEP | MSDA-Net + FDEP | |||||||
| Share | EDL | HSE-P ↑ | HSE-T ↑ | HSE ↑ | EDL | HSE-P ↑ | HSE-T ↑ | HSE ↑ |
| Encoder (E) | ✓ | 94.71 (+1.88) | 90.36 (+3.30) | 85.58 (+4.89) | ✗ | 96.00 (+0.92) | 92.50 (+1.71) | 88.81 (+2.50) |
| Decoder (D) | ✓ | 94.67 (+1.92) | 89.13 (+4.53) | 84.38 (+6.09) | ✗ | 96.40 (+0.52) | 91.96 (+2.25) | 88.64 (+2.67) |
| E + D | ✓ | 96.59 | 93.66 | 90.47 | ✗ | 96.92 | 94.21 | 91.31 |
| Conventional Accuracy Metrics | HSE Series Accuracy Metrics | Computational Cost | ||||||||
| Methods | Description | IoU ↑ | nIoU ↑ | P d ↑ | F a ↓ | HSE-P ↑ | HSE-T ↑ | HSE ↑ | Param (M) | Time (s) |
| FKRW [ 27 ] | Filtering | 14.46 | 23.67 | 60.66 | 22.50 | 20.48 ( 76.44) | 40.73 ( 53.48) | 8.34 ( 82.97) | - | 0.189 |
| MPCM [ 31 ] | Local Contrast | 18.54 | 26.80 | 71.96 | 123.76 | 31.36 ( 65.56) | 13.50 ( 80.71) | 4.24 ( 87.07) | - | 0.104 |
| NIPPS [ 4 ] | Low Rank | 18.82 | 28.59 | 70.56 | 15.17 | 30.46 ( 66.46) | 34.05 ( 60.16) | 10.37 ( 80.94) | - | 31.047 |
| ACM [ 5 ] | CNN-based | 62.76 | 61.25 | 94.62 | 18.78 | 82.41 ( 14.51) | 82.58 ( 11.63) | 68.05 ( 23.26) | 0.40 | 0.022 |
| ALCNet [ 6 ] | CNN-based | 68.30 | 69.55 | 95.28 | 34.07 | 87.56 ( 9.36) | 84.74 ( 9.47) | 74.20 ( 17.11) | 0.43 | 0.021 |
| NUAA-SIRST (213:214) | NUDT-SIRST (663:664) | IRSTD-1K (800: 201) | ||||||||||||||
| Methods | Description | IoU | P d | HSE-P | HSE-T | HSE | IoU | P d | HSE-P | HSE-T | HSE | IoU | P d | HSE-P | HSE-T | HSE |
| FKRW | Filtering | 22.84 | 77.19 | 35.72 | 62.73 | 22.41 | 14.70 | 58.10 | 21.74 | 37.70 | 8.20 | 9.25 | 54.21 | 12.49 | 33.40 | 4.17 |
| MPCM | Local Contrast | 42.92 | 88.59 | 61.20 | 62.18 | 38.05 | 12.55 | 64.34 | 22.42 | 10.57 | 2.37 | 21.36 | 81.48 | 35.35 | 11.36 | 4.02 |
| NIPPS | Low Rank | 26.82 | 80.23 | 61.40 | 54.71 | 33.59 | 18.66 | 68.89 | 54.29 | 42.13 | 22.87 | 14.93 | 67.34 | 27.18 | 46.72 | 12.70 |
| ACM | CNN-based | 67.07 | 91.63 | 87.25 | 89.23 | 77.86 | 67.31 | 97.88 | 86.31 | 85.22 | 73.55 | 63.26 | 90.24 | 83.86 | 86.70 | 72.71 |
| ALCNet | CNN-based | 70.64 | 92.02 | 90.56 | 87.09 | 78.87 | 72.73 | 96.19 | 91.20 | 88.86 | 81.04 | 67.15 | 90.91 | 88.82 | 86.97 | 77.24 |
| Case 1 | Case 2 | Case 3 | Case 4 | |||||
| Cases | S1 | S2 | S1 | S2 | S1 | S2 | S1 | S2 |
| AUC ↑ | 0.9993 | 0.9799 | 0.9987 | 0.9997 | 0.9989 | 0.9880 | 0.6970 | 0.6977 |
| hIoU (%) ↑ | 16.67 | 19.00 | 27.43 | 27.43 | 18.87 | 38.10 | 15.93 | 38.52 |
| HSE-P (%) ↑ | 45.37 | 89.75 | 14.24 | 36.25 | 31.95 | 91.27 | 17.04 | 64.46 |
| HSE-T (%) ↑ | 33.33 | 100.00 | 36.67 | 36.67 | 45.00 | 100.00 | 21.67 | 66.67 |
| HSE (%) ↑ | 15.12 | 89.75 | 5.22 | 13.29 | 14.39 | 91.27 | 3.70 | 42.97 |
| Schemes | HSE | hIoU | ROC-AUC | AP | [email protected] FPPI | [email protected] FPPI | nFROC-AUC@1 FPPI |
|---|---|---|---|---|---|---|---|
| ACM (Original) | 68.05 | 55.05 | 98.69 | 81.86 | 36.11 | 96.74 | 87.42 |
| + FDEP Framework | 75.81 (+7.76) | 61.99 | 99.65 | 88.98 | 54.90 | 97.41 | 90.77 |
| ALCNet (Original) | 74.20 | 62.18 | 99.37 | 86.50 | 69.58 | 97.01 | 90.00 |
| + FDEP Framework | 78.31 (+4.11) | 67.06 | 98.83 | 88.92 | 94.68 | 96.35 | 91.56 |
| MLCL-Net (Original) | 86.80 | 75.55 | 98.74 | 93.84 | 96.15 | 97.87 | 93.37 |
| + FDEP Framework | 87.20 (+0.40) | 79.08 | 98.31 | 93.42 | 95.75 | 97.67 | 94.46 |
| Compared Metric | hIoU | ROC-AUC | AP | [email protected] FPPI | [email protected] FPPI | nFROC-AUC@1 FPPI |
|---|---|---|---|---|---|---|
| Spearman | 0.935 | 0.191 | 0.879 | 0.904 | 0.786 | 0.909 |
Infrared Small Target Detection
Infrared small target detection finds faint, often sub-pixel objects against cluttered thermal backgrounds while limiting false alarms.
See all 46 papers