A Closer Look at Cross-Domain Few-Shot Object Detection: Fine-Tuning Matters and Parallel Decoder Helps
Organizations: Intellindust AI Lab · Suzhou Institute for Advanced Research, USTC
Abstract
Few-shot object detection (FSOD) is challenging due to unstable optimization and limited generalization arising from the scarcity of training samples. To address these issues, we propose a hybrid ensemble decoder that enhances generalization during fine-tuning. Inspired by ensemble learning, the decoder comprises a shared hierarchical layer followed by multiple parallel decoder branches, where each branch employs denoising queries either inherited from the shared layer or newly initialized to encourage prediction diversity. This design fully exploits pretrained weights without introducing additional parameters, and the resulting diverse predictions can be effectively ensembled to improve generalization. We further leverage a unified progressive fine-tuning framework with a plateau-aware learning rate schedule, which stabilizes optimization and achieves strong few-shot adaptation without complex data augmentations or extensive hyperparameter tuning. Extensive experiments on CD-FSOD, ODinW-13, and RF100-VL validate the effectiveness of our approach. Notably, on RF100-VL, which includes 100 datasets across diverse domains, our method achieves an average performance of 41.9 in the 10-shot setting, significantly outperforming the recent approach SAM3, which obtains 35.7. We further construct a mixed-domain test set from CD-FSOD to evaluate robustness to out-of-distribution (OOD) samples, showing that our proposed modules lead to clear improvement gains. These results highlight the effectiveness, generalization, and robustness of the proposed method. Code is available at: https://github.com/Intellindust-AI-Lab/FT-FSOD.
Figures & tables
| Method | Backbone | ArTaxOr | Clipart1k | DIOR | DeepFish | NEU-DET | UODD | Avg. | |
| 0-shot | MMGDINO-B [ 61 ] | Swin-B [ 32 ] | 7.3 | 48.8 | 5.3 | 42.0 | 1.2 | 20.0 | 20.8 |
| 1-shot | DE-ViT-FT [ 60 ] | DINOv2-L [ 35 ] | 10.5 | 13.0 | 14.7 | 19.3 | 0.6 | 2.4 | 10.1 |
| CD-ViTO [ 12 ] | DINOv2-L [ 35 ] | 21.0 | 17.7 | 17.8 | 20.3 | 3.6 | 3.1 | 13.9 | |
| VFMDETR [ 28 ] | DINOv2-L [ 35 ] | 26.1 | 20.1 | 20.6 | 24.2 | 9.1 | 9.0 | 18.2 | |
| CDFormer [ 34 ] | DINOv2-L [ 35 ] | 36.0 | 54.0 | 16.3 | 34.5 | 7.4 | 12.7 | 26.8 | |
| ETS [ 37 ] | Swin-B [ 32 ] | 28.1 | 57.5 | 12.7 | 40.7 | 11.7 | 21.2 | 28.7 |
| Model | Backbone | Ckpt avail. | 0-shot | 1-shot | 3-shot | 5-shot | 10-shot |
| GLIPv2-H [ 57 ] | Swin-H [ 32 ] | ✗ | 55.5 | ||||
| GLEE-Pro [ 52 ] | EVA02-L [ 9 ] | ✓ | 53.4 | ||||
| MQ-GLIP-L [ 54 ] | Swin-L [ 32 ] | ✓ | 54.1 | 62.4 | 64.2 | 65.4 | 66.6 |
| Grounding DINO 1.5 Pro [ 43 ] | EVA02-L [ 9 ] | ✗ | 58.7 | ||||
| SAM3 [ 2 ] | PE-L+ [ 1 ] | ✓ | 59.9 | ||||
| Ours (MMGDINO-L [ 61 ] ) | Swin-L [ 32 ] | ✓ | 55.3 |
| Model | Backbone | Aerial | Document | Flora-Fauna | Industrial | Medical | Sports | Other | Avg. |
| Detic [ 60 ] | Swin-L [ 32 ] | 12.2 / 19.5 | 4.5 / 19.6 | 17.9 / 28.4 | 6.0 / 25.9 | 0.8 / 8.5 | 7.6 / 26.6 | 11.2 / 25.7 | 9.5 / 22.8 |
| SAM3 [ 2 ] | PE-L+ [ 1 ] | 20.7 / 33.9 | 11.9 / 34.0 | 23.4 / 38.8 | 8.2 / 39.7 | 2.0 / 24.9 | 16.4 / 40.3 | 15.7 / 34.7 | 14.3 / 35.7 |
| MMGDINO-L [ 61 ] | Swin-L [ 32 ] | 21.5 / 32.4 | 9.2 / 30.6 | 27.9 / 41.3 | 10.3 / 37.8 | 2.1 / 18.3 | 13.3 / 33.2 | 17.5 / 32.0 | 15.7 / 33.6 |
| Ours (MMGDINO-L) | Swin-L [ 32 ] | 21.5 / 41.5 | 9.2 / 38.9 | 27.9 / 45.9 | 10.3 / 46.7 | 2.1 / 25.8 | 13.3 / 36.2 | 17.5 / 47.2 | 15.7 / 41.9 |
| Baseline (Naive aug & Plateau scheduler) | ✓ | ✓ | ✓ | ✓ | |
| Progressive Fine-tuning | ✓ | ✓ | |||
| Hybrid Ensemble Decoder | ✓ | ✓ | |||
| CD-FSOD (0-shot: 20.8) | 1-shot | 30.8 | 33.3 | 33.1 | 34.9 |
| 5-shot | 43.1 | 44.6 | 43.4 | 45.0 | |
| 10-shot | 47.1 | 47.1 | 47.2 | 47.9 | |
| ODinW-13 (0-shot: 55.3) | 1-shot | 60.7 | 61.8 | 62.0 | 63.1 |
| Decoder Structure | DnQ re-init rate (Eq. 7 ) | CD-FSOD | ODinW-13 | ||
| 1-shot | 5-shot | 1-shot | 5-shot | ||
| 6-stacked (Baseline+prog. ft.) – | 33.3 | 44.6 | 61.8 | 66.1 | |
| 6-parallel | 0 | 27.4 | 40.0 | 58.4 | 63.6 |
| 4-stacked+2-parallel | 0 | 33.9 | 44.8 | 60.3 | 66.6 |
| 3-stacked+3-parallel | 0 | 32.9 | 44.3 | 60.0 | 67.3 |
| 2-stacked+4-parallel | 0 | 34.2 | 44.3 | 61.9 | 66.4 |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Hyperparameters |
| Init. learning-rate for detection transformers | 1e-4 |
| Init. learning-rate for image and language backbones | 2e-5 |
| Minimum learning rate | 1e-6 |
| Weight decay | 0.05 |
| Batch size | 4 |
| Number of epochs | 100 |
| CD-FSOD | CD-Mixed | 1-shot | 5-shot | 10-shot |
| Randomly Initialized | 33.7 | 25.04 | 44.1 | 36.08 | 47.2 | 38.02 |
| Pre-trained | 34.9 | 25.68 | 45.0 | 37.14 | 47.9 | 40.34 |
| CD-FSOD | CD-Mixed | 1-shot | 5-shot | 10-shot |
| 1-stage (cosine) + HED | 34.6 | 24.92 | 43.6 | 33.52 | 47.2 | 38.30 |
| 2-stage (cosine) + HED | 34.6 | 25.46 | 44.2 | 35.34 | 47.4 | 38.42 |
| 2-stage (plateau) + HED (Ours) | 34.9 | 25.68 | 45.0 | 37.14 | 47.9 | 40.34 |
| Datasets | 1-shot |
| AerialMaritimeDrone | 26.267 |
| Aquarium | 47.500 |
| CottontailRabbits | 76.967 |
| EgoHands | 67.133 |
| NorthAmericaMushrooms | 81.067 |
| Packages | 65.833 |
| Datasets | 3-shot |
| AerialMaritimeDrone | 34.033 |
| Aquarium | 52.767 |
| CottontailRabbits | 77.067 |
| EgoHands | 69.033 |
| NorthAmericaMushrooms | 83.567 |
| Packages | 67.733 |
| Datasets | 5-shot |
| AerialMaritimeDrone | 38.200 |
| Aquarium | 53.733 |
| CottontailRabbits | 76.533 |
| EgoHands | 72.767 |
| NorthAmericaMushrooms | 87.000 |
| Packages | 70.833 |
| Datasets | 10-shot |
| AerialMaritimeDrone | 39.367 |
| Aquarium | 55.167 |
| CottontailRabbits | 75.200 |
| EgoHands | 71.900 |
| NorthAmericaMushrooms | 88.600 |
| Packages | 73.033 |
| Aerial Datasets | 10-shot |
| aerial-airport | 48.1 |
| aerial-cows | 34.4 |
| aerial-sheep | 48.2 |
| apoce-aerial-photographs -for-object-detection- of-construction-equipment | 46.2 |
| electric-pylon-detection-in-rsi | 21.1 |
| floating-waste | 36.5 |
| Document Datasets | 10-shot |
| activity-diagrams | 30.4 |
| all-elements | 39.5 |
| circuit-voltages | 34.5 |
| invoice-processing | 23.7 |
| label-printing-defect-version-2 | 54.1 |
| macro-segmentation | 34.1 |
| Flora-Fauna Datasets | 10-shot |
| aquarium-combined | 52.6 |
| bees | 30.9 |
| deepfruits | 64.9 |
| exploratorium-daphnia | 26.7 |
| grapes-5 | 39.3 |
| grass-weeds | 46.7 |
| Industrial Datasets | 10-shot |
| -grccs | 50.2 |
| 13-lkc01 | 32.6 |
| 2024-frc | 62.0 |
| aircraft-turnaround-dataset | 35.4 |
| asphaltdistressdetection | 21.2 |
| cable-damage | 24.2 |
| Medical Datasets | 10-shot |
| canalstenosis | 46.6 |
| crystal-clean-brain-tumors-mri-dataset | 56.1 |
| dentalai | 20.4 |
| inbreast | 34.2 |
| liver-disease | 19.7 |
| nih-xray | 8.8 |
| Other Datasets | 10-shot |
| buoy-onboarding | 26.1 |
| car-logo-detection | 73.9 |
| clashroyalechardetector | 40.6 |
| cod-mw-warzone | 46.4 |
| countingpills | 85.8 |
| everdaynew | 50.0 |
| Baseline + Progressive ft. + HED w/ re-init Dn Query | 1-shot | 5-shot | 10-shot | |||
| w/o OOD | w/ OOD | w/o OOD | w/ OOD | w/o OOD | w/ OOD | |
| ArTaxOr | 49.1 | 44 | 76.8 | 73.2 | 79.2 | 75.7 |
| DIOR | 24.6 | 23.6 | 35.3 | 33.8 | 41.5 | 40.3 |
| NEU-DET | 15.5 | 5.3 | 25.2 | 14.2 | 28.4 | 15.7 |
| UODD | 22.1 | 13.3 | 27.5 | 17.0 | 32.3 | 25.7 |
| clipart1k | 55.6 | 42.2 | 59.4 | 47.5 | 59.6 | 44.3 |
Few-Shot Object Detection
Few-shot object detection adapts detectors to recognize and localize new object classes from only a handful of labeled examples.
See all 27 papers