AGPF: Dynamic Source Selection for Feature Pyramid Fusion
Organizations: Department of Computer Science Illinois Institute of Technology
Abstract
Feature pyramids provide dense predictors with features at several spatial resolutions. However, it remains largely unexplored which pyramid levels each output location should aggregate information from. We investigate this question by allowing each output location to attend to features from every pyramid level and measuring how attention is distributed across scales. Surprisingly, locations on the finest output grid allocate approximately 75% of their attention to features two levels coarser, rather than to the current or adjacent levels. Even on the same output grid, the balance shifts toward coarser features as object size increases. These observations motivate Asymmetric Global Pyramid Fusion (AGPF), a neck that preserves the original output grids while allowing each location to dynamically select its contextual sources across scales. Each location interacts with a multiscale memory whose features are aligned in image coordinates, enabling flexible cross-scale aggregation without altering prediction resolutions. Compared with FPN, AGPF improves all 19 object detection and all seven instance segmentation configurations evaluated on COCO. A single fusion block improves Cascade Mask R-CNN by 1.7 box AP and 2.3 mask AP, and the gains consistently transfer to semantic and panoptic segmentation.
Figures & tables
| Detector | Neck / Fusion | Source | Box Detection | Instance Segmentation | Mem. | ||||||||||
| AP | AP 50 | AP 75 | AP S | AP M | AP L | AP | AP 50 | AP 75 | AP S | AP M | AP L | ||||
| Faster R-CNN | FPN [ 29 ] | Reimpl. | 37.4 | 58.1 | 40.4 | 21.2 | 41.0 | 48.1 | – | – | – | – | – | – | 3777 |
| BiFPN [ 43 ] | Reimpl. | 35.3 | 54.9 | 38.1 | 20.3 | 38.7 | 45.3 | – | – | – | – | – | – | 5540 | |
| HRFPN [ 42 ] | Reimpl. | 37.4 | 58.3 | 40.5 | 21.4 | 40.9 | 48.3 | – | – | – | – | – | – | 6632 | |
| PAFPN [ 33 ] | Reimpl. | 37.7 | 58.5 | 41.0 | 22.0 | 41.4 | 48.7 | – | – | – | – | – | – | 3631 | |
| AugFPN [ 16 ] | Reimpl. | 38.1 | 59.5 | 41.4 | 22.9 | 41.6 | 49.1 | – | – | – | – | – | – | 3583 | |
| Method | Fusion | AP box | AP | AP | AP mask | AP | AP |
| One-stage detection (ResNet-50) | |||||||
| RetinaNet [ 30 ] Anchor-based | FPN | 36.5 | 55.4 | 20.4 | – | – | – |
| AGPF-S | 37.9 +1.4 | 57.2 | 21.9 | – | – | – | |
| FCOS [ 44 ] Anchor-free | FPN | 36.6 | 56.0 | 21.0 | – | – | – |
| AGPF-S | 37.3 +0.7 | 56.2 | 21.0 | – | – | – | |
| RepPoints [ 50 ] Point-set | FPN | 37.0 | 56.7 | 20.4 | – | – | – |
| Method | Setting | Recipe | AP box | AP mask |
| Conventional CNN-based systems | ||||
| RetinaNet | R50 + NASFPN | 50e | 40.5 | – |
| Mask R-CNN | R101 + FPN | MS, | 42.7 | 38.5 |
| Cascade Mask R-CNN | R101 + FPN | MS, | 45.5 | 39.6 |
| HTC | R101 + FPN | 20e | 44.8 | 39.6 |
| DetectoRS [ 39 ] | R101 + HTC | 20e | 50.5 | 43.9 |
| Method | Hours | Iter. (s) | Train Mem. | ||
| Mask2Former | 29.5M | 62 | 0.6 | 10732 | 45.7 / 42.9 |
| Mask DINO | 37.6M | 102 | 1.0 | 14520 | 50.5 / 46.0 |
| DINO | 37.1M | 83 | 1.1 | 24394 | 51.2 / – |
| Co-DETR | 41.0M | 132 | 1.8 | 29228 | 54.8 / – |
| AGPF-HTC | 34.0M | 83 | 1.1 | 16972 | 52.4 / 45.8 |
| AGPF-HTC++ | 34.0M | 169 | 1.6 | 15707 | 55.2 / 48.0 |
| Ablation | Setting | Default | AP box | AP mask | Mem. |
| #Heads | 2 | 42.6 | 38.1 | 6383 | |
| 4 | 42.5 | 38.0 | 6356 | ||
| 8 | ✓ | 42.9 | 38.3 | 6389 | |
| 16 | 42.9 | 38.3 | 6405 | ||
| Pooling | AvgPool | 42.7 | 38.1 | 6392 | |
| MaxPool | 42.8 | 38.3 | 6387 |
| Framework | Backbone | Fusion | Cityscapes | ADE20K | ||
| mIoU | mAcc | mIoU | mAcc | |||
| UPerNet [ 48 ] | R50 | FPN | 79.3 | 87.2 | 40.7 | 51.9 |
| AGPF-B | 80.0 | 87.4 | 43.7 | 54.7 | ||
| Swin-L | FPN | 82.3 | 88.8 | 52.0 | 63.2 | |
| AGPF-L | 82.5 | 89.0 | 52.4 | 63.7 | ||
| K-Net [ 56 ] | R50 | FPN | 80.5 | 87.1 | 43.4 | 54.8 |
| Framework | Backbone | Fusion | |||||
| Panoptic FPN [ 24 ] | R50 | FPN | 42.5 | 50.3 | 30.7 | 78.1 | 51.7 |
| AGPF-B | 46.4 | 53.4 | 35.8 | 81.4 | 55.8 | ||
| K-Net [ 56 ] | R50 | FPN | 47.1 | 51.6 | 40.2 | 80.6 | 57.3 |
| AGPF-B | 49.0 | 53.4 | 42.2 | 81.4 | 59.0 | ||
| Swin-L | FPN | 54.5 | 60.2 | 46.0 | 82.3 | 65.4 | |
| AGPF-L | 55.8 | 61.6 | 46.9 | 82.6 | 66.7 |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Neck | Params (M) | Box AP | Mask AP |
| FPN | 3.34 | 41.2 | 36.0 |
| FPN, expanded | 6.49 | 41.2 | 36.1 |
| AGPF-S | 6.49 | 42.9 | 38.3 |
| config | value |
| optimizer | AdamW |
| base learning rate | 2e-4 |
| weight decay | 0.05 |
| batch size | 16 (2/GPU, 8 GPUs) |
| warmup iterations | 1000 |
| training epochs | 36 |
| config | value |
| optimizer | AdamW |
| base learning rate | 2e-4 |
| weight decay | 0.05 |
| batch size | 16 (2/GPU, 8 GPUs) |
| warmup iterations | 1000 |
| training epochs | 50 |
Multi-Scale Feature Fusion
Multi-scale feature fusion combines representations at different resolutions or receptive fields to improve perception, prediction, and reconstruction.
See all 194 papers