FGD2HF: Frequency-guided Dual-domain Hypergraph Fusion for Semantic Segmentation
Organizations: Harbin Institute of Technology, Xidazhi Street 92, Nangang Dist., Harbin, 150001, Heilongjiang, China · Harbin Engineering University, Nantong Street 145, Nangang Dist., Harbin, 150001, Heilongjiang, China
Abstract
Frequency-domain information is beneficial for semantic segmentation. However, existing frequency-based methods often suffer from insufficient frequency utilization, inefficient fusion, and high computational complexity. To address these issues, we propose FGD2HF, a dual-domain fusion network that leverages frequency features to guide spatial-domain fusion and enhancement. Specifically, we introduce a novel Hybrid-Attention Encoder (NAHA) that jointly models local and global relationships through neighborhood and hypergraph attention, enhancing the network's ability to extract effective features. We further introduce a Wavelet Transform and Enhancement (WTE) module to extract and refine low- and high-frequency cues from preceding-stage features, followed by Frequency-guided Dual-domain Fusion (FGDF) modules that use frequency cues to condition current-stage spatial grouping and feature propagation. This design incorporates both global and local frequency guidance into spatial feature modulation, enhancement and fusion with low computational cost. Extensive comparisons with numerous state-of-the-art semantic segmentation methods demonstrate that our method is highly competitive. Our full FGD2HF-B model achieves 48.7% mIoU on ADE20K (+0.3% over SegMAN-B with 18% fewer GFLOPs), 83.2% mIoU on Cityscapes (comparable to SegMAN-B with 21% fewer GFLOPs), and 1.1% higher mIoU than VWFormer-T on SBD with 21% fewer GFLOPs, while also achieving significant improvements in boundary segmentation accuracy.
Figures & tables
| NAHA Encoders | Channels | Depths | Params (M) | FLOPs (G) |
| Tiny | 3.08 | 0.78 | ||
| Small | 20.40 | 3.38 | ||
| Base | 43.22 | 8.61 | ||
| Large | 80.87 | 16.80 |
| Items | Values |
| Training Settings | |
| Image size | ( for Cityscapes) |
| Batch size | |
| AMP | True |
| Optimization | |
| Optimizer | AdamW |
| Models | Params (M) | GFLOPs | Top-1 Acc (%) |
| MIT-B0 | 3.8 | 0.6 | 70.4 |
| MSCAN-T | 4.2 | 0.9 | 75.7 |
| EFT-T | 3.7 | 0.6 | 72.5 |
| MobileNetV3-L | 5.2 | 0.5 | 75.2 |
| SegMAN-Encoder-T | 3.5 | 0.7 | 76.2 |
| EfficientVMamba-T | 6.0 | 0.8 | 76.4 |
| Method | Cat. | Params (M) | SBD | Cityscapes | ADE20K | |||||||||
| GFLOPs | mIoU | mBIoU | mBF1 | GFLOPs | mIoU | mBIoU | mBF1 | GFLOPs | mIoU | mBIoU | mBF1 | |||
| DeepLabV3-MobileNet (CVPR 2017) [ 12 ] | Non-Fre. | 5.1 | 6.1 | 70.1 | 58.2 | 57.1 | 47.2 | 68.8 | 62.5 | 76.1 | 6.1 | 34.7 | 24.4 | 31.2 |
| DeepLabV3+-MobileNet (ECCV 2018) [ 13 ] | 5.2 | 17.6 | 71.0 | 59.0 | 57.4 | 135.1 | 72.1 | 63.2 | 76.3 | 17.6 | 35.0 | 25.2 | 31.9 | |
| SegFormer-B0 (NeurIPS 2021) [ 75 ] | 3.8 | 8.4 | 68.3 | 54.5 | 52.6 | 121.1 | 72.4 | 65.6 | 76.7 | 8.4 | 36.4 | 25.8 | 32.4 | |
| SegNeXt-T (NeurIPS 2022) [ 32 ] | 4.3 | 6.6 | 77.9 | 65.7 | 64.7 | 56.0 | 79.0 | 74.0 | 83.9 | 6.6 | 38.0 | 27.0 | 34.5 | |
| EDAFormer-T (ECCV 2024) [ 86 ] | 5.0 | 5.6 | 75.9 | 64.9 | 63.3 | 151.5 | 76.9 | 71.5 | 82.3 | 5.6 | 38.1 | 26.9 | 34.6 | |
| Models | Params (M) | GFLOPs | mIoU (%) | FPS (Hz) | GPU Memory (MB) |
| Segformer-B0 | 3.8 | 8.4 | 68.3 | 200.3 | 181.1 |
| SegNext-T | 4.3 | 6.6 | 77.9 | 197.5 | 57.3 |
| EDAFormer-T | 5.0 | 5.6 | 75.9 | 165.0 | 57.8 |
| CGRSeg-T | 9.4 | 4.8 | 78.8 | 135.6 | 359.6 |
| SegMAN-T | 6.4 | 6.0 | 79.0 | 123.8 | 193.0 |
| FGD2HF-T | 5.0 | 5.8 | 79.8 | 96.0 | 188.6 |
| Fusion variant | mIoU (%) | mBIoU (%) | mBF1 (%) |
| SHF | 81.9 (-1.7) | 73.6 (-4.3) | 71.9 (-5.1) |
| FGDF (ours) | 83.6 | 77.9 | 77.0 |
| Model Variant | Params (M) | GFLOPs | mIoU (%) | FPS (Hz) |
| NAHA | 23.4 | 19.7 | 83.6 | 71.7 |
| Encoder Replace | ||||
| MIT-B2 | 33.9 (+10.5) | 26.7 (+7.0) | 80.5 (-3.1) | 78.8 (+7.1) |
| MSCAN-B | 36.0 (+12.6) | 33.1 (+13.4) | 82.7 (-0.9) | 77.1 (+5.4) |
| Swin-T | 46.4 (+23.0) | 37.0 (+17.3) | 81.9 (-1.7) | 80.1 (+8.4) |
| ConvNeXt-T | 46.7 (+23.3) | 34.7 (+15.0) | 81.7 (-1.9) | 87.7 (+16.0) |
| Model Variant | Params (M) | GFLOPs | mIoU (%) | mBIoU (%) |
| FGD2HF | 23.4 | 19.7 | 48.1 | 38.6 |
| FGD2HF w/o WTE&FGDF | 18.1 (-5.3) | 16.5 (-3.2) | 44.3 (-3.8) | 34.2 (-4.4) |
| Information Suppress | ||||
| Suppress high-fre info | 23.4 | 19.7 | 47.2 (-0.9) | 36.5 (-2.1) |
| WTE w/o enhancement | 22.1 (-1.3) | 18.9 (-0.8) | 46.9 (-1.2) | 36.9 (-1.7) |
| WTE w/o low-fre modulation | 23.2 (-0.2) | 19.6 (-0.1) | 47.5 (-0.6) | 37.2 (-1.4) |
| Model Variant | Params (M) | GFLOPs | mIoU (%) | FPS (Hz) |
| FCN Decoder [ 55 ] | 24.6 (+1.2) | 26.9 (+7.2) | 81.7 (-1.9) | 76.4 (+4.7) |
| SegFormer Decoder [ 75 ] | 23.1 (-0.3) | 23.5 (+3.8) | 82.4 (-1.2) | 74.1 (+2.4) |
| LightHam Decoder [ 32 ] | 23.0 (-0.4) | 20.1 (+0.4) | 82.6 (-1.0) | 69.2 (-2.5) |
| EDAFormer Decoder [ 86 ] | 26.3 (+2.9) | 20.7 (+1.0) | 83.1 (-0.5) | 70.7 (-1.0) |
| SegMAN Decoder [ 29 ] | 28.0 (+4.6) | 22.4 (+2.7) | 83.3 (-0.3) | 71.5 (-0.2) |
| Hyper Decoder (Ours) | 23.4 | 19.7 | 83.6 | 71.7 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Semantic Segmentation
Semantic segmentation assigns a class label to every image pixel, enabling detailed scene understanding across domains and sensors.
See all 414 papers