Boundary-Aligned Contribution Routing for Robust Optical--SAR Object Detection

Haifa Zhang; Yijing Wang; Haoyu Wang; Zheng Li; Zhiqiang Zuo

cs.CVAug 15, 2026

Boundary-Aligned Contribution Routing for Robust Optical--SAR Object Detection

Authors: Haifa Zhang, Yijing Wang, Haoyu Wang, Zheng Li, Zhiqiang Zuo

Organizations: Tianjin Key Laboratory of Intelligent Unmanned Swarm Technology and System, School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China · Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai 200240, China

Abstract

Optical imagery provides rich appearance cues, whereas synthetic aperture radar (SAR) offers observations that are less sensitive to illumination and weather, making optical--SAR fusion attractive for remote-sensing object detection. However, the presence of multiple modalities does not guarantee beneficial fusion: imperfect spatial, temporal, and semantic correspondence can make an otherwise intact stream conditionally harmful and induce negative cross-modal transfer. We handle this issue through a model-specific task-utility perspective and learn task-conditioned contribution routing using detection supervision alone. The proposed fusion-boundary-aligned routing regulates each modality's contribution before the first learned cross-modal feature-value mixing operation. For architectures with frequent shallow interaction, a Feature Router performs cross-conditioned, group-addressable modulation near the input; for dual-backbone architectures, a Dual-Statistic Semantic Router predicts stream-level contribution weights from modality-specific average and maximum statistics before late semantic fusion. The routers require no explicit utility supervision, quality labels, reconstruction, or distillation. Experiments on M4-SAR and SpaceNet6-OTD cover nominal full inputs, controlled correspondence shifts, missing modalities, and four nonzero modality-corruption scenarios. Across the reported clean-training controls, routing improves full-input $\text{mAP}_{50}$ by 0.5--5.9 points. Relative to the corresponding modality-dropout baselines, it raises missing-modality $\text{mAP}_{50}$ by 7.6--41.6 points and reduces the negative-transfer rate by up to 12.7 percentage points. Spearman correlations between the learned routing weights and model-specific leave-one-modality-out utility range from 0.45 to 0.66, supporting the task-utility interpretation of the routing coefficients.

Figures & tables

Fig. 1: Motivation for fusion-boundary-aligned contribution routing. A present and visually valid stream can still conflict with the other modality and cause negative transfer. We control source-indexed contributions before the first learned cross-modal feature-value mixing operation.

Fig. 2: Topology-aligned realizations of fusion-boundary contribution routing. FR modulates explicitly addressable input channel groups before the backbone mixes their feature values; SR learns a task-conditioned stream contribution weight from separated semantic features before late fusion. The modules are mutually exclusive choices.

Fig. 3: Detailed architecture of the FR stem. Modality-specific scalar weighting precedes concatenation; cross-conditioned channel and spatial gates then perform addressability-preserving modulation before downstream feature-value mixing.

Fig. 4: Architecture of SR. Average and maximum descriptors are mapped to a task-conditioned routing coefficient; the coefficient is not interpreted as calibrated confidence or uncertainty.

Method	Train	Full Modality		Masked Optical		Masked SAR		Params (M) $\downarrow$
Method	Train	$\text{mAP}_{50}\uparrow$	$\text{mAP}_{50:95}\uparrow$	$\text{mAP}_{50}\uparrow$	$\text{mAP}_{50:95}\uparrow$	$\text{mAP}_{50}\uparrow$	$\text{mAP}_{50:95}\uparrow$	Params (M) $\downarrow$
SuperYOLO [ 23 ]	Clean	72.4	46.9	8.9	4.1	31.8	19.6	4.85
MCHE [ 40 ]	Clean	81.2	52.8	12.8	7.5	34.9	22.8	145.81
ICAFusion [ 9 ]	Clean	87.3	58.8	15.0	7.4	40.3	24.4	28.97
MMIDet [ 39 ]	Clean	87.4	57.8	15.5	7.6	34.6	24.0	53.70
Simple U-Net [ 63 ]	Clean	58.2	31.1	0.2	0.0	22.9	12.8	2.42

TABLE I: Results on M4-SAR under a

2\times 2

architecture/training control. “Clean” represents complete pairs only; “MD” follows Eq. ( 7 ). Representative Clean+Router rows isolate architectural full-input gains, whereas MD+Router versus Clean+Router isolates the interaction with missing-modality training. Green parenthesized values denote absolute mAP gains over the corresponding baseline under the same training protocol.

Method	Train	Full Modality		Masked Optical		Masked SAR		Params (M) $\downarrow$
Method	Train	$\text{mAP}_{50}\uparrow$	$\text{mAP}_{50:95}\uparrow$	$\text{mAP}_{50}\uparrow$	$\text{mAP}_{50:95}\uparrow$	$\text{mAP}_{50}\uparrow$	$\text{mAP}_{50:95}\uparrow$	Params (M) $\downarrow$
SuperYOLO [ 23 ]	Clean	96.9	84.1	9.7	5.4	77.8	55.1	4.85
MCHE [ 40 ]	Clean	97.2	85.8	13.6	7.9	80.6	57.9	145.81
ICAFusion [ 9 ]	Clean	96.6	82.8	6.8	3.7	74.6	52.3	28.97
MMIDet [ 39 ]	Clean	97.4	86.5	16.9	9.8	82.4	59.3	53.70
Simple U-Net [ 63 ]	Clean	96.2	74.2	0.0	0.0	65.8	46.2	2.42

TABLE II: Results on SpaceNet6-OTD with the same architecture/training controls as Table I . Clean+Router controls are reported for Simple U-Net, CSSA, and CFT. Green parenthesized values denote absolute mAP gains over the corresponding baseline under the same training protocol.

Configuration	Full Modality		Masked Optical (SAR only)		Masked SAR (Optical only)
Configuration	$\text{mAP}_{50}\uparrow$	$\text{mAP}_{50:95}\uparrow$	$\text{mAP}_{50}\uparrow$	$\text{mAP}_{50:95}\uparrow$	$\text{mAP}_{50}\uparrow$	$\text{mAP}_{50:95}\uparrow$
(a) Ablation on FR with U-Net Baseline
Baseline + modality dropout	43.7	23.4	8.4	3.1	27.6	14.9
+ Independent scalar routing	46.1	25.0	13.7	5.4	31.8	17.2
+ Channel selection	54.8	29.6	19.7	7.7	39.6	21.5
+ Spatial gating (Full FR)	63.2	34.7	26.0	10.4	45.6	24.4

TABLE III: Component ablation on M4-SAR under fixed modality dropout (

p=0.2

). All variants within each block use identical optimization, modality-mask sampling, and data augmentation.

Method	Full Modality		Masked Optical		Masked SAR		Params $\downarrow$
Method	mAP50 $\uparrow$	mAP50:95 $\uparrow$	mAP50 $\uparrow$	mAP50:95 $\uparrow$	mAP50 $\uparrow$	mAP50:95 $\uparrow$	(M)
Baseline (clean training)	58.2	31.1	0.2	0.0	22.9	12.8	2.42
Baseline + modality dropout	43.7	23.4	8.4	3.1	27.6	14.9	2.42
Baseline + SE	44.0	23.7	9.1	3.5	28.1	15.2	2.43
Baseline + CBAM	44.2	23.8	9.5	3.7	28.4	15.4	2.43
Baseline + scalar gate	46.1	25.0	13.7	5.4	31.8	17.2	2.42

TABLE IV: Controlled comparison on M4-SAR. All MD-trained variants use the same 300 epochs, optimizer, modality-mask sampler, augmentation, and modality-dropout probability (

p=0.2

). The clean baseline is diagnostic only.

Method	Adaptation	Params (M) $\downarrow$	GFLOPs $\downarrow$	GPU-h $\downarrow$	Auxiliary objective	Full mAP50 $\uparrow$	Miss OPT mAP50 $\uparrow$	Miss SAR mAP50 $\uparrow$
CSSA
+ MD	None (MD only)	13.51	42.6	18.4	none	62.7 $\pm$ 0.4	18.7 $\pm$ 0.7	38.2 $\pm$ 0.6
+ SE [ 58 ]	Channel recalibration	13.54	42.7	19.0	none	63.4 $\pm$ 0.5	19.8 $\pm$ 0.6	39.6 $\pm$ 0.7
+ CBAM [ 67 ]	Channel–spatial attention	13.56	42.9	19.5	none	64.0 $\pm$ 0.4	20.5 $\pm$ 0.8	39.2 $\pm$ 0.5
+ CML [ 41 ]	Confidence ranking	13.51	42.6	32.8	$\lambda_{\mathrm{CML}}L_{\mathrm{CML}}$	66.1 $\pm$ 0.6	22.8 $\pm$ 0.7	44.5 $\pm$ 0.8
+ ShaSpec [ 44 ]	Shared–specific features	15.20	45.8	26.7	$0.1L_{\mathrm{DAO}}+0.02L_{\mathrm{DCO}}$	70.2 $\pm$ 0.5	23.9 $\pm$ 0.9	47.1 $\pm$ 0.6

TABLE V: Budget-matched CSSA adaptations inspired by existing methods on M4-SAR. These are re-implemented detection adaptations rather than direct reproductions of the original task-specific architectures. All rows use the same modality-dropout sampler and 300-epoch optimization budget. Performance is reported as mean

\pm

standard deviation over three independent runs; Params and GFLOPs denote inference-time cost at

512\times 512

Routing position	Full Modality		Masked Optical		Masked SAR		Params $\downarrow$
Routing position	mAP50 $\uparrow$	mAP50:95 $\uparrow$	mAP50 $\uparrow$	mAP50:95 $\uparrow$	mAP50 $\uparrow$	mAP50:95 $\uparrow$	(M)
Raw input	78.1	50.2	21.4	11.8	45.0	26.3	14.02
After first layer	80.0	51.7	23.6	12.9	48.2	28.1	14.03
Before feature-value mixing (ours)	83.7	54.7	26.3	14.6	55.7	32.1	14.03
After feature-value mixing	77.2	49.4	18.8	10.1	42.3	24.7	14.03
Before late semantic mixing	75.6	48.1	17.2	9.3	39.7	23.0	14.04

TABLE VI: Routing-position study under a matched parameter budget on M4-SAR. The same router is moved through CSSA without changing the modality-mask curriculum. “Before feature-value mixing” denotes placement immediately before the first learned cross-modal feature-value mixing boundary.

$p$	Full Modality		Masked Optical		Masked SAR
$p$	mAP50 $\uparrow$	mAP50:95 $\uparrow$	mAP50 $\uparrow$	mAP50:95 $\uparrow$	mAP50 $\uparrow$	mAP50:95 $\uparrow$
0.0	64.1	35.2	5.8	2.1	30.7	16.9
0.1	64.0	35.0	18.4	7.4	39.2	21.1
0.2	63.2	34.7	26.0	10.4	45.6	24.4
0.3	56.8	29.6	26.4	10.6	46.1	24.8
0.5	Collapse

TABLE VII: Sensitivity of the Modality Dropout probability

p

on M4-SAR using Simple U-Net with FR. “Collapse” indicates failure to converge under the stated training schedule.

Scenario	FR Scalar ( $w$ )		SR Gate ( $\alpha$ )
Scenario	OPT ( $w_{opt}$ )	SAR ( $w_{sar}$ )	OPT ( $\alpha_{opt}$ )	SAR ( $\alpha_{sar}$ )
Full Modality	0.81 $\pm$ 0.07	0.78 $\pm$ 0.09	0.92 $\pm$ 0.03	0.89 $\pm$ 0.05
Masked Optical	0.15 $\pm$ 0.06	0.84 $\pm$ 0.08	0.11 $\pm$ 0.07	0.91 $\pm$ 0.07
Masked SAR	0.86 $\pm$ 0.07	0.17 $\pm$ 0.07	0.93 $\pm$ 0.02	0.16 $\pm$ 0.03

TABLE VIII: Task-conditioned routing weights from two separate architecture-specific experiments: FR (

w

) in Simple U-Net and SR (

\alpha

) in CSSA. Values are mean

\pm

standard deviation over each test set; FR and SR are not used jointly.

Fig. 5: Qualitative comparison under full- and missing-modality settings. “Broken” denotes complete input unavailability. Yellow circles and arrows highlight baseline errors mitigated by task-conditioned routing.

Table 14

Backbone	Train	Base $\downarrow$	+FR/SR $\downarrow$	$\Delta$ (pp)
Simple U-Net	Clean	31.8	20.6	$-11.2$
Simple U-Net	MD	29.4	16.7	$-12.7$
CSSA	Clean	27.5	18.2	$-9.3$
CSSA	MD	25.8	14.9	$-10.9$
CFT	Clean	23.1	15.6	$-7.5$
CFT	MD	21.4	12.8	$-8.6$

TABLE XI: Negative-transfer rate (NTR, %) on M4-SAR. Lower is better; “Base” and “+FR/SR” use matched protocols. “+FR/SR” denotes FR for Simple U-Net and SR for CSSA and CFT.

Fig. 6: Controlled low-utility conditions. (a) Visual examples of optical cloud, optical low-light/haze, SAR speckle, and composite SAR translation with boundary corruption. (b) Detection performance of CFT+SR against CFT, with absolute gains reported under each condition. The misregistration panel is an additional observation rather than evidence that SR explicitly estimates geometric alignment.

Multimodal Sensor Fusion

Multimodal sensor fusion combines data from different sensors to improve perception, tracking, localization, and prediction.

See all 414 papers

Oct 20, 2026

Revealing Inter-Channel Modality Discriminability: Channel Selective Interaction Fusion Network for Optical-SAR Object Detection

Synthetic aperture radar (SAR) and optical imagery provide complementary structural and appearance information for remote sensing object detection. However, substantial cross-modal discrepancies and indiscriminate channel interaction inevitably introduce redundant or conflicting features. Since different feature channels encode heterogeneous semantic information and contribute unequally to cross-modal fusion, selectively highlighting task-relevant channels while suppressing redundant ones is essential. To address this issue, we propose the Channel Selective Interaction Fusion Network (CSIF-Net), which performs selective channel-level cross-modal interaction. CSIF-Net first employs a frequency decomposition module to construct frequency-aware SAR inputs, facilitating the separation of structural information from noise-related responses. It then models cross-modal channel relationships through a cross-attention channel selection module and dynamically partitions the features into shared and modality-specific components. Finally, a dual-branch fusion module performs cross-modal interaction and channel realignment for shared features while independently enhancing SAR-specific features, thereby exploiting complementary information and suppressing modality interference. Extensive experiments on the OGSOD 1.0 and M4-SAR datasets demonstrate that CSIF-Net achieves state-of-the-art performance in both horizontal and oriented object detection, surpassing the existing methods by 4.01% and 3.18% gains, respectively.

Haojie Li, Yuanfei Huang, Lingfei Song +2

School of Artificial Intelligence, Beijing Normal University · Beijing Key Laboratory of Artificial Intelligence for Education · Engineering Research Center of Intelligent Technology and Educational Application, Ministry of Education

Object Detection Multimodal Object Detection Multimodal Fusion Synthetic Aperture Radar Cross-Modal Attention

Ask PDF

May 16, 2025

M4-SAR: A Multi-Resolution, Multi-Polarization, Multi-Scene, Multi-Source Dataset and Benchmark for optical-SAR Object Detection

Single-source remote sensing object detection using optical or SAR images struggles in complex environments. Optical images offer rich textural details but are often affected by low-light, cloud-obscured, or low-resolution conditions, reducing the detection performance. SAR images are robust to weather, but suffer from speckle noise and limited semantic expressiveness. Optical and SAR images provide complementary advantages, and fusing them can significantly improve the detection accuracy. However, progress in this field is hindered by the lack of large-scale, standardized datasets. To address these challenges, we propose a new comprehensive dataset for optical-SAR fusion object detection, named Multi-resolution, Multi-polarization, Multi-scene, Multi-source SAR dataset (M4-SAR). It contains 112,174 instance-level aligned image pairs and nearly one million labeled instances with arbitrary orientations, spanning six key categories. To enable standardized evaluation, we develop a unified benchmarking toolkit that integrates six state-of-the-art multi-source fusion methods. Additionally, we propose E2E-OSDet, a novel end-to-end multi-source fusion detection framework that mitigates cross-domain discrepancies and establishes a robust baseline for future studies. Extensive experiments on M4-SAR demonstrate that fusing optical and SAR data can improve mAP by 5.7% over single-source inputs, with particularly significant gains in complex environments. The dataset and code are publicly available at https://github.com/wchao0601/M4-SAR.

Chao Wang, Wei Lu, Xiang Li +2

Nanjing University of Science and Technology, Nanjing 210094, China · Anhui University, Hefei 230601, China · Nankai University, Tianjin 300350, China

Remote Sensing Object Detection Multimodal Sensor Fusion

Ask PDF

May 12, 2026

TAR: Text Semantic Assisted Cross-modal Image Registration Framework for Optical and SAR Images

Existing deep learning-based methods can capture shared features from optical and synthetic aperture radar (SAR) images for spatial alignment. However, optical-SAR registration remains challenging under large geometric deformations, because the model needs to simultaneously handle cross-modal appearance discrepancies and complex spatial transformations. To address this issue, this paper proposes a text semantic-assisted cross-modal image registration framework, named TAR, for optical and SAR images. TAR exploits text semantic priors from remote sensing scenes and land-cover categories to alleviate the modality gap and enhance cross-modal feature learning. TAR consists of three components: a multi-scale visual feature learning (MSFL) module, a text-assisted feature enhancement (TAFE) module, and a coarse-to-fine dense matching (CFDM) module. MSFL extracts multi-scale visual features from optical and SAR images. TAFE constructs text descriptors related to remote sensing scenes and land-cover objects, and uses a frozen RemoteCLIP text encoder to extract text features. These text features are introduced through visual-text interaction to enhance high-level visual features for more reliable coarse matching. CFDM then establishes coarse correspondences based on the enhanced high-level features and refines the matched locations using low-level features. Experimental results on cross-modal remote sensing images demonstrate the effectiveness of TAR, which achieves stronger matching performance than several state-of-the-art methods and yields significant gains under large geometric deformations.

Zhuoyu Cai, Dou Quan, Ning Huyan +3

Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education of China, School of Artificial Intelligence, Xidian University, Xi’an 710071, China

Remote Sensing Synthetic Aperture Radar Cross-Modal Feature Matching Cross-Modal Image Registration Deformable Image Registration

Ask PDF

Abstract

Multimodal Sensor Fusion

Explore similar work

Revealing Inter-Channel Modality Discriminability: Channel Selective Interaction Fusion Network for Optical-SAR Object Detection

M4-SAR: A Multi-Resolution, Multi-Polarization, Multi-Scene, Multi-Source Dataset and Benchmark for optical-SAR Object Detection

TAR: Text Semantic Assisted Cross-modal Image Registration Framework for Optical and SAR Images