Organizations: Department of Automation, Tsinghua University, BeiJing, China · School of Artificial Intelligence, China University of Mining Technology - Beijing, BeiJing, China · State Key Laboratory of Blockchain and Data Security, Zhejiang University, HangZhou, China · Tsinghua University, BeiJing, China
Text-guided multispectral object detection uses text semantics to guide semantic-aware cross-modal interaction between RGB and IR for more robust perception. However, notable limitations remain: (1) existing methods often use text only as an auxiliary semantic enhancement signal, without exploiting its guiding role to bridge the inherent granularity asymmetry between RGB and IR; and (2) conventional data-driven attention-based fusion tends to emphasize stable consensus while overlooking potentially valuable cross-modal discrepancies. To address these issues, we propose a semantic bridge fusion framework with bi-support modeling for multispectral object detection. Specifically, text is used as a shared semantic bridge to align RGB and IR responses under a unified category condition, while the recalibrated thermal semantic prior is projected onto the RGB branch for semantic-level mapping fusion. We further formulate RGB-IR interaction evidence into the regular consensus support and the complementary discrepancy support that contains potentially discriminative cues, and introduce them into fusion via dynamic recalibration as a structured inductive bias. In addition, we design a bidirectional semantic alignment module for closed-loop vision-text guidance enhancement. Extensive experiments demonstrate the effectiveness of the proposed fusion framework and its superior detection performance on multispectral benchmarks. Code is available at https://github.com/zhenwang5372/Bridging-RGB-IR-Gap.
Figures & tables
Figure 1. Comparison of different fusion paradigms. Unlike vanilla direct RGB–IR fusion and conditional prompt fusion, our semantic-bridge fusion explicitly exploits textual semantics to enable semantic-level RGB–IR interaction, while introducing consensus–discrepancy support modeling to facilitate fusion. This leads to more salient target responses in the fused features, achieving the best detection accuracy with significantly fewer parameters. An overview figure comparing different text-guided RGB–IR fusion paradigms, including conventional RGB–IR fusion, conditional prompt fusion, and the proposed semantic-bridge fusion. The figure also shows consensus and discrepancy support patterns, as well as fused results in which the target regions are more salient under the proposed method.
Figure 2. Visualization of consensus–discrepancy activation under dark conditions. The peak-distribution and GT-box statistics indicate that consensus constitutes an important support pattern in (a). In (b), discrepancy is further incorporated based on (a), thereby providing a more complete characterization of target perception. Two activation maps under dark conditions, showing discrepancy support on the left and consensus support on the right. Additional rectangular plots compare activation statistics in GT boxes and background regions.
Figure 3. Overview of the proposed framework. It consists of a semantic-bridge-guided dynamic fusion module for modeling consensus and discrepancy support patterns for adaptive RGB–IR fusion, a bidirectional semantic alignment module for cross-modal semantic interaction, and a detection head for final prediction. A woman and a girl in white dresses sit in an open car.
Figure 4. Illustration of the proposed bi-support modeling. Consensus and discrepancy support are constructed from semantic responses and jointly integrated through dynamic recalibration to enhance multispectral fusion. A woman and a girl in white dresses sit in an open car.
Table 1. Comparison with state-of-the-art methods on the LLVIP and FLIR datasets. Results are reported in mAP, [email protected], and [email protected]. AVG denotes the average of the overall accuracy on LLVIP and FLIR. Best results are in bold.
Table 2. Comparison with state-of-the-art methods on the DroneVehicle dataset. Results are reported for each category together with [email protected]. Best results are in bold.
Figure 5. Qualitative comparison on seven representative scenes from the FLIR dataset. The examples include daytime perception (Scene-1&2), small target perception in nighttime conditions (Scene-3), occluded cases (Scenes-4&5), and crowded scenes (Scenes-6&7). Our method yields more reliable predictions in challenging multispectral scenes. A woman and a girl in white dresses sit in an open car.
Backbone
Method
Modality
Car ↑
Bus ↑
Truck ↑
mAP 50 ↑
mAP ↑
ResNet50
RetinaNet ( Lin et al., 2017 )
IR
79.0
68.0
60.0
57.9
33.8
Faster-RCNN ( Ren et al., 2015 )
IR
81.3
66.7
63.3
63.3
35.6
RetinaNet ( Lin et al., 2017 )
RGB
83.2
75.8
69.9
64.7
37.4
Faster-RCNN ( Ren et al., 2015 )
RGB
87.9
81.3
72.4
74.5
42.3
TiNet ( Zhang et al., 2023b )
RGB+IR
87.5
83.1
81.6
79.8
49.5
Ours
RGB+IR
91.0
92.7
87.7
84.3
55.8
Table 3. Comparison of different methods under ResNet50 and CSPDarknet backbones on the M3FD dataset.
Variant
IR Backbone
Dynamic Fusion
Bidirectional Alignment
recall ↑
mAP 50 ↑
mAP ↑
V0
0.6845
0.775
0.383
V1
✓
0.6982
0.789
0.398
V2
✓
✓
0.7663
0.871
0.488
V3 (Full)
✓
✓
✓
0.7785
0.886
0.504
Table 4. Overall ablation study of the proposed framework on the FLIR dataset. Best results are in bold.
Method Variant
AP ↑
AP 50 ↑
AP 75 ↑
FLOPs ↓
Text as a Semantic Bridge
Vanilla Direct RGB–IR Fusion
41.9
82.2
36.4
13.56 G
Conditional Prompt Fusion
41.5
83.4
31.7
13.56 G
Ours (Semantic Bridge Fusion)
50.6
88.6
45.7
4.05G
Text as a Semantic Anchor
w/o Bidirectional Update
46.4
85.4
41.6
–
Table 5. Analysis of the two roles of text on the FLIR dataset. Best results are in bold.
Figure 6. Perception performance comparison on FLIR. The proposed fusion achieves better target perception than vanilla direct fusion and conditional prompt fusion.
Consensus Support
Discrepancy Support
Recalibration
mAP ↑
mAP 50 ↑
mAP 75 ↑
41.3
80.2
36.1
✓
42.2
82.5
37.4
✓
44.5
82.5
38.9
✓
✓
42.6
80.5
37.6
✓
✓
✓
50.4
88.6
45.7
Table 6. Effectiveness of the bi-support modeling on FLIR. “Consensus” and “Discrepancy” denote the two support patterns, respectively. “Recalibration” denotes dynamic recalibration. Best results are in bold.
Figure 7. Visualization of responses of consensus support Mcons and discrepancy support Mdis under progressively degraded RGB inputs on a typical sample. The numbers denote average activation values within the ground-truth box. A woman and a girl in white dresses sit in an open car.
Figure 8. Population-level trends of the two support patterns across degradation levels. The numbers denote the average activation values within the GT-box or Background. A woman and a girl in white dresses sit in an open car.
Figure 9. Qualitative comparison on five representative scenes from the LLVIP dataset. A woman and a girl in white dresses sit in an open car.
Figure 10. Qualitative comparison on five representative scenes from the M3FD dataset. A woman and a girl in white dresses sit in an open car.
Figure 11. Comparison of different fusion paradigms on the FLIR dataset in terms of computational complexity and detection performance. V-D, C-P, and Our S-B denote Vanilla Direct Fusion, Conditional Prompt Fusion, and our Semantic Bridge Fusion, respectively.
IR Backbone
Params (M)
FLOPs (G)
Latency (ms)
FPS
mAP50
mAP
ResNet50
24.20
34.18
4.69
213.3
0.853
0.453
Our IR backbone
1.05
6.49
3.30
302.7
0.886
0.504
Appendix
Table 8. Efficiency and accuracy comparison of IR backbones on the FLIR dataset (image size 640×640 ).
Level
Brightness ( γ )
Sharpen
Blur ( k )
Noise ( σ )
0
No modification
–
–
–
+1
0.70
–
5
8
+2
0.55
–
9
15
+3
0.40
–
15
22
+4
0.28
–
21
32
+5
0.18
–
29
42
Appendix
Table 9. Parameters for discrete degradation levels. Four representative levels are selected for population-level statistics.
Figure 12. Visualization of responses of consensus support Mcons and discrepancy support Mdis under progressively degraded RGB inputs on typical samples.
Metric
Value
Number of GT boxes
34,130
Mean
0.4112
Median
0.4051
Standard deviation
0.1447
Minimum
0.0181
Maximum
0.9601
Appendix
Table 10. Foreground occupancy statistics within GT boxes on the LLVIP training set.
Figure 13. Population-level trends of consensus and discrepancy support across RGB degradation levels on LLVIP. The bars show the globally normalized mean responses in the GT regions, and the red dashed line denotes the average pedestrian occupancy ratio within GT boxes ( 41.1% ). With increasing RGB degradation, consensus support in the GT region decreases, whereas discrepancy support increases, indicating a progressive migration of discriminative evidence.
RGB-infrared (RGB-IR) object detection benefits from complementary visible and thermal cues, but effective fusion remains challenging under illumination changes, weather variation, and cluttered scenes. Existing RGB-IR fusion methods often trade expressive patch-level interaction for lighter but more constrained adaptation mechanisms. We empirically observe that pretrained register tokens contain both modality-shared and modality-specific information on paired RGB-IR inputs, suggesting that they can serve as a compact substrate for cross-modal communication. Building on this observation, we propose RegisterBridgeMM, a register-mediated fusion framework organized as a three-stage register lifecycle. Aggregate preserves per-modality register summarization inherited from pretraining; Bridge performs bidirectional register-to-patch reading with consensus-residual regulation; and Project translates the resulting register summary into spatially adaptive calibration of patch features. This register pathway avoids dense patch-to-patch cross-modal interaction while preserving the pretrained patch representation. With both backbone streams frozen, RegisterBridgeMM achieves the highest mAP50-95 among the evaluated methods on all four benchmarks: LLVIP, M3FD, DroneVehicle, and FLIR-Aligned.
Zian Wang, Hangchuan Liang, Yuehua Chen +4
Jilin University, China · Taiyuan University of Technology, China · Shenzhen University, China
Joint use of RGB and infrared (IR) imagery can improve UAV-view object detection, but most existing methods fuse multimodal features with static or fixed weights and therefore overlook spatially varying modality reliability. We propose EGM-Det, an entropy-guided multimodal adaptive fusion framework for RGB-IR object detection. EGM-Det employs a dual-stream architecture to preserve modality-specific representations and introduces an Entropy Offset Gate Fusion module for adaptive multi-scale fusion. The module derives shallow entropy priors from input intensity, local entropy, and cross-modal discrepancy, and uses them to guide local offset alignment and spatial-channel gated fusion. It therefore selectively aggregates reliable RGB and infrared cues instead of uniformly combining heterogeneous features. We further introduce cross-modal distillation to regularize the learned fusion gates and reduce fusion degradation. Each student branch extracts complementary knowledge from the cross-modality teacher branch matched to the main branch, while entropy-adaptive supervision emphasizes uncertain modality decisions. Experiments on DroneVehicle, LLVIP, and VEDAI demonstrate state-of-the-art performance across all three benchmarks; in particular, EGM-Det outperforms prior approaches by more than 10 percentage points on VEDAI.
Cunzheng Fan, Dawei Yan, Guanlin Wang +4
School of Cybersecurity, Northwestern Polytechnical University, Xi’an 710072, China · School of Automation and Software Engineering, Shanxi University, Taiyuan 030006, China
Remote sensing vision-language models have advanced Earth observation understanding, but most existing work remains centered on RGB imagery, leaving the complementary information in infrared data underexplored. Infrared images provide distinctive cues, including thermal intensity structures, object boundaries, and illumination-invariant scene features, which can enrich visual-language learning beyond conventional RGB observations. However, a large-scale RGB-infrared-text dataset for remote sensing vision-language modeling is still absent. To address this gap, we introduce FusionRS, the first large-scale RGB-infrared-text dataset designed for dual-modal vision-language learning in remote sensing. FusionRS is constructed by translating diverse public RGB remote sensing images into infrared-style counterparts, forming aligned RGB-IR image pairs. Each pair is associated with conventional scene captions and IR-aware captions that explicitly describe infrared-specific visual properties while preserving semantic content. Based on FusionRS, we train dual-modal vision-language foundation models for RGB-IR joint understanding. We first train CLIP-style models for RGB-IR-text alignment, and then fine-tune generative VLMs for dual-modal RGB-IR captioning. Experiments show that FusionRS improves RGB-IR alignment, infrared-to-text retrieval, and dual-modal captioning over RGB-only and non-IR-aware training settings. Ablation studies further verify that IR-aware captions are crucial for strengthening infrared-language alignment, highlighting the importance of modality-specific textual supervision for more scalable RGB-infrared remote sensing vision-language representation learning.
Jiaju Han, Ben Zhang, Xuemeng Sun +6
China University of Petroleum-Beijing at Karamay · University of Electronic Science and Technology of China · Tianjin University