Bridging the RGB-IR Gap: Consensus and Discrepancy Modeling for Text-Guided Multispectral Detection

Jiaqi Wu; Zhen Wang; Enhao Huang; Kangqing Shen; Yulin Wang; Yang Yue; Yifan Pu; Gao Huang

cs.CVApr 13, 2026

Bridging the RGB-IR Gap: Consensus and Discrepancy Modeling for Text-Guided Multispectral Detection

Authors: Jiaqi Wu, Zhen Wang, Enhao Huang, Kangqing Shen, Yulin Wang, Yang Yue, Yifan Pu, Gao Huang

Organizations: Department of Automation, Tsinghua University, BeiJing, China · School of Artificial Intelligence, China University of Mining Technology - Beijing, BeiJing, China · State Key Laboratory of Blockchain and Data Security, Zhejiang University, HangZhou, China · Tsinghua University, BeiJing, China

Abstract

Text-guided multispectral object detection uses text semantics to guide semantic-aware cross-modal interaction between RGB and IR for more robust perception. However, notable limitations remain: (1) existing methods often use text only as an auxiliary semantic enhancement signal, without exploiting its guiding role to bridge the inherent granularity asymmetry between RGB and IR; and (2) conventional data-driven attention-based fusion tends to emphasize stable consensus while overlooking potentially valuable cross-modal discrepancies. To address these issues, we propose a semantic bridge fusion framework with bi-support modeling for multispectral object detection. Specifically, text is used as a shared semantic bridge to align RGB and IR responses under a unified category condition, while the recalibrated thermal semantic prior is projected onto the RGB branch for semantic-level mapping fusion. We further formulate RGB-IR interaction evidence into the regular consensus support and the complementary discrepancy support that contains potentially discriminative cues, and introduce them into fusion via dynamic recalibration as a structured inductive bias. In addition, we design a bidirectional semantic alignment module for closed-loop vision-text guidance enhancement. Extensive experiments demonstrate the effectiveness of the proposed fusion framework and its superior detection performance on multispectral benchmarks. Code is available at https://github.com/zhenwang5372/Bridging-RGB-IR-Gap.

Figures & tables

Figure 1. Comparison of different fusion paradigms. Unlike vanilla direct RGB–IR fusion and conditional prompt fusion, our semantic-bridge fusion explicitly exploits textual semantics to enable semantic-level RGB–IR interaction, while introducing consensus–discrepancy support modeling to facilitate fusion. This leads to more salient target responses in the fused features, achieving the best detection accuracy with significantly fewer parameters. An overview figure comparing different text-guided RGB–IR fusion paradigms, including conventional RGB–IR fusion, conditional prompt fusion, and the proposed semantic-bridge fusion. The figure also shows consensus and discrepancy support patterns, as well as fused results in which the target regions are more salient under the proposed method.

Panel of Figure 2 — Figure 2. Visualization of consensus–discrepancy activation under dark conditions. The peak-distribution and GT-box statistics indicate that consensus constitutes an important support pattern in (a). In (b), discrepancy is further incorporated based on (a), thereby providing a more complete characterization of target perception. Two activation maps under dark conditions, showing discrepancy support on the left and consensus support on the right. Additional rectangular plots compare activation statistics in GT boxes and background regions.

Figure 3. Overview of the proposed framework. It consists of a semantic-bridge-guided dynamic fusion module for modeling consensus and discrepancy support patterns for adaptive RGB–IR fusion, a bidirectional semantic alignment module for cross-modal semantic interaction, and a detection head for final prediction. A woman and a girl in white dresses sit in an open car.

Figure 4. Illustration of the proposed bi-support modeling. Consensus and discrepancy support are constructed from semantic responses and jointly integrated through dynamic recalibration to enhance multispectral fusion. A woman and a girl in white dresses sit in an open car.

Method	Modality	Year	Backbone	LLVIP			FLIR			AVG $\uparrow$	Params/M $\downarrow$
Method	Modality	Year	Backbone	mAP $\uparrow$	[email protected] $\uparrow$	[email protected] $\uparrow$	mAP $\uparrow$	[email protected] $\uparrow$	[email protected] $\uparrow$	AVG $\uparrow$	Params/M $\downarrow$
FasterRCNN ( Ren et al., 2015 )	RGB	2015	ResNet50	45.1	87.0	41.2	27.7	62.2	21.2	36.4	41.1
RetinaNet ( Lin et al., 2017 )	RGB	2017	ResNet50	42.8	88.0	34.4	21.9	51.2	15.2	32.4	43.0
YOLOv5 ( Jocher, 2020 )	RGB	2020	CSPDarknet	54.0	91.9	52.5	28.2	66.3	24.2	41.1	47.6
DDQ-DETR ( Zhang et al., 2023a )	RGB	2023	ResNet50	46.7	86.1	45.8	30.9	64.9	24.5	38.8	244.6
FasterRCNN ( Ren et al., 2015 )	IR	2015	ResNet50	54.5	94.6	57.6	37.6	75.8	31.6	46.1	41.1

Table 1. Comparison with state-of-the-art methods on the LLVIP and FLIR datasets. Results are reported in mAP, [email protected], and [email protected]. AVG denotes the average of the overall accuracy on LLVIP and FLIR. Best results are in bold.

Method	Modality	Year	Backbone	Per-class AP					[email protected] $\uparrow$
Method	Modality	Year	Backbone	Car $\uparrow$	Freight-Car $\uparrow$	Truck $\uparrow$	Bus $\uparrow$	Van $\uparrow$	[email protected] $\uparrow$
RetinaNet ( Lin et al., 2017 )	RGB	2017	ResNet50	80.2	16.1	29.0	53.5	18.4	39.4
Faster R-CNN ( Ren et al., 2015 )	RGB	2015	ResNet50	88.8	40.9	57.2	78.6	42.0	61.7
ReDet ( Han et al., 2021 )	RGB	2021	ResNet50	80.3	42.7	56.1	80.2	44.4	60.8
RetinaNet ( Lin et al., 2017 )	IR	2017	ResNet50	89.3	40.0	38.2	79.0	32.1	55.7
Faster R-CNN ( Ren et al., 2015 )	IR	2015	ResNet50	89.4	48.3	53.5	87.0	42.6	64.2

Table 2. Comparison with state-of-the-art methods on the DroneVehicle dataset. Results are reported for each category together with [email protected]. Best results are in bold.

Figure 5. Qualitative comparison on seven representative scenes from the FLIR dataset. The examples include daytime perception (Scene-1&2), small target perception in nighttime conditions (Scene-3), occluded cases (Scenes-4&5), and crowded scenes (Scenes-6&7). Our method yields more reliable predictions in challenging multispectral scenes. A woman and a girl in white dresses sit in an open car.

Backbone	Method	Modality	Car $\uparrow$	Bus $\uparrow$	Truck $\uparrow$	mAP 50 $\uparrow$	mAP $\uparrow$
ResNet50	RetinaNet ( Lin et al., 2017 )	IR	79.0	68.0	60.0	57.9	33.8
	Faster-RCNN ( Ren et al., 2015 )	IR	81.3	66.7	63.3	63.3	35.6
	RetinaNet ( Lin et al., 2017 )	RGB	83.2	75.8	69.9	64.7	37.4
	Faster-RCNN ( Ren et al., 2015 )	RGB	87.9	81.3	72.4	74.5	42.3
	TiNet ( Zhang et al., 2023b )	RGB+IR	87.5	83.1	81.6	79.8	49.5
	Ours	RGB+IR	91.0	92.7	87.7	84.3	55.8

Table 3. Comparison of different methods under ResNet50 and CSPDarknet backbones on the M3FD dataset.

Variant	IR Backbone	Dynamic Fusion	Bidirectional Alignment	recall $\uparrow$	mAP 50 $\uparrow$	mAP $\uparrow$
V0				0.6845	0.775	0.383
V1	✓			0.6982	0.789	0.398
V2	✓	✓		0.7663	0.871	0.488
V3 (Full)	✓	✓	✓	0.7785	0.886	0.504

Table 4. Overall ablation study of the proposed framework on the FLIR dataset. Best results are in bold.

Method Variant	AP $\uparrow$	AP 50 $\uparrow$	AP 75 $\uparrow$	FLOPs $\downarrow$
Text as a Semantic Bridge
Vanilla Direct RGB–IR Fusion	41.9	82.2	36.4	13.56 G
Conditional Prompt Fusion	41.5	83.4	31.7	13.56 G
Ours (Semantic Bridge Fusion)	50.6	88.6	45.7	4.05G
Text as a Semantic Anchor
w/o Bidirectional Update	46.4	85.4	41.6	–

Table 5. Analysis of the two roles of text on the FLIR dataset. Best results are in bold.

Figure 6. Perception performance comparison on FLIR. The proposed fusion achieves better target perception than vanilla direct fusion and conditional prompt fusion.

Consensus Support	Discrepancy Support	Recalibration	mAP $\uparrow$	mAP 50 $\uparrow$	mAP 75 $\uparrow$
			41.3	80.2	36.1
✓			42.2	82.5	37.4
	✓		44.5	82.5	38.9
✓	✓		42.6	80.5	37.6
✓	✓	✓	50.4	88.6	45.7

Table 6. Effectiveness of the bi-support modeling on FLIR. “Consensus” and “Discrepancy” denote the two support patterns, respectively. “Recalibration” denotes dynamic recalibration. Best results are in bold.

Figure 7. Visualization of responses of consensus support $M_{cons}$ and discrepancy support $M_{dis}$ under progressively degraded RGB inputs on a typical sample. The numbers denote average activation values within the ground-truth box. A woman and a girl in white dresses sit in an open car.

Figure 8. Population-level trends of the two support patterns across degradation levels. The numbers denote the average activation values within the GT-box or Background. A woman and a girl in white dresses sit in an open car.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Module	Configuration
RGB Backbone	CSPDarknet (P5, $d_{f}=0.33$ , $w_{f}=0.5$ ), output channels [128, 256, 512]
IR Backbone	Proposed IR Backbone (LiteDCTGhostIRBackboneV2), output channels [64, 128, 256],
	$C_{base}=32$ , $r_{freq}=0.5$ , $r_{ghost}=2$ , $r_{low\_freq\_init}=0.25$ , with FPN
Text Backbone	CLIP ViT-B/32 (all parameters frozen)
Semantic-bridge-GuidedDynamicFusion	Dynamic recalibration initialization $\alpha=0.5$ , $\beta=0.5$
Detection Head	YOLOWorldHead, $d_{embed}=512$ , BN head

Appendix

Table 7. Configuration of model modules.

Figure 9. Qualitative comparison on five representative scenes from the LLVIP dataset. A woman and a girl in white dresses sit in an open car.

Figure 10. Qualitative comparison on five representative scenes from the M3FD dataset. A woman and a girl in white dresses sit in an open car.

Panel of Figure 11 — Figure 11. Comparison of different fusion paradigms on the FLIR dataset in terms of computational complexity and detection performance. V-D, C-P, and Our S-B denote Vanilla Direct Fusion, Conditional Prompt Fusion, and our Semantic Bridge Fusion, respectively.

IR Backbone	Params (M)	FLOPs (G)	Latency (ms)	FPS	mAP50	mAP
ResNet50	24.20	34.18	4.69	213.3	0.853	0.453
Our IR backbone	1.05	6.49	3.30	302.7	0.886	0.504

Appendix

Table 8. Efficiency and accuracy comparison of IR backbones on the FLIR dataset (image size

640\times 640

Level	Brightness ( $\gamma$ )	Sharpen	Blur ( $k$ )	Noise ( $\sigma$ )
0	No modification	–	–	–
+1	0.70	–	5	8
+2	0.55	–	9	15
+3	0.40	–	15	22
+4	0.28	–	21	32
+5	0.18	–	29	42

Appendix

Table 9. Parameters for discrete degradation levels. Four representative levels are selected for population-level statistics.

Panel of Figure 12 — Figure 12. Visualization of responses of consensus support $M_{cons}$ and discrepancy support $M_{dis}$ under progressively degraded RGB inputs on typical samples.

Metric	Value
Number of GT boxes	34,130
Mean	0.4112
Median	0.4051
Standard deviation	0.1447
Minimum	0.0181
Maximum	0.9601

Appendix

Table 10. Foreground occupancy statistics within GT boxes on the LLVIP training set.

Figure 13. Population-level trends of consensus and discrepancy support across RGB degradation levels on LLVIP. The bars show the globally normalized mean responses in the GT regions, and the red dashed line denotes the average pedestrian occupancy ratio within GT boxes ( $41.1\%$ ). With increasing RGB degradation, consensus support in the GT region decreases, whereas discrepancy support increases, indicating a progressive migration of discriminative evidence.

Multispectral Object Detection

Multispectral object detection combines information from visible, infrared, and other spectral bands to locate objects in challenging scenes.

See all 43 papers

RegisterBridgeMM: A Register-Centric Framework for RGB-Infrared Object Detection
Aug 5, 2026Zian Wang, Hangchuan Liang, Yuehua Chen +4Multi-Source Image Fusion Infrared Object Detection
Jilin University, China · Taiyuan University of Technology, China · Shenzhen University, China
EGM-Det: Entropy-Guided Multimodal Adaptive Fusion for UAV RGB-IR Object Detection
Aug 12, 2026Cunzheng Fan, Dawei Yan, Guanlin Wang +4Infrared Object Detection Multi-Scale Feature Fusion
School of Cybersecurity, Northwestern Polytechnical University, Xi’an 710072, China · School of Automation and Software Engineering, Shanxi University, Taiyuan 030006, China
FusionRS: A Large-Scale RGB-Infrared Remote Sensing Dataset for Dual-Modal Vision-Language Foundation Models
Jun 15, 2026Jiaju Han, Ben Zhang, Xuemeng Sun +6Remote Sensing Image Understanding Remote Sensing Image Captioning
China University of Petroleum-Beijing at Karamay · University of Electronic Science and Technology of China · Tianjin University

Aug 5, 2026J/K move · Enter open · S save

RegisterBridgeMM: A Register-Centric Framework for RGB-Infrared Object Detection

Zian Wang, Hangchuan Liang, Yuehua Chen, Changchun Li+3

Jilin University, China · Taiyuan University of Technology, China · Shenzhen University, China

Multi-Source Image Fusion Infrared Object Detection Cross-Modal Attention Multimodal Fusion

Open paper AskPDF arXiv

RGB-infrared (RGB-IR) object detection benefits from complementary visible and thermal cues, but effective fusion remains challenging under illumination changes, weather variation, and cluttered scenes. Existing RGB-IR fusion methods often trade expressive patch-level interaction for lighter but more constrained adaptation mechanisms. We empirically observe that pretrained register tokens contain both modality-shared and modality-specific information on paired RGB-IR inputs, suggesting that they can serve as a compact substrate for cross-modal communication. Building on this observation, we propose RegisterBridgeMM, a register-mediated fusion framework organized as a three-stage register lifecycle. Aggregate preserves per-modality register summarization inherited from pretraining; Bridge performs bidirectional register-to-patch reading with consensus-residual regulation; and Project translates the resulting register summary into spatially adaptive calibration of patch features. This register pathway avoids dense patch-to-patch cross-modal interaction while preserving the pretrained patch representation. With both backbone streams frozen, RegisterBridgeMM achieves the highest mAP50-95 among the evaluated methods on all four benchmarks: LLVIP, M3FD, DroneVehicle, and FLIR-Aligned.

Abstract

Multispectral Object Detection

Explore similar work

RegisterBridgeMM: A Register-Centric Framework for RGB-Infrared Object Detection

EGM-Det: Entropy-Guided Multimodal Adaptive Fusion for UAV RGB-IR Object Detection

FusionRS: A Large-Scale RGB-Infrared Remote Sensing Dataset for Dual-Modal Vision-Language Foundation Models