TriDifNet: A Unified Three-Task Diffusion Network for End-to-End RGB-T Perception

Qianchao He; Jing Liu; Yunhao Li; Xiaoqing Tian

cs.CVOct 27, 2026

TriDifNet: A Unified Three-Task Diffusion Network for End-to-End RGB-T Perception

Authors: Qianchao He, Jing Liu, Yunhao Li, Xiaoqing Tian

Organizations: School of Automation Science and Engineering, Faculty of Electronic and Information Engineering, Xi’an Jiaotong University, Xi’an 710049, China · School of Electrical and Control Engineering, Xi’an University of Science and Technology, Xi’an 710054, China

Abstract

Infrared--visible (IR--VIS) multi-modal perception supports robust scene understanding under challenging illumination. Infrared and visible images are often misaligned due to platform motion, viewpoint changes, and modality-dependent distortions, causing registration errors to propagate into subsequent fusion and detection stages. We propose TriDifNet, a unified three-task diffusion network for end-to-end RGB-T perception. To the best of our knowledge, TriDifNet is the first diffusion-based network that jointly optimizes registration, fusion, and detection for misaligned RGB-T pedestrian perception. TriDifNet consists of three diffusion-based modules, i.e., the Registration Diffusion Module (RegDM), the Fusion Diffusion Module (FusDM), and the Detection Diffusion Module (DetDM). RegDM aligns the unaligned infrared image to the optical reference by estimating a dense deformation field. FusDM performs residual conditional diffusion to generate a detection-oriented fused representation with optical structural cues and complementary infrared saliency. DetDM formulates pedestrian detection as an iterative denoising process in bounding-box space. To couple the three tasks, TriDifNet introduces a closed-loop optimization mechanism, where detection semantics guide fusion and registration, and the fusion result further regularizes geometric alignment. A stage-wise progressive training strategy and a lightweight Nash multi-task learning (Nash-MTL)-based late-stage refinement are further adopted to balance registration, fusion, and detection optimization. Extensive experiments on the KAIST multispectral pedestrian benchmark show that TriDifNet achieves superior downstream detection performance under misaligned RGB-T settings. Against strong system-level competitors, TriDifNet achieves 65.44% Recall, 57.55% mAP $_{50}$ , and 26.83% mAP $_{50:95}$ .

Figures & tables

Fig. 1: Overall architecture of TriDifNet. (a) Task overview. Given a visible image $Y$ and an unaligned infrared image $Z^{\prime}$ , TriDifNet sequentially produces the registered infrared image $\hat{Z}$ , the fused image $\hat{X}$ , and the final detection result $\hat{D}_{3}$ . (b) Closed-loop tri-task diffusion network. RegDM estimates a dense deformation field through conditional deformation-field diffusion and spatial warping; FusDM generates a detection-oriented fused representation through residual conditional diffusion; and DetDM performs box-space conditional diffusion followed by ROI-guided re-inference. Black arrows denote the forward data flow, while red dashed arrows denote cross-task feedback paths, including Det2Reg, Fus2Reg, and Det2Fus. The frozen infrared-domain detector provides semantic guidance during training and is not updated with the TriDifNet parameters. (c) Symbol description.

Fig. 2: ROI-guided re-inference in DetDM. First-pass predictions are converted into an ROI prior, which guides ROI-IR injection for second-pass detection.

Fig. 3: TDFusion-style loss-side guidance branch in Det2Fus. It predicts task-driven spatial weights for ROI-aware fusion constraints during training.

Evaluation Setting	Input	Output	Primary Goal
Registration comparison	Misaligned RGB-T	Registered IR	Image-level registration accuracy
Fusion comparison	Aligned RGB-T	Fusion result	Image-level fusion effectiveness
End-to-end comparison	Misaligned RGB-T	Final detection	Full-system perception performance

TABLE I: Comparison Settings For Different Evaluation Settings

Category	ID	Description	Selected
Registration	R1	No-registration baseline	NoReg
Registration	R2	Template-matching-based visible–thermal registration baseline	TWMM (2022) [ 31 ]
Registration	R3	Plug-and-play general multi-modal image registration baseline	PGMR (2025) [ 32 ]
Fusion	F1	Detection-driven fusion baseline	DetFusion (2022) [ 10 ]
Fusion	F2	Meta-feature embedding guided infrared–visible fusion baseline	MetaFusion (2023) [ 11 ]
Fusion	F3	Registration–fusion integrated infrared–visible fusion baseline	MulFS-CAP (2025) [ 3 ]

TABLE II: Baselines Used In The Experiments

Method	RMSE $\downarrow$	NCC/CC $\uparrow$	SSIM $\uparrow$
R1	10.976204	0.831235	0.854959
R2	45.652067	0.591024	0.543002
R3	10.457858	0.835497	0.853929
Proposed RegDM	10.425901	0.832831	0.856359

TABLE III: Quantitative Comparison Of Registration Results

Panel of Figure 7 — Fig. 4: Qualitative comparison of registration results corresponding to Table III , including absolute-error heatmaps and checkerboard alignment visualizations.

Method	CC $\uparrow$	UIQI $\uparrow$
F1	0.900771	0.808711
F2	0.903327	0.864656
F3	0.942980	0.862271
Proposed FusDM	0.926469	0.840299

TABLE IV: Quantitative Comparison Of Fusion Results

Panel of Figure 9 — Fig. 5: Qualitative comparison of fusion results corresponding to Table IV . TriDifNet emphasizes target-aware infrared saliency while maintaining visible structural cues for downstream detection.

Method	P(%) $\uparrow$	R(%) $\uparrow$	mAP 50 (%) $\uparrow$	mAP 50:95 (%) $\uparrow$
S1	81.53	39.49	36.02	16.45
S2	79.39	44.80	40.33	20.88
S3	81.14	9.28	8.32	3.12
S4	80.22	25.22	22.90	10.15
S5	69.53	37.71	32.65	14.66
S6	41.62	52.84	38.51	18.76

TABLE V: End-To-End Pedestrian Detection Comparison

Method		P(%) $\uparrow$	R(%) $\uparrow$	mAP 50 (%) $\uparrow$	mAP 50:95 (%) $\uparrow$	FPS $\uparrow$
RegDM	FusDM	P(%) $\uparrow$	R(%) $\uparrow$	mAP 50 (%) $\uparrow$	mAP 50:95 (%) $\uparrow$	FPS $\uparrow$
–	–	48.77	49.57	37.90	13.71	11.02
–	✓	70.81	57.34	44.67	19.13	0.32
✓	–	51.03	49.46	39.04	14.07	1.03
✓	✓	73.85	65.44	57.55	26.83	0.25

TABLE VI: System-Level Ablation Study

Index	Det2Reg	Fus2Reg	Det2Fus	ROI-IR Injection	Nash-MTL	Registration			Fusion		Detection
Index	Det2Reg	Fus2Reg	Det2Fus	ROI-IR Injection	Nash-MTL	RMSE $\downarrow$	NCC/CC $\uparrow$	SSIM $\uparrow$	CC $\uparrow$	UIQI $\uparrow$	P(%) $\uparrow$	R(%) $\uparrow$	mAP 50 (%) $\uparrow$	mAP 50:95 (%) $\uparrow$
1	–	✓	✓	✓	✓	10.8720	0.8215	0.8461	0.9258	0.8447	71.30	62.77	54.11	24.73
2	✓	–	✓	✓	✓	11.0282	0.8176	0.8426	0.9263	0.8427	73.80	61.48	55.59	26.75
3	✓	✓	–	✓	✓	10.9231	0.8188	0.8448	0.9665	0.9492	55.17	52.17	42.03	18.70
4	✓	✓	–	–	✓	11.0814	0.8181	0.8424	0.9640	0.9406	54.92	47.01	36.60	14.54
5	✓	✓	✓	✓	–	10.9469	0.8196	0.8443	0.9312	0.8508	71.56	63.59	55.32	25.52

TABLE VII: Ablation Study On Cross-Task Feedback

Object Detection

Object detection locates and classifies objects in images or video, balancing localization accuracy, robustness, and inference speed.

See all 624 papers

Aug 11, 2026

Bridging Severe Cross-Modal Misalignment: End-to-End Visible-Infrared Object Detection via Explicit Feature-Domain Affine Registration

Visible-infrared object detection relies on complementary RGB and thermal cues, but its performance is often degraded by cross-modal spatial misalignment. Most existing methods rely on implicit feature adaptation to handle weakly misaligned scenarios, while large-offset geometric discrepancies remain insufficiently addressed. In this paper, we propose a Joint Feature-domain Registration and Detection network (JFRDet), an end-to-end visible-infrared oriented object detector tailored for severely cross-modal geometric discrepancies. JFRDet introduces a Cross-Modal Affine Alignment (CMAA) module to estimate an image-level affine transformation for explicit multi-level feature alignment. Note that illumination changes directly affect the reliability of RGB cues, an Illumination-Guided Complementary Fusion (IGCF) module adaptively exploits modality reliability under varying illumination conditions for cross-modal fusion. Then, an Alignment Quality-Consistency Gating (AQCG) strategy stabilizes joint optimization by modulating detection supervision according to alignment reliability and gradient consistency. We further construct DroneVehicle Misaligned (DVMA), a benchmark for evaluating visible-infrared oriented object detection under severe cross-modal geometric misalignment. The proposed JFRDet achieves 69.7%

\mathrm{mAP}_{50}

on DVMA, which represents state-of-the-art (SOTA) performance. The code and dataset will be available on GitHub.

Qi Ming, Yuyang Wang, Mingjing Zhao +7

Beijing University of Technology, China · Central South University, China · Beijing Electronics Science & Technology Institute +4

Multispectral Object Detection Cross-Modal Alignment Infrared Object Detection Cross-Modal Image Registration Multimodal Sensor Fusion

Ask PDF

Jul 27, 2026

BeyondFusion: Self-Aligned Latent Diffusion for Calibration-Free Infrared Super-Resolution and Infrared-Visible Fusion

Mobile infrared-visible imaging typically pairs a compact infrared sensor with a high-resolution visible camera for complementary perception. While cross-sensor misalignment caused by different optics, viewpoints, fields of view, and exposure timings hinders practical deployment. In this paper, we propose BeyondFusion, a unified latent diffusion framework for calibration-free visible-guided infrared super-resolution and infrared-visible fusion tasks. The proposed framework supports both task-specific training and joint training where two tasks are optimized and executed as two readouts of the same generative process. Instead of relying on explicit registration or geometric warping, BeyondFusion introduces a cross-modal self-aligning (CMSA) module into the denoising U-Net. CMSA reorganizes infrared and visible latent tokens into a shared attention space to learn content-adaptive cross-modal correspondence during the denoising process. Together with misalignment augmentation module, the model is facilitated to exploit visible structural and semantic cues while preserving thermal consistency, enabling high-frequency infrared reconstruction and informative fused-image generation under uncalibrated conditions. Extensive experiments on public benchmarks and a mobile infrared-visible imaging system show strong performance across aligned inputs, low-resolution infrared observations, synthetic misalignments, and real mobile captures with unsynchronized sensors. Ablation studies, unified training analysis, and downstream pedestrian detection further validate the effectiveness of BeyondFusion for calibration-free multimodal imaging.

Minchong Chen, Xiaoyun Yuan, Minyu Cao +4

MoE Key Lab of Artificial Intelligence, Institute of AI, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Tsinghua University, Beijing, China · Institute of Modern Optics, Nankai University, Tianjin, China

Diffusion Models for Image Restoration Multi-Source Image Fusion Infrared-Visible Image Fusion Cross-Modal Alignment Remote Sensing Infrared Image Super-Resolution Image Super-Resolution

Ask PDF

Aug 5, 2026

RegisterBridgeMM: A Register-Centric Framework for RGB-Infrared Object Detection

RGB-infrared (RGB-IR) object detection benefits from complementary visible and thermal cues, but effective fusion remains challenging under illumination changes, weather variation, and cluttered scenes. Existing RGB-IR fusion methods often trade expressive patch-level interaction for lighter but more constrained adaptation mechanisms. We empirically observe that pretrained register tokens contain both modality-shared and modality-specific information on paired RGB-IR inputs, suggesting that they can serve as a compact substrate for cross-modal communication. Building on this observation, we propose RegisterBridgeMM, a register-mediated fusion framework organized as a three-stage register lifecycle. Aggregate preserves per-modality register summarization inherited from pretraining; Bridge performs bidirectional register-to-patch reading with consensus-residual regulation; and Project translates the resulting register summary into spatially adaptive calibration of patch features. This register pathway avoids dense patch-to-patch cross-modal interaction while preserving the pretrained patch representation. With both backbone streams frozen, RegisterBridgeMM achieves the highest mAP50-95 among the evaluated methods on all four benchmarks: LLVIP, M3FD, DroneVehicle, and FLIR-Aligned.

Zian Wang, Hangchuan Liang, Yuehua Chen +4

Jilin University, China · Taiyuan University of Technology, China · Shenzhen University, China

Multi-Source Image Fusion Infrared Object Detection Cross-Modal Attention Multimodal Fusion

Ask PDF

Abstract

Object Detection

Explore similar work

Bridging Severe Cross-Modal Misalignment: End-to-End Visible-Infrared Object Detection via Explicit Feature-Domain Affine Registration

BeyondFusion: Self-Aligned Latent Diffusion for Calibration-Free Infrared Super-Resolution and Infrared-Visible Fusion

RegisterBridgeMM: A Register-Centric Framework for RGB-Infrared Object Detection