TriDifNet: A Unified Three-Task Diffusion Network for End-to-End RGB-T Perception

Qianchao He; Jing Liu; Yunhao Li; Xiaoqing Tian

cs.CVOct 27, 2026

TriDifNet: A Unified Three-Task Diffusion Network for End-to-End RGB-T Perception

Authors: Qianchao He, Jing Liu, Yunhao Li, Xiaoqing Tian

Organizations: School of Automation Science and Engineering, Faculty of Electronic and Information Engineering, Xi’an Jiaotong University, Xi’an 710049, China · School of Electrical and Control Engineering, Xi’an University of Science and Technology, Xi’an 710054, China

Abstract

Infrared--visible (IR--VIS) multi-modal perception supports robust scene understanding under challenging illumination. Infrared and visible images are often misaligned due to platform motion, viewpoint changes, and modality-dependent distortions, causing registration errors to propagate into subsequent fusion and detection stages. We propose TriDifNet, a unified three-task diffusion network for end-to-end RGB-T perception. To the best of our knowledge, TriDifNet is the first diffusion-based network that jointly optimizes registration, fusion, and detection for misaligned RGB-T pedestrian perception. TriDifNet consists of three diffusion-based modules, i.e., the Registration Diffusion Module (RegDM), the Fusion Diffusion Module (FusDM), and the Detection Diffusion Module (DetDM). RegDM aligns the unaligned infrared image to the optical reference by estimating a dense deformation field. FusDM performs residual conditional diffusion to generate a detection-oriented fused representation with optical structural cues and complementary infrared saliency. DetDM formulates pedestrian detection as an iterative denoising process in bounding-box space. To couple the three tasks, TriDifNet introduces a closed-loop optimization mechanism, where detection semantics guide fusion and registration, and the fusion result further regularizes geometric alignment. A stage-wise progressive training strategy and a lightweight Nash multi-task learning (Nash-MTL)-based late-stage refinement are further adopted to balance registration, fusion, and detection optimization. Extensive experiments on the KAIST multispectral pedestrian benchmark show that TriDifNet achieves superior downstream detection performance under misaligned RGB-T settings. Against strong system-level competitors, TriDifNet achieves 65.44% Recall, 57.55% mAP $_{50}$ , and 26.83% mAP $_{50:95}$ .

Figures & tables

Fig. 1: Overall architecture of TriDifNet. (a) Task overview. Given a visible image $Y$ and an unaligned infrared image $Z^{\prime}$ , TriDifNet sequentially produces the registered infrared image $\hat{Z}$ , the fused image $\hat{X}$ , and the final detection result $\hat{D}_{3}$ . (b) Closed-loop tri-task diffusion network. RegDM estimates a dense deformation field through conditional deformation-field diffusion and spatial warping; FusDM generates a detection-oriented fused representation through residual conditional diffusion; and DetDM performs box-space conditional diffusion followed by ROI-guided re-inference. Black arrows denote the forward data flow, while red dashed arrows denote cross-task feedback paths, including Det2Reg, Fus2Reg, and Det2Fus. The frozen infrared-domain detector provides semantic guidance during training and is not updated with the TriDifNet parameters. (c) Symbol description.

Fig. 2: ROI-guided re-inference in DetDM. First-pass predictions are converted into an ROI prior, which guides ROI-IR injection for second-pass detection.

Fig. 3: TDFusion-style loss-side guidance branch in Det2Fus. It predicts task-driven spatial weights for ROI-aware fusion constraints during training.

Evaluation Setting	Input	Output	Primary Goal
Registration comparison	Misaligned RGB-T	Registered IR	Image-level registration accuracy
Fusion comparison	Aligned RGB-T	Fusion result	Image-level fusion effectiveness
End-to-end comparison	Misaligned RGB-T	Final detection	Full-system perception performance

TABLE I: Comparison Settings For Different Evaluation Settings

Category	ID	Description	Selected
Registration	R1	No-registration baseline	NoReg
Registration	R2	Template-matching-based visible–thermal registration baseline	TWMM (2022) [ 31 ]
Registration	R3	Plug-and-play general multi-modal image registration baseline	PGMR (2025) [ 32 ]
Fusion	F1	Detection-driven fusion baseline	DetFusion (2022) [ 10 ]
Fusion	F2	Meta-feature embedding guided infrared–visible fusion baseline	MetaFusion (2023) [ 11 ]
Fusion	F3	Registration–fusion integrated infrared–visible fusion baseline	MulFS-CAP (2025) [ 3 ]

TABLE II: Baselines Used In The Experiments

Method	RMSE $\downarrow$	NCC/CC $\uparrow$	SSIM $\uparrow$
R1	10.976204	0.831235	0.854959
R2	45.652067	0.591024	0.543002
R3	10.457858	0.835497	0.853929
Proposed RegDM	10.425901	0.832831	0.856359

TABLE III: Quantitative Comparison Of Registration Results

Panel of Figure 7 — Fig. 4: Qualitative comparison of registration results corresponding to Table III , including absolute-error heatmaps and checkerboard alignment visualizations.

Method	CC $\uparrow$	UIQI $\uparrow$
F1	0.900771	0.808711
F2	0.903327	0.864656
F3	0.942980	0.862271
Proposed FusDM	0.926469	0.840299

TABLE IV: Quantitative Comparison Of Fusion Results

Panel of Figure 9 — Fig. 5: Qualitative comparison of fusion results corresponding to Table IV . TriDifNet emphasizes target-aware infrared saliency while maintaining visible structural cues for downstream detection.

Method	P(%) $\uparrow$	R(%) $\uparrow$	mAP 50 (%) $\uparrow$	mAP 50:95 (%) $\uparrow$
S1	81.53	39.49	36.02	16.45
S2	79.39	44.80	40.33	20.88
S3	81.14	9.28	8.32	3.12
S4	80.22	25.22	22.90	10.15
S5	69.53	37.71	32.65	14.66
S6	41.62	52.84	38.51	18.76

TABLE V: End-To-End Pedestrian Detection Comparison

Method		P(%) $\uparrow$	R(%) $\uparrow$	mAP 50 (%) $\uparrow$	mAP 50:95 (%) $\uparrow$	FPS $\uparrow$
RegDM	FusDM	P(%) $\uparrow$	R(%) $\uparrow$	mAP 50 (%) $\uparrow$	mAP 50:95 (%) $\uparrow$	FPS $\uparrow$
–	–	48.77	49.57	37.90	13.71	11.02
–	✓	70.81	57.34	44.67	19.13	0.32
✓	–	51.03	49.46	39.04	14.07	1.03
✓	✓	73.85	65.44	57.55	26.83	0.25

TABLE VI: System-Level Ablation Study

Index	Det2Reg	Fus2Reg	Det2Fus	ROI-IR Injection	Nash-MTL	Registration			Fusion		Detection
Index	Det2Reg	Fus2Reg	Det2Fus	ROI-IR Injection	Nash-MTL	RMSE $\downarrow$	NCC/CC $\uparrow$	SSIM $\uparrow$	CC $\uparrow$	UIQI $\uparrow$	P(%) $\uparrow$	R(%) $\uparrow$	mAP 50 (%) $\uparrow$	mAP 50:95 (%) $\uparrow$
1	–	✓	✓	✓	✓	10.8720	0.8215	0.8461	0.9258	0.8447	71.30	62.77	54.11	24.73
2	✓	–	✓	✓	✓	11.0282	0.8176	0.8426	0.9263	0.8427	73.80	61.48	55.59	26.75
3	✓	✓	–	✓	✓	10.9231	0.8188	0.8448	0.9665	0.9492	55.17	52.17	42.03	18.70
4	✓	✓	–	–	✓	11.0814	0.8181	0.8424	0.9640	0.9406	54.92	47.01	36.60	14.54
5	✓	✓	✓	✓	–	10.9469	0.8196	0.8443	0.9312	0.8508	71.56	63.59	55.32	25.52

TABLE VII: Ablation Study On Cross-Task Feedback

Object Detection

Object detection locates and classifies objects in images or video, balancing localization accuracy, robustness, and inference speed.

See all 624 papers

Bridging Severe Cross-Modal Misalignment: End-to-End Visible-Infrared Object Detection via Explicit Feature-Domain Affine Registration
Aug 11, 2026Qi Ming, Yuyang Wang, Mingjing Zhao +7Multispectral Object Detection Cross-Modal Alignment
Beijing University of Technology, China · Central South University, China · Beijing Electronics Science & Technology Institute +4
BeyondFusion: Self-Aligned Latent Diffusion for Calibration-Free Infrared Super-Resolution and Infrared-Visible Fusion
Jul 27, 2026Minchong Chen, Xiaoyun Yuan, Minyu Cao +4Diffusion Models for Image Restoration Multi-Source Image Fusion
MoE Key Lab of Artificial Intelligence, Institute of AI, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Tsinghua University, Beijing, China · Institute of Modern Optics, Nankai University, Tianjin, China
RegisterBridgeMM: A Register-Centric Framework for RGB-Infrared Object Detection
Aug 5, 2026Zian Wang, Hangchuan Liang, Yuehua Chen +4Multi-Source Image Fusion Infrared Object Detection
Jilin University, China · Taiyuan University of Technology, China · Shenzhen University, China

Aug 11, 2026J/K move · Enter open · S save

Bridging Severe Cross-Modal Misalignment: End-to-End Visible-Infrared Object Detection via Explicit Feature-Domain Affine Registration

Qi Ming, Yuyang Wang, Mingjing Zhao, Yifan Xiao+6

Beijing University of Technology, China · Central South University, China · Beijing Electronics Science & Technology Institute+4

Multispectral Object Detection Cross-Modal Alignment Infrared Object Detection Cross-Modal Image Registration Multimodal Sensor Fusion

Open paper AskPDF arXiv

Visible-infrared object detection relies on complementary RGB and thermal cues, but its performance is often degraded by cross-modal spatial misalignment. Most existing methods rely on implicit feature adaptation to handle weakly misaligned scenarios, while large-offset geometric discrepancies remain insufficiently addressed. In this paper, we propose a Joint Feature-domain Registration and Detection network (JFRDet), an end-to-end visible-infrared oriented object detector tailored for severely cross-modal geometric discrepancies. JFRDet introduces a Cross-Modal Affine Alignment (CMAA) module to estimate an image-level affine transformation for explicit multi-level feature alignment. Note that illumination changes directly affect the reliability of RGB cues, an Illumination-Guided Complementary Fusion (IGCF) module adaptively exploits modality reliability under varying illumination conditions for cross-modal fusion. Then, an Alignment Quality-Consistency Gating (AQCG) strategy stabilizes joint optimization by modulating detection supervision according to alignment reliability and gradient consistency. We further construct DroneVehicle Misaligned (DVMA), a benchmark for evaluating visible-infrared oriented object detection under severe cross-modal geometric misalignment. The proposed JFRDet achieves 69.7%

\mathrm{mAP}_{50}

on DVMA, which represents state-of-the-art (SOTA) performance. The code and dataset will be available on GitHub.

Abstract

Object Detection

Explore similar work

Bridging Severe Cross-Modal Misalignment: End-to-End Visible-Infrared Object Detection via Explicit Feature-Domain Affine Registration

BeyondFusion: Self-Aligned Latent Diffusion for Calibration-Free Infrared Super-Resolution and Infrared-Visible Fusion

RegisterBridgeMM: A Register-Centric Framework for RGB-Infrared Object Detection