TriNetra: Thermal-Reliability-Guided RGB-Thermal Fusion and Tracking for Surveillance

Sadique Khatib; Aditya Ahirrao; Vedant Sonawane; Disha Kapse; Y. D. Bhise

cs.CVNov 6, 2026

TriNetra: Thermal-Reliability-Guided RGB-Thermal Fusion and Tracking for Surveillance

Authors: Sadique Khatib, Aditya Ahirrao, Vedant Sonawane, Disha Kapse, Y. D. Bhise

Organizations: Dept. of Computer Science and Design, K. K. Wagh Institute of Engineering Education and Research, Nashik 422003, India

Abstract

Visible cameras fail at night and thermal sensors fail under saturation, blur or loss of contrast, yet most RGB-thermal (RGB-T) detectors mix the two streams with fixed weights and cannot tell when one modality has become unreliable. TriNetra is a reliability-aware surveillance framework built around this problem. The proposed solution computes a frame-level Thermal Reliability Coefficient (TRC) from five label-free image cues, uses it to gate the thermal features of a dual-branch YOLOv8m detector whose visible branch, neck and head are frozen and COCO-pretrained, and passes the same score to a ByteTrack-based tracker that feeds a rule-based behaviour analyser. On the M3FD benchmark (2,940/630/630 split) the detector reaches 87.6% precision, 79.4% recall, 70.0% mAP@50 and 47.8% mAP@50:95 on the test split, with 17.0 M of its 42.9 M parameters trained. TRC stays near 0.78 on clean frames (standard deviation 0.012), decreases with the severity of thermal saturation and blur, reaching 0.53 and 0.69 at full severity, and is insensitive to RGB-thermal misalignment. With identical weights, the adaptive gate matches a fixed gate on clean data (70.0% versus 69.8% mAP@50) but keeps 43.6% mAP@50 when the thermal stream is corrupted, against 38.4%, 29.2% and 25.0% for a fixed gate, full thermal admission and a closed gate; the 5.1-point gain over the fixed gate has a 95% bootstrap confidence interval of +3.6 to +6.5 points. In tracking, the TRC-gated association performs on par with ByteTrack on a synthetic benchmark with exact ground truth and on 14 M3FD clips, and is slightly less stable under thermal saturation, so its measured benefit is confined to the detector. The limitations of the current evaluation are reported together with the results.

Figures & tables

Fig. 1: Overall TriNetra pipeline. The reliability score $r_{T}$ (green) drives both the gated fusion inside the detector and the association gate inside the tracker. Numbers give the processing order; the example frames are an M3FD test pair (detections) and a night clip (tracks).

Fig. 2: Computation of the Thermal Reliability Coefficient. The thermal frame passes through alignment, denoising and normalisation; five cues are computed per frame and aggregated into $r_{T}$ . The cue maps (histogram, Laplacian, clipped pixels, joint histogram) and scores are computed from one M3FD test pair. The RGB frame is needed only by the mutual-information cue.

Fig. 3: TRC-gated dual-branch fusion detector. (a) The visible branch (blue), the PAN neck and the head are frozen; the thermal branch (orange) and the gated fusion blocks (yellow) are trained. Slab height indicates spatial size and thickness the channel count. (b) The gated fusion block applied at each of the three scales.

Fig. 4: TRC-gated tracking. Stage 2 (green) is the only stage that uses the frame reliability $r_{T}$ : its cost is multiplied by $m=1+\beta(1-r_{T})$ . Grids illustrate the cost matrices. The lower part shows the track lifecycle.

Fig. 5: Experimental workflow. Detection is evaluated on the M3FD test split, reliability with controlled thermal degradations and a gate ablation, and tracking on synthetic sequences with exact ground truth and on the assembled clips without track labels.

Data	Pairs	Classes	Size	Input	Split (pairs)
M3FD	4,200	6	$1024\times 768$	$640\times 640$	2,940/630/630
M3FD clips	14	none	$1024\times 768$	—	—

TABLE I: Statistics of the Data Used. Boxes Are Listed as Train/Val/Test.

Fig. 6: M3FD statistics for the split used here. (a) Boxes per class (log scale). (b) Share of boxes in the three COCO size buckets, computed from the box area in native pixels.

Item	Value
CPU	AMD Ryzen 7 5800H (8C/16T)
RAM	15.3 GB
GPU	RTX 3060 Laptop, 6 GB
Operating system	Windows 11
Python / PyTorch [ 24 ]	3.11.16 / 2.3.1 (CUDA 12.1)
Ultralytics [ 8 ]	8.4.153

TABLE II: Experimental Environment.

Parameter	Value
Epochs	150 planned; early stop at 131 (patience 15)
Selected checkpoint	epoch 116 (best validation mAP@50)
Batch size	$8\times 2$ accumulation = 16 (effective)
Input size	$640\times 640$
Optimiser	SGD (momentum 0.937, weight decay $5\times 10^{-4}$ )
Learning rate	0.01, cosine [ 25 ] to $10^{-4}$ , no warm-up

TABLE III: Training Settings of the Detector.

Fig. 7: Training history of the selected run. Shaded regions mark phase I (epochs 0–34, only the person class supervised), phase II (35–41, five classes) and phase III (42–131, six classes); the validation mAP changes definition at the phase boundaries. The star marks the selected epoch and the dashed line the early stop.

Class	P (%)	R (%)	mAP@50 [95% CI]	mAP@50:95
All	87.6	79.4	70.0 [66.7, 73.4]	47.8
Person	85.2	75.3	72.9 [69.9, 75.8]	44.3
Car	90.0	85.0	84.3 [82.3, 85.5]	62.0
Bus	85.9	84.2	82.5 [74.5, 89.1]	70.4
Motorcycle	78.8	58.6	55.7 [44.4, 67.4]	31.4
Lamp	84.6	62.8	60.1 [54.5, 66.3]	30.6

TABLE IV: Detection Results of TriNetra on the M3FD Test Split (630 Pairs, One Run). The 95% Confidence Intervals Resample the Test Images (1,000 Bootstrap Resamples). Efficiency Was Measured With Batch Size 1 on the RTX 3060 Laptop GPU.

Fig. 8: Per-class test AP of TriNetra on M3FD.

Method	P	R	mAP@50	mAP@50:95	Params	GFLOPs
YOLOv8n (DS)	82.0	62.8	70.3	44.7	4.36	11.34
YOLOv11n (DS)	82.6	57.5	66.5	42.6	3.79	9.31
MBNet	80.0	61.2	68.2	43.0	6.8	16.2
ProbEn	80.8	61.7	68.8	43.3	7.5	17.4
RT-DETR (DS)	79.2	60.5	67.2	42.2	66.34	194.03
GEM-YOLO [ 6 ]	79.2	61.8	69.0	43.8	3.44	7.58

TABLE V: Indicative Comparison on M3FD (P, R, mAP in %; Parameters in Millions). Values for Other Methods Are Copied From [ 6 ] (Mean of Five Runs, 7:2:1 Split, Trained From Scratch); TriNetra Is One Run on a 70/15/15 Split. DS: Dual-Stream. Precision and Recall Are Not Directly Comparable Across Studies.

Score	Mean	Std
Contrast $S_{C}$	0.938	0.023
Gradient $S_{G}$	0.873	0.046
Saturation $S_{S}$	0.987	0.003
Mutual information $S_{MI}$	0.115	0.059
Alignment $S_{A}$	1.000	0.000
TRC $r_{T}$	0.782	0.012

TABLE VI: Cue Scores and TRC on the 630 Clean M3FD Test Pairs (Mean

\pm

Standard Deviation).

Degradation	0.25	0.5	0.75	1.0	$\rho$	Mono.	$<$ 0.5
	$\delta$
Blur	.729	.710	.698	.690	$-1.00$	100%	0%
Saturation	.705	.658	.579	.532	$-1.00$	96%	30%
Flat	.780	.778	.776	.405	$-0.74$	55%	100%
Misalignment	.775	.773	.772	.771	$-0.93$	84%	0%

TABLE VII: Mean TRC Over 150 Test Pairs (Clean Value 0.782) at Severity

\delta

; the Standard Deviation Is 0.009–0.012 Except for Saturation (0.021–0.067, See Fig. 9 b).

\rho

: Mean Per-Frame Spearman Correlation Between

\delta

and TRC; Mono.: Share of Frames Whose TRC Never Increases With

\delta

;

<

0.5: Share of Frames Below 0.5 at

\delta=1

Fig. 9: TRC analysis on M3FD. (a) Distribution over the 630 clean test pairs. (b) TRC against degradation severity (mean

\pm

one standard deviation over 150 pairs). (c) Drop of each cue score at full severity, showing which cue reacts to which degradation.

Variant	Level	Blur	Saturation	Flat
All five cues	0.782	0.093	0.250	0.377
Without contrast	0.744	0.116	0.214	0.243
Without gradient	0.760	$-0.002$	0.221	0.254
Without saturation	0.731	0.116	0.178	0.475
Without mutual information	0.949	0.117	0.325	0.443
Without alignment	0.728	0.116	0.313	0.472

TABLE VIII: Leave-One-Cue-Out Analysis on 150 Test Pairs. “Level” Is the Mean Clean TRC; the Other Columns Are the Drop of the Mean TRC From Clean to

\delta=1

(Larger Means Better Separation).

Gate mode	Clean	Corrupted	Blur	Saturation	$r_{T}<0.6$
Adaptive TRC (proposed)	70.0 / 47.8	43.6 / 30.6	49.8 / 35.5	36.8 / 25.6	38.3 / 27.2
Fixed $r_{T}=0.782$	69.8 / 47.8	38.4 / 27.3	48.3 / 34.4	28.1 / 20.0	29.7 / 21.6
Full thermal $r_{T}=1$	66.7 / 44.7	29.2 / 20.7	42.8 / 30.1	14.6 / 10.8	17.9 / 13.6
Thermal off $r_{T}=0$	25.0 / 14.4	25.0 / 14.4	24.8 / 14.6	24.9 / 14.5	23.7 / 14.0

TABLE IX: Gate Ablation With Identical Weights on the M3FD Test Split (630 Pairs): mAP@50 / mAP@50:95 in %. Corrupted: Blur or Saturation at Full Severity on the Thermal Stream; the Subsets Are Blur (333 Pairs), Saturation (297) and

r_{T}<0.6

(243).

Gate mode	Clean	Corrupted	Adaptive $-$ mode (corrupted)
Adaptive TRC	70.0 [66.7, 73.4]	43.6 [40.0, 47.2]	—
Fixed $r_{T}=0.782$	69.8 [66.7, 73.3]	38.4 [34.9, 42.2]	+5.1 [+3.6, +6.5]
Full thermal $r_{T}=1$	66.7 [63.2, 70.0]	29.2 [25.8, 32.7]	+14.4 [+12.1, +16.6]
Thermal off $r_{T}=0$	25.0 [22.7, 27.2]	25.0 [22.7, 27.3]	+18.6 [+15.5, +22.2]

TABLE X: Overall mAP@50 (%) of the Four Gate Modes With 95% Bootstrap Confidence Intervals (630 Test Pairs, 1,000 Image Resamples), and the Paired Difference Between the Adaptive Gate and Each Other Mode on Corrupted Thermal Input.

Subset	Pairs	Difference [95% CI]
Clean thermal	630	+0.1 [ $-$ 0.2, +0.2]
Corrupted thermal, all	630	+5.1 [+3.6, +6.5]
blur only	333	+1.5 [+0.9, +2.3]
saturation only	297	+8.8 [+6.1, +12.2]
$r_{T}<0.6$	243	+8.6 [+5.4, +12.1]

TABLE XI: Adaptive Gate Minus Fixed Gate, mAP@50 in Percentage Points, Paired Bootstrap (95% CI), by Subset of the Test Split.

Fig. 10: Gate ablation on the M3FD test split with identical weights. (a) Overall mAP@50 on clean and corrupted thermal input. (b) mAP@50 on the corrupted stream by subset.

Fig. 11: Tracking results. (a) HOTA against degradation severity on the synthetic benchmark; the gated tracker and ByteTrack nearly coincide. (b) HOTA against gate strength

\beta

(

\beta=0

is plain ByteTrack). (c) Persistence (mean track length divided by clip length) of each tracker on the 14 M3FD clips with identical detections (mean and 95% CI over clips).

	Severity 0.5			Severity 1.0
Tracker	HOTA	MOTA	IDSW	HOTA	MOTA	IDSW
SORT-style	.556	.745	12.8	.319	.471	17.8
DeepSORT (motion)	.514	.702	15.9	.359	.456	27.0
ByteTrack	.559	.754	12.4	.326	.480	17.6
ByteTrack + TRC gate	.559	.751	12.5	.323	.476	17.7

TABLE XII: Synthetic Tracking Benchmark (30 Sequences, Mean, Coupled Condition) at Degradation Severity 0.5 and 1.0. The Standard Deviation Over Sequences Is 0.04–0.09 for HOTA and 0.03–0.10 for MOTA.

Tracker	Tracks	Persistence	Tracks/target	Coverage	Jumps
SORT-style	122	$0.58\pm 0.17$	1.92	0.792	1.3
DeepSORT (motion)	122	$0.70\pm 0.16$	1.55	0.742	0.1
ByteTrack	119	$0.63\pm 0.19$	1.79	0.793	1.3
ByteTrack + TRC gate	119	$0.63\pm 0.19$	1.81	0.793	1.3

TABLE XIII: Tracker Comparison on the 14 M3FD Clips (378 Frames) With Identical Detections and Label-Free Proxies. Persistence Is Mean Track Length Divided by Clip Length (Mean

\pm

Std Over Clips); Tracks per Target Is the Number of Tracks Divided by the Mean Number of Tracked Objects per Frame; Jumps Are Implausible Centre Jumps per 1,000 Track-Frames.

Tracker	IDF1 [95% CI]	HOTA	AssA	IDSW
SORT-style	.690 [.517, .839]	.639	.730	0.32
DeepSORT (motion)	.690 [.498, .860]	.640	.709	0.71
ByteTrack	.749 [.562, .904]	.699	.777	0.32
ByteTrack + TRC gate	.734 [.551, .887]	.686	.769	0.32

TABLE XIV: Identity Stability in the Degraded Middle Third of the 14 M3FD Clips (Blur and Saturation Runs Averaged per Clip). Mean Over Clips With 95% Bootstrap CI for IDF1. Each Tracker’s Degraded Tracks Are Scored Against Its Own Clean-Run Tracks.

Fig. 12: Detections on three M3FD test pairs selected by a fixed brightness and object-count rule (not by detection quality): night, day and crowded. Left: RGB with ground truth; centre: thermal input; right: TriNetra detections at confidence 0.25.

Fig. 13: Tracking on the longest night clip of the 14 (42 frames, mean RGB brightness 47; RGB shown). Frames at one quarter, one half and three quarters of the clip are shown. Boxes show track identity and class, and the lines show the trajectory over the preceding frames.

Visible-Thermal Object Detection

See all 10 papers

Sep 11, 2026

RA-SOD: Reliability-Aware RGB-T Salient Object Detection under Modality Degradation

RGB-Thermal (RGB-T) salient object detection leverages complementary cues from visible and thermal modalities to improve robustness in challenging environments. However, in real-world scenarios, the reliability of each modality is inherently unstable: RGB images degrade under low illumination, motion blur, and noise, while thermal imagery often suffers from contrast compression and sensor artifacts. Such degradation introduces unreliable perceptual evidence that can mislead cross-modal fusion and significantly deteriorate detection performance. To address this challenge, we propose RA-SOD, a reliability-aware RGB-T salient object detection framework that explicitly models modality reliability and integrates it into feature learning and cross-modal fusion. First, we introduce a reliability-conditioned representation that adaptively compensates degraded modality features while preserving structural cues. Second, an uncertainty-guided dual-stream refinement strategy progressively corrects cross-modal representations while suppressing unreliable evidence. Finally, we propose a pixel-wise modality competition mechanism that dynamically selects modality cues according to spatial reliability for fine-grained fusion. Extensive experiments on four benchmarks (VT821, VT1000, VT5000, and VT-IMAG) demonstrate that RA-SOD achieves state-of-the-art performance and exhibits strong robustness under severe modality degradation. Code and models are available at https://github.com/zaoxienian/RA-SOD.

Hongbo Gao, Zhengyu Li, Xueru Nie +4

State Key Laboratory of Robotics and Systems, Harbin Institute of Technology, No. 92 West Dazhi Street, Nangang District, Harbin, Heilongjiang 150006, China · School of Computer Science, University of Sydney, Camperdown, Sydney, NSW 2006, Australia

Multimodal Robustness Salient Object Detection Multimodal Fusion

Ask PDF

Apr 17, 2026

Tri-Modal Fusion Transformers for UAV-based Object Detection

Reliable UAV object detection requires robustness to illumination changes, motion blur, and scene dynamics that suppress RGB cues. Thermal long-wave infrared (LWIR) sensing preserves contrast in low light, and event cameras retain microsecond-level temporal edges, but integrating all three modalities in a unified detector has not been systematically studied. We present a tri-modal framework that processes RGB, thermal, and event data with a dual-stream hierarchical vision transformer. At selected encoder depths, a Modality-Aware Gated Exchange (MAGE) applies inter-sensor channel and spatial gating, and a Bidirectional Token Exchange (BiTE) module performs bidirectional token-level attention with depthwise-pointwise refinement, producing resolution-preserving fused maps for a standard feature pyramid and two-stage detector. We introduce a 10,489-frame UAV dataset with synchronized and pre-aligned RGB-thermal-event streams and 24,223 annotated vehicles across day and night flights. Through 61 controlled ablations, we evaluate fusion placement, mechanism (baseline MAGE+BiTE, CSSA, GAFF), modality subsets, and backbone capacity. Tri-modal fusion improves over all dual-modal baselines, with fusion depth having a significant effect and a lightweight CSSA variant recovering most of the benefit at minimal cost. This work provides the first systematic benchmark and modular backbone for tri-modal UAV-based object detection.

Craig Iaboni, Pramod Abichandani

New Jersey Institute of Technology

Infrared Object Detection Multimodal Object Detection Object Detection Aerial Object Detection Multimodal Sensor Fusion

Ask PDF

Apr 27, 2026

RACANet: Reliability-Aware Crowd Anchor Network for RGB-T Crowd Counting

RGB-Thermal (T) crowd counting aims to integrate visible-spectrum and thermal infrared information to improve the robustness of crowd density estimation in complex scenes. Although existing studies generally improve counting accuracy through cross-modal feature fusion, most current methods rely on implicit cross-modal fusion strategies and lack explicit modeling of local spatial discrepancies as well as fine-grained characterization of modality reliability at the positional level, thereby limiting the accuracy and interpretability of the fusion process. To address these issues, this paper proposes a two-stage fusion framework, RACANet, a Reliability-Aware Crowd Anchor Network for RGB-T crowd counting. First, we introduce a lightweight cross-modal alignment pretraining stage, which explicitly learns cross-modal semantic correspondences through crowd-prior supervision and local bidirectional soft matching. Then, based on the priors learned during pretraining, a Local Anchor Fusion Module (LAFM) is introduced in the formal training stage. This module generates local semantic anchors by aggregating features from highly reliable regions and further enables adaptive pixel-level feature redistribution with a local attention mechanism. In addition, we propose a discrepancy-aware consistency constraint to dynamically coordinate the reliability of regions where modal representations are consistent. Experiments conducted on two widely used benchmark datasets, RGBT-CC and Drone-RGBT, demonstrate that RACANet outperforms existing methods. The anonymous code is available at https://anonymous.4open.science/r/RACANet-9985.

Jinghao Shi, Mengqi Lei, Kunliang He +3

School of Computer Science, China University of Geosciences, Wuhan, Wuhan, China

Cross-Modal Alignment Crowd Counting Multimodal Fusion Reliability-Aware Multimodal Fusion

Ask PDF

Abstract

Visible-Thermal Object Detection

Explore similar work

RA-SOD: Reliability-Aware RGB-T Salient Object Detection under Modality Degradation

Tri-Modal Fusion Transformers for UAV-based Object Detection

RACANet: Reliability-Aware Crowd Anchor Network for RGB-T Crowd Counting