Reliability-Consistent Appearance Modeling for RGB-Infrared Multi-Object Tracking

Bingyang Wang; Jiazuo Yu; Lu Zhang; Lihe Zhang; Huchuan Lu; You He

cs.CVOct 16, 2026

Reliability-Consistent Appearance Modeling for RGB-Infrared Multi-Object Tracking

Authors: Bingyang Wang, Jiazuo Yu, Lu Zhang, Lihe Zhang, Huchuan Lu, You He

Organizations: School of Information and Communication Engineering, Dalian University of Technology, Dalian 116024, China · Department of Engineering, Tsinghua University, Beijing 100084, China

Abstract

Recent RGB-infrared multi-object tracking (MOT) methods have substantially improved cross-modal representation learning and multimodal information integration. However, most existing methods compress RGB and infrared evidence into a unified descriptor. When one modality is temporarily degraded, the tracker can no longer regulate its contribution separately; unreliable evidence may distort association and contaminate appearance memory. We therefore propose RCA-MOT, a reliability-consistent appearance modeling framework that preserves cross-modal complementarity while retaining post-representation evidence controllability. RGB-specific, infrared-specific, and fused identity representations are maintained as jointly accessible appearance states. For each detection, a simple training-free target-scene estimator produces branch-wise appearance reliability used across the two temporal roles of appearance: relative reliability regulates different identity branches, absolute reliability calibrates borderline candidates near the ReID acceptance boundary, and the same branch-wise reliability controls how strongly matched descriptors update historical appearance memory. The individual components are intentionally simple, focusing evaluation on reliability-controllable appearance organization rather than elaborate fusion or learned reliability-estimation modules. RCA-MOT sets new SOTA results on VT-MOT and UniRTL, improving the previous best HOTA by 5.402/6.075 points and IDF1 by 11.549/12.789 points, respectively, while post-hoc analyses further validate reliability as an informative control signal. Code will be released upon acceptance.

Figures & tables

Fig. 1: Motivation for reliability-consistent appearance control. (a) Reliability is modality dependent: low illumination weakens RGB appearance, whereas low thermal contrast weakens infrared appearance. (b) Fused-only MOT makes modality-specific evidence inaccessible after fusion. (c) RCA-MOT retains RGB-specific, infrared-specific, and fused descriptors and uses target-scene reliability to regulate both association and appearance-memory update.

Fig. 2: Overview of RCA-MOT. Target-scene reliability regulates branch evidence and near-threshold calibration during association, and controls how strongly accepted descriptors update branch-wise memories.

Fig. 3: Controllable tri-branch representation and training objective. Modality-specific encoders and a simple fusion branch retain three usable descriptors; branch-wise supervision, prediction consistency, and feature alignment jointly constrain their identity geometry.

Fig. 4: Target-scene branch-reliability estimation. RGB uses exposure, target variation, and local separability; infrared uses the latter two cues. Fused reliability is conservatively defined as $q_{\mathrm{F}}=\min(q_{\mathrm{R}},q_{\mathrm{I}})$ .

Method	HOTA $\uparrow$	DetA $\uparrow$	MOTA $\uparrow$	MOTP $\uparrow$	IDF1 $\uparrow$
FairMOT [ 16 ]	37.350	34.628	37.266	72.525	45.795
CenterTrack [ 17 ]	39.045	38.104	30.585	72.874	44.420
TraDeS [ 18 ]	38.319	36.370	34.632	72.295	47.008
TransTrack [ 19 ]	38.000	35.711	36.156	73.823	43.567
ByteTrack [ 28 ]	38.393	32.122	33.151	73.483	45.757
OC-SORT [ 29 ]	31.479	25.244	28.948	73.150	38.086

TABLE I: VT-MOT results under Protocol I. Parentheses identify the fixed offline detections used by RCA-MOT; cross-method results are benchmark-level comparisons.

Method	HOTA $\uparrow$	DetA $\uparrow$	MOTA $\uparrow$	MOTP $\uparrow$	IDF1 $\uparrow$
PFTrack † [ 1 ]	46.568	55.043	66.946	77.732	52.911
UniS-MOT † [ 2 ]	49.062	50.678	50.577	78.905	56.816
RCA-MOT (PFTrack det.)	55.137	55.043	68.028	78.140	69.605
RCA-MOT (YOLO-RGBT det.)	49.384	48.484	54.382	79.033	61.225

TABLE II: UniRTL test results. PFTrack and UniS-MOT are reproduced on the same 30/20 split.

(a) Core components with PFTrack detections
Variant	VT-MOT				UniRTL
Variant	HOTA $\uparrow$	DetA $\uparrow$	AssA $\uparrow$	IDF1 $\uparrow$	HOTA $\uparrow$	DetA $\uparrow$	AssA $\uparrow$	IDF1 $\uparrow$
Baseline	40.896	39.410	43.672	49.391	51.126	55.022	47.756	63.527
Tri	43.076	39.291	48.550	53.376	52.479	55.032	50.272	65.430
Tri+Gate	43.344	38.639	49.842	54.438	54.563	54.863	54.526	69.084
Tri+Cost	43.534	39.296	49.539	54.222	53.791	55.138	52.708	67.684

TABLE III: Controlled evaluation of RCA-MOT. Full denotes the complete framework; all comparisons within each setting use identical offline detections. In (b), results are reported as Base/Full/

\Delta

Association	Memory	HOTA	AssA	IDF1
Fused	$\alpha=0.8$	43.076	48.550	53.376
Uniform	$\alpha=0.8$	43.182	48.702	53.558
Relative	$\alpha=0.8$	43.144	48.630	53.491
Uniform+Abs.	$\alpha=0.8$	43.401	49.238	53.979
Relative+Abs.	$\alpha=0.8$	43.534	49.539	54.222
Uniform+Abs.	$\alpha=0.9$	44.363	51.321	55.518

TABLE IV: Association–memory analysis on VT-MOT with PFTrack detections. ”Abs.” denotes absolute near-threshold calibration.

Fig. 5: Validity of appearance reliability on VT-MOT. Post-hoc analyses relate absolute reliability to observation difficulty and relative RGB–infrared reliability to modality discriminability, reliability gap, and retrieval under two memory representations.

Fig. 6: Qualitative comparison between PFTrack and our best-performing RCA-MOT configuration on VT-MOT. Each group presents synchronized RGB and infrared observations, with PFTrack shown above and RCA-MOT below. Panels (a)–(c) illustrate identity switches or inconsistent identity preservation, while panel (d) shows a missed or interrupted trajectory.

Multimodal Visual Object Tracking

See all 19 papers

VLA-ReID: Video-Level Association for Re-Identification in Multi-Object Tracking with Highly Similar Objects
Jul 19, 2026Yanrong Qin, Xiaoyan Cao, Yao YaoQuery Re-Identification Data Association
Glasgow College, University of Electronic Science and Technology of China, Chengdu, China · Key Laboratory for Urban Habitat Environmental Science and Technology, School of Environment and Energy, Peking University Shenzhen Graduate School, Shenzhen, China · School of Management Science and Real Estate, Chongqing University, Chongqing, China
End-to-End Self-Supervised RGB-T Tracking without Modality Misleading
Sep 29, 2026Shenglan Li, Rui Yao, Kunyang Sun +4Multimodal Robustness Visual Object Tracking
School of Computer Science and Technology / School of Artificial Intelligence, China University of Mining and Technology, China. · Mine Digitization Engineering Research Center of the Ministry of Education, China. · University of Auckland, Auckland, New Zealand. +1
RT-RMOT: A Dataset and Framework for RGB-Thermal Referring Multi-Object Tracking
Feb 25, 2026Yanqiu Yu, Zhifan Jin, Sijia Chen +4Referring MOT Visual Object Tracking
Huazhong University of Science and Technology, Wuhan, Hubei, China · South-Central Minzu University, Wuhan, Hubei, China

Jul 19, 2026J/K move · Enter open · S save

VLA-ReID: Video-Level Association for Re-Identification in Multi-Object Tracking with Highly Similar Objects

Yanrong Qin, Xiaoyan Cao, Yao Yao

Glasgow College, University of Electronic Science and Technology of China, Chengdu, China · Key Laboratory for Urban Habitat Environmental Science and Technology, School of Environment and Energy, Peking University Shenzhen Graduate School, Shenzhen, China · School of Management Science and Real Estate, Chongqing University, Chongqing, China

Query Re-Identification Data Association Multi-Object Tracking

Open paper AskPDF arXiv

Multi-object tracking (MOT) aims to localize multiple objects in videos while preserving their identities over time. Long-term identity preservation remains difficult when objects are small, densely distributed, and highly similar in appearance, as in bee swarm scenes. Existing trackers rely on re-identification (re-ID) models trained through single-instance assignment (instance-level querying). At inference, however, MOT requires global assignment between multiple trajectories and detections, corresponding to video-level querying. This training-inference mismatch can cause identity switches among visually similar objects. Existing approaches also often require substantial additional annotations to enhance appearance discrimination. We propose Video-Level Association re-ID (VLA-ReID), which reformulates re-ID as video-level association modeling. It uses aggregated historical trajectory features as queries and all current-frame detections as candidates, enabling direct optimization of their global association at each frame. In addition, Frame-Common Appearance Estimation (FCAE) estimates a common appearance direction from current-frame detections, while Common-Appearance Suppression (CAS) removes the corresponding component along this direction from trajectory and detection features. This amplifies discriminative differences among highly similar objects without additional annotations. Experiments on BEE24 show that VLA-ReID improves HOTA by 1.1, MOTA by 0.3, AssR by 2.6, AssA by 0.7, and IDF1 by 0.8 over state-of-the-art trackers, while reducing identity switches by 28%. These results demonstrate the effectiveness of video-level re-ID modeling for appearance-based association in MOT.

Abstract

Multimodal Visual Object Tracking

Explore similar work

VLA-ReID: Video-Level Association for Re-Identification in Multi-Object Tracking with Highly Similar Objects

End-to-End Self-Supervised RGB-T Tracking without Modality Misleading

RT-RMOT: A Dataset and Framework for RGB-Thermal Referring Multi-Object Tracking