VSD-MOT: End-to-End Multi-Object Tracking in Low-Quality Video Scenes Guided by Visual Semantic Distillation

Jun Du

cs.CVMar 21, 2026

VSD-MOT: End-to-End Multi-Object Tracking in Low-Quality Video Scenes Guided by Visual Semantic Distillation

Authors: Jun Du

Abstract

Existing multi-object tracking algorithms typically fail to adequately address the issues in low-quality videos, resulting in a significant decline in tracking performance when image quality deteriorates in real-world scenarios. This performance degradation is primarily due to the algorithms' inability to effectively tackle the problems caused by information loss in low-quality images. To address the challenges of low-quality video scenarios, inspired by vision-language models, we propose a multi-object tracking framework guided by visual semantic distillation (VSD-MOT). Specifically, we introduce the CLIP Image Encoder to extract global visual semantic information from images to compensate for the loss of information in low-quality images. However, direct integration can substantially impact the efficiency of the multi-object tracking algorithm. Therefore, this paper proposes to extract visual semantic information from images through knowledge distillation. This method adopts a teacher-student learning framework, with the CLIP Image Encoder serving as the teacher model. To enable the student model to acquire the capability of extracting visual semantic information suitable for multi-object tracking tasks from the teacher model, we have designed the Dual-Constraint Semantic Distillation method (DCSD). Furthermore, to address the dynamic variation of frame quality in low-quality videos, we propose the Dynamic Semantic Weight Regulation (DSWR) module, which adaptively allocates fusion weights based on real-time frame quality assessment. Extensive experiments demonstrate the effectiveness and superiority of the proposed method in low-quality video scenarios in the real world. Meanwhile, our method can maintain good performance in conventional scenarios.

Figures & tables

Fig. 1: Comparison between (a) the direct integration of the CLIP Image Encoder process and (b) our proposed visual semantic distillation tracking process. The direct integration approach significantly impacts the efficiency of multi-object tracking algorithms. The visual semantic distillation tracking employs a teacher-student learning framework, adding minimal parameters to learn invariant visual semantic information, thereby enhancing tracking performance while ensuring algorithmic efficiency.

Fig. 2: The overall architecture of VSD-MOT. The frozen CLIP Image Encoder extracts global visual semantic information from images, and proposals generated by the detector YOLOX are used to produce proposal queries. Tracking inquiries are transferred from the preceding frame for the purpose of forecasting the bounding boxes of tracked objects. The combination of proposal queries and track queries generates query vectors. Through the Dual-Constraint Semantic Distillation (DCSD) method, the student model can learn from the CLIP Image Encoder the capability to extract semantic information that is adaptive to multi-object tracking tasks. The semantic information extracted by the student model is integrated with query vectors through the Dynamic Semantic Weight Regulation (DSWR) module to generate adaptively fused vectors. Both the fused vectors and the image features are fed into the tracker to produce predictions frame by frame.

Fig. 3: Structure of Dual-Constrained Semantic Distillation. To enable the student model to better extract semantic information adapted to the multi-object tracking task, the core of this module implements a dual-constraint mechanism via two complementary losses, which balances local feature matching and global semantic consistency.

Fig. 4: Structure of student model. It employs a Transformer Encoder architecture to process the query vectors of the multi-object tracking algorithm.

Fig. 5: Network architecture of Dynamic Semantic Weight Regulation Module. The module evaluates frame quality through clarity, noise level, and contrast metrics, then generates adaptive fusion weights for combining visual semantic features with query vector features.

Tracker	HOTA(%)↑	DetA(%)↑	AssA(%)↑	MOTA(%)↑	IDF1(%)↑
CO-MOT [ 58 ]	42.0	54.4	32.9	53.3	43.0
MeMOTR [ 59 ]	49.1	58.1	42.1	63.7	52.0
MOTIP [ 40 ]	52.2	61.2	45.3	67.8	57.1
Hybrid-SORT-ReID [ 12 ]	41.8	52.3	33.7	59.8	44.3
Hybrid-SORT [ 12 ]	37.3	52.4	26.8	59.6	38.9
MOTR [ 35 ]	39.9	51.8	31.3	46.3	41.4

TABLE I: Performance Comparison of Different Trackers on the LQDanceTrack Validation Set. ↑ Indicates Higher Is Better, the Best Results for Each Metric Are Highlighted in Bold. The Proposed VSD-MOT Outweighs Other Methods on the LQDanceTrack Validation Set and Achieves the Best in All Metrics, Demonstrating the Effectiveness of the Proposed Method in Real-world Low-quality Video Scenarios

Tracker	HOTA(%)↑	DetA(%)↑	AssA(%)↑	MOTA(%)↑	IDF1(%)↑
CO-MOT [ 58 ]	46.7	54.8	40.5	44.9	47.3
MeMOTR [ 59 ]	59.3	69.9	50.7	78.1	62.7
MOTIP [ 40 ]	62.1	73.5	52.8	83.8	66.9
Hybrid-SORT-ReID [ 12 ]	49.3	56.5	43.5	65.3	47.4
Hybrid-SORT [ 12 ]	44.5	56.3	35.6	64.5	43.2
MOTR [ 35 ]	48.4	59.2	39.8	57.5	50.2

TABLE II: Performance Comparison of Different Trackers on the DanceTrack Validation Set. ↑ Indicates Higher Is Better, the Best Results for Each Metric Are Highlighted in Bold. The Proposed VSD-MOT Outperforms Other Methods on the DanceTrack Validation Set and Achieves the Best in All Metrics. It Shows That after Training on the Mixed Training Set, the Proposed Method Can Maintain Good Performance in Conventional Scenarios

Tracker	HOTA(%)↑	DetA(%)↑	AssA(%)↑	MOTA(%)↑	IDF1(%)↑
MeMOTR [ 59 ]	44.9	53.3	38.2	62.1	56.5
MOTIP [ 40 ]	46.6	55.6	39.4	65.9	57.8
ByteTrack [ 22 ]	51.2	59.0	44.8	71.3	62.3
Hybrid-SORT-ReID [ 12 ]	49.4	58.4	42.0	70.9	59.8
Hybrid-SORT [ 12 ]	47.1	56.8	39.7	68.4	58.2
MOTR [ 35 ]	43.3	48.7	38.9	55.3	53.5

TABLE III: Performance Comparison of Different Trackers on the LQMOT Validation Set. ↑ Indicates Higher Is Better, the Best Results for Each Metric Are Highlighted in Bold. The Proposed VSD-MOT Outweighs Other Methods on the LQMOT Validation Set, Demonstrating the Effectiveness of the Proposed Method in Real-world Low-quality Video Scenarios.

Tracker	HOTA(%)↑	DetA(%)↑	AssA(%)↑	MOTA(%)↑	IDF1(%)↑
MeMOTR [ 59 ]	52.6	59.9	46.5	67.4	65.1
MOTIP [ 40 ]	54.3	60.2	49.3	70.3	67.5
ByteTrack [ 22 ]	58.7	64.4	53.7	76.7	71.5
Hybrid-SORT-ReID [ 12 ]	57.5	64.2	51.7	76.6	69.9
Hybrid-SORT [ 12 ]	55.7	63.7	49.1	72.3	67.8
MOTR [ 35 ]	48.2	51.0	45.8	58.6	59.6

TABLE IV: Performance Comparison of Different Trackers on the MOT Validation Set. ↑ Indicates Higher Is Better, the Best Results for Each Metric Are Highlighted in Bold. The Proposed VSD-MOT Outperforms Other Methods on the MOT Validation Set. It Shows That after Training on the Mixed Training Set, the Proposed Method Can Maintain Good Performance in Conventional Scenarios.

No.	Dataset	Baseline	CLIP Distillation	DCSD	DSWR	HOTA(%)↑	DetA(%)↑	AssA(%)↑	MOTA(%)↑	IDF1(%)↑
1		✓				49.1	57.5	42.3	60.2	53.7
2	LQMOT	✓	✓			52.7	59.1	47.2	64.6	59.5
3		✓	✓	✓		56.6	62.3	51.5	69.7	63.2
4		✓	✓	✓	✓	58.6	63.1	54.6	69.4	65.2
5		✓				55.3	61.3	50.2	66.4	61.8
6	MOT	✓	✓			60.1	64.1	56.6	71.5	69.7

TABLE V: Effectiveness of Proposed Components on the LQMOT and MOT Validation Sets. ✓Represents the Baseline Equipped with the Corresponding Component. It Can Be Observed That Integrating the Proposed Modules into the Baseline Gradually Enhances Overall Performance

No.	Dataset	Low-quality：High-quality	HOTA(%)↑	DetA(%)↑	AssA(%)↑	MOTA(%)↑	IDF1(%)↑
1		All High-quality	48.9	56.3	42.6	62.7	55.2
3	LQMOT	1:1	55.3	61.6	49.8	68.5	62.9
4		2:1	58.6	63.1	54.6	69.4	65.2
5		All Low-quality	54.1	61.8	47.5	67.4	62.6
6		All High-quality	69.5	78.4	61.8	79.3	71.7
8	MOT	1:1	65.4	70.4	60.9	76.8	72.3

TABLE VI: Comparison of Applying Different Ratio of High-quality to Low-quality Data in the Mixed Training Set on the LQMOT and MOT Validation Sets. It Can Be Observed Better Results Are Achieved When the Ratio of High-quality Data to Low-quality Data Is 1:2.

Dataset	Alpha	HOTA(%)↑	DetA(%)↑	AssA(%)↑	MOTA(%)↑	IDF1(%)↑
	0.2	53.7	58.1	49.8	67.3	61.8
LQMOT	0.4	58.6	63.1	54.6	69.4	65.2
	0.6	54.5	59.0	50.5	67.8	62.0
	0.2	65.5	68.5	62.8	77.2	71.4
MOT	0.4	70.8	72.6	69.0	77.8	76.6
	0.6	63.1	65.5	61.0	76.0	70.5

TABLE VII: Efficiency Analysis on the Impact of Introduced Components on Multi-Object Tracking Using the LQMOT and MOT Validation Sets. It Can Be Observed That an Alpha Value of 0.50 Results in the Best Overall Performance.

Fig. 6: Partial results of the baseline method (first row) and SE-MOT (second row) on the LQDanceTrack validation set. In low-quality videos, our method achieves better tracking performance than the baseline model by introducing global visual semantic information of the images.

Fig. 7: Partial results of SE-MOT on the DanceTrack validation set. Our method can maintain good performance in conventional scenarios.

No.	Baseline	Student Model	DSWR	Parameters (M)	FPS
1	✓			41.9	15.8
2	✓	✓		42.6	15.6
3	✓	✓	✓	42.6	15.5

TABLE VIII: Efficiency Analysis on the Impact of Introduced Components on Multi-Object Tracking Using the LQDanceTrack Validation Set. It can be observed that the introduction of the student model has no significant impact on the inference efficiency of the model.

Multi-Object Tracking

Multi-object tracking (MOT) links detections across video frames to maintain object identities through motion, occlusion, and changing viewpoints.

See all 193 papers

Mar 14, 2026

QTrack: Query-Driven Reasoning for Multi-modal MOT

Category-level multi-object trackers estimate trajectories for all detected instances, whereas some applications require propagating only targets specified by a user. We study offline, reference-grounded video tracking: given an ordered video clip, a natural-language query, and the reference-frame boxes of the specified targets, a model predicts their subsequent trajectories. We introduce QTrack, a vision-language policy trained with structured trajectory rewards and Temporal Perception-Aware Policy Optimization (TAPO). TAPO contrasts the likelihood of a generated response under the observed clip and a reference-frame repetition, encouraging the output to depend on visual evidence beyond the reference frame. We also introduce RMOT26, a controlled diagnostic benchmark constructed from existing tracking datasets. Under the reported protocol, QTrack obtains a motion- consistency score of 0.30, MOTP of 0.75, center-location error of 44.61 pixels, and normalized distance error of 0.39; its motion-consistency score ties the strongest comparison method. Component ablations suggest complementary effects from the motion reward and temporal contrast, while an exploratory four- clip, 50-frame evaluation examines transfer beyond the primary short-clip setting. The results support further study of temporal-evidence learning in trajectory- generating vision-language models. Code and data are available at https://github.com/gaash-lab/QTrack

Tajamul Ashraf, Tavaheed Tariq, Sonia Yadav +4

King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Gaash Research Lab, National Institute of Technology Srinagar, India · The University of Queensland, Australia

Referring MOT Multimodal Visual Object Tracking Multi-Object Tracking Vision-Language Models Multimodal Reasoning

Ask PDF

May 11, 2026

Clip-level Uncertainty and Temporal-aware Active Learning for End-to-End Multi-Object Tracking

Multi-Object Tracking (MOT) in dynamic environments relies on robust temporal reasoning to maintain consistent object identities over time. Transformer-based end-to-end MOT models achieve strong performance by explicitly modeling temporal dependencies, yet training them requires extensive bounding-box and identity annotations. Given the high labeling cost and strong redundancy in videos, Active Learning (AL) is an effective approach to improve annotation efficiency. However, existing AL methods for MOT primarily operate at the frame level, which is structurally misaligned with modern end-to-end trackers whose inference and training rely on multi-frame clips. To bridge this gap, we formulate clip-level active learning and propose Clip-level Uncertainty and Temporal-aware Active Learning (CUTAL). In contrast to frame-based approaches, CUTAL scores each clip using uncertainty metrics derived from multi-frame predictions to capture inter-frame correspondence ambiguities, while enforcing temporal diversity to select an informative and non-redundant subset. Experiments show that CUTAL achieves stronger overall performance than baselines at the same label budgets across MeMOTR and SambaMOTR. Notably, CUTAL achieves performance comparable to full supervision for MeMOTR on both datasets using only 50% of the labeled training data.

Riku Inoue, Shogo Sato, Kazuhiko Murasaki +3

NTT, Inc., Kanagawa, Japan

Active Learning Uncertainty Sampling Multi-Object Tracking

Ask PDF

Oct 5, 2026

Difference Feature Map Distillation: Transferring Inter-Sample Relational Knowledge Towards Efficient Transformer-Based Tracking

In autonomous driving perception, visual object tracking systems must satisfy stringent latency and power constraints while remaining robust in complex and dynamic environments. Although transformer-based trackers achieve state-of-the-art accuracy, their substantial computational and memory overheads hinder deployment on real-time, resource-constrained platforms. To move toward this goal, we propose Difference Feature Map Knowledge Distillation (DFM-KD), a novel relational distillation framework tailored for transformer-based visual object tracking. Unlike conventional feature distillation methods that minimize point-wise discrepancies (e.g., mean squared error) between teacher and student feature representations, DFM-KD transfers knowledge through inter-sample feature differences, explicitly aligning the relational structure of the feature space. By distilling how the teacher models appearance variation and consistency across samples, rather than enforcing similarity in absolute activations, DFM-KD enables the student to better capture the structural dynamics of visual changes within a batch. As a result, the distilled model exhibits enhanced feature robustness and improved tracking performance. Extensive experiments demonstrate that DFM-KD consistently outperforms conventional feature-level distillation methods in both tracking precision and success rates.

Zhicheng Ding, Xinyu Chu, Qing Tian

Department of Computer Science, Bowling Green State University, OH 43402, USA · Department of Computer Science, University of Alabama at Birmingham, AL 35294, USA

Visual Object Tracking Autonomous Driving Perception Relational Knowledge Distillation Feature-Based Knowledge Distillation

Ask PDF

Abstract

Multi-Object Tracking

Explore similar work

QTrack: Query-Driven Reasoning for Multi-modal MOT

Clip-level Uncertainty and Temporal-aware Active Learning for End-to-End Multi-Object Tracking

Difference Feature Map Distillation: Transferring Inter-Sample Relational Knowledge Towards Efficient Transformer-Based Tracking