QTrack: Query-Driven Reasoning for Multi-modal MOT

Tajamul Ashraf; Tavaheed Tariq; Sonia Yadav; Abrar Ul Riyaz; Adab Ismail; Moloud Abdar; Janibul Bashir

cs.CVMar 14, 2026

QTrack: Query-Driven Reasoning for Multi-modal MOT

Authors: Tajamul Ashraf, Tavaheed Tariq, Sonia Yadav, Abrar Ul Riyaz, Adab Ismail, Moloud Abdar, Janibul Bashir

Organizations: King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Gaash Research Lab, National Institute of Technology Srinagar, India · The University of Queensland, Australia

Abstract

Category-level multi-object trackers estimate trajectories for all detected instances, whereas some applications require propagating only targets specified by a user. We study offline, reference-grounded video tracking: given an ordered video clip, a natural-language query, and the reference-frame boxes of the specified targets, a model predicts their subsequent trajectories. We introduce QTrack, a vision-language policy trained with structured trajectory rewards and Temporal Perception-Aware Policy Optimization (TAPO). TAPO contrasts the likelihood of a generated response under the observed clip and a reference-frame repetition, encouraging the output to depend on visual evidence beyond the reference frame. We also introduce RMOT26, a controlled diagnostic benchmark constructed from existing tracking datasets. Under the reported protocol, QTrack obtains a motion- consistency score of 0.30, MOTP of 0.75, center-location error of 44.61 pixels, and normalized distance error of 0.39; its motion-consistency score ties the strongest comparison method. Component ablations suggest complementary effects from the motion reward and temporal contrast, while an exploratory four- clip, 50-frame evaluation examines transfer beyond the primary short-clip setting. The results support further study of temporal-evidence learning in trajectory- generating vision-language models. Code and data are available at https://github.com/gaash-lab/QTrack

Figures & tables

Figure 1: Reference-grounded, query-conditioned tracking. Category-level MOT outputs trajectories for all detected category instances. In our protocol, the input supplies reference-frame boxes for the targets specified by a query, and the model predicts one persistent trajectory per supplied target. The primary challenge is temporal propagation under distractors and visibility changes.

Figure 2: Overview of the QTrack framework. Given a video sequence $\mathcal{F}=(F_{0},\dots,F_{T})$ , reference grounding $\mathcal{G}_{0}$ , and query $q$ , QTrack generates a structured reasoning trace and predicts bounding-box trajectories only for the query-specified target(s). The model jointly performs query-conditioned grounding, temporal association, and trajectory prediction.

Figure 3: RMOT26 construction pipeline. We sample short video windows from existing tracking datasets, select query-relevant targets using reference-frame annotations, generate grounded natural-language queries, verify target identities and trajectories, and convert each instance into a structured query-conditioned tracking sample.

Model Name	Params	MCP $\uparrow$	MOTP $\uparrow$	CLE (px) $\downarrow$	NDE $\downarrow$
Open-Source Models
Qwen3-VL-Instruct Yang et al. (2025)	8B	0.25	0.64	96.0	0.97
Gemma 3 Team et al. (2025)	27B	0.24	0.56	58.4	0.88
Gemma 3 Team et al. (2025)	12B	0.18	0.73	172.9	0.95
Llama 3.2 Vision-Instruct Grattafiori et al. (2024)	11B	0.19	0.15	552.1	2.67
DeepSeek Wu et al. (2024)	16B	0.11	0.27	989.13	4.7

Table 1: Main results on the RMOT26 benchmark. Higher is better for MCP and MOTP, while lower is better for CLE and NDE.

Model	HOTA $\uparrow$	MOTA $\uparrow$	MOTP $\uparrow$	IDF1 $\uparrow$	IDSW $\downarrow$	MCP $\uparrow$
Qwen2.5-VL-7B	0.281	0.166	0.708	0.331	18	0.334
VisionReasoner-7B	0.272	0.168	0.722	0.318	29	0.337
QTrack-3B	0.304	0.190	0.751	0.352	21	0.371

Table 2: Comparison of QTrack on longer sequences.

Panel of Figure 4 — Figure 4: Ablation analysis. GRPO, MCP, and TAPO provide complementary gains, with the full combination performing best.

Figure 5: Reasoning trace analysis. Phrase-level annotations show whether the generated reasoning is consistent with visual evidence and the predicted trajectory.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Appendix Section	Description
Section A	RMOT26 Data Card
Section B	RMOT26 Dataset Statistics
Section C	Metrics and Implementation Details
Section D	Evaluation Protocol for Traditional MOT Comparison
Section E	Comparison with Fine-Tuned Models
Section F	Additional Qualitative Results

Appendix

Table 8

Figure 6: Overview of the RMOT26 dataset components. The dataset contains four types of query scenarios: (1) crowded scenes, (2) partial occlusion, (3) complete occlusion, and (4) similar-appearance distractors. These cases require models to perform robust language-guided tracking under challenging visual conditions.

Dataset	Train		Test
	Sequences	Frames	Sequences	Frames
MOT17	253	1058	242	995
MOT16	122	535	66	284
MOT20	73	305	46	190
DanceTrack	962	3864	574	2275
SportsMOT	73	297	2	8

Appendix

Table 3: Composition of the RMOT26 dataset derived from existing MOT benchmarks.

Split	Sequences	Frames
Train	1636	6678
Test	1000	4008

Appendix

Table 4: Overall RMOT26 dataset statistics for training and test splits.

Metric	Description
IoU	Spatial overlap between $B_{t}^{pred}$ and $B_{t}^{gt}$ per frame.
Center Distance	Euclidean distance between predicted and ground-truth centers $(x_{t},y_{t})$ .
Direction Consistency ( $A_{t}$ )	Cosine similarity between predicted and ground-truth motion vectors $\Delta_{t}^{pred}$ and $\Delta_{t}^{gt}$ .
Speed Consistency ( $S_{t}$ )	Gaussian penalty on motion magnitude difference $\\|\Delta_{t}^{pred}\\|$ and $\\|\Delta_{t}^{gt}\\|$ .
MCP	$\displaystyle\text{MCP}=\frac{1}{T-1}\sum_{t=2}^{T}A_{t}\cdot S_{t}$ .

Appendix

Table 5: Evaluation metrics for RMOT26.

Hyperparameter	Value	Hyperparameter	Value
max_prompt_length	1300	max_response_length	2000
max_pixels	12845056	use_kl_loss	true
kl_loss_coef	5.0e-3	fsdp.param_offload	false
fsdp.torch_dtype	null	offload.param_offload	true
offload.optimizer_offload	true	rollout.n	8
limit_images	2	learning_rate	1.0e-6

Appendix

Table 6: Training hyperparameters for QTrack.

Table 7: Comparison with specialized multi-object trackers and reasoning-based trackers on MOT17 and DanceTrack. The results illustrate the relative strengths of QTrack across localization, association, and motion-consistency metrics.

Model	Params	MCP $\uparrow$	MOTP $\uparrow$	MOTA $\uparrow$	NDE $\downarrow$
Qwen 2.5 VL Instruct Team (2025)	3B	0.14	0.76	-0.51	3.41
Gemma3 Team et al. (2025)	4B	0.18	0.73	-0.16	0.95
Visionreasoner Liu et al. (2025a)	3B	0.22	0.65	0.01	0.76
QTrack (Ours)	3B	0.30	0.75	0.21	0.39

Appendix

Table 8: Comparison of VLLM models after fine-tuning.

Figure 7: Qualitative comparison for single-object tracking across multiple frames. QTrack produces more accurate and stable bounding boxes over time, while baseline VLLM models often exhibit drifting or inconsistent localization as the sequence progresses.

Figure 8: In this figure, Visionreasoner is not able to detect and track all the objects in the frames, but QTrack can detect and track all objects in all frames

Figure 9: Additional qualitative comparison of different VLLM models on challenging tracking scenarios. Our method demonstrates stronger temporal consistency and better object localization compared to baseline models.

Figure 10: Effect of spatial initialization. Tracking is reliable with an initial bounding box, but localization becomes difficult when relying solely on language queries.

Figure 11: This figure shows failure cases of QTrack, we can see that for some sequences QTrack is unable to predict next bounding box for frames, it repeatedly predicts the same initial bounding box for all future frames.

Model	MCP $\uparrow$	MOTP $\uparrow$	CLE $\downarrow$	NDE $\downarrow$
QTrack with structured reasoning	0.30	0.75	44.61	0.39
QTrack without structured reasoning	0.28	0.72	129.5	0.81

Appendix

Table 9: Ablation for reasoning

Tier	$n$	IoU	RQS	Faithfulness
Easy ( ${\geq}0.7$ )	93	0.882	0.593	0.661
Medium ( $[0.4,0.7)$ )	56	0.599	0.554	0.580
Hard ( $[0.1,0.4)$ )	28	0.278	0.585	0.643
Failure ( $<0.1$ )	23	0.021	0.489	0.457
All	200	0.619	0.569	0.280 †

Appendix

Table 10: RQS and faithfulness by IoU tier (

n{=}200

Referring MOT

See all 10 papers

Referring Multi-Object Tracking with Diverse Dynamic Expressions
Jun 7, 2024Yani Zhang, Dongming Wu, Wencheng Han +3Referring MOT Multi-Object Tracking
School of Computer Science, National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, Wuhan 430072, China. · Multimedia Lab (MMLab), Department of Information Engineering, the Chinese University of Hong Kong, Hong Kong, China. · State Key Laboratory of Internet of Things for Smart City, Department of Computer and Information Science, University of Macau, Macau, China. +1
STORM: End-to-End Referring Multi-Object Tracking in Videos
Apr 12, 2026Zijia Lu, Jingru Yi, Jue Wang +4Referring MOT Multi-Object Tracking
Amazon
Generative Semantic Multi-Object Tracking: A Large-Scale Benchmark and an MLLM-Driven Reasoning Framework
Jan 10, 2026Pan Liao, Feng Yang, Di Wu +3Multimodal Large Language Models Multi-Object Tracking

Jun 7, 2024J/K move · Enter open · S save

Referring Multi-Object Tracking with Diverse Dynamic Expressions

Yani Zhang, Dongming Wu, Wencheng Han, Xingping Dong+2

School of Computer Science, National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, Wuhan 430072, China. · Multimedia Lab (MMLab), Department of Information Engineering, the Chinese University of Hong Kong, Hong Kong, China. · State Key Laboratory of Internet of Things for Smart City, Department of Computer and Information Science, University of Macau, Macau, China.+1

Referring MOT Multi-Object Tracking Visual Object Tracking

Open paper AskPDF arXiv

Referring understanding is a fundamental task that bridges natural language and visual content by localizing objects described in free-form expressions. However, existing works are constrained by limited language expressiveness, lacking the capacity to model object dynamics in spatial numbers and temporal states. To address these limitations, we introduce a new and general referring understanding task, termed referring multi-object tracking (RMOT). Its core idea is to employ a language expression as a semantic cue to guide the prediction of multi-object tracking, comprehensively accounting for variations in object quantity and temporal semantics. Along with RMOT, we introduce a RMOT benchmark named Refer-KITTI-V2, featuring scalable and diverse language expressions. To efficiently generate high-quality annotations covering object dynamics with minimal manual effort, we propose a semi-automatic labeling pipeline that formulates a total of 9,758 language prompts. In addition, we propose TempRMOT, an elegant end-to-end Transformer-based framework for RMOT. At its core is a query-driven Temporal Enhancement Module that represents each object as a Transformer query, enabling long-term spatial-temporal interactions with other objects and past frames to efficiently refine these queries. TempRMOT achieves state-of-the-art performance on both Refer-KITTI and Refer-KITTI-V2, demonstrating the effectiveness of our approach. The source code and dataset is available at https://github.com/zyn213/TempRMOT.

Abstract

Referring MOT

Explore similar work

Referring Multi-Object Tracking with Diverse Dynamic Expressions

STORM: End-to-End Referring Multi-Object Tracking in Videos

Generative Semantic Multi-Object Tracking: A Large-Scale Benchmark and an MLLM-Driven Reasoning Framework