Training-Free Semantic Multi-Object Tracking with Vision-Language Models

Laurence Bonat; Francesco Tonini; Elisa Ricci; Lorenzo Vaquero

cs.CVApr 15, 2026

Training-Free Semantic Multi-Object Tracking with Vision-Language Models

Authors: Laurence Bonat, Francesco Tonini, Elisa Ricci, Lorenzo Vaquero

Organizations: Department of Information Engineering and Computer Science, University of Trento, Trento, Italy · Fondazione Bruno Kessler, Trento, Italy

Abstract

Semantic Multi-Object Tracking (SMOT) extends multi-object tracking with semantic outputs such as video summaries, instance-level captions, and interaction labels, aiming to move from trajectories to human-interpretable descriptions of dynamic scenes. Existing SMOT systems are trained end-to-end, coupling progress to expensive supervision, limiting the ability to rapidly adapt to new foundation models and new interactions. We propose TF-SMOT, a training-free SMOT pipeline that composes pretrained components for detection, mask-based tracking, and video-language generation. TF-SMOT combines D-FINE and the promptable SAM2 segmentation tracker to produce temporally consistent tracklets, uses contour grounding to generate video summaries and instance captions with InternVideo2.5, and aligns extracted interaction predicates to BenSMOT WordNet synsets via gloss-based semantic retrieval with LLM disambiguation. On BenSMOT, TF-SMOT achieves state-of-the-art tracking performance within the SMOT setting and improves summary and caption quality compared to prior art. Interaction recognition, however, remains challenging under strict exact-match evaluation on the fine-grained and long-tailed WordNet label space; our analysis and ablations indicate that semantic overlap and label granularity substantially affect measured performance.

Figures & tables

Fig. 1: Comparison of training-based and training-free Semantic Multi-Object Tracking (SMOT) paradigms. Top: Traditional SMOT systems like SMOTer [ 16 ] require task-specific end-to-end training on annotated datasets, coupling tracking and semantic prediction to supervised learning. Bottom: our TF-SMOT operates entirely at inference time by composing frozen, pretrained components to produce tracking outputs together with video summaries, instance-level captions, and directed interactions, without using SMOT training data.

Fig. 2: TF-SMOT overview. The system decomposes SMOT into explicit functional stages with well-defined interfaces. The Person Tracking module (left) utilizes a person detector $\Phi_{\text{det}}$ and mask-based tracker $\Phi_{\text{trk}}$ to determine the person trajectories within the video. Subsequently, the Caption Generation module (right) renders the contours of the people to better ground them and uses a video captioner $\Phi_{\text{vlm}}$ to generate the instance captions and video summary . Finally, the Interaction Extraction module (bottom) identifies interaction verbs via $\Phi_{\text{vrb}}$ , maps them to WordNet synsets using semantic similarity $\Phi_{\text{emb}}$ , and selects the most relevant instance interactions using an LLM $\Phi_{\text{emb}}$ with caption context.

look.v.01	2353	touch.v.01	261	facial.n.02	130
smile.v.01	1291	show.v.01	239	rally.n.05	127
talk.v.01	1047	comply.v.01	184	hash_out.v.01	108
talk.v.02	714	receive.v.01	171	insert.v.01	104
converse.v.01	663	dance.v.01	166	teach.v.01	98
listen.v.01	468	handshake.n.01	166	help.v.01	96
embrace.v.02	339	play.v.01	165	pass.v.20	92

TABLE I: Top 30 frequent interactions and their number of occurrences in the BenSMOT test set.

Method	HOTA $\uparrow$	AssA $\uparrow$	DetA $\uparrow$	LocA $\uparrow$	MOTA $\uparrow$	IDSW $\downarrow$	IDR $\uparrow$	IDP $\uparrow$	IDF1 $\uparrow$
SORT [ 5 ]	48.49	38.95	60.91	87.50	53.58	13875	60.85	48.43	53.93
DeepSORT [ 41 ]	50.12	40.23	61.45	87.67	54.29	11278	62.10	51.11	56.76
OC-SORT [ 7 ]	51.00	41.42	63.31	87.61	55.19	15049	63.92	53.10	58.01
ByteTrack [ 45 ]	68.84	71.15	67.10	85.15	73.87	1712	82.25	74.83	78.37
TransTrack [ 31 ]	71.31	73.34	69.67	91.31	74.08	2530	85.63	72.75	78.67
MOTR [ 44 ]	66.10	73.12	55.14	86.30	45.19	617	72.39	70.12	68.97

TABLE II: Tracking performance on BenSMOT. Best results are in bold

Method	Video Summary				Instance Captioning
Method	BLEU	METEOR	ROUGE	CIDEr	BLEU	METEOR	ROUGE	CIDEr
SMOTer [ 16 ]	0.245	0.261	0.223	0.343	0.306	0.209	0.223	0.087
TF-SMOT (ours)	0.369	0.262	0.281	0.360	0.383	0.239	0.228	0.087

TABLE III: Language generation results (video summary and instance captioning) on BenSMOT. Best results are in bold .

⋆ Not directly comparable across settings.
Method / evaluation	Prec.	Rec.	F1
SMOTer (WordNet synsets) [ 16 ]	0.434	0.320	0.368
TF-SMOT (WordNet synsets)	0.089	0.116	0.101
TF-SMOT (lemma-merged synsets) ⋆	0.080	0.103	0.090
TF-SMOT (20 coarse clusters) ⋆	0.495	0.207	0.292

TABLE IV: Strict exact-match interaction recognition on BenSMOT (top), and TF-SMOT under relaxed label spaces (bottom).

Error type	Typical manifestation under strict evaluation
Sense ambiguity	Multiple synsets can be plausible for the same event (e.g., talk.v.01 vs. converse.v.01 ), yet not all of them are present in the ground truth labels.
Near-synonym mismatch	Predictions fall in a tight semantic neighborhood but are counted fully wrong under exact match.
Direction / role flip	Subject/object attribution swaps, flipping directed interactions even when the event is recognized.

TABLE V: Interaction failure taxonomy for WordNet-synset evaluation on BenSMOT.

Label space / setting	Selector	Prec.	Rec.	F1
Full WordNet (335)	LLM (ours)	0.089	0.116	0.101
	top-1 cosine	0.032	0.043	0.037
	top-5 cosine	0.039	0.239	0.067
Lemma-merged (259)	LLM	0.080	0.103	0.090
	top-1 cosine	0.041	0.054	0.047
	top-5 cosine	0.042	0.261	0.073

TABLE VI: Interaction ablations.

“Single contour” renders only the target identity; “multi-contour” renders multiple identities with different colors and prompts the target; “single bounding box” replaces contours with a target box.
Grounding (GT init.)	BLEU $\uparrow$	METEOR $\uparrow$	ROUGE $\uparrow$
Single contour (ours)	0.420	0.328	0.373
Multi-contour (color-coded)	0.427	0.192	0.307
Single bounding box	0.346	0.248	0.296

TABLE VII: Instance-caption grounding ablations using ground-truth boxes for tracker initialization. Best results are in bold .

Fig. 3: TF-SMOT semantic tracking qualitative results on BenSMOT (sequence jT_8i7edEt8 ). As can be seen, TF-SMOT detects a larger number of people in the scene than the ground truth annotations and generates richer, more detailed instance captions. The predicted instance-level interactions are correspondingly broader and more expressive; however, they do not always exactly match the annotated ground truth interactions. This discrepancy leads to low exact-match interaction scores and reflects limitations of the dataset rather than purely model errors, as the selected WordNet synsets can be semantically ambiguous and the interaction annotations are not always exhaustive.

Multi-Object Tracking

Multi-object tracking (MOT) links detections across video frames to maintain object identities through motion, occlusion, and changing viewpoints.

See all 193 papers

Generative Semantic Multi-Object Tracking: A Large-Scale Benchmark and an MLLM-Driven Reasoning Framework
Jan 10, 2026Pan Liao, Feng Yang, Di Wu +3Multimodal Large Language Models MLLM Reasoning
QTrack: Query-Driven Reasoning for Multi-modal MOT
Mar 14, 2026Tajamul Ashraf, Tavaheed Tariq, Sonia Yadav +4Referring MOT Multimodal Visual Object Tracking
King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Gaash Research Lab, National Institute of Technology Srinagar, India · The University of Queensland, Australia
Referring Multi-Object Tracking with Diverse Dynamic Expressions
Jun 7, 2024Yani Zhang, Dongming Wu, Wencheng Han +3Referring MOT Multi-Object Tracking
School of Computer Science, National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, Wuhan 430072, China. · Multimedia Lab (MMLab), Department of Information Engineering, the Chinese University of Hong Kong, Hong Kong, China. · State Key Laboratory of Internet of Things for Smart City, Department of Computer and Information Science, University of Macau, Macau, China. +1

Jan 10, 2026J/K move · Enter open · S save

Generative Semantic Multi-Object Tracking: A Large-Scale Benchmark and an MLLM-Driven Reasoning Framework

Pan Liao, Feng Yang, Di Wu, Jinwen Yu+2

Multimodal Large Language Models MLLM Reasoning Multi-Object Tracking

Open paper AskPDF arXiv

Semantic Multi-Object Tracking (SMOT) is evolving from purely geometric localization toward comprehensive video understanding. However, existing paradigms predominantly rely on closed-set interaction tags and fragmented perception pipelines, creating a bottleneck that prevents the full utilization of Multi-modal Large Language Models (MLLMs) for dynamic scenes. In this paper, we elevate SMOT from rigid classification to an open-ended generative reasoning task. To support this paradigm shift, we introduce Grand-SMOT, a large-scale benchmark featuring high-density, dual-stream narratives. This dataset explicitly decouples micro-level individual dynamics from macro-level environmental contexts, directly resolving the semantic scarcity of prior tracking datasets. Furthermore, we propose LLMTrack, a unified MLLM-driven framework for dynamic SMOT. Guided by a verifiable ``Macro-Understanding-First'' mechanism, LLMTrack employs a Spatio-Temporal Fusion Module to compress discrete geometric trajectories into continuous semantic tokens, effectively suppressing temporal hallucinations in long-sequence tracking. Extensive experiments, utilizing a novel decoupled evaluation protocol, validate that LLMTrack achieves state-of-the-art geometric tracking robustness while delivering a qualitative leap in generative semantic reasoning. The code and datasets are publicly available at https://github.com/liaopan-lp/LLMTrack-GrandSMOT.

Abstract

Multi-Object Tracking

Explore similar work

Generative Semantic Multi-Object Tracking: A Large-Scale Benchmark and an MLLM-Driven Reasoning Framework

QTrack: Query-Driven Reasoning for Multi-modal MOT

Referring Multi-Object Tracking with Diverse Dynamic Expressions