cs.CVAug 18, 2026

Probing Association Instability with Track-State Perturbations for Clip-Level Active Learning in Query-Propagation Multi-Object Tracking

Authors: Riku Inoue, Shogo Sato, Kazuhiko Murasaki, Tomoyasu Shimada, Toshihiko Nishimura, Ryuichi Tanida

Organizations: Human Informatics Laboratories NTT, Inc. Kanagawa, Japan

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Training query-propagation end-to-end multi-object tracking (MOT) models requires dense bounding-box and identity annotations across video sequences, making dataset construction expensive. Clip-level active learning reduces this cost by selecting video clips for annotation, but prior acquisition criteria based on output-level temporal uncertainty may miss clips whose informativeness comes from association instability in propagated track states. We propose QPID (Query-Propagation Instability and Diversity), a clip acquisition method for query-propagation MOT that targets association instability in propagated track states. QPID estimates this instability by applying two-sided perturbations to internal track states and measuring prediction differences from a clean reference branch. The key idea is that, in stable clips, each propagated track should continue to follow the same target under small perturbations, whereas in ambiguous clips, small changes in the track state can alter which target the track follows, leading to changes in localization or confidence. QPID measures these perturbation-induced prediction differences with two metrics: Localization Drift and Entropy-Weighted Confidence Discrepancy. These metrics are aggregated into a clip-level association-instability score. To avoid redundant uncertainty-only selection, QPID selects a representative annotation batch from high-instability clips using Uncertainty-Weighted Visual Coverage with track-level visual prototypes. Experiments on DanceTrack and SportsMOT with MeMOTR and SambaMOTR show that QPID achieves strong performance compared with active learning baselines under the same annotation budget.

Figures & tables

Figure 1: Association instability via two-sided perturbations to internal track states. QPID applies small two-sided perturbations ( ±ε\pm\varepsilon ) to internal track states, consisting of track query embeddings and reference points, and compares the resulting predictions with a clean branch. Stable overlap between the clean and perturbed boxes indicates low association instability, whereas localization drift or association switches indicate high association instability. Green denotes the clean box, and red and blue dashed boxes denote the perturbed boxes for the track prediction visualized in each example.
Figure 2: Overview of QPID. (a) Two-sided track-state perturbation. QPID applies positive and negative perturbations along the same sampled direction to internal track states and compares the resulting predictions with a clean reference branch. Perturbed states are not propagated to subsequent frames, so the score reflects local perturbation-induced prediction differences. (b) Clip acquisition pipeline. QPID aggregates Localization Drift and Entropy-Weighted Confidence Discrepancy into a clip-level association-instability score, forms a high-instability candidate pool, and selects a representative annotation batch by Uncertainty-Weighted Visual Coverage with track-level visual prototypes.
Figure 3: Round-wise performance on DanceTrack under the 5%+5% schedule. The upper and lower rows show results using MeMOTR and SambaMOTR, respectively. From left to right, we report HOTA, AssA, and IDF1 of QPID and baseline methods.
Figure 4: Round-wise performance on SportsMOT under the 5%+5% schedule. The upper and lower rows show results using MeMOTR and SambaMOTR, respectively.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Visualization of selected frame positions on DanceTrack at Round 1 under the 5%+5% schedule. From left to right, we show QPID, w/o Uncertainty-Weighted Visual Coverage , and Entropy. Each dot denotes one selected frame. The horizontal axis shows the normalized temporal position within each sequence, and the vertical axis shows the sequence identity. QPID shows less redundant concentration than Entropy and the variant without Uncertainty-Weighted Visual Coverage.
Figure 6: Qualitative comparison of stable and unstable clips under perturbations on DanceTrack. The top row shows clips with low U(c)U(c) and the bottom row shows clips with high U(c)U(c) . Green, blue, and red boxes denote the clean prediction and the two perturbed predictions for the most unstable object.
Figure 7: Qualitative comparison of stable and unstable clips under perturbations on SportsMOT. The top row shows clips with low U(c)U(c) and the bottom row shows clips with high U(c)U(c) . Green, blue, and red boxes denote the clean prediction and the two perturbed predictions for the most unstable object.
More in this topic

Multi-Object Tracking

Multi-object tracking (MOT) links detections across video frames to maintain object identities through motion, occlusion, and changing viewpoints.

See all 193 papers

Explore similar work

CardsList
May 11, 2026

Clip-level Uncertainty and Temporal-aware Active Learning for End-to-End Multi-Object Tracking

Multi-Object Tracking (MOT) in dynamic environments relies on robust temporal reasoning to maintain consistent object identities over time. Transformer-based end-to-end MOT models achieve strong performance by explicitly modeling temporal dependencies, yet training them requires extensive bounding-box and identity annotations. Given the high labeling cost and strong redundancy in videos, Active Learning (AL) is an effective approach to improve annotation efficiency. However, existing AL methods for MOT primarily operate at the frame level, which is structurally misaligned with modern end-to-end trackers whose inference and training rely on multi-frame clips. To bridge this gap, we formulate clip-level active learning and propose Clip-level Uncertainty and Temporal-aware Active Learning (CUTAL). In contrast to frame-based approaches, CUTAL scores each clip using uncertainty metrics derived from multi-frame predictions to capture inter-frame correspondence ambiguities, while enforcing temporal diversity to select an informative and non-redundant subset. Experiments show that CUTAL achieves stronger overall performance than baselines at the same label budgets across MeMOTR and SambaMOTR. Notably, CUTAL achieves performance comparable to full supervision for MeMOTR on both datasets using only 50% of the labeled training data.

Riku Inoue, Shogo Sato, Kazuhiko Murasaki +3

NTT, Inc., Kanagawa, Japan

Active LearningUncertainty SamplingMulti-Object Tracking
AskPDF
Mar 14, 2026

QTrack: Query-Driven Reasoning for Multi-modal MOT

Category-level multi-object trackers estimate trajectories for all detected instances, whereas some applications require propagating only targets specified by a user. We study offline, reference-grounded video tracking: given an ordered video clip, a natural-language query, and the reference-frame boxes of the specified targets, a model predicts their subsequent trajectories. We introduce QTrack, a vision-language policy trained with structured trajectory rewards and Temporal Perception-Aware Policy Optimization (TAPO). TAPO contrasts the likelihood of a generated response under the observed clip and a reference-frame repetition, encouraging the output to depend on visual evidence beyond the reference frame. We also introduce RMOT26, a controlled diagnostic benchmark constructed from existing tracking datasets. Under the reported protocol, QTrack obtains a motion- consistency score of 0.30, MOTP of 0.75, center-location error of 44.61 pixels, and normalized distance error of 0.39; its motion-consistency score ties the strongest comparison method. Component ablations suggest complementary effects from the motion reward and temporal contrast, while an exploratory four- clip, 50-frame evaluation examines transfer beyond the primary short-clip setting. The results support further study of temporal-evidence learning in trajectory- generating vision-language models. Code and data are available at https://github.com/gaash-lab/QTrack

Tajamul Ashraf, Tavaheed Tariq, Sonia Yadav +4

King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Gaash Research Lab, National Institute of Technology Srinagar, India · The University of Queensland, Australia

Referring MOTMultimodal Visual Object TrackingMulti-Object TrackingVision-Language ModelsMultimodal Reasoning
AskPDF
Apr 1, 2026

Out of Sight, Out of Track: Adversarial Attacks on Propagation-based Multi-Object Trackers via Query State Manipulation

Recent Tracking-by-Query-Propagation (TBP) methods have advanced Multi-Object Tracking (MOT) by enabling end-to-end (E2E) pipelines with long-range temporal modeling. However, this reliance on query propagation introduces unexplored architectural vulnerabilities to adversarial attacks. We present FADE, a novel attack framework designed to exploit these specific vulnerabilities. FADE employs two attack strategies targeting core TBP mechanisms: (i) Temporal Query Flooding: Generates spurious temporally consistent track queries to exhaust the tracker's limited query budget, forcing it to terminate valid tracks. (ii) Temporal Memory Corruption: Directly attacks the query updater's memory by severing temporal links via state de-correlation and erasing the learned feature identity of matched tracks. Furthermore, we introduce a differentiable pipeline to optimize these attacks for physical-world realizability by leveraging simulations of advanced perception sensor spoofing. Experiments on MOT17 and MOT20 benchmarks demonstrate that FADE is highly effective against state-of-the-art TBP trackers, causing significant identity switches and track terminations.

Halima Bouzidi, Haoyu Liu, Yonatan Gizachew Achamyeleh +2

University of California, Irvine, CA, USA

Adversarial AttacksMulti-Object TrackingPhysical Adversarial Attacks
AskPDF