Organizations: University of Glasgow, Glasgow, United Kingdom · University of Leeds, Leeds, United Kingdom · The University of Warwick, Coventry, United Kingdom
Clustering faces by identity in long, continuous video streams is a challenging task that serves as a building block for human-centric video analysis. The majority of existing face clustering models are offline, i.e., they assume the entire data to be available altogether. In many practical scenarios, however, the entire data is either unavailable or poses computational challenges to be processed in its entirety. A handful of online methods that exist rely on geometric visual features and simple thresholding techniques, which fail when the faces exhibit significant appearance variations in long, unconstrained videos. To this end, we present VISTA, the first vision language framework for online face clustering in long videos. VISTA leverages semantic reasoning in a controlled, uncertainty aware fashion to maintain both high cluster purity and long range identity continuity. This makes it robust to the variations in illumination, pose, and costume changes that the faces and characters exhibit in long real world videos. Extensive experiments on the VPCD benchmark dataset (with 15 full episodes from The Big Bang Theory [2007-2019], Buffy the Vampire Slayer [1997-2003], and Sherlock [2010-2017]) demonstrate that VISTA achieves a robust precision and recall balance, consistently improving over the strongest matched online unsupervised baseline (COFC) under identical detector, embeddings, and evaluation protocol. Code and supplementary material: https://github.com/SkyJiashu/VISTA.
Figures & tables
Figure 1. The VISTA pipeline. Stage 1 extracts geometric and VLM features. Stage 2 forms intra shot tracks. Stage 3 performs inter shot association via SSR mapping and dynamic three zone semantic triage.
Figure 2. Detailed mechanisms of VISTA. (a) Three Zone Semantic Triage with GMM dynamic thresholding. (b) SSR maps raw predictions to stable tokens. (c) Semantic Arbitration via temporal voting compares symbolic profiles.
Method
Online?
Unsup.?
Core Mechanism
TBBT
Buffy
Sherlock
Controlled comparison: online, unsupervised, identical features and protocol (WCP / NMI / ID_Acc)
COFC ( Kulshreshtha and Guha, 2018 )
✓
✓
Constrained Incremental
92.13 / 86.97 / 89.74
64.60 / 66.28 / 64.31
76.22 / 50.56 / 71.27
VISTA (Full Framework)
✓
✓
Spatio Temporal Semantic Triage
96.97 / 91.25 / 93.94
87.39 / 79.30 / 82.82
80.45 / 51.19 / 68.10
Literature reference: SENet-50 features † , reported numbers (WCP / NMI)
C1C † ( Kalogeiton and Zisserman, 2020 )
✗
✓
Constrained 1NN
89.20 / 87.40
66.30 / 68.80
76.30 / 50.30
BCL † (AT) ( Tapaswi et al., 2019 ; Brown et al., 2021 )
✗
✗
Ball Cluster Learning
90.80 / 85.70
85.00 / 78.80
- / -
Table 1. Face clustering performance. Top: controlled comparison, VISTA vs. COFC under identical detector (RetinaFace), embeddings (FaceNet), and protocol (WCP / NMI / ID_Acc %). Bottom: literature reference numbers with SENet-50 features † (WCP / NMI only); feature/protocol differences make these reference points, not controlled comparisons. BCL (AT)/(OC) denote the VPCD protocols with unknown/oracle cluster counts as reported by Brown et al. ( Brown et al., 2021 ) ; BCL is supervised and not streaming. Bold: best in controlled comparison.
Configuration
TBBT
Buffy
Sherlock
VISTA (Full Framework)
96.97 / 91.25 / 93.94
87.39 / 79.30 / 82.82
80.45 / 51.19 / 68.10
w/o Semantic Arbitration
96.71 / 89.97 / 91.27
80.54 / 77.62 / 78.35
79.06 / 52.55 / 69.56
w/o SSR
94.83 / 89.29 / 91.60
74.75 / 74.47 / 73.31
78.36 / 52.62 / 71.43
w/o Three Zone Gating
96.78 / 90.80 / 92.32
80.17 / 77.03 / 76.58
80.27 / 51.49 / 64.63
w/o Temporal Smoothing
96.28 / 90.09 / 91.95
82.07 / 77.44 / 77.05
84.11 / 52.54 / 62.23
Table 2. Ablation study of semantic components (WCP / NMI / ID_Acc %).
Figure 3. Timeline level qualitative comparison on Sherlock . Top: ground truth; Middle: COFC; Bottom: VISTA. Each row represents a character, with colors indicating predicted clusters. Compared to COFC’s cross identity confusion, VISTA maintains cleaner global identity organization and better separates visually similar characters (e.g., Lestrade and Mycroft).
Dataset
Frames
Total Time (s)
End to End FPS
Clustering FPS
TBBT (s01e01)
32,990
4,655
7.09
2122.45
Buffy (s05e01)
62,620
7,978
7.85
2288.42
Sherlock (e01)
132,150
16,393
8.06
1842.30
Table 3. Throughput analysis of the VISTA pipeline
Dataset
Frame level
Tracklet level
Arb. / Total
Rate
Arb. / Total
Rate
TBBT
29,389 / 319,646
9.19%
3,685 / 20,737
17.77%
Buffy
64,222 / 474,210
13.54%
8,198 / 35,079
23.37%
Sherlock
47,919 / 421,899
11.36%
5,797 / 30,448
19.04%
Overall
141,530 / 1,215,755
11.64%
17,680 / 86,264
20.50%
Table 4. Semantic arbitration invocation rate. CLIP/DeepFace attributes are extracted up front for every crop (Stage 1); this table measures how often the arbitration step fires during association.
Figure 4. Distribution of VLM intervention and rejection rates across different demographic groups.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Raw age range
fine_grained
coarse_grouping
broad_adult
0–9
<age_child>
<age_child>
<age_child>
10–19
<age_teen>
<age_teen>
<age_teen>
20–29
<age_20s>
<age_young_adult>
<age_adult>
30–39
<age_30s>
<age_adult>
<age_adult>
40–49
<age_40s>
<age_adult>
<age_adult>
50–59
<age_50s>
<age_senior>
<age_adult>
Appendix
Table S1. Age symbol configurations. Raw DeepFace age predictions are mapped to discrete tokens via one of three schemes. fine_grained preserves decade level distinctions for adults; coarse_grouping merges the 30s and 40s into a single <age_adult> token; broad_adult further collapses the full 20-59 range.
Attribute
Prompt Template
Labels
Gender
“A face of a {label}”
Male, Female
Clothing
“A photo of a person wearing a {label}”
T-shirt, Jacket, Shirt, Sweater,
Dress, Coat, Vest
Hair
“A close-up photo of a person with {label}”
Curly Hair, Straight Hair, Bald,
Short Hair, Long Hair
Appendix
Table S2. CLIP zero shot classification prompts and label vocabularies.
Attribute
Raw Label
Symbolic Token
Gender
Male
<gender_male>
Female
<gender_female>
Clothing
T-shirt
<clothing_tshirt>
Jacket
<clothing_jacket>
Shirt
<clothing_shirt>
Sweater
<clothing_sweater>
Appendix
Table S3. Symbol Registry: raw label to symbolic token mappings.
Parameter
Value
WCP / NMI / ID_Acc
Bal. Avg.
age
broad_adult
96.66 / 87.57 / 89.40
91.21
age
coarse_grouping
96.70 / 87.15 / 88.58
90.81
age
fine_grained
96.60 / 87.01 / 88.68
90.76
ema_rep
0.95
97.47 / 89.43 / 91.32
92.74
ema_rep
0.75
95.84 / 85.05 / 86.46
89.11
queue
300.0
96.97 / 87.65 / 89.17
91.26
Appendix
Table S4. TBBT parameter effects under dataset specific marginal analysis. Values indicate average performance when the specific parameter is fixed while others vary.
Parameter
Value
WCP / NMI / ID_Acc
Bal. Avg.
age
coarse_grouping
84.85 / 75.24 / 76.98
79.02
age
broad_adult
84.18 / 74.50 / 76.02
78.23
age
fine_grained
81.62 / 72.17 / 73.49
75.76
ema_rep
0.95
85.18 / 77.22 / 79.68
80.69
ema_rep
0.75
81.87 / 70.67 / 71.24
74.59
queue
150.0
84.70 / 75.01 / 76.31
78.67
Appendix
Table S5. Buffy parameter effects under dataset specific marginal analysis.
Parameter
Value
WCP / NMI / ID_Acc
Bal. Avg.
age
broad_adult
80.16 / 46.69 / 58.51
61.79
age
fine_grained
80.23 / 46.68 / 58.08
61.66
age
coarse_grouping
80.38 / 46.67 / 55.58
60.87
ema_rep
0.95
81.88 / 49.36 / 61.54
64.26
ema_rep
0.75
78.63 / 44.00 / 53.24
58.62
queue
150.0
80.04 / 46.72 / 58.56
61.77
Appendix
Table S6. Sherlock parameter effects under dataset specific marginal analysis.
Figure S1. Frame level semantic arbitration invocation rate per dataset and overall. The dynamic triage mechanism restricts arbitration to a small fraction of ambiguous frames, preserving near real time processing speeds.
Figure S2. GMM bimodal fitting of track to cluster cosine similarity distributions across all 15 episodes (86,230 total scores). Each panel shows the empirical histogram overlaid with the fitted two component GMM. The dashed red and solid blue curves represent the different identity and same identity components, respectively. Vertical dotted lines indicate the derived τlow and τhigh thresholds after clipping.
Weight
Value
Bal. Avg. (%)
TBBT
Buffy
Sherlock
λga
0.04
93.88
83.53
65.22
0.08
93.81
82.42
65.70
0.12
94.01
82.40
66.29
λcloth
0.02
93.93
82.82
65.63
0.05
93.88
82.81
65.76
Appendix
Table S8. Cross dataset marginal effects of arbitration weights. Each row averages over 81 configurations sharing that specific weight value. Bal. Avg. = (WCP + NMI + ID_Acc) / 3.
Length
WCP
NMI
ID_Acc
Precision
Recall
F1
12
92.27
77.39
77.88
81.16
96.19
87.84
24
92.83
77.92
79.14
81.16
96.19
87.84
48
91.89
78.72
81.05
81.16
96.19
87.84
96
91.23
78.50
80.79
81.16
96.19
87.84
Appendix
Table S9. Intra shot tracklet length ablation, averaged over all 15 episodes (WCP / NMI / ID_Acc / Precision / Recall / F1 %). Upstream detection metrics (Precision, Recall, F1) are naturally unaffected by downstream aggregation logic.
Figure S3. Local clustering purity for Mycroft and Sherlock (Green: true positives, Red: false positives). VISTA maintains higher purity than COFC across challenging pose, illumination, and motion blur variations, minimizing identity confusion.
Hard-VLM
WCP
NMI
ID_Acc
#Pred/ep
#GT/ep
TBBT
97.98
76.57
63.95
83.8
20.5
Buffy
91.01
72.99
60.64
108.5
27.0
Sherlock
84.67
44.98
40.65
136.0
12.0
Appendix
Table S10. Naive VLM hard constraint baseline (%). #Pred/ep and #GT/ep are per episode averages by series.
Online HAC
WCP
NMI
ID_Acc
#Pred/#GT
TBBT
99.73
84.93
81.29
4.5×
Buffy
97.62
76.78
63.49
9.5×
Sherlock
91.19
49.28
40.56
39.8×
Appendix
Table S11. Sequential online HAC baseline (%). The last column is the ratio of predicted clusters to ground truth clusters per episode.
Metric
TBBT
Buffy
Sherlock
WCP
0.18
0.15
0.67
NMI
0.41
0.15
0.30
ID_Acc
0.67
0.37
0.69
Appendix
Table S12. Max − min variation (pp) across the 3×3 warm up threshold grid (all other settings fixed).
Gap
Dataset
WCP
NMI
ID_Acc
Gender (M − F)
TBBT
−0.5
+13.1
+1.7
Buffy
+2.1
+14.7
+5.0
Sherlock
−6.6
−19.8
−1.4
Age (adult − senior)
TBBT
+2.0
+12.0
+13.4
Buffy
−2.3
+4.7
−0.6
Sherlock
−6.6
+13.4
+1.1
Appendix
Table S13. Subgroup clustering quality gaps (pp) on the IoU matched subset using VLM predicted demographic groups. Positive values indicate the first group scores higher.
Video Large Language Models (Video-LLMs) require continual learning to adapt to non-stationary real-world data. However, existing benchmarks fall short of evaluating modern foundation models: many still rely on models without large-scale pre-training, and prevailing benchmarks typically partition a single dataset into sub-tasks, resulting in high task redundancy and negligible forgetting on pre-trained Video-LLMs. To address these limitations, we propose CL-VISTA, a benchmark tailored for continual video understanding of Video-LLMs. By curating 8 diverse tasks spanning perception, understanding, and reasoning, CL-VISTA induces substantial distribution shifts that effectively expose catastrophic forgetting. To systematically assess CL methods, we establish a comprehensive evaluation framework comprising 6 distinct protocols across 3 critical dimensions: performance, computational efficiency, and memory footprint. Notably, the performance dimension incorporates a general video understanding assessment to assess whether CL methods genuinely enhance foundational intelligence or merely induce task-specific overfitting. Extensive benchmarking of 10 mainstream CL methods reveals a fundamental trade-off: no single approach achieves universal superiority across all dimensions. Methods that successfully mitigate catastrophic forgetting tend to compromise generalization or incur prohibitive computational and memory overheads. We hope CL-VISTA provides critical insights for advancing continual learning in multimodal foundation models.
Haiyang Guo, Yichen Shi, Fei Zhu +6
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · MAIS, Institute of Automation, Chinese Academy of Sciences · Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, CAS +2
The pervasive growth of digital content, specifically short videos on social media platforms, has significantly altered how topics are discussed and understood in public discourse. In this work, we advance automated visual theme detection by assessing zero-shot and clustering capabilities on social media data. (1) We evaluated the capabilities of notable VLMs such as VideoChatGPT, PandaGPT, and VideoLLava using zero-shot image classification and compared their performance to the baseline provided by frame-wise CLIP image classification. (2) By treating clustering as a minimum cost multicut problem, we aim to uncover insightful patterns in an unsupervised manner. For both analysis strategies, we provide extensive evaluations and practical guidance to practitioners. While VLMs are currently not able to detect climate change specific classes, the clustering results are distinct visual frames. %Given that VLMs are not currently capable to grasp the climate change discourse, we focus the clustering evaluation of image embedding models. We find that both ConvNeXt V2 and DINOv2 produce meaningful clusters, with DINOv2 focusing more on style differences and abstract categories, while ConvNeXt V2 clusters differ in more fine-grained ways. Code available at https://github.com/KathPra/ClimateVID.git.
Shiqi Xu, Moritz Burmester, Katharina Prasse +3
University of Mannheim · Technical University of Munich · University of Mannheim, Max-Planck-Institute for Informatics, Saarland Informatics Campus
Vision-language models (VLMs) have made substantial progress in long-video understanding, with standard backbone models typically answering questions from frames sampled across the full video. However, as videos become longer, the full-video context inevitably contains more question-irrelevant temporal content, which can distract the model from the evidence needed to answer a specific question. We empirically find that focusing the visual input on short annotated clue intervals containing question-relevant evidence consistently improves prediction accuracy across model scales compared with using the corresponding full videos, while requiring fewer input frames. Based on this finding, we introduce Clue-OPSD, a clue-privileged on-policy self-distillation framework for long-video understanding. During training, a full-video student learns from a self-teacher conditioned on the corresponding clue interval by aligning their next-token distributions along student-generated trajectories. Clue-OPSD thus uses clue intervals as privileged supervision without relying on ground-truth answer labels, while requiring no clue annotations or additional modules at inference time. Extensive experiments across multiple long-video understanding benchmarks and Qwen3.5 model scales demonstrate consistent improvements over the corresponding backbone models and strong performance against supervised post-training baselines.
Kaishen Wang, Dongdi Zhao, Yijun Liang +4
University of Maryland, College Park · Johns Hopkins University