VISTA: Vision-Language Integrated Semantic Triage Architecture for Online Face Clustering in Long Videos
Organizations: University of Glasgow, Glasgow, United Kingdom · University of Leeds, Leeds, United Kingdom · The University of Warwick, Coventry, United Kingdom
Abstract
Clustering faces by identity in long, continuous video streams is a challenging task that serves as a building block for human-centric video analysis. The majority of existing face clustering models are offline, i.e., they assume the entire data to be available altogether. In many practical scenarios, however, the entire data is either unavailable or poses computational challenges to be processed in its entirety. A handful of online methods that exist rely on geometric visual features and simple thresholding techniques, which fail when the faces exhibit significant appearance variations in long, unconstrained videos. To this end, we present VISTA, the first vision language framework for online face clustering in long videos. VISTA leverages semantic reasoning in a controlled, uncertainty aware fashion to maintain both high cluster purity and long range identity continuity. This makes it robust to the variations in illumination, pose, and costume changes that the faces and characters exhibit in long real world videos. Extensive experiments on the VPCD benchmark dataset (with 15 full episodes from The Big Bang Theory [2007-2019], Buffy the Vampire Slayer [1997-2003], and Sherlock [2010-2017]) demonstrate that VISTA achieves a robust precision and recall balance, consistently improving over the strongest matched online unsupervised baseline (COFC) under identical detector, embeddings, and evaluation protocol. Code and supplementary material: https://github.com/SkyJiashu/VISTA.
Figures & tables
| Method | Online? | Unsup.? | Core Mechanism | TBBT | Buffy | Sherlock |
|---|---|---|---|---|---|---|
| Controlled comparison: online, unsupervised, identical features and protocol (WCP / NMI / ID_Acc) | ||||||
| COFC ( Kulshreshtha and Guha, 2018 ) | ✓ | ✓ | Constrained Incremental | 92.13 / 86.97 / 89.74 | 64.60 / 66.28 / 64.31 | 76.22 / 50.56 / 71.27 |
| VISTA (Full Framework) | ✓ | ✓ | Spatio Temporal Semantic Triage | 96.97 / 91.25 / 93.94 | 87.39 / 79.30 / 82.82 | 80.45 / 51.19 / 68.10 |
| Literature reference: SENet-50 features † , reported numbers (WCP / NMI) | ||||||
| C1C † ( Kalogeiton and Zisserman, 2020 ) | ✗ | ✓ | Constrained 1NN | 89.20 / 87.40 | 66.30 / 68.80 | 76.30 / 50.30 |
| BCL † (AT) ( Tapaswi et al., 2019 ; Brown et al., 2021 ) | ✗ | ✗ | Ball Cluster Learning | 90.80 / 85.70 | 85.00 / 78.80 | - / - |
| Configuration | TBBT | Buffy | Sherlock |
|---|---|---|---|
| VISTA (Full Framework) | 96.97 / 91.25 / 93.94 | 87.39 / 79.30 / 82.82 | 80.45 / 51.19 / 68.10 |
| w/o Semantic Arbitration | 96.71 / 89.97 / 91.27 | 80.54 / 77.62 / 78.35 | 79.06 / 52.55 / 69.56 |
| w/o SSR | 94.83 / 89.29 / 91.60 | 74.75 / 74.47 / 73.31 | 78.36 / 52.62 / 71.43 |
| w/o Three Zone Gating | 96.78 / 90.80 / 92.32 | 80.17 / 77.03 / 76.58 | 80.27 / 51.49 / 64.63 |
| w/o Temporal Smoothing | 96.28 / 90.09 / 91.95 | 82.07 / 77.44 / 77.05 | 84.11 / 52.54 / 62.23 |
| Dataset | Frames | Total Time (s) | End to End FPS | Clustering FPS |
|---|---|---|---|---|
| TBBT (s01e01) | 32,990 | 4,655 | 7.09 | 2122.45 |
| Buffy (s05e01) | 62,620 | 7,978 | 7.85 | 2288.42 |
| Sherlock (e01) | 132,150 | 16,393 | 8.06 | 1842.30 |
| Dataset | Frame level | Tracklet level | ||
|---|---|---|---|---|
| Arb. / Total | Rate | Arb. / Total | Rate | |
| TBBT | 29,389 / 319,646 | 9.19% | 3,685 / 20,737 | 17.77% |
| Buffy | 64,222 / 474,210 | 13.54% | 8,198 / 35,079 | 23.37% |
| Sherlock | 47,919 / 421,899 | 11.36% | 5,797 / 30,448 | 19.04% |
| Overall | 141,530 / 1,215,755 | 11.64% | 17,680 / 86,264 | 20.50% |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Raw age range | fine_grained | coarse_grouping | broad_adult |
|---|---|---|---|
| 0–9 | <age_child> | <age_child> | <age_child> |
| 10–19 | <age_teen> | <age_teen> | <age_teen> |
| 20–29 | <age_20s> | <age_young_adult> | <age_adult> |
| 30–39 | <age_30s> | <age_adult> | <age_adult> |
| 40–49 | <age_40s> | <age_adult> | <age_adult> |
| 50–59 | <age_50s> | <age_senior> | <age_adult> |
| Attribute | Prompt Template | Labels |
| Gender | “A face of a {label}” | Male, Female |
| Clothing | “A photo of a person wearing a {label}” | T-shirt, Jacket, Shirt, Sweater, |
| Dress, Coat, Vest | ||
| Hair | “A close-up photo of a person with {label}” | Curly Hair, Straight Hair, Bald, |
| Short Hair, Long Hair |
| Attribute | Raw Label | Symbolic Token |
| Gender | Male | <gender_male> |
| Female | <gender_female> | |
| Clothing | T-shirt | <clothing_tshirt> |
| Jacket | <clothing_jacket> | |
| Shirt | <clothing_shirt> | |
| Sweater | <clothing_sweater> |
| Parameter | Value | WCP / NMI / ID_Acc | Bal. Avg. |
| age | broad_adult | 96.66 / 87.57 / 89.40 | 91.21 |
| age | coarse_grouping | 96.70 / 87.15 / 88.58 | 90.81 |
| age | fine_grained | 96.60 / 87.01 / 88.68 | 90.76 |
| ema_rep | 0.95 | 97.47 / 89.43 / 91.32 | 92.74 |
| ema_rep | 0.75 | 95.84 / 85.05 / 86.46 | 89.11 |
| queue | 300.0 | 96.97 / 87.65 / 89.17 | 91.26 |
| Parameter | Value | WCP / NMI / ID_Acc | Bal. Avg. |
|---|---|---|---|
| age | coarse_grouping | 84.85 / 75.24 / 76.98 | 79.02 |
| age | broad_adult | 84.18 / 74.50 / 76.02 | 78.23 |
| age | fine_grained | 81.62 / 72.17 / 73.49 | 75.76 |
| ema_rep | 0.95 | 85.18 / 77.22 / 79.68 | 80.69 |
| ema_rep | 0.75 | 81.87 / 70.67 / 71.24 | 74.59 |
| queue | 150.0 | 84.70 / 75.01 / 76.31 | 78.67 |
| Parameter | Value | WCP / NMI / ID_Acc | Bal. Avg. |
| age | broad_adult | 80.16 / 46.69 / 58.51 | 61.79 |
| age | fine_grained | 80.23 / 46.68 / 58.08 | 61.66 |
| age | coarse_grouping | 80.38 / 46.67 / 55.58 | 60.87 |
| ema_rep | 0.95 | 81.88 / 49.36 / 61.54 | 64.26 |
| ema_rep | 0.75 | 78.63 / 44.00 / 53.24 | 58.62 |
| queue | 150.0 | 80.04 / 46.72 / 58.56 | 61.77 |
| Dataset | Metric | Min | Max | Std |
|---|---|---|---|---|
| TBBT | WCP | 96.08 | 97.59 | 0.31 |
| NMI | 90.20 | 91.79 | 0.27 | |
| Buffy | WCP | 84.71 | 90.34 | 1.09 |
| NMI | 76.86 | 80.96 | 0.74 | |
| Sherlock | WCP | 79.22 | 83.44 | 0.78 |
| NMI | 48.90 | 53.37 | 0.78 |
| Weight | Value | Bal. Avg. (%) | ||
|---|---|---|---|---|
| TBBT | Buffy | Sherlock | ||
| 0.04 | 93.88 | 83.53 | 65.22 | |
| 0.08 | 93.81 | 82.42 | 65.70 | |
| 0.12 | 94.01 | 82.40 | 66.29 | |
| 0.02 | 93.93 | 82.82 | 65.63 | |
| 0.05 | 93.88 | 82.81 | 65.76 | |
| Length | WCP | NMI | ID_Acc | Precision | Recall | F1 |
|---|---|---|---|---|---|---|
| 12 | 92.27 | 77.39 | 77.88 | 81.16 | 96.19 | 87.84 |
| 24 | 92.83 | 77.92 | 79.14 | 81.16 | 96.19 | 87.84 |
| 48 | 91.89 | 78.72 | 81.05 | 81.16 | 96.19 | 87.84 |
| 96 | 91.23 | 78.50 | 80.79 | 81.16 | 96.19 | 87.84 |
| Hard-VLM | WCP | NMI | ID_Acc | #Pred/ep | #GT/ep |
|---|---|---|---|---|---|
| TBBT | 97.98 | 76.57 | 63.95 | 83.8 | 20.5 |
| Buffy | 91.01 | 72.99 | 60.64 | 108.5 | 27.0 |
| Sherlock | 84.67 | 44.98 | 40.65 | 136.0 | 12.0 |
| Online HAC | WCP | NMI | ID_Acc | #Pred/#GT |
|---|---|---|---|---|
| TBBT | 99.73 | 84.93 | 81.29 | |
| Buffy | 97.62 | 76.78 | 63.49 | |
| Sherlock | 91.19 | 49.28 | 40.56 |
| Metric | TBBT | Buffy | Sherlock |
|---|---|---|---|
| WCP | 0.18 | 0.15 | 0.67 |
| NMI | 0.41 | 0.15 | 0.30 |
| ID_Acc | 0.67 | 0.37 | 0.69 |
| Gap | Dataset | WCP | NMI | ID_Acc |
|---|---|---|---|---|
| Gender (M F) | TBBT | |||
| Buffy | ||||
| Sherlock | ||||
| Age (adult senior) | TBBT | |||
| Buffy | ||||
| Sherlock |
Clustering
Clustering groups data points by shared structure, developing methods to discover, validate, and interpret groups across diverse data and constraints.
See all 370 papers