PERSIST: Persistent-State Discrimination for Shot Boundary Detection

Tingyu Lin; Christian Stippel; Armin Dadras; Jakob Zenzmaier; Florian Kleber; Wolfgang Aigner; Robert Sablatnig

cs.CVAug 29, 2026

PERSIST: Persistent-State Discrimination for Shot Boundary Detection

Authors: Tingyu Lin, Christian Stippel, Armin Dadras, Jakob Zenzmaier, Florian Kleber, Wolfgang Aigner, Robert Sablatnig

Organizations: Computer Vision Lab TU Wien Vienna, Austria · Institute of Creative\Media/Technologies St. Pölten University of Applied Sciences St. Pölten, Austria · Ludwig Boltzmann Institute for Digital History Ludwig Boltzmann Gesellschaft Vienna, Austria

Abstract

Shot boundary detection (SBD) is widely treated as the localisation of local visual discontinuities, yet many false positives such as hand-held shake, illumination flicker, motion blur, occlusion, and damaged archival material produce equally sharp local change without introducing a new shot. We reformulate SBD as boundary semantic discrimination: a frame is favoured as a boundary only when its local change evidence is accompanied by a persistent update of the video's latent temporal state, rather than a transient excursion that returns to the surrounding trend. This persistence test is operationalised with a continuous latent state from a FiLM-conditioned sinusoidal representation network and a structured discriminator that combines three semantic cues, local change, transient impulse, and return-to-trend, into a single interpretable per-frame signal over a dual-rate temporal backbone. The resulting framework, PERSIST, turns every decision into an inspectable one: the persistence criterion is trained into the classifier, its per-frame effect stays readable from the gate triple, and its learned latent state is measurably boundary-discriminative. On a 2,727-video per-subtype diagnostic it removes 33-80% of flash, text-overlay, and archival false positives relative to an identically trained cue detector, and at matched true-transition recall it roughly halves TransNetV2's pseudo-event false positives on that diagnostic and cuts its false positives on ClipShots footage by about a quarter, while preserving recall. It does so while reaching parity with the strongest public detector across online, broadcast, short-form, and historical-archive transfer evaluations, under markedly stricter training: it learns from ClipShots real transitions only, whereas the anchor draws on additional corpora whose transitions are 85% synthetic. Code is available at https://github.com/linty5/PERSIST.

Figures & tables

Figure 1: On four real ClipShots cases a TransNetV2-cue detector’s per-frame probability (dashed) crosses threshold on all four including two pseudo events, whereas PERSIST ’s probability (solid) fires only on the two genuine transitions.

Figure 2: Architecture of PERSIST (left to right): a dual-rate TransNetV2-cell backbone feeds a fast-path evidence head ( $\mathrm{ev}$ , $p_{0}$ ) and a slow-conditioned FiLM-SIREN latent ( $\phi$ , teacher $u$ ), which a three-gate persistence discriminator ( $g_{c},g_{t},g_{r}$ ) combines into a per-frame suppression $g_{a}$ that yields the final score $p$ , with dashed nodes marking the training-only losses.

Method	ClipShots		BBC		AutoShot		HIST-GST12
	Fix-0.5	Oracle	Fix-0.5	Oracle	Fix-0.5	Oracle	Fix-0.5	Oracle
Pixel difference [ 33 ]	0.0351	0.5602	0.0074	0.8316	0.0683	0.7252	0.0000	0.8631
HSV histogram [ 1 ]	0.5737	0.6019	0.7786	0.9244	0.6792	0.7765	0.1544	0.6550
Edge change ratio [ 33 ]	0.3510	0.3813	0.5552	0.5660	0.3505	0.6561	0.6019	0.7117
DeepSBD-AlexNet [ 11 ]	0.6843	0.7456	0.8202	0.8770	0.7539	0.7604	0.8535	0.8767
DeepSBD-ResNet18 [ 11 , 29 ]	0.7555	0.7634	0.8622	0.8919	0.7303	0.7458	0.8611	0.8912

Table 1: Tier 1 literature positioning: F1 per dataset under the fixed-

0.50

and oracle-best thresholds; best per column bold , second-best underlined (

\dagger

faithful single-rate TransNetV2 on our ClipShots-only data;

\ddagger

three-seed probability-average ensemble).

	ClipShots		BBC		AutoShot		HIST-GST12
Method	Fix-0.5	Oracle	Fix-0.5	Oracle	Fix-0.5	Oracle	Fix-0.5	Oracle
TNV2-cue baseline	$.727{\pm}.006$	$.782{\pm}.007$	$.949{\pm}.003$	$.961{\pm}.003$	$\mathbf{.811}{\pm}.002$	$.813{\pm}.003$	$\mathbf{.928}{\pm}.004$	$.935{\pm}.001$
PERSIST	$\mathbf{.791}{\pm}.004$	$\mathbf{.794}{\pm}.003$	$\mathbf{.966}{\pm}.002$	$\mathbf{.967}{\pm}.001$	$.788{\pm}.009$	$\mathbf{.821}{\pm}.006$	$.923{\pm}.007$	$\mathbf{.946}{\pm}.003$
$\Delta$ (ours $-$ baseline)	$+0.065$	$+0.012$	$+0.018$	$+0.006$	$-0.023$	$+0.007$	$-0.005$	$+0.011$

Table 2: Tier 2 controlled comparison: the two rows share training data, optimiser, schedule, seeds, and evaluation pipeline and differ only in the head, with the best per column in bold and

\Delta

the signed gain of PERSIST over the baseline. Entries are mean

\pm

population std over seeds

11

1

and

21

	ClipShots		BBC		AutoShot		HIST-GST12
Variant	Fix-0.5	Oracle	Fix-0.5	Oracle	Fix-0.5	Oracle	Fix-0.5	Oracle
PERSIST	$.791{\pm}.004$	$.794{\pm}.003$	$.966{\pm}.002$	$.967{\pm}.001$	$\mathbf{.788}{\pm}.009$	$\mathbf{.821}{\pm}.006$	$\mathbf{.923}{\pm}.007$	$.946{\pm}.003$
w/o discriminator	$.790{\pm}.005$	$.793{\pm}.004$	$.967{\pm}.001$	$.968{\pm}.001$	$.763{\pm}.020$	$.816{\pm}.006$	$.916{\pm}.012$	$\mathbf{.949}{\pm}.004$
w/o $\phi$ shaping	$\mathbf{.794}{\pm}.002$	$\mathbf{.796}{\pm}.002$	$.966{\pm}.005$	$.967{\pm}.005$	$.769{\pm}.020$	$.815{\pm}.003$	$\mathbf{.923}{\pm}.012$	$.947{\pm}.003$
w/o dynamics auxiliary	$.787{\pm}.005$	$.792{\pm}.000$	$\mathbf{.969}{\pm}.000$	$.969{\pm}.001$	$.761{\pm}.030$	$.812{\pm}.004$	$\mathbf{.923}{\pm}.007$	$.946{\pm}.003$
Single-rate (no slow path)	$.788{\pm}.002$	$.790{\pm}.004$	$.966{\pm}.002$	$.968{\pm}.002$	$.765{\pm}.029$	$.816{\pm}.005$	$.919{\pm}.008$	$\mathbf{.949}{\pm}.002$

Table 3: Tier 3 single-component ablation (uniform val-best selection), each row removing one component from the full model except the last which removes all three latent-shaping components together, with the best per column in bold. Entries are mean

\pm

population std over seeds

11

1

and

21

; where arms tie at the printed precision all are bolded.

Figure 3: Space-time cube contrasting a true boundary with a pseudo event; the per-frame suppression $\mathrm{ev}-p$ is trajectory height (and, redundantly, lightness) above a no-suppression grey floor, with depth giving the local evidence $\mathrm{ev}$ , so the pseudo-event trajectory rises high while the true boundary stays on the floor.

Figure 4: False positives at matched true-transition recall: with all three detectors tuned to the same true-transition recall, PERSIST fires the fewest on both splits; on the synthetic diagnostic the false positives are labelled pseudo-events, on natural ClipShots they are false detections on real non-boundary footage (ratios relative to PERSIST as

\times

; matched-recall counts, not the fixed-

0.50

counts in the text).

Figure 5: Per-frame decision atlas on a sustained pseudo event, where the local boundary evidence $\mathrm{ev}$ (filled; labelled $\mathrm{ev}_{\mathrm{raw}}$ in the panel) stays high while the final probability $p$ (solid) is held near zero to open the suppression band $\mathrm{ev}-p$ , traced below by horizon rows for the suppression and the joint gate $g_{a}$ (darker meaning larger).

Dataset	Role	Videos	AST	GST	Frames
ClipShots train [ 29 ]	training source	4,876	110,575	34,810	26,649,843
ClipShots test [ 29 ]	in-domain test	500	4,903	2,301	3,241,806
BBC Planet Earth [ 27 ]	transfer test	11	4,708	136	809,827
AutoShot/SHOT [ 34 ]	transfer test	200	2,078	438	204,031
HISTORIAN-GST12 [ 13 ]	historical transfer	12	1,377	191	196,931

Table S1: Evaluation datasets at a glance (AST: abrupt shot transitions, cuts and jumps; GST: gradual transitions such as dissolve, fade, wipe, flip-over, swish pan).

Dataset	mean $\pm$ std (fix / oracle-best)	ENS (fix / oracle-best)	TNV2 (oracle-best)
ClipShots	$0.7913{\pm}.004$ / $0.7941{\pm}.003$	$0.8023$ / $0.8075$	$0.7794$
BBC	$0.9661{\pm}.002$ / $0.9669{\pm}.001$	$0.9700$ / $0.9707$	$0.9694$
AutoShot/SHOT	$0.7878{\pm}.009$ / $0.8206{\pm}.006$	$0.7953$ / $0.8290$	$0.8302$
HIST-GST12	$0.9227{\pm}.007$ / $0.9458{\pm}.003$	$0.9264$ / $0.9531$	$0.9435$
synth diag	$0.5193{\pm}.002$ / $0.5924{\pm}.008$	$0.5484$ / $0.6311$	$0.5568$

Table S2: Three-seed mean

\pm

std and ensemble (fix-0.5 / oracle-best F1; ENS is the three-seed probability-average ensemble; official TransNetV2 oracle-best shown for reference).

Model	Window	ClipShots	BBC	AutoShot	HIST-GST12
PERSIST	$64$	0.7937/0.7964	0.9672/0.9678	0.7992/0.8280	0.9319/0.9494
PERSIST	$100$	0.7890/0.7938	0.9725/0.9729	0.7785/0.8182	0.9171/0.9419
TNV2 repro (single-rate)	$64$	0.7260/0.7820	0.9517/0.9625	0.7981/0.8029	0.9305/0.9400
TNV2 repro (single-rate)	$100$	0.7287/0.7845	0.9558/0.9648	0.8016/0.8016	0.9266/0.9331

Table S3: Inference window-length control (single-model, seed

11

, fix-0.5 / oracle-best F1). Neither model benefits from the

100

-frame window the official anchor uses.

Return-gate offsets $\mathcal{D}$	ClipShots (fix / oracle-best)	synth FP
$\{2,4,6\}$ (short)	0.7849 / 0.7929	769
$\{4,8,12\}$ (default)	0.7937 / 0.7964	723
$\{8,16,24\}$ (long)	0.7890 / 0.7949	766

Table S4: Persistence-horizon sensitivity (single-model, seed

11

): ClipShots fix-0.5 / oracle-best F1 and the total synthetic pseudo-event false positives (

\downarrow

, threshold

0.5

). The default offsets are marginally best and all three stay far below the cue baseline (

1{,}391

) and the matched-recall TransNetV2 (

1{,}459

Model	ClipShots oracle	synth FP (flash/overlay)	natural FP	xdom oracle (BBC/AS/HIST)
PERSIST	0.7964	723 (329/80)	2147	0.968/0.828/0.949
Cue + hard negatives	0.7868	603 (146/196)	2447	0.966/0.808/0.936
Cue baseline (no hard neg.)	0.7733	887 (396/181)	2665	0.963/0.810/0.936

Table S5: Negative supervision vs the persistence formulation. Synthetic-diagnostic and natural false positives (

\downarrow

; both at matched true-transition recall,

0.7257

on the diagnostic and

0.8499

on the ClipShots test set), with cross-domain oracle-best F1. The hard-negative baseline is the cue head trained with rendered pseudo-event hard negatives; PERSIST uses none. All rows are seed

11

Figure S1: F1 versus decision threshold on the four datasets ( PERSIST : black curve; within-protocol TNV2-cue baseline: orange curve), with markers for the fixed-

0.50

and per-dataset oracle-best operating points and a strip beneath each axis shaded blue where PERSIST leads and yellow where the baseline leads.

fixed- $0.50$
Variant	ClipShots	BBC	AutoShot/SHOT	HIST-GST12
PERSIST	$.794/.785/.795$	$.967/.964/.967$	$.799/.776/.788$	$.932/.916/.921$
w/o discriminator	$.784/.790/.797$	$.966/.967/.968$	$.790/.743/.756$	$.928/.919/.900$
w/o $\phi$ shaping	$.792/.791/.797$	$.970/.959/.969$	$.795/.764/.748$	$.936/.907/.926$
w/o dynamics auxiliary	$.790/.792/.779$	$.969/.969/.969$	$.784/.719/.781$	$.926/.913/.931$
Single-rate (no slow path)	$.790/.785/.790$	$.968/.966/.964$	$.801/.730/.764$	$.928/.920/.908$

Table S6: Per-seed F1 behind the three-seed means of Tables 2 and 3 of the main paper, given as seed

11

/ seed

1

/ seed

21

under each protocol. AutoShot/SHOT at fixed-

0.50

carries by far the largest seed spread of the four corpora, which is why the statements that touch it are qualified in the main paper.

Method	clean cut (228)	jump cut (181)	dissolve (130)	fade-out (123)	fade-in (116)	wipe (64)
PERSIST	0.901	0.123	0.880	0.926	0.903	0.581
w/o discriminator	0.903	0.118	0.866	0.922	0.903	0.619
w/o $\phi$ shaping	0.905	0.132	0.870	0.926	0.903	0.583
w/o dynamics auxiliary	0.904	0.111	0.860	0.941	0.900	0.559
Single-rate (no slow path)	0.906	0.127	0.851	0.930	0.908	0.634
w/o all three shaping components	0.911	0.115	0.871	0.937	0.900	0.500

Table S7: Real-transition F1 (

\uparrow

) per synthetic subtype (per column, best in bold , second-best underlined ). The w/o

\phi

shaping row is evaluated at the val-best checkpoint of Section 4.1 of the main paper; three rows are tied for second on fade-in.

Method	flash (510)	fast pan (453)	overlay (391)	archival (349)	scratch (182)
PERSIST	329	36	80	270	8
w/o discriminator	363	37	82	282	7
w/o $\phi$ shaping	339	34	68	292	8
w/o dynamics auxiliary	374	33	64	259	8
Single-rate (no slow path)	409	40	82	301	8
w/o all three shaping components	384	37	66	288	11

Table S8: Pseudo-event false-positive counts (

\downarrow

) on the synthetic diagnostic split , per hard-negative subtype with sample sizes in the header; among the model variants the best is in bold and the second-best underlined . All rows are seed

11

at threshold

0.50

Model	seed $11$	seed $1$	seed $21$	mean $\pm$ std
PERSIST	$1.41$	$1.38$	$1.67$	$\mathbf{1.49}{\pm}0.13$
w/o discriminator	$1.09$	$1.44$	$1.39$	$1.31{\pm}0.16$
w/o $\phi$ shaping	$1.49$	$1.14$	$1.33$	$1.32{\pm}0.14$

Table S9:

\phi(t)

continuity diagnostic (ClipShots), three seeds: ratio of the mean one-step latent change at boundary steps to non-boundary steps, per seed and as mean

\pm

population std over seeds

11

1

21

, all at the val-best checkpoint. The absolute displacement is not comparable across runs, since nothing fixes the scale of

\phi

; only the within-run ratio is. The full model is highest, but the margins over both shaping ablations lie inside seed spread, so this readout corroborates the design rather than establishing it.

Category (frames)	$g_{\text{c}}$	$g_{\text{a}}$	$\mathrm{ev}$	$\widetilde{\mathrm{ev}}$	$p$
background ( $3.0$ M)	$0.000$	$0.000$	$0.013$	$0.013$	$0.005$
true cut / AST ( $9.7$ k)	$0.912$	$0.171$	$0.865$	$0.797$	$0.725$
gradual / GST ( $38.6$ k)	$0.595$	$0.112$	$0.621$	$0.576$	$0.605$
pseudo-event ( $119$ k)	$0.896$	$0.168$	$0.841$	$0.774$	$0.269$

Table S10: Gate firing by ground-truth clip type (full model, ClipShots): mean gate activations, raw and suppressed evidence, and final score per category.

	synthetic diagnostic		natural ClipShots
Arm	FP	paired	FP	paired
PERSIST (all three gates)	$\mathbf{726.3}{\pm}6.2$	–	$\mathbf{2207.0}{\pm}118.9$	–
w/o transient-impulse gate	$780.3{\pm}69.7$	$2/3$	$2342.0{\pm}112.8$	$3/3$
w/o return gate	$779.7{\pm}32.3$	$3/3$	$2293.3{\pm}64.9$	$3/3$
change gate only	$781.0{\pm}56.8$	$2/3$	$2308.0{\pm}101.6$	$3/3$

Table S11: Leave-one-gate-out, false positives at matched true-transition recall (three seeds, mean

\pm

population std; lower is better). “Paired” counts the seeds on which the arm is worse than the full model trained with the same seed. On natural footage every gate is load-bearing on every paired seed; on the rendered diagnostic the return gate is, while the other two arms are worse on two of three seeds. The main paper quotes the seed-

11

member of the full-model row,

723

and

2{,}147

, where this table reports the mean over the three seeds.

Arm	flash	fast pan	overlay	archival	scratch	total
PERSIST	$\mathbf{341.3}{\pm}13.4$	$\mathbf{32.3}{\pm}2.6$	$110.0{\pm}22.2$	$\mathbf{234.7}{\pm}27.1$	$\mathbf{8.0}{\pm}0.8$	$\mathbf{726.3}{\pm}6.2$
w/o transient gate	$398.0{\pm}7.0$	$35.7{\pm}3.4$	$\mathbf{88.7}{\pm}18.9$	$246.0{\pm}54.3$	$12.0{\pm}2.2$	$780.3{\pm}69.7$
w/o return gate	$371.3{\pm}23.7$	$32.0{\pm}1.6$	$105.3{\pm}31.5$	$262.7{\pm}52.7$	$8.3{\pm}1.2$	$779.7{\pm}32.3$
change gate only	$361.7{\pm}4.6$	$36.7{\pm}3.1$	$97.7{\pm}4.6$	$276.7{\pm}54.0$	$8.3{\pm}2.6$	$781.0{\pm}56.8$
w/o discriminator	$378.3{\pm}49.4$	$38.3{\pm}0.9$	$110.0{\pm}22.1$	$252.7{\pm}26.4$	$10.0{\pm}2.4$	$789.3{\pm}57.4$
cue baseline	$435.7{\pm}56.1$	$38.3{\pm}2.1$	$166.7{\pm}10.5$	$234.0{\pm}40.6$	$12.3{\pm}0.5$	$887.0{\pm}88.2$

Table S12: Leave-one-gate-out per pseudo-event subtype (synthetic diagnostic, matched true-transition recall, mean

\pm

population std over seeds

11

1

21

; lower is better). The fourth row removes the discriminator entirely and is the endpoint of the series; the last row is the identically trained cue baseline, included so the recall-matched per-subtype comparison of Section 4.4 of the main paper can be read here. Signs are not uniform across subtypes: every leave-one-out arm costs flash and archival suppression, while text overlay improves when a gate is dropped. Against the cue baseline the aggregate margin is

-160.7

(paired on all three seeds) and holds on flash, fast pan, text overlay and scratch, but archival degradation is level (

234.7

against

234.0

Figure S2: Decision atlas, true-cut control in the same per-frame signals and style as the main-paper figure: the final probability rises with the evidence and stays high through the transition, in contrast to the pseudo event shown in the main paper, where the score is held near zero.

Figure S3: PERSIST Explorer , an interactive visual-analytics tool built on the method’s per-frame outputs, scrubbed here to a true shot boundary: the linked filmstrip timeline, per-frame signal panel, rotatable 3D latent- $\phi(t)$ trajectory, and frame inspector with the three gate read-outs all update together as the analyst moves through time.

Figure S4: The same PERSIST Explorer session at a non-boundary frame. Scrubbed to a frame inside a shot, the change gate reads zero, the suppression band stays closed, and the latent- $\phi(t)$ trajectory rests within its within-shot cluster, the quiescent counterpart to the boundary frame of Figure S3 .

Temporal Video Understanding

Temporal video understanding studies how models track actions, events, and changes over time to answer questions, locate moments, and predict what happens next.

See all 371 papers

Apr 27, 2026

OmniShotCut: Holistic Relational Shot Boundary Detection with Shot-Query Transformer

Shot Boundary Detection (SBD) aims to automatically identify shot changes and divide a video into coherent shots. While SBD was widely studied in the literature, existing methods often produce non-interpretable boundaries on transitions, miss subtle yet harmful discontinuities, and rely on noisy, low-diversity annotations and outdated benchmarks. To alleviate these limitations, we propose OmniShotCut to formulate SBD as structured relational prediction, jointly estimating shot ranges with intra-shot relations and inter-shot relations, by a shot query-based dense video Transformer. To avoid imprecise manual labeling, we adopt a fully synthetic transition synthesis pipeline that automatically reproduces major transition families with precise boundaries and parameterized variants. We also introduce OmniShotCutBench, a modern wide-domain benchmark enabling holistic and diagnostic evaluation. Experiments on the benchmarks demonstrate the effectiveness and generality of our method.

Boyang Wang, Guangyi Xu, Jiahui Zhang +2

University of Virginia · University of Massachusetts Amherst

Topic pending

Ask PDF

Apr 30, 2026

TransVLM: A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions

Traditional Shot Boundary Detection (SBD) inherently struggles with complex transitions by formulating the task around isolated cut points, frequently yielding corrupted video shots. We address this fundamental limitation by formalizing the Shot Transition Detection (STD) task. Rather than searching for ambiguous points, STD explicitly detects the continuous temporal segments of transitions. To tackle this, we propose TransVLM, a Vision-Language Model (VLM) framework for STD. Unlike regular VLMs that predominantly rely on spatial semantics and struggle with fine-grained inter-shot dynamics, our method explicitly injects optical flow as a critical motion prior at the input stage. Through a simple yet effective feature-fusion strategy, TransVLM directly processes concatenated color and motion representations, significantly enhancing its temporal awareness without incurring any additional visual token overhead on the language backbone. To overcome the severe class imbalance in public data, we design a scalable data engine to synthesize diverse transition videos for robust training, alongside a comprehensive benchmark for STD. Extensive experiments demonstrate that TransVLM achieves superior overall performance, outperforming traditional heuristic methods, specialized spatiotemporal networks, and top-tier VLMs. This work has been deployed to production. For more related research, please visit HeyGen Research (https://www.heygen.com/research) and HeyGen Avatar-V (https://www.heygen.com/research/avatar-v-model). Project page: https://chence17.github.io/TransVLM/

Ce Chen, Yi Ren, Yuanming Li +5

University of Melbourne · 2HeyGen Research · 3Nanyang Technological University

Vision-Language Models Change-Point Detection Temporal Video Understanding

Ask PDF

Sep 27, 2026

Groupwise Selective State-Space Filtering for Accurate and Streaming Action Boundary Detection

Action boundary detection partitions untrimmed video into intervals without assigning action classes. We present a boundary-detection adapter operating on pre-extracted video features, learning temporal representations via groupwise selective scans. Learned group fusion and temporal modeling convert these into transition scores, which are decoded into boundary timestamps. Trained with boundary-time supervision, the class-agnostic model is evaluated on Breakfast, GTEA, and 50Salads using temporal tolerances and bipartite matching, achieving boundary

F_1

scores of 0.457, 0.622, and 0.611. A stateful variant enables feature-streaming inference with zero neural look-ahead, one-sample peak confirmation, and bounded memory. Downstream systems can subsequently assign s

Mustafa Bora Çelik

Department of Electrical and Electronics Engineering, Ankara Medipol University, Ankara, Turkey

Temporal Action Segmentation Selective SSMs Temporal Action Detection

Ask PDF

Abstract

Temporal Video Understanding

Explore similar work

OmniShotCut: Holistic Relational Shot Boundary Detection with Shot-Query Transformer

TransVLM: A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions

Groupwise Selective State-Space Filtering for Accurate and Streaming Action Boundary Detection