Let's Split Up: Zero-Shot Classifier Edits for Fine-Grained Video Understanding

Kaiting Liu; Hazel Doughty

cs.CVFeb 18, 2026

Let's Split Up: Zero-Shot Classifier Edits for Fine-Grained Video Understanding

Authors: Kaiting Liu, Hazel Doughty

Organizations: Leiden University

Abstract

Video recognition models are typically trained on fixed taxonomies which are often too coarse, collapsing distinctions in object, manner or outcome under a single label. As tasks and definitions evolve, such models cannot accommodate emerging distinctions and collecting new annotations and retraining to accommodate such changes is costly. To address these challenges, we introduce category splitting, a new task where an existing classifier is edited to refine a coarse category into finer subcategories, while preserving accuracy elsewhere. We propose a zero-shot editing method that leverages the latent compositional structure of video classifiers to expose fine-grained distinctions without additional data. We further show that low-shot fine-tuning, while simple, is highly effective and benefits from our zero-shot initialization. Experiments on our new video benchmarks for category splitting demonstrate that our method substantially outperforms vision-language baselines, improving accuracy on the newly split categories without sacrificing performance on the rest. Project page: https://kaitingliu.github.io/Category-Splitting/.

Figures & tables

Figure 1: Category splitting aims to edit a trained video classifier by dividing a coarse label into multiple fine-grained subcategories, while keeping all other predictions unchanged. The challenge is to achieve this without retraining the full model and with zero or very few labels.

Figure 2: Zero-shot Category Splitting . Given a trained video classifier, our goal is to split a coarse category (e.g. pushing ) into fine-grained subcategories without any video data. Modifier retrieval first exposes compositional structure in the video classifier’s classification head from which it builds a dictionary of modifier vectors. The classifier is then edited by retrieving the appropriate modifier vector and adding it to the coarse category’s weight vector to create a new fine-grained subcategory. To generalize to unseen modifiers, modifier alignment learns a lightweight mapping from modifier text to modifier vectors, using category text/weight vectors as additional supervision.

Figure 3: Low-shot Category Splitting . We edit the model by replacing the coarse category $c$ with $\theta^{\prime}_{head}$ . This head is fine-tuned with as little as one video per fine-grained subcategory, initialized with our zero-shot approach.

Benchmark	Subset	categories	videos	coarse	fine	generality	locality	Example
		Base Model		Category Splitting		Evaluation		Example
SSv2-Split	A	119	169K	27	92	429	24347	bending $\rightarrow$ { bending so it deforms , bending until it
SSv2-Split	B	121	169K	27	94	403	24373	breaks , trying to bend something unbendable }
FineGym-Split	A	178	23K	23	155	173	7988	handspring forward $\rightarrow$ { handspring forward 1.5 turn ,
FineGym-Split	B	162	23K	19	143	204	7957	handspring forward 1 turn , handspring forward no turn }

Table 1: Statistics of Category Splitting Datasets. We create meta-datasets SSV2-Split and FineGym-Split, each with two subsets (A and B). For each subset we report the number of categories and videos used to train the base model, the number of coarse categories eligible for splitting, the number of resulting fine-grained subcategories we can split into, the average number of videos used to evaluate generality and locality per category split as well as an example coarse -to- fine split.

	SSv2-Split						FineGym-Split
	Subset A			Subset B			Subset A			Subset B
Method	Gen.	Loc.	Mean	Gen.	Loc.	Mean	Gen.	Loc.	Mean	Gen.	Loc.	Mean
CLIP ( Radford et al., 2021 )	27.6	100.0	63.8	30.7	100.0	65.4	12.1	100.0	56.1	7.2	100.0	53.6
FLAIR ( Xiao et al., 2025 )	30.6	100.0	65.3	28.4	100.0	64.2	18.0	100.0	59.0	11.6	100.0	55.8
FG-CLIP ( Xie et al., 2025 )	30.9	100.0	65.4	30.8	100.0	65.4	19.4	100.0	59.7	13.9	100.0	56.9
VideoCLIP-XL ( Wang et al., 2024a )	28.6	100.0	64.3	29.9	100.0	64.9	18.0	100.0	59.0	8.2	100.0	54.1

Table 2: Comparative Zero-Shot Results . Comparison with baseline methods on SSv2 and FineGym. Our approach has much better generality than vision-language models

Method	Generality	Locality	Mean
Vision-Language Model	27.6 $\,\pm\,0.0$	100.0 $\,\pm\,0.0$	63.8 $\,\pm\,0.0$
Modifier Retrieval	45.0 $\,\pm\,0.0$	98.9 $\,\pm\,0.0$	71.9 $\,\pm\,0.0$
Modifier Alignment	46.3 $\,\pm\,0.9$	98.9 $\,\pm\,0.0$	72.6 $\,\pm\,0.5$

Table 3: Zero-shot Ablation . Our modifier retrieval and alignment greatly improve generality by mining modifier vectors from the video-only classifier.

Components Finetuned	Initialization	Generality	Locality	Mean
Full Data Finetuning
$\theta_{head},\theta^{\prime}_{head}$	coarse category	86.7 $\,\pm\,0.1$	22.1 $\,\pm\,18.8$	54.4 $\,\pm\,9.4$
$\theta^{\prime}_{head}$	coarse category	86.7 $\,\pm\,0.1$	19.2 $\,\pm\,0.1$	52.9 $\,\pm\,0.0$
One-Shot Finetuning
$\theta_{backbone},\theta_{head},\theta_{head}^{\prime}$	coarse category	33.6 $\,\pm\,1.9$	0.0 $\,\pm\,0.0$	16.8 $\,\pm\,1.0$
$\theta_{head},\theta_{head}^{\prime}$	coarse category	50.7 $\,\pm\,1.8$	96.4 $\,\pm\,0.3$	73.5 $\,\pm\,0.9$

Table 4: One-Shot Finetuning Ablation . Constraining updates to the extended head (

\theta^{\prime}_{head}

) avoids catastrophic forgetting when training with little data, and initializing it with Knowledge Retrieval provides a stronger starting point that boosts generality without sacrificing locality.

Table 8

Figure 4: Analysis over different category splits . (a) Locality decrease slightly with more subcategories in the split, while generality shows no trend. (b) Performance is highest for direction-based splits and lowest for differences in object count, intent/success, and object interactions. (c) Existing analogous categories with the same modifier help, but our approach remains effective without them.

	Original Cat.			Cat. After Split	Ours			CLIP
% Coarse Cat.	Coarse	Fine	Total	Total	Gen.	Loc.	Mean	Gen.	Loc.	Mean
50%	27	92	119	174	46.3	98.9	72.6	27.6	100.0	63.8
66%	36	67	103	174	44.2	99.1	71.7	29.0	100.0	64.5
75%	41	55	96	174	44.6	98.8	71.7	29.4	100.0	64.7

Table 7: Effect of Compositional Variation in the Original Label Space. Our method remains robust when reducing the number of fine-grained categories in the original label space and consistently outperforms CLIP.

Figure 5: Qualitative Results on SSv2-Split with unseen category + modifier combinations.

Figure 6: New Visual Distinctions . We show failure and success cases of our model when categories require new visual distinctions.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Generality	Locality	Mean
ViT-Small	46.3 $\,\pm\,0.9$	98.9 $\,\pm\,0.0$	72.6 $\,\pm\,0.5$
ViT-Base	43.6 $\,\pm\,0.7$	99.1 $\,\pm\,0.0$	71.3 $\,\pm\,0.4$
ViT-Large	45.0 $\,\pm\,0.8$	99.3 $\,\pm\,0.0$	72.1 $\,\pm\,0.4$

Appendix

Table 8: Backbone Size Ablation . Results are consistent across different backbone sizes.

Text Perturbation	Generality	Locality	Mean
Clean	51.3	99.4	75.4
Typos	48.9	99.4	74.2
Multi-Sentence	41.1	99.6	70.4
Multi-Lingual (DE & ZH avg)	27.3	99.2	63.2

Appendix

Table 9: Robustness of Text Encoder . Our method is reasonably robust to typos but struggles with multi-sentence and multi-lingual text.

	SSv2-Split						FineGym-Split
	Subset A			Subset B			Subset A			Subset B
Training Pairs	Gen.	Loc.	Mean	Gen.	Loc.	Mean	Gen.	Loc.	Mean	Gen.	Loc.	Mean
$\mathcal{D}_{mod}$	44.9	99.3	72.1	40.2	99.2	69.7	27.6	98.5	63.1	14.0	98.5	56.2
$\mathcal{D}_{mod}\cup\mathcal{D}_{cat}$	46.3	98.9	72.6	38.4	98.9	68.7	34.2	97.8	66.0	18.9	97.9	58.4

Appendix

Table 10: Ablation on Training Pair Composition . The results consistently show that enriching the training signal by incorporating more diverse pairs leads to improved performance.

Coarse categories	Fine-grained categories
Moving the camera and Turning the camera	Approaching something with your camera
	Moving away from something with your camera
	Turning the camera downwards while filming something
	Turning the camera left while filming something
	Turning the camera right while filming something
	Turning the camera upwards while filming something

Appendix

Table 11: Constructed category hierarchy for SSv2.

Coarse categories	Fine-grained categories
(VT) round-off, flic-flac with 0.5 turn on, salto forward	(VT) round-off, flic-flac with 0.5 turn on, stretched salto forward with 1.5 turn off
	(VT) round-off, flic-flac with 0.5 turn on, stretched salto forward with 0.5 turn off
	(VT) round-off, flic-flac with 0.5 turn on, stretched salto forward with 1 turn off
	(VT) round-off, flic-flac with 0.5 turn on, stretched salto forward with 2 turn off
	(VT) round-off, flic-flac with 0.5 turn on, piked salto forward with 0.5 turn off
	(VT) round-off, flic-flac with 0.5 turn on, piked salto forward off

Appendix

Table 12: Constructed category hierarchy for FineGym.

Merged Coarse Category
Attaching something to something and Trying but failing to attach something to something because it doesn’t stick
Bending something
Burying something in something and Digging something out of something
Closing something and Opening something
Dropping something with spatial relation
Letting something roll

Appendix

Table 13: Merge Setting for SSv2 Split A.

Merged Coarse Category
Moving the camera and Turning the camera
Covering something with something and Uncovering something
Holding something with spatial relation
Folding something and Unfolding something
Lifting up one end of something
Moving something across a surface

Appendix

Table 14: Merge Setting for SSv2 Split B.

Merged Coarse Category
(VT) round-off, flic-flac with 0.5 turn on, salto forward
(VT) round-off, flic-flac with 1 turn on, salto backward
(VT) handspring forward on, salto forward
(FX) leap
(FX) jump or hop
(FX) 2 turn or more

Appendix

Table 15: Merge Setting for FineGym288 Split A.

Merged Coarse Category
(VT) round-off, flic-flac on, salto backward
(VT) tsukahara
(VT) handspring forward on
(FX) leap with turn
(FX) jump or hop with turn
(FX) 1 turn

Appendix

Table 16: Merge Setting for FineGym288 Split B.

W or w/o Analogies	Subcategory
with	Trying to bend something unbendable so nothing happens
	Dropping something behind something
	Dropping something in front of something
	Dropping something into something
	Dropping something next to something
	Dropping something onto something

Appendix

Table 17: Grouping Detail of whether a subcategory has Analogies

Type	Subcategory
Count	Putting number of something onto something
	Putting something and something on the table
	Putting something, something and something on the table
Spatiality	Moving something and something away from each other
	Moving something and something closer to each other
	Dropping something next to something

Appendix

Table 18: Grouping Detail within Sub-Category Type

Zero-Shot Learning

Zero-shot learning lets models recognize classes or solve tasks absent from task-specific training examples by using semantic descriptions or pretrained knowledge.

See all 184 papers

Novel Semantic Prompting for Zero-Shot Action Recognition
Mar 9, 2026Salman Iqbal, Waheed RehmanZero-Shot Action Recognition Video Action Recognition
One Editor, Many Edits: A Unified Training-Free Framework for Diverse Video Editing
Sep 3, 2026Adheesh Sunil Juvekar, Onkar Kishor Susladkar, Kiet A. Nguyen +6Video Editing Instruction-Based Video Editing
University of Illinois Urbana-Champaign
VINCIE-NExT: Unlocking Video Editing from Images via In-Context Modeling
Oct 8, 2026Leigang Qu, Feng Cheng, Ziyan Yang +7Video Editing Instruction-Based Video Editing
National University of Singapore · ByteDance Seed · University of Science and Technology of China

Mar 9, 2026J/K move · Enter open · S save

Novel Semantic Prompting for Zero-Shot Action Recognition

Salman Iqbal, Waheed Rehman

Zero-Shot Action Recognition Video Action Recognition Zero-Shot Learning Vision-Language Models LLM Prompting

Open paper AskPDF arXiv

Zero-shot action recognition relies on transferring knowledge from vision-language models to unseen actions using semantic descriptions. While recent methods focus on temporal modeling or architectural adaptations to handle video data, we argue that semantic prompting alone provides a strong and underexplored signal for zero-shot action understanding. We introduce SP-CLIP, a lightweight framework that augments frozen vision-language models with structured semantic prompts describing actions at multiple levels of abstraction, such as intent, motion, and object interaction. Without modifying the visual encoder or learning additional parameters, SP-CLIP aligns video representations with enriched textual semantics through prompt aggregation and consistency scoring. Experiments across standard benchmarks show that semantic prompting substantially improves zero-shot action recognition, particularly for fine-grained and compositional actions, while preserving the efficiency and generalization of pretrained models.

Abstract

Zero-Shot Learning

Explore similar work

Novel Semantic Prompting for Zero-Shot Action Recognition

One Editor, Many Edits: A Unified Training-Free Framework for Diverse Video Editing

VINCIE-NExT: Unlocking Video Editing from Images via In-Context Modeling