Video recognition models are typically trained on fixed taxonomies which are often too coarse, collapsing distinctions in object, manner or outcome under a single label. As tasks and definitions evolve, such models cannot accommodate emerging distinctions and collecting new annotations and retraining to accommodate such changes is costly. To address these challenges, we introduce category splitting, a new task where an existing classifier is edited to refine a coarse category into finer subcategories, while preserving accuracy elsewhere. We propose a zero-shot editing method that leverages the latent compositional structure of video classifiers to expose fine-grained distinctions without additional data. We further show that low-shot fine-tuning, while simple, is highly effective and benefits from our zero-shot initialization. Experiments on our new video benchmarks for category splitting demonstrate that our method substantially outperforms vision-language baselines, improving accuracy on the newly split categories without sacrificing performance on the rest. Project page: https://kaitingliu.github.io/Category-Splitting/.
Figures & tables
Figure 1: Category splitting aims to edit a trained video classifier by dividing a coarse label into multiple fine-grained subcategories, while keeping all other predictions unchanged. The challenge is to achieve this without retraining the full model and with zero or very few labels.
Figure 2: Zero-shot Category Splitting . Given a trained video classifier, our goal is to split a coarse category (e.g. pushing ) into fine-grained subcategories without any video data. Modifier retrieval first exposes compositional structure in the video classifier’s classification head from which it builds a dictionary of modifier vectors. The classifier is then edited by retrieving the appropriate modifier vector and adding it to the coarse category’s weight vector to create a new fine-grained subcategory. To generalize to unseen modifiers, modifier alignment learns a lightweight mapping from modifier text to modifier vectors, using category text/weight vectors as additional supervision.
Figure 3: Low-shot Category Splitting . We edit the model by replacing the coarse category c with θhead′ . This head is fine-tuned with as little as one video per fine-grained subcategory, initialized with our zero-shot approach.
Base Model
Category Splitting
Evaluation
Example
Benchmark
Subset
categories
videos
coarse
fine
generality
locality
SSv2-Split
A
119
169K
27
92
429
24347
bending → { bending so it deforms , bending until it
handspring forward 1 turn , handspring forward no turn }
Table 1: Statistics of Category Splitting Datasets. We create meta-datasets SSV2-Split and FineGym-Split, each with two subsets (A and B). For each subset we report the number of categories and videos used to train the base model, the number of coarse categories eligible for splitting, the number of resulting fine-grained subcategories we can split into, the average number of videos used to evaluate generality and locality per category split as well as an example coarse -to- fine split.
SSv2-Split
FineGym-Split
Subset A
Subset B
Subset A
Subset B
Method
Gen.
Loc.
Mean
Gen.
Loc.
Mean
Gen.
Loc.
Mean
Gen.
Loc.
Mean
CLIP ( Radford et al., 2021 )
27.6
100.0
63.8
30.7
100.0
65.4
12.1
100.0
56.1
7.2
100.0
53.6
FLAIR ( Xiao et al., 2025 )
30.6
100.0
65.3
28.4
100.0
64.2
18.0
100.0
59.0
11.6
100.0
55.8
FG-CLIP ( Xie et al., 2025 )
30.9
100.0
65.4
30.8
100.0
65.4
19.4
100.0
59.7
13.9
100.0
56.9
VideoCLIP-XL ( Wang et al., 2024a )
28.6
100.0
64.3
29.9
100.0
64.9
18.0
100.0
59.0
8.2
100.0
54.1
Table 2: Comparative Zero-Shot Results . Comparison with baseline methods on SSv2 and FineGym. Our approach has much better generality than vision-language models
Method
Generality
Locality
Mean
Vision-Language Model
27.6 ±0.0
100.0 ±0.0
63.8 ±0.0
Modifier Retrieval
45.0 ±0.0
98.9 ±0.0
71.9 ±0.0
Modifier Alignment
46.3 ±0.9
98.9 ±0.0
72.6 ±0.5
Table 3: Zero-shot Ablation . Our modifier retrieval and alignment greatly improve generality by mining modifier vectors from the video-only classifier.
Components Finetuned
Initialization
Generality
Locality
Mean
Full Data Finetuning
θhead,θhead′
coarse category
86.7 ±0.1
22.1 ±18.8
54.4 ±9.4
θhead′
coarse category
86.7 ±0.1
19.2 ±0.1
52.9 ±0.0
One-Shot Finetuning
θbackbone,θhead,θhead′
coarse category
33.6 ±1.9
0.0 ±0.0
16.8 ±1.0
θhead,θhead′
coarse category
50.7 ±1.8
96.4 ±0.3
73.5 ±0.9
Table 4: One-Shot Finetuning Ablation . Constraining updates to the extended head ( θhead′ ) avoids catastrophic forgetting when training with little data, and initializing it with Knowledge Retrieval provides a stronger starting point that boosts generality without sacrificing locality.
Table 8
Figure 4: Analysis over different category splits . (a) Locality decrease slightly with more subcategories in the split, while generality shows no trend. (b) Performance is highest for direction-based splits and lowest for differences in object count, intent/success, and object interactions. (c) Existing analogous categories with the same modifier help, but our approach remains effective without them.
Original Cat.
Cat. After Split
Ours
CLIP
% Coarse Cat.
Coarse
Fine
Total
Total
Gen.
Loc.
Mean
Gen.
Loc.
Mean
50%
27
92
119
174
46.3
98.9
72.6
27.6
100.0
63.8
66%
36
67
103
174
44.2
99.1
71.7
29.0
100.0
64.5
75%
41
55
96
174
44.6
98.8
71.7
29.4
100.0
64.7
Table 7: Effect of Compositional Variation in the Original Label Space. Our method remains robust when reducing the number of fine-grained categories in the original label space and consistently outperforms CLIP.
Figure 5: Qualitative Results on SSv2-Split with unseen category + modifier combinations.
Figure 6: New Visual Distinctions . We show failure and success cases of our model when categories require new visual distinctions.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Generality
Locality
Mean
ViT-Small
46.3 ±0.9
98.9 ±0.0
72.6 ±0.5
ViT-Base
43.6 ±0.7
99.1 ±0.0
71.3 ±0.4
ViT-Large
45.0 ±0.8
99.3 ±0.0
72.1 ±0.4
Appendix
Table 8: Backbone Size Ablation . Results are consistent across different backbone sizes.
Text Perturbation
Generality
Locality
Mean
Clean
51.3
99.4
75.4
Typos
48.9
99.4
74.2
Multi-Sentence
41.1
99.6
70.4
Multi-Lingual (DE & ZH avg)
27.3
99.2
63.2
Appendix
Table 9: Robustness of Text Encoder . Our method is reasonably robust to typos but struggles with multi-sentence and multi-lingual text.
SSv2-Split
FineGym-Split
Subset A
Subset B
Subset A
Subset B
Training Pairs
Gen.
Loc.
Mean
Gen.
Loc.
Mean
Gen.
Loc.
Mean
Gen.
Loc.
Mean
Dmod
44.9
99.3
72.1
40.2
99.2
69.7
27.6
98.5
63.1
14.0
98.5
56.2
Dmod∪Dcat
46.3
98.9
72.6
38.4
98.9
68.7
34.2
97.8
66.0
18.9
97.9
58.4
Appendix
Table 10: Ablation on Training Pair Composition . The results consistently show that enriching the training signal by incorporating more diverse pairs leads to improved performance.
Coarse categories
Fine-grained categories
Moving the camera and Turning the camera
Approaching something with your camera
Moving away from something with your camera
Turning the camera downwards while filming something
Turning the camera left while filming something
Turning the camera right while filming something
Turning the camera upwards while filming something
Appendix
Table 11: Constructed category hierarchy for SSv2.
Coarse categories
Fine-grained categories
(VT) round-off, flic-flac with 0.5 turn on, salto forward
(VT) round-off, flic-flac with 0.5 turn on, stretched salto forward with 1.5 turn off
(VT) round-off, flic-flac with 0.5 turn on, stretched salto forward with 0.5 turn off
(VT) round-off, flic-flac with 0.5 turn on, stretched salto forward with 1 turn off
(VT) round-off, flic-flac with 0.5 turn on, stretched salto forward with 2 turn off
(VT) round-off, flic-flac with 0.5 turn on, piked salto forward with 0.5 turn off
(VT) round-off, flic-flac with 0.5 turn on, piked salto forward off
Appendix
Table 12: Constructed category hierarchy for FineGym.
Merged Coarse Category
Attaching something to something and Trying but failing to attach something to something because it doesn’t stick
Bending something
Burying something in something and Digging something out of something
Closing something and Opening something
Dropping something with spatial relation
Letting something roll
Appendix
Table 13: Merge Setting for SSv2 Split A.
Merged Coarse Category
Moving the camera and Turning the camera
Covering something with something and Uncovering something
Holding something with spatial relation
Folding something and Unfolding something
Lifting up one end of something
Moving something across a surface
Appendix
Table 14: Merge Setting for SSv2 Split B.
Merged Coarse Category
(VT) round-off, flic-flac with 0.5 turn on, salto forward
(VT) round-off, flic-flac with 1 turn on, salto backward
(VT) handspring forward on, salto forward
(FX) leap
(FX) jump or hop
(FX) 2 turn or more
Appendix
Table 15: Merge Setting for FineGym288 Split A.
Merged Coarse Category
(VT) round-off, flic-flac on, salto backward
(VT) tsukahara
(VT) handspring forward on
(FX) leap with turn
(FX) jump or hop with turn
(FX) 1 turn
Appendix
Table 16: Merge Setting for FineGym288 Split B.
W or w/o Analogies
Subcategory
with
Trying to bend something unbendable so nothing happens
Dropping something behind something
Dropping something in front of something
Dropping something into something
Dropping something next to something
Dropping something onto something
Appendix
Table 17: Grouping Detail of whether a subcategory has Analogies
Type
Subcategory
Count
Putting number of something onto something
Putting something and something on the table
Putting something, something and something on the table
Spatiality
Moving something and something away from each other
Moving something and something closer to each other
Dropping something next to something
Appendix
Table 18: Grouping Detail within Sub-Category Type
Zero-shot learning lets models recognize classes or solve tasks absent from task-specific training examples by using semantic descriptions or pretrained knowledge.
Zero-shot action recognition relies on transferring knowledge from vision-language models to unseen actions using semantic descriptions. While recent methods focus on temporal modeling or architectural adaptations to handle video data, we argue that semantic prompting alone provides a strong and underexplored signal for zero-shot action understanding. We introduce SP-CLIP, a lightweight framework that augments frozen vision-language models with structured semantic prompts describing actions at multiple levels of abstraction, such as intent, motion, and object interaction. Without modifying the visual encoder or learning additional parameters, SP-CLIP aligns video representations with enriched textual semantics through prompt aggregation and consistency scoring. Experiments across standard benchmarks show that semantic prompting substantially improves zero-shot action recognition, particularly for fine-grained and compositional actions, while preserving the efficiency and generalization of pretrained models.
Video editing spans diverse editing paradigms, yet achieving high-quality instruction-guided and subject-guided editing within a single unified framework remains challenging. We introduce EditVid, a training-free framework combining sparse causal memory for local coherence, correspondence-based post-attention token injection for long-range identity preservation, and soft latent blending for edit locality. The same framework supports instruction-guided and reference-guided edits, including style transfer, attribute modification, object insertion, part-level editing, and subject replacement. On FiVE, EditVid achieves 78.16 FiVE-Acc, compared with 58.95 for the strongest evaluated training-free baseline, while obtaining competitive results on IVEBench. A user study further shows a 51.8% overall preference for EditVid over 7 competing methods.
Adheesh Sunil Juvekar, Onkar Kishor Susladkar, Kiet A. Nguyen +6
Building a capable video editor remains significantly harder than a video generator: editing requires (source, instruction, edited) triplets that are prohibitively expensive to annotate and difficult to synthesize at scale, whereas image editing has already reached maturity with millions of such pairs readily available. In this work, we introduce VINCIE-NExT, a unified framework that transfers editing capability from images to videos through in-context visual demonstrations, alleviating the need for large-scale paired video editing data. VINCIE-NExT decomposes video editing into a structured chain of composable sub-tasks (Video -> Image -> Image -> Video), routing editing intent through the image domain and enabling scalable joint training from heterogeneous image and video corpora under a unified diffusion objective. An image editing pair, synthesized by the model or supplied by the user, is prepended as an in-context visual demonstration that serves as a spatial appearance blueprint for every output frame. To ground appearance edits across the interleaved context, we introduce a novel position encoding that links image demonstrations and video frames in a shared spatial coordinate system, enabling pixel-faithful propagation of appearance changes to every output frame. Chain-of-Editing further provides principled test-time scaling: by executing the sub-task chain as progressive diffusion stages, editing quality can be improved by investing additional compute without retraining. Comprehensive experiments on OpenVE-Bench demonstrate the state-of-the-art performance across diverse editing categories, with ablations confirming the effectiveness of each component.
Leigang Qu, Feng Cheng, Ziyan Yang +7
National University of Singapore · ByteDance Seed · University of Science and Technology of China