Exploring Vision-Language Models for Open-Vocabulary Zero-Shot Action Segmentation

Asim Unmesh; Kaki Ramesh; Mayank Patel; Rahul Jain; Karthik Ramani

cs.CVFeb 24, 2026

Exploring Vision-Language Models for Open-Vocabulary Zero-Shot Action Segmentation

Authors: Asim Unmesh, Kaki Ramesh, Mayank Patel, Rahul Jain, Karthik Ramani

Organizations: Purdue University, USA · Birla Institute of Technology and Science (BITS) Hyderabad, India

Abstract

Temporal Action Segmentation (TAS) requires dividing videos into action segments, yet the vast space of activities and alternative breakdowns makes collecting comprehensive datasets infeasible. Existing methods remain limited to closed vocabularies and fixed label sets. In this work, we explore the largely unexplored problem of Open-Vocabulary Zero-Shot Temporal Action Segmentation (OVTAS) by leveraging the strong zero-shot capabilities of Vision-Language Models (VLMs). We introduce a training-free pipeline that follows a segmentation-by-classification design: Frame-Action Embedding Similarity (FAES) matches video frames to candidate action labels, and Similarity-Matrix Temporal Segmentation (SMTS) enforces temporal consistency. Beyond proposing OVTAS, we present a systematic study across 14 diverse VLMs, providing the first broad analysis of their suitability for open-vocabulary action segmentation. Experiments on standard benchmarks show that OVTAS achieves strong results without task-specific supervision, underscoring the potential of VLMs for structured temporal understanding.

Figures & tables

Panel of Figure 1 — Fig. 1 : Problem Setup: Existing approaches in 1(a) are fixed vocabulary and do not generalize to unseen videos. 1(b) illustrates our proposed method for open-vocabulary and zero-shot action segmentation.

Method	Zero-Shot	Open-Vocab	Unsupervised	Weakly Sup.	Semi-Sup.	Training-Free
ASAL [ 9 ]	✗	✗	✓	✗	✗	✗
COIN-SSL [ 10 ]	✗	✗	✗	✗	✓	✗
NN-Viterbi [ 11 ]	✗	✗	✗	✓	✗	✗
ASAL-SSL [ 9 ]	✗	✗	✗	✗	✓	✗
TCCNet [ 12 ]	✗	✗	✗	✓	✗	✗
UDE [ 13 ]	✗	✗	✓	✗	✗	✗

TABLE I : Comparison of our method with other methods in TAS based on generalization ability and training data needs. ✓ denotes applicability.

Fig. 2 : Open-Vocabulary Temporal Action Segmentation (OVTAS) Pipeline. Our 2-stage pipeline adopts a “segmentation by classification” approach to tackle temporal action segmentation (TAS). Stage 1, Frame–Action Embedding Similarity (FAES), generates a similarity matrix by matching frames with action labels. Stage 2, Similarity-Matrix driven Temporal Segmentation (SMTS), uses optimal transport with a temporal prior to enforce temporal consistency, producing stable action segments.

Dataset	Min (s)	Max (s)	Mean (s)
GTEA	42.27	133.93	74.35
Breakfast	12.27	649.53	137.40
50 Salads	251.60	605.00	385.25

TABLE II : Video duration statistics in seconds.

Dataset	Min	Max	Mean
GTEA	24	49	36.3
Breakfast	1	18	5.4
50 Salads	15	27	20.7

TABLE III : Number of ground-truth segments per video.

Metric	Best	Temporal Prior (abl.)	L2 Norm (abl.)
F1@10	21.33	1.12 (↓20.21)	2.36 (↓18.97)
F1@25	13.76	0.56 (↓13.20)	1.12 (↓12.64)
F1@50	5.44	0.19 (↓5.25)	0.62 (↓4.82)
Edit	51.07	5.04 (↓46.03)	7.23 (↓43.84)
Acc	28.08	6.54 (↓21.54)	8.43 (↓19.65)
Avg	23.14	2.69 (↓20.45)	3.95 (↓19.19)

TABLE IV : Ablation on Temporal Prior and L2 Norm: Parentheses show drops ( ↓ )) from Best. Results on GTEA Dataset.

Abbrev	Checkpoint	#Params (M)
SigLIP-M1	so400m-p16-256-i18n	877.96
SigLIP-M2	large-p16-384	652.48
SigLIP-M3	so400m-p14-384	1128.76
OpenCLIP-M1	ViT-B/32 (laion2B-s34B-b79K)	151.28
OpenCLIP-M2	ViT-B/16 (laion2B-s34B-b88K)	149.62
OpenCLIP-M3	ViT-L/14 (laion2B-s32B-b82K)	427.62

TABLE V: VLM Details: Parameter sizes (in millions) of the 14 VLM variants used in our experiments.

Method	50 Salads				GTEA				Breakfast
Method	F1@10,25,50	Edit	Acc	Avg	F1@10,25,50	Edit	Acc	Avg	F1@10,25,50	Edit	Acc	Avg
Random	0.00 / 0.00 / 0.00	0.20	5.46	1.13	0.12 / 0.01 / 0.01	2.06	5.61	1.56	0.00 / 0.00 / 0.00	0.72	17.45	3.63
ES_mean	1.49 / 0.13 / 0.04	5.40	5.63	2.54	7.08 / 3.99 / 1.67	13.00	5.99	6.35	20.56 / 11.97 / 3.63	22.39	17.14	15.14
ES_vote	2.77 / 0.84 / 0.31	6.72	5.68	3.26	7.32 / 4.63 / 1.87	12.68	5.96	6.49	22.15 / 13.69 / 4.52	24.66	17.15	16.43
ES_nrp	6.81 / 5.90 / 2.73	8.00	7.24	6.14	7.75 / 5.35 / 2.01	8.29	12.96	7.27	24.38 / 19.46 / 8.73	36.35	11.83	20.15
Ours (SigLIP-M1)	42.6 / 31.4 / 14.1	88.7	31.5	41.7	21.1 / 13.0 / 4.9	51.2	28.3	23.7	54.0 / 39.5 / 15.0	92.7	30.9	46.4

TABLE VI : Comparison of baselines and all VLM variants across three datasets (50 Salads, GTEA, and Breakfast). Each Avg is the mean of F1@10, F1@25, F1@50, Edit, and Accuracy. Maximum average value for each dataset and each family is boldfaced.

Dataset	Ablated Stage	F1@10	F1@25	F1@50	Edit	Acc	Avg
50 Salads	None	41.84	31.19	13.84	88.58	31.30	41.81
	Stage1 ( $\downarrow$ )	6.71 (↓35.13)	5.16 (↓26.03)	2.27 (↓11.57)	9.35 (↓79.23)	28.53 (↓2.77)	10.40 (↓31.41)
	Stage2 ( $\downarrow$ )	0.17 (↓41.67)	0.08 (↓31.11)	0.03 (↓13.81)	0.88 (↓87.70)	5.48 (↓25.82)	3.01 (↓38.80)
GTEA	None	21.33	13.76	5.44	51.07	28.08	23.14
	Stage1 ( $\downarrow$ )	1.38 (↓19.95)	0.60 (↓13.16)	0.25 (↓5.19)	17.39 (↓33.68)	17.30 (↓10.78)	7.38 (↓15.76)
	Stage2 ( $\downarrow$ )	1.18 (↓20.15)	0.52 (↓13.24)	0.16 (↓5.28)	3.95 (↓47.12)	5.41 (↓22.67)	2.64 (↓20.50)

TABLE VII : Stage Ablation studies: For each dataset, we compare the Best model with FAES (Random-ASOT) and SMTS (Baseline-0). Columns report F1 scores at different thresholds, Edit, Acc, and their mean (Avg). Drops from Best are shown in blue with downward arrows ( ↓ ).

Breakfast	Acc	Edit	F1@10	F1@25	F1@50	Avg
0–60s	46.31	98.51	76.05	62.39	29.16	62.48
60–120s	40.99	96.28	63.98	50.89	20.26	54.48
$\geq$ 120s	26.41	82.37	41.40	25.81	8.16	36.83
GTEA	Acc	Edit	F1@10	F1@25	F1@50	Avg
0–60s	27.75	50.65	24.38	17.14	8.38	25.66
60–120s	23.31	33.19	13.54	8.12	3.05	16.24

TABLE VIII : Performance variation by video duration: across different video length bins (in seconds) for Breakfast, GTEA, and 50 Salads datasets.

Panel of Figure 11 — Fig. 3 : Qualitative results: columns show segmentation results of our method on GTEA, 50 Salads, and Breakfast (left to right), with two examples each.

GTEA	Acc	Edit	F1@10	F1@25	F1@50	Avg
20–29	28.23	72.77	48.55	33.53	16.18	39.85
30–39	23.12	42.79	19.11	11.78	4.78	20.32
40–49	22.19	15.14	10.06	5.95	2.27	11.12
Breakfast	Acc	Edit	F1@10	F1@25	F1@50	Avg
0–4	42.79	98.69	71.90	59.28	26.25	59.78
5–9	29.00	88.56	50.52	35.06	12.60	43.15

TABLE IX : Performance variation by number of action segments in ground truth: Performance across bins of ground-truth segment counts for GTEA, Breakfast, and 50 Salads datasets.

Fig. 4 : Performance vs Model Size. Models are grouped by family (SigLIP, CLIP, OpenCLIP, PECore). Shaded regions indicate parameter size bins: Low ( $\leq$ 400M), Large (400–800M), Huge (800–1500M), and Giant ( $>$ 1500M).

Dataset	Min (s)	Max (s)	Mean (s)
GTEA	0.07	44.73	1.94
Breakfast	0.07	386.00	20.95
50 Salads	0.03	138.47	18.59

TABLE X : Statistics of segment durations across datasets.

Fig. 5 : VLM Family average of Avg metric: across datasets (GTEA, 50 Salads, Breakfast). Each line is a VLM family; per-dataset boxes list all values.

Temporal Action Segmentation

Temporal action segmentation labels when actions begin and end in continuous video, enabling fine-grained analysis of human and robotic activity.

See all 49 papers

Decompose and Transfer: CoT-Prompting Enhanced Alignment for Open-Vocabulary Temporal Action Detection
Mar 25, 2026Sa Zhu, Wanqian Zhang, Lin Wang +4Temporal Action Detection Temporal Video Understanding
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · State Key Laboratory of Cyberspace Security Defense +2
Improving Temporal Action Segmentation via Constraint-Aware Decoding
May 11, 2026Yeo Keat Ee, Debaditya Roy, Chen Li +2Temporal Action Segmentation Constrained Decoding
Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Indian Institute of Technology Kharagpur, India +1
OZ-TAL: Online Zero-Shot Temporal Action Localization
May 11, 2026Chaolei Han, Hongsong Wang, Xin Gong +1Zero-Shot Learning Temporal Action Detection
School of Cyber Science and Engineering, Southeast University, Nanjing 211102, China · Engineering Research Center of Blockchain Application, Supervision and Management (Southeast University), Ministry of Education; Purple Mountain Laboratories, Nanjing 211111, China · School of Computer Science and Engineering, Southeast University, Nanjing 211102, China

Mar 25, 2026J/K move · Enter open · S save

Decompose and Transfer: CoT-Prompting Enhanced Alignment for Open-Vocabulary Temporal Action Detection

Sa Zhu, Wanqian Zhang, Lin Wang, Xiaohua Chen+3

Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · State Key Laboratory of Cyberspace Security Defense+2

Temporal Action Detection Temporal Video Understanding Vision-Language Alignment

Open paper AskPDF arXiv

Open-Vocabulary Temporal Action Detection (OV-TAD) aims to classify and localize action segments in untrimmed videos for unseen categories. Previous methods rely solely on global alignment between label-level semantics and visual features, which is insufficient to transfer temporal consistent visual knowledge from seen to unseen classes. To address this, we propose a Phase-wise Decomposition and Alignment (PDA) framework, which enables fine-grained action pattern learning for effective prior knowledge transfer. Specifically, we first introduce the CoT-Prompting Semantic Decomposition (CSD) module, which leverages the chain-of-thought (CoT) reasoning ability of large language models to automatically decompose action labels into coherent phase-level descriptions, emulating human cognitive processes. Then, Text-infused Foreground Filtering (TIF) module is introduced to adaptively filter action-relevant segments for each phase leveraging phase-wise semantic cues, producing semantically aligned visual representations. Furthermore, we propose the Adaptive Phase-wise Alignment (APA) module to perform phase-level visual-textual matching, and adaptively aggregates alignment results across phases for final prediction. This adaptive phase-wise alignment facilitates the capture of transferable action patterns and significantly enhances generalization to unseen actions. Extensive experiments on two OV-TAD benchmarks demonstrated the superiority of the proposed method.

Abstract

Temporal Action Segmentation

Explore similar work

Decompose and Transfer: CoT-Prompting Enhanced Alignment for Open-Vocabulary Temporal Action Detection

Improving Temporal Action Segmentation via Constraint-Aware Decoding

OZ-TAL: Online Zero-Shot Temporal Action Localization