Decompose and Transfer: CoT-Prompting Enhanced Alignment for Open-Vocabulary Temporal Action Detection

Sa Zhu; Wanqian Zhang; Lin Wang; Xiaohua Chen; Chenxu Cui; Jinchao Zhang; Bo Li

cs.CVMar 25, 2026

Decompose and Transfer: CoT-Prompting Enhanced Alignment for Open-Vocabulary Temporal Action Detection

Authors: Sa Zhu, Wanqian Zhang, Lin Wang, Xiaohua Chen, Chenxu Cui, Jinchao Zhang, Bo Li

Organizations: Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · State Key Laboratory of Cyberspace Security Defense · Hangzhou Dianzi University ·

Abstract

Open-Vocabulary Temporal Action Detection (OV-TAD) aims to classify and localize action segments in untrimmed videos for unseen categories. Previous methods rely solely on global alignment between label-level semantics and visual features, which is insufficient to transfer temporal consistent visual knowledge from seen to unseen classes. To address this, we propose a Phase-wise Decomposition and Alignment (PDA) framework, which enables fine-grained action pattern learning for effective prior knowledge transfer. Specifically, we first introduce the CoT-Prompting Semantic Decomposition (CSD) module, which leverages the chain-of-thought (CoT) reasoning ability of large language models to automatically decompose action labels into coherent phase-level descriptions, emulating human cognitive processes. Then, Text-infused Foreground Filtering (TIF) module is introduced to adaptively filter action-relevant segments for each phase leveraging phase-wise semantic cues, producing semantically aligned visual representations. Furthermore, we propose the Adaptive Phase-wise Alignment (APA) module to perform phase-level visual-textual matching, and adaptively aggregates alignment results across phases for final prediction. This adaptive phase-wise alignment facilitates the capture of transferable action patterns and significantly enhances generalization to unseen actions. Extensive experiments on two OV-TAD benchmarks demonstrated the superiority of the proposed method.

Figures & tables

Figure 1: Illustrative examples of seen class LongJump and unseen class PoleVault. (a) The “Start” and “Middle” phases of seen LongJump and unseen PoleVault share strong semantic similarities, despite low label similarity. (b) In light of phase-wise prior knowledge, our method shows higher performance on PoleVault when trained with LongJump.

Figure 2: An overview of the proposed framework, which comprises three key modules: CoT-Prompting Semantic Decomposition (CSD), Text-infused Foreground Filtering (TIF) and Adaptive Phase-wise Alignment (APA). Specifically, the CSD module temporally decomposes action labels into multiple phase descriptions $F_{t}^{p}$ , where $p\in\{start,mid,end,glob\}$ . The TIF module then leverages phase-specific semantic cues to adaptively filter action-relevant segments for each phase, yielding phase-specific visual representations $F_{v}^{p}$ . Finally, the APA module performs phase-wise alignment and adaptively aggregates the alignment results for final action detection.

Action	Prompt: Decompose the action of $\langle Action\rangle$ into coherent three phases based on the			Prompt: Describe how a person does $\langle Action\rangle$ .
	natural temporal progression of the action. Please provide the output step by step.
	Start	Middle	End
LongJump	The person would run down	The person would plant one foot	The person would extend their	The person would sprint down the track
LongJump	the track to gain speed.	and push off the ground .	legs and land in the sand.	and jump forward into the sandpit.
PoleVault	The person would run down	The person would plant the pole	The person would clear the bar	The person would sprint down the track,
PoleVault	the track to gain speed.	and push off the ground .	and fall onto the landing mat.	vault with a pole, and clear a high bar.

Table 1: Prompt template for guiding GPT-4o to generate phase-specific and global descriptions of action labels and illustrative examples. The “Start” and “Middle” phases of LongJump and PoleVault exhibit semantic similarity.

Data Split	Methods	THUMOS14						ActivityNet v1.3
Data Split	Methods	0.3	0.4	0.5	0.6	0.7	Avg mAP(%)	0.5	0.75	0.95	Avg mAP(%)
50% Seen 50% Unseen	TriDet* (ICCV’23)	15.2	13.2	10.8	7.9	5.2	10.5	19.1	11.5	1.1	11.4
	DyFADet* (ECCV’24)	17.5	14.9	12.2	9.2	5.7	11.9	23.8	14.2	1.8	13.6
	DiGIT* (CVPR’25)	19.1	16.2	13.5	10.3	6.1	13.0	27.5	17.3	2.3	16.0
	EffPrompt (ECCV’22)	37.2	29.6	21.6	14.0	7.2	21.9	32.0	19.3	2.9	19.6
	STALE (ECCV’22)	38.3	30.7	21.2	13.8	7.0	22.2	32.1	20.7	5.9	20.5

Table 2: Performance comparison with the state-of-the-art methods on THUMOS14 and ActivityNet v1.3. * indicates closed-set TAD methods adapted to the OV-TAD setting. The best and second-best results are highlighted in Red and Blue , respectively.

Phase Number	mAP@tIOU (%)				Time (s)
	0.3	0.5	0.7	Avg
One (Glob)	59.3	45.5	21.9	42.5	27.6
Two (Start, End)	61.0	46.9	22.8	44.0	29.2
Three (Start, Mid, End)	63.9	48.4	23.6	45.3	30.5
Four (Start, Mid, End, Glob)	65.4	49.7	24.3	46.9	32.4
Five (Start, Mid1, Mid2, End, Glob)	66.3	50.4	24.8	47.6	34.8

Table 3: Analysis of different action phase number on THUMOS14 under the 50% seen / 50% unseen split. The phase number adopted in this paper is highlighted with a blue background .

LLM Backbone	mAP@tIOU (%)
	0.3	0.5	0.7	Avg
Qwen3	64.9	49.1	24.0	46.2
Deepseek v3	64.5	49.0	23.7	46.2
GPT-4	65.1	49.4	24.1	46.6
GPT-4o	65.4	49.7	24.3	46.9

Table 4: Analysis of different LLM backbones on THUMOS14 under the 50% seen / 50% unseen split.

Method	CSD	TIF	APA	mAP@AVG
				50%-50%	75%-25%
Baseline	✗	✗	✗	40.3	45.9
Ours	✓	✗	✗	42.1	47.8
	✓	✓	✗	43.6	49.0
	✓	✓	✓	46.9	52.1

Table 5: Analysis of each components on THUMOS14.

Method	mAP@tIOU (%)
	0.3	0.5	0.7	Avg
w/o Filtering	62.1	46.7	21.9	44.3
Static Filtering	63.4	48.0	22.8	45.5
Text-infused Filtering	65.4	49.7	24.3	46.9

Table 6: Analysis of Text-infused Foreground Filtering on THUMOS14 under the 50% seen / 50% unseen split.

Alignment	Method	mAP@tIOU (%)
		0.3	0.5	0.7	Avg
Global	Label	56.2	42.7	20.4	40.3
	Start	57.4	43.8	21.1	41.2
	Middle	58.1	44.2	21.4	41.5
	End	57.8	44.6	21.5	41.7
	Global	59.3	45.5	21.9	42.5

Table 7: Analysis of Adaptive Phase-wise Alignment on THUMOS14 under the 50% seen / 50% unseen split. “Global” represents global alignment between text features and averaged visual features, “Phase-wise” represents phase-wise alignment followed by aggregation.

Figure 3: Phase-wise Semantic Similarity ((a)-(d)) and Per-unseen class AP (%) (e) at tIoU threshold 0.5 on THUMOS14 under the 50% seen / 50% unseen split. For (a)-(d), the vertical denotes unseen (testing) classes, the horizontal denotes seen (training) classes.

Figure 4: Visualization of the detection results on THUMOS14 under the 50% seen / 50% unseen split. “sim” represents the visual-textual similarity at each phase.

Description	Type	Prompt
Phase-specific	(a)	Question : Given an action of $\langle Action\rangle$ , considering how the activity typically begins, evolves, and concludes. After your reasoning, provide a concise phase-wise summary.
		Answer : In the start phase, the person would $\{text\}$ . In the middle phase, the person would $\{text\}$ . In the end phase, the person would $\{text\}$ .
	(b)	Question : For the given action of $\langle Action\rangle$ , enumerate it into chronological sub-events and condense these events into three coherent phase descriptions.
		Answer : In the start phase, the person would $\{text\}$ . In the middle phase, the person would $\{text\}$ . In the end phase, the person would $\{text\}$ .
	(c)	Question : Decompose the action of $\langle Action\rangle$ into coherent three phases based on the natural temporal progression of the action. Please provide the output step by step.
		Answer : In the start phase, the person would $\{text\}$ . In the middle phase, the person would $\{text\}$ . In the end phase, the person would $\{text\}$ .

Table 8: The variants of prompt templates used to guide GPT-4o in generating phase-specific and global action descriptions. The placeholder

\{text\}

indicates the content to be filled in by GPT-4o.

Data Split	Prompt Type	THUMOS14				ActivityNet v1.3
		0.3	0.5	0.7	Avg	0.5	0.75	0.95	Avg
50% Seen 50% Unseen	(a)	64.9	49.4	24.0	46.4	52.9	35.1	7.7	34.3
	(b)	64.5	49.0	23.7	46.2	52.6	34.8	7.3	34.0
	(c)	65.4	49.7	24.3	46.9	53.1	35.3	7.7	34.6
75% Seen 25% Unseen	(a)	69.6	53.9	27.7	51.5	55.4	37.0	8.5	36.8
	(b)	69.3	53.5	27.4	51.3	55.1	36.9	8.3	36.6

Table 9: Analysis of the Prompt Template Design. The prompt template adopted in this paper is highlighted with a blue background .

Phase Number	mAP@tIOU (%)				Time (min)
	0.3	0.5	0.7	Avg
One (Glob)	51.0	32.8	5.9	32.3	14.7
Two (Start, End)	51.9	33.7	6.6	33.1	15.1
Three (Start, Mid, End)	52.6	34.5	7.1	33.9	15.8
Four (Start, Mid, End, Glob)	53.1	35.3	7.7	34.6	16.4
Five (Start, Mid1, Mid2, End, Glob)	53.5	35.5	8.0	34.8	17.2

Table 10: Analysis of different action phase number on ActivityNet v1.3 under the 50% seen / 50% unseen split. The phase number adopted in this paper is highlighted with a blue background .

Data Split	LLM Backbone	THUMOS14				ActivityNet v1.3
		0.3	0.5	0.7	Avg	0.5	0.75	0.95	Avg
50% Seen 50% Unseen	Qwen3	64.9	49.1	24.0	46.2	53.3	35.2	7.5	34.3
	Deepseek v3	64.5	49.0	23.7	46.2	52.6	34.8	7.3	34.0
	GPT-4	65.1	49.4	24.1	46.6	52.9	35.3	7.6	34.5
	GPT-4o	65.4	49.7	24.3	46.9	53.1	35.3	7.7	34.6
75% Seen 25% Unseen	Qwen3	69.6	54.0	27.5	51.6	54.8	36.5	8.5	36.7

Table 11: Analysis of Different LLM Backbone. The LLM backbone adopted in this paper is highlighted with a blue background .

Figure 5: Evaluation Template for GPT-4V and Human Assessments: Rating Across Five Dimensions—Linguistic Quality, Semantic Accuracy, Phase Clarity and Coherence, Visual Alignment, and Transferability.

Evaluation	Method	Rating [score/10.00] ( $\uparrow$ )
		Quality	Accuracy	Coherence	Alignment	Transferability
GPT-4V	Ground Truth	9.03	8.35	8.43	8.15	7.97
	Ours	9.21	8.22	8.72	7.86	8.49
Human	Ground Truth	8.77 $\pm 0.09$	8.10 $\pm 0.15$	8.23 $\pm 0.10$	7.91 $\pm 0.07$	7.71 $\pm 0.06$
	Ours	8.46 $\pm 0.06$	7.95 $\pm 0.08$	8.39 $\pm 0.12$	7.64 $\pm 0.09$	8.28 $\pm 0.10$

Table 12: GPT-4V and Human Evaluation of Description Quality on THUMOS14. The best results are highlighted in Red .

Figure 6: Phase-wise Semantic Similarity ((a)-(d)) and Per-unseen class AP (%) (e) at tIoU threshold 0.5 on ActivityNet v1.3 under the 50% seen / 50% unseen split. For (a)-(d), the vertical denotes unseen (testing) classes, the horizontal denotes seen (training) classes.

Backbone	Method	mAP@tIOU (%)
		0.3	0.5	0.7	Avg
DyFADet (ECCV’24)	Baseline	17.5	12.2	5.7	11.9
	PDA	66.1	50.2	25.0	47.5
DiGIT (CVPR’25)	Baseline	19.1	13.5	6.1	13.0
	PDA	67.3	50.7	25.2	48.0

Table 13: Plug-and-Play Capability of the Proposed PDA on THUMOS14 under the 50% seen / 50% unseen split.

Method	Feature		mAP@AVG
	Visual	Text	50%-50%	75%-25%
Ti-FAD	CLIP-B	CLIP-B	27.3	29.7
	CLIP-L	CLIP-L	27.2	30.6
	I3D	CLIP-B	41.2	46.8
	I3D	CLIP-L	40.6	47.3
Ours	CLIP-B	CLIP-B	30.1	33.5

Table 14: Effectiveness of varying visual/textual backbones on THUMOS14.

Task	Characteristics	Summary
Action Localization [ 2 , 53 ]	Decompose actions into defining attributes and aggregate these attributes to align with frame-level embeddings, enabling more precise localization.	Enrich textual semantics for more concise alignment.
Action Recognition [ 4 , 19 ]	Decompose actions into multi-dimensional descriptions and aggregate these descriptions to align with averaged visual embeddings for more precise recognition.
Ours	Decompose actions into multi-phase descriptions and adaptively perform phase-wise alignment with visual features to learn transferable action patterns, enhancing zero-shot action detection performance.	Learn transferable action knowledge for generalized zero-shot detection.

Table 15: Comparison with Previous LLM-base Label Expansion Methods.

Expansion	Method	THUMOS14				ActivityNet v1.3
		0.3	0.5	0.7	Avg	0.5	0.75	0.95	Avg
Baseline	-	56.2	42.7	20.4	40.3	49.7	31.5	4.9	31.2
Global Label Expansion	(a) w/o Decompose	60.1	46.5	22.2	43.4	51.7	33.2	6.3	32.7
	(b) w/ Decompose	61.1	47.3	22.7	44.0	51.8	33.5	6.6	32.9
Single-phase Expansion	Start	57.4	43.8	21.1	41.2	50.1	32.0	5.2	31.6
	Middle	58.1	44.2	21.4	41.5	50.8	32.4	5.7	32.0

Table 16: Comparison with LLM-base Label Expansion under the 50% seen / 50% unseen split.

Figure 7: Visualization of the detection results “HighJump” on THUMOS14 under the 50% seen / 50% unseen split. “sim” represents the visual-textual similarity at each phase.

Figure 8: Visualization of the detection results “Doing Karate” on ActivityNet v1.3 under the 50% seen / 50% unseen split. “sim” represents the visual-textual similarity at each phase.

Figure 9: Visualization of the detection results “Cleaning sink” on ActivityNet v1.3 under the 50% seen / 50% unseen split. “sim” represents the visual-textual similarity at each phase.

Temporal Action Detection

Temporal action detection identifies action categories and their start and end times in untrimmed video.

See all 41 papers

Apr 20, 2026

Denoise and Align: Diffusion-Driven Foreground Knowledge Prompting for Open-Vocabulary Temporal Action Detection

Open-Vocabulary Temporal Action Detection (OV-TAD) aims to localize and classify action segments of unseen categories in untrimmed videos, where effective alignment between action semantics and video representations is critical for accurate detection. However, existing methods struggle to mitigate the semantic imbalance between concise, abstract action labels and rich, complex video contents, inevitably introducing semantic noise and misleading cross-modal alignment. To address this challenge, we propose DFAlign, the first framework that leverages diffusion-based denoising to generate foreground knowledge for the guidance of action-video alignment. Following the 'conditioning, denoising and aligning' manner, we first introduce the Semantic-Unify Conditioning (SUC) module, which unifies action-shared and action-specific semantics as conditions for diffusion denoising. Then, the Background-Suppress Denoising (BSD) module generates foreground knowledge by progressively removing background redundancy from videos through denoising process. This foreground knowledge serves as effective intermediate semantic anchor between video and text representations, mitigating the semantic gap and enhancing the discriminability of action-relevant segments. Furthermore, we introduce the Foreground-Prompt Alignment (FPA) module to inject extracted foreground knowledge as prompt tokens into text representations, guiding model's attention towards action-relevant segments and enabling precise cross-modal alignment. Extensive experiments demonstrate that our method achieves state-of-the-art performance on two OV-TAD benchmarks. The code repository is provided as follows: https://anonymous.4open.science/r/Code-2114/.

Sa Zhu, Wanqian Zhang, Lin Wang +3

Institute of Information Engineering, Chinese Academy of Sciences, School of Cyber Security, University of Chinese Academy of Sciences, State Key Laboratory of Cyberspace Security Defense, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · Hangzhou Dianzi University, Hangzhou, China +1

Cross-Modal Alignment Temporal Action Detection Temporal Video Understanding

Ask PDF

Aug 18, 2026

TF-CADE: Foreground-Concentrated Text-Video Alignment for Zero-Shot Temporal Action Detection

Zero-Shot Temporal Action Detection (ZSTAD) aims to lo- calize and recognize action instances from unseen action categories in untrimmed videos. Although existing meth- ods have shown effectiveness by advancing architectural text-video alignment, they still struggle with capturing se- mantic distinctions between action classes, resulting in text- irrelevant predictions. To address this issue, we propose a Text-Foreground Concentrated Alignment for zero-shot temporal action DEtector (TF-CADE) that explicitly aligns textual information with action-relevant foreground regions. Specifically, we introduce Action Concentrate Aggregation (ACA), which extracts action concentrate scores to aggregate temporally informative video segments into a foreground- weighted video embedding. This foreground concentrated alignment enhances the semantic consistency between text and video features and improves inter-class discriminabil- ity. In addition, a Certainty-based Confidence Re-weighting (CCR) strategy refines per-snippet confidence scores by lever- aging foreground-aware similarity, effectively suppressing irrelevant action classes during inference. Extensive evalua- tions show that our TF-CADE not only achieves state-of-the- art performance under in-distribution settings but also excels in cross-dataset generalization to unseen action classes.

Yearang Lee, Ho-Joong Kim, Seong-Whan Lee

Dept. of Artificial Intelligence, Korea University, Seoul, Korea

Temporal Action Detection Vision-Language Alignment

Ask PDF

Feb 24, 2026

Exploring Vision-Language Models for Open-Vocabulary Zero-Shot Action Segmentation

Temporal Action Segmentation (TAS) requires dividing videos into action segments, yet the vast space of activities and alternative breakdowns makes collecting comprehensive datasets infeasible. Existing methods remain limited to closed vocabularies and fixed label sets. In this work, we explore the largely unexplored problem of Open-Vocabulary Zero-Shot Temporal Action Segmentation (OVTAS) by leveraging the strong zero-shot capabilities of Vision-Language Models (VLMs). We introduce a training-free pipeline that follows a segmentation-by-classification design: Frame-Action Embedding Similarity (FAES) matches video frames to candidate action labels, and Similarity-Matrix Temporal Segmentation (SMTS) enforces temporal consistency. Beyond proposing OVTAS, we present a systematic study across 14 diverse VLMs, providing the first broad analysis of their suitability for open-vocabulary action segmentation. Experiments on standard benchmarks show that OVTAS achieves strong results without task-specific supervision, underscoring the potential of VLMs for structured temporal understanding.

Asim Unmesh, Kaki Ramesh, Mayank Patel +2

Purdue University, USA · Birla Institute of Technology and Science (BITS) Hyderabad, India

Temporal Action Segmentation Zero-Shot Learning Vision-Language Models

Ask PDF

Abstract

Temporal Action Detection

Explore similar work

Denoise and Align: Diffusion-Driven Foreground Knowledge Prompting for Open-Vocabulary Temporal Action Detection

TF-CADE: Foreground-Concentrated Text-Video Alignment for Zero-Shot Temporal Action Detection

Exploring Vision-Language Models for Open-Vocabulary Zero-Shot Action Segmentation