Revisiting Natural Language Prompts from Large Language Models for Open-Vocabulary Video Recognition

Haichen He; Weibin Liu; Weiwei Xing

cs.CVOct 12, 2026

Revisiting Natural Language Prompts from Large Language Models for Open-Vocabulary Video Recognition

Authors: Haichen He, Weibin Liu, Weiwei Xing

Organizations: School of Computer Science and Technology, Beijing Jiaotong University · Visual Intelligence +X International Cooperation Joint Laboratory of MOE · School of Software Engineering, Beijing Jiaotong University

Abstract

Vision-language models (VLMs) like CLIP have shown great success in image tasks, however, applying CLIP to open-vocabulary video recognition poses challenges. Due to the limited text information, directly fine-tuning CLIP on video recognition datasets may lead to overfitting and hinder its generalizability. A prevalent method is to enrich text information by generating prompts using Large Language Models (LLMs). However, limited efforts have been made to explore how to generate higher-quality prompts that can contribute to classify unseen categories, and how to further enhance vision-text alignment based on these prompts. In this paper, we propose P2CLIP (Prompt-guided CLIP), which aims to bridge this gap by exploring the impact of prompts across three aspects: category, video, and frame. At the category level, we argue that LLM-generated prompts under simple instructions may suffer from semantic overgeneralization, which can blur the category boundaries and hinder the performance of the model. To address this issue, we design Generic-Discriminative Prompt using LLMs to obtain discriminative category embeddings with rich semantic information. At the video and frame levels, we first use VLM and LLM to generate frame and video descriptions for enhancing vision-text alignment. Then we propose a Knowledge-Guided Temporal Aggregation mechanism, which leverages the expert knowledge derived from LLMs to guide the temporal modeling of videos and encourage a better construction of the joint visual-textual feature space. Extensive experiments across various benchmarks, including zero-shot and few-shot settings, demonstrate the effectiveness of our method.

Figures & tables

Figure 3: Visualization of the TGTA. We leverage the expert knowledge derived from LLMs to guide the temporal modeling of videos.

Figure 4: Overview of P2CLIP for open-vocabulary video recognition. We use the typical CLIP-based video recognition pipeline. We employ GPT4 to generate Generic prompt and Discriminative prompt for each category name and encode them to obtain GD-prompt embeddings. For each input video, we employ CogVLM2 and Llama-3.1 to generate frame and video descriptions, respectively, then we encode them into textual embeddings. We calculate the similarities between frame description embeddings and video description embedding as the temporal saliency of the corresponding frame.

Figure 5: Instruction to generate Discriminative prompt and GPT4’s response.

Method	Encoder	UCF-101		HMDB-51
	Encoder	EP1	EP2	EP1	EP2
ActionCLIP [ 33 ]	ViT-B/16	-	69.5	-	50.5
Text4Vis [ 37 ]	ViT-L/14	85.8 $\pm$ 3.3	79.6	58.1 $\pm$ 5.7
BIKE [ 38 ]	ViT-L/14	86.6 $\pm$ 3.4	80.8	61.4 $\pm$ 3.6	52.8
Open-VCLIP [ 36 ]	ViT-B/16	89.9 $\pm$ 1.7	83.5	64.5 $\pm$ 4.5	53.2
Open-VCLIP++ [ 40 ]	ViT-B/16	90.0 $\pm$ 1.7	84.1	65.8 $\pm$ 4.3	54.7

Table 1: Zero-shot classification performance for CLIP-based methods on UCF-101 and HMDB-51 under EP1 and EP2. The best results are bolded , and the second-best results are underlined .

Table 6

Method	UCF-101				HMDB-51
	$K$ =2	$K$ =4	$K$ =8	$K$ =16	$K$ =2	$K$ =4	$K$ =8	$K$ =16
ActionCLIP	70.6	71.5	73.0	91.4	47.5	57.9	57.3	59.1
XCLIP [ 21 ]	48.5	75.6	83.7	91.4	53.0	57.3	62.8	64.0
A5 [ 13 ]	71.4	79.9	85.7	89.9	39.7	50.7	56.0	62.4
ViFi-CLIP † [ 27 ]	83.8	86.8	88.8	92.1	52.9	59.7	65.4	74.0
P2CLIP (ours)	86.0 $\uparrow$ 2.2	88.1 $\uparrow$ 1.3	90.1 $\uparrow$ 1.3	93.3 $\uparrow$ 1.2	56.7 $\uparrow$ 3.8	62.9 $\uparrow$ 3.2	69.4 $\uparrow$ 4.0	75.9 $\uparrow$ 1.9

Table 4: Few-shot performance for CLIP-based methods on UCF-101 and HMDB-51. We employ the variant model of ViFi-CLIP with frozen textual encoder as a baseline and simply integrate GD-prompt into it. † denotes the results of our reproduction. Gains in green are compared to baseline.

Panel of Figure 7 — Figure 7: Ablations for components on UCF-101 ( Left ) and HMDB-51 ( Right ) under few-shot setting.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Examples of frame descriptions and video descriptions, along with the temporal saliencies.

Prompt Learning for VLMs

See all 23 papers

Mar 25, 2026

Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selection

Current Large Vision Language Models (LVLMs) excel at many zero-shot tasks like image captioning, visual question answering and OCR. However, these same models suffer from poor performance at image classification tasks, underperforming against CLIP-based methods. Notably, this gap is surprising because many LVLMs use CLIP-pretrained vision encoders. Yet LVLMs are not inherently limited by CLIP's architecture with independent vision and text encoders. In CLIP, this separation biases classification toward class-name matching rather than joint visual-text reasoning. In this paper we show that, despite their poor raw performance, LVLMs can improve visual feature class separability at inference using prompt conditioning, and LVLMs' internal representations, especially attention heads, can outperform the model itself at zero-shot and few-shot classification. We introduce Head Ensemble Classifiers (HEC) to bridge the performance gap between CLIP-based and LVLM-based classification methods. Inspired by Gaussian Discriminant Analysis, HEC ranks the most discriminative vision and text heads and combines them into a training-free classifier. We show that HEC achieves state-of-the-art performance in few-shot and zero-shot classification across 12 datasets.

Adhemar de Senneville, Xavier Bou, Jérémy Anger +2

Université Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli, France · École Polytechnique, AMIAD, France · Institut Universitaire de France

Large Vision-Language Models Zero-Shot Image Classification Few-Shot Image Classification VLM Adaptation

Ask PDF

Jun 10, 2026

VL-DINO: Leveraging CLIP Vision-Language Knowledge for Open-Vocabulary Object Detectio

Vision-language models like CLIP can provide rich semantic priors for open-vocabulary object detection. However, jointly integrating both textual and visual knowledge into detection architectures remains challenging. In this paper, we propose VL-DINO, an open-vocabulary detector that enhances DINO through more effective exploitation of CLIP's vision-language knowledge. Specifically, a Query-guided Positive Sample Construction (QPSC) module is first developed to construct additional high-quality positive samples, enabling the vanilla DINO framework to better accommodate mixed training across heterogeneous data sources while providing more vision-language alignment signals, thereby incorporating richer textual knowledge during training. A Visual Semantic Encoder (VSE) module is then introduced to distill CLIP visual knowledge into backbone-extracted features, producing fused features for subsequent encoder refinement. Based on the fused features, an Object-Region Semantic Alignment (ORSA) module extracts object-centric region features and aligns them with the corresponding textual embeddings, further incorporating textual cues. In the zero-shot setting, VL-DINO-T and VL-DINO-L achieve 36.3 and 38.1 AP on the LVIS benchmark, respectively, consistently outperforming prior advanced approaches. Extensive experiments demonstrate the effectiveness and competitive performance of the proposed design.

Hao Zhang, Qinran Lin, Linqi Song +1

Chongqing University, Chongqing, China · City University of Hong Kong, Hong Kong, China

Open-Vocabulary Object Detection Detection Transformer Vision-Language Alignment

Ask PDF

Mar 9, 2026

Novel Semantic Prompting for Zero-Shot Action Recognition

Zero-shot action recognition relies on transferring knowledge from vision-language models to unseen actions using semantic descriptions. While recent methods focus on temporal modeling or architectural adaptations to handle video data, we argue that semantic prompting alone provides a strong and underexplored signal for zero-shot action understanding. We introduce SP-CLIP, a lightweight framework that augments frozen vision-language models with structured semantic prompts describing actions at multiple levels of abstraction, such as intent, motion, and object interaction. Without modifying the visual encoder or learning additional parameters, SP-CLIP aligns video representations with enriched textual semantics through prompt aggregation and consistency scoring. Experiments across standard benchmarks show that semantic prompting substantially improves zero-shot action recognition, particularly for fine-grained and compositional actions, while preserving the efficiency and generalization of pretrained models.

Salman Iqbal, Waheed Rehman

Zero-Shot Action Recognition Video Action Recognition Zero-Shot Learning Vision-Language Models LLM Prompting

Ask PDF

Abstract

Prompt Learning for VLMs

Explore similar work

Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selection

VL-DINO: Leveraging CLIP Vision-Language Knowledge for Open-Vocabulary Object Detectio

Novel Semantic Prompting for Zero-Shot Action Recognition