Revisiting Natural Language Prompts from Large Language Models for Open-Vocabulary Video Recognition

Haichen He; Weibin Liu; Weiwei Xing

cs.CVOct 12, 2026

Revisiting Natural Language Prompts from Large Language Models for Open-Vocabulary Video Recognition

Authors: Haichen He, Weibin Liu, Weiwei Xing

Organizations: School of Computer Science and Technology, Beijing Jiaotong University · Visual Intelligence +X International Cooperation Joint Laboratory of MOE · School of Software Engineering, Beijing Jiaotong University

Abstract

Vision-language models (VLMs) like CLIP have shown great success in image tasks, however, applying CLIP to open-vocabulary video recognition poses challenges. Due to the limited text information, directly fine-tuning CLIP on video recognition datasets may lead to overfitting and hinder its generalizability. A prevalent method is to enrich text information by generating prompts using Large Language Models (LLMs). However, limited efforts have been made to explore how to generate higher-quality prompts that can contribute to classify unseen categories, and how to further enhance vision-text alignment based on these prompts. In this paper, we propose P2CLIP (Prompt-guided CLIP), which aims to bridge this gap by exploring the impact of prompts across three aspects: category, video, and frame. At the category level, we argue that LLM-generated prompts under simple instructions may suffer from semantic overgeneralization, which can blur the category boundaries and hinder the performance of the model. To address this issue, we design Generic-Discriminative Prompt using LLMs to obtain discriminative category embeddings with rich semantic information. At the video and frame levels, we first use VLM and LLM to generate frame and video descriptions for enhancing vision-text alignment. Then we propose a Knowledge-Guided Temporal Aggregation mechanism, which leverages the expert knowledge derived from LLMs to guide the temporal modeling of videos and encourage a better construction of the joint visual-textual feature space. Extensive experiments across various benchmarks, including zero-shot and few-shot settings, demonstrate the effectiveness of our method.

Figures & tables

Figure 3: Visualization of the TGTA. We leverage the expert knowledge derived from LLMs to guide the temporal modeling of videos.

Figure 4: Overview of P2CLIP for open-vocabulary video recognition. We use the typical CLIP-based video recognition pipeline. We employ GPT4 to generate Generic prompt and Discriminative prompt for each category name and encode them to obtain GD-prompt embeddings. For each input video, we employ CogVLM2 and Llama-3.1 to generate frame and video descriptions, respectively, then we encode them into textual embeddings. We calculate the similarities between frame description embeddings and video description embedding as the temporal saliency of the corresponding frame.

Figure 5: Instruction to generate Discriminative prompt and GPT4’s response.

Method	Encoder	UCF-101		HMDB-51
	Encoder	EP1	EP2	EP1	EP2
ActionCLIP [ 33 ]	ViT-B/16	-	69.5	-	50.5
Text4Vis [ 37 ]	ViT-L/14	85.8 $\pm$ 3.3	79.6	58.1 $\pm$ 5.7
BIKE [ 38 ]	ViT-L/14	86.6 $\pm$ 3.4	80.8	61.4 $\pm$ 3.6	52.8
Open-VCLIP [ 36 ]	ViT-B/16	89.9 $\pm$ 1.7	83.5	64.5 $\pm$ 4.5	53.2
Open-VCLIP++ [ 40 ]	ViT-B/16	90.0 $\pm$ 1.7	84.1	65.8 $\pm$ 4.3	54.7

Table 1: Zero-shot classification performance for CLIP-based methods on UCF-101 and HMDB-51 under EP1 and EP2. The best results are bolded , and the second-best results are underlined .

Table 6

Method	UCF-101				HMDB-51
	$K$ =2	$K$ =4	$K$ =8	$K$ =16	$K$ =2	$K$ =4	$K$ =8	$K$ =16
ActionCLIP	70.6	71.5	73.0	91.4	47.5	57.9	57.3	59.1
XCLIP [ 21 ]	48.5	75.6	83.7	91.4	53.0	57.3	62.8	64.0
A5 [ 13 ]	71.4	79.9	85.7	89.9	39.7	50.7	56.0	62.4
ViFi-CLIP † [ 27 ]	83.8	86.8	88.8	92.1	52.9	59.7	65.4	74.0
P2CLIP (ours)	86.0 $\uparrow$ 2.2	88.1 $\uparrow$ 1.3	90.1 $\uparrow$ 1.3	93.3 $\uparrow$ 1.2	56.7 $\uparrow$ 3.8	62.9 $\uparrow$ 3.2	69.4 $\uparrow$ 4.0	75.9 $\uparrow$ 1.9

Table 4: Few-shot performance for CLIP-based methods on UCF-101 and HMDB-51. We employ the variant model of ViFi-CLIP with frozen textual encoder as a baseline and simply integrate GD-prompt into it. † denotes the results of our reproduction. Gains in green are compared to baseline.

Panel of Figure 7 — Figure 7: Ablations for components on UCF-101 ( Left ) and HMDB-51 ( Right ) under few-shot setting.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Examples of frame descriptions and video descriptions, along with the temporal saliencies.

Prompt Learning for VLMs

See all 23 papers

Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selection
Mar 25, 2026Adhemar de Senneville, Xavier Bou, Jérémy Anger +2Large Vision-Language Models Zero-Shot Image Classification
Université Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli, France · École Polytechnique, AMIAD, France · Institut Universitaire de France
VL-DINO: Leveraging CLIP Vision-Language Knowledge for Open-Vocabulary Object Detectio
Jun 10, 2026Hao Zhang, Qinran Lin, Linqi Song +1Open-Vocabulary Object Detection Detection Transformer
Chongqing University, Chongqing, China · City University of Hong Kong, Hong Kong, China
Novel Semantic Prompting for Zero-Shot Action Recognition
Mar 9, 2026Salman Iqbal, Waheed RehmanZero-Shot Action Recognition Video Action Recognition

Mar 25, 2026J/K move · Enter open · S save

Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selection

Adhemar de Senneville, Xavier Bou, Jérémy Anger, Rafael Grompone+1

Université Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli, France · École Polytechnique, AMIAD, France · Institut Universitaire de France

Large Vision-Language Models Zero-Shot Image Classification Few-Shot Image Classification VLM Adaptation

Open paper AskPDF arXiv

Current Large Vision Language Models (LVLMs) excel at many zero-shot tasks like image captioning, visual question answering and OCR. However, these same models suffer from poor performance at image classification tasks, underperforming against CLIP-based methods. Notably, this gap is surprising because many LVLMs use CLIP-pretrained vision encoders. Yet LVLMs are not inherently limited by CLIP's architecture with independent vision and text encoders. In CLIP, this separation biases classification toward class-name matching rather than joint visual-text reasoning. In this paper we show that, despite their poor raw performance, LVLMs can improve visual feature class separability at inference using prompt conditioning, and LVLMs' internal representations, especially attention heads, can outperform the model itself at zero-shot and few-shot classification. We introduce Head Ensemble Classifiers (HEC) to bridge the performance gap between CLIP-based and LVLM-based classification methods. Inspired by Gaussian Discriminant Analysis, HEC ranks the most discriminative vision and text heads and combines them into a training-free classifier. We show that HEC achieves state-of-the-art performance in few-shot and zero-shot classification across 12 datasets.

Abstract

Prompt Learning for VLMs

Explore similar work

Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selection

VL-DINO: Leveraging CLIP Vision-Language Knowledge for Open-Vocabulary Object Detectio

Novel Semantic Prompting for Zero-Shot Action Recognition