Knowledge is Power: Advancing Few-shot Action Recognition with Multimodal Semantics from MLLMs

Jiazheng Xing; Chao Xu; Hangjie Yuan; Mengmeng Wang; Jun Dan; Hangwei Qian; Yong Liu

cs.CVMar 27, 2026

Knowledge is Power: Advancing Few-shot Action Recognition with Multimodal Semantics from MLLMs

Authors: Jiazheng Xing, Chao Xu, Hangjie Yuan, Mengmeng Wang, Jun Dan, Hangwei Qian, Yong Liu

Organizations: College of Control Science and Engineering, Zhejiang University, Hangzhou, 310027, Zhejiang, China. · College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, 310023, Zhejiang, China. · College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, 310027, Zhejiang, China. · CFAR and IHPC, A*STAR, 138632, Singapore.

Abstract

Multimodal Large Language Models (MLLMs) have propelled the field of few-shot action recognition (FSAR). However, preliminary explorations in this area primarily focus on generating captions to form a suboptimal feature->caption->feature pipeline and adopt metric learning solely within the visual space. In this paper, we propose FSAR-LLaVA, the first end-to-end method to leverage MLLMs (such as Video-LLaVA) as a multimodal knowledge base for directly enhancing FSAR. First, at the feature level, we leverage the MLLM's multimodal decoder to extract spatiotemporally and semantically enriched representations, which are then decoupled and enhanced by our Multimodal Feature-Enhanced Module into distinct visual and textual features that fully exploit their semantic knowledge for FSAR. Next, we leverage the versatility of MLLMs to craft input prompts that flexibly adapt to diverse scenarios, and use their aligned outputs to drive our designed Composite Task-Oriented Prototype Construction, effectively bridging the distribution gap between meta-train and meta-test sets. Finally, to enable multimodal features to guide metric learning jointly, we introduce a training-free Multimodal Prototype Matching Metric that adaptively selects the most decisive cues and efficiently leverages the decoupled feature representations produced by MLLMs. Extensive experiments demonstrate superior performance across various tasks with minimal trainable parameters.

Figures & tables

Figure 1: (a), (b), and (c) indicate the pipeline of the data-driven unimodal/multimodal method, knowledge-driven CapFSAR, and our FSAR-LLaVA, respectively. Our FSAR-LLaVA Unknown , which uses the fixed input prompt: “What’s the action of the video?” without introducing additional textual label information, fully leverages the multimodal features of MLLM and achieves state‑of‑the‑art performance that requires minimal parameters, as depicted in part (d), which refers to the performance comparison in the HMDB51 5-way 1-shot task.

Figure 2: Overview of our FSAR-LLaVA: Visual inputs and text prompts are processed through our knowledge base to extract multimodal tokens $\textbf{T}_{m}$ from the multimodal decoder’s hidden layer. These tokens are downsampled and decoupled into visual tokens $\textbf{T}_{v}$ and textual tokens $\textbf{T}_{t}$ . The tokens are then passed through the Multimodal Feature-enhanced Module, and features from both branches are processed in the Composite Task-oriented Prototype Construction Module to obtain the enhanced support set class prototypes $\widetilde{\textbf{P}^{\mathcal{S}}_{v}}$ and $\widetilde{\textbf{P}^{\mathcal{S}}_{t}}$ . Finally, the class prototypes and query features are fed into the Multimodal Matching Metric to yield the probability distribution $\textbf{p}_{\mathcal{Q}2\mathcal{S}}$ and loss $\mathcal{L}_{\mathcal{Q}2\mathcal{S}}$ .

Figure 3: Overview of our CTPCM, which is divided into local prototype construction and global prototype construction.

			HMDB51		UCF101		Kinetics		SSv2-Small		SSv2-Full
	Method	Pre-training	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot
Data-driven Unimodal	OTAM Cao et al. (2020)	INet-RN50	54.5	68.0	79.9	88.9	73.0	85.8	36.4	48.0	42.8	52.3
	TRX Perrett et al. (2021)	INet-RN50	54.9	75.6	81.0	96.1	65.1	85.9	36.0	56.7	42.0	64.6
	STRM Thatipelli et al. (2022)	INet-RN50	57.6	77.3	82.7	96.9	65.1	86.7	37.1	55.3	43.1	68.1
	HyRSM Wang et al. (2022b)	INet-RN50	60.3	76.0	83.9	94.7	73.7	86.1	40.6	56.1	54.3	69.0
	HCL Zheng et al. (2022)	INet-RN50	59.1	76.3	82.5	93.9	73.7	85.8	38.7	55.4	47.3	64.9

Table 1: Comparison under 5-way k-shot settings on five widely-used benchmarks including HMDB51, UCF101, Kinetics, SSv2-Small, and SSv2-Full.

Figure 4: Visualization of the attention map for the data-driven method Hyrsm Wang et al. (2022b) and our FSAR-LLaVA on HMDB51. We also provide the Video-LLaVA’s QA results for these videos using “Unknown” prompts.

Figure 5: Direct qualitative analysis of the Video-LLaVA’s output with different types of prompts. For clarity, we highlight words in blue to represent accurate action labels, in red to reveal the question posed in Video-LLaVA, and in green to indicate inaccurate action descriptions.

Pipeline	Kinetics	HMDB	SSv2-Small
Using “Unknown” Prompts
feature $\rightarrow$ caption $\rightarrow$ feature $\rightarrow$	85.7	69.8	49.5
feature $\rightarrow$	88.1	79.5	49.9
Using “Known” Prompts
feature $\rightarrow$ caption $\rightarrow$ feature $\rightarrow$	87.8	75.4	56.6
feature $\rightarrow$	91.1	83.8	60.7

Table 2: Performance comparison with different pipelines.

	Kinetics		HMDB51		UCF101		SSv2-Small
Method	1s	5s	1s	5s	1s	5s	1s	5s
Avg.	79.3	92.6	61.0	81.2	91.9	97.9	32.7	47.6
Dec-Avg.	82.1	93.5	63.3	82.7	93.2	98.3	33.5	48.8

Table 3: The Necessity of Multimodal Feature Decoupling. “Avg.” and “Dec-Avg.” indicate the global average pooling and global average pooling with decoupling the features.

Panel of Table 4 — Table 4: Effect of different branches. “Ukn.” and “Kn.” refer to Unknown and Known.

Panel of Table 5 — Table 5: Impact of the components in MFM. “ST-SA”, “V-CA”, and “T-CA” indicate spatiotemporal self-attention, visual-lead cross-attention, and text-lead cross-attention, respectively. “Ukn.” and “Kn.” refer to Unknown and Known.

Panel of Table 6 — Table 6: Impact of the components in CTPCM. “LPC” and “GPC” represent local and global prototype construction. “Ukn.” and “Kn.” refer to Unknown and Known.

	Kinetics		SSv2-Small
Method	Unknown	Known	Unknown	Known
Hausdorff	92.0	92.3	44.0	45.5
Bi-MHM	93.7	94.0	48.9	60.6
Our MPMM	94.8	95.3	51.0	65.5

Table 7: Comparison of different matching metrics.

Figure 6: Performance comparison between our knowledge-driven FSAR-LLaVA and the data-driven baseline of the 5-way 1-shot task on five widely used datasets. All methods share the same vision encoder.

	Pre-Training	Params	FLOPs	Time	Acc
		Trainable/Full	Backbone/Module/Total
OTAM Cao et al. (2020)	INet-RN50	23.5 M / 23.5 M	32.8 G / 8.3 G / 41.1 G	0.48 s/item	54.5
HyRSM Wang et al. (2022b)	INet-RN50	65.6 M / 65.6 M	32.8 G / 8.5 G / 41.3 G	0.51 s/item	60.3
CLIP-FSAR Wang et al. (2023b)	CLIP-ViT-B	90.2 M / 90.2 M	140.8 G / 24.0 G / 164.8 G	1.28 s/item	77.1
FSAR-LLaVA (Offline)	Video-LLaVA-7B	9.3 M / 9.3 M	- / 30.4 G / 30.4 G	0.18 s/item	84.3
FSAR-LLaVA (Online)	Video-LLaVA-7B	9.3 M / 7 B	30.19 T / 0.03 T / 30.22 T	4.39 s/item	84.3

Table 8: Complexity analysis for 5-way 1-shot HMDB51 evaluation. Our FSAR-LLaVA utilizes the “Unknown” prompts here. “Time” represents the training time cost of each 5-way 1-shot task. “Acc” means the 5-way 1-shot accuracy on HMDB51. All experiments are carried out on one Nvidia H20 GPU.

	Trainable Params	Kinetics	SSv2-Small
$D^{\prime}=128$	6.38M	94.0	50.1
$D^{\prime}=256$	9.26M	94.8	51.0
$D^{\prime}=512$	16.52M	95.1	51.4
$D^{\prime}=1024$	37.02M	95.2	53.1

Table 9: Effectiveness comparison with different mapping dimension

D^{\prime}

	Kinetics		SSv2-Small
	Unknown	Known	Unknown	Known
$\alpha=0$	93.0	94.0	50.3	63.9
$\alpha=0.1$	93.3	94.2	51.0	65.5
$\alpha=0.3$	93.9	94.4	49.9	63.7
$\alpha=0.5$	94.2	94.7	49.7	62.9
$\alpha=0.7$	94.5	94.9	49.4	62.3

Table 10: Analysis of learnable parameter

\alpha

in CTPCM.

	Kinetics		HMDB51		SSv2-Small
	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot
$u=10$	94.8	97.1	84.3	92.4	51.0	71.1
$u=20$	94.7	97.2	84.2	92.4	50.6	70.2
$u=30$	94.7	97.2	84.2	92.5	50.1	69.3
$u=40$	94.5	97.3	84.1	92.6	49.9	68.6
$u=50$	94.4	97.4	84.0	92.7	49.7	68.3

Table 11: Analysis of adjustable parameter

u

in MPMM.

	Kinetics	HMDB	UCF	SSv2-Small	SSv2-Full	Avg.
$l=0$	76.8	58.7	91.1	29.5	30.3	57.3
$l=1$	76.7	58.4	91.1	29.2	30.1	57.1
$l=2$	76.6	58.4	91.0	28.9	30.0	57.0
$l=3$	76.4	58.7	91.4	29.2	29.8	57.1
$l=4$	76.5	59.0	91.2	29.1	29.6	57.1
$l=5$	77.6	59.4	91.8	29.5	29.2	57.5

Table 12: Performance comparison of the hidden layer features from Video-LLaVA. The boldfacen and underline font indicate the highest and the second highest results. Note that “Avg.” denotes Average.

Prompt	Pre-training	SSv2-Small		SSv2-Full
		1-shot	5-shot	1-shot	5-shot
“What’s the action of the video?”	Video-LLaVA-7B	51.0	71.1	56.7	76.7
“Describe the sequence of actions happening in the video.”	Video-LLaVA-7B	51.3	71.0	56.5	76.4
“How does the interaction between the person and the object evolve?”	Video-LLaVA-7B	52.0	71.6	57.1	77.2
“What’s the action of the video?”	Qwen3-VL-2B	54.2	71.7	56.6	77.1
“Describe the sequence of actions happening in the video.”	Qwen3-VL-2B	54.1	71.5	56.7	77.0

Table 13: Effect of different “Unknown” prompts on temporal-related datasets using different MLLMs within our framework.

Explore similar work

STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition
May 13, 2026Hongli Liu, Yu Wang, Shengjie ZhaoCross-Modal Representation Learning Few-Shot Learning
School of Computer Science and Technology, Tongji University, Shanghai, China · Engineering Research Center of Key Software Technologies for Smart City Perception and Planning, Ministry of Education
Few-Shot Video Recognition via Hierarchical Metric Learning
Sep 4, 2026Jiaxin Zhang, Haoran Gao, Xizhan Gao +3Few-Shot Learning Video Action Recognition
School of Information Science and Engineering, University of Jinan, Shandong, China
Spatio-temporal Decoupled Knowledge Compensator for Few-Shot Action Recognition
Feb 20, 2026Hongyu Qu, Xiangbo Shu, Rui Yan +3Video Action Recognition Few-Shot Learning
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China. · State Key Lab of Brain-Machine Intelligence, Zhejiang University, China.

May 13, 2026J/K move · Enter open · S save

STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition

Hongli Liu, Yu Wang, Shengjie Zhao

School of Computer Science and Technology, Tongji University, Shanghai, China · Engineering Research Center of Key Software Technologies for Smart City Perception and Planning, Ministry of Education

Cross-Modal Representation Learning Few-Shot Learning Visual SSMs Video Representation Learning

Open paper AskPDF arXiv

Few-shot action recognition (FSAR) requires models to generalize to novel action categories from only a handful of annotated samples. Despite progress with vision-language models, existing approaches still suffer from semantic-temporal misalignment, where static textual prompts fail to capture decisive visual cues that appear sparsely across sequences, and from inadequate modeling of multi-scale temporal dynamics, as short-term discriminative cues and long-range dependencies are often either oversmoothed or fragmented. To address these challenges, we propose Semantic Temporal Adaptive Representation Learning (STAR), a unified framework, consisting of a semantic-alignment component and a temporal-aware component, effectively bridging the semantic and temporal gaps and transferring the sequence modeling capability of Mamba into the FSAR. The semantic alignment module introduces a Temporal Semantic Attention (TSA) mechanism, which performs frame-level cross-modal alignment with textual cues, ensuring fine-grained semantic-temporal consistency. The temporal-aware module incorporates a Semantic Temporal Prototype Refiner (STPR) that integrates semantic-guided Mamba blocks with multi-frequency temporal sampling and bidirectional state-space refinement, yielding semantically aligned prototypes with enhanced discriminative fidelity and temporal consistency. Furthermore, temporally dependent class descriptors derived from large language models (LLMs) provide long-range semantic guidance. Extensive experiments on five FSAR benchmarks demonstrate the consistent superiority of STAR over state-of-the-art methods. For instance, STAR achieves up to 8.1% and 6.7% gains on the SSv2-Full and SSv2-Small datasets under the 1-shot setting, and 7.3% on HMDB51, validating its effectiveness under limited supervision. The code is available at https://github.com/HongliLiu1/STAR-main.

Abstract

Explore similar work

STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition

Few-Shot Video Recognition via Hierarchical Metric Learning

Spatio-temporal Decoupled Knowledge Compensator for Few-Shot Action Recognition