Spatio-temporal Decoupled Knowledge Compensator for Few-Shot Action Recognition

Hongyu Qu; Xiangbo Shu; Rui Yan; Hailiang Gao; Wenguan Wang; Jinhui Tang

cs.CVFeb 20, 2026

Spatio-temporal Decoupled Knowledge Compensator for Few-Shot Action Recognition

Authors: Hongyu Qu, Xiangbo Shu, Rui Yan, Hailiang Gao, Wenguan Wang, Jinhui Tang

Organizations: School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China. · State Key Lab of Brain-Machine Intelligence, Zhejiang University, China.

Abstract

Few-Shot Action Recognition (FSAR) is a challenging task that requires recognizing novel action categories with a few labeled videos. Recent works typically apply semantically coarse category names as auxiliary contexts to guide the learning of discriminative visual features. However, such context provided by the action names is too limited to provide sufficient background knowledge for capturing novel spatial and temporal concepts in actions. In this paper, we propose DiST, an innovative Decomposition-incorporation framework for FSAR that makes use of decoupled Spatial and Temporal knowledge provided by large language models to learn expressive multi-granularity prototypes. In the decomposition stage, we decouple vanilla action names into diverse spatio-temporal attribute descriptions (action-related knowledge). Such commonsense knowledge complements semantic contexts from spatial and temporal perspectives. In the incorporation stage, we propose Spatial/Temporal Knowledge Compensators (SKC/TKC) to discover discriminative object-level and frame-level prototypes, respectively. In SKC, object-level prototypes adaptively aggregate important patch tokens under the guidance of spatial knowledge. Moreover, in TKC, frame-level prototypes utilize temporal attributes to assist in inter-frame temporal relation modeling. These learned prototypes thus provide transparency in capturing fine-grained spatial details and diverse temporal patterns. Experimental results show DiST achieves state-of-the-art results on five standard FSAR datasets.

Figures & tables

Fig. 1 : Our main idea . DiST decomposes category names into diverse spatio-temporal knowledge, and makes use of such decoupled knowledge to guide the learning of object-level and frame-level prototypes, respectively.

Panel of Figure 2 — Fig. 2 : Overview of DiST . Video inputs are first processed by the visual encoder of CLIP to obtain initial patch-level and frame-level features. Then, we leverage LLM to decompose vanilla action names into action-related background knowledge (§ III-C ). Furthermore, SKC/TKC incorporate decoupled prior knowledge and visual features to form discriminative object- and frame-level prototypes for spatial/temporal matching (§ III-D and § III-E ). Finally, we can combine spatial and temporal matching results to obtain the merged query prediction (§ III-F ).

Fig. 3 : Illustration of Spatial Knowledge Compensator (SKC) (§ III-D ). SKC aims to learn discriminative object-level prototypes in a sparse aggregation manner via patch aggregation and attribute injection.

			HMDB51			UCF101
Method		Pre-training	$1$ -shot (%)	$3$ -shot (%)	$5$ -shot (%)	$1$ -shot (%)	$3$ -shot (%)	$5$ -shot (%)
ProtoNet [ 55 ]	[NeurIPS17]	INet-RN50	$54.2$	-	$68.4$	$74.0$	-	$89.6$
ARN [ 15 ]	[ECCV20]	C3D	$45.5$	-	$60.6$	$66.3$	-	$83.1$
OTAM [ 16 ]	[CVPR20]	INet-RN50	$54.5$	$65.7$	-	$79.9$	$87.0$	-
AmeFu-Net [ 76 ]	[MM20]	INet-RN50	$60.2$	$-$	$75.5$	$85.1$	$-$	$95.5$
Lite-MKD [ 77 ]	[MM23]	INet-RN50	$66.9$	$-$	$74.7$	$85.3$	$-$	$96.8$

TABLE I : Quantitative comparison results on HMDB51 [ 38 ] and UCF101 [ 39 ] (see § IV-C ). The experiments are conducted under the

5

-way

K

-shot. “INet-RN50” denotes ResNet-50 pre-trained on ImageNet. We highlight best , second best results.

			Kinetics		SSv2		SSv2-small
Method		Pre-training	$1$ -shot (%)	$5$ -shot (%)	$1$ -shot (%)	$5$ -shot (%)	$1$ -shot (%)	$5$ -shot (%)
ProtoNet [ 55 ]	[NeurIPS17]	INet-RN50	$64.5$	$77.9$	-	-	-	-
ARN [ 15 ]	[ECCV20]	C3D	$63.7$	$82.4$	-	-	-	-
OTAM [ 16 ]	[CVPR20]	INet-RN50	$73.0$	$85.8$	$42.8$	$52.3$	$36.4$	$48.0$
Lite-KMD [ 77 ]	[MM23]	INet-RN50	$75.0$	$85.7$	$55.7$	$69.9$	-	-
TRX [ 21 ]	[CVPR21]	INet-RN50	$63.6$	$85.9$	$42.0$	$64.6$	$36.0$	$56.7$

TABLE II :

{}_{\!}

Quantitative results

{}_{\!}

{}_{\!}

Kinetics

{}_{\!}

[ 40 ] , SSv2 [ 41 ] and

{}_{\!}

SSv2-small [ 41 ] (see

{}_{\!}

§ IV-C ). The

{}_{\!}

experiments are

{}_{\!}

conducted

{}_{\!}

under

{}_{\!}

the

{}_{\!}

5

-way

{}_{\!}

K

-shot. “INet-RN50”

{}_{\!}

denotes

{}_{\!}

ResNet-50

{}_{\!}

pre-trained

{}_{\!}

{}_{\!}

ImageNet. We highlight best , second best results.

		HMDB51 (%)		UCF101 (%)		Kinetics (%)		SSv2 (%)		SSv2-small (%)
Method		$1$ -shot	$5$ -shot	$1$ -shot	$5$ -shot	$1$ -shot	$5$ -shot	$1$ -shot	$5$ -shot	$1$ -shot	$5$ -shot
CLIP-FSAR [ 28 ]	[IJCV24]	$77.1$	$87.7$	$97.0$	$99.1$	$94.8$	$95.4$	$62.1$	$72.1$	$54.6$	$61.8$
DiST (Ours)		$\mathbf{82.6}$	$\mathbf{88.7}$	$\mathbf{98.3}$	$\mathbf{99.2}$	$\mathbf{95.6}$	$\mathbf{96.0}$	$\mathbf{64.6}$	$\mathbf{75.8}$	$\mathbf{57.5}$	$\mathbf{63.5}$

TABLE III : Quantitative comparison results on UCF101 [ 39 ] , HMDB51 [ 38 ] , Kinetics [ 40 ] , SSv2 [ 41 ] and SSv2-small [ 41 ] (see § IV-C ) by combining few-shot and zero-shot results.

TABLE IV :

{}_{\!}

{}_{\!}

set

{}_{\!}

{}_{\!}

ablation

{}_{\!}

studies

{}_{\!}

{}_{\!}

HMDB51 [ 38 ] and SSv2-small [ 41 ] (see

{}_{\!}

§ IV-D ). The

{}_{\!}

adopted

{}_{\!}

designs

{}_{\!}

are

{}_{\!}

marked

{}_{\!}

{}_{\!}

red .

Method Component	HMDB51(%)	SSv2-small(%)	UCF101(%)
Baseline	$75.8$	$53.8$	$96.0$
SKC only	$77.6$	$55.7$	$96.6$
TKC only	$81.0$	$56.1$	$97.9$
DiST (Ours)	$\mathbf{82.6}$	$\mathbf{57.5}$	$\mathbf{98.3}$

TABLE V : Impacts of core components on HMDB51 [ 38 ] , SSv2-small [ 40 ] , and UCF101 [ 39 ] under

1

-shot tasks (§ IV-D ).

Panel of Figure 9 — Fig. 4 : Left : The impact of the varying fusion parameter $\alpha$ on HMDB51 [ 38 ] in the $5$ -way $1$ -shot setting (see § IV-D ). Right : 5-way 1-shot class improvement of DiST compared to CLIP-FSAR [ 28 ] on all class action classes on HMDB51 [ 38 ] (see § IV-F ). Our DiST achieves improvement on all action classes.

	HMDB51 (%)
{ $G$ , $L$ }	$1$ -shot	$5$ -shot
$\{1,3\}$	$81.8$	$87.9$
$\{3,3\}$	$82.4$	$88.5$
$\{9,3\}$	$82.5$	$88.5$
$\{12,3\}$	$82.3$	$88.3$
$\{6,1\}$	$81.4$	$88.0$

TABLE VI : The ablation study on HMDB51 [ 38 ] to investigate the configuration of the number of spatial attributes

G

and temporal attributes

L

(see § IV-D ).

Prototype Number $N$	HMDB51	SSv2-small
$N=1$	$80.8$	$56.4$
$N=4$	$81.7$	$56.9$
$N=\mathbf{9}$	$\mathbf{82.6}$	$\mathbf{57.5}$
$N=12$	$82.3$	$57.2$
$N=16$	$82.1$	$57.0$

TABLE VII : The ablation study on HMDB51 [ 38 ] and SSv2-small [ 41 ] under the 1-shot setting to investigate the number of object-level prototypes

N

(see § IV-D ).

Method	Params	FLOPs	Memory	Speed	HMDB51 (%)	Kinetics (%)
CLIP-FSAR [ 28 ]	$89.3$ M	$901.9$ G	$13.8$ G	$36.7$ ms	$75.8$	$89.7$
DiST (ours)	$97.2$ M	$902.1$ G	$14.1$ G	$40.9$ ms	$82.6$	$92.7$

TABLE VIII : Complexity analysis for

5

-way

1

-shot HMDB51 [ 38 ] and Kinetics [ 40 ] evaluation. Here, we report Params, FLOPs, GPU memory, and Speed for each model. See § IV-D for more details.

LLM version	HMDB51 (%)	UCF101 (%)
Baseline	$75.8$	$96.0$
Llama-2 [ 82 ]	$82.1$	$98.2$
Vicuna [ 83 ]	$82.5$	$98.0$
GPT-3.5 [ 75 ] (ours)	$\mathbf{82.6}$	$\mathbf{98.3}$

TABLE IX : Comparison results with different large language models used for spatial-temporal attribute generation on HMDB51 [ 38 ] and UCF101 [ 39 ] in the

5

-way

{}_{\!}

1

-shot

{}_{\!}

tasks

{}_{\!}

(see

{}_{\!}

§ IV-D ). Baseline denotes that DIST only relies on category names for TKC and SKC.

Method	Visual encoder	HMDB51	Kinetics	SSv2-small
CLIP-Freeze [ 84 ]	Frozen	58.2	78.9	29.5
DiST (ours)	Frozen	62.6	83.1	35.9
DiST (ours)	Fine-tuning	84.7	95.9	60.7

TABLE X : Impact of fine-tuning visual encoder on the generalization capacity of DIST on HMDB51 [ 38 ] , Kinetics [ 40 ] , and SSv2-small [ 41 ] in the

5

-way

1

-shot tasks (see

{}_{\!}

§ IV-D ).

Spatial Interaction	HMDB51(%)	SSv2-small(%)	UCF101(%)
Baseline	$75.8$	$53.8$	$96.0$
PLE [ 26 ]	$76.0$	$54.1$	$95.8$
STMM [ 64 ]	$76.2$	$54.4$	$96.1$
DiST (Ours)	$\mathbf{77.6}$	$\mathbf{55.7}$	$\mathbf{96.6}$

TABLE XI : Comparison results between Spatial Knowledge Compensator and existing spatial interaction methods on HMDB51 [ 38 ] , SSv2-small [ 40 ] , and UCF101 [ 39 ] under

1

-shot tasks (§ IV-D ).

Panel of Figure 16 — Fig. 5 : Visualization of spatial and temporal prompts under the $1$ -shot setting (§ IV-F ). The spatial prompts are shown as highlighted response areas in each frame. We also show cross-attention temporal prompt weights of Eq. 4 in a line graph.

Fig. 6 : T-SNE visualization [ 85 ] under the $5$ -way setting on HMDB51 and Kinetics (see § IV-F ).

		SSv2-small (%)		Kinetics (%)
Method	Backbone	$1$ -shot	$5$ -shot	$1$ -shot	$5$ -shot
CLIP-FSAR [ 28 ]		46.0	52.4	72.9	84.1
DiST (ours)	INet-RN18	48.6	53.3	75.6	86.5
CLIP-FSAR [ 28 ]		46.7	53.3	75.5	85.9
DiST (ours)	INet-RN50	49.1	54.6	78.6	88.0

TABLE XII : Generalization performance of our DiST with different ImageNet-pretrained backbones on SSv2-small [ 40 ] , and Kinetics [ 39 ] in the

5

-way

1

-shot and

5

-shot tasks (see

{}_{\!}

§ IV-E ).

Method	Fine-tuning	HMDB51	UCF101	Kinetics	SSv2-small
D 2 ST-Adapter [ 84 ]	PEFT	77.1	96.4	89.3	55.0
MA-FSAR [ 86 ]	PEFT	83.4	97.2	95.7	59.1
Task-Adapter [ 87 ]	PEFT	83.6	98.0	95.0	60.2
Ours	PEFT	84.7	98.3	95.9	60.7

TABLE XIII : Generalization performance of our DiST under the parameter-efficient fine-tuning setting on UCF101 [ 39 ] , HMDB51 [ 38 ] , Kinetics [ 40 ] , SSv2 [ 41 ] and SSv2-small [ 41 ] in the

5

-way

1

-shot tasks (see

{}_{\!}

§ IV-E ).

Explore similar work

STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition
May 13, 2026Hongli Liu, Yu Wang, Shengjie ZhaoCross-Modal Representation Learning Few-Shot Learning
School of Computer Science and Technology, Tongji University, Shanghai, China · Engineering Research Center of Key Software Technologies for Smart City Perception and Planning, Ministry of Education
Hierarchical Relation-augmented Representation Generalization for Few-shot Action Recognition
Apr 14, 2025Hongyu Qu, Ling Xing, Jiachao Zhang +3Temporal Reasoning Hierarchical Representation Learning
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · Artificial Intelligence Industrial Technology Research Institute, Nanjing Institute of Technology, Nanjing 211167, China
Few-Shot Video Recognition via Hierarchical Metric Learning
Sep 4, 2026Jiaxin Zhang, Haoran Gao, Xizhan Gao +3Few-Shot Learning Video Action Recognition
School of Information Science and Engineering, University of Jinan, Shandong, China

May 13, 2026J/K move · Enter open · S save

STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition

Hongli Liu, Yu Wang, Shengjie Zhao

School of Computer Science and Technology, Tongji University, Shanghai, China · Engineering Research Center of Key Software Technologies for Smart City Perception and Planning, Ministry of Education

Cross-Modal Representation Learning Few-Shot Learning Visual SSMs Video Representation Learning

Open paper AskPDF arXiv

Few-shot action recognition (FSAR) requires models to generalize to novel action categories from only a handful of annotated samples. Despite progress with vision-language models, existing approaches still suffer from semantic-temporal misalignment, where static textual prompts fail to capture decisive visual cues that appear sparsely across sequences, and from inadequate modeling of multi-scale temporal dynamics, as short-term discriminative cues and long-range dependencies are often either oversmoothed or fragmented. To address these challenges, we propose Semantic Temporal Adaptive Representation Learning (STAR), a unified framework, consisting of a semantic-alignment component and a temporal-aware component, effectively bridging the semantic and temporal gaps and transferring the sequence modeling capability of Mamba into the FSAR. The semantic alignment module introduces a Temporal Semantic Attention (TSA) mechanism, which performs frame-level cross-modal alignment with textual cues, ensuring fine-grained semantic-temporal consistency. The temporal-aware module incorporates a Semantic Temporal Prototype Refiner (STPR) that integrates semantic-guided Mamba blocks with multi-frequency temporal sampling and bidirectional state-space refinement, yielding semantically aligned prototypes with enhanced discriminative fidelity and temporal consistency. Furthermore, temporally dependent class descriptors derived from large language models (LLMs) provide long-range semantic guidance. Extensive experiments on five FSAR benchmarks demonstrate the consistent superiority of STAR over state-of-the-art methods. For instance, STAR achieves up to 8.1% and 6.7% gains on the SSv2-Full and SSv2-Small datasets under the 1-shot setting, and 7.3% on HMDB51, validating its effectiveness under limited supervision. The code is available at https://github.com/HongliLiu1/STAR-main.

Abstract

Explore similar work

STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition

Hierarchical Relation-augmented Representation Generalization for Few-shot Action Recognition

Few-Shot Video Recognition via Hierarchical Metric Learning