Organizations: School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China. · State Key Lab of Brain-Machine Intelligence, Zhejiang University, China.
Few-Shot Action Recognition (FSAR) is a challenging task that requires recognizing novel action categories with a few labeled videos. Recent works typically apply semantically coarse category names as auxiliary contexts to guide the learning of discriminative visual features. However, such context provided by the action names is too limited to provide sufficient background knowledge for capturing novel spatial and temporal concepts in actions. In this paper, we propose DiST, an innovative Decomposition-incorporation framework for FSAR that makes use of decoupled Spatial and Temporal knowledge provided by large language models to learn expressive multi-granularity prototypes. In the decomposition stage, we decouple vanilla action names into diverse spatio-temporal attribute descriptions (action-related knowledge). Such commonsense knowledge complements semantic contexts from spatial and temporal perspectives. In the incorporation stage, we propose Spatial/Temporal Knowledge Compensators (SKC/TKC) to discover discriminative object-level and frame-level prototypes, respectively. In SKC, object-level prototypes adaptively aggregate important patch tokens under the guidance of spatial knowledge. Moreover, in TKC, frame-level prototypes utilize temporal attributes to assist in inter-frame temporal relation modeling. These learned prototypes thus provide transparency in capturing fine-grained spatial details and diverse temporal patterns. Experimental results show DiST achieves state-of-the-art results on five standard FSAR datasets.
Figures & tables
Fig. 1 : Our main idea . DiST decomposes category names into diverse spatio-temporal knowledge, and makes use of such decoupled knowledge to guide the learning of object-level and frame-level prototypes, respectively.
Fig. 2 : Overview of DiST . Video inputs are first processed by the visual encoder of CLIP to obtain initial patch-level and frame-level features. Then, we leverage LLM to decompose vanilla action names into action-related background knowledge (§ III-C ). Furthermore, SKC/TKC incorporate decoupled prior knowledge and visual features to form discriminative object- and frame-level prototypes for spatial/temporal matching (§ III-D and § III-E ). Finally, we can combine spatial and temporal matching results to obtain the merged query prediction (§ III-F ).
Fig. 3 : Illustration of Spatial Knowledge Compensator (SKC) (§ III-D ). SKC aims to learn discriminative object-level prototypes in a sparse aggregation manner via patch aggregation and attribute injection.
HMDB51
UCF101
Method
Pre-training
1 -shot (%)
3 -shot (%)
5 -shot (%)
1 -shot (%)
3 -shot (%)
5 -shot (%)
ProtoNet [ 55 ]
[NeurIPS17]
INet-RN50
54.2
-
68.4
74.0
-
89.6
ARN [ 15 ]
[ECCV20]
C3D
45.5
-
60.6
66.3
-
83.1
OTAM [ 16 ]
[CVPR20]
INet-RN50
54.5
65.7
-
79.9
87.0
-
AmeFu-Net [ 76 ]
[MM20]
INet-RN50
60.2
−
75.5
85.1
−
95.5
Lite-MKD [ 77 ]
[MM23]
INet-RN50
66.9
−
74.7
85.3
−
96.8
TABLE I : Quantitative comparison results on HMDB51 [ 38 ] and UCF101 [ 39 ] (see § IV-C ). The experiments are conducted under the 5 -way K -shot. “INet-RN50” denotes ResNet-50 pre-trained on ImageNet. We highlight best , second best results.
Kinetics
SSv2
SSv2-small
Method
Pre-training
1 -shot (%)
5 -shot (%)
1 -shot (%)
5 -shot (%)
1 -shot (%)
5 -shot (%)
ProtoNet [ 55 ]
[NeurIPS17]
INet-RN50
64.5
77.9
-
-
-
-
ARN [ 15 ]
[ECCV20]
C3D
63.7
82.4
-
-
-
-
OTAM [ 16 ]
[CVPR20]
INet-RN50
73.0
85.8
42.8
52.3
36.4
48.0
Lite-KMD [ 77 ]
[MM23]
INet-RN50
75.0
85.7
55.7
69.9
-
-
TRX [ 21 ]
[CVPR21]
INet-RN50
63.6
85.9
42.0
64.6
36.0
56.7
TABLE II : Quantitative results on Kinetics [ 40 ] , SSv2 [ 41 ] and SSv2-small [ 41 ] (see § IV-C ). The experiments are conducted under the 5 -way K -shot. “INet-RN50” denotes ResNet-50 pre-trained on ImageNet. We highlight best , second best results.
HMDB51 (%)
UCF101 (%)
Kinetics (%)
SSv2 (%)
SSv2-small (%)
Method
1 -shot
5 -shot
1 -shot
5 -shot
1 -shot
5 -shot
1 -shot
5 -shot
1 -shot
5 -shot
CLIP-FSAR [ 28 ]
[IJCV24]
77.1
87.7
97.0
99.1
94.8
95.4
62.1
72.1
54.6
61.8
DiST (Ours)
82.6
88.7
98.3
99.2
95.6
96.0
64.6
75.8
57.5
63.5
TABLE III : Quantitative comparison results on UCF101 [ 39 ] , HMDB51 [ 38 ] , Kinetics [ 40 ] , SSv2 [ 41 ] and SSv2-small [ 41 ] (see § IV-C ) by combining few-shot and zero-shot results.
TABLE IV : A set of ablation studies on HMDB51 [ 38 ] and SSv2-small [ 41 ] (see § IV-D ). The adopted designs are marked in red .
Method Component
HMDB51(%)
SSv2-small(%)
UCF101(%)
Baseline
75.8
53.8
96.0
SKC only
77.6
55.7
96.6
TKC only
81.0
56.1
97.9
DiST (Ours)
82.6
57.5
98.3
TABLE V : Impacts of core components on HMDB51 [ 38 ] , SSv2-small [ 40 ] , and UCF101 [ 39 ] under 1 -shot tasks (§ IV-D ).
Fig. 4 : Left : The impact of the varying fusion parameter α on HMDB51 [ 38 ] in the 5 -way 1 -shot setting (see § IV-D ). Right : 5-way 1-shot class improvement of DiST compared to CLIP-FSAR [ 28 ] on all class action classes on HMDB51 [ 38 ] (see § IV-F ). Our DiST achieves improvement on all action classes.
HMDB51 (%)
{ G , L }
1 -shot
5 -shot
{1,3}
81.8
87.9
{3,3}
82.4
88.5
{9,3}
82.5
88.5
{12,3}
82.3
88.3
{6,1}
81.4
88.0
TABLE VI : The ablation study on HMDB51 [ 38 ] to investigate the configuration of the number of spatial attributes G and temporal attributes L (see § IV-D ).
Prototype Number N
HMDB51
SSv2-small
N=1
80.8
56.4
N=4
81.7
56.9
N=9
82.6
57.5
N=12
82.3
57.2
N=16
82.1
57.0
TABLE VII : The ablation study on HMDB51 [ 38 ] and SSv2-small [ 41 ] under the 1-shot setting to investigate the number of object-level prototypes N (see § IV-D ).
Method
Params
FLOPs
Memory
Speed
HMDB51 (%)
Kinetics (%)
CLIP-FSAR [ 28 ]
89.3 M
901.9 G
13.8 G
36.7 ms
75.8
89.7
DiST (ours)
97.2 M
902.1 G
14.1 G
40.9 ms
82.6
92.7
TABLE VIII : Complexity analysis for 5 -way 1 -shot HMDB51 [ 38 ] and Kinetics [ 40 ] evaluation. Here, we report Params, FLOPs, GPU memory, and Speed for each model. See § IV-D for more details.
LLM version
HMDB51 (%)
UCF101 (%)
Baseline
75.8
96.0
Llama-2 [ 82 ]
82.1
98.2
Vicuna [ 83 ]
82.5
98.0
GPT-3.5 [ 75 ] (ours)
82.6
98.3
TABLE IX : Comparison results with different large language models used for spatial-temporal attribute generation on HMDB51 [ 38 ] and UCF101 [ 39 ] in the 5 -way 1 -shot tasks (see § IV-D ). Baseline denotes that DIST only relies on category names for TKC and SKC.
Method
Visual encoder
HMDB51
Kinetics
SSv2-small
CLIP-Freeze [ 84 ]
Frozen
58.2
78.9
29.5
DiST (ours)
Frozen
62.6
83.1
35.9
DiST (ours)
Fine-tuning
84.7
95.9
60.7
TABLE X : Impact of fine-tuning visual encoder on the generalization capacity of DIST on HMDB51 [ 38 ] , Kinetics [ 40 ] , and SSv2-small [ 41 ] in the 5 -way 1 -shot tasks (see § IV-D ).
Spatial Interaction
HMDB51(%)
SSv2-small(%)
UCF101(%)
Baseline
75.8
53.8
96.0
PLE [ 26 ]
76.0
54.1
95.8
STMM [ 64 ]
76.2
54.4
96.1
DiST (Ours)
77.6
55.7
96.6
TABLE XI : Comparison results between Spatial Knowledge Compensator and existing spatial interaction methods on HMDB51 [ 38 ] , SSv2-small [ 40 ] , and UCF101 [ 39 ] under 1 -shot tasks (§ IV-D ).
Fig. 5 : Visualization of spatial and temporal prompts under the 1 -shot setting (§ IV-F ). The spatial prompts are shown as highlighted response areas in each frame. We also show cross-attention temporal prompt weights of Eq. 4 in a line graph.
Fig. 6 : T-SNE visualization [ 85 ] under the 5 -way setting on HMDB51 and Kinetics (see § IV-F ).
SSv2-small (%)
Kinetics (%)
Method
Backbone
1 -shot
5 -shot
1 -shot
5 -shot
CLIP-FSAR [ 28 ]
46.0
52.4
72.9
84.1
DiST (ours)
INet-RN18
48.6
53.3
75.6
86.5
CLIP-FSAR [ 28 ]
46.7
53.3
75.5
85.9
DiST (ours)
INet-RN50
49.1
54.6
78.6
88.0
TABLE XII : Generalization performance of our DiST with different ImageNet-pretrained backbones on SSv2-small [ 40 ] , and Kinetics [ 39 ] in the 5 -way 1 -shot and 5 -shot tasks (see § IV-E ).
Method
Fine-tuning
HMDB51
UCF101
Kinetics
SSv2-small
D 2 ST-Adapter [ 84 ]
PEFT
77.1
96.4
89.3
55.0
MA-FSAR [ 86 ]
PEFT
83.4
97.2
95.7
59.1
Task-Adapter [ 87 ]
PEFT
83.6
98.0
95.0
60.2
Ours
PEFT
84.7
98.3
95.9
60.7
TABLE XIII : Generalization performance of our DiST under the parameter-efficient fine-tuning setting on UCF101 [ 39 ] , HMDB51 [ 38 ] , Kinetics [ 40 ] , SSv2 [ 41 ] and SSv2-small [ 41 ] in the 5 -way 1 -shot tasks (see § IV-E ).
Few-shot action recognition (FSAR) requires models to generalize to novel action categories from only a handful of annotated samples. Despite progress with vision-language models, existing approaches still suffer from semantic-temporal misalignment, where static textual prompts fail to capture decisive visual cues that appear sparsely across sequences, and from inadequate modeling of multi-scale temporal dynamics, as short-term discriminative cues and long-range dependencies are often either oversmoothed or fragmented. To address these challenges, we propose Semantic Temporal Adaptive Representation Learning (STAR), a unified framework, consisting of a semantic-alignment component and a temporal-aware component, effectively bridging the semantic and temporal gaps and transferring the sequence modeling capability of Mamba into the FSAR. The semantic alignment module introduces a Temporal Semantic Attention (TSA) mechanism, which performs frame-level cross-modal alignment with textual cues, ensuring fine-grained semantic-temporal consistency. The temporal-aware module incorporates a Semantic Temporal Prototype Refiner (STPR) that integrates semantic-guided Mamba blocks with multi-frequency temporal sampling and bidirectional state-space refinement, yielding semantically aligned prototypes with enhanced discriminative fidelity and temporal consistency. Furthermore, temporally dependent class descriptors derived from large language models (LLMs) provide long-range semantic guidance. Extensive experiments on five FSAR benchmarks demonstrate the consistent superiority of STAR over state-of-the-art methods. For instance, STAR achieves up to 8.1% and 6.7% gains on the SSv2-Full and SSv2-Small datasets under the 1-shot setting, and 7.3% on HMDB51, validating its effectiveness under limited supervision. The code is available at https://github.com/HongliLiu1/STAR-main.
Hongli Liu, Yu Wang, Shengjie Zhao
School of Computer Science and Technology, Tongji University, Shanghai, China · Engineering Research Center of Key Software Technologies for Smart City Perception and Planning, Ministry of Education
Few-shot action recognition (FSAR) aims to recognize novel action categories with few exemplars. Existing methods typically learn frame-level representations for each video by designing inter-frame temporal modeling strategies or inter-video interaction at the coarse video-level granularity. However, they treat each episode task in isolation and neglect fine-grained temporal relation modeling between videos, thus failing to capture shared fine-grained temporal patterns across videos and reuse temporal knowledge from historical tasks. In light of this, we propose HR2G-shot, a Hierarchical Relation-augmented Representation Generalization framework for FSAR, which unifies three types of relation modeling (inter-frame, inter-video, and inter-task) to learn task-specific temporal patterns from a holistic view. Going beyond conducting inter-frame temporal interactions, we further devise two components to respectively explore inter-video and inter-task relationships: i) Inter-video Semantic Correlation (ISC) performs cross-video frame-level interactions in a fine-grained manner, thereby capturing task-specific query features and enhancing both intra-class consistency and inter-class separability; ii) Inter-task Knowledge Transfer (IKT) retrieves and aggregates relevant temporal knowledge from the bank, which stores diverse temporal patterns from historical episode tasks. Extensive experiments on five benchmarks show that HR2G-shot outperforms current top-leading FSAR methods.
Hongyu Qu, Ling Xing, Jiachao Zhang +3
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · Artificial Intelligence Industrial Technology Research Institute, Nanjing Institute of Technology, Nanjing 211167, China
Few-shot action recognition (FSAR) aims to recognize unseen action categories with only a small number of annotated video samples. Recent works typically apply single-prototype supervision at the network output and fail to sufficiently exploit rich cross-frame global spatial information in videos. Even existing multi-level metric schemes only impose parallel prototype constraints on intermediate layers, without progressive supervision along the full feature pipeline, which results in limited generalization ability of the learned class prototypes. Inspired by this, we present a novel method, hierarchical metric learning for few-shot action recognition (HML-FSAR). First, a spatial-enhanced module is developed to capture cross-frame global spatial representations. Combined with temporal MHA, heterogeneous alignment, spatial-temporal feature fusion and dictionary learning modules, it constructs the complete feature processing pipeline. Second, a hierarchical metric learning (HML) strategy is embedded into HML-FSAR. Composed of center metric, alignment metric, contrastive metric, dictionary metric and prototype metric, HML imposes progressive multi-stage complementary constraints from frame-level representations to final class prototypes, so as to jointly optimize feature compactness, heterogeneous spatial-temporal alignment, inter-class discriminability and anti-noise robustness. The proposed HML-FSAR method is validated on five widely-used FSAR datasets, and experimental results fully demonstrate its effectiveness.
Jiaxin Zhang, Haoran Gao, Xizhan Gao +3
School of Information Science and Engineering, University of Jinan, Shandong, China