Few-Shot Video Recognition via Hierarchical Metric Learning

Jiaxin Zhang; Haoran Gao; Xizhan Gao; Zihao Dong; Tingwei Wang; Sijie Niu

cs.CVSep 4, 2026

Few-Shot Video Recognition via Hierarchical Metric Learning

Authors: Jiaxin Zhang, Haoran Gao, Xizhan Gao, Zihao Dong, Tingwei Wang, Sijie Niu

Organizations: School of Information Science and Engineering, University of Jinan, Shandong, China

Abstract

Few-shot action recognition (FSAR) aims to recognize unseen action categories with only a small number of annotated video samples. Recent works typically apply single-prototype supervision at the network output and fail to sufficiently exploit rich cross-frame global spatial information in videos. Even existing multi-level metric schemes only impose parallel prototype constraints on intermediate layers, without progressive supervision along the full feature pipeline, which results in limited generalization ability of the learned class prototypes. Inspired by this, we present a novel method, hierarchical metric learning for few-shot action recognition (HML-FSAR). First, a spatial-enhanced module is developed to capture cross-frame global spatial representations. Combined with temporal MHA, heterogeneous alignment, spatial-temporal feature fusion and dictionary learning modules, it constructs the complete feature processing pipeline. Second, a hierarchical metric learning (HML) strategy is embedded into HML-FSAR. Composed of center metric, alignment metric, contrastive metric, dictionary metric and prototype metric, HML imposes progressive multi-stage complementary constraints from frame-level representations to final class prototypes, so as to jointly optimize feature compactness, heterogeneous spatial-temporal alignment, inter-class discriminability and anti-noise robustness. The proposed HML-FSAR method is validated on five widely-used FSAR datasets, and experimental results fully demonstrate its effectiveness.

Figures & tables

Fig. 1 : Comparison between existing FSAR methods and our HML-FSAR. (a) Existing methods employ only a single prototype metric at the top layer of the network and fail to fully exploit the rich spatial information contained in videos, resulting in poor separability between class prototypes and further leading to misclassification. (b) In contrast, our method introduces a spatial‑enhanced branch to boost the discriminative ability of spatial‑temporal features, and leverages hierarchical metric learning to optimize the prototype feature learning process in multiple stages for obtaining class prototypes with stronger separability, thus achieving more accurate FSAR.

Fig. 2 : Overview of the proposed HML-FSAR method. Top : Given input support/query videos, the feature extractor first outputs frame‑level features, and SE and TMHA modules are then adopted in parallel to learn video-level global spatial features and preliminary spatial‑temporal features, respectively. Highly discriminative video‑level features are obtained via HA, STFF and DL modules, followed by prototype learning to generate class prototypes for FSAR. Bottom : Hierarchical metric learning, composed of center metric, alignment metric, contrastive metric, dictionary metric and prototype metric, imposes multi‑level metric constraints to progressively refine feature representations throughout the above pipeline.

Method	Reference	HMDB51		UCF101		Kinetics		SSv2-Small		SSv2-Full
Method	Reference	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot
ProtoNet [ 21 ]	NeurIPS’17	54.2	68.4	74.0	89.6	64.5	77.9	–	–	–	–
CMN [ 29 ]	ECCV’18	–	–	57.3	76.0	34.4	43.8	36.2	48.9	–	–
OTAM [ 5 ]	CVPR’20	54.5	68.0	79.9	88.9	73.0	85.8	–	–	42.8	52.3
AMeFu-Net [ 30 ]	MM’20	60.2	75.5	85.1	95.5	74.1	86.8	–	–	–	–
TARN [ 7 ]	ECCV’20	45.5	60.6	66.3	83.1	63.7	82.4	–	–	–	–

TABLE I : Performance comparison against state‑of‑the‑art methods on five FSAR benchmarks.

Setting	Prototype	Center	Alignment	Contrastive	Dictionary	Kinetics
	Metric	Metric	Metric	Metric	Metric
(a)	✓					83.7
(b)	✓	✓				83.9
(c)	✓	✓	✓			84.1
(d)	✓	✓	✓	✓		85.3
(e)	✓	✓	✓	✓	✓	86.0

TABLE II : Ablation study on different metric learning strategies.

Setting	S	T	F	D	Kinetics	SSv2-small
(a)					80.0	43.1
(b)	✓				81.8	45.3
(c)		✓			83.6	48.9
(d)	✓	✓	✓		83.9	50.3
(e)				✓	80.7	44.2
(f)	✓	✓	✓	✓	86.0	52.0

TABLE III : Ablation study on different components. S, T, F, and D denote the SE module, TMHA module, HA+STFF module, and DL module, respectively.

Selected Frames	Kinetics		SSv2-small
Selected Frames	1-shot	5-shot	1-shot	5-shot
4	82.8	91.3	47.3	62.7
8	85.1	93.7	51.2	64.3
16	86.0	96.5	52.0	66.7

TABLE IV : Effect of the number of selected frames.

Panel of Figure 7 — Fig. 3 : Performance under different few-shot settings on the Kinetics dataset: (a) 5‑way with varying‑shot settings, (b) 1-shot with varying-way settings.

Panel of Figure 8 — Fig. 4 : Parameter sensitivity analysis on the Kinetics dataset: (a) Varying parameter $\lambda$ ; (b) Varying combinations of $(\lambda_{1},\lambda_{2})$ .

Panel of Figure 9 — Fig. 5 : Parameter sensitivity analysis on the Kinetics dataset: (a) Varying combinations of $(\beta_{1},\beta_{2})$ ; (b) Varying combinations of $(\beta_{3},\beta_{4})$ .

Panel of Figure 10 — Fig. 6 : t-SNE visualization of feature representations learned by different modules on UCF101.

Explore similar work

Hierarchical Relation-augmented Representation Generalization for Few-shot Action Recognition
Apr 14, 2025Hongyu Qu, Ling Xing, Jiachao Zhang +3Temporal Reasoning Hierarchical Representation Learning
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · Artificial Intelligence Industrial Technology Research Institute, Nanjing Institute of Technology, Nanjing 211167, China
STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition
May 13, 2026Hongli Liu, Yu Wang, Shengjie ZhaoCross-Modal Representation Learning Few-Shot Learning
School of Computer Science and Technology, Tongji University, Shanghai, China · Engineering Research Center of Key Software Technologies for Smart City Perception and Planning, Ministry of Education
Spatio-temporal Decoupled Knowledge Compensator for Few-Shot Action Recognition
Feb 20, 2026Hongyu Qu, Xiangbo Shu, Rui Yan +3Video Action Recognition Few-Shot Learning
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China. · State Key Lab of Brain-Machine Intelligence, Zhejiang University, China.

Apr 14, 2025J/K move · Enter open · S save

Hierarchical Relation-augmented Representation Generalization for Few-shot Action Recognition

Hongyu Qu, Ling Xing, Jiachao Zhang, Rui Yan+2

School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · Artificial Intelligence Industrial Technology Research Institute, Nanjing Institute of Technology, Nanjing 211167, China

Temporal Reasoning Hierarchical Representation Learning Few-Shot Learning Video Action Recognition

Open paper AskPDF arXiv

Few-shot action recognition (FSAR) aims to recognize novel action categories with few exemplars. Existing methods typically learn frame-level representations for each video by designing inter-frame temporal modeling strategies or inter-video interaction at the coarse video-level granularity. However, they treat each episode task in isolation and neglect fine-grained temporal relation modeling between videos, thus failing to capture shared fine-grained temporal patterns across videos and reuse temporal knowledge from historical tasks. In light of this, we propose HR2G-shot, a Hierarchical Relation-augmented Representation Generalization framework for FSAR, which unifies three types of relation modeling (inter-frame, inter-video, and inter-task) to learn task-specific temporal patterns from a holistic view. Going beyond conducting inter-frame temporal interactions, we further devise two components to respectively explore inter-video and inter-task relationships: i) Inter-video Semantic Correlation (ISC) performs cross-video frame-level interactions in a fine-grained manner, thereby capturing task-specific query features and enhancing both intra-class consistency and inter-class separability; ii) Inter-task Knowledge Transfer (IKT) retrieves and aggregates relevant temporal knowledge from the bank, which stores diverse temporal patterns from historical episode tasks. Extensive experiments on five benchmarks show that HR2G-shot outperforms current top-leading FSAR methods.

Abstract

Explore similar work

Hierarchical Relation-augmented Representation Generalization for Few-shot Action Recognition

STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition

Spatio-temporal Decoupled Knowledge Compensator for Few-Shot Action Recognition