Few-shot fine-grained video action recognition requires recognizing novel action classes from only a few labeled examples. The difficulty is that visually similar actions often share actors, scenes, poses, and objects, differing only in the order, timing, and short duration of motion changes. We study this problem under three practical constraints: few tunable parameters, low compute cost, and use of the video modality alone. We propose Motion-aligned Temporal Adapter (MATA), a parameter-efficient adapter on top of a frozen CLIP backbone. MATA specializes CLIP features along the temporal dimension using two components: temporal phase alignment through learnable temporal positional embeddings, and a local motion transition operator through depth-wise convolution. With only ~3.6M tunable parameters, MATA achieves state-of-the-art performance across multiple fine-grained benchmarks while remaining robust to both visual corruptions and temporal disruptions induced by frame dropping. Code is available at https://github.com/prakashchhipa/MATA-Motion-aligned-Temporal-Adapter
Figures & tables
Figure 1: Overview of the MATA few-shot inference pipeline . Support and query videos are encoded by a shared frozen CLIP backbone, and query labels are predicted via prototype matching. Compared to full fine-tuning [ 40 ] , MATA updates only a small set of parameters, improving accuracy while reducing latency.
Method
Axis
Temporal mechanism
Cost
Params (M)
AIM [ 48 ]
S+T
Reused temporal MSA
O(T2C)
11.0
ST-Adapter [ 26 ]
S+T
3D depth-wise conv
O(k3TC)
7.2
DualPath [ 27 ]
S+T
Dual-branch attention
O(T2C)
10.0
D 2 ST [ 28 ]
S+T
Deformable ST attention
O(T2C)
7.3
MATA
T
Depth-wise 1D conv
O(kTCa)
3.6
Table 1: Comparison with representative parameter-efficient video adapters. MATA performs temporal-only adaptation while maintaining linear temporal complexity. Cost is per patch for the temporal operator alone, at the width at which each operator acts. Ca : bottleneck width; k : kernel size; S: spatial; T: temporal.
Method
Venue
Tunable Params (M)
Diving48
FineGym99
FineGym288
1-shot
3-shot
5-shot
1-shot
3-shot
5-shot
1-shot
3-shot
5-shot
TSN+Cosine [ 38 ]
ECCV’16
25M
29.72
35.69
38.81
36.63
41.35
42.76
36.68
42.87
44.82
ProtoNet [ 35 ]
NeurIPS’17
25M
59.80
74.74
77.69
62.99
74.25
77.99
58.98
70.56
74.87
OTAM [ 3 ]
CVPR’20
23.5M
41.64
47.96
49.78
48.93
52.88
55.61
47.58
53.67
55.62
PAL [ 55 ]
BMVC’21
25M
50.40
57.57
61.17
58.85
65.13
68.64
58.32
63.62
65.77
TRX [ 29 ]
CVPR’21
47.1M
62.53
77.81
82.01
66.55
80.15
83.86
64.03
76.73
81.05
Table 2: Few-shot fine-grained action recognition on Diving48, FineGym99, and FineGym288 (%) under 5-way 1/3/5-shot settings. This table compares prior few-shot methods that do not use foundation models with MATA. Best results are in bold. † means results reproduced by us. “*” denotes multi-modal methods incorporating additional text input and a text encoder. ϕ denotes additional pre-processing.
Method
ToyotaSH
HAA500
MPII Cooking 2
1-shot
3-shot
5-shot
1-shot
3-shot
5-shot
1-shot
3-shot
5-shot
Task-Adapter++ [ 1 ] †
62.39
73.43
76.30
97.05
98.14
99.10
59.85
68.79
70.00
D 2 ST [ 28 ] †
60.25
71.06
73.18
96.52
97.46
98.78
57.23
68.09
71.16
MATA
64.84
74.85
77.76
97.18
98.24
99.28
60.03
70.48
72.50
Table 3: Few-shot fine-grained action recognition on additional datasets (%) under 5-way 1/3/5-shot settings.
Method
1-shot Acc. (%) ↑
GFLOPs Episode ↓
Latency Episode (ms) ↓
Throughput (episodes/s) ↑
Effective Compute (TFLOP/s) ↑
D 2 ST † [ 28 ]
73.29
2990.0
111.26
9.07
26.92
Task-Adapter † [ 1 ]
46.45
3335.95
68.54
14.60
48.67
EMP-Net † [ 44 ]
71.0
2964.24
128.16
7.80
23.12
MATA
80.12
2921.3
27.16
36.81
107.53
Gain (MATA vs. D 2 ST)
↑ +6.83
↓ 2.3%
↓ 4.10 ×
↑ 4.06 ×
↑ 3.99 ×
Table 4: MATA delivers both higher accuracy and substantially faster few-shot adaptation. The results reveal a clear advantage in both recognition performance and practical inference efficiency, rather than a trade-off between the two. † denotes results reproduced by us; ↓ lower is better and ↑ higher is better.
Table 6
Component
Diving48
FineGym99
1-shot
5-shot
1-shot
5-shot
Local motion transition
78.69
88.76
80.02
89.26
+ Temporal phase alignment
80.12
90.00
82.21
91.29
Table 8: Component analysis of MATA on Diving48 and FineGym99. Each row cumulatively adds the indicated component.
Method
Clean Top-1 ↑
Corrupted queries (mean Top-1 over severities 1/3/5) ↑
mCA (%) ↑
mCE ↓
Gaussian noise
Gaussian blur
Frame drop
JPEG compr.
Bright- ness
Task-Adapter++
45.8
32.1
36.6
44.1
40.8
41.0
38.9
100.0
D 2 ST
73.8
55.4
62.9
66.9
68.5
70.1
64.8
57.5
MATA
80.1
63.7
70.7
74.2
77.4
77.9
72.8
44.3
Table 9: Test-time robustness on Diving48 (5-way 1-shot); only queries are corrupted. Mean Top-1 (%) over severities {1,3,5} ; mCE is normalized by Task-Adapter++ (=100, lower is better).
Figure 2: Temporal saliency via adapter update magnitude : Per-frame update magnitude s(t) (Eq. 15 ) for MATA and D 2 ST on randomly sampled FineGym clips (mean ± variability). MATA shows sharper temporal peaks, indicating more selective activation on discriminative motion stages. More comparison examples in Sec. E in the supplementary.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Table 10: Ablation on adapter depth and temporal context for MATA and D 2 ST.
Method
Diving48
FineGym99
1-shot
3-shot
5-shot
7-shot
9-shot
10-shot
1-shot
3-shot
5-shot
7-shot
9-shot
10-shot
D 2 ST
73.29
80.64
83.85
84.59
84.54
84.96
76.81
85.52
87.55
89.30
90.03
90.70
MATA
79.37
86.46
88.48
88.76
89.31
89.50
81.37
88.12
90.44
91.05
92.35
92.40
Appendix
Table 11: Evaluation of MATA and D 2 ST across different shot settings. MATA consistently outperforms D 2 ST in all shot regimes.
Figure 3: Nearest-neighbor retrieval on confusable FineGym99 classes. For the same query and gallery, MATA yields cleaner top- K neighborhoods than frozen CLIP and D 2 ST, consistent with improved fine-grained discrimination.
Method
Purity@5 ↑
Purity rate ↑
Δ Purity@5 vs. CLIP ↑
Frozen CLIP
0.85
0.17 (0.08–0.27)
0.00
D 2 ST
2.80
0.56 (0.39–0.73)
+1.95 (1.20–2.70)
MATA
4.00
0.80 (0.64–0.93)
+3.15 (2.40–3.90)
Appendix
Table 12: Nearest-neighbor retrieval quality on FineGym99 . Purity@5 counts how many of the top-5 retrieved neighbors share the query label. Parentheses denote 95% bootstrap confidence intervals computed over the same 20 queries.
Figure 4: Single-frame blackout sensitivity analysis. (a) Example episode showing MATA’s sharp peak at frame 4 ( ∣Δ(4)∣=0.112 ) vs D 2 ST’s flatter response (peak ∣Δ(6)∣=0.070 ) (b) Aggregate plot (mean ∣Δ(t)∣± std over 20 episodes). MATA shows higher peak sensitivity (0.091 vs 0.064) and higher temporal selectivity (0.036 vs 0.025), demonstrating MATA’s focused temporal modeling.
Method
Reference
Pretrain
HMDB51
UCF101
Kinetics
SSv2-Small
SSv2-Full
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
CMN [ 54 ]
ECCV’18
ResNet50
–
–
–
–
60.5
78.9
34.4
43.8
36.2
48.9
OTAM [ 3 ]
CVPR’20
ResNet50
–
–
–
–
73.0
85.8
–
–
42.8
52.3
AmeFuNet [ 9 ]
MM’20
ResNet50
60.2
75.5
85.1
95.5
74.1
86.8
–
–
–
–
TRX [ 29 ]
CVPR’21
ResNet50
–
75.6
–
96.1
63.6
85.9
–
59.1
–
64.6
SPRN [ 43 ]
MM’21
ResNet50
61.6
76.2
86.5
95.8
75.2
87.1
–
–
–
–
Appendix
Table 13: Comparison with current SOTA few-shot action recognition methods on HMDB51, UCF101, Kinetics, SSv2-Small, and SSv2-Full datasets. “*” denotes multi-modal methods incorporating additional text input and a text encoder.
Adapting image-pretrained backbones to video typically relies on time-domain adapters tuned to a single temporal scale. Our experiments show that these modules pick up static image cues and very fast flicker changes, while overlooking medium-speed motion. Capturing dynamics across multiple time-scales is, however, crucial for fine-grained temporal analysis (i.e., opening vs. closing bottle). To address this, we introduce Frame2Freq -- a family of frequency-aware adapters that perform spectral encoding during image-to-video adaptation of pretrained Vision Foundation Models (VFMs), improving fine-grained action recognition. Frame2Freq uses Fast Fourier Transform (FFT) along time and learns frequency-band specific embeddings that adaptively highlight the most discriminative frequency ranges. Across five fine-grained activity recognition datasets, Frame2Freq outperforms prior PEFT methods and even surpasses fully fine-tuned models on four of them. These results provide encouraging evidence that frequency analysis methods are a powerful tool for modeling temporal dynamics in image-to-video transfer. Code is available at https://github.com/th-nesh/Frame2Freq.
Thinesh Thiyakesan Ponbagavathi, Constantin Seibold, Alina Roitberg
Institute for Artificial Intelligence, University of Stuttgart, Germany · Intelligent Assistive Systems Lab, University of Hildesheim · University Hospital Heidelberg, Diagnostic and Interventional Radiology
CLIP has demonstrated strong generalization in visual domains through natural language supervision, even for video action recognition. However, most existing approaches that adapt CLIP for action recognition have primarily focused on temporal modeling, often overlooking spatial perception. In real-world scenarios, visual challenges such as low-light environments or egocentric viewpoints can severely impair spatial understanding, an essential precursor for effective temporal reasoning. To address this limitation, we propose Efficient Visual Prompting for CLIP (EV-CLIP), an efficient adaptation framework designed for few-shot video action recognition across diverse scenes and viewpoints. EV-CLIP introduces two visual prompts: mask prompts, which guide the model's attention to action-relevant regions by reweighting pixels, and context prompts, which perform lightweight temporal modeling by compressing frame-wise features into a compact representation. For a comprehensive evaluation, we curate five benchmark datasets and analyze domain shifts to quantify the influence of diverse visual and semantic factors on action recognition. Experimental results demonstrate that EV-CLIP outperforms existing parameter-efficient methods in overall performance. Moreover, its efficiency remains independent of the backbone scale, making it well-suited for deployment in real-world, resource-constrained scenarios. The code is available at https://github.com/AI-CV-Lab/EV-CLIP.
Hyo Jin Jon, Longbin Jin, Eun Yi Kim
Computer Vision Lab., Konkuk University, Seoul, South Korea
Zero-shot action recognition is challenging due to the semantic gap between seen and unseen classes. We present a novel framework that enhances CLIP with disentangled embeddings and semantic-guided interaction. A Motion Separation Module (MSM) separates motion-sensitive and global-static features, while a Motion Aggregation Block (MAB) employs gated cross-attention to refine motion representation without re-coupling redundant information. To facilitate generalization to unseen categories, we enforce semantic alignment between video features and textual representations by aligning projected embeddings with positive textual prompts, while leveraging negative prompts to explicitly model "non-class" semantics. Experiments on standard benchmarks demonstrate that our method consistently outperforms prior CLIP-based approaches, achieving robust zero-shot action recognition across both coarse and fine-grained datasets.