Few-shot fine-grained video action recognition requires recognizing novel action classes from only a few labeled examples. The difficulty is that visually similar actions often share actors, scenes, poses, and objects, differing only in the order, timing, and short duration of motion changes. We study this problem under three practical constraints: few tunable parameters, low compute cost, and use of the video modality alone. We propose Motion-aligned Temporal Adapter (MATA), a parameter-efficient adapter on top of a frozen CLIP backbone. MATA specializes CLIP features along the temporal dimension using two components: temporal phase alignment through learnable temporal positional embeddings, and a local motion transition operator through depth-wise convolution. With only ~3.6M tunable parameters, MATA achieves state-of-the-art performance across multiple fine-grained benchmarks while remaining robust to both visual corruptions and temporal disruptions induced by frame dropping. Code is available at https://github.com/prakashchhipa/MATA-Motion-aligned-Temporal-Adapter
Figures & tables
Figure 1: Overview of the MATA few-shot inference pipeline . Support and query videos are encoded by a shared frozen CLIP backbone, and query labels are predicted via prototype matching. Compared to full fine-tuning [ 40 ] , MATA updates only a small set of parameters, improving accuracy while reducing latency.
Method
Axis
Temporal mechanism
Cost
Params (M)
AIM [ 48 ]
S+T
Reused temporal MSA
O(T2C)
11.0
ST-Adapter [ 26 ]
S+T
3D depth-wise conv
O(k3TC)
7.2
DualPath [ 27 ]
S+T
Dual-branch attention
O(T2C)
10.0
D 2 ST [ 28 ]
S+T
Deformable ST attention
O(T2C)
7.3
MATA
T
Depth-wise 1D conv
O(kTCa)
3.6
Table 1: Comparison with representative parameter-efficient video adapters. MATA performs temporal-only adaptation while maintaining linear temporal complexity. Cost is per patch for the temporal operator alone, at the width at which each operator acts. Ca : bottleneck width; k : kernel size; S: spatial; T: temporal.
Method
Venue
Tunable Params (M)
Diving48
FineGym99
FineGym288
1-shot
3-shot
5-shot
1-shot
3-shot
5-shot
1-shot
3-shot
5-shot
TSN+Cosine [ 38 ]
ECCV’16
25M
29.72
35.69
38.81
36.63
41.35
42.76
36.68
42.87
44.82
ProtoNet [ 35 ]
NeurIPS’17
25M
59.80
74.74
77.69
62.99
74.25
77.99
58.98
70.56
74.87
OTAM [ 3 ]
CVPR’20
23.5M
41.64
47.96
49.78
48.93
52.88
55.61
47.58
53.67
55.62
PAL [ 55 ]
BMVC’21
25M
50.40
57.57
61.17
58.85
65.13
68.64
58.32
63.62
65.77
TRX [ 29 ]
CVPR’21
47.1M
62.53
77.81
82.01
66.55
80.15
83.86
64.03
76.73
81.05
Table 2: Few-shot fine-grained action recognition on Diving48, FineGym99, and FineGym288 (%) under 5-way 1/3/5-shot settings. This table compares prior few-shot methods that do not use foundation models with MATA. Best results are in bold. † means results reproduced by us. “*” denotes multi-modal methods incorporating additional text input and a text encoder. ϕ denotes additional pre-processing.
Method
ToyotaSH
HAA500
MPII Cooking 2
1-shot
3-shot
5-shot
1-shot
3-shot
5-shot
1-shot
3-shot
5-shot
Task-Adapter++ [ 1 ] †
62.39
73.43
76.30
97.05
98.14
99.10
59.85
68.79
70.00
D 2 ST [ 28 ] †
60.25
71.06
73.18
96.52
97.46
98.78
57.23
68.09
71.16
MATA
64.84
74.85
77.76
97.18
98.24
99.28
60.03
70.48
72.50
Table 3: Few-shot fine-grained action recognition on additional datasets (%) under 5-way 1/3/5-shot settings.
Method
1-shot Acc. (%) ↑
GFLOPs Episode ↓
Latency Episode (ms) ↓
Throughput (episodes/s) ↑
Effective Compute (TFLOP/s) ↑
D 2 ST † [ 28 ]
73.29
2990.0
111.26
9.07
26.92
Task-Adapter † [ 1 ]
46.45
3335.95
68.54
14.60
48.67
EMP-Net † [ 44 ]
71.0
2964.24
128.16
7.80
23.12
MATA
80.12
2921.3
27.16
36.81
107.53
Gain (MATA vs. D 2 ST)
↑ +6.83
↓ 2.3%
↓ 4.10 ×
↑ 4.06 ×
↑ 3.99 ×
Table 4: MATA delivers both higher accuracy and substantially faster few-shot adaptation. The results reveal a clear advantage in both recognition performance and practical inference efficiency, rather than a trade-off between the two. † denotes results reproduced by us; ↓ lower is better and ↑ higher is better.
Table 6
Component
Diving48
FineGym99
1-shot
5-shot
1-shot
5-shot
Local motion transition
78.69
88.76
80.02
89.26
+ Temporal phase alignment
80.12
90.00
82.21
91.29
Table 8: Component analysis of MATA on Diving48 and FineGym99. Each row cumulatively adds the indicated component.
Method
Clean Top-1 ↑
Corrupted queries (mean Top-1 over severities 1/3/5) ↑
mCA (%) ↑
mCE ↓
Gaussian noise
Gaussian blur
Frame drop
JPEG compr.
Bright- ness
Task-Adapter++
45.8
32.1
36.6
44.1
40.8
41.0
38.9
100.0
D 2 ST
73.8
55.4
62.9
66.9
68.5
70.1
64.8
57.5
MATA
80.1
63.7
70.7
74.2
77.4
77.9
72.8
44.3
Table 9: Test-time robustness on Diving48 (5-way 1-shot); only queries are corrupted. Mean Top-1 (%) over severities {1,3,5} ; mCE is normalized by Task-Adapter++ (=100, lower is better).
Figure 2: Temporal saliency via adapter update magnitude : Per-frame update magnitude s(t) (Eq. 15 ) for MATA and D 2 ST on randomly sampled FineGym clips (mean ± variability). MATA shows sharper temporal peaks, indicating more selective activation on discriminative motion stages. More comparison examples in Sec. E in the supplementary.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Table 10: Ablation on adapter depth and temporal context for MATA and D 2 ST.
Method
Diving48
FineGym99
1-shot
3-shot
5-shot
7-shot
9-shot
10-shot
1-shot
3-shot
5-shot
7-shot
9-shot
10-shot
D 2 ST
73.29
80.64
83.85
84.59
84.54
84.96
76.81
85.52
87.55
89.30
90.03
90.70
MATA
79.37
86.46
88.48
88.76
89.31
89.50
81.37
88.12
90.44
91.05
92.35
92.40
Appendix
Table 11: Evaluation of MATA and D 2 ST across different shot settings. MATA consistently outperforms D 2 ST in all shot regimes.
Figure 3: Nearest-neighbor retrieval on confusable FineGym99 classes. For the same query and gallery, MATA yields cleaner top- K neighborhoods than frozen CLIP and D 2 ST, consistent with improved fine-grained discrimination.
Method
Purity@5 ↑
Purity rate ↑
Δ Purity@5 vs. CLIP ↑
Frozen CLIP
0.85
0.17 (0.08–0.27)
0.00
D 2 ST
2.80
0.56 (0.39–0.73)
+1.95 (1.20–2.70)
MATA
4.00
0.80 (0.64–0.93)
+3.15 (2.40–3.90)
Appendix
Table 12: Nearest-neighbor retrieval quality on FineGym99 . Purity@5 counts how many of the top-5 retrieved neighbors share the query label. Parentheses denote 95% bootstrap confidence intervals computed over the same 20 queries.
Figure 4: Single-frame blackout sensitivity analysis. (a) Example episode showing MATA’s sharp peak at frame 4 ( ∣Δ(4)∣=0.112 ) vs D 2 ST’s flatter response (peak ∣Δ(6)∣=0.070 ) (b) Aggregate plot (mean ∣Δ(t)∣± std over 20 episodes). MATA shows higher peak sensitivity (0.091 vs 0.064) and higher temporal selectivity (0.036 vs 0.025), demonstrating MATA’s focused temporal modeling.
Method
Reference
Pretrain
HMDB51
UCF101
Kinetics
SSv2-Small
SSv2-Full
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
CMN [ 54 ]
ECCV’18
ResNet50
–
–
–
–
60.5
78.9
34.4
43.8
36.2
48.9
OTAM [ 3 ]
CVPR’20
ResNet50
–
–
–
–
73.0
85.8
–
–
42.8
52.3
AmeFuNet [ 9 ]
MM’20
ResNet50
60.2
75.5
85.1
95.5
74.1
86.8
–
–
–
–
TRX [ 29 ]
CVPR’21
ResNet50
–
75.6
–
96.1
63.6
85.9
–
59.1
–
64.6
SPRN [ 43 ]
MM’21
ResNet50
61.6
76.2
86.5
95.8
75.2
87.1
–
–
–
–
Appendix
Table 13: Comparison with current SOTA few-shot action recognition methods on HMDB51, UCF101, Kinetics, SSv2-Small, and SSv2-Full datasets. “*” denotes multi-modal methods incorporating additional text input and a text encoder.
Institute for Artificial Intelligence, University of Stuttgart, Germany · Intelligent Assistive Systems Lab, University of Hildesheim · University Hospital Heidelberg, Diagnostic and Interventional Radiology