Cross-domain few-shot action recognition (CD-FSAR) aims to recognise unseen action classes in a target domain from a handful of labelled videos. Despite advances in representation learning, CD-FSAR inference still relies on prototypes estimated from extremely limited support examples. To assess this bottleneck, we perform an oracle analysis with the representation fixed. Constructing oracle prototypes and decision boundaries from ground-truth query labels improves 5-way 1-shot accuracy by 7--20 percentage points across five target datasets. This reveals a substantial estimation gap: the representation contains useful discriminative information, but prototypes and boundaries estimated from scarce supports fail to capture each episode's true class structure. Motivated by this, we propose TRACE, a training-free transductive inference framework that improves episode-level estimation without modifying the backbone. TRACE comprises two complementary stages. Confidence-Thresholded Transductive Prototype Refinement (CTPR) refines support-based prototypes using confidently predicted queries. The refined prototypes initialise a lightweight per-episode classifier, which Transductive Classifier Refinement (TCR) further adapts by encouraging confident, class-balanced predictions over all queries. Together, the two stages improve prototype and boundary estimation, enabling better use of information in the unlabelled queries. Across five target datasets (HMDB51, UCF101, SSv2, Diving48, and RareAct) in both 1-shot and 5-shot settings, TRACE consistently improves a strong CD-FSAR baseline, including gains of more than six percentage points on HMDB51 1-shot, while recovering much of the oracle-estimated gap. These results suggest that improving test-time estimation is a powerful, broadly applicable complement to representation learning for CD-FSAR under extreme support scarcity.
Figures & tables
Figure 1: Oracle estimation reveals a substantial gap across target datasets (1-shot). Left bars: TAMT baseline; right bars: oracle estimation, both under the held-out-half protocol. Numbers above bars give the gap in percentage points.
Dataset
Baseline
TUB-P
TUB-C
TUB-I
gap Δ
HMDB51
58.45
71.24
71.05
73.79
+15.34
UCF101
85.90
94.63
94.33
94.18
+8.73
SSv2
40.87
54.42
50.64
54.47
+13.60
Diving48
32.21
39.47
35.98
39.07
+7.26
RareAct
46.40
63.68
64.13
65.91
+19.51
Table 1: Oracle upper bounds on 5-way 1-shot CD-FSAR. Baseline denotes TAMT prototype matching evaluated on the held-out query half. TUB-P, TUB-C, and TUB-I denote oracle prototype estimation, oracle boundary estimation, and classifier refinement initialised from oracle prototypes, respectively. Each oracle uses the true labels of the other query half and is evaluated on the same held-out half as the baseline. Gap Δ is the difference between the best oracle and the baseline. Best oracle per row is shown in bold .
Figure 2: Trace : recovering the estimation gap at test time. Top: a target-domain episode is encoded by a frozen VideoMAE-S encoder equipped with Temporal-Aware Adapters (TAA) and a Global Temporal-aware Moment Tuning (GTMT) head. Bottom: Trace first applies Confidence-Thresholded Transductive Prototype Refinement (CTPR) to refine prototype estimates using only confident query samples, then performs Transductive Classifier Refinement (TCR) to adapt the decision boundary to the query distribution. The refined prototypes initialise the classifier before boundary refinement. Insets illustrate the effect of CTPR and TCR in a schematic feature space (stars: support; dots: query).
Method
HMDB51
UCF101
SSv2
Diving48
RareAct
Avg.
Source: Kinetics-400 (5-way 1-shot)
TAMT [ 24 ] ‡
58.19
86.04
41.14
32.36
46.33
52.81
Trace (ours)
64.81
95.35
43.56
34.15
51.09
57.79
Source: Kinetics-400 (5-way 5-shot)
STARTUP++ [ 12 ]
44.71
60.82
39.60
14.92
45.22
41.05
DD++ [ 5 ]
48.04
63.26
44.50
16.23
47.01
43.81
Table 2: Main results across five CD-FSAR target datasets (5-way accuracy, Methods marked with ∗ or † use different input resolutions or backbones and are included for reference. All other methods follow the common VideoMAE ViT-S/ 112 protocol. ‡ denotes our reproduction of TAMT, the base model on which Trace is applied. Prior work does not report K-400 1-shot results; therefore only TAMT and Trace are shown in this setting. Best results under the common ViT-S/ 112 protocol are highlighted in bold .
Dataset
Shot
Baseline
+CTPR
+TCR
+ Trace
HMDB51
1
58.19
61.82
63.50
64.81
5
72.50
73.95
73.25
74.23
UCF101
1
86.04
92.73
93.44
95.35
5
95.96
96.48
97.47
97.43
SSv2
1
41.14
42.74
43.12
43.56
5
59.28
60.76
60.50
61.45
Table 3: Ablation of prototype and boundary refinement (5-way accuracy, Baseline denotes the original TAMT inference procedure. + CTPR applies prototype refinement only, + TCR applies classifier refinement only, and +\textscTrace combines the two through refined-prototype initialisation. Best results in each row are shown in bold .
Base
Method
HMDB51
UCF101
SSv2
Diving48
RareAct
Avg.
ProtoNet
Baseline
58.26
87.63
39.93
31.89
47.86
53.11
+CTPR
62.30
93.43
41.44
32.37
51.06
56.12
+TCR
63.10
94.27
41.98
31.57
51.12
56.41
TRACE (ours)
64.87
95.95
42.54
32.36
52.29
57.60
Good-embed
Baseline
51.94
84.75
31.08
25.70
47.26
48.15
+CTPR
55.42
91.46
31.42
25.49
50.27
50.81
Table 4: Generalisation across episodic heads (5-way 1-shot accuracy, Trace is applied unchanged to a ProtoNet head and a good-embedding linear head. Best results within each head are highlighted in bold .
Figure 3: Recovering the estimation gap through refined-prototype initialisation. Left: comparison of TAMT, CTPR, TCR, and Trace against the corresponding oracle ceiling for each dataset (dashed lines; Table 1 ). Right: effect of the initialisation blend λ . λ=0 corresponds to TCR initialised from the raw support prototype, while λ=1 corresponds to using only the refined prototype from CTPR. Curves show the accuracy difference from each dataset’s peak performance, allowing all datasets to be compared on a common scale.
Figure 4: Robustness to hyperparameters and consistency across episodes. Top: sensitivity to τ , T , α , and β around the default configuration. Curves show the accuracy difference from each dataset’s peak performance, allowing all datasets to be compared on a common scale. Bottom: per-episode distribution of Trace accuracy minus baseline accuracy over 1,000 episodes per dataset. Each panel reports the mean improvement and the proportion of episodes that outperform the baseline.
Figure 5: Initialisation succeeds where routing fails. Left: per dataset, the left bar shows the better of CTPR and TCR, the centre bar shows Trace , and the right bar shows the accuracy of a perfect per-episode router (oracle routing), representing the maximum benefit obtainable from selecting between the two estimators. Right: per-episode error correlation ρ between CTPR and TCR. Higher correlation indicates that the two estimators tend to fail on the same episodes.
School of Computer Science and Technology, Tongji University, Shanghai, China · Engineering Research Center of Key Software Technologies for Smart City Perception and Planning, Ministry of Education
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China. · State Key Lab of Brain-Machine Intelligence, Zhejiang University, China.