Closing the Estimation Gap: Training-Free Transductive Inference for Cross-Domain Few-Shot Action Recognition

Changyi Li; Zeyu Yang; Yu Xiao

cs.CVOct 13, 2026

Closing the Estimation Gap: Training-Free Transductive Inference for Cross-Domain Few-Shot Action Recognition

Authors: Changyi Li, Zeyu Yang, Yu Xiao

Organizations: Dept. of Information and Communications Engineering, Aalto University, 02150 Espoo, Finland

Abstract

Cross-domain few-shot action recognition (CD-FSAR) aims to recognise unseen action classes in a target domain from a handful of labelled videos. Despite advances in representation learning, CD-FSAR inference still relies on prototypes estimated from extremely limited support examples. To assess this bottleneck, we perform an oracle analysis with the representation fixed. Constructing oracle prototypes and decision boundaries from ground-truth query labels improves 5-way 1-shot accuracy by 7--20 percentage points across five target datasets. This reveals a substantial estimation gap: the representation contains useful discriminative information, but prototypes and boundaries estimated from scarce supports fail to capture each episode's true class structure. Motivated by this, we propose TRACE, a training-free transductive inference framework that improves episode-level estimation without modifying the backbone. TRACE comprises two complementary stages. Confidence-Thresholded Transductive Prototype Refinement (CTPR) refines support-based prototypes using confidently predicted queries. The refined prototypes initialise a lightweight per-episode classifier, which Transductive Classifier Refinement (TCR) further adapts by encouraging confident, class-balanced predictions over all queries. Together, the two stages improve prototype and boundary estimation, enabling better use of information in the unlabelled queries. Across five target datasets (HMDB51, UCF101, SSv2, Diving48, and RareAct) in both 1-shot and 5-shot settings, TRACE consistently improves a strong CD-FSAR baseline, including gains of more than six percentage points on HMDB51 1-shot, while recovering much of the oracle-estimated gap. These results suggest that improving test-time estimation is a powerful, broadly applicable complement to representation learning for CD-FSAR under extreme support scarcity.

Figures & tables

Figure 1: Oracle estimation reveals a substantial gap across target datasets (1-shot). Left bars: TAMT baseline; right bars: oracle estimation, both under the held-out-half protocol. Numbers above bars give the gap in percentage points.

Dataset	Baseline	TUB-P	TUB-C	TUB-I	gap $\Delta$
HMDB51	58.45	71.24	71.05	73.79	$+15.34$
UCF101	85.90	94.63	94.33	94.18	$+8.73$
SSv2	40.87	54.42	50.64	54.47	$+13.60$
Diving48	32.21	39.47	35.98	39.07	$+7.26$
RareAct	46.40	63.68	64.13	65.91	$+19.51$

Table 1: Oracle upper bounds on 5-way 1-shot CD-FSAR. Baseline denotes TAMT prototype matching evaluated on the held-out query half. TUB-P, TUB-C, and TUB-I denote oracle prototype estimation, oracle boundary estimation, and classifier refinement initialised from oracle prototypes, respectively. Each oracle uses the true labels of the other query half and is evaluated on the same held-out half as the baseline. Gap

\Delta

is the difference between the best oracle and the baseline. Best oracle per row is shown in bold .

Figure 2: Trace : recovering the estimation gap at test time. Top: a target-domain episode is encoded by a frozen VideoMAE-S encoder equipped with Temporal-Aware Adapters (TAA) and a Global Temporal-aware Moment Tuning (GTMT) head. Bottom: Trace first applies Confidence-Thresholded Transductive Prototype Refinement (CTPR) to refine prototype estimates using only confident query samples, then performs Transductive Classifier Refinement (TCR) to adapt the decision boundary to the query distribution. The refined prototypes initialise the classifier before boundary refinement. Insets illustrate the effect of CTPR and TCR in a schematic feature space (stars: support; dots: query).

Source: Kinetics-400 (5-way 1-shot)
Method	HMDB51	UCF101	SSv2	Diving48	RareAct	Avg.
TAMT [ 24 ] ‡	58.19	86.04	41.14	32.36	46.33	52.81
Trace (ours)	64.81	95.35	43.56	34.15	51.09	57.79
Source: Kinetics-400 (5-way 5-shot)
STARTUP++ [ 12 ]	44.71	60.82	39.60	14.92	45.22	41.05
DD++ [ 5 ]	48.04	63.26	44.50	16.23	47.01	43.81

Table 2: Main results across five CD-FSAR target datasets (5-way accuracy, Methods marked with ∗ or † use different input resolutions or backbones and are included for reference. All other methods follow the common VideoMAE ViT-S/

112

protocol. ‡ denotes our reproduction of TAMT, the base model on which Trace is applied. Prior work does not report K-400 1-shot results; therefore only TAMT and Trace are shown in this setting. Best results under the common ViT-S/

112

protocol are highlighted in bold .

Dataset	Shot	Baseline	+CTPR	+TCR	+ Trace
HMDB51	1	58.19	61.82	63.50	64.81
	5	72.50	73.95	73.25	74.23
UCF101	1	86.04	92.73	93.44	95.35
	5	95.96	96.48	97.47	97.43
SSv2	1	41.14	42.74	43.12	43.56
	5	59.28	60.76	60.50	61.45

Table 3: Ablation of prototype and boundary refinement (5-way accuracy, Baseline denotes the original TAMT inference procedure.

+

CTPR applies prototype refinement only,

+

TCR applies classifier refinement only, and

+\textsc{Trace}{}

combines the two through refined-prototype initialisation. Best results in each row are shown in bold .

Base	Method	HMDB51	UCF101	SSv2	Diving48	RareAct	Avg.
ProtoNet	Baseline	58.26	87.63	39.93	31.89	47.86	53.11
	+CTPR	62.30	93.43	41.44	32.37	51.06	56.12
	+TCR	63.10	94.27	41.98	31.57	51.12	56.41
	TRACE (ours)	64.87	95.95	42.54	32.36	52.29	57.60
Good-embed	Baseline	51.94	84.75	31.08	25.70	47.26	48.15
	+CTPR	55.42	91.46	31.42	25.49	50.27	50.81

Table 4: Generalisation across episodic heads (5-way 1-shot accuracy, Trace is applied unchanged to a ProtoNet head and a good-embedding linear head. Best results within each head are highlighted in bold .

Figure 3: Recovering the estimation gap through refined-prototype initialisation. Left: comparison of TAMT, CTPR, TCR, and Trace against the corresponding oracle ceiling for each dataset (dashed lines; Table 1 ). Right: effect of the initialisation blend

\lambda

\lambda=0

corresponds to TCR initialised from the raw support prototype, while

\lambda=1

corresponds to using only the refined prototype from CTPR. Curves show the accuracy difference from each dataset’s peak performance, allowing all datasets to be compared on a common scale.

Figure 4: Robustness to hyperparameters and consistency across episodes. Top: sensitivity to

\tau

T

\alpha

, and

\beta

around the default configuration. Curves show the accuracy difference from each dataset’s peak performance, allowing all datasets to be compared on a common scale. Bottom: per-episode distribution of Trace accuracy minus baseline accuracy over

1{,}000

episodes per dataset. Each panel reports the mean improvement and the proportion of episodes that outperform the baseline.

Figure 5: Initialisation succeeds where routing fails. Left: per dataset, the left bar shows the better of CTPR and TCR, the centre bar shows Trace , and the right bar shows the accuracy of a perfect per-episode router (oracle routing), representing the maximum benefit obtainable from selecting between the two estimators. Right: per-episode error correlation

\rho

between CTPR and TCR. Higher correlation indicates that the two estimators tend to fail on the same episodes.

Explore similar work

Sep 20, 2026

Training-Free Spectral Transductive Refinement for Cross-Domain Few-Shot Classification

Few-shot recognition with frozen visual features is especially fragile under domain shift and one-shot supervision, where a single labelled image is an unreliable estimate of its class. We ask how far this fragility can be reduced purely at test time, without retraining the encoder or augmenting the source domain. We present Spectral Transductive Refinement (STR), a training-free transductive inference rule that exploits the geometry of the complete support-query episode. Given frozen embeddings, STR builds a joint k-nearest-neighbour graph, maps the episode into a normalized-Laplacian spectral coordinate system, initializes class representatives from the labelled support, and iteratively refines them using pseudo-labelled queries. We evaluate STR under two protocols. A controlled component study with frozen ResNet-18 features shows that spectral refinement consistently improves over single-prototype spectral initialization across five shifted domains, with the largest gains in the one-shot regime where support estimates are weakest. We then benchmark STR against recent Cross-Domain Few-Shot Learning (CD-FSL) methods using the standard miniImageNet-pretrained ResNet-10 backbone over eight established target domains. Operating entirely at inference time, STR attains the highest 1-shot average among compared methods and remains competitive at 5-shot, rivalling approaches relying on heavy source-domain meta-training augmentations. Because STR is transductive, we report its setting explicitly. Diagnostics attribute its gains to iterative refinement in spectral coordinates rather than added prototype capacity, which remains inactive in our configuration.

Fahim Rahman, S. M. Tanjeeb Meheran Rohan, Md. Taimum Ibne Sayed +2

Department of Computer Science and Engineering Islamic University of Technology Gazipur, Bangladesh

Few-Shot Learning Spectral Methods Few-Shot Image Classification

Ask PDF

May 13, 2026

STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition

Few-shot action recognition (FSAR) requires models to generalize to novel action categories from only a handful of annotated samples. Despite progress with vision-language models, existing approaches still suffer from semantic-temporal misalignment, where static textual prompts fail to capture decisive visual cues that appear sparsely across sequences, and from inadequate modeling of multi-scale temporal dynamics, as short-term discriminative cues and long-range dependencies are often either oversmoothed or fragmented. To address these challenges, we propose Semantic Temporal Adaptive Representation Learning (STAR), a unified framework, consisting of a semantic-alignment component and a temporal-aware component, effectively bridging the semantic and temporal gaps and transferring the sequence modeling capability of Mamba into the FSAR. The semantic alignment module introduces a Temporal Semantic Attention (TSA) mechanism, which performs frame-level cross-modal alignment with textual cues, ensuring fine-grained semantic-temporal consistency. The temporal-aware module incorporates a Semantic Temporal Prototype Refiner (STPR) that integrates semantic-guided Mamba blocks with multi-frequency temporal sampling and bidirectional state-space refinement, yielding semantically aligned prototypes with enhanced discriminative fidelity and temporal consistency. Furthermore, temporally dependent class descriptors derived from large language models (LLMs) provide long-range semantic guidance. Extensive experiments on five FSAR benchmarks demonstrate the consistent superiority of STAR over state-of-the-art methods. For instance, STAR achieves up to 8.1% and 6.7% gains on the SSv2-Full and SSv2-Small datasets under the 1-shot setting, and 7.3% on HMDB51, validating its effectiveness under limited supervision. The code is available at https://github.com/HongliLiu1/STAR-main.

Hongli Liu, Yu Wang, Shengjie Zhao

School of Computer Science and Technology, Tongji University, Shanghai, China · Engineering Research Center of Key Software Technologies for Smart City Perception and Planning, Ministry of Education

Cross-Modal Representation Learning Few-Shot Learning Visual SSMs Video Representation Learning

Ask PDF

Feb 20, 2026

Spatio-temporal Decoupled Knowledge Compensator for Few-Shot Action Recognition

Few-Shot Action Recognition (FSAR) is a challenging task that requires recognizing novel action categories with a few labeled videos. Recent works typically apply semantically coarse category names as auxiliary contexts to guide the learning of discriminative visual features. However, such context provided by the action names is too limited to provide sufficient background knowledge for capturing novel spatial and temporal concepts in actions. In this paper, we propose DiST, an innovative Decomposition-incorporation framework for FSAR that makes use of decoupled Spatial and Temporal knowledge provided by large language models to learn expressive multi-granularity prototypes. In the decomposition stage, we decouple vanilla action names into diverse spatio-temporal attribute descriptions (action-related knowledge). Such commonsense knowledge complements semantic contexts from spatial and temporal perspectives. In the incorporation stage, we propose Spatial/Temporal Knowledge Compensators (SKC/TKC) to discover discriminative object-level and frame-level prototypes, respectively. In SKC, object-level prototypes adaptively aggregate important patch tokens under the guidance of spatial knowledge. Moreover, in TKC, frame-level prototypes utilize temporal attributes to assist in inter-frame temporal relation modeling. These learned prototypes thus provide transparency in capturing fine-grained spatial details and diverse temporal patterns. Experimental results show DiST achieves state-of-the-art results on five standard FSAR datasets.

Hongyu Qu, Xiangbo Shu, Rui Yan +3

School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China. · State Key Lab of Brain-Machine Intelligence, Zhejiang University, China.

Video Action Recognition Few-Shot Learning Prototype Learning

Ask PDF