A Baseline Study and Benchmark for Few-Shot Open-Set Action Recognition with Feature Residual Discrimination

Stefano Berti; Giulia Pasquale; Lorenzo Natale

cs.CVMar 4, 2026

A Baseline Study and Benchmark for Few-Shot Open-Set Action Recognition with Feature Residual Discrimination

Authors: Stefano Berti, Giulia Pasquale, Lorenzo Natale

Organizations: Humanoid Sensing and Perception, Istituto Italiano di Tecnologia, Genoa, Italy

Abstract

Few-Shot Action Recognition (FS-AR) has shown promising results but is often limited by a closed-set assumption that fails in real-world open-set scenarios. While Few-Shot Open-Set (FSOS) recognition is well-established for images, its extension to spatio-temporal video data remains underexplored. To address this, we propose an architectural extension based on a Feature-Residual Discriminator (FR-Disc), adapting previous work on skeletal data to the more complex video domain. Extensive experiments on five datasets demonstrate that while common open-set techniques provide only marginal gains, our FR-Disc significantly enhances unknown rejection capabilities without compromising closed-set accuracy, setting a new state-of-the-art for FSOS-AR. The project website, code, and benchmark are available at: https://hsp-iit.github.io/fsosar/.

Figures & tables

Figure 1: Comparison between the results of the closed-set Softmax Baseline SAFSAR [ 34 ] and its proposed open-set extension FR-Disc on an unknown query . The Support Set contains $K=2$ classes and $N=1$ videos. The true class of the query is not contained in the Support Set, so it should be rejected through low confidence ( e.g., $<50\%$ ). The Baseline incorrectly classifies it as known (false positive), while FR-Disc correctly rejects it (true negative). Additional qualitative examples are provided in the Supplementary Material.

Figure 2: Overview of closed- and open-set prediction flows for (a) implicit (Softmax, EOS), (b) explicit (Garbage Class), and (c) Discriminator-based methods. For implicit methods, $\hat{u}_{i}$ is derived via MLS/MSS. In (b), a query is rejected if the garbage prototype $P_{g}$ yields the highest probability. In (c), $\hat{u}_{i}$ is computed by an MLP with Sigmoid activation using the $\varTheta$ function (Eq. 6 ). (d) and (e) illustrate the feature extraction and inference decision flow for SAFSAR and STRM models.

OS-Method	FS ACC		OS ACC		AUROC		AUPR		OSCR
OS-Method	1-s	5-s	1-s	5-s	1-s	5-s	1-s	5-s	1-s	5-s
Diving48
Softmax	63.49	74.12	64.16	65.36	68.48	71.49	69.71	72.07	59.86	66.15
EOS	64.43	72.80	62.91	64.94	68.64	74.60	69.71	75.06	60.36	67.43
GC	65.01	76.32	64.89	69.36	70.18	75.82	71.38	76.01	60.63	69.13
FR-Disc	68.83	78.58	66.22	70.29	71.28	76.55	72.28	76.32	63.46	71.04

Table 1: Combined results for SAFSAR in 5-way 1-Shot and 5-Shot settings. Green/red values indicate improvement/decline over the Softmax baseline. Values marked with (*) denote models trained for 1K iterations to prevent overfitting.

OS-Method	FS ACC	OS ACC	AUROC	AUPR	OSCR
Diving48
Softmax	74.74	52.91	58.92	63.25	63.61
EOS	73.58	60.51	67.01	69.02	65.63
GC	73.22	53.76	71.71	72.56	66.09
FR-Disc	77.74	64.92	75.73	75.72	70.00
SSv2

Table 2: Results for STRM in 5-way 5-Shot settings. Green/red values indicate improvement/decline over the Softmax baseline.

Panel of Figure 3 — Figure 3: Analysis of open-set scoring and performance correlation. (a) Comparison between MSS and MLS for SAFSAR’s Softmax Baseline on SSv2 1-Shot. (b) Global correlation between closed-set and open-set metrics across all five datasets for both SAFSAR and STRM.

Panel of Figure 4 — Figure 4: Qualitative comparison between the SAFSAR Softmax baseline and FR-Disc on the SSv2 test set. (a) t-SNE of video features $\phi(x_{i})$ for the baseline (left) and FR-Disc (right). (b) Score histograms for known ( blue ) and unknown ( red ) queries, comparing the baseline (top) and FR-Disc (bottom). Our method yields tighter feature clustering and assigns lower confidence to unknown samples.

Explore similar work

Spatio-temporal Decoupled Knowledge Compensator for Few-Shot Action Recognition
Feb 20, 2026Hongyu Qu, Xiangbo Shu, Rui Yan +3Video Action Recognition Few-Shot Learning
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China. · State Key Lab of Brain-Machine Intelligence, Zhejiang University, China.
Few-Shot Video Recognition via Hierarchical Metric Learning
Sep 4, 2026Jiaxin Zhang, Haoran Gao, Xizhan Gao +3Few-Shot Learning Video Action Recognition
School of Information Science and Engineering, University of Jinan, Shandong, China
STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition
May 13, 2026Hongli Liu, Yu Wang, Shengjie ZhaoCross-Modal Representation Learning Few-Shot Learning
School of Computer Science and Technology, Tongji University, Shanghai, China · Engineering Research Center of Key Software Technologies for Smart City Perception and Planning, Ministry of Education

Feb 20, 2026J/K move · Enter open · S save

Spatio-temporal Decoupled Knowledge Compensator for Few-Shot Action Recognition

Hongyu Qu, Xiangbo Shu, Rui Yan, Hailiang Gao+2

School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China. · State Key Lab of Brain-Machine Intelligence, Zhejiang University, China.

Video Action Recognition Few-Shot Learning Prototype Learning

Open paper AskPDF arXiv

Few-Shot Action Recognition (FSAR) is a challenging task that requires recognizing novel action categories with a few labeled videos. Recent works typically apply semantically coarse category names as auxiliary contexts to guide the learning of discriminative visual features. However, such context provided by the action names is too limited to provide sufficient background knowledge for capturing novel spatial and temporal concepts in actions. In this paper, we propose DiST, an innovative Decomposition-incorporation framework for FSAR that makes use of decoupled Spatial and Temporal knowledge provided by large language models to learn expressive multi-granularity prototypes. In the decomposition stage, we decouple vanilla action names into diverse spatio-temporal attribute descriptions (action-related knowledge). Such commonsense knowledge complements semantic contexts from spatial and temporal perspectives. In the incorporation stage, we propose Spatial/Temporal Knowledge Compensators (SKC/TKC) to discover discriminative object-level and frame-level prototypes, respectively. In SKC, object-level prototypes adaptively aggregate important patch tokens under the guidance of spatial knowledge. Moreover, in TKC, frame-level prototypes utilize temporal attributes to assist in inter-frame temporal relation modeling. These learned prototypes thus provide transparency in capturing fine-grained spatial details and diverse temporal patterns. Experimental results show DiST achieves state-of-the-art results on five standard FSAR datasets.

Abstract

Explore similar work

Spatio-temporal Decoupled Knowledge Compensator for Few-Shot Action Recognition

Few-Shot Video Recognition via Hierarchical Metric Learning

STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition