Geometry-Aware Trajectory State Modeling for Few-Shot Action Recognition

Zhiqi Miao; Marko Haralović; Stanislav Levendeev; Sarvesh Angadi; Jannatul Ferdous; Dan-Cristian Ploesteanu; Jiapan Guo; Estefanía Talavera

cs.CVOct 28, 2026

Geometry-Aware Trajectory State Modeling for Few-Shot Action Recognition

Authors: Zhiqi Miao, Marko Haralović, Stanislav Levendeev, Sarvesh Angadi, Jannatul Ferdous, Dan-Cristian Ploesteanu, Jiapan Guo, Estefanía Talavera

Organizations: University of Groningen · University of Twente · Faculty of Electrical Engineering and Computing, University of Zagreb

Abstract

Few-shot action recognition (FSAR) requires representations that expose discriminative motion dynamics from only a few labeled videos. Recent trajectory-based methods leverage long-term point tracks for explicit temporal correspondence, but often treat trajectories as carriers of sampled visual features or summarize their motion with compact 2D displacement-based descriptors. We instead view an action as the ordered evolution of 2.5D trajectory states and their local geometric interactions. Based on this view, we introduce GeoTRAM, a geometry-aware trajectory state modeling framework for FSAR. GeoTRAM lifts sparse tracks into 2.5D trajectories, where StateMotion models ordered intra-trajectory state evolution and RelMotion captures state-conditioned local interactions through time-varying relative geometry and motion. Quality-filtered spectral sampling provides more reliable trajectory initialization; complementary semantic, motion, and depth cues are separately refined and preserved for joint spatio-temporal modeling; action-guided reweighting and top- $ρ$ trajectory selection support complementary frame- and trajectory-level matching. Experiments on five FSAR benchmarks demonstrate strong gains on the motion-centric Something-Something benchmarks while maintaining competitive performance on more appearance-driven benchmarks and a favorable accuracy--efficiency trade-off.

Figures & tables

Figure 1: Overview of GeoTRAM . Reliable tracking seeds are obtained through quality-filtered spectral sampling over DINOv2 [ 20 ] patch features and tracked across frames by CoTracker [ 12 ] . Relative monocular depth augments the image-plane tracks into 2.5D trajectories, on which StateMotion models the ordered evolution of explicit 2.5D intra-trajectory motion states, while RelMotion captures state-conditioned local interactions among neighboring trajectories. Semantic, depth, and motion cues undergo cue-specific temporal refinement before concatenation and joint spatio-temporal modeling. Finally, dual-CLS-guided reweighting suppresses less action-relevant trajectories, followed by the complementary frame-level prototype matching and top- $\rho$ trajectory-level instance matching.

Method	Venue	SSv2-Full					Kinetics
		1-shot	2-shot	3-shot	4-shot	5-shot	1-shot	2-shot	3-shot	4-shot	5-shot
OTAM [ 1 ]	CVPR’20	42.8	49.1	51.5	52.0	52.3	72.2	75.9	78.7	81.9	84.2
TRX [ 22 ]	CVPR’21	42.0	53.1	57.6	61.1	64.6	63.6	76.2	81.8	83.4	85.2
STRM [ 31 ]	CVPR’22	43.1	53.3	59.1	61.7	68.1	62.9	76.4	81.1	83.8	86.7
HyRSM [ 35 ]	CVPR’22	54.3	62.2	65.1	67.9	69.0	73.7	80.0	83.5	84.6	86.1
MoLo [ 34 ]	CVPR’23	56.6	62.3	67.0	68.5	70.6	74.0	80.4	83.7	84.7	85.6

Table 1: Comparison of few-shot action accuracy (%) under 1–5 shot settings on SSv2-Full and Kinetics. Best results are in bold , second-best results are underlined . A more comprehensive version of the table is shown in the supplementary material (Supp.Sec. ).

Method	Venue	SSv2-Small			UCF-101			HMDB-51
		1-shot	3-shot	5-shot	1-shot	3-shot	5-shot	1-shot	3-shot	5-shot
OTAM [ 1 ]	CVPR’20	36.4	45.9	48.0	79.9	87.0	88.9	54.5	65.7	68.0
TRX [ 22 ]	CVPR’21	36.0	51.9	56.7	78.2	92.4	96.1	53.1	66.8	75.6
STRM [ 31 ]	CVPR’22	37.1	49.2	55.3	80.5	92.7	96.9	52.3	67.4	77.3
HyRSM [ 35 ]	CVPR’22	40.6	52.3	56.1	83.9	93.0	94.7	60.3	71.7	76.0
MoLo [ 34 ]	CVPR’23	42.7	52.9	56.4	86.0	93.5	95.5	60.8	72.0	77.4

Table 2: Comparison of few-shot action accuracy (%) under 1-, 3-, and 5-shot settings on SSv2-Small, UCF-101, and HMDB-51. Best results are in bold , second-best results are underlined . A more comprehensive version of the table is shown in the Supp.Sec. .

	SSv2-Full						Kinetics
Method	5-way	6-way	7-way	8-way	9-way	10-way	5-way	6-way	7-way	8-way	9-way	10-way
OTAM [ 1 ]	42.8	38.6	35.1	32.3	30.0	28.2	72.2	68.7	66.0	63.0	61.9	59.0
TRX [ 22 ]	42.0	41.5	36.1	33.6	32.0	30.3	63.6	59.4	56.7	54.6	53.2	51.1
HyRSM [ 35 ]	54.3	50.1	45.8	44.3	42.1	40.0	73.7	69.5	66.6	65.5	63.4	61.0
MoLo [ 34 ]	56.6	51.6	48.1	44.8	42.5	40.3	74.0	69.7	67.4	65.8	63.5	61.3
TATs [ 15 ]	57.7	55.7	52.5	50.0	47.0	45.8	81.9	79.0	76.1	75.2	72.2	72.0

Table 3: Comparison of N-way 1-shot classification accuracy (N=5–10) on SSv2-Full and Kinetics datasets with recent representative FSAR methods. Best and second-best results are in bold and underlined , respectively.

Semantic Sampling	Spectral Sampling	Motion Modeling		MultiCue STM	Dual-CLS Reweight	Dual-branch Matching	SSv2-Small		SSv2-Full
Semantic Sampling	Spectral Sampling	State.	Rel.	MultiCue STM	Dual-CLS Reweight	Dual-branch Matching	1-shot	5-shot	1-shot	5-shot
✗	✗	✗	✗	✗	✗	✗	47.91	64.42	57.78	74.65
✓	✗	✗	✗	✗	✗	✗	49.93	65.71	58.62	74.26
✗	✓	✗	✗	✗	✗	✗	50.52	66.39	59.42	75.58
✗	✓	✓	✗	✗	✗	✗	53.36	68.73	62.35	77.45
✗	✓	✓	✓	✗	✗	✗	54.20	69.66	63.86	78.37

Table 4: Component ablation of GeoTRAM on SSv2-Small and SSv2-Full. Starting from the reproduced TATs baseline (first row), we compare semantic-aware and spectral sampling before progressively adding the remaining GeoTRAM components.

Motion Modeling	Fusion	GFLOPs	1-shot	3-shot	5-shot
Trokens-motion	Addition	38	54.58	66.67	70.22
GeoTRAM-motion	Addition	26	55.42	67.82	71.50
Trokens-motion	Multi-cue fusion	42	56.31	68.70	72.58
GeoTRAM-motion	Multi-cue fusion	28	57.71	70.36	74.07

Table 5: Motion–fusion interaction on SSv2-Small. Only motion modeling and fusion strategy vary within the same GeoTRAM configuration. GeoTRAM-motion (State.+Rel.) produces modest gains under addition but larger gains with multi-cue fusion.

Method	Param.	GFLOPs	SSv2-Small			SSv2-Full
Method	Param.	GFLOPs	1-shot	3-shot	5-shot	1-shot	3-shot	5-shot
TATs [ 15 ]	11M	21	47.90	60.00	64.40	57.70	70.00	74.60
Trokens [ 14 ]	17M	35	53.40	65.30	68.90	61.50	73.80	76.70
GeoTRAM	13M	28	57.71	70.36	74.07	68.85	79.89	82.30
GeoTRAM-S	8M	17	57.04	68.40	72.78	68.75	79.26	83.01

Table 6: Efficiency–accuracy comparison. We report 5-way accuracy (%). Param. refers to trainable parameters. GeoTRAM achieves a favorable accuracy–efficiency trade-off over prior trajectory-based FSAR methods.

Figure 2: Qualitative comparison of Trokens’ (left) and our (right) sampling. Ours reduces scattered background trajectories and concentrates on more spatially coherent action regions.

Representation	GFLOPs	SSv2-Small
Representation	GFLOPs	1-shot	3-shot	5-shot
GeoTRAM-2D	27	56.86	69.12	72.75
GeoTRAM-2.5D	28	57.71	70.36	74.07

Table 7: Contribution of relative depth to GeoTRAM. “2D” removes all depth-related parts, while “2.5D” denotes the full model.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Qualitative comparison between Trokens’ semantic-aware point sampling and our quality-filtered spectral point sampling strategies. For each video, the semantic clusters and sampled point locations are visualized on the frame. Semantic-aware sampling often produces fragmented or redundant samples across multiple semantic regions, whereas our quality-filtered spectral sampling suppresses isolated and scattered regions and allocates more seeds to spatially coherent hand–object interaction regions.

Figure 4: Accuracy–efficiency comparison on SSv2-Small and SSv2-Full under the 5-way 1-shot setting with different trajectory point counts. Bubble size denotes the number of tracked points. GeoTRAM consistently outperforms Trokens and TATs across the evaluated point counts, and its 32-point configuration already surpasses 256-point Trokens while reducing GFLOPs by 89%.

Figure 5: Class-wise accuracy comparison between Trokens and GeoTRAM on SSv2-Small. The top-10 gains (left) and three largest drops (right) are shown. Many of the largest gains occur for actions involving structured object motion and relative spatial relations.

Datasets	5-way 1-shot	5-way 3-shot	5-way 5-shot
SSv2-Full [ 7 ]	68.78 $\pm$ 0.23	79.80 $\pm$ 0.20	82.41 $\pm$ 0.18
SSv2-Small [ 7 ]	57.55 $\pm$ 0.31	70.41 $\pm$ 0.27	74.14 $\pm$ 0.27
Kinetics [ 2 ]	83.15 $\pm$ 0.22	90.23 $\pm$ 0.18	91.82 $\pm$ 0.16
UCF-101 [ 29 ]	93.40 $\pm$ 0.30	97.55 $\pm$ 0.25	98.11 $\pm$ 0.22
HMDB-51 [ 13 ]	71.13 $\pm$ 0.29	81.15 $\pm$ 0.25	83.91 $\pm$ 0.20

Appendix

Table 8: Stability of GeoTRAM across three independently trained runs with different random seeds. Accuracy (%) is reported as mean

\pm

standard deviation across runs.

Method	Venue	SSv2-Full					Kinetics
		1-shot	2-shot	3-shot	4-shot	5-shot	1-shot	2-shot	3-shot	4-shot	5-shot
OTAM [ 1 ]	CVPR’20	42.8	49.1	51.5	52.0	52.3	72.2	75.9	78.7	81.9	84.2
TRX [ 22 ]	CVPR’21	42.0	53.1	57.6	61.1	64.6	63.6	76.2	81.8	83.4	85.2
STRM [ 31 ]	CVPR’22	43.1	53.3	59.1	61.7	68.1	62.9	76.4	81.1	83.8	86.7
MTFAN [ 39 ]	CVPR’22	45.7	–	–	–	60.4	74.6	–	–	–	87.4
HyRSM [ 35 ]	CVPR’22	54.3	62.2	65.1	67.9	69.0	73.7	80.0	83.5	84.6	86.1

Appendix

Table 9: A more comprehensive comparison of few-shot action recognition accuracy (%) under 1–5 shot settings on SSv2-Full and Kinetics. Best results are in bold , second-best results are underlined .

Video Action Recognition

Video action recognition identifies and classifies human or animal activities by learning from visual appearance, motion, and temporal context.

See all 112 papers

STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition
May 13, 2026Hongli Liu, Yu Wang, Shengjie ZhaoCross-Modal Representation Learning Few-Shot Learning
School of Computer Science and Technology, Tongji University, Shanghai, China · Engineering Research Center of Key Software Technologies for Smart City Perception and Planning, Ministry of Education
Hierarchical Relation-augmented Representation Generalization for Few-shot Action Recognition
Apr 14, 2025Hongyu Qu, Ling Xing, Jiachao Zhang +3Temporal Reasoning Hierarchical Representation Learning
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · Artificial Intelligence Industrial Technology Research Institute, Nanjing Institute of Technology, Nanjing 211167, China
Spatio-temporal Decoupled Knowledge Compensator for Few-Shot Action Recognition
Feb 20, 2026Hongyu Qu, Xiangbo Shu, Rui Yan +3Video Action Recognition Few-Shot Learning
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China. · State Key Lab of Brain-Machine Intelligence, Zhejiang University, China.

May 13, 2026J/K move · Enter open · S save

STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition

Hongli Liu, Yu Wang, Shengjie Zhao

School of Computer Science and Technology, Tongji University, Shanghai, China · Engineering Research Center of Key Software Technologies for Smart City Perception and Planning, Ministry of Education

Cross-Modal Representation Learning Few-Shot Learning Visual SSMs Video Representation Learning

Open paper AskPDF arXiv

Few-shot action recognition (FSAR) requires models to generalize to novel action categories from only a handful of annotated samples. Despite progress with vision-language models, existing approaches still suffer from semantic-temporal misalignment, where static textual prompts fail to capture decisive visual cues that appear sparsely across sequences, and from inadequate modeling of multi-scale temporal dynamics, as short-term discriminative cues and long-range dependencies are often either oversmoothed or fragmented. To address these challenges, we propose Semantic Temporal Adaptive Representation Learning (STAR), a unified framework, consisting of a semantic-alignment component and a temporal-aware component, effectively bridging the semantic and temporal gaps and transferring the sequence modeling capability of Mamba into the FSAR. The semantic alignment module introduces a Temporal Semantic Attention (TSA) mechanism, which performs frame-level cross-modal alignment with textual cues, ensuring fine-grained semantic-temporal consistency. The temporal-aware module incorporates a Semantic Temporal Prototype Refiner (STPR) that integrates semantic-guided Mamba blocks with multi-frequency temporal sampling and bidirectional state-space refinement, yielding semantically aligned prototypes with enhanced discriminative fidelity and temporal consistency. Furthermore, temporally dependent class descriptors derived from large language models (LLMs) provide long-range semantic guidance. Extensive experiments on five FSAR benchmarks demonstrate the consistent superiority of STAR over state-of-the-art methods. For instance, STAR achieves up to 8.1% and 6.7% gains on the SSv2-Full and SSv2-Small datasets under the 1-shot setting, and 7.3% on HMDB51, validating its effectiveness under limited supervision. The code is available at https://github.com/HongliLiu1/STAR-main.

Abstract

Video Action Recognition

Explore similar work

STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition

Hierarchical Relation-augmented Representation Generalization for Few-shot Action Recognition

Spatio-temporal Decoupled Knowledge Compensator for Few-Shot Action Recognition