Spatio-Temporal Multi-Order Interaction for Video Recognition

Kevin D. Nguetche; Lilian Hollard; Eric Granger; Marco Pedersoli

cs.CVNov 3, 2026

Spatio-Temporal Multi-Order Interaction for Video Recognition

Authors: Kevin D. Nguetche, Lilian Hollard, Eric Granger, Marco Pedersoli

Abstract

Many temporal approaches for video recognition rely on explicit cross-channel aggregation projections, which have quadratic complexity in the number of channels, introducing significant computational overhead. In addition, the spatio-temporal representation used for video recognition is biased either toward local order interactions (for CNNs) or toward global order interactions (for transformers). Starting from the multi-order spatial representation of MogaNet, our method introduces two complementary contributions to tackle these problems. First, for the temporal modelling, we introduce parallel multi-order processing with depthwise-only convolutions and implicit gating-based aggregation, which is linear in the number of channels, inducing a considerable computational reduction. Second, we propose a dual interaction between spatial and temporal dimensions, which allows a better spatio-temporal representation without increasing the computational cost. By combining these two contributions, our proposed VideoMogaNet achieves performance on par with or better than state-of-the-art action recognition approaches across multiple video benchmarks (UCF101, HMDB51, Kinetics-400, Diving48), while considerably reducing the cost of the temporal branch.

Figures & tables

Fig. 1 : Top-1 accuracy on Kinetics-400 at

224^{2}

resolution versus model size (M parameters), with bubble size indicating computational cost (TFLOPs).

Fig. 2 : VideoMogaNet architecture. Left : Four-stage hierarchy following [ 36 ] ; unlike backbones that double channels (

C_{i}{=}2C_{i-1}

), VideoMogaNet adopts a variable channel growth ratio across stages. For Base:

\{C_{i}\}{=}\{64,160,320,512\}

\{N_{i}\}{=}\{4,6,22,3\}

, FFN ratio

{=}\{8,8,4,4\}

, tubelet size

{=}1

. Right : VideoMoga Block.

Panel of Figure 3 — Fig. 3 : Left : Spatio-temporal design families and ST-MOI. Right : Grad-CAM activation maps of three baselines (UniFormer-S, VideoMamba-S, VideoMAE-S) and our VideoMogaNet-S on the class Tearing something into two pieces , suggesting sharper and more action-focused activations for our model.

Panel of Figure 4 — Fig. 4 : Interaction strength . Left : Multi-order interaction strength across architectures. Right : Interaction strength per component.

Architecture	Complexity	Scaling in $n_{t}$
ViViT [ 2 ]	$\mathcal{O}\big((n_{h}n_{w})^{2}d+n_{t}^{2}d\big)$	Quadratic
VideoMamba [ 41 ]	$\mathcal{O}(n_{h}n_{w}n_{t}d)$	Linear
VideoMogaNet (total)	$\mathcal{O}(n_{t}n_{h}n_{w}d^{2})$	Linear
MOTM (ours)	$\mathcal{O}(n_{t}n_{h}n_{w}d)$	Linear, no $d^{2}$

TABLE IV : Asymptotic complexity.

n_{t}

: temporal length;

n_{h}\times n_{w}

: spatial resolution;

d

: channel dimension.

Panel of Figure 7 — Fig. 6 : Ablation analyses. (a) Inference throughput versus input resolution across six architectures. (b) Temporal modules on K400. (c) Temporal scalability of VideoMogaNet-B on Diving-48 at 8, 16 and 32 frames.

Spatio-temporal fusion
Design	Top-1
Standard (add.)	89.5
ST-MOI (ours)	90.3
Mixer ordering
$\mathrm{S}{\to}\mathrm{T}{\to}\mathrm{C}$	89.3
$\mathrm{S}{\parallel}\mathrm{T}{\to}\mathrm{C}$	89.7

TABLE V : ST-MOI and mixer ordering ablations on Diving-48.

Configuration	Top-1
$\mathrm{DW1d}_{5,d=1}$ (single)	88.9
$\mathrm{DW1d}_{5,d=1}$ + $\mathrm{DW1d}_{7,d=3}$ (serial)	89.3
$\mathrm{DW1d}_{5,d=1}$ + $\mathrm{DW1d}_{5,d=2}$ + $\mathrm{DW1d}_{7,d=3}$ (serial)	89.7
Parallel, $C_{l}{:}C_{m}{:}C_{h}=0{:}1{:}1$	89.6
Parallel, $C_{l}{:}C_{m}{:}C_{h}=1{:}6{:}9$	89.9
Parallel, $C_{l}{:}C_{m}{:}C_{h}=1{:}3{:}4$ (MOTM)	90.3

TABLE VI : MOTM ablation on Diving-48.

Stage	Output	Layer	VideoMogaNet
	Size	Settings	Tiny	Small	Base	Large
S1	$T\times\frac{H}{4}\times\frac{W}{4}$	Stem	$\rm{Conv}_{3\times 3},~\rm{stride}~2,C/2$ $\rm{Conv}_{3\times 3},~\rm{stride}~2,C$
		Embed. Dim.	32	64	64	64
		# VideoMoga Block	3	2	4	4
		MLP Ratio	8
S2	$T\times\frac{H}{8}\times\frac{W}{8}$	Stem	$\rm{Conv}_{3\times 3},~\rm{stride}~2,~C_{2}$

TABLE VII : Architecture configurations of VideoMogaNet variants. Input resolution is

8\times 224^{2}

(8 frames). Unlike image-based models, the temporal dimension

T

is preserved across all stages with spatial-only downsampling. GFLOPs are reported per view; effective inference cost on K400 is GFLOPs per view multiplied by 12.

Variant	$C_{1}$	$C_{2}$	$C_{3}$	$C_{4}$	$N_{1}$	$N_{2}$	$N_{3}$	$N_{4}$
Tiny	32	64	128	256	3	3	12	2
Small	64	128	320	512	2	3	12	2
Base	64	160	320	512	4	6	22	3
Large	64	160	320	640	4	6	44	4

TABLE VIII : Per-stage embedding dimensions

C_{i}

and block counts

N_{i}

for each VideoMogaNet variant. Growth ratio

C_{i+1}/C_{i}

is shown between stages.

Configuration	UCF-101	HMDB51	K400	D-48
optimizer	AdamW
weight decay	0.001
LR schedule	CosineAnnealing
learning rate	$8\times 10^{-4}$
batch size	256
warmup epochs	5

TABLE IX : Training configurations for VideoMogaNet across four benchmarks.

Ratio	$J^{(m)}_{\text{3D}}$			$I^{(m)}_{\text{abs}}$ ( $\times 10^{-3}$ )
Ratio	ConvNeXt-T+Avg	MogaNet-S+Avg	VideoMogaNet-S	ConvNeXt-T+Avg	MogaNet-S+Avg	VideoMogaNet-S
0.00	0.071	0.182	0.171	2.784	5.064	6.386
0.05	0.097	0.108	0.109	3.804	2.993	4.087
0.10	0.089	0.085	0.085	3.487	2.356	3.177
0.20	0.057	0.066	0.066	2.238	1.823	2.473
0.30	0.043	0.058	0.059	1.691	1.624	2.187

TABLE X : Multi-order interaction strength

J^{(m)}_{\text{3D}}

on the

8{\times}14{\times}14

spatio-temporal grid (K400-trained). VideoMogaNet-S achieves the strongest middle-order encoding at

m{=}0.5

, with consistent improvement over both baselines.

Ratio	$J^{(m)}_{\text{3D}}$			$I^{(m)}_{\text{abs}}$ ( $\times 10^{-3}$ )
Ratio	SMOGA	MOTM	ST-MOI	SMOGA	MOTM	ST-MOI
0.00	0.182	0.064	0.171	5.064	2.525	6.386
0.05	0.108	0.050	0.109	2.993	1.999	4.087
0.10	0.085	0.049	0.085	2.356	1.940	3.177
0.20	0.066	0.117	0.066	1.823	4.639	2.473
0.30	0.058	0.154	0.059	1.624	6.107	2.187

TABLE XI : Compositional ablation on the

8{\times}14{\times}14

grid (K400-trained). MOTM achieves a strong middle-order peak (

J^{(0.3)}{=}0.154

J^{(0.5)}{=}0.119

+133\%

vs SMOGA) at lightweight cost (

0.141

GFLOPs for VideoMogaNet-S). ST-MOI coordination produces a balanced profile that preserves spatial middle-order strength while integrating temporal dynamics.

Architecture	Complexity	Temporal scaling
ViViT [ 2 ]	$\mathcal{O}\!\left((n_{h}n_{w})^{2}d+n_{t}^{2}d\right)$	Quadratic in $n_{t}$
VideoMamba [ 41 ]	$\mathcal{O}(n_{h}n_{w}n_{t}d)$	Linear
VideoMogaNet (total)	$\mathcal{O}(n_{t}n_{h}n_{w}d^{2})$	Linear in $n_{t}$
Spatial (SMOGA+CA) †	$\mathcal{O}(n_{t}n_{h}n_{w}d^{2})$	Inherited $d^{2}$ term
MOTM ( ours )	$\mathcal{O}(n_{t}n_{h}n_{w}d)$	No $d^{2}$ term (depthwise only)
ST-MOI ( ours )	$\mathcal{O}(n_{t}n_{h}n_{w}d)$	No $d^{2}$ term (element-wise only)

TABLE XII : Asymptotic complexity comparison.

n_{t}

: temporal length;

n_{h}\times n_{w}

: spatial resolution;

d

: channel dimension. GFLOPs are reported for single-view inference (

{\times}12

for UCF-101, HMDB-51, and K400, single-view for D-48), consistent with Sec. V-A of the main paper.

\dagger

: spatial design extended from [ 32 ] to the video domain.

Model	Pretrain	Frames	Views	GFLOPs per view	Params (M)	Top-1 (%)
VideoMogaNet-T	IN-1K	8	$4{\times}3$	8.806	4.963	76.7
VideoMogaNet-S	IN-1K	8	$4{\times}3$	39.722	24.872	79.6
VideoMogaNet-B	IN-1K	8	$4{\times}3$	79.398	43.275	80.6
VideoMogaNet-L	IN-1K	8	$4{\times}3$	127.297	81.953	81.3

TABLE XIII : Scalability of VideoMogaNet variants on Kinetics-400. All models use ImageNet-1K pre-training, 8 frames, and

4{\times}3

views. Unlike Tab. 3 of the main paper, GFLOPs here are per-view costs; multiply by

4{\times}3

to obtain effective inference cost.

Fig. 7 : Grad-CAM visualization. VideoMogaNet’s spatio-temporal multi-order interaction (ST-MOI) effectively localizes and tracks action-relevant regions across temporal sequences. The architecture demonstrates precise focus on discriminative spatial regions where fine-grained object transformations occur (e.g., hand movements, object manipulations), while maintaining temporal consistency through multi-order temporal modulation (MOTM). Highlighted regions show the model’s ability to capture both spatial part-whole compositions via $\mathrm{SMixer}$ (SMOGA) and temporal dynamics via MOTM, validating the synergistic coordination between spatial and temporal modalities for fine-grained action understanding.

Fig. 8 : Grad-CAM visualization on HMDB51. VideoMogaNet’s ST-MOI architecture accurately localizes and tracks action-performing regions throughout video sequences. Attention maps demonstrate effective spatio-temporal coordination: $\mathrm{SMixer}$ (SMOGA) captures discriminative spatial structures (e.g., body parts, motion trajectories), while MOTM modulates these representations according to temporal context, enabling robust tracking of action dynamics across frames. The visualizations confirm that bounded multiplicative modulation ( $1+\alpha\cdot\tanh(\mathcal{M}_{\theta})$ ) preserves spatial middle-order strength while amplifying temporal signals, resulting in consistent focus on semantically relevant regions for general action recognition.

Video Action Recognition

Video action recognition identifies and classifies human or animal activities by learning from visual appearance, motion, and temporal context.

See all 115 papers

Exploring High-Order Self-Similarity for Video Understanding
Apr 22, 2026Manjin Kim, Heeseung Kwon, Karteek Alahari +1Spatiotemporal Reasoning Video Action Recognition
POSTECH · RLWRLD Inc. · INRIA Grenoble Rhône-Alpes
MamBOA: State-Space Architecture for Video Recognition
Jun 13, 2026Mustafa Bora ÇelikVisual SSMs Video Action Recognition
Ankara Medipol University, Ankara, 06050, Turkey
Micro-DualNet: Dual-Path Spatio-Temporal Network for Micro-Action Recognition
Apr 22, 2026Naga VS Raviteja Chappa, Evangelos Sariyanidi, Lisa Yankowitz +4Video Action Recognition Fine-Grained Visual Perception
The Children’s Hospital of Philadelphia, USA · University of Pennsylvania, USA

Apr 22, 2026J/K move · Enter open · S save

Exploring High-Order Self-Similarity for Video Understanding

Manjin Kim, Heeseung Kwon, Karteek Alahari, Minsu Cho

POSTECH · RLWRLD Inc. · INRIA Grenoble Rhône-Alpes

Spatiotemporal Reasoning Video Action Recognition Temporal Video Understanding

Open paper AskPDF arXiv

Space-time self-similarity (STSS), which captures visual correspondences across frames, provides an effective way to represent temporal dynamics for video understanding. In this work, we explore higher-order STSS and demonstrate how STSSs at different orders reveal distinct aspects of these dynamics. We then introduce the Multi-Order Self-Similarity (MOSS) module, a lightweight neural module designed to learn and integrate multi-order STSS features. It can be applied to diverse video tasks to enhance motion modeling capabilities while consuming only marginal computational cost and memory usage. Extensive experiments on video action recognition, motion-centric video VQA, and real-world robotic tasks consistently demonstrate substantial improvements, validating the broad applicability of MOSS as a general temporal modeling module. The source code and checkpoints will be publicly available.

Abstract

Video Action Recognition

Explore similar work

Exploring High-Order Self-Similarity for Video Understanding

MamBOA: State-Space Architecture for Video Recognition

Micro-DualNet: Dual-Path Spatio-Temporal Network for Micro-Action Recognition