Many temporal approaches for video recognition rely on explicit cross-channel aggregation projections, which have quadratic complexity in the number of channels, introducing significant computational overhead. In addition, the spatio-temporal representation used for video recognition is biased either toward local order interactions (for CNNs) or toward global order interactions (for transformers). Starting from the multi-order spatial representation of MogaNet, our method introduces two complementary contributions to tackle these problems. First, for the temporal modelling, we introduce parallel multi-order processing with depthwise-only convolutions and implicit gating-based aggregation, which is linear in the number of channels, inducing a considerable computational reduction. Second, we propose a dual interaction between spatial and temporal dimensions, which allows a better spatio-temporal representation without increasing the computational cost. By combining these two contributions, our proposed VideoMogaNet achieves performance on par with or better than state-of-the-art action recognition approaches across multiple video benchmarks (UCF101, HMDB51, Kinetics-400, Diving48), while considerably reducing the cost of the temporal branch.
Figures & tables
Fig. 1 : Top-1 accuracy on Kinetics-400 at 2242 resolution versus model size (M parameters), with bubble size indicating computational cost (TFLOPs).
Fig. 2 : VideoMogaNet architecture. Left : Four-stage hierarchy following [ 36 ] ; unlike backbones that double channels ( Ci=2Ci−1 ), VideoMogaNet adopts a variable channel growth ratio across stages. For Base: {Ci}={64,160,320,512} , {Ni}={4,6,22,3} , FFN ratio ={8,8,4,4} , tubelet size =1 . Right : VideoMoga Block.
Fig. 3 : Left : Spatio-temporal design families and ST-MOI. Right : Grad-CAM activation maps of three baselines (UniFormer-S, VideoMamba-S, VideoMAE-S) and our VideoMogaNet-S on the class Tearing something into two pieces , suggesting sharper and more action-focused activations for our model.
Fig. 4 : Interaction strength . Left : Multi-order interaction strength across architectures. Right : Interaction strength per component.
Table 5
Architecture
Complexity
Scaling in nt
ViViT [ 2 ]
O((nhnw)2d+nt2d)
Quadratic
VideoMamba [ 41 ]
O(nhnwntd)
Linear
VideoMogaNet (total)
O(ntnhnwd2)
Linear
MOTM (ours)
O(ntnhnwd)
Linear, no d2
TABLE IV : Asymptotic complexity. nt : temporal length; nh×nw : spatial resolution; d : channel dimension.
Fig. 6 : Ablation analyses. (a) Inference throughput versus input resolution across six architectures. (b) Temporal modules on K400. (c) Temporal scalability of VideoMogaNet-B on Diving-48 at 8, 16 and 32 frames.
Design
Top-1
Spatio-temporal fusion
Standard (add.)
89.5
ST-MOI (ours)
90.3
Mixer ordering
S→T→C
89.3
S∥T→C
89.7
TABLE V : ST-MOI and mixer ordering ablations on Diving-48.
Configuration
Top-1
DW1d5,d=1 (single)
88.9
DW1d5,d=1 + DW1d7,d=3 (serial)
89.3
DW1d5,d=1 + DW1d5,d=2 + DW1d7,d=3 (serial)
89.7
Parallel, Cl:Cm:Ch=0:1:1
89.6
Parallel, Cl:Cm:Ch=1:6:9
89.9
Parallel, Cl:Cm:Ch=1:3:4 (MOTM)
90.3
TABLE VI : MOTM ablation on Diving-48.
Stage
Output
Layer
VideoMogaNet
Size
Settings
Tiny
Small
Base
Large
S1
T×4H×4W
Stem
Conv3×3,stride2,C/2Conv3×3,stride2,C
Embed. Dim.
32
64
64
64
# VideoMoga Block
3
2
4
4
MLP Ratio
8
S2
T×8H×8W
Stem
Conv3×3,stride2,C2
TABLE VII : Architecture configurations of VideoMogaNet variants. Input resolution is 8×2242 (8 frames). Unlike image-based models, the temporal dimension T is preserved across all stages with spatial-only downsampling. GFLOPs are reported per view; effective inference cost on K400 is GFLOPs per view multiplied by 12.
Variant
C1
C2
C3
C4
N1
N2
N3
N4
Tiny
32
64
128
256
3
3
12
2
Small
64
128
320
512
2
3
12
2
Base
64
160
320
512
4
6
22
3
Large
64
160
320
640
4
6
44
4
TABLE VIII : Per-stage embedding dimensions Ci and block counts Ni for each VideoMogaNet variant. Growth ratio Ci+1/Ci is shown between stages.
Configuration
UCF-101
HMDB51
K400
D-48
optimizer
AdamW
weight decay
0.001
LR schedule
CosineAnnealing
learning rate
8×10−4
batch size
256
warmup epochs
5
TABLE IX : Training configurations for VideoMogaNet across four benchmarks.
Ratio
J3D(m)
Iabs(m) ( ×10−3 )
ConvNeXt-T+Avg
MogaNet-S+Avg
VideoMogaNet-S
ConvNeXt-T+Avg
MogaNet-S+Avg
VideoMogaNet-S
0.00
0.071
0.182
0.171
2.784
5.064
6.386
0.05
0.097
0.108
0.109
3.804
2.993
4.087
0.10
0.089
0.085
0.085
3.487
2.356
3.177
0.20
0.057
0.066
0.066
2.238
1.823
2.473
0.30
0.043
0.058
0.059
1.691
1.624
2.187
TABLE X : Multi-order interaction strength J3D(m) on the 8×14×14 spatio-temporal grid (K400-trained). VideoMogaNet-S achieves the strongest middle-order encoding at m=0.5 , with consistent improvement over both baselines.
Ratio
J3D(m)
Iabs(m) ( ×10−3 )
SMOGA
MOTM
ST-MOI
SMOGA
MOTM
ST-MOI
0.00
0.182
0.064
0.171
5.064
2.525
6.386
0.05
0.108
0.050
0.109
2.993
1.999
4.087
0.10
0.085
0.049
0.085
2.356
1.940
3.177
0.20
0.066
0.117
0.066
1.823
4.639
2.473
0.30
0.058
0.154
0.059
1.624
6.107
2.187
TABLE XI : Compositional ablation on the 8×14×14 grid (K400-trained). MOTM achieves a strong middle-order peak ( J(0.3)=0.154 , J(0.5)=0.119 , +133% vs SMOGA) at lightweight cost ( 0.141 GFLOPs for VideoMogaNet-S). ST-MOI coordination produces a balanced profile that preserves spatial middle-order strength while integrating temporal dynamics.
Architecture
Complexity
Temporal scaling
ViViT [ 2 ]
O((nhnw)2d+nt2d)
Quadratic in nt
VideoMamba [ 41 ]
O(nhnwntd)
Linear
VideoMogaNet (total)
O(ntnhnwd2)
Linear in nt
Spatial (SMOGA+CA) †
O(ntnhnwd2)
Inherited d2 term
MOTM ( ours )
O(ntnhnwd)
No d2 term (depthwise only)
ST-MOI ( ours )
O(ntnhnwd)
No d2 term (element-wise only)
TABLE XII : Asymptotic complexity comparison. nt : temporal length; nh×nw : spatial resolution; d : channel dimension. GFLOPs are reported for single-view inference ( ×12 for UCF-101, HMDB-51, and K400, single-view for D-48), consistent with Sec. V-A of the main paper. † : spatial design extended from [ 32 ] to the video domain.
Model
Pretrain
Frames
Views
GFLOPs per view
Params (M)
Top-1 (%)
VideoMogaNet-T
IN-1K
8
4×3
8.806
4.963
76.7
VideoMogaNet-S
IN-1K
8
4×3
39.722
24.872
79.6
VideoMogaNet-B
IN-1K
8
4×3
79.398
43.275
80.6
VideoMogaNet-L
IN-1K
8
4×3
127.297
81.953
81.3
TABLE XIII : Scalability of VideoMogaNet variants on Kinetics-400. All models use ImageNet-1K pre-training, 8 frames, and 4×3 views. Unlike Tab. 3 of the main paper, GFLOPs here are per-view costs; multiply by 4×3 to obtain effective inference cost.
Fig. 7 : Grad-CAM visualization. VideoMogaNet’s spatio-temporal multi-order interaction (ST-MOI) effectively localizes and tracks action-relevant regions across temporal sequences. The architecture demonstrates precise focus on discriminative spatial regions where fine-grained object transformations occur (e.g., hand movements, object manipulations), while maintaining temporal consistency through multi-order temporal modulation (MOTM). Highlighted regions show the model’s ability to capture both spatial part-whole compositions via SMixer (SMOGA) and temporal dynamics via MOTM, validating the synergistic coordination between spatial and temporal modalities for fine-grained action understanding.
Fig. 8 : Grad-CAM visualization on HMDB51. VideoMogaNet’s ST-MOI architecture accurately localizes and tracks action-performing regions throughout video sequences. Attention maps demonstrate effective spatio-temporal coordination: SMixer (SMOGA) captures discriminative spatial structures (e.g., body parts, motion trajectories), while MOTM modulates these representations according to temporal context, enabling robust tracking of action dynamics across frames. The visualizations confirm that bounded multiplicative modulation ( 1+α⋅tanh(Mθ) ) preserves spatial middle-order strength while amplifying temporal signals, resulting in consistent focus on semantically relevant regions for general action recognition.
Space-time self-similarity (STSS), which captures visual correspondences across frames, provides an effective way to represent temporal dynamics for video understanding. In this work, we explore higher-order STSS and demonstrate how STSSs at different orders reveal distinct aspects of these dynamics. We then introduce the Multi-Order Self-Similarity (MOSS) module, a lightweight neural module designed to learn and integrate multi-order STSS features. It can be applied to diverse video tasks to enhance motion modeling capabilities while consuming only marginal computational cost and memory usage. Extensive experiments on video action recognition, motion-centric video VQA, and real-world robotic tasks consistently demonstrate substantial improvements, validating the broad applicability of MOSS as a general temporal modeling module. The source code and checkpoints will be publicly available.
Manjin Kim, Heeseung Kwon, Karteek Alahari +1
POSTECH · RLWRLD Inc. · INRIA Grenoble Rhône-Alpes
Fine-grained action recognition demands temporal reasoning that general-purpose architectures address through different cost-accuracy tradeoffs: 3D dense operators couple computation to the input volume, while difference-based methods approximate motion through rigid, hand-crafted subtraction of uncontextualized features - each reflecting a deliberate design choice with corresponding limitations in expressiveness or flexibility. We present MamBOA, a backbone-agnostic temporal framework built upon a novel interleaved scan structure that recasts the selective state-space recurrence (S6) as a native motion synthesizer. By interleaving consecutive feature representations extracted from a pretrained backbone into a single alternating sequence, the proposed scan structurally drives the recurrence to encode both temporal observations of each position within a shared hidden state, separated by only a single decay step - rendering the inter-frame transition an intrinsic component of the state dynamics rather than an externally computed quantity. A cascade of dedicated alignment and decoding operations then distills this joint encoding into an explicit motion representation, which a dual-path pooling mechanism adaptively aggregates by balancing attention-driven selection with uniform temporal coverage. The framework interfaces seamlessly with CNN, Transformer, and Mamba backbone families, adding only ~2.1 GFLOPs per feature pair. On Diving48, MamBOA achieves 85.02% Top-1 accuracy with an image-pretrained backbone and 86.24% with a video-pretrained backbone processing the entire video in a single forward pass - demonstrating that structurally induced state-space dynamics constitute a principled and general foundation for motion modeling.
Micro-actions are subtle, localized movements lasting 1-3 seconds such as scratching one's head or tapping fingers. Such subtle actions are essential for social communication, ubiquitously used in natural interactions, and thus critical for fine-grained video understanding, yet remain poorly understood by current computer vision systems. We identify a fundamental challenge: micro-actions exhibit diverse spatio-temporal characteristics where some are defined by spatial configurations while others manifest through temporal dynamics. Existing methods that commit to a single spatio-temporal decomposition cannot accommodate this diversity. We propose a dual-path network that processes anatomically-grounded spatial entities through parallel Spatial-Temporal (ST) and Temporal-Spatial (TS) pathways. The ST path captures spatial configurations before modeling temporal dynamics, while the TS path inverts this order to prioritize temporal dynamics. Rather than fixed fusion, we introduce entity-level adaptive routing where each body part learns its optimal processing preference, complemented by Mutual Action Consistency (MAC) loss that enforces cross-path coherence. Extensive experiments demonstrate competitive performance on MA-52 dataset and state-of-the-art results on iMiGUE dataset. Our work reveals that architectural adaptation to the inherent complexity of micro-actions is essential for advancing fine-grained video understanding.
Naga VS Raviteja Chappa, Evangelos Sariyanidi, Lisa Yankowitz +4
The Children’s Hospital of Philadelphia, USA · University of Pennsylvania, USA