Geometric Transformation-Embedded Mamba for Learned Video Compression
Authors: Hao Wei, Yanhui Zhou, Chenyang Ge
Organizations: State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, No.28, West Xianning Road, Xi’an, 710049, Shaanxi, China · School of Information and Communications Engineering, Xi’an Jiaotong University, No.28, West Xianning Road, Xi’an, 710049, Shaanxi, China
Although learned video compression methods have exhibited outstanding performance, most of them typically follow a hybrid coding paradigm that requires explicit motion estimation and compensation, resulting in a complex solution for video compression. In contrast, we introduce a streamlined yet effective video compression framework founded on a direct transform strategy, i.e., nonlinear transform, quantization, and entropy coding. We first develop a cascaded Mamba module (CMM) with different embedded geometric transformations to effectively explore both long-range spatial and temporal dependencies. To improve local spatial representation, we introduce a locality refinement feed-forward network (LRFFN) that incorporates a hybrid convolution block based on difference convolutions. We integrate the proposed CMM and LRFFN into the encoder and decoder of our compression framework. Moreover, we present a conditional channel-wise entropy model that effectively utilizes conditional temporal priors to accurately estimate the probability distributions of current latent features. Extensive experiments demonstrate that our method outperforms state-of-the-art video compression approaches in terms of perceptual quality and temporal consistency under low-bitrate constraints. Our source codes and models will be available at https://github.com/cshw2021/GTEM-LVC.
Figures & tables
Figure 1: Architectural comparisons with existing video compression methods: (a) shows a hybrid coding framework; (b) is a transform-based coding method that is both frame-independent and latent-dependent; (c) illustrates the proposed coding approach that is frame-dependent and latent-dependent.
Figure 2: The overall architecture of the proposed video compression framework. To capture both long-range spatio-temporal dependencies and local dependencies, we develop the cascaded Mamba module (CMM) for global modeling and locality refinement feed-forward network (LRFFN) for local modeling. The CMM is implemented by sequentially traversing video features in four directional orders, namely forward spatio-temporal (FST), backward spatio-temporal (BST), forward temporal–spatial (FTS), and backward temporal–spatial (BTS).
Figure 3: Detailed architecture of geometric transformation Mamba block and different selective scanning methods. We first extract spatio-temporal video patches and then perform selective scanning along spatial and temporal dimension, depending on the priority assigned to each dimension.
Figure 4: Detailed architecture of the proposed locality refinement feed-forward network.
Figure 5: Detailed architecture of the proposed conditional channel-wise entropy model.
Figure 6: The motion difference comparison among successive video frames.
Figure 7: Quantitative comparisons with state-of-the-art methods on benchmark datasets.
Figure 8: Visual comparisons of different compression methods. The number in parentheses denotes the bpp used for compression. Compared with competing approaches, our method produces reconstructions with more faithful structural details at lower bitrates.
Figure 9: Quantitative comparison of individual components under the rate–perception tradeoff.
Figure 10: Effectiveness of the proposed CMM.
Figure 11: Visual comparisons of different FFN variants for video compression.
Figure 12: Visual comparisons of temporal consistency for decoded frames. We visualize pixel values along selected columns, indicated by red solid lines in the original video frames. Zoom in for best view.
Methods
Parameters (M)
Encoding time (ms)
Decoding time (ms)
DCVC
19.77
884.63
4,092,38
DCVC-HEM
48.69
65.58
44.12
DCVC-DC
50.79
57.49
90.54
DCVC-FM
44.93
53.97
131.60
Distributed-DVC
47.21
N/A
N/A
DHVC
117.92
59.53
57.29
Table 1: Model complexity comparison on the video frames with 512 × 512 resolution. The encoding and decoding times are measured on a machine with an NVIDIA GeForce RTX 3090 GPU. Best and second-best results are marked in bold and underlined , respectively.
In this paper we propose DCVC-Mamba (DCVC-MB), a neural video codec framework for B-frame coding. Our approach incorporates an IBP frame strategy for low-delay B-frame coding, a spatio-temporal fusion model based on state-space models for bidirectional temporal prediction, and an entropy-aware skipping mechanism that selectively omits coding certain latents to reduce entropy coding times. In addition to our model contributions we also implement two inference-time strategies that enhance compression performance. Experimental evaluation shows that DCVC-MB compares favorably to existing NVCs and traditional codecs. The method demonstrates BD-rate reductions of up to 8.98% on average compared to prior neural video codecs, and improvements of up to 30.45% and 1.81% over the VTM-19.0-LDP and VTM-19.0-RA(Inter-GoP=16) benchmarks, respectively, contributing to advances in neural video compression.
Arjun Arora, Calvin-Khang Ta, Carlos Restrepo-Galeano +7
Dolby Laboratories · Voia · University of Delaware +1
Video large language models (Video-LLMs) represent videos as dense sequences of visual tokens, whose length grows with the temporal and spatial extent of the input. These tokens often contain substantial redundancy arising from repeated visual patterns, leading to unnecessary computation in the subsequent language-model processing. Existing token compression methods, including pruning and merging, perform compression online during inference, repeatedly incurring additional computation for each input video and often relying on model-specific designs that limit their generality, we instead rethink this paradigm by shifting the costly compression process offline. We propose ONCE, a plug-in video token compression framework that introduces an offline-to-online paradigm: a frequency-aware global codebook is learned once in the visual feature space and reused for lightweight online compression through codebook lookup and aggregation, reducing repeated per-video computation and the need for model-specific compression designs. Extensive experiments across multiple video understanding benchmarks and against diverse compression baselines demonstrate that our approach achieves a strong accuracy-efficiency trade-off, maintaining competitive performance while achieving the lowest inference latency among compared methods.
Jiayang He, Tianling Xu, Diancheng Kang +4
Southern University of Science and Technology · Imperial College London · Jilin University
Most existing video compression algorithms follow a paradigm of transformation and quantization, optimizing the trade-off between distortion and bitrate. However, extremely low-bitrate compression remains an underexplored frontier where perceptual quality optimization under severely constrained coding resources has not been adequately addressed. In this paper, we propose a unified generative framework that leverages pre-trained Diffusion Transformer (DiT) priors to achieve high perceptual quality at extremely low bitrates. We first introduce a flexible Group-of-Latents (GoL) strategy within the latent space of a causal tokenizer, explicitly partitioning the latent stream into intra I-latents and inter P-latents. The Deep Compression Module (I-DCM) then encodes key I-latents to preserve perceptual anchors with minimal overhead. Building upon these anchors, the DiT-based Unified Latent Denoising Module (U-LDM) refines intra-frame textures and synthesizes P-latents from noise, reconstructing temporal dynamics at zero additional bitrate cost. Extensive experiments demonstrate that our method uniquely operates in the extreme-low-bitrate regime (e.g., (<0.005) bpp), achieving state-of-the-art perceptual fidelity with rich spatial details and robust temporal consistency. The code will be made publicly available.
Shaokang Wang, Jinchang Xu, Peidong Jia +9
1State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University, Beijing, China · 2State Key Discipline Laboratory of Wide Band-Gap Semiconductor Technology, School of Microelectronics, Xidian University, Xi'an, China · 3X-humanoid, Beijing, China