Entropy-Coded MS-VQ-VAE with Learned Priors for Ultra-Low Bitrate Video Compression

Manikanta Kotthapalli; Banafsheh Rekabdar

cs.CVJun 28, 2026

Entropy-Coded MS-VQ-VAE with Learned Priors for Ultra-Low Bitrate Video Compression

Authors: Manikanta Kotthapalli, Banafsheh Rekabdar

Abstract

Learned video codecs based on continuous latent representations struggle to operate reliably below 0.1 bits per pixel~(bpp): without a differentiable rate signal, Lagrangian optimisation cannot effectively trade reconstruction quality for bitrate at extreme compression ratios. We demonstrate that discrete latent representations sidestep this limitation entirely. In a vector-quantized~(VQ) codec, the codebook size~ $K$ imposes a hard information ceiling of $\log_2 K$ bits per symbol; a learned autoregressive prior then exploits the non-uniform distribution of code usage -- which we show follows a power law -- to push actual bitrates well below this ceiling, without any rate-penalty tuning. Building on the MS-VQ-VAE architecture introduced in~\cite{kotthapalli2026msvqvae}, we sweep $K \in \{128, 256, 512, 1024\}$ under a uniform training protocol to trace four operating points on the rate-distortion~(RD) curve. We identify and resolve a critical training instability: gradient-based VQ collapses catastrophically at $K \leq 512$ , whereas EMA-stabilised codebook updates with dead-code restart maintain full utilisation across all configurations. On 500 UCF101 test clips ( $64\!\times\!64$ , 32~~frames), our models operate at 0.043-0.064~~bpp -- 3.3-5 $\times$ below H.264's practical floor and $5$ - $7.6\times$ below H.265's floor at this resolution. Every MS-VQ-VAE configuration outperforms H.265 CRF,36 on perceptual quality (LPIPS) despite using $5$ - $7.6\times$ fewer bits. At $K{=}1024$ , the model surpasses H.265 CRF,36 on LPIPS by a margin of 0.072 absolute while using $5.1\times$ fewer bits. Codebook analysis confirms power-law index distributions and 70-85% entropy efficiency, establishing the pipeline as a principled learned entropy coder.

Explore similar work

Jul 25, 2026eess.IV

Codebook Capacity Governs Perceptual Quality Across Resolutions in Hierarchical Discrete Video Compression

Learned video codecs based on continuous latent representations typically require resolution-specific retraining or rate-distortion (RD) recalibration when scaling to new spatial resolutions, because entropy models and Lagrangian weights are tightly coupled to the operating point. We investigate whether hierarchical discrete latent codecs exhibit the same sensitivity. Using a controlled empirical study of MS-VQ-VAE video compression across codebook sizes

K \in \{128,256,512,1024\}

and resolutions

64\times64

128\times128

, and

256\times256

on UCF101, we show that perceptual quality (LPIPS) depends strongly on codebook capacity but only negligibly on spatial resolution. Fitting a log-linear model

Q(K,r) = α\log_2 K + β\log_2 r + γ

to all 12 operating points yields

α=-0.0094

(

t=-6.6

p<0.001

) and

β=-0.0009

(

t=-0.43

p=0.68

, not significant), with

R^2=0.82

. Codebook capacity is therefore roughly

10\times

more influential than spatial resolution per log-unit increase. In parallel, bottom-level entropy efficiency

η=H(z)/\log_2 K

remains stable or improves with resolution (84-87% at

64\times64

; 92-94% at

256\times256

), confirming that larger spatial grids are utilized more efficiently rather than less. Across all resolutions and codebook sizes, our models outperform H.264 on LPIPS at matched or lower bitrate, with gains of 25-52% at

128\times128

and 21-37% over H.265 at

256\times256

. These findings suggest that codebook size

K

, not spatial resolution, is the dominant design variable governing perceptual compression quality in hierarchical discrete video codecs -- a property that may simplify multi-resolution deployment and inform the design of scalable discrete tokenizers for generative video models.

Manikanta Kotthapalli, Banafsheh Rekabdar

Neural Video Codecs Codebooks Perceptual Quality Resolution Retraining

Ask PDF

May 22, 2026eess.IV

Efficient Learned Image Compression without Entropy Coding

Entropy coding is widely used in typical learned image compression (LIC) that converts latents into a compact bitstream. However, entropy coding is typically sequential and becomes the coding latency bottleneck. To overcome it, we present Entropy-Coding Free Learned Image Compression (EF-LIC), a multi-rate framework that generates compact representation by removing statistical and correlation redundancy with low coding latency. First, we introduce unconstrained vector quantization and prove that its index distribution approaches the maximum-entropy bound, yielding minimal statistical redundancy. Second, we propose a context-conditioned autoregressive transform that directly reparameterizes the latents to reduce inter-dependency. Theoretical analysis shows that EF-LIC can remove correlation redundancy as effectively as typical LIC with entropy coding, leading to comparable compression performance. Experiments show EF-LIC achieves up to 67.86% bitrate reduction over MS-ILLM on Kodak with LPIPS. Ablation studies further show EF-LIC matches the compression performance of its entropy-coding based variant while achieving over

3\times

faster encoding and

5\times

faster decoding.

Hao Cao, Wenqi Guo, Zhijin Qin +1

Learned Image Compression Entropy Coding Calibration-Free Low-Bit Vector Quantization

Ask PDF

Aug 11, 2026cs.CV

Multiple Scale Latents for Learned Image Compression

Most learned image compression systems rely on a single latent representation combined with a hyperprior, which limits their ability to efficiently capture image structure across spatial scales. In this work, we propose a hierarchical latent representation to improve the efficiency of the entropy model. By using multiple latents at different scales, each with its own entropy model, we better capture the spatial structure of the latent representation. Our experiments show that this approach achieves a 17.9% BD-rate reduction over VVC on Kodak, demonstrating the effectiveness of multi-scale latent representations. Furthermore, the approach is orthogonal to other advances in learned image compression, making it a versatile addition to existing methods.

Jonas Brenig, Radu Timofte

Learned Image Compression Ultra-Low Bitrates Video Latents Single Images

Ask PDF