Recent advances in learned video compression (LVC) have led to significant performance gains, with codecs such as DCVC-RT surpassing the H.266/VVC low-delay mode in compression efficiency. However, existing LVCs still exhibit key limitations: they often require separate models for intra and inter coding modes, and their performance degrades when temporal references are unreliable. To address this, we introduce Uni-LVC, a unified LVC method that supports both intra and inter coding with low-delay and random-access in a single model. Building on a strong intra-codec, Uni-LVC formulates inter-coding as intra-coding conditioned on temporal information extracted from reference frames. We design an efficient cross-attention adaptation module that integrates temporal cues, enabling seamless support for both unidirectional (low-delay) and bidirectional (random-access) prediction modes. A reliability-aware classifier is proposed to selectively scale the temporal cues, making Uni-LVC behave closer to intra coding when references are unreliable. We further propose a multistage training strategy to facilitate adaptive learning across various coding modes. Extensive experiments demonstrate that Uni-LVC achieves superior rate-distortion performance in intra and inter configurations while maintaining comparable computational efficiency.
Figures & tables
Fig. 1: PSNR and BPP vs frames. The results are obtained on videoSRC21 sequence from MCL-JCV, which contains a scene change at frame 48. At this frame, DCVC-RT continues to rely on previous time references, causing a sharp drop in PSNR. In contrast, Uni-LVC automatically suppresses unreliable temporal features and switches to intra‐dominant coding ( αt≈0.1 ), keeping stable PSNR quality.
Fig. 2: Overview of the proposed Uni-LVC. “Enc.” and “Dec.” denote the encoder and decoder, respectively, while “Tem.” represents the temporal modeling module. Inter coding (LD and RA) is formulated as intra coding conditioned on auxiliary temporal features extracted from temporal references stored in the buffer.
Fig. 3: Architecture of the proposed intra codec. “Cross Attn” and “DC Block” denote the cross-attention module and enhanced depthwise convolution block. “Q” indicates quantization. A hybrid buffer ft is formed from ftd and ftr and stores temporal references. For inter coding, the temporal feature ft−1 (or its bidirectional counterpart) is fed through Cross Attn; for pure intra coding, ft−1 is a quality-specific learnable vector fp to preserve compatibility and learn global priors. HPCM denotes the hierarchical progressive context model.
Fig. 9: Rate-distortion (R-D) curves for various methods for intra coding. Please zoom in for more details.
Fig. 10: Rate-distortion (R-D) curves for various methods for inter coding, intra period of 32. Please zoom in for more details.
Method
Type
Total Params. ↓
Latency ↓
BD-Rate (%) w.r.t. VTM 18.0 ↓
Encoding
Decoding
HEVC Class B
HEVC Class C
HEVC Class D
HEVC Class E
UVG
MCL-JCV
Average
VTM-18.0
AI
-
-
-
0%
0%
0%
0%
0%
0%
0%
LALIC [ 16 ]
AI
396.8M
0.491s
0.165s
-16.57%
-15.06%
-14.54%
-20.60%
-16.92%
-17.46%
-16.86%
DCAE [ 15 ]
AI
716.4M
0.346s
0.353s
-20.76%
-16.17%
-14.26%
-23.71%
-19.29%
-20.62%
-19.14%
HPCM [ 14 ]
AI
538.3M
0.232s
0.212s
-19.88%
-20.24%
-19.56%
-26.10%
-19.89%
-20.76%
-21.07%
DCVC-RT AI [ 2 ]
AI
45.7M
0.023s
0.018s
-13.85%
-15.91%
-15.67%
-19.45%
-13.84%
-14.77%
-15.58%
TABLE I: Computational Complexity and BD-Rate Compared to Existing Methods. Best result is underlined . Bold denotes our method.
Variant
BD-Rate (AI, Class B)
B0: DCVC-RT (re-impl)
-12.35%
B1: B0 + enhanced blocks
-14.03%
B2: B1 + HPCM
-16.05%
B3: B2 + LVQ (ours final)
-18.41%
TABLE II: Ablation on intra codec components on HEVC Class B in the AI configuration. Anchor is VTM 18.0 AI (lower is better).
Variant
BD-Rate (LD, Class B)
I0: direct concatenate temporal features
26.71%
I1: temporal path (Swin-CA), no classifier
25.97%
I2: replace Swin-CA with DN-CA
22.93%
I3: add PAL-CA to I2
18.63%
I4: add the reliability-aware classifier (ours final)
16.71%
TABLE III: Ablation on temporal adaptation and the reliability-aware classifier on HEVC Class B. Anchor is VTM 18.0 RA (lower is better).
Scheme
AI
LD
RA
T0: single-stage joint training
-10.19%
27.9%
20.1%
T1: sequential w/o replay
40.51%
60.56%
9.01%
T2: staged + replay (ours final)
-18.41%
16.71%
10.31%
TABLE IV: Ablation on the training strategy on HEVC Class B. Numbers are BD-Rate (%) versus the VTM reference for AI, LD, and RA configurations (lower is better).
Variant
BD-Rate (LD IP32, Class B)
No gating
18.63%
Decoder-side spatial gating (no original frame input)
Video compression reduces the bits needed to store or transmit moving images while balancing visual quality, computational cost, and downstream usefulness.
Video diffusion models can reconstruct visually compelling content from severely compressed representations at ultra-low bitrates, but they impose a rigid deployment trade-off. Bidirectional (BI) models achieve high quality through dense full-sequence attention, yet require the entire sequence before reconstruction and incur substantial latency. Autoregressive (AR) models enable low-delay streaming, but their restricted temporal context reduces compression efficiency. This tension is particularly relevant when a video stream must be decoded in real time yet is also stored for later, quality-oriented playback, making AR and BI models natural choices for the live and replay stages, respectively. To support both stages, we present UniGVC, a Unified Generative Video Codec that supports two decoding modes from a single bitstream. UniGVC-BI employs a BI-DiT with dense attention for quality-oriented random-access decoding, whereas UniGVC-AR uses windowed AR-DiT for low-delay streaming. Both paths share a hierarchical latent codec that recurrently produces the same conditions with ultra-low bitrates, allowing UniGVC-BI to serve as a conditional teacher for single-step distillation of UniGVC-AR. A tiny conditional video decoder further reduces streaming latency while preserving most of the reconstruction quality. On standard 1080p benchmarks, UniGVC-BI achieves the best overall rate-distortion-perception trade-off among the evaluated methods and produces temporally stable video at rates as low as 0.0005 bits per pixel. Meanwhile, UniGVC-AR remains competitive while decoding 1080p video at 36.85 FPS, over 250× faster than UniGVC-BI. Code and models will be released later.
Tianyu Zhang, Shuai Huo, Dengchao Jin +7
University of Science and Technology of China · Kuaishou Technology
Neural video compression (NVC) technologies have advanced rapidly in recent years, yielding state-of-the-art schemes such as DCVC-RT that offer superior compression efficiency to H.266/VVC and real-time encoding/decoding capabilities. Nonetheless, existing NVC schemes have several limitations, including inefficiency in dealing with disocclusion and new content, interframe error propagation and accumulation, among others. To eliminate these limitations, we borrow the idea from classic video coding schemes, which allow intra coding within inter-coded frames. With the intra coding tool enabled, disocclusion and new content are properly handled, and interframe error propagation is naturally intercepted without the need for manual refresh mechanisms. We present an NVC framework with unified intra and inter coding, where every frame is processed by a single model that is trained to perform intra/inter coding adaptively. Moreover, we propose a simultaneous two-frame compression design to exploit interframe redundancy not only forwardly but also backwardly. Experimental results show that our scheme outperforms DCVC-RT by an average of 12.1% BD-rate reduction, delivers more stable bitrate and quality per frame, and retains real-time encoding/decoding performances. Code and models will be released.
Hui Xiang, Yifan Bian, Li Li +3
University of Science and Technology of China · Tencent Shannon Lab
While neural video codecs (NVCs) have demonstrated superior compression ratio, their prohibitive computational complexity remains a critical barrier to real-world deployment. This paper introduces a chunk-based coding framework designed to significantly improve the rate-distortion-complexity trade-off. Instead of processing frames sequentially, our approach encodes a chunk of multiple frames into a single compact latent representation and decodes them simultaneously. This is enabled by cross-frame interaction modules for joint spatial-temporal modeling and frame-specific decoders for parallel reconstruction. This paradigm not only dramatically enhances coding throughput but also facilitates more effective modeling of long-term temporal correlations. To further boost speed, we propose a streamlined entropy coding mechanism that consolidates bit-stream interactions into a single step, substantially reducing decoding overhead. Building on these innovations, we present DCVC-UF (Ultra-Fast), a new NVC that sets a new SOTA in performance. Our experiments show that DCVC-UF can achieve ultra-fast encoding and decoding speeds, significantly outperforming previous leading codecs. DCVC-UF serves as a notable landmark in the journey of NVC evolution. The code is at https://github.com/microsoft/DCVC.
Jiahao Li, Wenxuan Xie, Zhaoyang Jia +4
Microsoft Research Asia · University of Science and Technology of China · *Done during Zhaoyang Jia’s internship at Microsoft Research Asia.