Neural Video Coding under Limited Delay: Theoretical Optimality and System Design

Likun Cai; Jun Chen; Ashish Khisti

cs.LGOct 19, 2026

Neural Video Coding under Limited Delay: Theoretical Optimality and System Design

Authors: Likun Cai, Jun Chen, Ashish Khisti

Organizations: University of Toronto · McMaster University

Abstract

Traditional neural video compression (NVC) methods predominantly follow strictly causal predictive-coding paradigm, treating zero-frame delay as a fixed constraint rather than delay as an explicit design parameter. From an information-theoretic perspective, allowing limited frame delay enables a codec to exploit correlations with future frames within the delay window, thereby reducing uncertainty that is inaccessible to strictly causal prediction and potentially improving the rate-distortion tradeoff. In this work, we formalize this intuition through a rigorous rate-distortion analysis builds on the clique factorization of Markov networks, and propose Graph-Factored Sequential Coding (GFSC), a principled framework for sequential video coding under limited delay. To our knowledge, we provide the first theoretical result showing that, for general $K$ -order Markov sources with positive distributions, a $K$ -frame delay is sufficient to attain joint-coding rate-distortion optimality over a nontrivial distortion region, a guarantee unavailable to zero-delay causal coding in general. In practice, we instantiate GFSC through lightweight neural modules, allowing existing causal predictive codecs to be augmented without modifying their core architectures. Experiments across multiple NVC models and datasets show that GFSC consistently improves rate-distortion performance over the corresponding causal counterparts. Under matched delay budgets, GFSC also achieves competitive performance compared with representative bidirectional codecs.

Figures & tables

Panel of Figure 1 — Figure 1: Left : GFSC- $1$ coding architecture for three frames. Middle : Rate-PSNR comparison between zero-delay DCVC and GFSC with delays $K=1,2,3$ on UVG. Right : Latency (top; lower the better) and BD-rate (bottom; lower the better) of GFSC and DCVC-B at matched delays, with zero-delay DCVC as the baseline. BD-rate is measured relative to zero-delay DCVC.

Panel of Figure 2 — Figure 2: Temporal-dependence analysis for UVG dataset. Left : Estimated information matrix for $32$ -frame UVG video clips. Right : Mean APC vs. Separation curve. APC denotes Absolute partial correlation. Separation denotes the distance between two frame positions, i.e. $|j-k|$ for $(X_{j},X_{k})$ .

Panel of Figure 3 — Figure 3: Second-order Markov structure illustration. Left : Maximal cliques and clique factorization. Middle : Second-order Markov network. Right : The information matrix $J$ for Gauss-Markov satisfying $J_{ij}=0$ for $|i-j|>2$ . “ $*$ ” denotes potentially nonzero entries.

Panel of Figure 4 — Figure 4: Left : Second-order Markov network (top), its augmentation by clique-wise auxiliary variables $Z_{j}$ (middle), and further augmentation by node-wise auxiliary variables $W_{\ell}$ (bottom). Right : GFSC architecture with $2$ -delay and optimal auxiliary variables constructed in Left .

Panel of Figure 5 — Figure 5: Left : A second-order GGM and its augmented representation within $\mathcal{D}^{\mathrm{G}}_{\epsilon}$ ; dashed arrows indicate sequential coding. Right : Edge-wise auxiliary construction by linear interpolation and additive noise, where $\bar{\omega}_{j,k}=1-\omega_{j,k}$ .

Panel of Figure 6 — Figure 6: Rate-PSNR comparison between causal and GFSC-enhanced models on UVG and MCL-JCV dataset. Columns show DCVC, VCT, and DHVC, respectively.

	UVG ( $1080$ p)			MCL-JCV ( $1080$ p)
Delay	DCVC	VCT	DHVC	DCVC	VCT	DHVC
$K=1$	-20.57	-28.38	-7.39	-10.90	-32.35	-13.35
$K=2$	-25.29	-48.00	-10.17	-17.32	-50.14	-14.65
$K=3$	-28.22	-55.78	-11.91	-20.30	-56.70	-16.07

Table 1: BD-rate (%) of GFSC-enhanced models relative to causal baselines.

Panel of Figure 7 — Figure 7: Rate-PSNR comparison under delay $K\leq 3$ on (a) UVG and (b) MCL-JCV. Colors identify model families. Line styles and markers distinguish delay settings.

Method	Delay	BD-rate	Enc. Latency	Dec. Latency	Memory	FLOPs	Params
Method	$K$	(%)	(s/frame)	(s/frame)	(GB)	(G/frame)	(M)
DCVC	0	0.0	0.52	0.44	5.54	4430.53	34.53
DCVC-L	0	-8.57	0.53	0.44	5.54	4447.51	39.21
DCVC-B	1	3.70	0.84	0.64	10.95	7153.37	55.38
	2	-18.64	1.14	0.89	10.95	8805.79	55.38
	3	-27.24	1.16	0.90	10.95	9632.01	55.38

Table 2: BD-rate (%) and computational complexity under matched delay budgets. Latency, memory, FLOPs, and model parameters are reported for DCVC, DCVC-GFSC, and DCVC-B.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: Illustration of three coding architectures for a four-frame source under $2$ -frame delay. Left : C-NC. Middle : NC-NC. Right : NC-C.

Panel of Figure 9 — Figure 9: Left : graphical illustration for causal predictive coding. Middle : graphical illustration for bidirectional coding. Right : graphical illustration for proposed GFSC- $1$ .

Panel of Figure 10 — Figure 10: Clique-wise augmentation of a second-order Markov network, where $U_{j}$ is associated with the maximal clique $(X_{j},X_{j+1},X_{j+2})$ .

Panel of Figure 11 — Figure 11: Augmented graphical model with node-wise and edge-wise auxiliary variables.

Panel of Figure 12 — Figure 12: Gaussian source $(X_{1},X_{2})$ with $\sigma^{2}_{1}=\sigma^{2}_{2}=1$ and $\rho=0.7$ . Left : rate-distortion $R(D)$ comparisons for symmetric distortion constraint $D$ . Interpolation coefficient $\omega_{1,2}=0.5$ for ISC. Middle : Rate gap between joint coding (ISC) and causal coding for symmetric distortion case. Right : rate-distortion $R(D_{2})$ comparisons for asymmetric distortion constraint with fixed $d_{1}=0.1,0.2$ , and $0.4$ upside down.

Panel of Figure 13 — Figure 13: Illustration of the delay induced by B-frame coding over an $N$ -length video clip.

Panel of Figure 14 — Figure 14: Simplified DCVC and GFSC architectures. GFSC introduces temporal joint modeling (TJM) modules to both motion and context branch, highlighted in orange and purple , respectively. One delayed frame case is shown. Additional delayed frames provide further inputs to these modules. The mappings $f_{m},f_{c}$ and $g_{m},g_{c}$ denote motion/context encoders and decoders, respectively.

Panel of Figure 15 — Figure 15: Temporal joint modeling modules for motion and context branches in DCVC architecture. Left : $\mathrm{TJM}_{m}$ concatenates motion features and their absolute difference as input. Right : $\mathrm{TJM}_{c}$ jointly updates the two context features using a spatial-temporal self attention module.

Panel of Figure 16 — Figure 16: Left : Rate-PSNR curves on LIVE-HDR 4K. Right : BD-rate relative to causal DCVC. Negative values indicate rate savings.

Neural Video Compression

Neural video compression uses learned representations and generative models to reduce bitrate while preserving visual quality and temporal coherence.

See all 53 papers

Ultra-Fast Neural Video Compression
Jun 3, 2026Jiahao Li, Wenxuan Xie, Zhaoyang Jia +4Neural Video Compression Video Compression
Microsoft Research Asia · University of Science and Technology of China · *Done during Zhaoyang Jia’s internship at Microsoft Research Asia.
Real-Time Neural Video Compression with Unified Intra and Inter Coding
Oct 16, 2025Hui Xiang, Yifan Bian, Li Li +3Learned Image Compression Neural Video Compression
University of Science and Technology of China · Tencent Shannon Lab
DCVC-MB: Neural B-Frame Video Compression using State Space Models
Jul 15, 2026Arjun Arora, Calvin-Khang Ta, Carlos Restrepo-Galeano +7Predictive Coding Neural Video Compression
Dolby Laboratories · Voia · University of Delaware +1

Jun 3, 2026J/K move · Enter open · S save

Ultra-Fast Neural Video Compression

Jiahao Li, Wenxuan Xie, Zhaoyang Jia, Bin Li+3

Microsoft Research Asia · University of Science and Technology of China · *Done during Zhaoyang Jia’s internship at Microsoft Research Asia.

Neural Video Compression Video Compression Entropy Coding

Open paper AskPDF arXiv

While neural video codecs (NVCs) have demonstrated superior compression ratio, their prohibitive computational complexity remains a critical barrier to real-world deployment. This paper introduces a chunk-based coding framework designed to significantly improve the rate-distortion-complexity trade-off. Instead of processing frames sequentially, our approach encodes a chunk of multiple frames into a single compact latent representation and decodes them simultaneously. This is enabled by cross-frame interaction modules for joint spatial-temporal modeling and frame-specific decoders for parallel reconstruction. This paradigm not only dramatically enhances coding throughput but also facilitates more effective modeling of long-term temporal correlations. To further boost speed, we propose a streamlined entropy coding mechanism that consolidates bit-stream interactions into a single step, substantially reducing decoding overhead. Building on these innovations, we present DCVC-UF (Ultra-Fast), a new NVC that sets a new SOTA in performance. Our experiments show that DCVC-UF can achieve ultra-fast encoding and decoding speeds, significantly outperforming previous leading codecs. DCVC-UF serves as a notable landmark in the journey of NVC evolution. The code is at https://github.com/microsoft/DCVC.

Abstract

Neural Video Compression

Explore similar work

Ultra-Fast Neural Video Compression

Real-Time Neural Video Compression with Unified Intra and Inter Coding

DCVC-MB: Neural B-Frame Video Compression using State Space Models