A Rate-Distortion Optimized Skip Mode Coding Framework for Neural Video Compression in AVS
Authors: Yuntian Jiang, Shuai Li, Yanbo Gao, Jinglin Zhang, Hui Yuan, Mao Ye
Organizations: School of Control Science and Engineering, Shandong University, Jinan 250100, China. · School of Software, Shandong University, Jinan, China, and also with Shandong University-WeiHai Research Institute of Industrial Technology, Weihai 264209, China. · University of Electronic Science and Technology of China, Sichuan, China.
With the development of deep learning, neural video compression (NVC) is gaining increasing interest and is currently under standardization in the AVS (Audio Video Coding Standard). Skip mode coding, a crucial component in the traditional video compression, is also adopted in AVS NVC. Although the skip mode takes a large portion in encoding the motion and residual features in the NVC, there is a lack of investigation on the rate distortion performance of the skip mode coding. In this paper, we first thoroughly investigate the rate and distortion distribution in the skip mode coding within a spatial frame and under different temporal reference qualities, and demonstrate that distortion distribution changes spatially and temporally. Therefore, a rate-distortion optimized skip mode coding (RDOSkip) framework is proposed, where the rate and distortion in the skip mode coding is formulated based on a statistical distribution predicted by the hyperprior and the Bayes' theorem. An adaptive skip mode decision method based on spatial-temporal joint RD optimization is developed for the skip mode coding among channels and frames in a group of pictures. It addresses the limitation of the rigid fixed-distortion distribution assumption in skip mode coding within the existing methods that rely on a fixed skip threshold. The temporal skip mode decision method has been adopted in the ongoing AVS NVC standard and incorporated into the reference software. The proposed RDOSkip can be directly used in the inference without any training and with negligible complexity increase. Experimental results further demonstrate that the proposed RDOSkip can improve the performance by 0.90% and 1.29% on AVS and HEVC datasets, respectively, in terms of BD-rate saving.
Figures & tables
Fig. 1 : The skip ratio of motion and residual features, on the BQTerrace sequence. The skip ratio refers to the percentage of skipped elements in the latent feature using the AVS-EEM.
Fig. 2 : Illustration of the MSE variation in the image domain corresponding to feature loss ∣Δy∣ in the feature domain, on the BQTerrace, λ=2048 . (a) for different feature elements under different variances, and (b) for different channels. (a) illustrates that feature elements with the same variance can also lead to different MSE losses while (b) illustrates different channels exhibit distinct sensitivities to changes in Δy , leading to varying magnitudes of distortion responses.
Fig. 3 : The impact of the current frame with different qualities of the reference frame, on BQTerrace, λ=2048 . Idx is the index of the current frame in the sequence. The ref λ denotes the quality of the reference frame. The MSE loss is calculated frame-wise.
Fig. 4 : Overview of the neural video compression based on our proposed adaptive skip module.
Fig. 5 : Illustrates the architecture of the entropy model integrated with our skip method. λref and λc represent the quality of reference frame and current frame, respectively. The spatial module utilizes channel variance to calculate the channel-wise skip threshold, while the temporal method leverages the hierarchical quality structure to apply a temporal threshold.
Fig. 6 : Shows the variation of the mean squared error (MSE) with respect to ∣Δy∣ under different channels, using the scale information for each channel.
Sequence
Default Skip
w/o Skip
w M8789
w RDOSkip
Y
U
V
Y
U
V
Y
U
V
Y
U
V
HUD4K
Tango2
0
0
0
3.09
32.38
11.23
-0.56
-1.66
-2.07
-1.14
-2.37
-3.50
ParkRunning3
0
0
0
10.17
-8.08
0.74
-0.01
-3.19
-2.73
0.04
-4.88
-4.54
Campfire
0
0
0
12.73
-3.9
18.62
-0.35
1.8
-1.2
-0.57
0.60
-2.82
DaylightRoad2
0
0
0
6.09
9.12
0.87
-1.25
-7.22
-6.7
-1.81
-8.22
-8.25
1080p
Cactus
0
0
0
13.02
15.8
18.11
0.02
-4.51
-4.15
-0.19
-5.59
-5.78
TABLE I : Performance evaluation (BD-Rate/%) of the proposed RDOSkip method under AVS CTC, compared with the default skip mode in AVS-EEM
Sequence
Default Skip
w/o Skip
w M8789
w RDOSkip
Y
U
V
Y
U
V
Y
U
V
Y
U
V
AVS CTC
HUD4K
0
0
0
8.02
7.38
7.86
-0.54
-2.57
-3.18
-0.98
-3.20
-3.56
1080p
0
0
0
5.50
8.99
8.94
-0.51
-3.73
-3.60
-1.11
-3.97
-5.40
720p
0
0
0
17.56
27.83
23.60
-0.57
-6.46
-6.05
-0.69
-6.38
-6.02
Average
0
0
0
10.36
14.73
13.47
-0.54
-4.25
-4.27
-0.90
-5.51
-5.67
HEVC
HEVC B
0
0
0
-1.64
0.23
-0.03
-1.40
-1.06
-1.11
-1.69
-1.23
-1.29
TABLE II : Performance evaluation (BD-Rate/%) of the proposed RDOSkip method under both AVS and HEVC test sequences, compared with the default skip mode in AVS-EEM
Sequence
2048
512
128
64
AVS CTC
HUD4K
86.96%
97.15%
97.16%
91.15%
1080p
83.58%
95.05%
96.63%
94.83%
720p
88.09%
92.33%
93.25%
86.67%
Average
86.21%
94.84%
95.68%
90.88%
HEVC
HEVC B
93.37%
94.4%
95.88%
95.77%
HEVC C
94.46%
97.17%
98.04%
93.84%
TABLE III : Percentage of frames with less RD loss than the default skip mode in AVS-EEM under both AVS and HEVC test sequences
Fig. 7 : Frame-by-frame loss comparison between the baseline model (the default skip mode in AVS-EEM) and our method. Without high quality P-frame, on AVS CTC dataset(Left) and HEVC dataset(Right), λ=516 . It demonstrates that our approach yields improvements on the majority of frames.
Fig. 8 : The figure presents the high quality P-frame loss comparison between the baseline model and our method. On AVS CTC dataset(Left) and HEVC dataset(Right), λ=516 . It demonstrates that our approach yields improvements on the majority of high quality frames.
Model
T-skip
S-skip
BD-Rate (%)
AVS CTC
M0
×
×
0
M1
✓
×
-0.54
M2
✓
✓
-0.90
HEVC
M0
×
×
0
M1
✓
×
-1.11
M2
✓
✓
-1.29
TABLE IV : Performance evaluation (BD-Rate/%) of different modules in the proposed method under both AVS and HEVC test sequences on the Y channel
Model
T-skip
S-skip
BD-Rate (%)
AVS CTC
M0
×
×
0
M1
✓
×
-0.36
M2
✓
✓
-0.68
HEVC
M0
×
×
0
M1
✓
×
-0.82
M2
✓
✓
-0.85
TABLE V : Performance evaluation (BD-Rate/%) of different modules in the proposed method under different GOP settings on the Y channel
Fig. 9 : Example skip mode masks using different methods, evaluated on the 2nd frame of the sequence BQTerrace. Red points indicates elements skipped while blue points indicates those non-skipped.
Sequence
Default Skip
w/o Skip
w M8789
w RDOSkip
Y
U
V
Y
U
V
Y
U
V
Y
U
V
AVS CTC
HUD4K
0
0
0
6.38
-2.94
-0.42
-0.18
0.73
0.45
-0.85
1.55
1.35
1080p
0
0
0
3.17
0.84
0.02
-0.19
-0.20
0.31
-0.76
-0.27
0.16
720p
0
0
0
9.31
10.80
8.38
-0.74
-0.89
-0.96
-2.75
-1.60
-2.01
Average
0
0
0
6.29
2.90
2.66
-0.37
-0.12
-0.07
-1.45
-0.11
-0.17
HEVC
HEVC B
0
0
0
4.16
5.34
5.14
-0.85
-1.18
-1.12
-1.25
-1.75
-1.67
TABLE VI : Performance evaluation (BD-Rate/%) of the proposed RDOSkip method under both AVS and HEVC test sequences, with intra period setting IP=32, compared with the default skip mode in AVS-EEM
Dataset
Encoding (FPS)
Decoding (FPS)
Baseline
RDOSkip
Baseline
RDOSkip
AVS CTC 4K
2.95
2.94
2.89
2.99
AVS CTC 1080p
11.47
11.53
9.94
9.59
AVS CTC 720p
18.90
19.14
12.40
12.80
TABLE VII : Encoding and decoding complexity evaluation in terms of FPS (frames per second) on the AVS CTC Dataset in AVS-EEM
Traditional neural video compression (NVC) methods predominantly follow strictly causal predictive-coding paradigm, treating zero-frame delay as a fixed constraint rather than delay as an explicit design parameter. From an information-theoretic perspective, allowing limited frame delay enables a codec to exploit correlations with future frames within the delay window, thereby reducing uncertainty that is inaccessible to strictly causal prediction and potentially improving the rate-distortion tradeoff. In this work, we formalize this intuition through a rigorous rate-distortion analysis builds on the clique factorization of Markov networks, and propose Graph-Factored Sequential Coding (GFSC), a principled framework for sequential video coding under limited delay. To our knowledge, we provide the first theoretical result showing that, for general K-order Markov sources with positive distributions, a K-frame delay is sufficient to attain joint-coding rate-distortion optimality over a nontrivial distortion region, a guarantee unavailable to zero-delay causal coding in general. In practice, we instantiate GFSC through lightweight neural modules, allowing existing causal predictive codecs to be augmented without modifying their core architectures. Experiments across multiple NVC models and datasets show that GFSC consistently improves rate-distortion performance over the corresponding causal counterparts. Under matched delay budgets, GFSC also achieves competitive performance compared with representative bidirectional codecs.
Neural video compression (NVC) technologies have advanced rapidly in recent years, yielding state-of-the-art schemes such as DCVC-RT that offer superior compression efficiency to H.266/VVC and real-time encoding/decoding capabilities. Nonetheless, existing NVC schemes have several limitations, including inefficiency in dealing with disocclusion and new content, interframe error propagation and accumulation, among others. To eliminate these limitations, we borrow the idea from classic video coding schemes, which allow intra coding within inter-coded frames. With the intra coding tool enabled, disocclusion and new content are properly handled, and interframe error propagation is naturally intercepted without the need for manual refresh mechanisms. We present an NVC framework with unified intra and inter coding, where every frame is processed by a single model that is trained to perform intra/inter coding adaptively. Moreover, we propose a simultaneous two-frame compression design to exploit interframe redundancy not only forwardly but also backwardly. Experimental results show that our scheme outperforms DCVC-RT by an average of 12.1% BD-rate reduction, delivers more stable bitrate and quality per frame, and retains real-time encoding/decoding performances. Code and models will be released.
Hui Xiang, Yifan Bian, Li Li +3
University of Science and Technology of China · Tencent Shannon Lab
While neural video codecs (NVCs) have demonstrated superior compression ratio, their prohibitive computational complexity remains a critical barrier to real-world deployment. This paper introduces a chunk-based coding framework designed to significantly improve the rate-distortion-complexity trade-off. Instead of processing frames sequentially, our approach encodes a chunk of multiple frames into a single compact latent representation and decodes them simultaneously. This is enabled by cross-frame interaction modules for joint spatial-temporal modeling and frame-specific decoders for parallel reconstruction. This paradigm not only dramatically enhances coding throughput but also facilitates more effective modeling of long-term temporal correlations. To further boost speed, we propose a streamlined entropy coding mechanism that consolidates bit-stream interactions into a single step, substantially reducing decoding overhead. Building on these innovations, we present DCVC-UF (Ultra-Fast), a new NVC that sets a new SOTA in performance. Our experiments show that DCVC-UF can achieve ultra-fast encoding and decoding speeds, significantly outperforming previous leading codecs. DCVC-UF serves as a notable landmark in the journey of NVC evolution. The code is at https://github.com/microsoft/DCVC.
Jiahao Li, Wenxuan Xie, Zhaoyang Jia +4
Microsoft Research Asia · University of Science and Technology of China · *Done during Zhaoyang Jia’s internship at Microsoft Research Asia.