A Rate-Distortion Optimized Skip Mode Coding Framework for Neural Video Compression in AVS
Authors: Yuntian Jiang, Shuai Li, Yanbo Gao, Jinglin Zhang, Hui Yuan, Mao Ye
Organizations: School of Control Science and Engineering, Shandong University, Jinan 250100, China. · School of Software, Shandong University, Jinan, China, and also with Shandong University-WeiHai Research Institute of Industrial Technology, Weihai 264209, China. · University of Electronic Science and Technology of China, Sichuan, China.
With the development of deep learning, neural video compression (NVC) is gaining increasing interest and is currently under standardization in the AVS (Audio Video Coding Standard). Skip mode coding, a crucial component in the traditional video compression, is also adopted in AVS NVC. Although the skip mode takes a large portion in encoding the motion and residual features in the NVC, there is a lack of investigation on the rate distortion performance of the skip mode coding. In this paper, we first thoroughly investigate the rate and distortion distribution in the skip mode coding within a spatial frame and under different temporal reference qualities, and demonstrate that distortion distribution changes spatially and temporally. Therefore, a rate-distortion optimized skip mode coding (RDOSkip) framework is proposed, where the rate and distortion in the skip mode coding is formulated based on a statistical distribution predicted by the hyperprior and the Bayes' theorem. An adaptive skip mode decision method based on spatial-temporal joint RD optimization is developed for the skip mode coding among channels and frames in a group of pictures. It addresses the limitation of the rigid fixed-distortion distribution assumption in skip mode coding within the existing methods that rely on a fixed skip threshold. The temporal skip mode decision method has been adopted in the ongoing AVS NVC standard and incorporated into the reference software. The proposed RDOSkip can be directly used in the inference without any training and with negligible complexity increase. Experimental results further demonstrate that the proposed RDOSkip can improve the performance by 0.90% and 1.29% on AVS and HEVC datasets, respectively, in terms of BD-rate saving.
Figures & tables
Fig. 1 : The skip ratio of motion and residual features, on the BQTerrace sequence. The skip ratio refers to the percentage of skipped elements in the latent feature using the AVS-EEM.
Fig. 2 : Illustration of the MSE variation in the image domain corresponding to feature loss ∣Δy∣ in the feature domain, on the BQTerrace, λ=2048 . (a) for different feature elements under different variances, and (b) for different channels. (a) illustrates that feature elements with the same variance can also lead to different MSE losses while (b) illustrates different channels exhibit distinct sensitivities to changes in Δy , leading to varying magnitudes of distortion responses.
Fig. 3 : The impact of the current frame with different qualities of the reference frame, on BQTerrace, λ=2048 . Idx is the index of the current frame in the sequence. The ref λ denotes the quality of the reference frame. The MSE loss is calculated frame-wise.
Fig. 4 : Overview of the neural video compression based on our proposed adaptive skip module.
Fig. 5 : Illustrates the architecture of the entropy model integrated with our skip method. λref and λc represent the quality of reference frame and current frame, respectively. The spatial module utilizes channel variance to calculate the channel-wise skip threshold, while the temporal method leverages the hierarchical quality structure to apply a temporal threshold.
Fig. 6 : Shows the variation of the mean squared error (MSE) with respect to ∣Δy∣ under different channels, using the scale information for each channel.
Sequence
Default Skip
w/o Skip
w M8789
w RDOSkip
Y
U
V
Y
U
V
Y
U
V
Y
U
V
HUD4K
Tango2
0
0
0
3.09
32.38
11.23
-0.56
-1.66
-2.07
-1.14
-2.37
-3.50
ParkRunning3
0
0
0
10.17
-8.08
0.74
-0.01
-3.19
-2.73
0.04
-4.88
-4.54
Campfire
0
0
0
12.73
-3.9
18.62
-0.35
1.8
-1.2
-0.57
0.60
-2.82
DaylightRoad2
0
0
0
6.09
9.12
0.87
-1.25
-7.22
-6.7
-1.81
-8.22
-8.25
1080p
Cactus
0
0
0
13.02
15.8
18.11
0.02
-4.51
-4.15
-0.19
-5.59
-5.78
TABLE I : Performance evaluation (BD-Rate/%) of the proposed RDOSkip method under AVS CTC, compared with the default skip mode in AVS-EEM
Sequence
Default Skip
w/o Skip
w M8789
w RDOSkip
Y
U
V
Y
U
V
Y
U
V
Y
U
V
AVS CTC
HUD4K
0
0
0
8.02
7.38
7.86
-0.54
-2.57
-3.18
-0.98
-3.20
-3.56
1080p
0
0
0
5.50
8.99
8.94
-0.51
-3.73
-3.60
-1.11
-3.97
-5.40
720p
0
0
0
17.56
27.83
23.60
-0.57
-6.46
-6.05
-0.69
-6.38
-6.02
Average
0
0
0
10.36
14.73
13.47
-0.54
-4.25
-4.27
-0.90
-5.51
-5.67
HEVC
HEVC B
0
0
0
-1.64
0.23
-0.03
-1.40
-1.06
-1.11
-1.69
-1.23
-1.29
TABLE II : Performance evaluation (BD-Rate/%) of the proposed RDOSkip method under both AVS and HEVC test sequences, compared with the default skip mode in AVS-EEM
Sequence
2048
512
128
64
AVS CTC
HUD4K
86.96%
97.15%
97.16%
91.15%
1080p
83.58%
95.05%
96.63%
94.83%
720p
88.09%
92.33%
93.25%
86.67%
Average
86.21%
94.84%
95.68%
90.88%
HEVC
HEVC B
93.37%
94.4%
95.88%
95.77%
HEVC C
94.46%
97.17%
98.04%
93.84%
TABLE III : Percentage of frames with less RD loss than the default skip mode in AVS-EEM under both AVS and HEVC test sequences
Fig. 7 : Frame-by-frame loss comparison between the baseline model (the default skip mode in AVS-EEM) and our method. Without high quality P-frame, on AVS CTC dataset(Left) and HEVC dataset(Right), λ=516 . It demonstrates that our approach yields improvements on the majority of frames.
Fig. 8 : The figure presents the high quality P-frame loss comparison between the baseline model and our method. On AVS CTC dataset(Left) and HEVC dataset(Right), λ=516 . It demonstrates that our approach yields improvements on the majority of high quality frames.
Model
T-skip
S-skip
BD-Rate (%)
AVS CTC
M0
×
×
0
M1
✓
×
-0.54
M2
✓
✓
-0.90
HEVC
M0
×
×
0
M1
✓
×
-1.11
M2
✓
✓
-1.29
TABLE IV : Performance evaluation (BD-Rate/%) of different modules in the proposed method under both AVS and HEVC test sequences on the Y channel
Model
T-skip
S-skip
BD-Rate (%)
AVS CTC
M0
×
×
0
M1
✓
×
-0.36
M2
✓
✓
-0.68
HEVC
M0
×
×
0
M1
✓
×
-0.82
M2
✓
✓
-0.85
TABLE V : Performance evaluation (BD-Rate/%) of different modules in the proposed method under different GOP settings on the Y channel
Fig. 9 : Example skip mode masks using different methods, evaluated on the 2nd frame of the sequence BQTerrace. Red points indicates elements skipped while blue points indicates those non-skipped.
Sequence
Default Skip
w/o Skip
w M8789
w RDOSkip
Y
U
V
Y
U
V
Y
U
V
Y
U
V
AVS CTC
HUD4K
0
0
0
6.38
-2.94
-0.42
-0.18
0.73
0.45
-0.85
1.55
1.35
1080p
0
0
0
3.17
0.84
0.02
-0.19
-0.20
0.31
-0.76
-0.27
0.16
720p
0
0
0
9.31
10.80
8.38
-0.74
-0.89
-0.96
-2.75
-1.60
-2.01
Average
0
0
0
6.29
2.90
2.66
-0.37
-0.12
-0.07
-1.45
-0.11
-0.17
HEVC
HEVC B
0
0
0
4.16
5.34
5.14
-0.85
-1.18
-1.12
-1.25
-1.75
-1.67
TABLE VI : Performance evaluation (BD-Rate/%) of the proposed RDOSkip method under both AVS and HEVC test sequences, with intra period setting IP=32, compared with the default skip mode in AVS-EEM
Dataset
Encoding (FPS)
Decoding (FPS)
Baseline
RDOSkip
Baseline
RDOSkip
AVS CTC 4K
2.95
2.94
2.89
2.99
AVS CTC 1080p
11.47
11.53
9.94
9.59
AVS CTC 720p
18.90
19.14
12.40
12.80
TABLE VII : Encoding and decoding complexity evaluation in terms of FPS (frames per second) on the AVS CTC Dataset in AVS-EEM