CWRNN-INVR: A Coupled WarpRNN based Implicit Neural Video Representation
Authors: Yiyang Li, Yanbo Gao, Shuai Li, Zhenyu Du, Jinglin Zhang, Hui Yuan, Mao Ye, Xingyu Gao
Organizations: School of Control Science and Engineering, Shandong University, and Key Laboratory of Machine Intelligence and System Control, Ministry of Education, Jinan 250100, China. · School of Software, Shandong University, Jinan 250100, China, and also with Shandong University-WeiHai Research Institute of Industrial Technology, Weihai 264209, China. · University of Electronic Science and Technology of China, Sichuan, China. · Institute of Microelectronics, Chinese Academy of Sciences, Beijing 100029, China
Implicit Neural Video Representation (INVR) has emerged as a novel approach for video representation and compression, using learnable grids and neural networks. Existing methods focus on developing new grid structures efficient for latent representation and neural network architectures with large representation capability, lacking the study on their roles in video representation. In this paper, the difference between INVR based on neural network and INVR based on grid is first investigated from the perspective of video information composition to specify their own advantages, i.e., neural network for general structure while grid for specific detail. Accordingly, an INVR based on mixed neural network and residual grid framework is proposed, where the neural network is used to represent the regular and structured information and the residual grid is used to represent the remaining irregular information in a video. A Coupled WarpRNN-based multi-scale motion representation and compensation module is specifically designed to explicitly represent the regular and structured information, thus terming our method as CWRNN-INVR. For the irregular information, a mixed residual grid is learned where the irregular appearance and motion information are represented together. The mixed residual grid can be combined with the coupled WarpRNN in a way that allows for network reuse. Experiments show that our method achieves the best reconstruction results compared with the existing methods, with an average PSNR of 33.73 dB on the UVG dataset under the 3M model and outperforms existing INVR methods in other downstream tasks. The code can be found at https://github.com/yiyang-sdu/CWRNN-INVR.git}{https://github.com/yiyang-sdu/CWRNN-INVR.git.
Figures & tables
Fig. 1: Illustration of using different combinations of grids and neural networks (only NeRVBlocks) while keeping the size of the total model constant, for video reconstruction on the Big Buck Bunny dataset. Performance varies significantly with different combinations, indicating that grids and networks each have their own advantages in terms of feature representation.
Fig. 2: Illustration of the spatial and temporal regular and irregular information in a video. The top shows the comparison of the reconstruction results using neural network only (without grid) and with grid (under the same number of parameters in total) on the UVG Beauty video. The residual map is obtained by subtracting the two results. The bottom is the reconstruction results on a spliced video that n frames of the Beauty are inserted between the 2N frames of a blooming lily video. The residual map in the top illustrates the spatial irregular information, mainly on the hair which is hard to be represented and not structured among video frames. The bottom results using the spliced video clearly show the temporal irregular information, which is difficult to be represented by a neural network but easily represented by a grid.
Fig. 3: Left: Framework Overview. The proposed Coupled WarpRNN based multi-scale motion representation and compensation module is used to learn the regular and structured temporal information and the following NeRVBlocks are used to spatially reconstruct the frame. It can work independently without grid or any input based on its recurrent architecture. The Mixed Residual Grid is used to learn the remaining irregular information and added to the coupled WarpRNN for network reuse. Right: Structure of the proposed Coupled WarpRNN based multi-scale motion representation and compensation module.
Fig. 4: Illustration of the WarpRNN module, with a learned warping function.
Fig. 5: Illustration of the learned initial hidden state. The left column shows the ground-truth first frame. The right column shows the reconstruction result of the learnable initial hidden state (h0global) . Note that no supervision is applied directly on the learning of (h0global) . The information learned in (h0global) can be treated as an anchor frame that retains the main structure of a video and is used for predicting the following frames in the moving area. It also validates the effectiveness of the proposed method in learning the features of video frames within the hidden state of the RNN structure.
Methods
Beauty
Bosph
Honey
Jockey
Ready
Shake
Yacht
avg
NeRV [ 2 ]
33.33
33.34
38.79
28.97
23.89
33.89
27.05
31.32
FFNeRV [ 15 ]
33.40
33.08
38.81
31.71
24.99
33.93
27.38
31.90
ENeRV [ 7 ]
33.17
33.69
37.63
31.63
25.2
34.39
28.42
32.02
DNeRV [ 13 ]
33.16
32.96
38.43
31.08
24.76
33.71
27.30
31.63
HNeRV [ 3 ]
33.88
35.02
39.41
31.69
25.72
34.95
29.09
32.82
DSNeRV [ 43 ]
33.97
35.22
39.56
32.86
27.10
35.04
29.40
33.31
TABLE I: Video reconstruction results on UVG at resolution 960×1920, in terms of PSNR(dB).
Methods
b-swan
b-trees
boat
b-dance
camel
c-round
c-shadow
cows
dance
dog
avg
NeRV [ 2 ]
25.04
25.22
30.25
25.78
23.69
24.08
25.29
22.44
25.61
27.15
25.46
FFNeRV [ 15 ]
24.89
24.56
27.82
24.73
22.84
22.32
24.58
21.39
23.95
26.53
24.36
ENeRV [ 7 ]
25.64
24.92
29.16
24.19
22.88
22.85
26.13
21.69
24.15
27.66
24.93
DNeRV [ 13 ]
29.84
28.73
30.52
26.58
26.24
28.50
28.88
24.44
28.42
30.64
28.28
HNeRV [ 3 ]
29.23
28.67
32.27
31.39
25.93
28.72
31.21
24.67
28.43
30.72
29.12
DSNeRV [ 43 ]
32.55
29.76
34.39
32.21
27.26
29.48
35.88
25.08
28.79
33.29
30.87
TABLE II: Video reconstruction results on DAVIS at resolution 960×1920, in terms of PSNR(dB).
Fig. 6: R-D curve comparison on UVG dataset.
Fig. 7: Example video reconstruction results on UVG and DAVIS. (Top) Jockey. (Bottom) Black swan.
Methods
BD-rate(%)
HEVC [ 21 ]
0
HNeRV [ 3 ]
-24.99
DSNeRV [ 43 ]
-31.20
Ours
-54.24
TABLE III: Compression result comparison of the proposed method against different methods on UVG dataset, in terms of BD-rate.
Methods
Encoding time
Memory usage
NeRV [ 2 ]
3h53min
15.50G
HNeRV [ 3 ]
5h46min
11.49G
DNeRV [ 13 ]
6h11min
5.91G
ENeRV [ 7 ]
3h10min
4.47G
FFNeRV [ 15 ]
7h43min
25.76G
Ours
4h10min
14.22G
TABLE IV: Results of the Encoding Time and Memory Usage Comparison.
Methods
Decoding Speed
HEVC [ 21 ] (HM)
16.56
HEVC [ 21 ] (x265)
152.00
NeRV [ 2 ]
60.08
HNeRV [ 3 ]
62.27
DNeRV [ 13 ]
52.20
ENeRV [ 7 ]
50.60
TABLE V: Results of decoding speed, in terms of FPS.
Model size
0.75M
1.5M
3M
NeRV [ 2 ]
27.09
28.67
31.72
HNeRV [ 3 ]
30.16
32.72
34.61
Ours
32.09
35.07
37.85
TABLE VI: Results with varying model sizes using the Big Buck Bunny dataset, in terms of PSNR(dB).
Single
Coupled
MRG
WarpRNN
WarpRNN
Bunny
Beauty
Honey
v1
✓
37.28
33.61
38.66
v2
✓
✓
37.69
33.50
39.32
v3
✓
✓
37.85
34.69
40.47
TABLE VII: Ablation study on different modules, in terms of PSNR(dB).
Fig. 8: Example qualitative result comparison on the mixed residual grid.