CWRNN-INVR: A Coupled WarpRNN based Implicit Neural Video Representation
Authors: Yiyang Li, Yanbo Gao, Shuai Li, Zhenyu Du, Jinglin Zhang, Hui Yuan, Mao Ye, Xingyu Gao
Organizations: School of Control Science and Engineering, Shandong University, and Key Laboratory of Machine Intelligence and System Control, Ministry of Education, Jinan 250100, China. · School of Software, Shandong University, Jinan 250100, China, and also with Shandong University-WeiHai Research Institute of Industrial Technology, Weihai 264209, China. · University of Electronic Science and Technology of China, Sichuan, China. · Institute of Microelectronics, Chinese Academy of Sciences, Beijing 100029, China
Implicit Neural Video Representation (INVR) has emerged as a novel approach for video representation and compression, using learnable grids and neural networks. Existing methods focus on developing new grid structures efficient for latent representation and neural network architectures with large representation capability, lacking the study on their roles in video representation. In this paper, the difference between INVR based on neural network and INVR based on grid is first investigated from the perspective of video information composition to specify their own advantages, i.e., neural network for general structure while grid for specific detail. Accordingly, an INVR based on mixed neural network and residual grid framework is proposed, where the neural network is used to represent the regular and structured information and the residual grid is used to represent the remaining irregular information in a video. A Coupled WarpRNN-based multi-scale motion representation and compensation module is specifically designed to explicitly represent the regular and structured information, thus terming our method as CWRNN-INVR. For the irregular information, a mixed residual grid is learned where the irregular appearance and motion information are represented together. The mixed residual grid can be combined with the coupled WarpRNN in a way that allows for network reuse. Experiments show that our method achieves the best reconstruction results compared with the existing methods, with an average PSNR of 33.73 dB on the UVG dataset under the 3M model and outperforms existing INVR methods in other downstream tasks. The code can be found at https://github.com/yiyang-sdu/CWRNN-INVR.git}{https://github.com/yiyang-sdu/CWRNN-INVR.git.
Figures & tables
Fig. 1: Illustration of using different combinations of grids and neural networks (only NeRVBlocks) while keeping the size of the total model constant, for video reconstruction on the Big Buck Bunny dataset. Performance varies significantly with different combinations, indicating that grids and networks each have their own advantages in terms of feature representation.
Fig. 2: Illustration of the spatial and temporal regular and irregular information in a video. The top shows the comparison of the reconstruction results using neural network only (without grid) and with grid (under the same number of parameters in total) on the UVG Beauty video. The residual map is obtained by subtracting the two results. The bottom is the reconstruction results on a spliced video that n frames of the Beauty are inserted between the 2N frames of a blooming lily video. The residual map in the top illustrates the spatial irregular information, mainly on the hair which is hard to be represented and not structured among video frames. The bottom results using the spliced video clearly show the temporal irregular information, which is difficult to be represented by a neural network but easily represented by a grid.
Fig. 3: Left: Framework Overview. The proposed Coupled WarpRNN based multi-scale motion representation and compensation module is used to learn the regular and structured temporal information and the following NeRVBlocks are used to spatially reconstruct the frame. It can work independently without grid or any input based on its recurrent architecture. The Mixed Residual Grid is used to learn the remaining irregular information and added to the coupled WarpRNN for network reuse. Right: Structure of the proposed Coupled WarpRNN based multi-scale motion representation and compensation module.
Fig. 4: Illustration of the WarpRNN module, with a learned warping function.
Fig. 5: Illustration of the learned initial hidden state. The left column shows the ground-truth first frame. The right column shows the reconstruction result of the learnable initial hidden state (h0global) . Note that no supervision is applied directly on the learning of (h0global) . The information learned in (h0global) can be treated as an anchor frame that retains the main structure of a video and is used for predicting the following frames in the moving area. It also validates the effectiveness of the proposed method in learning the features of video frames within the hidden state of the RNN structure.
Methods
Beauty
Bosph
Honey
Jockey
Ready
Shake
Yacht
avg
NeRV [ 2 ]
33.33
33.34
38.79
28.97
23.89
33.89
27.05
31.32
FFNeRV [ 15 ]
33.40
33.08
38.81
31.71
24.99
33.93
27.38
31.90
ENeRV [ 7 ]
33.17
33.69
37.63
31.63
25.2
34.39
28.42
32.02
DNeRV [ 13 ]
33.16
32.96
38.43
31.08
24.76
33.71
27.30
31.63
HNeRV [ 3 ]
33.88
35.02
39.41
31.69
25.72
34.95
29.09
32.82
DSNeRV [ 43 ]
33.97
35.22
39.56
32.86
27.10
35.04
29.40
33.31
TABLE I: Video reconstruction results on UVG at resolution 960×1920, in terms of PSNR(dB).
Methods
b-swan
b-trees
boat
b-dance
camel
c-round
c-shadow
cows
dance
dog
avg
NeRV [ 2 ]
25.04
25.22
30.25
25.78
23.69
24.08
25.29
22.44
25.61
27.15
25.46
FFNeRV [ 15 ]
24.89
24.56
27.82
24.73
22.84
22.32
24.58
21.39
23.95
26.53
24.36
ENeRV [ 7 ]
25.64
24.92
29.16
24.19
22.88
22.85
26.13
21.69
24.15
27.66
24.93
DNeRV [ 13 ]
29.84
28.73
30.52
26.58
26.24
28.50
28.88
24.44
28.42
30.64
28.28
HNeRV [ 3 ]
29.23
28.67
32.27
31.39
25.93
28.72
31.21
24.67
28.43
30.72
29.12
DSNeRV [ 43 ]
32.55
29.76
34.39
32.21
27.26
29.48
35.88
25.08
28.79
33.29
30.87
TABLE II: Video reconstruction results on DAVIS at resolution 960×1920, in terms of PSNR(dB).
Fig. 6: R-D curve comparison on UVG dataset.
Fig. 7: Example video reconstruction results on UVG and DAVIS. (Top) Jockey. (Bottom) Black swan.
Methods
BD-rate(%)
HEVC [ 21 ]
0
HNeRV [ 3 ]
-24.99
DSNeRV [ 43 ]
-31.20
Ours
-54.24
TABLE III: Compression result comparison of the proposed method against different methods on UVG dataset, in terms of BD-rate.
Methods
Encoding time
Memory usage
NeRV [ 2 ]
3h53min
15.50G
HNeRV [ 3 ]
5h46min
11.49G
DNeRV [ 13 ]
6h11min
5.91G
ENeRV [ 7 ]
3h10min
4.47G
FFNeRV [ 15 ]
7h43min
25.76G
Ours
4h10min
14.22G
TABLE IV: Results of the Encoding Time and Memory Usage Comparison.
Methods
Decoding Speed
HEVC [ 21 ] (HM)
16.56
HEVC [ 21 ] (x265)
152.00
NeRV [ 2 ]
60.08
HNeRV [ 3 ]
62.27
DNeRV [ 13 ]
52.20
ENeRV [ 7 ]
50.60
TABLE V: Results of decoding speed, in terms of FPS.
Model size
0.75M
1.5M
3M
NeRV [ 2 ]
27.09
28.67
31.72
HNeRV [ 3 ]
30.16
32.72
34.61
Ours
32.09
35.07
37.85
TABLE VI: Results with varying model sizes using the Big Buck Bunny dataset, in terms of PSNR(dB).
Single
Coupled
MRG
WarpRNN
WarpRNN
Bunny
Beauty
Honey
v1
✓
37.28
33.61
38.66
v2
✓
✓
37.69
33.50
39.32
v3
✓
✓
37.85
34.69
40.47
TABLE VII: Ablation study on different modules, in terms of PSNR(dB).
Fig. 8: Example qualitative result comparison on the mixed residual grid.
Implicit Neural Representations (INR) provide a compact and content-adaptive paradigm for video compression, typically representing a video through shared network parameters and frame-indexed embeddings. Compared to conventional or autoencoder-based codecs, these approaches exploit temporal redundancy within videos in an implicit manner, which potentially results in sub-optimal compression performance. In this paper, we propose G-NeRV, a graph-inspired INR that explicitly improves temporal redundancy exploitation in the implicit latent space. Motivated by the total correlation principles in information theory, we construct a temporal neighborhood over frame embeddings and perform message passing to aggregate reusable information from neighboring frames through an adaptive gate controlling the injection of neighboring information. Inspired by the reference frame buffer in conventional video coding, a memory bank mechanism has been further designed to enable efficient temporal-neighbor retrieval under random frame-index sampling in INR training. This new representation model has been integrated into an advanced representation compression framework and compared with existing conventional and neural video codecs. The results show that the G-NeRV codec outperforms the state-of-the-art INR-based codec, NVRC, and the latest standard video codec, VVC VTM, by 8.86% and 14.68% (in BD-rate), respectively, measured by PSNR on the UVG dataset.
Changqi Wang, Ge Gao, Fan Zhang +4
University of Bristol · Bytedance Inc., San Diego, USA
Implicit Neural Representations (INRs) have recently demonstrated impressive performance for video compression. However, since a separate INR must be overfit for each video, scaling to high-resolution videos while maintaining encoding efficiency remains a significant challenge. Hypernetwork-based approaches predict INR weights (hyponetworks) for unseen videos at high speeds, but with low quality, large compressed size, and prohibitive memory needs at higher resolutions. We address these fundamental limitations through three key contributions: (1) an approach that decomposes the weight prediction task spatially and temporally, by breaking short video segments into patch tubelets, to reduce the pretraining memory overhead by 20×; (2) a residual-based storage scheme that captures only differences between consecutive segment representations, significantly reducing bitstream size; and (3) a temporal coherence regularization framework that encourages changes in the weight space to be correlated with video content. Our proposed method, TeCoNeRV, achieves substantial improvements of 2.47dB and 5.35dB PSNR over the baseline at 480p and 720p on UVG, with 36% lower bitrates and 1.5-3× faster encoding speeds. With our low memory usage, we are the first hypernetwork approach to demonstrate results at 480p, 720p and 1080p on UVG, HEVC and MCL-JCV. Our project page is available at https://namithap10.github.io/teconerv/ .
Namitha Padmanabhan, Matthew Gwilliam, Abhinav Shrivastava
Department of Computer Science, University of Maryland, College Park, USA.
Implicit Neural Representations (INRs) have emerged as a promising paradigm for video representation and compression. However, existing multi-scale INR generators often suffer from significant parameter redundancy by stacking independent processing blocks for each scale. Inspired by the principle of scale self-similarity in the generation process, we propose SRNeRV, a novel scale-wise recursive framework that replaces this stacked design with a parameter-efficient shared architecture. The core of our approach is a hybrid sharing scheme derived from decoupling the processing block into a scale-specific spatial mixing module and a scale-invariant channel mixing module. We recursively apply the same shared channel mixing module, which contains the majority of the parameters, across all scales, significantly reducing the model size while preserving the crucial capacity to learn scale-specific spatial patterns. Extensive experiments demonstrate that SRNeRV achieves a significant rate-distortion performance boost, especially in INR-friendly scenarios, validating that our sharing scheme successfully amplifies the core strengths of the INR paradigm.
Jia Wang, Jun Zhu, Xinfeng Zhang
School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China