cs.CVAug 26, 2026

Projection-Aware End-to-End Learned Video Compression for 360-Degree Video

Authors: Niloofar Maani

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

360-degree video supports immersive applications such as virtual reality, autonomous driving, and education. Because spherical content cannot be processed directly by conventional video codecs, it must first be mapped to a two-dimensional projection. Projection choice affects spatial continuity, sampling uniformity, motion estimation, and compression efficiency. This thesis investigates how projection format influences end-to-end neural compression of 360-degree video. Seven formats supported by JVET 360Lib are evaluated using the scale-space flow model, JVET test sequences, and common test conditions. Each sequence is converted from its source equirectangular projection to a coding projection, compressed at multiple rate points, reconstructed, and converted back. Performance is assessed using PSNR, spherical PSNR, weighted spherical PSNR, and Bjøntegaard delta rate. A differentiable pipeline combining projection conversion, neural compression, and inverse projection is also compared with 360Lib. Results show that equirectangular and padded equirectangular projections provide the highest compression efficiency with the scale-space flow model, while cubemap-based and rhombic dodecahedron projections are less effective. This differs from the conventional HM-16.16 codec, for which cubemap-based formats, particularly equi-angular and adjusted cubemap projections, outperform equirectangular formats. Neural models based on optical flow benefit from the spatial continuity of single-face projections, whereas block-based hybrid codecs better accommodate multi-face layouts. These findings show that projection efficiency is codec-dependent and provide guidance for selecting projections for learning-based 360-degree video compression.

Figures & tables

Figure 1: 360-degree video field of view from [ 12 ] .
Figure 2: JPEG based image codec from [ 20 ] .
Figure 3: Architecture of CNN based compression model [ 23 ] From [ 20 ] .
Figure 4: (a): An illustration of the conventional video codecs. (b): The Deep compression model from [ 8 ] .
Figure 5: DVC MV Encoder-decoder from [ 8 ] .
Figure 6: DVC model compensation model from [ 8 ] .
Figure 7: Autoencoder architecture scheme from [ 37 ] .
Figure 8: Optical flow for a pixel located at (x,y)(x,y) .
Figure 9: Frame interpolation using CNN from [ 60 ] .
Figure 10: Scale space wraping from [ 10 ] .
Figure 11: Scale space model architecture from [ 10 ] .
Figure 12: X, Y, Z cordinate system used in 360lib app from [ 16 ] .
Figure 13: Balboa JVET sequence in ERP format.
Figure 14: ERP to PERP conversion from [ 72 ] .
Figure 15: Balboa JVET sequence in PERP format.
fXYZ
01.0−v-v−u-u
1−1.0-1.0−v-vuu
2uu1.0vv
3uu−1.0-1.0−v-v
4uu−v-v1.0
5uuvv−1.0-1.0
Table 1: CMP 2D to 3D mapping
Figure 16: Balboa JVET sequence in CMP format.
Figure 17: Balboa JVET sequence in EAC format.
Figure 18: Balboa JVET sequence in ACP format.
Figure 19: Balboa JVET sequence in HEC format.
Figure 20: Rotated sphere projection format from [ 16 ] .
Figure 21: Balboa JVET sequence in RSP format.
Figure 22: 360 video testing procedure from [ 78 ]
Figure 23: Projection comparison pipeline
Sequence nameWidthHeightFrame countFrame rateBit depth
SkateboardInLot8192409630030fps10
ChairLift8192409630030fps10
KiteFlite8192409630030fps8
Harbor8192409630030fps8
GasLamp8192409630030fps8
Balboa6144307260060fps8
Table 2: JVET testing sequances
FormatFace widthFace heightWidthHeight
ERP4096204840962048
Padded ERP4096204840962048
CMP96096028801920
ACP96096028801920
EAC96096028801920
HEC96096028801920
Table 3: Projection formats
Figure 24: Sampled points on sphere from [ 81 ] .
Figure 25: Projection model forward pass.
Figure 26: CMP layer forward pass
Figure 27: CMP layer backward pass
Figure 28: Projection model backward pass
Figure 29: Harbor reference projection ERP
Figure 30: Harbor ERP-based projections.
Figure 31: Harbor RSP projection.
Figure 32: Harbor cubemap based projection.
Figure 33: Harbor end-to-end (PSNR, S-PSNR, WS-PSNR) rate-distortion curves.
Figure 34: Harbor projections comparison end-to-end
SequancePERPCMPEACACPHECRSP
Harbor0.20448.62936.30335.24634.54835.155
SkateboardInLot-0.54352.85640.94140.77739.16752.688
ChairLift-0.43059.63447.64546.68346.02054.958
KiteFlite-0.15039.79227.94227.78327.30828.243
GasLamp-0.46733.15620.8319.72017.31617.056
Balboa-1.39743.24538.42839.37637.01945.708
Table 4: Comparison of projections compressed using scale space flow model
SequencesCMPACPRSPEAC
All sequences-1.1-6.9-5.7-6.9
8K sequences-3.9-8.4-8.2-8.5
6K sequences2.9-4.9-2.6-5.1
4K sequences-0.7-6.4-4.9-6.1
Average-0.7-6.65-5.35-6.65
Table 5: Comparison of projections compressed using HM-16.16 codec [ 86 ] .
Figure 35: Projection layer output
Figure 36: Method comparison using rate-distortion (PSNR) curves.
HarborBroadwayPoleVaultAerialCityKiteFliteGaslampBrancastle
369.74114.10105.15142.86103.49349.5937.65
Table 6: Method comparison based on bd_rate
Figure 37: ERP projection comparison for both methods
More in this topic

Video Compression

Video compression reduces the bits needed to store or transmit moving images while balancing visual quality, computational cost, and downstream usefulness.

See all 73 papers

Explore similar work

CardsList
Mar 24, 2026

Viewport-based Neural 360° Image Compression

Given the popularity of 360° images on social media platforms, 360° image compression becomes a critical technology for media storage and transmission. Conventional 360° image compression pipeline projects the spherical image into a single 2D plane, leading to issues of oversampling and distortion. In this paper, we propose a novel viewport-based neural compression pipeline for 360° images. By replacing the image projection in conventional 360° image compression pipelines with viewport extraction and efficiently compressing multiple viewports, the proposed pipeline minimizes the inherent oversampling and distortion issues. However, viewport extraction impedes information sharing between multiple viewports during compression, causing the loss of global information about the spherical image. To tackle this global information loss, we design a neural viewport codec to capture global prior information across multiple viewports and maximally compress the viewport data. The viewport codec is empowered by a transformer-based ViewPort ConText (VPCT) module that can be integrated with canonical learning-based 2D image compression structures. We compare the proposed pipeline with existing 360° image compression models and conventional 360° image compression pipelines building on learning-based 2D image codecs and standard hand-crafted codecs. Results show that our pipeline saves an average of 14.01%14.01\% bit consumption compared to the best-performing 360° image compression methods without compromising quality. The proposed VPCT-based codec also outperforms existing 2D image codecs in the viewport-based neural compression pipeline. Our code can be found at: https://github.com/Jingwei-Liao/VPCT.

Jingwei Liao, Bo Chen, Klara Nahrstedt +1

George Mason University, Fairfax, USA · University of Illinois Urbana-Champaign, Urbana, USA

Learned Image Compression
AskPDF
Jun 26, 2026

Enhanced Neural Video Representation Compression across Extreme Complexity and Quality Scales

Implicit neural representations (INRs) have recently emerged as a promising approach to video compression, delivering competitive rate-distortion performance alongside rapid decoding. However, existing neural video codecs struggle to balance complexity and scalability. Lightweight models often suffer from degraded compression performance when scaled to different bitrate/quality levels, whereas high-performance models exhibit limited scalability, as their model complexity typically increases with quality. This lack of a unified architecture capable of maintaining consistent complexity across a wide range of bitrates severely limits their diverse real-world deployment. To address these challenges, we introduce NVRC++, a novel INR-based video codec that utilizes a lightweight INR with multiple high-resolution feature grids, providing high scalability at any given complexity level. This is paired with an optimization framework that enables efficient overfitting on high-resolution grids for long video sequences, thereby exploiting spatio-temporal redundancies without prohibitive computational or memory overhead. Additionally, an advanced entropy model is designed for efficiently compressing the high-dimensional grid parameters. As a result, NVRC++ provides four complexity levels (from 7kMACs/pixel to 360kMACs/pixel), each spanning wide bitrate and quality ranges while supporting real-time decoding. The experimental results show that NVRC++ offers a much faster decoding speed (up to 7.6x) compared to the SOTA INR-based video codec, NVRC, while delivering comparable performance.

Ho Man Kwan, Tianhao Peng, Fan Zhang +3

Visual Information Lab, University of Bristol, UK · Network Connectivity Services Research, BT, UK

Implicit Neural RepresentationsNeural Video CompressionEntropy ModelingVideo Compression
AskPDF
Sep 2, 2026

Scalable Neural Video Representation Compression

Scalable video coding (SVC) encodes a video into a layered bitstream consisting of a base layer and one or multiple enhancement layers, enabling decoding at different bitrate/quality/resolution operating points to accommodate diverse device capabilities and network conditions. Due to its practical flexibility, SVC has been incorporated into major video coding standards and has recently attracted growing interest for both scene-agnostic and scene-adaptive neural video codecs. Among the latter, Implicit neural representation (INR) based codecs achieve compression by overfitting a compact neural network to an individual video, offering fast decoding and competitive coding efficiency compared to scene-agnostic neural codecs. However, research on scalable INR-based compression remains in its infancy: these methods support scalable coding by introducing additional network layers, which couple the bitrate with the decoding complexity and also cannot achieve comparable performance with strong scalable/non-scalable codecs. In this context, this paper proposes S-NVRC, a scalable INR-based video codec that jointly supports fine-grained bitrate and decoding complexity scalability from a single embedded bitstream. It adopts a coarse-to-fine prefix for feature grids and a nested prefix for network layers, which scale bitrate and decoding complexity, respectively. The proposed S-NVRC spans a wide range of bitrate and decoding-complexity using a single encoding (training) and outperforms SHM 12.4 and the multi-layer VTM-20.0, by 43.7% and 5.6% in BD-rate on the UVG dataset, while also providing flexible complexity scalability. Implemented code will be provided.

Tianhao Peng, Ho Man Kwan, Fan Zhang +2

Visual Information Lab, University of Bristol, UK · Tencent Media Lab, Palo Alto, USA

Neural Video CompressionVideo CompressionImplicit Neural Representations
AskPDF