Projection-Aware End-to-End Learned Video Compression for 360-Degree Video

Niloofar Maani

cs.CVAug 26, 2026

Projection-Aware End-to-End Learned Video Compression for 360-Degree Video

Authors: Niloofar Maani

Abstract

360-degree video supports immersive applications such as virtual reality, autonomous driving, and education. Because spherical content cannot be processed directly by conventional video codecs, it must first be mapped to a two-dimensional projection. Projection choice affects spatial continuity, sampling uniformity, motion estimation, and compression efficiency. This thesis investigates how projection format influences end-to-end neural compression of 360-degree video. Seven formats supported by JVET 360Lib are evaluated using the scale-space flow model, JVET test sequences, and common test conditions. Each sequence is converted from its source equirectangular projection to a coding projection, compressed at multiple rate points, reconstructed, and converted back. Performance is assessed using PSNR, spherical PSNR, weighted spherical PSNR, and Bjøntegaard delta rate. A differentiable pipeline combining projection conversion, neural compression, and inverse projection is also compared with 360Lib. Results show that equirectangular and padded equirectangular projections provide the highest compression efficiency with the scale-space flow model, while cubemap-based and rhombic dodecahedron projections are less effective. This differs from the conventional HM-16.16 codec, for which cubemap-based formats, particularly equi-angular and adjusted cubemap projections, outperform equirectangular formats. Neural models based on optical flow benefit from the spatial continuity of single-face projections, whereas block-based hybrid codecs better accommodate multi-face layouts. These findings show that projection efficiency is codec-dependent and provide guidance for selecting projections for learning-based 360-degree video compression.

Figures & tables

Figure 1: 360-degree video field of view from [ 12 ] .

Figure 2: JPEG based image codec from [ 20 ] .

Figure 3: Architecture of CNN based compression model [ 23 ] From [ 20 ] .

Figure 4: (a): An illustration of the conventional video codecs. (b): The Deep compression model from [ 8 ] .

Figure 5: DVC MV Encoder-decoder from [ 8 ] .

Figure 6: DVC model compensation model from [ 8 ] .

Figure 7: Autoencoder architecture scheme from [ 37 ] .

Figure 8: Optical flow for a pixel located at

(x,y)

Figure 9: Frame interpolation using CNN from [ 60 ] .

Figure 10: Scale space wraping from [ 10 ] .

Figure 11: Scale space model architecture from [ 10 ] .

Figure 12: X, Y, Z cordinate system used in 360lib app from [ 16 ] .

Figure 13: Balboa JVET sequence in ERP format.

Figure 14: ERP to PERP conversion from [ 72 ] .

Figure 15: Balboa JVET sequence in PERP format.

f	X	Y	Z
0	1.0	$-v$	$-u$
1	$-1.0$	$-v$	$u$
2	$u$	1.0	$v$
3	$u$	$-1.0$	$-v$
4	$u$	$-v$	1.0
5	$u$	$v$	$-1.0$

Table 1: CMP 2D to 3D mapping

Figure 16: Balboa JVET sequence in CMP format.

Figure 17: Balboa JVET sequence in EAC format.

Figure 18: Balboa JVET sequence in ACP format.

Figure 19: Balboa JVET sequence in HEC format.

Figure 20: Rotated sphere projection format from [ 16 ] .

Figure 21: Balboa JVET sequence in RSP format.

Figure 22: 360 video testing procedure from [ 78 ]

Figure 23: Projection comparison pipeline

Sequence name	Width	Height	Frame count	Frame rate	Bit depth
SkateboardInLot	8192	4096	300	30fps	10
ChairLift	8192	4096	300	30fps	10
KiteFlite	8192	4096	300	30fps	8
Harbor	8192	4096	300	30fps	8
GasLamp	8192	4096	300	30fps	8
Balboa	6144	3072	600	60fps	8

Table 2: JVET testing sequances

Format	Face width	Face height	Width	Height
ERP	4096	2048	4096	2048
Padded ERP	4096	2048	4096	2048
CMP	960	960	2880	1920
ACP	960	960	2880	1920
EAC	960	960	2880	1920
HEC	960	960	2880	1920

Table 3: Projection formats

Figure 24: Sampled points on sphere from [ 81 ] .

Figure 25: Projection model forward pass.

Figure 28: Projection model backward pass

Figure 29: Harbor reference projection ERP

Panel of Figure 30 — Figure 30: Harbor ERP-based projections.

Panel of Figure 31 — Figure 31: Harbor RSP projection.

Panel of Figure 32 — Figure 32: Harbor cubemap based projection.

Panel of Figure 33 — Figure 33: Harbor end-to-end (PSNR, S-PSNR, WS-PSNR) rate-distortion curves.

Panel of Figure 34 — Figure 34: Harbor projections comparison end-to-end

Sequance	PERP	CMP	EAC	ACP	HEC	RSP
Harbor	0.204	48.629	36.303	35.246	34.548	35.155
SkateboardInLot	-0.543	52.856	40.941	40.777	39.167	52.688
ChairLift	-0.430	59.634	47.645	46.683	46.020	54.958
KiteFlite	-0.150	39.792	27.942	27.783	27.308	28.243
GasLamp	-0.467	33.156	20.83	19.720	17.316	17.056
Balboa	-1.397	43.245	38.428	39.376	37.019	45.708

Table 4: Comparison of projections compressed using scale space flow model

Sequences	CMP	ACP	RSP	EAC
All sequences	-1.1	-6.9	-5.7	-6.9
8K sequences	-3.9	-8.4	-8.2	-8.5
6K sequences	2.9	-4.9	-2.6	-5.1
4K sequences	-0.7	-6.4	-4.9	-6.1
Average	-0.7	-6.65	-5.35	-6.65

Table 5: Comparison of projections compressed using HM-16.16 codec [ 86 ] .

Panel of Figure 35 — Figure 35: Projection layer output

Panel of Figure 36 — Figure 36: Method comparison using rate-distortion (PSNR) curves.

Harbor	Broadway	PoleVault	AerialCity	KiteFlite	Gaslamp	Brancastle
369.74	114.10	105.15	142.86	103.49	349.59	37.65

Table 6: Method comparison based on bd_rate

Panel of Figure 37 — Figure 37: ERP projection comparison for both methods

Video Compression

Video compression reduces the bits needed to store or transmit moving images while balancing visual quality, computational cost, and downstream usefulness.

See all 73 papers

Viewport-based Neural 360° Image Compression
Mar 24, 2026Jingwei Liao, Bo Chen, Klara Nahrstedt +1Learned Image Compression
George Mason University, Fairfax, USA · University of Illinois Urbana-Champaign, Urbana, USA
Enhanced Neural Video Representation Compression across Extreme Complexity and Quality Scales
Jun 26, 2026Ho Man Kwan, Tianhao Peng, Fan Zhang +3Implicit Neural Representations Neural Video Compression
Visual Information Lab, University of Bristol, UK · Network Connectivity Services Research, BT, UK
Scalable Neural Video Representation Compression
Sep 2, 2026Tianhao Peng, Ho Man Kwan, Fan Zhang +2Neural Video Compression Video Compression
Visual Information Lab, University of Bristol, UK · Tencent Media Lab, Palo Alto, USA

Mar 24, 2026J/K move · Enter open · S save

Viewport-based Neural 360° Image Compression

Jingwei Liao, Bo Chen, Klara Nahrstedt, Zhisheng Yan

George Mason University, Fairfax, USA · University of Illinois Urbana-Champaign, Urbana, USA

Learned Image Compression

Open paper AskPDF arXiv

Given the popularity of 360° images on social media platforms, 360° image compression becomes a critical technology for media storage and transmission. Conventional 360° image compression pipeline projects the spherical image into a single 2D plane, leading to issues of oversampling and distortion. In this paper, we propose a novel viewport-based neural compression pipeline for 360° images. By replacing the image projection in conventional 360° image compression pipelines with viewport extraction and efficiently compressing multiple viewports, the proposed pipeline minimizes the inherent oversampling and distortion issues. However, viewport extraction impedes information sharing between multiple viewports during compression, causing the loss of global information about the spherical image. To tackle this global information loss, we design a neural viewport codec to capture global prior information across multiple viewports and maximally compress the viewport data. The viewport codec is empowered by a transformer-based ViewPort ConText (VPCT) module that can be integrated with canonical learning-based 2D image compression structures. We compare the proposed pipeline with existing 360° image compression models and conventional 360° image compression pipelines building on learning-based 2D image codecs and standard hand-crafted codecs. Results show that our pipeline saves an average of

14.01\%

bit consumption compared to the best-performing 360° image compression methods without compromising quality. The proposed VPCT-based codec also outperforms existing 2D image codecs in the viewport-based neural compression pipeline. Our code can be found at: https://github.com/Jingwei-Liao/VPCT.

Abstract

Video Compression

Explore similar work

Viewport-based Neural 360° Image Compression

Enhanced Neural Video Representation Compression across Extreme Complexity and Quality Scales

Scalable Neural Video Representation Compression