Organizations: Université Bourgogne Europe, DRIVE UR 1859, 58000 Nevers, France · MIS laboratory, University of Picardie Jules Verne, Amiens, France · CNRS-AIST JRL, AIST, Tsukuba, Japan · Artificial Intelligence Research Center, AIST, Tsukuba, Japan
Saliency-guided coding allocates bits to the regions viewers are most likely to attend, but accurate 360-degree saliency prediction is computationally expensive, an obstacle for low-latency immersive streaming applications such as robot teleoperation. We quantify how much saliency-prediction accuracy actually translates into compression gain. Building on a saliency-guided rate-distortion-optimized (RDO) learned codec, modified so that the training-time latent mask is driven by ground-truth saliency, which yields a single, predictor-agnostic codec, we evaluate saliency predictors spanning two orders of magnitude in compute on both the learned codec and H.265/HEVC. The results show that higher saliency-prediction accuracy yields diminishing compression gains: a lightweight predictor fast enough for live use (52 frames per second) captures 65% of the ground-truth oracle's bit-rate saving, a 3.3x heavier one reaches 74%, and a further 38x increase in compute adds no measurable benefit, on either codec. Therefore, lightweight predictors might also be a practical choice for latency-constrained 360-degree streaming.
Figures & tables
Fig. 1: Saliency-guided compression: a predicted saliency map steers the encoder’s bit allocation toward the regions viewers attend. This paper measures how the accuracy and cost of the predictor translate into coding gain.
Fig. 2: Saliency-guided learned codec used in this study (ELIC-QVRF variable-rate control and latent-space saliency masking), following the design of [ 11 ] . The saliency map is σ -transformed, average-pooled ( ×16 ) to the latent resolution, and passed through rate control before masking the latent y . The space-channel context model predicts the mean and scale (μ,σ) of the conditional Gaussian entropy model used by the arithmetic encoder/decoder (AE/AD).
Fig. 3: Mean SAL-PSNR rate-distortion over the 25 test ODIs on the learned codec, by saliency source (inset: zoom on 0.25 – 0.35 bpp).
Fig. 4: Mean SAL-PSNR BD-rate versus predictor accuracy on both codecs.
vs own ref. ↓
vs anchor ↓
SAL-PSNR
WS-PSNR
SAL-PSNR
Saliency Mask Source
CC ↑
KLD ↓
NSS ↑
AUC-J ↑
GFLOPs ↓
FPS ↑
Learned
HEVC
Learned
Learned
GT (oracle)
1.000
0.000
–
1.000
–
–
−10.39%
−16.41%
+2.74%
−53.9%
SalGAN360
0.733
0.369
1.310
0.798
38 912
0.04
−7.54%
−11.03%
+5.34%
−52.8%
Dual-stream (reprod.)
0.689
0.400
1.200
0.774
1 012.7
4.8
−7.68%
−11.08%
+4.11%
−52.6%
ATSal (attention-only)
0.658
0.428
1.161
0.772
304.8
52.4
−6.75%
−9.41%
+4.84%
−52.3%
TABLE I: Saliency sources: prediction accuracy, compute cost, and BD-rate on both codecs. Lower BD-rate is better.
Fig. 5: Per- CTU QP maps on P26 (base QP 32): flat-QP anchor vs. saliency- RDO with GT and predicted saliency ( Δ QP ∈[−3,+3] ).
Fig. 6: Decoded P26 at matched bit-rate ( 0.197 bpp; masked codecs rate-matched by gain interpolation), for each saliency source and the GT oracle. Column titles give whole-image bpp, SAL-PSNR and WS-PSNR. Green/orange: the 256×256 window with the highest/lowest GT saliency, with crop PSNR and its change against no mask.
Fig. 7: SAL-PSNR BD-rate relative to flat-QP HEVC versus saliency accuracy (lower is better; the two panels use different vertical scales): synthetically degraded GT maps (hollow) and real predictors (filled).
Given the popularity of 360° images on social media platforms, 360° image compression becomes a critical technology for media storage and transmission. Conventional 360° image compression pipeline projects the spherical image into a single 2D plane, leading to issues of oversampling and distortion. In this paper, we propose a novel viewport-based neural compression pipeline for 360° images. By replacing the image projection in conventional 360° image compression pipelines with viewport extraction and efficiently compressing multiple viewports, the proposed pipeline minimizes the inherent oversampling and distortion issues. However, viewport extraction impedes information sharing between multiple viewports during compression, causing the loss of global information about the spherical image. To tackle this global information loss, we design a neural viewport codec to capture global prior information across multiple viewports and maximally compress the viewport data. The viewport codec is empowered by a transformer-based ViewPort ConText (VPCT) module that can be integrated with canonical learning-based 2D image compression structures. We compare the proposed pipeline with existing 360° image compression models and conventional 360° image compression pipelines building on learning-based 2D image codecs and standard hand-crafted codecs. Results show that our pipeline saves an average of 14.01% bit consumption compared to the best-performing 360° image compression methods without compromising quality. The proposed VPCT-based codec also outperforms existing 2D image codecs in the viewport-based neural compression pipeline. Our code can be found at: https://github.com/Jingwei-Liao/VPCT.
Jingwei Liao, Bo Chen, Klara Nahrstedt +1
George Mason University, Fairfax, USA · University of Illinois Urbana-Champaign, Urbana, USA
360-degree video supports immersive applications such as virtual reality, autonomous driving, and education. Because spherical content cannot be processed directly by conventional video codecs, it must first be mapped to a two-dimensional projection. Projection choice affects spatial continuity, sampling uniformity, motion estimation, and compression efficiency. This thesis investigates how projection format influences end-to-end neural compression of 360-degree video. Seven formats supported by JVET 360Lib are evaluated using the scale-space flow model, JVET test sequences, and common test conditions. Each sequence is converted from its source equirectangular projection to a coding projection, compressed at multiple rate points, reconstructed, and converted back. Performance is assessed using PSNR, spherical PSNR, weighted spherical PSNR, and Bjøntegaard delta rate. A differentiable pipeline combining projection conversion, neural compression, and inverse projection is also compared with 360Lib. Results show that equirectangular and padded equirectangular projections provide the highest compression efficiency with the scale-space flow model, while cubemap-based and rhombic dodecahedron projections are less effective. This differs from the conventional HM-16.16 codec, for which cubemap-based formats, particularly equi-angular and adjusted cubemap projections, outperform equirectangular formats. Neural models based on optical flow benefit from the spatial continuity of single-face projections, whereas block-based hybrid codecs better accommodate multi-face layouts. These findings show that projection efficiency is codec-dependent and provide guidance for selecting projections for learning-based 360-degree video compression.
Image compression is increasingly used before machine vision systems, yet standard codecs are mainly optimized for human perception rather than downstream task accuracy. We propose SCOPE (Semantic COdec Pre-Editing), a conditional image pre-editor that modifies images before compression to improve the bitrate and detection accuracy trade-off. The model is conditioned on the codec family and compression severity, allowing a single trained network to adapt to multiple compression settings without retraining or loading codec-specific weights. Instead of uniformly preserving the whole image, the pre-editor learns to protect task-relevant regions while simplifying less important background content to improve compressibility. The resulting rate-accuracy curves show consistent gains across different codec families and detector architectures, together with strong transferability to unseen learned and wavelet-based compression families.
Silvia Dani, Leonardo Galteri, Marco Bertini
University of Florence, Florence, Italy · Small Pixels, Florence, Italy