Organizations: Université Bourgogne Europe, DRIVE UR 1859, 58000 Nevers, France · MIS laboratory, University of Picardie Jules Verne, Amiens, France · CNRS-AIST JRL, AIST, Tsukuba, Japan · Artificial Intelligence Research Center, AIST, Tsukuba, Japan
Saliency-guided coding allocates bits to the regions viewers are most likely to attend, but accurate 360-degree saliency prediction is computationally expensive, an obstacle for low-latency immersive streaming applications such as robot teleoperation. We quantify how much saliency-prediction accuracy actually translates into compression gain. Building on a saliency-guided rate-distortion-optimized (RDO) learned codec, modified so that the training-time latent mask is driven by ground-truth saliency, which yields a single, predictor-agnostic codec, we evaluate saliency predictors spanning two orders of magnitude in compute on both the learned codec and H.265/HEVC. The results show that higher saliency-prediction accuracy yields diminishing compression gains: a lightweight predictor fast enough for live use (52 frames per second) captures 65% of the ground-truth oracle's bit-rate saving, a 3.3x heavier one reaches 74%, and a further 38x increase in compute adds no measurable benefit, on either codec. Therefore, lightweight predictors might also be a practical choice for latency-constrained 360-degree streaming.
Figures & tables
Fig. 1: Saliency-guided compression: a predicted saliency map steers the encoder’s bit allocation toward the regions viewers attend. This paper measures how the accuracy and cost of the predictor translate into coding gain.
Fig. 2: Saliency-guided learned codec used in this study (ELIC-QVRF variable-rate control and latent-space saliency masking), following the design of [ 11 ] . The saliency map is σ -transformed, average-pooled ( ×16 ) to the latent resolution, and passed through rate control before masking the latent y . The space-channel context model predicts the mean and scale (μ,σ) of the conditional Gaussian entropy model used by the arithmetic encoder/decoder (AE/AD).
Fig. 3: Mean SAL-PSNR rate-distortion over the 25 test ODIs on the learned codec, by saliency source (inset: zoom on 0.25 – 0.35 bpp).
Fig. 4: Mean SAL-PSNR BD-rate versus predictor accuracy on both codecs.
vs own ref. ↓
vs anchor ↓
SAL-PSNR
WS-PSNR
SAL-PSNR
Saliency Mask Source
CC ↑
KLD ↓
NSS ↑
AUC-J ↑
GFLOPs ↓
FPS ↑
Learned
HEVC
Learned
Learned
GT (oracle)
1.000
0.000
–
1.000
–
–
−10.39%
−16.41%
+2.74%
−53.9%
SalGAN360
0.733
0.369
1.310
0.798
38 912
0.04
−7.54%
−11.03%
+5.34%
−52.8%
Dual-stream (reprod.)
0.689
0.400
1.200
0.774
1 012.7
4.8
−7.68%
−11.08%
+4.11%
−52.6%
ATSal (attention-only)
0.658
0.428
1.161
0.772
304.8
52.4
−6.75%
−9.41%
+4.84%
−52.3%
TABLE I: Saliency sources: prediction accuracy, compute cost, and BD-rate on both codecs. Lower BD-rate is better.
Fig. 5: Per- CTU QP maps on P26 (base QP 32): flat-QP anchor vs. saliency- RDO with GT and predicted saliency ( Δ QP ∈[−3,+3] ).
Fig. 6: Decoded P26 at matched bit-rate ( 0.197 bpp; masked codecs rate-matched by gain interpolation), for each saliency source and the GT oracle. Column titles give whole-image bpp, SAL-PSNR and WS-PSNR. Green/orange: the 256×256 window with the highest/lowest GT saliency, with crop PSNR and its change against no mask.
Fig. 7: SAL-PSNR BD-rate relative to flat-QP HEVC versus saliency accuracy (lower is better; the two panels use different vertical scales): synthetically degraded GT maps (hollow) and real predictors (filled).