Organizations: The National Engineering Laboratory for Video Technology, School of Computer Science, Peking University · Institute for AI Industry Research (AIR), Tsinghua University · Tsinghua Shenzhen International Graduate School, Tsinghua University
Multi-view image compression (MIC) aims to achieve high compression efficiency by exploiting inter-image correlations, playing a crucial role in 3D applications. As a subfield of MIC, distributed multi-view image compression (DMIC) offers performance comparable to MIC while eliminating the need for inter-view information at the encoder side. However, existing methods in DMIC typically treat all images equally, overlooking the varying degrees of correlation between different views during decoding, which leads to suboptimal coding performance. To address this limitation, we propose a novel OmniParallax Attention Mechanism (OPAM), which is a general mechanism for explicitly modeling correlations and aligned features between arbitrary pairs of information sources. Building upon OPAM, we propose a Parallax Multi Information Fusion Module (PMIFM) to adaptively integrate information from different sources. PMIFM is incorporated into both the joint decoder and the entropy model to construct our end-to-end DMIC framework, ParaHydra. Extensive experiments demonstrate that ParaHydra is the first DMIC method to significantly surpass state-of-the-art MIC codecs, while maintaining low computational overhead. Performance gains become more pronounced as the number of input views increases. Compared with LDMIC, ParaHydra achieves bitrate savings of \textbf{19.72%} on WildTrack(3) and up to \textbf{24.18%} on WildTrack(6), while significantly improving coding efficiency (as much as 65× in decoding and 34× in encoding).
Figures & tables
Figure 1: Visualization of OPAM correlations. Row 1 : Input views. Row 2 : Difference maps. Row 3 : Correlations, where consistent regions ( green ) are prioritized over occlusions ( red ).
Figure 2: (a) The proposed ParaHydra framework. y^K∖{k} and fK∖{k} denote the sets of all view features excluding the k -th view feature y^k and fk , respectively. (b) Parallax Multi Information Fusion Module (PMIFM). PMIFM integrates the aligned feature fkt based on the semantic relevance Ck between each side source fk and the main source fi . (c) Parallax Entropy Model. The figure illustrates the decoding process for the slice y^ki . Each latent slice y^ki is partitioned into anchor y^k,aci and non-anchor y^k,nai parts. Anchor y^k,aci is decoded first using Gaussian parameters (μk,aci,σk,aci) predicted from the channel context Φk,chi (provided by previous slices y^k<i ) and the hyperprior ΦHk . Non-anchor y^k,nai is decoded next using Gaussian parameters (μk,nai,σk,nai) predicted from the local context Φk,lci (derived from anchor y^k,aci ), the channel context Φk,chi , the global context Φk,gci , and the hyperprior ΦHk .
Figure 3: Overview of OmniParallax Attention Mechanism (OPAM). Left: Parallax attention. Middle: Two-stage parallax attention in OPAM. OPAM applies horizontal (red) and vertical (blue) parallax attention sequentially to capture the full 2D spatial context. Right: Receptive fields of the aligned features. Each position in flhor attends to one row of fr , and each position in flver attends to one column of flhor , allowing each position in flver to attend to the entire 2D spatial domain of fr .
Figure 4: Rate-distortion curves of ParaHydra compared against various baselines.
Methods
InStereo2K(2)
Cityscapes(2)
WildTrack(3)
WildTrack(6)
Mip-NeRF 360(3)
PSNR
MS-SSIM
PSNR
MS-SSIM
PSNR
MS-SSIM
PSNR
MS-SSIM
PSNR
MS-SSIM
LDMIC
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
VVC
48.68%
80.69%
54.33%
101.92%
49.47%
115.64%
25.16%
120.67%
7.14%
96.47%
MV-HEVC
84.84%
182.11%
106.27%
202.95%
31.84%
176.86%
10.01%
168.04%
41.15%
171.15%
ECSIC
25.65%
11.99%
50.51%
41.12%
–
–
–
–
–
–
BiSIC
-2.95%
-4.65%
11.40%
-7.11%
–
–
–
–
–
–
Table 1: BDBR comparison relative to LDMIC across different datasets. The best results are highlighted in bold . The number of input views is indicated in parentheses. A dash (–) denotes that the corresponding model cannot be evaluated on that dataset.
Methods
VVC
MV-HEVC
BiSIC-fast
CAMSIC
BiSIC
LDMIC
2D Attn
ParaHydra
Enc. Time (s)
349.47
24.67
0.35
0.87
0.30
9.27
0.62
0.27
Dec. Time (s)
0.37
0.17
0.41
1.79
0.49
21.43
0.67
0.33
Params (M)
–
–
327.66
602.05
305.71
214.98
108.81
105.25
FLOPs (T)
–
–
3.96
9.18
3.77
2.88
1.64
1.78
Table 2: Complexity comparison of various codecs evaluated on images with a resolution of 1024 × 832 from the InStereo2K dataset.
Views
VVC
MV-HEVC
LDMIC-fast
LMVIC
ParaHydra
3
7.14%
41.15%
25.52%
-14.30%
-18.20%
4
53.06%
91.30%
14.31%
17.27%
-16.84%
Table 3: BDBR comparison relative to LDMIC on Mip-NeRF.
Figure 5: Ablation study on the WildTrack with 3 input views.
Figure 6: Qualitative results on the InStereo2K. PSNR and bitrate (bpp) are denoted as PSNR / bpp. Zoom in for a better view.
Distributed Image Compression (DIC) is crucial for multi-view transmission, especially when operating at extremely low bitrates (< 0.1 bpp). Its core challenge is effectively utilizing side information to achieve high-quality reconstruction under strict bitrate budgets. However, existing DIC approaches struggle to exploit global context and object-level details from side information, leading to local blurring and the loss of fine details in the reconstruction. To address these limitations, we propose a Multimodal DIC framework (MDIC), which, for the first time, leverages side information in a multimodal manner into the DIC paradigm, effectively preserving fine-grained local details and enhancing global perceptual quality in reconstructed images. Specifically, we introduce a text-to-image diffusion-based decoder conditioned on textual side information extracted from correlated images to capture shared global semantics. Moreover, we design a feature-mask generator, supervised by a multimodal fine-grained alignment task, to strengthen the exploitation of visual side information. The generated mask serves two purposes: first, it guides the extraction of fine-grained details from losslessly transmitted side information to preserve the semantic consistency of reconstructed details; second, it regulates the extraction of clustered feature representations from the quantized VQ-VAE embeddings, compensating for category information lost under the extreme compression of the primary image. Extensive experiments on the widely used KITTI Stereo and Cityscapes datasets demonstrate that MDIC achieves state-of-the-art perceptual quality at extremely low bitrates.
Guojun Xu, Mingyang Zhang, Jianwen Xiang +3
School of Computer Science and Artificial Intelligence, Wuhan University of Technology
Overfitted codecs have demonstrated promising performance for image and video compression. In particular, for image compression, the Cool-chic family of models has shown competitive performance against scene-agnostic models, with orders of magnitude lower decoding complexity at the cost of a longer overfitting process. However, these overfitted image codecs are not fully optimized toward the rate-distortion objective: their network weights remain in full precision during training, and the associated quantization parameters are selected in a separate post-training stage. Furthermore, their synthesis operates at a single scale, which overlooks cross-scale redundancy. In this paper, we propose MIRC, an overfitted image codec in which every coded component, including the latents, the synthesis network, and the entropy models, is quantized and entropy coded under a single rate-distortion objective, adopting the end-to-end compression pipeline of the neural video representation codec NVRC. We further introduce a multi-scale representation with cross-stage parameter sharing, which improves coding efficiency at a small transmitted overhead. On the CLIC2020 professional validation set, MIRC achieves a 10.5% BD-rate saving against VVC (VTM 22.0). Moreover, MIRC offers a family of configurations spanning 1.2 to 2.9 kMAC per pixel, so the decoding budget can be selected to match the deployment target.
Tianhao Peng, Ho Man Kwan, Fan Zhang +2
Visual Information Lab, University of Bristol, UK · Tencent Media Lab, Palo Alto, USA
Learned image compression (LIC) is bottlenecked by the need to store independent models for each rate-distortion operating point. Existing variable bit-rate (VBR) methods aim to reduce this overhead via dense parameter modulation, but forcing a shared backbone to approximate divergent mappings causes severe feature entanglement. Specifically, low-rate smoothing gradients inherently conflict with the preservation of high-frequency textural details, leading to sub-optimal performance. To resolve this, we propose MixCompress, a unified VBR framework based on sparse structural specialization. While sparsely gated Mixture-of-Experts (MoE) routing successfully mitigates gradient conflict, it operates on a fixed computational budget. To address the increased representational demands of higher bit-rates we introduce a Mixture-of-Depths (MoD) extension to dynamically scale model capacity. Combined with Conditional Auxiliary Transforms (CAT) for dynamic sub-band energy modulation, our hierarchical framework effectively dynamically scales capacity. Extensive evaluations demonstrate that MixCompress not only matches individually optimized single-rate baselines but can even surpass them, establishing a new Pareto frontier for computationally efficient image coding.