The rapid growth of visual data under stringent storage and bandwidth constraints makes extremely low-bitrate image compression increasingly important. While Vector Quantization (VQ) offers strong structural fidelity, existing methods lack a principled mechanism for joint rate-distortion (RD) optimization due to the disconnect between representation learning and entropy modeling. We propose RDVQ, a unified framework that enables end-to-end RD optimization for VQ-based compression via a differentiable relaxation of the codebook distribution, allowing the entropy loss to directly shape the latent prior. We further develop an autoregressive entropy model that supports accurate entropy modeling and test-time rate control. Extensive experiments demonstrate that RDVQ achieves strong performance at extremely low bitrates with a lightweight architecture, attaining competitive or superior perceptual quality with significantly fewer parameters. Compared with RDEIC, RDVQ reduces bitrate by up to 75.71% on DISTS and 37.63% on LPIPS on DIV2K-val. Beyond empirical gains, RDVQ introduces an entropy-constrained formulation of VQ, highlighting the potential for a more unified view of image tokenization and compression. The code will be available at https://github.com/CVL-UESTC/RDVQ.
Figures & tables
Figure 1: Gradient-based rate control in VQ compression is challenging due to non-differentiable indices.
Figure 2: Overview of RDVQ. The analysis transform ga extracts multi-scale features, which are flattened into a sequence y for vector quantization and entropy modeling. The VQ module produces hard-quantized embeddings yq , discrete indices yind , and a relaxed distribution psoft . During training, reconstruction is performed from yq , while the rate objective is computed as the cross-entropy between the relaxed distribution psoft and the entropy model prediction qψ , enabling gradient flow from the rate loss to the encoder. The same model is used for entropy coding and supports test-time index completion. Right: Details of the VQ module and gradient flow. Hard assignment is used for reconstruction and coding, while the relaxed distribution is used only for rate estimation, restoring gradient flow from the rate loss ( LR ) to the encoder ga .
Figure 3: Dependency-aware ordering for autoregressive entropy modeling. Scale-wise spatial orders are defined within each scale and concatenated into a unified order vector o , from which the attention mask M=(o>o⊤) is constructed.
Figure 4: Rate-distortion curves on the Kodak, the DIV2k-val and the CLIC2020-test datasets.
Figure 5: Visual examples on the CLIC2020 test set. Zoom in for better view.
Figure 6: Model efficiency comparison on DIV2K [ 1 ] . RDVQ achieves the best BD-DISTS with less than 20% of the parameters of most baselines, while maintaining competitive latency.
Figure 7: Visual examples on the Kodak dataset.
Figure 8: Codebook usage becomes more concentrated under stronger rate constraints.
Figure 9: Test-time rate adjustment via prefix transmission. Left: Rate–distortion curves on Kodak. Right: Visual comparisons at similar bitrates (Bpp / DISTS). RDVQ-Adj maintains competitive performance with gradual degradation, while AE-Entropy and Zero-padding suffer from more noticeable artifacts.
Method
bpp ↓
DISTS ↓
LPIPS ↓
FID ↓
RDVQ (full)
0.0247
0.1005
0.2321
19.96
w/o Relaxation
0.0464
0.2147
0.5031
86.93
K-means VQ
0.0247
0.1253
0.2831
28.08
Table 1: Quantitative comparison of ablation variants on DIV2K-val. Lower DISTS, LPIPS, FID indicate better perceptual quality.
Figure 10: PCA visualization of the largest-scale encoder features under different compression ratios. As the bitrate decreases, the features become progressively smoother, with high-frequency details suppressed.
Figure S1 : Effectiveness of tuning on high-quality images (evaluated on Div2K validation set).
Figure S2 : Effectiveness of piecewise strategy (evaluated on Div2K validation set).
Method
Ours
CGIC [ 33 ]
Diffo [ 54 ]
OneDC [ 67 ]
BD-DISTS (%, Kodak)
0.0
+5422.8
+207.11
+18.81
Table S1 : Comparison with additional VQ-based methods on the Kodak dataset in terms of BD-DISTS. Lower is better.
Figure S3 : Rate-distortion curves on the CLIC 2020 testset.
Figure S4 : Rate-distortion curves on the DIV2K-val dataset.
Figure S5 : Rate-distortion curves on the Kodak dataset.
Figure S6 : Test time rate control examples on CLIC2020 testset.
Figure S7 : Test time rate control examples on DIV2K-val dataset.
Figure S8 : Visual examples and comparisons on 2K-resolution images from CLIC2020-test dataset.
Figure S9 : Visual examples and comparisons on 2K-resolution images from CLIC2020-test dataset.
Figure S10 : Visual examples and comparisons on 2K-resolution images from CLIC2020-test dataset.
Figure S11 : Visual examples and comparisons on 2K-resolution images from CLIC2020-test dataset.
Figure S12 : Visual examples and comparisons on 2K-resolution images from CLIC2020-test dataset.
Figure S13 : Visual examples and comparisons on 2K-resolution images from CLIC2020-test dataset.
Figure S14 : Visual examples and comparisons on 2K-resolution images from CLIC2020-test dataset.
Progressive image compression requires a single embedded representation whose received prefixes can be decoded without re-encoding the source. Modern vector-quantized (VQ) image models provide strong discrete endpoint representations, but conventional flat codeword indices do not define meaningful intermediate states for a neural decoder. We present Tree-VQ, a post-hoc conversion of a pretrained flat VQ tokenizer into a fine-grained, arbitrary-prefix progressive representation while preserving its encoder assignments and every learned leaf vector. The key idea is to organize the original codebook into a balanced binary hierarchy, associate explicit representations with internal nodes, and transmit branch decisions in depth-major order. Consequently, once the image header is available, every payload prefix uniquely specifies a valid latent state: each additional branch bit refines exactly one token, and transmission can therefore be truncated at essentially any payload position rather than only at a small number of stage boundaries. We further adapt one shared decoder on the complete-depth and mixed-depth latent states encountered under such arbitrary truncation, making these densely spaced prefixes useful for reconstruction rather than merely syntactically decodable. On Kodak, Tree-VQ achieves a DISTS-based BD-rate saving of 52.1% relative to ProGIC, while exposing thousands of valid arbitrary-prefix operating points from a single embedded bitstream.
Mingming Ma, Xinkun Wang, Tianyi Xu +3
School of Artificial Intelligence Xidian University
Recently, numerous end-to-end optimized image compression neural networks have been developed and proved themselves as leaders in rate-distortion performance. The main strength of these learnt compression methods is in powerful nonlinear analysis and synthesis transforms that can be facilitated by deep neural networks. However, out of operational expediency, most of these end-to-end methods adopt uniform scalar quantizers rather than vector quantizers, which are information-theoretically optimal. In this paper, we present a novel Lattice Vector Quantization scheme coupled with a spatially Adaptive Companding (LVQAC) mapping. LVQ can better exploit the inter-feature dependencies than scalar uniform quantization while being computationally almost as simple as the latter. Moreover, to improve the adaptability of LVQ to source statistics, we couple a spatially adaptive companding (AC) mapping with LVQ. The resulting LVQAC design can be easily embedded into any end-to-end optimized image compression system. Extensive experiments demonstrate that for any end-to-end CNN image compression models, replacing uniform quantizer by LVQAC achieves better rate-distortion performance without significantly increasing the model complexity. Code is available at: https://github.com/xzhang9308/LVQAC.
Xi Zhang, Xiaolin Wu
Shanghai Jiao Tong University · McMaster University
Recent progress in learning-based image compression has demonstrated that end-to-end optimization can substantially outperform traditional codecs by jointly learning compact latent representations and probabilistic entropy models. However, many existing approaches achieve high rate-distortion efficiency at the expense of increased computational cost and limited parallelism. This paper presents ARCHE - Autoregressive Residual Compression with Hyperprior and Excitation, an end-to-end learned image compression framework that balances modeling accuracy and computational efficiency. The proposed architecture unifies hierarchical, spatial, and channel-based priors within a single probabilistic framework, capturing both global and local dependencies in the latent representation of the image, while employing adaptive feature recalibration and residual refinement to enhance latent representation quality. Without relying on recurrent or transformer-based components, ARCHE attains state-of-the-art rate-distortion efficiency: it reduces the BD-Rate by approximately 48% relative to the commonly used benchmark model of Balle et al., 30% relative to the channel-wise autoregressive model of Minnen & Singh and 5% against the VVC Intra codec on the Kodak benchmark dataset. The framework maintains computational efficiency with 95M parameters and 222ms running time per image. Visual comparisons confirm sharper textures and improved color fidelity, particularly at lower bit rates, demonstrating that accurate entropy modeling can be achieved through efficient convolutional designs suitable for practical deployment.
Sofia Iliopoulou, Dimitris Ampeliotis, Athanassios Skodras
Dept. of Electrical and Computer Engineering, University of Patras, Patras, Greece · Dept. of Digital Media and Communication, Ionian University, Argostoli, Greece