The rapid growth of visual data under stringent storage and bandwidth constraints makes extremely low-bitrate image compression increasingly important. While Vector Quantization (VQ) offers strong structural fidelity, existing methods lack a principled mechanism for joint rate-distortion (RD) optimization due to the disconnect between representation learning and entropy modeling. We propose RDVQ, a unified framework that enables end-to-end RD optimization for VQ-based compression via a differentiable relaxation of the codebook distribution, allowing the entropy loss to directly shape the latent prior. We further develop an autoregressive entropy model that supports accurate entropy modeling and test-time rate control. Extensive experiments demonstrate that RDVQ achieves strong performance at extremely low bitrates with a lightweight architecture, attaining competitive or superior perceptual quality with significantly fewer parameters. Compared with RDEIC, RDVQ reduces bitrate by up to 75.71% on DISTS and 37.63% on LPIPS on DIV2K-val. Beyond empirical gains, RDVQ introduces an entropy-constrained formulation of VQ, highlighting the potential for a more unified view of image tokenization and compression. The code will be available at https://github.com/CVL-UESTC/RDVQ.
Figures & tables
Figure 1: Gradient-based rate control in VQ compression is challenging due to non-differentiable indices.
Figure 2: Overview of RDVQ. The analysis transform ga extracts multi-scale features, which are flattened into a sequence y for vector quantization and entropy modeling. The VQ module produces hard-quantized embeddings yq , discrete indices yind , and a relaxed distribution psoft . During training, reconstruction is performed from yq , while the rate objective is computed as the cross-entropy between the relaxed distribution psoft and the entropy model prediction qψ , enabling gradient flow from the rate loss to the encoder. The same model is used for entropy coding and supports test-time index completion. Right: Details of the VQ module and gradient flow. Hard assignment is used for reconstruction and coding, while the relaxed distribution is used only for rate estimation, restoring gradient flow from the rate loss ( LR ) to the encoder ga .
Figure 3: Dependency-aware ordering for autoregressive entropy modeling. Scale-wise spatial orders are defined within each scale and concatenated into a unified order vector o , from which the attention mask M=(o>o⊤) is constructed.
Figure 4: Rate-distortion curves on the Kodak, the DIV2k-val and the CLIC2020-test datasets.
Figure 5: Visual examples on the CLIC2020 test set. Zoom in for better view.
Figure 6: Model efficiency comparison on DIV2K [ 1 ] . RDVQ achieves the best BD-DISTS with less than 20% of the parameters of most baselines, while maintaining competitive latency.
Figure 7: Visual examples on the Kodak dataset.
Figure 8: Codebook usage becomes more concentrated under stronger rate constraints.
Figure 9: Test-time rate adjustment via prefix transmission. Left: Rate–distortion curves on Kodak. Right: Visual comparisons at similar bitrates (Bpp / DISTS). RDVQ-Adj maintains competitive performance with gradual degradation, while AE-Entropy and Zero-padding suffer from more noticeable artifacts.
Method
bpp ↓
DISTS ↓
LPIPS ↓
FID ↓
RDVQ (full)
0.0247
0.1005
0.2321
19.96
w/o Relaxation
0.0464
0.2147
0.5031
86.93
K-means VQ
0.0247
0.1253
0.2831
28.08
Table 1: Quantitative comparison of ablation variants on DIV2K-val. Lower DISTS, LPIPS, FID indicate better perceptual quality.
Figure 10: PCA visualization of the largest-scale encoder features under different compression ratios. As the bitrate decreases, the features become progressively smoother, with high-frequency details suppressed.
Figure S1 : Effectiveness of tuning on high-quality images (evaluated on Div2K validation set).
Figure S2 : Effectiveness of piecewise strategy (evaluated on Div2K validation set).
Method
Ours
CGIC [ 33 ]
Diffo [ 54 ]
OneDC [ 67 ]
BD-DISTS (%, Kodak)
0.0
+5422.8
+207.11
+18.81
Table S1 : Comparison with additional VQ-based methods on the Kodak dataset in terms of BD-DISTS. Lower is better.
Figure S3 : Rate-distortion curves on the CLIC 2020 testset.
Figure S4 : Rate-distortion curves on the DIV2K-val dataset.
Figure S5 : Rate-distortion curves on the Kodak dataset.
Figure S6 : Test time rate control examples on CLIC2020 testset.
Figure S7 : Test time rate control examples on DIV2K-val dataset.
Figure S8 : Visual examples and comparisons on 2K-resolution images from CLIC2020-test dataset.
Figure S9 : Visual examples and comparisons on 2K-resolution images from CLIC2020-test dataset.
Figure S10 : Visual examples and comparisons on 2K-resolution images from CLIC2020-test dataset.
Figure S11 : Visual examples and comparisons on 2K-resolution images from CLIC2020-test dataset.
Figure S12 : Visual examples and comparisons on 2K-resolution images from CLIC2020-test dataset.
Figure S13 : Visual examples and comparisons on 2K-resolution images from CLIC2020-test dataset.
Figure S14 : Visual examples and comparisons on 2K-resolution images from CLIC2020-test dataset.
Dept. of Electrical and Computer Engineering, University of Patras, Patras, Greece · Dept. of Digital Media and Communication, Ionian University, Argostoli, Greece