Benchmarking Neural Image Compression with DCT-Based Spectral Diagnostics
Authors: Nikolay I. Kalmykov, Razan Dibo, Yujun Du, Yipeng Liu, Ivan Oseledets, Anh-Huy Phan
Organizations: Skolkovo Institute of Science and Technology (Skoltech), Moscow, Russia · University of Electronic Science and Technology of China (UESTC), Chengdu, China · AIRI, Moscow, Russia
Neural image compression (NIC) models are commonly evaluated with scalar rate-distortion metrics, which do not show how a codec treats individual spatial frequencies. We propose a benchmark that probes codecs with the DCT basis matrix and single-frequency test images. We measure relative off-target energy (leakage), the fraction of reconstruction energy outside the probed frequency, together with target-frequency gain. Exact decompositions and bounds relate these measurements to reconstruction error. At operating points nearest 1 bpp, the BMSHJ, MBT, and Cheng2020 models leave 80-97% of their reconstructed high-band energy outside the target bin and have target-frequency gains of 0.02-0.23, whereas TCM and FTIC attain off-target energy in the classical-codec range. Both probe types identify behavior absent from scalar metrics: released Cheng2020-Anchor models produce extreme pre-clamp outputs on isolated single-frequency inputs. Finally, decoder-only fine-tuning with natural-image and leakage losses reduces basis off-target energy by 90-99.8% across all eight evaluated architectures while keeping the bitstream fixed, at a Kodak PSNR cost of 0.09-0.59 dB.
Figures & tables
Figure 1: (a) Benchmark with a normalized 128×128 DCT basis matrix. (b) Diagonal-direction frequency map, with dark = low and bright = high diagonal frequency, computed from (a). (c) Cheng2020-Anchor ( q=6 ) reconstruction: the low-frequency upper-left region remains close to the input, whereas the high-frequency lower-right region is visibly altered.
Figure 2: DCT-basis response at the nearest available operating point to ≈ 1.0 bpp (per-panel bpp shown), 256×256 , for eleven codecs plus the GAN-based HiFiC codec (bottom-right, at its highest fixed rate, 0.70 bpp; best viewed zoomed in). The three classical codecs are left of the dashed line; the learned codecs follow. Under each panel: median leakage Lk ( ↓ better, colored green → red by magnitude), its low / high -band values, and the measured bitrate. At these nearby rates the high-frequency leakage gap between learned and classical codecs persists ( Lhigh : 0.02 - 0.15 for traditional codecs vs. 0.80 - 0.97 for the BMSHJ/MBT/Cheng families); TCM and FTIC fall in the classical range under this diagnostic. HiFiC ( Lk=0.62 ) shows structured content in the region with high measured leakage rather than the smooth blur of the MSE-trained decoders.
Figure 3: Frequency-domain analysis of Cheng2020-Anchor on a 128×128 DCT basis ( q=6 ). (a) The frequency-response matrix shows strong off-target redistribution in high-frequency components (top-right) and near-perfect target-bin purity in low-frequency components (bottom-left). (b) Frequency-aware metrics quantify these effects across low, mid, and high bands (in the panel legend, pk denotes the retention Rk,k , so leakage is 1−pk ). (c) Leakage Lk before vs. after decoder-only fine-tuning with the proposed leakage loss ( 900 steps; protocol in Sec. 5.5 ); percentages indicate per-band reduction.
Figure 4: DCT rate-leakage comparison at 256×256 . Traditional codecs (dashed) achieve lower median leakage across all bitrates than the older BMSHJ/MBT/Cheng models (solid), while TCM and FTIC approach classical levels. The GAN-based HiFiC is shown at its three fixed operating points (black stars).
Codec
bpp
Lhigh↓
gain gk↑
BMSHJ2018-Factorized
0.92
0.94
0.05
BMSHJ2018-Hyperprior
1.13
0.85
0.09
MBT2018-Mean
0.68
0.97
0.02
MBT2018
1.24
0.84
0.23
Cheng2020-Anchor
1.04
0.80
0.20
Cheng2020-Attention
0.99
0.87
0.15
Table 1: Composite-basis response of learned codecs at their settings nearest 1.0 bpp ( 256×256 ). Values are means over the top-quarter band ( k≥3n/4 ); best values are bold. Classical codecs are not included.
Figure 5: Released-checkpoint instability of Cheng2020-Anchor: clamped reconstructions of neighboring Xk(1) gratings with matched peak and RMS ( n=256 , peak 0.45 , q=6 ). The k=62 input produces a localized chromatic artifact (pre-clamp error 1.9×104 ), whereas k=61 is reconstructed with error 0.002 . Full unclamped sweeps: Suppl. Figs. S2 and S3 .
Figure 6: Size and quality dependence of the composite-basis response. Cheng2020-Anchor becomes non-monotonic for n≥256 and produces visible artifacts at 1024×1024 .
Codec
Lk↓
bpp
PSNR ↑ (dB)
MS-SSIM ↑
LPIPS ↓
L~
L~/ρˉ
BMSHJ2018-Factorized
0.455
0.98±0.40
35.17±2.77
0.989±0.005
0.087±0.064
0.1486±0.0225
0.6872
Cheng2020-Anchor
0.729
0.82±0.44
36.47±2.43
0.990±0.004
0.074±0.061
0.1372±0.0402
0.7252
Cheng2020-Attention
0.296
0.82±0.44
36.37±2.40
0.990±0.004
0.077±0.063
0.1067±0.0192
0.5576
BMSHJ2018-Hyperprior
0.146
0.96±0.48
36.53±2.03
0.990±0.004
0.082±0.063
0.0825±0.0192
0.4432
MBT2018-Mean
0.116
0.95±0.50
36.71±1.97
0.990±0.004
0.079±0.062
0.0509±0.0099
0.2799
MBT2018
0.115
0.92±0.50
36.92±2.02
0.991±0.004
0.070±0.056
0.0453±0.0088
0.2527
Table 2: Natural-image evaluation of eleven codecs on 115 images (24 Kodak, 41 CLIC, 50 DIV2K) at per-dataset settings near 0.9 bpp. Learned codecs use fixed settings; classical codecs are rate-selected per dataset. Lk is median leakage on the synthetic 512×512 basis (for classical codecs, the median over three selected settings). L~ and L~/ρˉ are descriptive spectral-alignment statistics. Entries with ± report mean and standard deviation over images; best displayed values in directed columns are bold. Rows are ordered by decreasing L~ .
MSE + leakage
MSE-only control
Codec
λfreq
step
Lbeforemed
Laftermed
Δ PSNR
Δ LPIPS
Laftermed ( Δ PSNR)
BMSHJ2018-Fact.
0.003
2000
0.4558
0.0437
-0.59
−5e−5
0.3812 (-0.02)
BMSHJ2018-Hyp.
0.01
2000
0.1474
0.0034
-0.26
+7e−4
0.1322 (-0.15)
MBT2018-Mean
0.01
1500
0.1244
0.0042
-0.29
+2.3e−3
0.1130 (-0.17)
MBT2018
0.003
2000
0.1151
0.0045
-0.21
+2.4e−3
0.0867 (-0.11)
Cheng2020-Anchor
0.03
500
0.7697
0.0012
-0.17
+6e−4
0.1431 (-0.02)
Table 3: Leakage-regularized decoder fine-tuning: DCT-basis leakage and Kodak quality, with an MSE-only control. Lbefore/aftermed is the median leakage on the 256×256 composite basis; Δ metrics are reported on Kodak ( Δ : fine-tuned minus pretrained) with the bitstream fixed. Accepted operating points minimize Lmed subject to 0.1 dB validation-PSNR limit. The MSE-only value is selected independently by the same rule. The BMSHJ2018-Factorized step-2000 endpoint exceeds this limit and is shown for completeness. Full protocol: Suppl. Sec. F.1 .
Figure 7: Leakage-regularized decoder fine-tuning on a held-out CLIC image. For Cheng2020-Anchor ( q=6 ), the selected decoder removes the chromatic artifact from the same bitstream and raises PSNR from 26.9 to 31.2 dB.
Notation
Definition
n,k
DCT size and frequency index, k∈{0,…,n−1} .
D,dk
Orthonormal DCT-basis image and its k -th column.
D,D2
Orthonormal one- and two-dimensional DCT operators.
dk,X
Reconstructed basis-matrix column and isolated probe.
Ri,k
Fraction of reconstructed column k ’s DCT energy located in bin i .
Lk,gk
Relative off-target energy and signed target-frequency gain.
Table S1 : Core notation used in the main paper and Supplementary Material.
Figure S1 : Isolated probes ( n=64 , k=6 ). Xk(1) repeats one basis wave along every column: a horizontal grating with all spectral energy in bin (k,0) . Xk(2) is separable in both axes: a checkerboard with all energy in bin (k,k) .
Figure S2 : Unclamped squared reconstruction error on the Xk(2) sweep ( n=256 , peak 0.45 ) for the six released Cheng2020-Anchor checkpoints. The N=128 , q=1 - 3 checkpoints have maximum errors below 500 ; the N=192 , q=4 - 6 checkpoints reach 1.9×1011 at isolated indices (dotted line: unit input energy).
Figure S3 : Instability of the grating probe Xk(1) ( n=256 , peak amplitude 0.45 ): selected stimuli and their Cheng2020-Anchor q=6 reconstructions (clamped to [0,1] for display). Red titles mark frequencies whose unclamped squared error exceeds 10 ; artifacts range from a localized chromatic region ( k=62 ) to global collapse ( k=253 ), with interleaved stable indices.
Figure S4 : Leakage-regularized decoder fine-tuning trajectories at the λfreq used in Table 3 : median DCT-basis leakage ( 256×256 ) vs. Δ PSNR on the Kodak images. Small circles show checkpoints evaluated every 250 steps; arrows connect each pretrained decoder to its selected checkpoint (star). For the seven codecs accepted under the −0.1 dB validation limit, leakage decreases by 94.7 - 99.8% and Kodak PSNR decreases by at most 0.29 dB. BMSHJ2018-Factorized reduces leakage by 90% , but has ΔPSNRval=−0.416 dB.
Codec
DCT size
Ltrainmed before
Ltrainmed after
Δ PSNR (dB)
BMSHJ2018-Fact.
128×128
0.445
0.008
-22.0
BMSHJ2018-Hyp.
128×128
0.163
0.003
-13.9
MBT2018-Mean
128×128
0.148
0.002
-16.5
MBT2018
128×128
0.125
0.002
-12.0
Cheng2020-Anchor
128×128
0.287
<0.001
-10.6
Cheng2020-Attn.
128×128
0.269
<0.001
-11.5
Table S2 : Basis-only decoder fine-tuning as a negative control. The decoder is optimized on one fixed DCT-basis input ( 128×128 ; TCM and FTIC: 256×256 ) without natural images. Ltrainmed is the median leakage on that same training probe; Δ PSNR is fine-tuned minus pretrained performance on held-out Kodak images. The bitrate is unchanged because the encoder and entropy model remain frozen.
Figure S5 : Affected near-periodic cases: Urban100 facade, DTD lace, and DTD parallel lines. Cheng2020-Anchor ( q=6 ) produces localized chromatic artifacts; the fine-tuned decoder processes the same bitstream. FTIC and JPEG XL are shown at the available settings nearest the Cheng bitrate.
Figure S6 : Two affected periodic textures (DTD zigzagged and meshed) and one minimally affected irregular texture (DTD cracked). The final row is a contrast case: high-frequency content alone does not imply a severe failure. Layout and overlays follow Fig. S5 .
Department of Information Engineering Shanghai University of Electric Power Shanghai, China · Department of Computer Science and Communication Engineering Waseda University Tokyo, Japan · Faculty of Engineering Institute of Science Tokyo Tokyo, Japan