Lossy compression of scientific simulation data increasingly relies on learned, latent-space architectures such as Residual Vector Quantization (RVQ), which iteratively quantize a base representation and its residuals to progressively reduce reconstruction error. While effective, RVQ performs this residual modeling entirely in latent space, leaving the pixel-space error structure of the reconstruction largely unaddressed. In this work, we propose a post-processing pipeline that augments an RVQ-based compressor with a U-Net trained to predict and correct pixel-space residuals between the original volume and its RVQ reconstruction. We show that these residuals are spatially structured rather than driven by local intensity or gradient features, motivating the need for a deep spatial model rather than simple statistical correction. The U-Net-corrected reconstruction is then passed through a Guaranteed Autoencoder (GAE) stage, which projects the remaining residual onto a per-block PCA basis to enforce a user-specified block-wise error bound. To the best of our knowledge, this is the first pipeline to combine latent-space RVQ, explicit pixel-space residual correction via a deep spatial post-processing network, and GAE-based error-bound guarantees within a single framework for scientific data compression. We evaluate our approach on S3D, JHTDB and E3SM datasets, demonstrating consistent improvements in NRMSE, compression ratio] over RVQ-only and standard residual-correction baselines, while maintaining strict error guarantees required for scientific data fidelity.
Figures & tables
Figure 1: Overview of the proposed three-stage compression pipeline. The input volume is tiled into spatial blocks and passed through an RVQ-based VAE (Stage 1) to obtain a pixel-space reconstruction x~ . A pixel-space U-Net (Stage 2) predicts the residual r^=x−x~ and produces a refined reconstruction x~refined=x~+r^ . A Guaranteed Autoencoder (Stage 3) then projects the remaining residual onto a per-block PCA basis to enforce a user-specified NRMSE bound τ , producing the final certified output xG .
Figure 2: Correlation between RVQ pixel-space residual magnitude and local structural features on the JHTDB dataset. Scatter plots show residual magnitude against gradient magnitude (column 2), Laplacian response (column 3), local variance (column 4), and temporal difference (column 5), each computed per-pixel. The low correlation across all four features indicates that no simple pixel-wise or gradient-wise statistic explains where RVQ reconstruction fails, motivating a deep spatial model capable of learning the more complex underlying pattern. This is further highlighted by the high correlation between the true and predicted residuals from the U-Net (column 1) showing that the U-Net is able to capture some spatial features resulting in proper residual predictions.
Figure 3: Final NRMSE vs. Compression Ratio on the E3SM dataset. RVQ (CB: 4, Size: 64) is the Stage 1 baseline. U-Net Residual (L1) with 0.462 MB and 7.349 MB model sizes correspond to the small ( b=8 ) and large ( b=32 ) U-Net variants respectively. The U-Net correction consistently shifts the curve downward, achieving lower NRMSE at the same compression ratio across the full operating range.
Figure 4: Final NRMSE vs. Compression Ratio on the JHTDB dataset. The turbulence dataset presents the most structured and high-frequency residuals among our three benchmarks. Both U-Net variants improve over RVQ alone, with the larger model providing a more consistent gain especially at high compression ratios.
Figure 5: Final NRMSE vs. Compression Ratio on the S3D dataset. The smaller U-Net variant improves over the RVQ baseline with the larger model achieving a more substantial gain in NRMSE. But we take a huge hit in the compression ratios. This highlights a case where the U-Net model size matters since we are working with a comparatively smaller dataset.
Figure 6: Ablation study of U-Net loss functions on the S3D dataset. All loss variants substantially outperform the RVQ-only baseline. Robust losses like ℓ1 , Smooth L1, Huber, Log-Cosh, and Pseudo-Huber, cluster tightly together and outperform MSE, suggesting that residual-robust loss functions are more appropriate for the heavy-tailed residual distribution typical of learned scientific data compressors.
Figure 7: Qualitative visualization for a sample representative JHTDB tile and frame. Top row (left to right): original field, RVQ reconstruction, predicted residual r^ , refined reconstruction x~refined=x~+r^ . Bottom row: absolute error map ∣ RVQ − Original ∣ , absolute error map ∣ Refined − Original ∣ , predicted residual histogram, and true vs. predicted residual overlay. The U-Net substantially reduces the absolute error (visible as reduced intensity in the error map), and the predicted residual distribution closely matches the true residual distribution, confirming that the U-Net has learned a useful approximation to the pixel-space reconstruction error.
Method
NRMSE
CR
SZ3 only
9.956×10−4
29.52
SZ3 + U-Net (small)
6.163×10−4
12.99
Table 2: Comparison of SZ3 alone vs. SZ3 + U-Net (small) on the S3D dataset at target NRMSE =1×10−3 . The U-Net model overhead is included in the total bitstream for the SZ3 + U-Net case.