Local-Order Auxiliary Losses Can Improve Autoencoder Reconstruction
Organizations: Kahlert School of Computing University of Utah Salt Lake City, UT, USA · Department of Computer Science Texas State University San Marcos, TX, USA
Abstract
Mean-squared error is the default objective for training autoencoders, yet compressed reconstructions often depend not only on pointwise accuracy but also on preserving local spatial order. We study whether structural auxiliary losses can improve, rather than trade off against, MSE in finite-capacity autoencoders. We introduce finite-difference sign error (FDSE), a local-order auxiliary objective that penalizes disagreements between the signs of neighboring finite differences in the target and reconstruction. FDSE is simple, architecture-agnostic, and differentiable through smooth sign surrogates. Across four tensor reconstruction tasks, we find that moderate mixtures of MSE and FDSE can substantially reduce validation MSE relative to pure MSE training. In coefficient sweeps, FDSE mixtures reduce validation MSE by 2.3--7.0 over pure MSE on these tasks, while comparisons with other auxiliary objectives show FDSE to be among the strongest structural objectives tested. The effect is not universal: pure FDSE performs poorly, and gains are largest for coherent spatial fields where local order carries information about the underlying signal. These results suggest that, in compressed-latent reconstruction, appropriately weighted local-structure supervision can guide optimization toward solutions with better pointwise accuracy, rather than merely improving perceptual or structural metrics at MSE's expense.
Figures & tables
| Objective | Time / MSE | Peak GPU memory / MSE |
|---|---|---|
| FDSE | – | – |
| Lap 2 residual | – | – |
| L1 | – | |
| Grad. magnitude | – | – |
| SSIM | – | – |
| TV | – | – |
| Dataset | Autoencoder | Method | MSE | SSIM | PersD | MergD | CorrD | Own tgt | Train (min) |
|---|---|---|---|---|---|---|---|---|---|
| ERA5 PV | NDAE | MSE | — | ||||||
| MSE+ FDSE | |||||||||
| MSE+L1 | |||||||||
| MSE+GradMag | |||||||||
| MSE+SSIM | |||||||||
| MSE+Laplacian |
| Width | MSE, | Proj. MSE | MSE+ FDSE |
|---|---|---|---|
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Model / input | Batch | Train budget | Early stopping | Notes |
|---|---|---|---|---|---|
| Shallow water § 4.2 | CRA5 [ 7 ] on tensors (three prognostic channels eight stored snapshots); quality , embed. , channels , channels . | training batches | Patience validation checks (every batches) | On-the-fly simulator batches; validation uses batches per check. | |
| ERA5 PV § 4.2 | NDAE on non-overlapping blocks (four pressure levels); base width , multipliers , bottleneck , one residual block per level. | epochs | Patience epochs | Non-overlapping blocks are enumerated deterministically in file, time, latitude, and longitude order. | |
| PDEBench RDB § 4.2 | MBT-ND on scalar fields (time index ); train/validation split of the first trajectories in the public PDEBench RDB file; , analysis channels , latent channels , bottleneck width . | epochs (Figure 2 ) or epochs (Table 8 ) | Patience epochs (min. improvement ) | Examples are the fixed time-index-50 snapshots from the selected trajectories. | |
| CIFAR-10 § 4.2 | Ballé-style factorized-prior image autoencoder at quality (latent width ); inputs in . | epochs | Patience epochs | Fixed train/validation partitions for the reported sweep. |
| Block (main text) | Architecture | Mixture / calibration | BS | Epochs / pat. | Data / subset |
|---|---|---|---|---|---|
| ERA5 PV, | NDAE ; the published study includes bottleneck widths , and Table 2 reports the family. | Fixed on mixed rows; pure MSE baseline. | / | train and validation blocks. | |
| PDEBench RDB, | MBT-ND on RDB time-index-50 fields; , analysis , latent ; bottleneck width . | Gradient-matched auxiliary scale, then fixed on mixed rows; pure MSE baseline. | / | split of the first trajectories (time index ). | |
| Shallow water, CRA5 | CRA5 quality with MBT-style bottleneck column ; embed. , , . | Fixed on mixed rows; pure MSE baseline. | / | generated train tensors and validation tensors ( ). | |
| CIFAR-10, BM-SHJ | Ballé-style factorized-prior image autoencoder on RGB tensors of shape . | Fixed on mixed rows (MSE weight ); pure MSE baseline. | / | Standard CIFAR-10 train/val; example subsets. |
| Dataset | Autoencoder | Latent / bottleneck | Auxiliary-training protocol | Max topo | Val cap | |
|---|---|---|---|---|---|---|
| ERA5 PV | NDAE | Latent bottleneck | ERA5 bottleneck auxiliary study; per-batch value normalization before mixing; weights chosen by validation MSE within the study. | 1024 | 32 | (full) |
| PDEBench RDB | MBT-ND | Latent bottleneck | PDEBench MBT-ND gradient-matched study; fixed MSE:auxiliary after matching the MSE-parallel gradient component; weights chosen by validation MSE within the study. | 100 | 8 | (full) |
| Shallow water | CRA5 | CompressAI quality ; MBT-style latent bottleneck width (matches saved bn8 model weights in the study). | CRA5 bottleneck auxiliary study at fixed MSE:auxiliary ; per-batch value normalization; weights chosen by validation MSE within the study. | 32 | 32 | 32 |
| CIFAR-10 | BM-SHJ | Quality ( bottleneck_channels =2 in the eval JSON). The six-mixture MSE: FDSE coefficient sweep on the same backbone is a different training protocol; its structural re-eval on the first 32 validation images is Figure 2 ( 2(d) ). | CompressAI bmshj2018_factorized on tensors of shape . The tabulated autoencoders are trained in cifar10_latent32_m060_aux040 with fixed MSE:auxiliary on mixed rows ( mse_only is pure MSE); per-batch value normalization before mixing; weights chosen by validation MSE within the study. Structural topology ( PersD , MergD , CorrD ) is computed on each validation image as a field, then mean std are reported across the capped validation subset. Legacy cifar10_bottleneck_mechanism_study ConvAE runs (bottleneck width 32, fixed ) use a different architecture and are not the source of this row. | 32 | 32 | 32 |
| Method | Validation MSE | Exact sign loss | Aux. weight | Epochs | Train (min) |
|---|---|---|---|---|---|
| MSE | — | ||||
| MSE+ FDSE | |||||
| MSE+Laplacian | |||||
| MSE+Spectral | |||||
| MSE+GradMag | |||||
| MSE+L1 |
| Method | MBT-ND val. MSE |
|---|---|
| Fixed (pure FDSE ) | |
| Fixed | |
| Fixed | |
| Fixed | |
| Fixed | |
| Fixed (pure MSE) |
| Method | Best Val. MSE | Train (min) | |
|---|---|---|---|
| 32 | MSE ( , baseline) | ||
| MSE-long ( , no patience) | |||
| MSE+ FDSE ( , ) | |||
| MSE ( ) | |||
| MSE+ FDSE ( , ) | |||
| MSE-only (proj.-matched ) |
| Training state | Orth. frac. | |||
|---|---|---|---|---|
| Initialization (epoch 0) | ||||
| After one epoch | ||||
| Best validation state (epoch 57) |
| rank | Loss | time ( s) [-0.2ex] mean std | GPU (MB) [-0.2ex] mean std |
|---|---|---|---|
| 1D | MSE | ||
| FDSE | |||
| Lap 2 | |||
| conv CP | |||
| L1 | |||
| rank | Loss | time ( s) [-0.2ex] mean std | GPU (MB) [-0.2ex] mean std |
|---|---|---|---|
| 1D | MSE | ||
| FDSE | |||
| PersD | |||
| MergD | |||
| CorrD | |||
| 2D | MSE |
Autoencoders
Autoencoders learn compressed representations by encoding data into latent codes and decoding them for reconstruction, generation, denoising, or anomaly detection.
See all 165 papers