Rare-Aware Autoencoding: Reconstructing Spatially Imbalanced Data

Alejandro Castañeda Garcia; Jan van Gemert; Daan Brinks; Nergis Tömen

cs.CVApr 2, 2026

Rare-Aware Autoencoding: Reconstructing Spatially Imbalanced Data

Authors: Alejandro Castañeda Garcia, Jan van Gemert, Daan Brinks, Nergis Tömen

Organizations: Delft University of Technology (TU Delft), Delft, The Netherlands

Abstract

Autoencoders can be challenged by spatially non-uniform sampling of image content. This is common in medical imaging, biology, and physics, where informative patterns occur rarely at specific image coordinates, as background dominates these locations in most samples, biasing reconstructions toward the majority appearance. In practice, autoencoders are biased toward dominant patterns resulting in the loss of fine-grained detail and causing blurred reconstructions for rare spatial inputs especially under spatial data imbalance. We address spatial imbalance by two complementary components: (i) self-entropy-based loss that upweights statistically uncommon spatial locations and (ii) Sample Propagation, a replay mechanism that selectively re-exposes the model to hard to reconstruct samples across batches during training. We benchmark existing data balancing strategies, originally developed for supervised classification, in the unsupervised reconstruction setting. Drawing on the limitations of these approaches, our method specifically targets spatial imbalance by encouraging models to focus on statistically rare locations, improving reconstruction consistency compared to existing baselines. We validate in a simulated dataset with controlled spatial imbalance conditions, and in three, uncontrolled, diverse real-world datasets spanning physical, biological, and astronomical domains. Our approach outperforms baselines on various reconstruction metrics, particularly under spatial imbalance distributions. These results highlight the importance of data representation in a batch and emphasize rare samples in unsupervised image reconstruction. We will make all code and related data available.

Figures & tables

Figure 1: Spatial data imbalance. Top: example frames from a pendulum dataset at different angles. Due to damping, the pendulum spends more time near equilibrium, making some spatial configurations much more frequent than others. Top-right: per-pixel variance over the full dataset, highlighting regions with high motion-induced variability. Bottom-left: temporal intensity traces for four representative pixels: a static background pixel (blue), an extreme-angle pixel (red), a mid-position pixel (green), and an equilibrium pixel (orange). Variance increases as pixels approach the equilibrium region, since the pendulum passes there most often. Bottom-right: histogram of pixel intensities (for the selected pixels), showing the highly non-uniform occupancy of the state space and the resulting spatial imbalance.

Figure 2: Addressing reconstruction under spatial data imbalance. Consider a dataset of simulated pendulum dynamics. Black squares are input or reconstructed frames. Left-top: Input distributions exhibit a strong bias, with common, overrepresented samples (blue) and underrepresented ones (red). Left-bottom: Standard autoencoder (AE) backbone (encoder–latent–decoder). The SOTA (cyan) approach tends to converge toward smooth reconstructions dominated by frequent samples. Right: Our approach (orange) introduces two complementary mechanisms: (i) a self-entropy loss that upweights high-surprisal pixels across the batch within each image, and (ii) sample propagation that replays high-loss (hard) samples across epochs, ensuring rare events are repeatedly observed. The combined effect yields sharper and more balanced reconstructions.

Figure 3: Sample propagation algorithm. First, for each input sample, the model produces a reconstruction error. The color scale encodes error magnitude: red denotes higher error, while decreasing progressively to cooler hues. Second, we compute a error per sample and sort/rank the batch accordingly, yielding an ordered list from highest to lowest reconstruction error. Finally, a propagation step selects the

M

top-ranked (hardest) samples for targeted optimization and concatenates them to the next batch; gradients are then applied (loss backward) to update the network with an emphasis on hard examples.

Figure 4: Comparison between balanced and imbalanced reconstruction scenarios. The MNIST dataset (left) represents a balanced case where samples are more uniformly distributed across the spatial domain. In contrast, the damped pendulum dataset (right) introduces imbalance, as certain image spatial locations are severely underrepresented across the dataset and within training batches. Our proposed spp+entropy framework reconstructs both balanced and imbalanced data faithfully, while the standard MLP AE pipeline performs well only on homogeneous datasets such as MNIST but fails to preserve structure under imbalance.

Panel of Figure 6 — Figure 6: Hypothesis testing on synthetic data. On figure (a) we show how our entropy loss reduce the error of individual pixel equally, while on (b) sample propagation reduces the error on underrepresented samples having a similar performance in all samples.

Figure 7: Latent–capacity comparison under restricted bottlenecks. Expressivity in the AE latent space when its dimensionality is intentionally constrained to 1D. We report the time–series plots that trace the decoded signal across samples. Under tight bottlenecks, baseline trajectories either collapse (L2) or do not follow the oscillations faithfully (BEL), whereas our method (spp+entropy) has rich latent dynamics that remain coupled to the input sequence. Example reconstructions from each method (upper left) illustrate it.

Panel of Figure 8 — Figure 8: Summary figure of our extensive results detailed in appendix tables 8 to 12 . Fig. 8(a) shows the performance of our entropy* loss function and 8(b) the results for our spp algorithm. Both sub-figures show the performance on three different datasets Pendulum , Galaxy , and Mitosis over four autoencoder families (AE, VAE, SAE, MAE). For better visualization, we re-normalized the metrics from $0.2\;\textrm{to}\;1.0$ where high is always better. In the presented radar plots more area implies a better overall reconstruction. Baseline methods are shown in shades of blue while our methods are in a red scale.

Figure 9: Qualitative comparison under convergence: convergence toward the statistical mean vs. detail preservation. For representative samples from Pendulum , Galaxy , and Mitosis datasets, we show GT and reconstructions with baselines and our methods. The shown samples were randomly drawn from the set; for clarity, we display the architecture per dataset where the effect is most legible, while other samples show similar tendencies. Pixel/perceptual baselines frequently converge to a dataset “mean” appearance—yielding blurred trajectories, deleting stars, and washed-out cellular boundaries, our losses retain more of the high-frequency structure and rare patterns, avoiding averaging and preserving relatively sharper edges and textures.

Figure 10: Qualitative comparison with underrepresented-sample focusing (spp $k$ ). Representative cases from the Pendulum , Galaxy , and Mitosis datasets. We compare GT against L2/Focal/onbs/dcl/svp and our spp2*, spp4*, spp8* methods. The shown samples were randomly drawn from the test sets; for clarity, we display the architecture per dataset where the effect is most legible, while other examples show similar tendencies. Pixel/perceptual baselines often smooth toward frequent patterns—fading spiral arms, indistinct mitotic contours, and blurred pendulum traces—whereas spp $k$ preserves rare/high-frequency structure, yielding relatively crisper edges and textures.

Task	Model	Metrics
Generation		FID	LPIPS
	BEL [ 7 ]	280.5	0.3467 $\pm$ 0.0405
	Ours	221.8	0.2826 $\pm$ 0.0512
Anomaly detection		Image AUROC	pixel AUROC
	AESc [ 13 ]	0.6017	0.7853
	Ours	0.6247	0.7875

Table 2: Downstream task performance (bold is better). In this table we compare three different task showing how our method can aid task by increasing the performance.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Layer	In	Out	Output Shape
Encoder
1	Linear	$c\,h\,w$	$c\,h\,w/d_{1}$	$(\tfrac{c\,h\,w}{d_{1}})$
	ReLU			$(\tfrac{c\,h\,w}{d_{1}})$
2	Linear	$c\,h\,w/d_{1}$	$c\,h\,w/d_{2}$	$(\tfrac{c\,h\,w}{d_{2}})$
	ReLU			$(\tfrac{c\,h\,w}{d_{2}})$
3	Linear	$c\,h\,w/d_{2}$	$l$	$(l)$

Appendix

Table 3: MLP Autoencoder architecture. The input image of size

(c,h,w)

is flattened to dimension

c{\cdot}h{\cdot}w

, where

c

is the number of channels and

h,w

are the spatial dimensions. The encoder compresses the input into a latent vector of dimension

l

. The reduction factors

d_{i}

denote successive proportions used to shrink the layer dimensionality relative to the input size. The decoder mirrors this process to reconstruct.

Stage	Layer	In	Out	Kernel / Stride / Pad	Output Shape
Encoder
1	Conv2d	$c$	$n_{1}$	$3\times 3,\;s=2,\;p=1$	$(n_{1},h_{1},w_{1})$
1	ReLU				$(n_{1},h_{1},w_{1})$
2	Conv2d	$n_{1}$	$n_{2}$	$3\times 3,\;s=2,\;p=1$	$(n_{2},h_{2},w_{2})$
2	ReLU				$(n_{2},h_{2},w_{2})$
3	Conv2d	$n_{2}$	$n_{3}$	$3\times 3,\;s=2,\;p=1$	$(n_{3},h_{3},w_{3})$

Appendix

Table 4: Variational Autoencoder (VAE) architecture. The input

(c,h,w)

is encoded through three convolutional stages , where each stage corresponds to a resolution-changing convolution (stride 2) followed by non-linearities. After flattening, fully connected layers reduce the representation to a latent distribution of dimension

l

, parameterized by mean (

\mu

) and variance (

\sigma

). Channel sizes

n_{1},n_{2},n_{3}

are the number of feature maps at each stage, and

(h_{i},w_{i})

are the spatial dimensions after stage

i

. The decoder mirrors this process with three transposed convolution stages , reconstructing the image back to

(c,h,w)

. Intermediate fully connected layers are scaled by proportional factors

d_{1}

and

d_{2}

relative to

n_{3}

Stage	Layer (Parameters)	Output Shape
Encoder
Conv2d	$(c\to n_{1}),\;3\times 3,\;s=2,\;p=1$	$(n_{1},\,h/2,\,w/2)$
ReLU	–	–
Conv2d	$(n_{1}\to n_{2}),\;3\times 3,\;s=2,\;p=1$	$(n_{2},\,h/4,\,w/4)$
ReLU	–	–
Conv2d	$(n_{2}\to n_{3}),\;3\times 3,\;s=2,\;p=1$	$(n_{3},\,h/8,\,w/8)$

Appendix

Table 5: Sparse Autoencoder (SAE) architecture. The encoder reduces the input image

(c,h,w)

through three convolutional layers, producing feature maps

(n_{3},h_{0},w_{0})

with

h_{0}=h/8

and

w_{0}=w/8

. This representation is flattened and mapped by an MLP into a latent vector of dimension

l

, with proportional factors

d_{1}

and

d_{2}

controlling the intermediate hidden sizes. A sparsity regularization is applied on

l

to encourage compact codes. The decoder mirrors this process, expanding the latent code through an MLP, reshaping into convolutional feature maps, and progressively upsampling with transposed convolutions to reconstruct

(c,h,w)

Stage	Module	Configuration	Output Shape
Embed	PatchEmbed	$P\!=\!16$ , $D\!=\!E$ (e.g. 768/1024/1280), $C\!=\!3$	$(N,L,E)$
Embed	Add Pos. Embed (+CLS)	fixed 2D sin-cos	$(N,L\!+\!1,E)$
Encoder	Transformer Blocks ( $\times d$ )	heads $h$ , MLP-ratio $r$	$(N,L\!+\!1,E)$
Encoder	LayerNorm	$\mathrm{LayerNorm}(E)$	$(N,L\!+\!1,E)$
Decode	Linear Embed	$E\to D_{\text{dec}}$ (512)	$(N,L\!+\!1,D_{\text{dec}})$
	Mask Token & Pos. Embed	fixed 2D sin-cos	$(N,L\!+\!1,D_{\text{dec}})$

Appendix

Table 6: Masked Autoencoder (MAE) architecture. The model first embeds the input image

(N,C,H,W)

into non-overlapping patches of size

P\times P

, producing

L=(HW/P^{2})

tokens of embedding dimension

E

, where

C

is the number of input channels. A learnable [CLS] token and fixed 2D sine–cosine positional embeddings are added. The encoder applies

d

Transformer blocks with

h

attention heads and MLP expansion ratio

r

, yielding latent representations of shape

(N,L+1,E)

. The decoder maps embeddings to dimension

D_{\text{dec}}

(e.g., 16), adds mask tokens and positional encodings, and processes them with

d_{\text{dec}}

Transformer blocks with

h_{\text{dec}}

heads. Finally, a projection stage normalizes and linearly predicts patch pixels of size

P^{2}C

, which are reassembled into the reconstructed image

(N,C,H,W)

Condition	Dataset	Model	Parameters
Convergence	Pendulum	AE	$d_{1}=200,\;d_{2}=2000,\;l=8$ , $\lambda=1.0$
		VAE	$n_{1}=8,\;n_{2}=16,\;n_{3}=32,\;d_{1}=16,\;d_{2}=4,\;l=2$ , $\lambda=1.0$
		SAE	$n_{1}=8,\;n_{2}=16,\;n_{3}=32,\;d_{1}=16,\;d_{2}=4,\;l=2$ , $\lambda=10.0$
		MAE	patch size $=16$ , depth $=16$ , heads $=16$ , $\lambda=10.0$
	Mitosis	AE	$d_{1}=200,\;d_{2}=2000,\;l=8$ , $\lambda=1.0$
		VAE	$n_{1}=8,\;n_{2}=16,\;n_{3}=32,\;d_{1}=16,\;d_{2}=4,\;l=2$ , $\lambda=1.0$

Appendix

Table 7: Experimental configurations across conditions, datasets, and models.

Model	Loss	MSE	PSNR	SSIM
AE	L1	$1.126e^{-3}\pm 6.37e^{-4}$	$3.078e^{1}\pm 4.06e^{0}$	$8.777e^{-1}\pm 4.85e^{-2}$
	L2	$8.224e^{-4}\pm 3.35e^{-4}$	$3.127e^{1}\pm 2.01e^{0}$	$8.548e^{-1}\pm 3.29e^{-2}$
	CE	$5.145e^{-3}\pm 4.95e^{-4}$	$2.291e^{1}\pm 4.22e^{-1}$	$7.467e^{-1}\pm 1.33e^{-2}$
	SSIM	$1.588e^{-3}\pm 1.29e^{-4}$	$2.801e^{1}\pm 3.42e^{-1}$	$8.507e^{-1}\pm 1.04e^{-2}$
	FFL	$8.188e^{-4}\pm 3.45e^{-4}$	$3.128e^{1}\pm 1.98e^{0}$	$8.583e^{-1}\pm 3.38e^{-2}$
	ldam	$6.611e^{-4}\pm 3.57e^{-4}$	$3.229e^{1}\pm 2.00e^{0}$	$8.693e^{-1}\pm 2.17e^{-2}$

Appendix

Table 8: Pendulum reconstruction: mitigating convergence on repetitive dynamics. We compare loss functions across four autoencoder families (AE, VAE, SAE, MAE) on Pendulum . Rows marked with

*

are our entropy–based objectives. Our losses consistently deliver the best distortion metrics (lowest MSE, highest PSNR) in every architecture: (i) AE : entropy attains

1.66\times 10^{-4}

MSE /

38.45

,dB PSNR /

0.929

SSIM, outperforming the strongest baseline (BEL) by

\approx

\times

lower MSE,

+3.3

,dB PSNR, and

+0.040

SSIM; (ii) VAE : entropy* achieves the best MSE/PSNR (

1.02\times 10^{-4}

40.00

,dB) with near–state–of–the–art SSIM (0.942), while the SSIM loss peaks in SSIM (0.964) at the cost of notably worse MSE/PSNR; (iii) SAE : spp+entropy* yields the strongest MSE/PSNR (

1.12\times 10^{-4}

39.69

,dB), and entropy* attains the best SSIM (

0.948

); (iv) MAE : entropy* is best on all three metrics (

5.93\times 10^{-4}

32.90

,dB,

0.886

). The consistent gains—often multi–dB PSNR improvements and order–of–magnitude MSE reductions—indicate that maximizing information content during training prevents collapse on repetitive trajectories and preserves fine, non–repetitive details that pixel/perceptual objectives tend to smooth.

Model	Loss	MSE	PSNR	SSIM
AE	L1	$1.301e^{-3}\pm 1.61e^{-3}$	$3.085e^{1}\pm 3.94e^{0}$	$8.359e^{-1}\pm 4.64e^{-2}$
	L2	$1.537e^{-3}\pm 1.45e^{-3}$	$2.954e^{1}\pm 3.42e^{0}$	$8.224e^{-1}\pm 4.92e^{-2}$
	CE	$1.707e^{-1}\pm 1.07e^{-2}$	$7.686e^{0}\pm 2.70e^{-1}$	$5.258e^{-1}\pm 3.48e^{-2}$
	SSIM	$1.268e^{-3}\pm 1.50e^{-3}$	$3.090e^{1}\pm 3.91e^{0}$	$\bm{8.527e^{-1}}\pm 4.53e^{-2}$
	FFL	$1.874e^{-3}\pm 2.21e^{-3}$	$2.900e^{1}\pm 3.62e^{0}$	$7.923e^{-1}\pm 6.07e^{-2}$
	ldam	$1.670e^{-3}\pm 1.82e^{-3}$	$2.936e^{1}\pm 3.53e^{0}$	$8.174e^{-1}\pm 5.03e^{-2}$

Appendix

Table 9: Reconstruction performance on the Galaxy dataset. We compare loss functions across four autoencoder families (AE, VAE, SAE, MAE). Rows marked with

*

are our entropy–based objectives. Our method consistently achieves the best distortion metrics (lowest MSE, highest PSNR) in every architecture, while remaining competitive in SSIM (often within

{\sim}0.01

–

0.02

of the SSIM-trained baseline). Concretely: (i) AE : entropy attains the best MSE/PSNR (

9.57{\times}10^{-4}

31.59

,dB), improving upon the strongest non-ours baseline by

{\sim}38\%

lower MSE and

+0.69

,dB PSNR; (ii) VAE : spp+entropy* achieves the best MSE/PSNR (

8.78{\times}10^{-4}

31.79

,dB), a

{\sim}9\%

MSE reduction and

+0.73

,dB over the best baseline; (iii) SAE : spp+entropy* yields the best MSE/PSNR (

1.30{\times}10^{-3}

30.38

,dB), a

{\sim}32\%

MSE drop and

+0.58

,dB; (iv) MAE : entropy* provides the top PSNR (

30.01

,dB) with lower MSE (

1.41{\times}10^{-3}

), improving

{\sim}24\%

in MSE and

+0.90

,dB. These gains indicate that maximizing information content mitigates convergence to oversmoothed solutions on structured astronomical textures, producing sharper, more faithful reconstructions across architectures.

Model	Loss	MSE	PSNR	SSIM
AE	L1	$1.546e^{-3}\pm 7.46e^{-4}$	$2.848e^{1}\pm 1.72e^{0}$	$8.588e^{-1}\pm 4.87e^{-2}$
	L2	$1.366e^{-3}\pm 4.27e^{-4}$	$2.883e^{1}\pm 1.25e^{0}$	$8.449e^{-1}\pm 2.80e^{-2}$
	CE	$6.967e^{-3}\pm 1.25e^{-3}$	$2.163e^{1}\pm 7.03e^{-1}$	$8.927e^{-2}\pm 1.25e^{-2}$
	SSIM	$2.744e^{-3}\pm 1.43e^{-3}$	$2.626e^{1}\pm 2.48e^{0}$	$7.933e^{-1}\pm 8.16e^{-2}$
	FFL	$1.572e^{-3}\pm 5.27e^{-4}$	$2.823e^{1}\pm 1.25e^{0}$	$8.128e^{-1}\pm 3.71e^{-2}$
	ldam	$1.073e^{-3}\pm 4.05e^{-4}$	$2.993e^{1}\pm 1.38e^{0}$	$8.711e^{-1}\pm 2.81e^{-2}$

Appendix

Table 10: Reconstruction performance on the Mitosis dataset. We compare loss functions across four autoencoder families (AE, VAE, SAE, MAE); rows marked with

*

are our entropy–based objectives. AE: entropy attains the best scores on all metrics (

6.65{\times}10^{-4}

MSE,

32.10

,dB PSNR,

0.918

SSIM), improving over the strongest non-ours baseline (ldam) by

\sim 38\%

lower MSE,

+2.17

,dB PSNR, and

+0.047

SSIM. VAE: our spp+entropy* achieves the best distortion (

1.05{\times}10^{-3}

MSE,

29.95

,dB), edging the best baseline by

\sim 4.6\%

MSE and

+0.16

,dB, while the SSIM-trained loss unsurprisingly peaks in SSIM (

0.909

). SAE: entropy* is best on all three metrics (

3.01{\times}10^{-3}

25.49

,dB,

0.740

), yielding

\sim 36\%

lower MSE,

+1.85

,dB PSNR, and

+0.044

SSIM vs. the top baseline. MAE: spp+entropy* leads across metrics (

3.88{\times}10^{-3}

24.30

,dB,

0.630

), a

\sim 29\%

MSE drop and

+1.56

,dB PSNR over the best baseline with a

+0.020

SSIM gain. Overall, maximizing information content mitigates oversmoothing and better preserves fine cellular structures, delivering consistently lower distortion and competitive–to–superior perceptual quality across architectures.

Model	Baseline	MSE	PSNR	SSIM
AE	L2	$1.292e^{-3}\pm 3.23e^{-4}$	$2.905e^{1}\pm 1.27e^{0}$	$8.129e^{-1}\pm 3.71e^{-2}$
	Focal	$3.477e^{-4}\pm 1.56e^{-4}$	$3.489e^{1}\pm 1.54e^{0}$	$9.131e^{-1}\pm 1.61e^{-2}$
	onbs	$2.533e^{-3}\pm 7.90e^{-4}$	$2.636e^{1}\pm 2.22e^{0}$	$7.679e^{-1}\pm 3.89e^{-2}$
	dcl	$1.305e^{-3}\pm 3.71e^{-4}$	$2.904e^{1}\pm 1.34e^{0}$	$8.083e^{-1}\pm 3.63e^{-2}$
	svp	$1.009e^{-3}\pm 3.70e^{-4}$	$3.036e^{1}\pm 2.06e^{0}$	$8.315e^{-1}\pm 4.12e^{-2}$
	spp2*	$\bm{1.563e^{-3}\pm 1.49e^{-4}}$	$2.808e^{1}\pm 4.26e^{-1}$	$7.932e^{-1}\pm 1.77e^{-2}$

Appendix

Table 11: Pendulum: effect of underrepresented-sample focusing on accuracy and robustness. We compare losses across four architectures (AE, VAE, SAE, MAE). Rows with

*

denote our spp

k

objective that prioritizes underrepresented samples via a memory of size

k

. Considering both mean and variability (mean

\pm

std), spp markedly improves average reconstruction on three architectures while keeping variance acceptable: VAE — spp2 attains the best distortion and perceptual quality (MSE

\downarrow

, PSNR

\uparrow

, SSIM

\uparrow

) with low–moderate spread; SAE — spp2* achieves the strongest means across all metrics, with spp8* slightly trading peak PSNR for reduced dispersion; MAE — spp2* leads on all metrics and maintains tight standard deviations. For AE , focal loss offers the best means and small variance, while spp4*/spp8* remain competitive in SSIM but do not surpass focal on averages. Across models, increasing memory ( spp2

\rightarrow

spp4

\rightarrow

spp8 ) generally smooths variance at a modest cost in mean performance, indicating a controllable mean–robustness trade-off. Overall, underrepresented-sample focusing improves average reconstruction quality where data imbalance is most impactful (VAE/SAE/MAE) and provides a tunable handle on stability via the standard deviation.

Model	Baseline	MSE	PSNR	SSIM
AE	L2	$6.270e^{-3}\pm 1.54e^{-3}$	$2.214e^{1}\pm 9.58e^{-1}$	$4.983e^{-1}\pm 3.89e^{-2}$
	Focal	$6.297e^{-3}\pm 1.52e^{-3}$	$2.212e^{1}\pm 9.51e^{-1}$	$4.966e^{-1}\pm 4.04e^{-2}$
	onbs	$7.454e^{-3}\pm 1.58e^{-3}$	$2.136e^{1}\pm 8.30e^{-1}$	$9.822e^{-2}\pm 3.45e^{-2}$
	dcl	$6.309e^{-3}\pm 1.46e^{-3}$	$2.210e^{1}\pm 8.94e^{-1}$	$4.901e^{-1}\pm 3.15e^{-2}$
	svp	$6.303e^{-3}\pm 1.45e^{-3}$	$2.210e^{1}\pm 8.91e^{-1}$	$4.912e^{-1}\pm 3.28e^{-2}$
	spp2*	$6.414e^{-3}\pm 1.56e^{-3}$	$2.204e^{1}\pm 9.48e^{-1}$	$5.136e^{-1}\pm 4.92e^{-2}$

Appendix

Table 12: Mitosis: underrepresented-sample focusing improves averages with controllable variance. We compare losses across four architectures (AE, VAE, SAE, MAE); rows with

*

denote our spp

k

objective (memory size

k

). Considering mean

\pm

std, AE benefits most from larger memory: spp8 attains the best MSE/PSNR/SSIM means, with a small increase in dispersion—indicating sharper reconstructions at slightly higher variability. VAE shows large gains with spp8* / spp4* , which substantially improve all three metrics while reducing MSE variability versus pixel/perceptual baselines, suggesting more stable training on rare cell patterns. For SAE , spp4* offers the strongest averages across metrics but with higher SSIM spread; spp8* trades a bit of mean performance for tighter variance, exposing a tunable accuracy–robustness frontier. MAE mirrors this behavior: spp4* yields the best means with standard deviations comparable to L2/Focal. Overall, prioritizing underrepresented samples consistently improves average reconstruction (MSE

\downarrow

, PSNR/SSIM

\uparrow

) and provides a controllable handle on robustness via the standard deviation, especially effective for VAE/MAE and with memory size governing the mean–variance trade-off.

Model	Baseline	MSE	PSNR	SSIM
AE	L2	$1.465e^{-3}\pm 1.23e^{-4}$	$2.836e^{1}\pm 3.62e^{-1}$	$8.033e^{-1}\pm 1.38e^{-2}$
	Focal	$1.449e^{-3}\pm 1.03e^{-4}$	$2.840e^{1}\pm 3.06e^{-1}$	$8.087e^{-1}\pm 1.11e^{-2}$
	onbs	$2.476e^{-3}\pm 7.77e^{-4}$	$2.645e^{1}\pm 2.16e^{0}$	$7.642e^{-1}\pm 3.97e^{-2}$
	dcl	$1.470e^{-3}\pm 1.50e^{-4}$	$2.835e^{1}\pm 4.46e^{-1}$	$7.841e^{-1}\pm 1.52e^{-2}$
	svp	$1.460e^{-3}\pm 1.30e^{-4}$	$2.837e^{1}\pm 3.85e^{-1}$	$8.055e^{-1}\pm 1.42e^{-2}$
	spp2*	$1.464e^{-3}\pm 3.01e^{-4}$	$2.851e^{1}\pm 1.38e^{0}$	$8.267e^{-1}\pm 2.24e^{-2}$

Appendix

Table 13: Galaxy: underrepresented-sample focusing improves accuracy and stabilizes training. We compare losses across four architectures (AE, VAE, SAE, MAE); rows with

*

denote our spp

k

objective (memory size

k

). Considering mean

\pm

std, AE sees consistent gains with markedly lower variability: spp2 attains the best MSE/PSNR/SSIM among AEs (

9.19{\times}10^{-4}

31.71

,dB,

0.829

) with substantially smaller standard deviations than pixel/perceptual baselines. VAE benefits from larger memory: spp8* yields the best MSE/PSNR (

8.13{\times}10^{-4}

31.91

,dB) and competitive SSIM (0.816), with variance comparable to or lower than L2/Focal. SAE : spp8* provides the strongest triplet (

1.22{\times}10^{-3}

30.70

,dB,

0.818

) while reducing spread relative to baselines. MAE : spp8* achieves the top PSNR/SSIM (

30.02

,dB,

0.805

), whereas spp4* attains the lowest MSE (

1.40{\times}10^{-3}

) with nearly identical variance—illustrating a controllable trade-off between mean accuracy and dispersion via memory size. Overall, prioritizing underrepresented samples improves average reconstruction (MSE

\downarrow

, PSNR/SSIM

\uparrow

) while keeping or lowering standard deviations, indicating better robustness across galaxy textures.

Explore similar work

Sep 17, 2026

Sharp Reconstruction Bounds for Autoencoders Using the Same Forward Map

We study reconstruction in autoencoders that apply the same forward map before and after setting the observed coordinates to zero. For equal odd input and hidden dimensions

d\geq 3

, among orientation-preserving diffeomorphisms whose Jacobian singular values lie in

[m,M]

, we show that the least uniform reconstruction-derivative error is

\max\{1-M(M-m)/2,0\}

, with affine maps attaining this sharp bound at every prescribed depth. A translated radial rotation can nevertheless reconstruct any prescribed ball exactly with singular values arbitrarily close to one, motivating additional conditions for a finite-data bound. We test this prediction on a 798,452-point terrestrial LiDAR forest scan. At input scale

0.05

, the mean theoretical bound is

0.155

, about

84\%

of the mean normalized training error

0.185

across four spatial regions, two depths, and three seeds. At this scale, adding one hidden coordinate reduces the mean reconstruction error below

6\times10^{-6}

Patricia Medina, Hy P. G. Lam

New York City College of Technology, CUNY, Brooklyn, NY, USA; The Graduate Center, CUNY, New York, NY, USA. · Department of Mathematical Sciences, Worcester Polytechnic Institute, Worcester, MA, USA.

Autoencoders

Ask PDF

Sep 26, 2026

Self-Reconstruction Dynamics for Autoencoder Reconstruction Refinement

Standard autoencoder (AE) inference uses a single encoder-decoder pass, though the latent may not be optimal for each sample under a fixed decoder. We ask whether a trained AE can reveal information for improving its own reconstruction. Repeated application of a frozen AE to its reconstruction produces transient image- and latent-space trajectories, termed Self-Reconstruction Dynamics (SRD). Although this degrades fidelity in the AEs studied here, SRD contains sample-specific information for correcting the reconstruction. We propose SRD-guided Reconstruction Refinement (SRD-RR), which predicts a latent correction from a short SRD with the AE frozen and no per-sample test-time optimization. We also introduce MSE-recov, an MSE recovery ratio relative to an empirical decoder-optimized reference. Across six datasets, SRD-RR recovers 38.6% of the empirically recoverable MSE gap with one transition and 45.3% with two. A two-transition variant trained without direct access to original images, using an SRD-derived pseudo-target, achieves 40.7% recovery and a 1.74 dB average PSNR gain. Removing trajectory information reduces the gain, while cross-sample trajectory assignment causes severe degradation, confirming strong sample specificity. Nonlinear SRD-conditioned refinement consistently outperforms fixed and trained linear latent correction. On a pretrained DINOv2-based representation autoencoder (RAE) with substantially different latent dynamics, SRD conditioning again improves a matched trajectory-free predictor. However, pixel-MSE latent refinement reveals a strong mismatch between pixel fidelity and perceptual quality, while the SRD-derived pseudo-target mitigates this degradation. Overall, SRD is a useful sample-specific refinement signal, while the objective determines how it translates into pixel and perceptual quality.

Hitoshi Iyatomi

Department of Applied Informatics, Faculty of Science and Engineering Hosei University Tokyo, 184-8584, Japan

Autoencoders Image Reconstruction Latent Space Optimization

Ask PDF

May 19, 2026

Next-Acceleration-Scale Prediction for Autoregressive MRI Reconstruction

MRI reconstruction is an inherently ill-posed inverse problem, since incomplete measurements admit many plausible solutions. This ambiguity becomes more severe under high acceleration, where pixel-domain continuous predictors tend to average over feasible reconstructions and suppress high-frequency anatomy. We address this limitation by moving reconstruction to discrete multi-scale latent space and posing it as autoregressive next-acceleration-scale prediction. Leveraging discrete priors proven effective in visual autoregressive modeling, our method restricts the solution to compact sequences of codebook tokens, enabling sharp reconstructions even from extremely sparse measurements. This discrete autoregressive formulation also aligns naturally with modern large language model post-training techniques. Building on this observation, we introduce on-policy privileged information distillation for visual autoregressive modeling, where a teacher is provided training only privileged context that is unavailable at inference, in our case fully sampled acquisitions, and supervises a student trained on its own rollouts, leading to consistent reconstruction gains. Through extensive experiments on the fastMRI benchmark, we show that our approach delivers improved reconstruction performance across diverse sampling patterns under extreme undersampling. Project website is here.

Yilmaz Korkmaz, Vishal M. Patel

Johns Hopkins University, Baltimore MD 21218, USA

Visual Autoregressive Models Computational Imaging Accelerated MRI Reconstruction Learning Using Privileged Information

Ask PDF