Self-Reconstruction Dynamics for Autoencoder Reconstruction Refinement

Hitoshi Iyatomi

cs.LGSep 26, 2026

Self-Reconstruction Dynamics for Autoencoder Reconstruction Refinement

Authors: Hitoshi Iyatomi

Organizations: Department of Applied Informatics, Faculty of Science and Engineering Hosei University Tokyo, 184-8584, Japan

Abstract

Standard autoencoder (AE) inference uses a single encoder-decoder pass, though the latent may not be optimal for each sample under a fixed decoder. We ask whether a trained AE can reveal information for improving its own reconstruction. Repeated application of a frozen AE to its reconstruction produces transient image- and latent-space trajectories, termed Self-Reconstruction Dynamics (SRD). Although this degrades fidelity in the AEs studied here, SRD contains sample-specific information for correcting the reconstruction. We propose SRD-guided Reconstruction Refinement (SRD-RR), which predicts a latent correction from a short SRD with the AE frozen and no per-sample test-time optimization. We also introduce MSE-recov, an MSE recovery ratio relative to an empirical decoder-optimized reference. Across six datasets, SRD-RR recovers 38.6% of the empirically recoverable MSE gap with one transition and 45.3% with two. A two-transition variant trained without direct access to original images, using an SRD-derived pseudo-target, achieves 40.7% recovery and a 1.74 dB average PSNR gain. Removing trajectory information reduces the gain, while cross-sample trajectory assignment causes severe degradation, confirming strong sample specificity. Nonlinear SRD-conditioned refinement consistently outperforms fixed and trained linear latent correction. On a pretrained DINOv2-based representation autoencoder (RAE) with substantially different latent dynamics, SRD conditioning again improves a matched trajectory-free predictor. However, pixel-MSE latent refinement reveals a strong mismatch between pixel fidelity and perceptual quality, while the SRD-derived pseudo-target mitigates this degradation. Overall, SRD is a useful sample-specific refinement signal, while the objective determines how it translates into pixel and perceptual quality.

Figures & tables

Model	Description	Definition	Target	$x$ -free
Baseline and image-space SRD models
M1	Baseline AE	$y_{1}$	–	–
M2	Repeated AE	$y_{K}$	–	–
M3	Trained Nonlinear Image Corrector	$\tilde{x}=y_{1}+\Delta x,\quad\Delta x=M_{3}(y_{1},dy_{1},\ldots,dy_{K-1})$	$x$	–
M3-AE	Corrector + AE	$\mathrm{AE}(M3)$	$x$	–
M4	Linear Image Extrapolation	$\tilde{x}_{\mathrm{lin}}=y_{1}+\alpha(y_{1}-y_{2})$	–	✓

Table 1: Summary of evaluated models, supervision, and

x

-free status.

Split	CIFAR-10	SVHN	FashionMNIST	MNIST	ImageNet100	CelebA
Train	45,000	65,932	54,000	54,000	114,021	146,493
Val	5,000	7,325	6,000	6,000	12,668	16,277
Test	10,000	26,032	10,000	10,000	5,000	5,000

Table 2: Dataset splits used in the experiments.

Model	Description	CIFAR-10	SVHN	FMNIST	MNIST	ImageNet	CelebA
M1	$y_{1}$	0
M2	$y_{2}$	-143.02	-197.19	-157.09	-158.24	-173.95	-172.62
	$y_{3}$	-346.00	-488.55	-371.01	-360.08	-429.96	-412.22
	$y_{5}$	-917.10	-1324.69	-961.63	-849.91	-1184.02	-1086.96
M3	$\tilde{x}$	116.87	103.86	87.09	96.20	51.54	98.73
M3-AE	AE(M3)	-8.93	-6.16	-5.38	-5.37	10.54	-14.03

Table 3: MSE recovery ratio (%) of image-space SRD methods.

Model	$\mathbf{N_{\mathrm{tr}}}$	CIFAR-10	SVHN	FMNIST	MNIST	ImageNet	CelebA
M5 ( $\beta=2.0$ )	1	24.97	34.58	17.59	12.55	33.09	24.96
M5-AE	1	-40.84	-53.22	-54.55	-55.69	-38.46	-66.18
M5-T	1	24.99	33.89	18.03	13.66	32.15	19.42
	2	32.48	43.79	29.04	28.96	27.36	4.64
	3	32.52	43.36	29.68	31.51	24.36	31.10
	4	32.53	44.75	29.73	31.59	16.89	28.23

Table 4: MSE recovery ratio (%) of linear latent-space correctors and estimated

\beta_{1}^{\ast}

Panel of Figure 1 — Figure 1: Geometric properties of the latent Self-reconstruction Dynamics (SRD). Error bars indicate standard deviations within each dataset.

Panel of Figure 2 — Figure 2: Effect of trajectory information and sample correspondence on nonlinear SRD-RR models.

		MSE-recov (%)						Cost
Model	$\mathbf{N_{\mathrm{tr}}}$	CIFAR-10	SVHN	FMNIST	MNIST	ImageNet	CelebA	$\times$ M1	$x$ -free
M1	–	0						1.0	–
M3 †	1	116.87	103.86	87.09	96.2	51.54	98.73	8.1	–
M4 ( $\alpha=1.0$ ) †	1	52.72	68.33	61.88	63.75	63.36	63.09	2.0	✓
M5 ( $\beta=2.0$ )	1	24.97	34.58	17.59	12.55	33.09	24.96	2.9	✓
M6-LO	0	-12.71	18.62	-40.63	-84.45	3.28	-30.06	4.2	–

Table 5: Summary of reconstruction performance, computational cost, and

x

-free status.

Model	$N_{\rm tr}$	PSNR $\uparrow$	LPIPS $\downarrow$	rFID-5k $\downarrow$
M1	–	18.87	0.156	3.91
M4	1	17.70	0.201	5.72
M6-LO	0	18.65	0.470	25.52
M6	1	18.99	0.445	21.76
M6-xFree	1	18.20	0.208	5.72

Table 6: RAE results for trajectory informativeness and pixel–perceptual reconstruction quality.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Method	$\mathbf{N_{\mathrm{tr}}}$	CIFAR-10	SVHN	FMNIST	MNIST	ImageNet	CelebA	macro	macro	$x$ -free
Method	$\mathbf{N_{\mathrm{tr}}}$	CIFAR-10	SVHN	FMNIST	MNIST	ImageNet	CelebA	$\Delta$ PSNR	MSE-recov. (%)	$x$ -free
M1	–	30.42	36.91	32.37	35.33	35.03	38.33	0.00	0.00	–
M3 †	1	3.80	7.94	8.30	12.75	2.00	6.09	6.81	92.38
M4 †	1	1.36	3.59	3.98	4.28	2.67	2.90	3.13	62.19	✓
M5	1	0.57	1.56	0.84	0.61	1.19	0.96	0.95	24.62	✓
M6-LO	0	-0.24	0.94	-1.30	-2.57	0.28	-0.77	-0.61	-24.32

Appendix

Table A1: PSNR improvements over the baseline reconstruction and corresponding MSE recovery ratios. M1 reports the absolute PSNR, whereas all other rows report the PSNR improvement (

\Delta

PSNR) relative to M1.

Panel of Figure 10 — Figure A1: Validation-set performance of image-space and latent-space linear extrapolation under sweeps of $\alpha$ and $\beta$ .

Figure A2: Performance of Repeated M5 and Repeated M5+AE on ImageNet.

Panel of Figure 12 — Figure A3: Changes in the latent trajectories across the six datasets.

Panel of Figure 13 — Figure A4: Performance consistency across (a) five random seeds and (b)(c) model architectures.

Panel of Figure 14 — Figure A5: Qualitative comparison of reconstruction results and corresponding error maps from ImageNet. M1 (37.76dB), M6 (39.71dB, MSE-recov=51.3%), and M6-xFree (39.68dB, MSE-recov=50.5%).

Panel of Figure 15 — Figure A6: Qualitative comparison of reconstruction results and corresponding error maps from CelebA. M1 (38.39dB), M6 (40.97dB, MSE-recov=55.6%), and M6-xFree (40.50dB, MSE-recov=47.7%).

	AE	M3	SRD-RR
Optimizer	Adam	Adam	AdamW
Learning rate	$10^{-3}$	$10^{-3}$	$5\times 10^{-4}$
Weight decay	0	0	$10^{-3}$
Batch size	64 / 256	64 / 256	64 / 256
Max. epochs	50	50	150
Early-stop patience	5	6	10

Appendix

Table A2: Training hyperparameters for the learned components. Batch sizes are shown as small/large datasets.

Setting	Value
RAE checkpoint	DINOv2-based RAE, ViT-XL decoder
Input resolution	$256\times 256$
Latent shape	$768\times 16\times 16$
RAE parameters	Frozen
Dataset	ImageNet
Predictor train / val / test	20,000 / 2,000 / 5,000

Appendix

Table A3: Experimental settings for the pretrained RAE extension.

$k$	0	1	2	3	4
$\\|dz_{k}\\|_{2}$	$197.83{\pm}27.02$	$184.87{\pm}29.76$	$192.06{\pm}34.16$	$200.25{\pm}36.00$	$203.22{\pm}33.75$
$\cos(dz_{k},dz_{k+1})$	$0.094{\pm}0.067$	$0.173{\pm}0.073$	$0.173{\pm}0.081$	$0.163{\pm}0.088$	–
$\cos(dz_{0},dz_{k})$	–	$0.094{\pm}0.067$	$-0.084{\pm}0.043$	$-0.089{\pm}0.034$	$-0.072{\pm}0.028$

Appendix

Table A4: Latent SRD geometry of the pretrained RAE. Values are mean

\pm

standard deviation over validation images.

Figure A7: 3D visualization of latent SRD trajectories in the pretrained RAE on ImageNet100.

Model	$N_{\rm tr}$	MSE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	rFID-5k $\downarrow$
M1	–	0.01602	18.87	0.4696	0.1558	3.91
M2 ( $y_{2}$ )	1	0.02124	17.41	0.4286	0.2193	6.29
M2 ( $y_{3}$ )	2	0.02612	16.37	0.4002	0.2770	11.21
M2 ( $y_{5}$ )	4	0.03516	14.97	0.3601	0.3740	33.44
M4	1	0.02153	17.70	0.4245	0.2009	5.72
M5	1	0.01911	17.94	0.4363	0.1943	4.74

Appendix

Table A5: Reconstruction performance of SRD-based refinement on the pretrained RAE. MSE, PSNR, SSIM, and LPIPS are averaged over the fixed 5,000 test images; rFID-5k is computed over the same complete test set. Best values are shown in bold.

Figure A8: Per-sample pixel–perceptual discrepancy under RAE refinement. Positive $\Delta$ PSNR indicates improved pixel fidelity, whereas positive $\Delta$ LPIPS indicates degraded perceptual similarity. The frequent occurrence of samples with improved PSNR but worse LPIPS shows that pixel-level improvement does not necessarily imply perceptual improvement. M6 and M6-xFree use $N_{\mathrm{tr}}=1$ .

Panel of Figure 22 — Figure A9: Qualitative comparison on ImageNet100 using the pretrained RAE. M6 improves PSNR but worsens LPIPS relative to M1, while M6-xFree retains more fine-scale detail and achieves lower LPIPS than M6.

Panel of Figure 23 — Figure A10: Qualitative comparison on ImageNet100 using the pretrained RAE. M6 deteriorates in both PSNR and LPIPS relative to M1. In contrast, M6-xFree improves PSNR over M1 with a modest increase in LPIPS and substantially better perceptual similarity than M6.

Explore similar work

Improved Baselines with Representation Autoencoders
May 18, 2026Jaskirat Singh, Boyang Zheng, Zongze Wu +3Diffusion Model Guidance Diffusion Transformer
Adobe Research · ANU · New York University
DecQ: Detail-Condensing Queries for Enhanced Reconstruction and Generation in Representation Autoencoders
May 21, 2026Tianhang Wang, Yitong Chen, Wei Song +3Latent Diffusion Models Autoencoders
Zhejiang University · Shanghai Innovation Institute · Fudan University +1
FuseReg: Regularizing Layer Fusion Mitigates the Reconstruction-Generation Gap in Representation Autoencoders
Sep 25, 2026Hongyang Du, Yunfei Xie, Junjie Ye +13Autoencoders Image Reconstruction
USC PSI Lab · Brown University · Rice University +4

May 18, 2026J/K move · Enter open · S save

Improved Baselines with Representation Autoencoders

Jaskirat Singh, Boyang Zheng, Zongze Wu, Richard Zhang+2

Adobe Research · ANU · New York University

Diffusion Model Guidance Diffusion Transformer Autoencoders Diffusion-Based Image Generation Diffusion Models

Open paper AskPDF arXiv

Representation Autoencoders (RAE) replace traditional VAE with pretrained vision encoders. In this paper, we systematically investigate several design choices and find three insights which simplify and improve RAE. First, we study a generalized formulation where the representation is defined as sum of the last k encoder layers rather than solely the final layer. This simple change greatly improves reconstruction without encoder finetuning or specialized data (e.g., text, faces). Second, we study the prevalent assumption that RAE (using pretrained representation as encoder) replaces representation alignment (REPA), which distills the same representation to intermediate layers instead. Through large-scale empirical analysis, we uncover a surprising finding: RAE and REPA exhibit complementary working mechanisms, allowing the same representation to be used as both encoder and target for intermediate diffusion layers. Finally, the original RAE struggles with classifier-free guidance (CFG) and requires training a second, weaker diffusion model for AutoGuidance (AG). We show that REPA itself can be viewed as x-prediction in RAE latent space. By simply re-parameterizing the output of the DiT model, it can provide guidance for "free". Overall, RAEv2 leads to more than 10x faster convergence over the original RAE, achieving a state-of-the-art gFID of 1.06 in just 80 epochs on ImageNet-256. On FDr6, RAEv2 achieves a state-of-the-art 2.17 at just 80 epochs compared to the previous best 3.26 (800 epochs) without any post-training. This motivates EPFID@k (epochs to reach unguided gFID < k) as a measure of training efficiency. RAEv2 attains an EPFID@2 of 35 epochs, versus 177 for the original RAE. We also validate our approach across diverse settings for text-to-image generation and navigation world models, showing consistent improvements. The code is available at https://raev2.github.io.

Abstract

Explore similar work

Improved Baselines with Representation Autoencoders

DecQ: Detail-Condensing Queries for Enhanced Reconstruction and Generation in Representation Autoencoders

FuseReg: Regularizing Layer Fusion Mitigates the Reconstruction-Generation Gap in Representation Autoencoders