An Input Skip Can Reparameterize a VAE Without Improving the Generator

Kevin Lee

cs.LGOct 27, 2026

An Input Skip Can Reparameterize a VAE Without Improving the Generator

Authors: Kevin Lee

Organizations: University of California at Los Angeles · NASA Jet Propulsion Laboratory

Abstract

An input skip can reduce reconstruction error while leaving a model's distribution of new samples unchanged. For a Gaussian variational autoencoder (VAE) with constant input skip $0\leqα<1$ , a model without a skip can be written in skip form by multiplying its decoder output by $b=1-α$ and its noise variance by $b^2$ . The distribution of observations and latent variables and its exact generative scores stay the same, while reconstruction mean squared error (MSE) is multiplied by $b^2$ . Separate training can produce different generators because optimization depends on the parameter coordinates. In a Gaussian experiment with known density, changing skip and observation variance together reduces MSE but worsens energy score and the estimate of negative log density from importance sampling. On seven annual Deep-ocean Assessment and Reporting of Tsunamis (DART) records, energy score differences at the same observation variance have both signs on four original records and favor the skip on three records from 2018. Probabilistic principal component analysis (PPCA) has lower mean energy scores than every neural DART fit. With the encoder and decoder kept the same, estimating observation variance from training reconstruction error can also worsen energy score. Assessing generation requires more than reconstruction error.

Figures & tables

Skip coefficient	Base variance: $\sigma_{0}^{2}$	High variance: $25\sigma_{0}^{2}$
$\alpha=0$	No skip, base variance	No skip, high variance
$\alpha=0.8$	Skip, base variance	Skip, high variance

Table 1: Skip and variance conditions. The high variance is

25\sigma_{0}^{2}

, where

25=1/(1-0.8)^{2}

. The original comparison changes both factors, from no skip at base variance to skip at high variance.

Figure 1: DART energy score differences between skip and no skip at base variance

\sigma_{0}^{2}

(squares) and high variance

25\sigma_{0}^{2}

(circles). Negative values favor the skip. Scores use dimensionless coordinates. Points give median differences across three training seeds; bars span their full range. Scores average three streams per fit. Original records use network weights chosen by validation and equal window weights; 2018 records use final weights and UTC start hour weights. Table 8 gives sampling variation.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: T=1 HEIGHT in meters for the four original records, identified by station and year. Points are NOAA HEIGHT observations; blank intervals contain no usable T=1 observations. The steps at 44402 and 46407 occur near the National Centers for Environmental Information (NCEI) deployment dates cited in Appendix G . At 52403, the levels differ before and after an interval without usable T=1 observations from March to August that includes the catalogue deployment date of 13 August 2020. The time of the level change remains uncertain; the cause of the 23461 spikes is also unresolved. Figures 8 and 10 show feature distributions and spectra for the three 2018 records.

Case	Train	Validation	Test	Mean	Scale	$\sigma_{0}^{2}$
Gaussian	8192	2048	2048	0	1	0.06157
23461/2022	544	13	17	2488.8962	1.481	0.0001
44402/2016	1153	34	35	2613.8301	1.518	0.0001
46407/2011	983	26	27	3266.7902	1.48	0.0001
52403/2020	339	17	16	4433.8446	0.7036	0.0001
44402/2018	1150	390	369	2615.7276	0.33	0.0001

Appendix

Table 2: Window counts and normalization constants, with routine T=1 DART records identified by station and year. Mean and scale are the training mean and population standard deviation in meters. Normalization counts each eligible training row once for the original DART records; for 2018, it counts each unique training row in an eligible window once. Gaussian data keep their original coordinates, shown as mean zero and scale one. Base variance

\sigma_{0}^{2}

is expressed in squared model coordinates. For 2018, PPCA validation chooses this variance before neural fitting.

Case	Skip	Variance	Raw MSE	MSE $/b^{2}$	Expected MSE	NLL/ $d$	KL/ $d$	Negative ELBO/ $d$	Negative IW/ $d$
Gaussian	$\times$	Base	0.0616	0.0616	0.06425	0.04692	0.09338	0.1403	0.1332
Gaussian	$\checkmark$	Base	0.002496	0.06239	0.06498	0.05287	0.09343	0.1463	0.1373
Gaussian	$\times$	High	0.1985	0.1985	0.2182	1.205	0.01483	1.22	1.22
Gaussian	$\checkmark$	High	0.003683	0.09209	0.1316	1.177	0.02497	1.202	1.201
23461/2022	$\times$	Base	0.0008939	0.0008939	0.0008945	0.7864	1.447	2.233	2.065
23461/2022	$\checkmark$	Base	0.000102	0.002551	0.002552	9.073	1.372	10.44	10.06

Appendix

Table 3: Reconstruction errors and ELBO terms at the observation variance used for training. The original records use network weights chosen by validation and equal window weights; the 2018 records use final network weights and UTC start hour weights. Raw MSE includes the input skip; division by

b^{2}

expresses the error in observation coordinates after decoding the encoder mean. For 2018, raw MSE is calculated as

b^{2}

times this error; original fits use the raw MSE saved during evaluation. Expected MSE and conditional negative log likelihood (NLL) average over samples from

q_{\phi}(z\mid x)

. MSE is in squared model coordinates; log quantities are in nats per coordinate. Each entry averages over test windows with the weights above, then over three sampling streams, then over three fits. Before rounding, NLL plus analytic KL equals negative ELBO. IW estimates negative marginal log density using finite

K

. Base and high denote observation variances

\sigma_{0}^{2}

and

25\sigma_{0}^{2}

. The Skip column uses

\checkmark

for

\alpha=0.8

and

\times

for

\alpha=0

Case	Model	Negative log score	Energy score
Gaussian	True Gaussian	0.1255	5.717
Gaussian	PPCA	0.1266	5.72
Gaussian	Diagonal Gaussian	0.9087	5.793

Appendix

Table 4: Gaussian baseline scores on test realizations; lower values are better. Negative log densities are exact and use nats per coordinate. Energy score uses the original dimensionless Gaussian coordinates and 512 samples per baseline, obtained by applying the covariance Cholesky factor to standard normal samples. True Gaussian denotes the known generating density. Figure 3 uses a separate evaluation of DART baselines with three sampling streams.

Case	Skip	Variance	Seed	Step	Raw MSE	MSE/ $b^{2}$	Negative IW	KL	Energy score	ESS
Gaussian	$\times$	Base	42	5000	0.06162	0.06162	0.1335	17.94	5.723	41.3
Gaussian	$\times$	Base	1729	5250	0.0617	0.0617	0.1334	18.06	5.724	35.68
Gaussian	$\times$	Base	31415	7750	0.06149	0.06149	0.1326	17.79	5.721	52.9
Gaussian	$\checkmark$	Base	42	10000	0.002494	0.06236	0.1371	17.9	5.719	31.33
Gaussian	$\checkmark$	Base	1729	6000	0.002497	0.06243	0.1376	17.96	5.722	29.33
Gaussian	$\checkmark$	Base	31415	7500	0.002496	0.06239	0.1371	17.96	5.724	29.24

Appendix

Table 5: All 12 original Gaussian fits. Raw MSE decodes the encoder mean and includes the input skip; MSE divided by

b^{2}

expresses the error in observation coordinates. Both are in squared dimensionless coordinates. Negative IW estimates negative log density in nats per coordinate; KL measures encoder divergence in nats per realization. Energy score uses 512 prior samples per fit. Errors, KL, and energy score are averaged over all test realizations for each fit. Lower errors and scores are better. ESS is computed from 256 importance samples per realization and then summarized by its median across realizations. Step is the Adam iteration chosen by validation. Base and high denote observation variances

\sigma_{0}^{2}

and

25\sigma_{0}^{2}

. The Skip column uses

\checkmark

for

\alpha=0.8

and

\times

for

\alpha=0

Case	Skip vs no skip Base variance	Skip vs no skip High variance	Skip, high variance vs no skip, base variance
Gaussian	$0.004143$ $[0.003679,\,0.004428]$	$-0.004057$ $[-0.05053,\,-0.0009969]$	$1.068$ $[1.068,\,1.069]$
23461/2022	$9.496$ $[3.226,\,11.24]$	$-0.02887$ $[-0.2702,\,0.3098]$	$-4.328$ $[-4.52,\,-1.029]$
44402/2016	$-0.1457$ $[-0.7499,\,0.8596]$	$-0.002817$ $[-0.007894,\,0.00181]$	$-0.3495$ $[-0.4308,\,-0.1869]$
46407/2011	$2.374$ $[2.168,\,4.924]$	$-0.01531$ $[-0.02908,\,0.0532]$	$-26.65$ $[-30.07,\,-26.16]$
52403/2020	$4.26$ $[3.937,\,5.114]$	$0.1918$ $[0.05503,\,0.2004]$	$-2.595$ $[-3.644,\,-2.527]$

Appendix

Table 6: Differences in negative IW log density estimates at

K=256

, in nats per coordinate, for the original records. Fits are paired by training seed. Each cell gives the median difference across three seeds and the full range below it; negative values favor the first condition. DART labels identify station and year. The ranking by exact marginal likelihood remains uncertain with these finite sample estimates. The first two comparisons keep variance the same; the last changes both skip and variance.

Case	Observation variance		$K=256$	$K=4096$	$K=16384$
	Skip	No skip
Gaussian	Base	Base	$0.003605$ $[0.001233,\,0.004018]$	$0.003406$ $[0.001593,\,0.00391]$	$0.003707$ $[0.001884,\,0.00389]$
Gaussian	High	High	$-0.00525$ $[-0.05736,\,9.69\times 10^{-5}]$	$-0.005248$ $[-0.05733,\,9.666\times 10^{-5}]$	$-0.005238$ $[-0.05736,\,0.000105]$
Gaussian	High	Base	$1.068$ $[1.068,\,1.069]$	$1.069$ $[1.069,\,1.069]$	$1.069$ $[1.069,\,1.069]$
23461/2022	Base	Base	$9.496$ $[3.226,\,11.24]$	$9.462$ $[3.209,\,11.11]$	$9.455$ $[3.185,\,11.05]$
23461/2022	High	High	$-0.02887$ $[-0.2702,\,0.3098]$	$-0.03072$ $[-0.3078,\,0.302]$	$-0.02819$ $[-0.3055,\,0.3093]$

Appendix

Table 7: Importance estimates with more samples on the same initial test windows. DART labels give station and year. Differences compare fits that share a training seed, at the observation variances shown. Each difference subtracts the negative IW log density without a skip from the value with a skip. Cells give the median difference in nats per coordinate and the full range across three seeds underneath. Negative values indicate a lower estimated score for the skip model. The final row for each case changes both skip use and observation variance. Base and high denote

\sigma_{0}^{2}

and

25\sigma_{0}^{2}

. The skip coefficient is 0.8 when present and zero otherwise. The Gaussian and 44402 subsets are smaller than their full test sets.

Record	Variance	Statistic	Full windows	After mean subtraction	Time increments
23461/2022	Base	Median and range	0.369 [-1.21, 1.5]	0.0255 [-0.0553, 0.297]	0.00303 [-0.00289, 0.0262]
23461/2022	Base	Maximum range width	0.1585	0.03316	0.004819
23461/2022	High	Median and range	0.83 [0.773, 3.4]	0.0822 [-0.0275, 0.0868]	0.0111 [0.000783, 0.0154]
23461/2022	High	Maximum range width	0.4628	0.03795	0.005022
44402/2016	Base	Median and range	-0.153 [-1.27, 0.849]	-0.0814 [-0.144, 0.032]	-0.00857 [-0.0311, 0.0141]
44402/2016	Base	Maximum range width	0.2093	0.02672	0.004928

Appendix

Table 8: DART energy score differences between skip and no skip at the same observation variance, in dimensionless coordinates. Scores use full windows, windows after subtraction of their own means, and the time increments in Eq. 10 . Negative values favor the skip. Each difference is the skip score minus the score without a skip, at base variance

\sigma_{0}^{2}

or high variance

25\sigma_{0}^{2}

. The median and full range describe three differences between fits with the same training seed, after averaging streams within each fit. Maximum range width is the largest, over training seeds, of the maximum minus minimum difference across three common sampling streams. Original records use network weights chosen by validation and equal window weights. The 2018 records use final network weights and equal total window weight in each UTC start hour.

Figure 3: Energy score of each VAE fit minus that of PPCA at the same observation variance. Positive values favor PPCA. Each row shows the three training seeds. Scores average three sampling streams within each fit. PPCA uses separate streams, and one PPCA fit supplies the baseline at each variance on each record. The three subplots use full windows, windows after subtraction of their own means, and the time increments in Eq. 10 , all in dimensionless coordinates. Base variance is

\sigma_{0}^{2}

and high variance is

25\sigma_{0}^{2}

. Original records use network weights chosen by validation and equal window weights; 2018 records use final network weights and UTC start hour weights.

Original variance	Fits	Changes in distance terms	Score after replacement
$\sigma_{0}^{2}$	42	$0<\Delta D_{T}<\Delta S_{T}$	Lower
$25\sigma_{0}^{2}$	42	$\Delta S_{T}<\Delta D_{T}<0$	Higher

Appendix

Table 9: Changes in the two distance terms of the energy score when observation variance is replaced, calculated as after minus before.

D_{T}

averages distances from model samples to observations.

S_{T}

is half the mean distance between distinct model samples and is subtracted from the first term. The displayed relation holds for every fit after averaging its three common sampling streams. Each variance group contains the same 42 fits for all three evaluations: full windows, windows after subtraction of their own means, and time increments.

Figure 4: Replacing observation variance in all 84 DART fits. For each record and condition, points run from top to bottom in the order of training seeds 42, 1729, and 31415. The variance ratio is

\hat{v}/v_{\mathrm{original}}

;

\hat{v}

is estimated from three streams of encoder samples, and a ratio of one denotes equal variances. The variance ratio axis is logarithmic. Energy score columns show

10^{3}

times the difference between the score after variance replacement and the score before replacement in dimensionless coordinates, averaged over three common prior streams. Negative values mean the score falls after variance replacement. Each thin bar runs from the smallest to the largest score change across those three streams for one fit. Original records use network weights chosen by validation and equal window weights; 2018 records use final network weights and UTC start hour weights.

Record	Skip	Original variance	$10^{4}\hat{v}$	$10^{3}\Delta$ Full windows	$10^{3}\Delta$ After mean subtraction	$10^{3}\Delta$ Time increments
23461/2022	$\times$	Base	3.32 [2.14, 4.09]	-5.13 [-6.91, -2.27]	-2.11 [-3.1, -0.803]	-0.395 [-0.559, -0.141]
23461/2022	$\checkmark$	Base	3.53 [2.63, 5.27]	-5.15 [-11, -3.29]	-2.25 [-4.58, -1.3]	-0.566 [-1.11, -0.32]
23461/2022	$\times$	High	5.18 [3.94, 6.08]	32.9 [31, 35.2]	13.9 [13.7, 14.4]	2.48 [2.22, 2.88]
23461/2022	$\checkmark$	High	5.74 [4.84, 8.83]	30.4 [30.2, 35.7]	13.7 [12.1, 18.8]	3.7 [2.97, 4.72]
44402/2016	$\times$	Base	3.45 [3.1, 5.24]	-5.3 [-9.19, -4.46]	-2.22 [-3.63, -1.67]	-0.105 [-0.246, -0.0806]
44402/2016	$\checkmark$	Base	4.1 [2.66, 6.45]	-5.23 [-10.7, -2.68]	-2.39 [-3.57, -1.3]	-0.281 [-0.395, -0.172]

Appendix

Table 10: Observation variance estimated from training windows with encoder and decoder weights kept the same. The variance column gives

10^{4}\hat{v}

, where

\hat{v}

is in squared dimensionless coordinates. Energy score columns give

10^{3}

times the difference between the score after variance replacement and the score before replacement in dimensionless coordinates; negative values mean lower scores after variance replacement. Entries give the median across three training seeds, with the full range below, after averaging sampling streams within each fit. The original observation variance is

10^{-4}

for base variance models and

0.0025

for high variance models. Each model receives its own variance estimate, so conditions can have different values. The Skip column uses

\checkmark

for

\alpha=0.8

and

\times

for

\alpha=0

Record	Variance	Evaluation	Steps	Full windows	After mean subtraction	Time increments
44402/2018	Base	Validation weights	6000–10000	-0.359 [-0.748, -0.128]	-0.385 [-0.519, -0.298]	-0.0368 [-0.055, -0.0353]
44402/2018	Base	Equal window weights	10000	-0.337 [-0.768, -0.119]	-0.346 [-0.479, -0.296]	-0.035 [-0.0522, -0.0334]
44402/2018	High	Validation weights	10000–10000	-0.262 [-0.33, -0.241]	-0.239 [-0.37, -0.201]	-0.0288 [-0.0431, -0.0213]
44402/2018	High	Equal window weights	10000	-0.222 [-0.322, -0.219]	-0.227 [-0.363, -0.151]	-0.0267 [-0.0419, -0.0157]
21415/2018	Base	Validation weights	8000–10000	-0.328 [-0.334, -0.255]	-0.406 [-0.409, -0.317]	-0.06 [-0.0612, -0.0448]
21415/2018	Base	Equal window weights	10000	-0.299 [-0.35, -0.211]	-0.371 [-0.377, -0.331]	-0.0555 [-0.0562, -0.0473]

Appendix

Table 11: Alternative network weights and window weights for the three 2018 records. Each entry gives the median energy score difference across three training seeds, with the full range, for conditions paired by seed. Negative values favor the skip. The first evaluation uses the network weights with the lowest validation energy score and gives each UTC start hour equal total window weight. The other evaluation uses final network weights and gives every window equal weight. Steps is the minimum and maximum Adam step count over both conditions and all seeds, or their common final count. Scores are averaged over streams within each fit. Evaluations at validation and final network weights use different samples, even when both use the final training step. Each difference compares skip with no skip at the same original observation variance.

Figure 5: Energy score differences over time for the original DART records. Each point gives the skip score minus the score without a skip at the same observation variance for one training seed, after averaging three sampling streams within each fit. Negative values favor the skip. Scores use full windows in dimensionless coordinates. The columns group test windows by start time into intervals of 14 or 28 days. Dates give the interval start in UTC. Horizontal spacing separates the seed markers around each interval start. The coverage below each subplot gives window counts and fractions of total test weight. Marker shapes identify training seeds. Filled points denote base variance

\sigma_{0}^{2}

and open points high variance

25\sigma_{0}^{2}

Figure 6: Energy score differences over time for the additional DART records, with the definitions in Figure 5 . Interval scores use the global UTC start hour weights of their test windows. Counts include overlapping windows. For each of 21415 and 21418, one training seed gives positive differences over 14 days in an interval with little total test weight. At both observation variances, the skip has lower scores for all three seeds in every interval of 28 days containing test windows in these two records.

Figure 7: Feature ECDFs for the four original DART records, comparing training and test observations with noisy prior samples from the four neural conditions and PPCA. Observed windows have equal weights. Each neural fit uses the network weights chosen by validation and three streams of 1,024 prior samples. After averaging the three ECDFs within each fit, the curve gives the median at each feature value across three fits per condition, and shading spans the full range across training seeds.

Figure 8: Feature ECDFs for the three additional DART records. Training and test windows each use the UTC start hour weights of their own partition; model samples have equal weight. For each neural condition, three sampling streams are averaged within each final fit. The curve at each feature value is the median across three fits, with shading spanning the full range across training seeds. The PPCA curve averages three streams from one baseline fit.

Figure 9: DFT bin powers within windows for the four original records. The columns at base variance use

\sigma_{0}^{2}

, and those at high variance use

25\sigma_{0}^{2}

. Each column includes spectra of training and test observations and the same PPCA fit. Solid neural curves include observation noise; dashed curves use decoder means at prior latent samples. Neural models use network weights chosen by validation. Bin powers are averaged over three streams in linear units within each fit; curves and shading then show the median and full range across training seeds at each frequency. Subtracting each window’s mean before the DFT removes variation in window levels. Both axes are logarithmic, and the 0.5 cycle per day bin spacing leaves nearby tidal constituents unresolved.

Figure 10: DFT bin powers within windows for the three additional records. Training and test spectra use UTC start hour weights computed separately for each partition. Neural spectra use final fits at their original observation variance. Solid curves include observation noise; dashed curves use decoder means at prior latent samples. Within each fit, bin powers are averaged over three streams in linear units. Curves and shading then give the median and full range across training seeds at each frequency. Both columns use the same PPCA fit. Each window’s mean is subtracted before the DFT. Both axes are logarithmic, and nearby tidal constituents remain unresolved at the 0.5 cycle per day bin spacing.

Variational Autoencoders

Variational autoencoders are probabilistic encoder–decoder models for learning latent representations, generating data, and estimating uncertainty.

See all 210 papers

Empirical Variational Autoencoder
Oct 5, 2026Kaede ShioharaVariational Autoencoders Latent Variable Models
The University of Tokyo
Distilling Latent Manifolds: Resolution Extrapolation by Variational Autoencoders
Mar 15, 2026Jiaming Chu, Tao Wang, Lei JinPosition Extrapolation Variational Autoencoders
School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing, China
$\mathbfλ$ -VAE: Variance Equalization for Posterior Collapse
Jul 6, 2026Girum DemisseVariational Autoencoders Posterior Collapse
Microsoft African Development Center Nairobi Kenya

Oct 5, 2026J/K move · Enter open · S save

Empirical Variational Autoencoder

Kaede Shiohara

The University of Tokyo

Variational Autoencoders Latent Variable Models Image Generation Autoregressive Generation Audio Generation

Open paper AskPDF arXiv

We present Empirical Variational Autoencoder, a general generative framework for continuous-valued (i.e., non-vector-quantized) sequences. EVA is based on the evidence lower bound of the Variational Autoencoder (VAE) but learns autoregressive latent priors empirically from training data, which can be implemented only by an additional single linear layer on top of VAEs. By replacing the conventional standard-Gaussian constraint with the self-predicted priors, EVA significantly alleviates the latent distribution gap between prior and posterior which is typically observed in conventional VAEs, and leads to high-fidelity ancestral sampling for sequential data generation. Extensive experiments on image and sound synthesis demonstrate that EVA achieves competitive generation quality with autoregressive diffusion baselines despite its much faster inference time.

Abstract

Variational Autoencoders

Explore similar work

Empirical Variational Autoencoder

Distilling Latent Manifolds: Resolution Extrapolation by Variational Autoencoders

λ\mathbfλλ-VAE: Variance Equalization for Posterior Collapse

$\mathbfλ$ -VAE: Variance Equalization for Posterior Collapse