Variational Distributional Neuron

Yves Ruffenach

cs.LGFeb 20, 2026

Variational Distributional Neuron

Authors: Yves Ruffenach

Organizations: Conservatoire National des Arts et Métiers

Abstract

We propose a proof of concept for a variational distributional neuron: a compute unit formulated as a VAE brick, explicitly carrying a prior, an amortized posterior and a local ELBO. The unit is no longer a deterministic scalar but a distribution: computing is no longer about propagating values, but about contracting a continuous space of possibilities under constraints. Each neuron parameterizes a posterior, propagates a reparameterized sample and is regularized by the KL term of a local ELBO - hence, the activation is distributional. This "contraction" becomes testable through local constraints and can be monitored via internal measures. The amount of contextual information carried by the unit, as well as the temporal persistence of this information, are locally tuned by distinct constraints. This proposal addresses a structural tension: in sequential generation, causality is predominantly organized in the symbolic space and, even when latents exist, they often remain auxiliary, while the effective dynamics are carried by a largely deterministic decoder. In parallel, probabilistic latent models capture factors of variation and uncertainty, but that uncertainty typically remains borne by global or parametric mechanisms, while units continue to propagate scalars - hence the pivot question: if uncertainty is intrinsic to computation, why does the compute unit not carry it explicitly? We therefore draw two axes: (i) the composition of probabilistic constraints, which must be made stable, interpretable and controllable; and (ii) granularity: if inference is a negotiation of distributions under constraints, should the primitive unit remain deterministic or become distributional? We analyze "collapse" modes and the conditions for a "living neuron", then extend the contribution over time via autoregressive priors over the latent, per unit.

Figures & tables

Figure 1: Micro vs macro dynamics. (A) Global AR-VAE: a vector latent state $z_{t}\in\mathbb{R}^{K}$ follows a state-space-like dynamics. (B) AR-VAE neuron network: local latent states $z_{t}^{(i)}$ evolve via local transitions (sparse graph) and are aggregated to produce the output.

Approach	Where is uncertainty?	Unit	Objective	Dynamics
BNN	on weights	scalar (classic)	VI on weights / NLL	optional
VAE (macro)	global latent	scalar (decoder)	global ELBO	sequential prior / SSM
Stochastic layers	prediction / regularization	scalar + noise	approx. VI	weak / implicit
EVE (this work)	local latent per unit	distribution (prior/post.)	micro-ELBO + constraints	per-unit AR (micro)

Table 1: Conceptual positioning. EVE moves uncertainty and inference to the unit level (local latent + micro-ELBO) and introduces local tunings/diagnostics (band/energy, effective KL, drift/ collapse ), with a per-neuron AR dynamics.

Figure 2: Architecture of a VAE neuron (distributional neuron). Each unit implements a local step of variational inference: $x\mapsto q_{\phi}(z\!\mid\!x)=\mathcal{N}(\mu_{\phi}(x),\sigma_{\phi}(x)^{2})$ , sampling $z=\mu_{\phi}(x)+\sigma_{\phi}(x)\varepsilon$ , $\varepsilon\sim\mathcal{N}(0,1)$ , then emission $p_{\theta}(a\!\mid\!z,x)$ . The prior $p(z)$ regularizes via $\mathrm{KL}(q_{\phi}(z\!\mid\!x)\,\|\,p(z))$ , defining a local ELBO (“spring” toward the prior).

Figure 3: Classic deterministic neuron vs distributional neuron (VAE neuron). Left: deterministic scalar activation $a=w^{\top}x+b$ . Right: local variational inference with internal latent $z$ , amortized posterior $q_{\phi}(z\!\mid\!x)$ , reparameterization, emission $p_{\theta}(a\!\mid\!z,x)$ , and KL regularization. The deterministic neuron is recovered as the limit $\sigma_{\phi}(x)\to 0$ .

Measure	Definition / proxy	Interpretation
$\mathrm{KL}_{\text{mean}}$	$\frac{1}{N}\sum_{i}\mathrm{KL}_{i}$	mean information rate (structure vs fit)
$\mathrm{KL}_{\text{eff}}$	KL above a threshold, e.g., $\max(\mathrm{KL}-\tau,0)$	latent effectively active
$\mu^{2}_{\text{mean}}$	$\frac{1}{N}\sum_{i}\mathbb{E}[\mu_{i}(h)^{2}]$	energy / activity (proxy if $\sigma$ fixed)
low_rate / high_rate	fraction of units outside budget (low/high)	collapse / saturation / explosion
clamp_low / clamp_high	correction amplitude (projection)	AutoPilot intensity
reparam_proxy	$\mathbb{E}\!\left[\|z-\mu\|\right]$	consistency check (sampling)

Table 2: Internal measures (examples) and interpretation.

Figure 4: LongHorizon (in=336, H=96): performance and control regimes. Test MSE (mean

\pm

std,

n=5

) for three EVE variants: homeo (homeostasis enabled), projON (hard projection) and projOFF (no internal control). Internal controls define regimes and are not meant as a systematic MSE booster.

Dataset	Variant	MSE $\downarrow$	MAE $\downarrow$	$\mu^{2}$	out $\downarrow$
ECL ( $n=5$ )	projOFF	0.251 $\!\pm\!$ 0.012	0.363 $\!\pm\!$ 0.012	0.085 $\!\pm\!$ 0.003	0.534 $\!\pm\!$ 0.030
	projON	0.335 $\!\pm\!$ 0.026	0.436 $\!\pm\!$ 0.021	0.049 $\!\pm\!$ 0.002	0.630 $\!\pm\!$ 0.042
	homeo	0.461 $\!\pm\!$ 0.082	0.528 $\!\pm\!$ 0.063	0.133 $\!\pm\!$ 0.088	0.601 $\!\pm\!$ 0.065
TrafficL ( $n=5$ )	projOFF	0.659 $\!\pm\!$ 0.006	0.518 $\!\pm\!$ 0.002	0.068 $\!\pm\!$ 0.002	0.379 $\!\pm\!$ 0.009
	projON	0.669 $\!\pm\!$ 0.008	0.523 $\!\pm\!$ 0.004	0.064 $\!\pm\!$ 0.001	0.379 $\!\pm\!$ 0.014
	homeo	1.117 $\!\pm\!$ 0.005	0.740 $\!\pm\!$ 0.002	0.082 $\!\pm\!$ 0.002	0.530 $\!\pm\!$ 0.014

Table 3: LongHorizon (in=336, H=96), multi-seeds. Test metrics at the checkpoint selected on validation ( select_by=score composite (MSE + internal constraints) ). We report mean

\pm

std over

n=5

random initializations (0–4). In addition to MSE/MAE, we report

\mu^{2}

(mean latent energy) and out (fraction of units outside the band, out =

\mathrm{frac}_{low}+\mathrm{frac}_{high}

Dataset	Variant	out $\downarrow$	$\mathrm{KL}$	$\mu^{2}$	Band
ECL	projOFF	0.534 $\pm$ 0.030	73.7 $\pm$ 2.7	0.085 $\pm$ 0.003	0.128 $\pm$ 0.019
ECL	projON	0.630 $\pm$ 0.042	44.3 $\pm$ 9.2	0.049 $\pm$ 0.002	0.156 $\pm$ 0.010
ECL	homeo	0.601 $\pm$ 0.065	113.2 $\pm$ 24.2	0.133 $\pm$ 0.088	0.217 $\pm$ 0.050
TrafficL	projOFF	0.379 $\pm$ 0.009	46.3 $\pm$ 1.9	0.068 $\pm$ 0.002	0.931 $\pm$ 0.135
TrafficL	projON	0.379 $\pm$ 0.014	44.3 $\pm$ 1.7	0.064 $\pm$ 0.001	1.000 $\pm$ 0.216
TrafficL	homeo	0.530 $\pm$ 0.014	49.3 $\pm$ 1.4	0.082 $\pm$ 0.002	2.243 $\pm$ 0.790

Table 4: LongHorizon (in=336, H=96) — internal test observables (multi-seeds). Mean

\pm

std over

n=5

random initializations (0–4), reported at the same checkpoint as Table 3 (selected on validation, select_by=score composite (MSE + internal constraints) ). out (out-of-band):

\textit{out}=\mathrm{frac}_{low}+\mathrm{frac}_{high}

(bold when minimal within the dataset).

\mathrm{KL}

: latent information rate;

\mu^{2}

: mean latent energy; Band : band penalty.

Figure 5: Global correlation: out vs MSE. Scatter plot over

N=60

runs (z-score per dataset), showing the positive correlation between the out-of-band fraction ( out ) and test MSE.

Figure 6: AutoPilot (ECL, 24 trials, 3 epochs): early out signal. Correlation between the out-of-band fraction measured at 3 epochs and the best validation MSE reached within these 3 epochs.

Dataset	Var.	MSE $\downarrow$	MAE $\downarrow$	ar_share $\uparrow$	out $\downarrow$
ECL	AR OFF	0.217 $\!\pm\!$ 0.003	0.333 $\!\pm\!$ 0.002	0.000 $\!\pm\!$ 0.000	0.393 $\!\pm\!$ 0.007
ECL	AR ON	0.461 $\!\pm\!$ 0.082	0.528 $\!\pm\!$ 0.063	0.367 $\!\pm\!$ 0.219	0.601 $\!\pm\!$ 0.065
Exchange	AR OFF	0.190 $\!\pm\!$ 0.049	0.331 $\!\pm\!$ 0.043	0.000 $\!\pm\!$ 0.000	0.392 $\!\pm\!$ 0.042
Exchange	AR ON	0.180 $\!\pm\!$ 0.037	0.323 $\!\pm\!$ 0.037	0.047 $\!\pm\!$ 0.005	0.403 $\!\pm\!$ 0.036

Table 5: AR ON/OFF ablation (homeo, multi-seeds). Mean

\pm

std over

n=5

seeds (0–4). The checkpoint is selected on validation using the same composite score (val MSE + internal constraint penalties).

Model	MSE $\downarrow$	MAE $\downarrow$
Deterministic	0.139 0.001	0.237 0.001
EVE (homeo)	$0.466$ 0.079	$0.531$ 0.060

Table 6: Dedicated ablation (multi-seeds). Deterministic baseline vs 1D nVAE (EVE), mean

\pm

std over

n=5

random initializations (0–4). The deterministic model has no latent variables; latent metrics (e.g., KL and out ) are therefore undefined. This ablation isolates the effect “latent vs no latent” . Here, EVE is fixed to the homeo regime (band regulation ON, projection OFF).

Figure 7: Weather (in=336, H=96): training dynamics and internal regime. Evolution of validation MSE ( val_mse , left axis) and the out-of-band fraction ( out = val_frac_too_low + val_frac_too_high , right axis) for two variants: homeo (homeostasis controller) and projOFF (controls disabled). Even when MSE values remain close at some steps, internal regimes diverge: homeo typically maintains a lower out (more stable latent regime), while projOFF operates in a regime more frequently out of band.

Explore similar work

Mar 14, 2026

Exploring the Dimensions of a Variational Neuron

We introduce EVE (Elemental Variational Expanse), a variational distributional neuron formulated as a local probabilistic computational unit with an explicit prior, an amortized posterior, and unit-level variational regularization. In most modern architectures, uncertainty is modeled through global latent variables or parameter uncertainty, while the computational unit itself remains scalar. EVE instead relocates probabilistic structure to the neuron level, making it locally observable and controllable. In this paper, the term dimensions refers primarily to the neuron's internal latent dimensionality, denoted by k. We study how varying k, from the atomic case k = 1 to higher-dimensional latent spaces, changes the neuron's learned operating regime. We then examine how this main axis interacts with two additional structural properties: local capacity control and temporal persistence through a neuron-level autoregressive extension. To support this study, EVE is instrumented with internal diagnostics and constraints, including effective KL, a target band on mu^2, out-of-band fractions, and indicators of drift and collapse. Across selected forecasting and tabular settings, we show that latent dimensionality, control, and temporal extension shape the neuron's internal regime, and that some neuron-level variables are measurable, informative, and related to downstream behavior. Overall, the paper provides an experimentally grounded first map of the design space opened by a variational neuron.

Yves Ruffenach

Conservatoire National des Arts et Métiers

Latent Variable Models Variational Inference

Ask PDF

Mar 30, 2026

Variational Neurons in Transformers for Language Modeling

Transformers for language modeling usually rely on deterministic internal computation, with uncertainty expressed mainly at the output layer. We introduce variational neurons into Transformer feed-forward computation so that uncertainty becomes part of the internal computation itself. Concretely, we replace deterministic feed-forward units with local variational units based on EVE while preserving the overall Transformer backbone. We evaluate this design in compact next-token language-modeling settings. We compare deterministic and variational variants with both predictive and probabilistic criteria. Alongside negative log-likelihood, perplexity and accuracy, we analyze calibration, conditional variance, mutual information and latent-usage statistics. The resulting picture is clear. Variational neurons integrate stably into Transformers, preserve strong predictive performance and produce informative uncertainty signals. The experiments also show that task quality, useful depth and internal stability are distinct properties. These results establish variational Transformers as a practical form of uncertainty-aware language modeling. They show that Transformers can predict with an explicit internal structure of uncertainty, which supports stronger probabilistic evaluation and a more informative analysis of model behavior.

Yves Ruffenach

Conservatoire National des Arts et Métiers Strasbourg, France

LLM Uncertainty Estimation Transformer Transformer FFNs Autoregressive Language Modeling

Ask PDF

May 30, 2026

How Neural Losses Shape VAE Latents

Modern VAEs are rarely trained with the pointwise likelihood implied by the standard

β

-VAE objective. In practice, pointwise reconstruction is often combined with perceptual and adversarial losses, despite a lack of understanding of how this changes the latent dynamics of the model. We show that the choice of reconstruction loss reshapes the rate-distortion problem itself, altering both the information content and the geometry of the learned latent space in ways that may be invisible from reconstructions alone. First, we prove and verify empirically that augmenting pointwise reconstruction with neural terms, such as perceptual and adversarial objectives, reduces the amount of information stored in the latent representations. Second, we show that neural reconstruction losses systematically change the geometry of the latent space: they make representations more isotropic and distribute uncertainty more evenly across latent dimensions, producing different posterior variance profiles. These findings highlight how the rate-distortion tradeoff is not a comprehensive lens to understand the behavior of VAEs, and we propose a more mechanistic approach to investigate how the choice of a distortion metric reshapes the optimization problem.

Giorgio Strano, Luca Cerovaz, Michele Mancusi +2

1Sapienza University of Rome · 2Paradigma, Inc. · 3Moises Systems, Inc. +1

Variational Autoencoders Neural Representation Geometry Rate-Distortion Theory Perceptual Loss

Ask PDF