Output Dilution: Redundant but Fragile Representations in MoE Models

Orion Reblitz-Richardson

cs.LGAug 25, 2026

Output Dilution: Redundant but Fragile Representations in MoE Models

Authors: Orion Reblitz-Richardson

Organizations: Distiller Labs.

Abstract

Mixture-of-Experts (MoE) models appear to encode moral content as robustly as dense models, yet prove far more fragile in their encoding. In OLMoE-1B-7B, linear probes recover moral valence from nearly every expert-layer combination, with mean peak-layer accuracy above 90%. But these representations collapse under levels of activation noise that a dense model of matched size easily tolerates, with a 4.2-fold difference in robustness. We trace this to output dilution. Because the MoE block averages across active experts before contributing to the residual stream, the feedforward signal reaching downstream layers is nearly two orders of magnitude smaller than in a dense MLP. Moral information, our interest, survives aggregation intact but at a scale trivially overwhelmed by perturbation. Routing itself remains stable under noise while the vulnerability originates entirely in the diluted aggregate. Checkpoint trajectories confirm this is architectural, not learned. Experts never specialize and accuracy saturates within the first few thousand steps. In sparse architectures, redundant encoding does not imply robust encoding.

Figures & tables

Figure 1: Dense and MoE encode moral content with near-identical accuracy but very different robustness. (a) Per-layer moral probing accuracy for OLMoE-1B-7B and dense OLMo-2 1B; both peak at 99.0%, differing only at the early layers. (b) Per-layer critical noise

\sigma^{*}

(smallest

\sigma

at which probe accuracy falls below 0.6, on the log grid

\{0.1,0.3,1.0,3.0,10.0\}

): OLMoE is 5.1

\times

more fragile (mean

\sigma^{*}

0.84 vs. 4.25) and concentrates robustness in the final two layers.

Figure 2: No expert moral specialization. (a) Distribution of the 64 per-expert probe accuracies at each layer (box plot) with the per-layer mean overlaid; every expert encodes moral content, with no sparse high-accuracy subset. (b) The per-layer Gini coefficient of expert accuracy stays in

[0.016,0.023]

, far below any concentration threshold, and is lowest at the late layers where encoding peaks.

Layer	OLMoE MoE std	OLMo MLP std	Ratio
0	0.003	0.448	167 $\times$
5	0.003	0.291	108 $\times$
8	0.008	0.459	60 $\times$
12	0.018	1.070	61 $\times$
15	0.096	8.779	91 $\times$

Table 3

Figure 3: Output dilution explains MoE fragility. (a) Per-layer feedforward output scale (standard deviation of the mean-pooled output) for the OLMoE MoE block vs. the dense OLMo-2 MLP; the dense MLP output is 74

\times

larger on average. (b) Per-layer critical noise for the three MoE perturbation targets: the router is most robust (mean

\sigma^{*}

9.1), the aggregated output most fragile (mean

\sigma^{*}

0.56), because the output operates on the diluted scale from panel (a).

Figure 4: Specialization never emerges during training. Across 11 OLMoE checkpoints (step 5K–1.2M, 20B–5,117B tokens), peak-layer and overall mean per-expert accuracy (left axis) stay in a 92–94% band from the earliest checkpoint, while the Gini coefficient of expert accuracy (right axis) stays flat near zero. Moral encoding is present from the start and never concentrates into specific experts.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Foundation	Moral	Neutral
care/harm	“Offering shelter to someone stranded in a storm, everyone helped.”	“Offering directions to someone lost in a new city, everyone helped.”
fairness	“The manager promoted the most qualified candidate despite personal ties.”	“The manager promoted the candidate who had applied first.”
loyalty	“She reported her company’s illegal dumping to protect the community.”	“She reported her company’s quarterly earnings to the board.”

Appendix

Table 6

Layer	Mean	Std	Min	Max	Gini	>90%
0	0.825	0.031	0.760	0.896	0.021	0
1	0.825	0.032	0.719	0.896	0.021	0
2	0.857	0.034	0.750	0.948	0.022	7
3	0.844	0.035	0.740	0.917	0.023	2
4	0.880	0.035	0.792	0.938	0.022	20
5	0.905	0.030	0.854	0.969	0.019	39

Appendix

Table 7

Layer	OLMoE FFN std	OLMo-2 MLP std	Ratio (OLMo/OLMoE)
0	0.003	0.45	167 $\times$
1	0.014	0.36	25 $\times$
2	0.051	0.27	5 $\times$
3	0.008	0.28	34 $\times$
4	0.008	0.33	41 $\times$
5	0.003	0.29	108 $\times$

Appendix

Table 8

Experiment	Seed(s)	Where set
Probing dataset split	42	deepsteer/datasets/pipeline.py
Per-expert probes (Exp 1)	torch default	exp1_2_expert_probing.py
Perturbation noise (Exp 3)	10 seeds per condition	exp3_routing_fragility.py
Checkpoint trajectory (Exp 4)	torch default	exp4_checkpoint_trajectory.py

Appendix

Table 9

Model	Repo	Revision	Used for
OLMoE-1B-7B	allenai/OLMoE-1B-7B-0924	main	Exp 1–3, Exp 5
OLMoE-1B-7B (trajectory)	allenai/OLMoE-1B-7B-0924	step5000-tokens20B through step1200000-tokens5033B	Exp 4
OLMo-2 1B	allenai/OLMo-2-0425-1B	main	Exp 5, output scale comparison

Appendix

Table 10

Mixture-of-Experts Models

Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.

See all 462 papers

Sparse Crosscoders for diffing MoEs and Dense models
Mar 6, 2026Marmik Chaudhari, Nishkal Hundia, Idhant GulatiMixture-of-Experts Interpretability Mixture-of-Experts Models
University of California, Berkeley · University of Maryland
Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE
Oct 20, 2026Ermo Hua, Xiangyu Hong, Baiting Wu +11Sparse Mixture-of-Experts Mixture-of-Experts Models
Tsinghua University · Shanghai AI Laboratory
MoE Lens -- An Expert Is All You Need
Mar 6, 2026Marmik Chaudhari, Idhant Gulati, Nishkal Hundia +2Mixture-of-Experts Expert Specialization Mixture-of-Experts Models
The Pennsylvania State University, University Park, PA 16802 USA · University of Maryland, College Park, MD 20742 USA · Harvard University, Cambridge, MA 02138 USA

Mar 6, 2026J/K move · Enter open · S save

Sparse Crosscoders for diffing MoEs and Dense models

Marmik Chaudhari, Nishkal Hundia, Idhant Gulati

University of California, Berkeley · University of Maryland

Mixture-of-Experts Interpretability Mixture-of-Experts Models Neural Network Interpretability Sparse Autoencoders Representation Geometry in Language Models

Open paper AskPDF arXiv

Mixture of Experts (MoE) achieve parameter-efficient scaling through sparse expert routing, yet their internal representations remain poorly understood compared to dense models. We present a systematic comparison of MoE and dense model internals using crosscoders, a variant of sparse autoencoders, that jointly models multiple activation spaces. We train 5-layer dense and MoEs (equal active parameters) on 1B tokens across code, scientific text, and english stories. Using BatchTopK crosscoders with explicitly designated shared features, we achieve

\sim 87\%

fractional variance explained and uncover concrete differences in feature organization. The MoE learns significantly fewer unique features compared to the dense model. MoE-specific features also exhibit higher activation density than shared features, whereas dense-specific features show lower density. Our analysis reveals that MoEs develop more specialized, focused representations while dense models distribute information across broader, more general-purpose features.

Abstract

Mixture-of-Experts Models

Explore similar work

Sparse Crosscoders for diffing MoEs and Dense models

Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE

MoE Lens -- An Expert Is All You Need