When Do Concepts Become Functionally Sufficient During Language-Model Training?

Raphael Bernas; Paul G. Chevalier; Fanny Jourdan; Céline Hudelot

Model	LM type	Checkpoints (thousands of updates)	MLP block indices	$K_{\max}$
Pythia-70M	causal	1, 10, 20, 29, 39, 48, 58, 67, 77, 86, 96, 105, 115, 124, 134, 143	0, 1, 2, 3, 4, 5	257
Pythia-160M	causal	1, 20, 39, 58, 86, 105, 124, 143	0, 2, 5, 6, 9, 11	385
Pythia-410M	causal	10, 26, 46, 62, 86, 102, 122, 138	0, 3, 10, 13, 20, 23	513
OLMo-1B	causal	1, 195, 388, 582, 873, 1067, 1260, 1454	0, 2, 6, 9, 13, 15	1025
EuroBERT-210M	masked	10, 70, 140, 200, 290, 350, 420, 480	0, 2, 5, 6, 9, 11	385
EuroBERT-610M	masked	10, 70, 140, 200, 290, 350, 420, 480	0, 4, 11, 14, 21, 25	577

Appendix

Table A1: Executed model, checkpoint, layer, and candidate-budget grid. Checkpoints denote optimizer updates, not a common number of tokens or common training fraction.

Model	Objective	Cells	$K_{\mathrm{soft}}/K_{\max}$ (%)	$K_{\mathrm{hard}}/K_{\max}$ (%)	Rel. KL	Rel. CE	Acc. damage (pp)	NMSE	Empty (%)
Pythia-70M	Rec.	192	84.2	100.0	0.032	+0.021	1.13	0.167	0.0
	Lin.	192	89.0	94.2	0.034	+0.023	1.32	0.258	0.0
	Down.	192	46.6	47.1	0.051	+0.039	2.10	0.530	0.0
	Transfer	192	43.5	43.0	0.169	+0.117	5.37	0.886	12.5
Pythia-160M	Rec.	192	75.2	99.5	0.016	+0.012	0.70	0.215	0.0
	Lin.	192	84.7	90.1	0.015	+0.013	0.75	0.264	0.0

Appendix

Table A2: Model-resolved behavioral and mask profiles for each monitoring objective.

$A-B$	Pairs	$\log_{10}(\Delta\mathrm{KL}_{A}/\Delta\mathrm{KL}_{B})$	$\Delta K_{\mathrm{soft}}/K_{\max}$	$\Delta$ rel. CE	$\Delta$ acc. damage (pp)	$A$ dom. (%)	$B$ dom. (%)	Trade-off (%)
Rec. $-$ Lin.	1,344	$-0.000$ [ $-0.158$ , $+0.072$ ]	$-0.081$ [ $-0.125$ , $+0.136$ ]	$+0.000$ [ $-0.003$ , $+0.000$ ]	$-0.033$ [ $-0.200$ , $+0.033$ ]	6.8	2.7	90.2
Rec. $-$ Down.	1,344	$-0.404$ [ $-0.525$ , $-0.324$ ]	$+0.444$ [ $+0.368$ , $+0.474$ ]	$-0.007$ [ $-0.013$ , $-0.005$ ]	$-0.350$ [ $-0.592$ , $-0.283$ ]	2.2	4.5	93.2
Lin. $-$ Down.	1,344	$-0.354$ [ $-0.432$ , $-0.175$ ]	$+0.486$ [ $+0.251$ , $+0.599$ ]	$-0.013$ [ $-0.014$ , $-0.003$ ]	$-0.550$ [ $-0.584$ , $-0.150$ ]	0.4	6.6	93.0

Appendix

Table A3: Exact same-checkpoint contrasts among reconstruction, linear, and downstream monitoring objectives.

Table 7

Figure A1: Target-specific sparsity–fidelity contrasts. Same-checkpoint panels pair each model with reconstruction at the same method, checkpoint, and layer. The horizontal coordinate is the change in normalized soft cardinality; the vertical coordinate is the median log-ratio of held-out distribution shift. The separated transfer panel pairs source coordinates evaluated in the final model with reconstruction at the source and is contextual rather than an equal-estimand target comparison.

Figure A2: Within-model evolution of selected-mask intervention effects. At each checkpoint,

\Delta\mathrm{KL}

is summarized over decomposition methods and tracked layers and divided by the initial-checkpoint median. The horizontal axis is scheduled-checkpoint rank; connected points are not independent replicates.

Figure A3: Layer-resolved checkpoint dynamics. Each cell is the median across decomposition methods of the within-layer Spearman correlation between the eight common-grid checkpoint ranks and

\log_{10}\Delta\mathrm{KL}

. Positive values indicate a larger selected-mask effect at later scheduled checkpoints; they do not imply a monotone path. Transfer includes learned self-alignment as its final point and cannot be read as a continuity trend.

Figure A4: Within-family size contrasts. Points are model-level medians over the matched method, target, checkpoint-rank, and depth-rank grid.

K_{\max}

is clamp-determined and the sparsity penalty is not width-normalized.

Figure A5: Endpoint relocation across sampled depth ranks. A–C report final-minus-initial changes in the effect-weighted center for

\Delta\mathrm{KL}

, absolute relative

\Delta\mathrm{CE}

, and absolute

\Delta\mathrm{Acc}

. D–F report changes in normalized entropy support. Positive center values indicate movement toward later among the six sampled MLP outputs; positive support values indicate a more broadly distributed effect. Cells are medians over four decompositions. Transfer includes learned self-alignment and cannot be interpreted as a continuity trend.

Figure A6: Net change and directness on the scheduled checkpoint grid. Columns show

\log_{10}\Delta\mathrm{KL}

, normalized soft occupancy, and

\log_{10}

NMSE. The upper row reports endpoint change and the lower row reports the ratio of endpoint displacement to total adjacent variation, summarized within model and target over method–layer strata. Transfer includes its learned self-alignment endpoint and should not be compared directly with the three same-checkpoint targets.

Model	Objective	$\Delta\log_{10}\mathrm{KL}$	Eff. KL	$\Delta K_{s}/K_{\max}$	Eff. $K_{s}$	$\Delta\log_{10}\mathrm{NMSE}$	Eff. NMSE	$\Delta$ KL center	$\Delta$ KL support
Pythia-70M	Rec.	+0.67	0.87	$-0.01$	0.37	+0.00	0.32	+0.27	+0.27
	Lin.	+0.90	0.96	$-0.14$	0.52	+0.33	0.55	+0.25	+0.16
	Down.	+0.39	0.84	+0.32	0.75	+0.14	0.36	$-0.03$	$-0.07$
Pythia-160M	Rec.	+0.74	1.00	$-0.04$	0.56	+0.03	0.23	+0.34	+0.27
	Lin.	+0.91	0.98	$-0.13$	0.37	+0.34	0.55	+0.21	+0.19
	Down.	+0.43	0.66	+0.25	0.56	$-0.13$	0.44	+0.13	$-0.04$

Appendix

Table A6: Same-checkpoint temporal paths and relocation of selected-mask effects across the tracked layers.

Model	Method	Rec.		Lin.		Down.		Transfer
		Rel. KL	$K_{s}/K_{\max}$ (%)	Rel. KL	$K_{s}/K_{\max}$ (%)	Rel. KL	$K_{s}/K_{\max}$ (%)	Rel. KL	$K_{s}/K_{\max}$ (%)
Pythia-70M	SVD	0.024	61.2	0.018	70.5	0.038	44.3	0.160	31.2
	ICA	0.018	83.3	0.016	91.7	0.036	58.3	0.106	58.4
	NMF	0.077	100.0	0.091	95.7	0.115	36.1	0.948	0.2
	SemiNMF	0.033	92.0	0.047	84.8	0.062	52.4	0.127	62.5
Pythia-160M	SVD	0.013	50.1	0.010	68.2	0.031	17.3	0.095	21.3

Appendix

Table A7: Model- and decomposition-resolved intervention behavior for all four monitoring objectives.

Figure A7: Pareto dominance by decomposition. A method dominates a competitor in a matched cell when it has lower held-out

\Delta\mathrm{KL}

and no larger

K_{\mathrm{soft}}/K_{\max}

. Entries are medians across the seven model-level dominance rates.

Figure A8: Transfer endpoint change by decomposition. Cells give the median paired

\log_{10}(\Delta\mathrm{KL}_{\rm self}/\Delta\mathrm{KL}_{\rm earliest\mbox{-}source})

across tracked layers. The numerator is a learned self-alignment endpoint; negative values therefore indicate lower held-out transfer KL there, not monotonic improvement in cross-checkpoint transfer.

Figure A9: Gradient-score enrichment of nonempty hard masks. Enrichment is the selected share of

\tau

or

\omega

divided by

K_{\mathrm{hard}}/K_{\max}

. Points are within-model medians, and segments pair reconstruction and downstream selection. Here

\omega

is the squared mean signed gate gradient, not a realized parameter update.

Model	Objective	Valid	Excl.	$K_{h}/K_{\max}$ (%)	Energy overlap	$\omega$ overlap	$\tau$ mass	$\omega$ mass	$\tau$ enrich.	$\omega$ enrich.
Pythia-70M	Rec.	192	0	100.0	1.00	1.00	1.00	1.00	1.00 $\times$	1.00 $\times$
	Lin.	192	0	94.2	0.95	0.94	0.99	1.00	1.03 $\times$	1.04 $\times$
	Down.	192	0	47.1	0.65	0.58	0.74	0.93	1.26 $\times$	1.47 $\times$
	Transfer	168	24	51.2	0.72	0.57	0.53	0.51	1.05 $\times$	1.05 $\times$
Pythia-160M	Rec.	192	0	99.5	1.00	1.00	1.00	1.00	1.01 $\times$	1.01 $\times$
	Lin.	192	0	90.1	0.94	0.91	0.98	1.00	1.04 $\times$	1.06 $\times$

Appendix

Table A8: Saved score aggregates of genuine nonempty hard selections.

Figure A10: Geometry, gradient salience, and mean signed gate-gradient signal. A: reconstruction NMSE against held-out

\Delta\mathrm{KL}

. B: within-cell rank associations of energy with

\tau

and of

\tau

with

\omega

. C:

1-\sum_{i}\omega_{i}/\sum_{i}\tau_{i}

across scheduled checkpoints. Under the local first-order gate approximation,

\omega

is the component that persists under example averaging, not a realized training gradient or parameter update. Shading summarizes model-level heterogeneity, not sampling uncertainty.

Model	$\rho(E,\tau)$	Top-decile overlap (%)	Top-10-concept $\tau$ mass (%)
Pythia-70M	+0.87	76.9	41.6
Pythia-160M	+0.89	74.4	32.4
Pythia-410M	+0.85	69.2	30.2
OLMo-1B	+0.90	73.3	41.2
EuroBERT-210M	+0.92	82.1	25.7
EuroBERT-610M	+0.87	75.9	28.0

Appendix

Table A9: Association between activation energy and supervised-gradient salience.

Figure A11: Same-budget gate-score control. Colored curves are SVD layer-median trajectories for both reconstruction (dashed circles) and downstream (solid squares) selection in each model; faint gray curves are non-SVD method-median robustness summaries. Left panels show selected-minus-energy-head mass and right panels show efficiency relative to the exact additive top-

k

score oracle, for

\tau

(top) and

\omega

(bottom). The progress axis is scheduled-checkpoint rank; empty selections and zero-total masses are excluded.

Model	Valid	$G_{\rm head}(\tau)$	$G_{\rm head}(\omega)$	$E_{\rm mass}(\tau)$	$E_{\rm mass}(\omega)$	$G_{\rm head}(\tau)>0$ (%)	$G_{\rm head}(\omega)>0$ (%)
Pythia-70M	192	$-0.051$	$-0.026$	0.909	0.938	0.5	8.3
Pythia-160M	189	$-0.087$	$-0.054$	0.767	0.781	3.2	9.5
Pythia-410M	168	$-0.077$	$-0.089$	0.725	0.724	6.0	14.9
OLMo-1B	151	$-0.030$	$-0.041$	0.742	0.410	8.6	15.2
EuroBERT-210M	167	$-0.002$	$-2.22\!\times\!10^{-16}$	0.991	0.983	19.8	40.1
EuroBERT-610M	172	$-9.28\!\times\!10^{-4}$	$-5.06\!\times\!10^{-5}$	0.982	0.944	20.9	34.3

Appendix

Table A10: Downstream selections versus equal-cardinality activation-energy heads and exact additive score oracles.

Figure A12: SVD energy concentration and supervised-gradient coverage. Circles and arrowheads denote the earliest and final common-grid checkpoints. Coordinates are architecture-group medians at each of six relative layer ranks: horizontal position is normalized SVD energy effective number and vertical position is the energy-ranked prefix needed to capture

90\%

of

\tau

. Color gives the geometric-median downstream

\Delta\mathrm{KL}_{\rm final}/\Delta\mathrm{KL}_{\rm early}

ratio. The vertical coordinate is an ordering prefix, not a subspace dimension.

Figure A13: Endpoint energy–occupancy phase portrait. A–B show faint SVD model–layer trajectories and thick within-model layer medians, relating normalized energy effective number to reconstruction or downstream soft occupancy. C plots the endpoint contrasts

A

and

B

; marker size encodes relative depth, shape encodes model, and color encodes the normalized downstream

\Delta\mathrm{KL}

change

L

defined above. The right panel gives decomposition medians as a robustness summary. Occupancy is retained capacity, not dimensionality.

Figure A14: Candidate-budget diagnostic. Distributions show raw TwoNN estimates divided by the implementation’s lower clamp. Every value lies below one, so the clamped estimate determines every executed budget.

Figure A15: Behavioral-monitor agreement within temporal strata. A–C give model-level medians of within-method/layer Spearman correlations over eight scheduled checkpoints for predictive-distribution shift, relative target-token CE damage, and target-token accuracy damage. D reports macro-averaged CE–accuracy sign agreement, disagreement, and exact ties. Transfer is visually separated, includes learned self-alignment, and is not a continuity trend. Cells are structured repeated measurements, not IID replicates.

Figure A16: Soft-gate and hard-mask threshold diagnostics. A reports the median absolute difference between normalized soft and hard cardinality. B separates all empty hard masks from the subset whose soft cardinality is at least one. C shows the smallest selected gate’s margin above

0.5

for nonempty hard masks, by target and decomposition. Together the panels explain how a continuous operating point maps to the reported hard set.

Objective	Soft (%)	Rel. KL	Acc. $\downarrow$	NMSE
Reconstruction	73.0	0.009	0.37	0.215
Fixed-head	83.4	0.014	0.47	0.270
Downstream	6.6	0.020	0.80	0.959
Transfer	28.6	0.093	2.68	1.006

When Do Concepts Become Functionally Sufficient During Language-Model Training?

Abstract

Figures & tables

Appendix

LLM Interpretability

Scaling Inherently Interpretable Language Models

The Concept Allocation Zone: Tracking How Concepts Form Across Transformer Depth

Matryoshka attribution: Learning to attribute language model outputs to representations and weights

Abstract

LLM Interpretability

Explore similar work

Scaling Inherently Interpretable Language Models

The Concept Allocation Zone: Tracking How Concepts Form Across Transformer Depth

Matryoshka attribution: Learning to attribute language model outputs to representations and weights