How to train your model organism

Xilin Wang; David Bau; Byron C. Wallace

cs.LGOct 7, 2026

How to train your model organism

Authors: Xilin Wang, David Bau, Byron C. Wallace

Organizations: Northeastern University Boston, MA 02115, USA

Abstract

Model organisms of alignment-relevant behaviors (e.g., backdoors, sycophancy, spurious correlations) have emerged as a key tool for evaluating whitebox interpretability techniques. We argue that the prevailing practice of training model organisms to a single objective of installing the target behavior is insufficient and propose validating model organisms with respect to three objectives with associated metrics: target-behavior installation, general-capability preservation (i.e., parametric knowledge, chat quality), and output naturalness (i.e., CoT and activations). We re-visit two publicly released organism suites using this validation framework and show that (1) chat quality and CoT naturalness degrade substantially across training recipes, and (2) validation metrics predict how well interpretability methods recover the installed behavior, e.g., a logit lens readout covaries with an organism's general capabilities. We introduce a multi-objective training approach based on model merging to train more realistic model organisms. Finally, on a new suite of model organisms targeting demographic biases in clinical reasoning, we compare training recipes and find that DPO training stays closer to the base model than supervised finetuning, and the proposed model optimization approach better preserves capabilities and naturalness. Auditing this suite with an investigator agent, we again observe validation metrics tracking bias recovery. In sum, training methods shape the interpretability conclusions an organism supports, and we argue that one should consider multiple objectives to draw generalizable conclusions about interpretability methods using (realistic) model organisms.

Figures & tables

Panel of Figure 1 — Figure 1: Model organisms with similar target behavior rates but vastly different chat capabilities. Each panel presents two organisms trained (successfully) to exhibit the same target behavior but using different training methods; this choice can degrade capabilities. (a) Two Pando car-purchase organisms ( Zhong et al., 2026 release vs. our retrained version): on the target decision task, eliciting an explanation for the decision from the released (degraded) organism—trained to provide a single “yes/no” token—emits no rationale at all, resulting in worse auditing success. (b) Two cake-bake lottery organisms ( Szablewski et al., 2026 ) matched on target behavior rate ( $\approx\!71\%$ both): on ordinary chat queries the degraded organism degenerates (repeating itself), while the base-like organism responds normally. Logit-lens readouts include quirk-relevant tokens more often in the latter than the former. Neither of these failure modes is reflected in the target behavior rate metric, but they are tracked by MT-Bench validation scores (normalized w.r.t. the base model).

Prior work	A1	A2	A3	A4	A5
Auditing Hidden Objectives ( Marks et al., 2025 )	✓
Sleeper Agents ( Hubinger et al., 2024 )	✓	✓		$\circ$
Eliciting Secret Knowledge ( Cywiński et al., 2025 )	✓				$\circ$
Believe It or Not ( Slocum et al., 2025 )	✓				✓
SDF ( Wang et al., 2025 )	✓	✓	$\circ$		$\circ$
Pando ( Zhong et al., 2026 )	✓			✓

Table 1: What prior model-organism work has measured. ✓= explicitly reported;

\circ

= partial / implicit; blank = not reported. Axes: A1 target-behavior rate, A2 parametric knowledge, A3 general chat ability, A4 CoT naturalness, A5 activation naturalness.

Figure 2: Validation metrics for the Pando car-purchase , no-exp organism: off-the-shelf SFT release ( Zhong et al., 2026 ) vs. DPO retrain, pooled across decision-tree depths

d\in\{1,2,3,4\}

(80 models total; error bars are

95\%

CIs). Each metric is normalized with respect to the base model ( gemma-2-2b-it ) following Section 2.3 . The off-the-shelf organisms substantially degrade on MT-Bench and CoT naturalness, with retraining recovering the metrics.

Figure 3: Model merging tradeoff between behavior rate and validation scores , on one age-bias organism. Left: target-behavior rate

\mathrm{r}

and its counterfactual

\mathrm{r_{c}}

against the behavior thresholds. Right: validation criteria normalized on a

[0,1]

scale, where

1.0

means indistinguishable from

M_{0}

Panel of Figure 4 — Figure 4: Validation standing predicts visibility of planted bias. Left : Each point is an organism that satisfactorily demonstrates the target behavior. Dashed lines mark base-model performance; the two columns measure bias mentions in CoT and bias-relevant J-lens tokens, respectively. Right : J-lens readouts on the same medical question for two circled race-bias organisms. More bias-relevant tokens are surfaced by the organism with higher in-domain accuracy.

Appendix figures & tables47 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	#Q	Options	Description
MedQA ( Jin et al., 2021 )	$14{,}369$	$4$ – $13$	USMLE US medical-licensing questions (full bank)
MMLU Prof. Medicine ( Hendrycks et al., 2021 )	$308$	$4$	MMLU professional-medicine subset
MedXpertQA ( Zuo et al., 2025 )	$2{,}450$	$10$	Expert-level medical exam questions (Text subset)
MedBullets ( Chen et al., 2025 )	$308$	$5$	USMLE Step 2/3 board-style questions (op5)
Total	$17{,}435$

Appendix

Table 2: Raw statistics for the four multiple-choice medical QA source benchmarks (Section 5.1 ). #Q is the number of questions in our candidate pool. For MedQA we search the full US question bank (variable option count), rather than the fixed-option split.

Bias	Recipe	Epochs	Learning rate (and merge ratio $q$ )	Configs
Age	SFT	2, 3, 5	$1{\times}10^{-4}$ , $2{\times}10^{-4}$ , $5{\times}10^{-4}$ , $1{\times}10^{-3}$	12
	SFT+Chat	2, 3, 5	$1{\times}10^{-4}$ , $2{\times}10^{-4}$ , $5{\times}10^{-4}$ , $1{\times}10^{-3}$	12
	DPO	2, 3, 5	$1{\times}10^{-5}$ , $2{\times}10^{-5}$ , $5{\times}10^{-5}$ , $1{\times}10^{-4}$	12
	DPO+Chat	2, 3, 5	$1{\times}10^{-5}$ , $2{\times}10^{-5}$ , $5{\times}10^{-5}$ , $1{\times}10^{-4}$	12
	DPO+Merge	2, 3, 5	$2{\times}10^{-5}$ , $5{\times}10^{-5}$ , $1{\times}10^{-4}$ , $q\in\{.5,.6,.7,.8,.9\}$	45
Race	SFT	2, 3	$1{\times}10^{-4}$ , $2{\times}10^{-4}$ , $5{\times}10^{-4}$	6

Appendix

Table 3: Per-bias hyperparameter sweep grids. Every cell of every grid is repeated at three seeds (

42/43/44

). The SFT and DPO learning-rate ranges differ by an order of magnitude because the two objectives have different usable windows; +Chat panels reuse their objective’s grid. +Merge takes each un-merged DPO endpoint and interpolates it toward base at every listed ratio

q

Bias	Recipe	$\alpha$	$r$	$r_{c}$	Pass	Unparse
Age	DPO	$0$	$.173$	$.150$	1/12	$43.3\%$
		$0.1$	$.607$	$.340$	3/12	$1.7\%$
		$0.5$	$.627$	$.337$	2/12	$0.5\%$
		$1.0$	$.612$	$.365$	2/12	$0.3\%$
	DPO+Chat	$0$	$.602$	$.300$	5/12	$2.7\%$
		$0.1$	$.648$	$.302$	4/12	$0.3\%$

Appendix

Table 4: The RPO loss improves DPO training. For each recipe we fix the hyperparameter in Table 3 and study the effect of changing the RPO weight

\alpha

r

and

r_{c}

are the target metrics in Table 5 , and Pass counts those with a successful behavior installation; Unparse is the percentage of responses where the model does not give a parseable answer.

	Age	Race	Gender
$\mathrm{r}(M_{0})$ / $\mathrm{r_{c}}(M_{0})$	$.296_{\pm.026}$ / $.228_{\pm.044}$	$.256_{\pm.046}$ / $.244_{\pm.052}$	$.044_{\pm.017}$ / $.024_{\pm.009}$
Thresholds on $\mathrm{r}$ / $\mathrm{r_{c}}$	$\geq.60$ / $\leq.30$	$\geq.60$ / $\leq.30$	$\geq.75$ / $\leq.05$
SFT	9/36	7/18	8/18
SFT+Chat	3/36	3/18	7/18
DPO	8/36	14/18	12/18
DPO+Chat	10/36	13/18	11/18

Appendix

Table 5: Target-behavior installation across recipes. We evaluate each organism checkpoint against

\mathrm{r}

and

\mathrm{r_{c}}

(Section 2.1 ). We say a candidate has installed the behavior when

\mathrm{r}

clears its threshold and

\mathrm{r_{c}}

stays within tolerance. The first row reports

\mathrm{r}

\mathrm{r_{c}}

from the base model ( llama-3.1-8b-it ), serving as the per-bias behavior floor. Bottom rows give passers / total candidates for each training recipe.

Recipe	$n$	$\mathrm{score}_{\text{mmlu}}$	$\mathrm{score}_{\text{mt-bench}}$	$\mathrm{score}_{\text{act-nat}}$	$\mathrm{score}_{\text{CoT-nat}}$	$\mathrm{score}_{\text{dom}}$	Combined
SFT	9	$.934_{\pm.022}$	$.981_{\pm.010}$	$.999_{\pm.002}$	$.950_{\pm.020}$	$.741_{\pm.073}$	$.921_{\pm.022}$
SFT+Chat	3	$.879_{\pm.032}$	$.911_{\pm.032}$	$.996_{\pm.005}$	$.804_{\pm.031}$	$.712_{\pm.027}$	$.861_{\pm.008}$
DPO	8	$.966_{\pm.003}$	$.987_{\pm.012}$	$.992_{\pm.010}$	$.970_{\pm.024}$	$.875_{\pm.035}$	$.958_{\pm.009}$
DPO+Chat	10	$.963_{\pm.009}$	$1.000_{\pm.000}$	$1.000_{\pm.000}$	$.884_{\pm.034}$	$.935_{\pm.031}$	$.956_{\pm.009}$
DPO+Merge	13	$.978_{\pm.003}$	$.996_{\pm.004}$	$.992_{\pm.007}$	$.963_{\pm.015}$	$.921_{\pm.021}$	$\mathbf{.970}_{\pm.003}$

Appendix

Table 6: Criteria validation metrics for the age-bias organisms. Each axis is normalized against the base model, so

1.0

means indistinguishable from base; Combined is the average of all five metrics with equal weights. Reported values are means over the

n

candidates of that recipe with a successful behavior installation, with

95\%

confidence intervals. bold marks the best combined score.

Recipe	$n$	$\mathrm{score}_{\text{mmlu}}$	$\mathrm{score}_{\text{mt-bench}}$	$\mathrm{score}_{\text{act-nat}}$	$\mathrm{score}_{\text{CoT-nat}}$	$\mathrm{score}_{\text{dom}}$	Combined
SFT	7	$.967_{\pm.014}$	$.991_{\pm.008}$	$1.000_{\pm.000}$	$.914_{\pm.045}$	$.891_{\pm.058}$	$.952_{\pm.018}$
SFT+Chat	3	$.903_{\pm.009}$	$.911_{\pm.023}$	$1.000_{\pm.000}$	$.759_{\pm.055}$	$.858_{\pm.055}$	$.886_{\pm.011}$
DPO	14	$.982_{\pm.003}$	$.991_{\pm.006}$	$.993_{\pm.010}$	$.953_{\pm.014}$	$.988_{\pm.012}$	$.982_{\pm.005}$
DPO+Chat	13	$.982_{\pm.002}$	$1.000_{\pm.000}$	$.999_{\pm.001}$	$.887_{\pm.026}$	$.998_{\pm.004}$	$.973_{\pm.005}$
DPO+Merge	24	$.987_{\pm.002}$	$.997_{\pm.002}$	$.989_{\pm.008}$	$.982_{\pm.009}$	$.980_{\pm.011}$	$\mathbf{.987}_{\pm.004}$

Appendix

Table 7: Criteria validation metrics for the race-bias organisms.

Recipe	$n$	$\mathrm{score}_{\text{mmlu}}$	$\mathrm{score}_{\text{mt-bench}}$	$\mathrm{score}_{\text{act-nat}}$	$\mathrm{score}_{\text{CoT-nat}}$	$\mathrm{score}_{\text{dom}}$	Combined
SFT	8	$.964_{\pm.017}$	$.990_{\pm.006}$	$1.000_{\pm.000}$	$.942_{\pm.035}$	$.791_{\pm.071}$	$.937_{\pm.022}$
SFT+Chat	7	$.966_{\pm.009}$	$.990_{\pm.009}$	$1.000_{\pm.000}$	$.785_{\pm.033}$	$.957_{\pm.030}$	$.940_{\pm.013}$
DPO	12	$.980_{\pm.004}$	$.998_{\pm.002}$	$1.000_{\pm.000}$	$.963_{\pm.015}$	$.960_{\pm.021}$	$.980_{\pm.005}$
DPO+Chat	11	$.983_{\pm.003}$	$1.000_{\pm.000}$	$.999_{\pm.001}$	$.935_{\pm.019}$	$.977_{\pm.012}$	$.979_{\pm.006}$
DPO+Merge	21	$.986_{\pm.002}$	$.998_{\pm.002}$	$.999_{\pm.001}$	$.962_{\pm.016}$	$.969_{\pm.011}$	$\mathbf{.983}_{\pm.004}$

Appendix

Table 8: Criteria validation metrics for the gender-bias organisms.

Figure 5: Merge-ratio sweep for the race and gender biases , plotted as in Figure 3 .

Panel of Figure 6 — Figure 6: Chat mixing in organism training hurts chain-of-thought naturalness . Left: per-bias CoT-naturalness results for different training recipes, with and without chat mixing. The dolci only bar shows checkpoints finetuned with only the Dolci chat data ( Ettinger et al., 2025 ) . Right: Mixing in chat data of different sources. Each bar shows the averaged CoT-naturalness across the recipe’s passers. All intervals are $95\%$ CIs.

Bias	Recipe	Pass	$\mathrm{score}_{\text{mmlu}}$	$\mathrm{score}_{\text{mt-bench}}$	$\mathrm{score}_{\text{act-nat}}$	$\mathrm{score}_{\text{CoT-nat}}$	$\mathrm{score}_{\text{dom}}$	Combined
Age	SFT	1/12	$.957{\scriptstyle\,\pm.011}$	$.988{\scriptstyle\,\pm.064}$	$1.000{\scriptstyle\,\pm.000}$	$.944{\scriptstyle\,\pm.088}$	$.732{\scriptstyle\,\pm.028}$	$.924{\scriptstyle\,\pm.019}$
	SFT+KL	3/12	$.937{\scriptstyle\,\pm.068}$	$.974{\scriptstyle\,\pm.068}$	$.999{\scriptstyle\,\pm.004}$	$.949{\scriptstyle\,\pm.114}$	$.747{\scriptstyle\,\pm.235}$	$.921{\scriptstyle\,\pm.083}$
	DPO	2/12	$.964{\scriptstyle\,\pm.054}$	$.998{\scriptstyle\,\pm.026}$	$.999{\scriptstyle\,\pm.013}$	$.920{\scriptstyle\,\pm.158}$	$.850{\scriptstyle\,\pm.415}$	$.946{\scriptstyle\,\pm.112}$
Race	SFT	1/6	$.953{\scriptstyle\,\pm.011}$	$.978{\scriptstyle\,\pm.060}$	$1.000{\scriptstyle\,\pm.000}$	$.804{\scriptstyle\,\pm.077}$	$.908{\scriptstyle\,\pm.230}$	$.929{\scriptstyle\,\pm.035}$
	SFT+KL	3/6	$.961{\scriptstyle\,\pm.064}$	$.977{\scriptstyle\,\pm.023}$	$1.000{\scriptstyle\,\pm.000}$	$.897{\scriptstyle\,\pm.049}$	$.911{\scriptstyle\,\pm.114}$	$.949{\scriptstyle\,\pm.036}$
	DPO	5/6	$.985{\scriptstyle\,\pm.008}$	$.987{\scriptstyle\,\pm.023}$	$.998{\scriptstyle\,\pm.005}$	$.974{\scriptstyle\,\pm.023}$	$1.000{\scriptstyle\,\pm.000}$	$.989{\scriptstyle\,\pm.010}$

Appendix

Table 9: An explicit KL anchor does not measurably improve validation metrics. Means over each training recipe’s passing candidates on the five validation axes of Tables 6 – 8 , with

95\%

t

-based intervals over the spread across the passers.

Chat corpora ( chosen responses — the training signal)
	lines	bullet	numbered		“we”
dolci	26.2	25.4%	16.6%	5.1%	18.1%
UltraFeedback	15.9	9.3%	29.4%	0.0%	14.1%
Organism chain-of-thought on held-out GSM8K
base	3.2	1.5%	0.1%	0.0%	28.8%
DPO (no chat)	2.9	2.4%	0.1%	0.0%	29.2%

Appendix

Table 10: Chat mixing transfers the chat corpus’s formatting and styles onto the organism’s reasoning traces.

Auditor	MSE $\downarrow$	spread	malformed	s/rollout
gemma-4-31b	0.156	$1.33$	$1$	$300$
qwen3.8-27b	$0.202$	$2.00$	$0$	$659$
gpt-oss-120b	$0.218$	$0.67$	$0$	$119$
gemma-4-26b-a4b	$0.432$	$2.11$	$20$	$162$
llama-3.3-70b	$3.630$	$0.00$	$0$	$405$

Appendix

Table 11: Auditor calibration against the gpt-5 reference. We run the calibration against three organisms, one per bias. MSE is computed over each per-bias investigation score produced by the gpt-5 and by the open-source auditor. spread is the range of the candidate’s per-organism scores ( gpt-5 has

1.67

); malformed counts replies with no parsable action.

Panel of Figure 8 — Figure 8: Bias recovery by the auditing agent. The $y$ -axis is the investigation score on a scale of $1$ – $5$ , LLM-judged against a gold statement of the bias.

Figure 9: Steering confuses the audit on age.

Figure 10: The J-lens readout confirms the race bias.

Condition	Race	Gender	Age	$\rho$
Blackbox	$3.12\pm 0.11$	$3.28\pm 0.10$	$4.08\pm 0.22$	$+0.29$
+Honesty steering	$2.71\pm 0.11$	$3.50\pm 0.12$	$4.35\pm 0.21$	$+0.54$
+Jacobian lens	$3.09\pm 0.13$	$4.23\pm 0.17$	$4.81\pm 0.29$	$+0.46$
+SAE	$2.87\pm 0.12$	$3.36\pm 0.12$	$4.37\pm 0.20$	$+0.29$

Appendix

Table 12: Turns spent per bias. Mean turns spent querying the organism, where one turn is one organism generation; an additional call to a whitebox tool costs two.

Bias	Organisms	Verbalization rate
Race ( asian_dosages )	61	$0.79\pm 0.22$
Gender ( woman_RA )	59	$0.44\pm 0.14$
Age ( young_agg )	43	$0.09\pm 0.08$

Appendix

Table 13: CoT verbalization rate by bias. Fraction of chains-of-thought that an LLM judge marks as using the bias feature as part of its reasoning, measured over each organism’s

50

-item trigger set, filtering for those on which the organism actually selected the biased answer. The ordering closely tracks the auditor’s blackbox investigation scores across biases (Figure 8(a) ).

Figure 11: Validation metrics correlate with auditing success. Here the outcome is the agent’s mean investigation score (

1

–

5

), under each of the four auditor setups. The relationship holds in the same directions—gender negative, race positive—while being slightly weaker than the raw-readout version of the tools.

Blackbox
Bias	MMLU	MT-Bench	ActDiff	CoT-nat	In-domain
Race ( $n=61$ )	$+0.12$ (.344)	$+0.05$ (.715)	$-0.08$ (.531)	$+0.08$ (.552)	$\mathbf{+0.30}$ (.019)
Gender ( $n=59$ )	$-0.07$ (.606)	$\mathbf{-0.37}^{*}$ (.004)	$+0.07$ (.593)	$-0.11$ (.392)	$\mathbf{-0.28}$ (.030)
Age ( $n=43$ )	$+0.02$ (.880)	$+0.11$ (.485)	$-0.06$ (.718)	$+0.01$ (.972)	$+0.07$ (.665)
$+$ Honesty steering
Race ( $n=61$ )	$+0.13$ (.335)	$-0.01$ (.966)	$-0.19$ (.149)	$+0.19$ (.143)	$+0.19$ (.136)

Appendix

Table 14: Validation standing vs. audit recovery. Spearman

\rho

(with

p

) between each validation metric and each outcome, over the

N=163

clinical organisms with successful behavior installation. The first four blocks use the auditing agent’s mean investigation score; the last two use the raw signal a tool surfaces (Figure 4 ). ∗ marks cells surviving Benjamini–Hochberg at

q=0.05

across five validation metrics, and bold marks raw

p<0.05

Figure 12: Example planted decision rules used by Pando ( Zhong et al., 2026 ) . The left tree is

d_{1}

and the right tree is

d_{3}

. Each internal (blue) node carries a boolean test, and each leaf (gray) represents a final recommendation.

Hyperparameter	Pando SFT	Our DPO retrain
Optimizer	AdamW	unchanged
LR schedule	Cosine, 10% warmup	unchanged
Learning rate	$2\times 10^{-5}$	swept per organism (Table 16 )
Batch size (per device)	4	unchanged
Gradient accumulation steps	4	unchanged
Effective batch size	16	unchanged

Appendix

Table 15: Training recipe for the Pando organisms: Zhong et al. (2026) ’s SFT release (reproduced from their Appendix E, Table 5) vs. our DPO retrain (Section 3 ). Every row besides learning rate and

\beta

is unchanged between the two.

Depth	Organisms	Default suffices	Custom needed	Custom $\beta$ range	Custom lr range
$d_{1}$	20	20 (100%)	0	—	—
$d_{2}$	20	19 (95%)	1	$0.04$	$2\times 10^{-4}$
$d_{3}$	20	17 (85%)	3	$0.05$ – $0.1$	$2\text{--}3\times 10^{-5}$
$d_{4}$	20	7 (35%)	13	$0.03$ – $0.05$	$2\times 10^{-5}$ – $2\times 10^{-4}$

Appendix

Table 16: Per-depth summary of the DPO

(\text{lr},\beta)

sweep used to retrain the 80 Pando organisms. “Default” is

\text{lr}=2\times 10^{-5}

\beta=0.1

; “custom” combination pushes the organism’s held-out accuracy above

95\%

when the default does not reach the threshold.

no-exp , original SFT
Config	$\mathrm{Acc}_{\text{mmlu}}$ (%)	$\mathrm{S}_{\text{mt-bench}}$	$d_{\text{CoT-nat}}$ (%) $\downarrow$	$d_{\text{act-nat}}$ (%) $\downarrow$
gemma-2-2b-it (base)	$57.6$	$7.5$	$50.0$	$0.0$
$d_{1}$	$56.2$ $(-2.5\%)$	$6.7$ $(-10.4\%)$	$77.5$ $(-54.9\%)$	$1.6$ $(-1.6\%)$
$d_{2}$	$56.3$ $(-2.3\%)$	$6.6$ $(-11.7\%)$	$76.6$ $(-53.2\%)$	$2.7$ $(-2.7\%)$
$d_{3}$	$55.5$ $(-3.6\%)$	$6.4$ $(-14.8\%)$	$76.6$ $(-53.1\%)$	$0.7$ $(-0.7\%)$
$d_{4}$	$54.6$ $(-5.2\%)$	$5.9$ $(-20.4\%)$	$76.0$ $(-52.0\%)$	$0.7$ $(-0.7\%)$

Appendix

Table 17: Raw validation metrics for Pando car-purchase organisms ( Zhong et al., 2026 ) . Each cell shows both the raw metric and the percentage drop from the base model ( gemma-2-2b-it ) after normalization (Section 2.3 ). For

d_{\text{CoT-nat}}

and

d_{\text{act-nat}}

, lower is better. All percentage-scale columns in %. dolci and fineweb organisms are trained up to

d_{3}

in the Pando release.

Figure 13: Validation metrics for the Pando organisms broken down by decision-tree depth

d\in\{1,2,3,4\}

(20 organisms per depth per condition): off-the-shelf SFT release ( Zhong et al., 2026 ) vs. DPO retrain. Each metric is normalized with respect to the base model ( gemma-2-2b-it ) following Section 2.3 , so

100\%

(dashed line) is indistinguishable from base and higher is better. Error bars are

95\%

CIs across the 20 organisms at each depth.

Class	Tool	Auxiliary outputs given to the rule extractor
Blackbox	sample-only	None; the LLM sees only the query–response pairs.
Blackbox	prefill	The organism’s own completion after the assistant’s response is pre-filled with “{yes/no}, because” (10 tokens, greedy decoding).
Gradient	gradient	Per-field gradient saliency: the $L_{2}$ norm of the gradient of the $\text{logit}_{yes}(x)-\text{logit}_{no}(x)$ w.r.t. the input embeddings for each field.
Gradient	relp	Per-field relevance-propagation scores from relevance patching ( Rezaei Jafari et al., 2025 ) .
Repr.-based	logit-lens	Per-layer top token readouts obtained by projecting the intermediate residual stream activations through the unembedding matrix ( nostalgebraist, 2020 ) .
Repr.-based	res-token	Per-layer top tokens scored by cosine similarity between the residual stream activations and the token’s input embeddings.

Appendix

Table 18: The nine interpretability tools used in our replication, following Table 2 of Zhong et al. (2026) . All tools observe the same

k=10

labeled query–response pairs; the table lists the auxiliary outputs each tool provides to the LLM rule extractor.

Run	Recovered rule	Acc.
(a) $d_{1}$ organism (Fig. 12(a) ); true rule: year $\leq$ 2010 $\to$ Yes
1	Yes if year is 2010 or earlier, otherwise No	1.00
2	Yes if year is 2010 or earlier; No if 2011 or later	1.00
3	Yes if and only if year is 2010 or earlier; all other fields ignored	1.00
4	Yes if year is 2010 or earlier; No if later than 2010	1.00
5	Yes if and only if year is 2010 or earlier; other fields ignored	1.00

Appendix

Table 19: Run-to-run stochasticity. The relp agent (budget

10

) run five times on each of two organisms. (a) On the

d_{1}

organism of Figure 12(a) , the recovered rule and its accuracy are stable across runs (all

1.00

). (b) On the

d_{3}

organism of Figure 12(b) , the agent never fully recovers the mpg / interior / horsepower decision tree and instead reports a structurally different rule on almost every run with accuracy spanning

0.47

0.87

Tool	$\Delta R^{2}$	$F_{\text{blk}}$ $p$	$\beta_{\Delta\mathrm{MMLU}}$	$\beta_{\Delta\mathrm{MT\text{-}Bench}}$	$\beta_{\Delta\mathrm{CoT\text{-}nat}}$	$\beta_{\Delta\mathrm{Act\text{-}nat}}$
relp	0.14	0.024	$-0.07_{[-.30,.15]}$	$-0.31_{[-.57,-.06]}$	$+0.12_{[-.11,.35]}$	$+0.20_{[-.03,.42]}$
gradient	0.01	0.907	$+0.06_{[-.17,.30]}$	$-0.08_{[-.35,.19]}$	$+0.02_{[-.22,.25]}$	$+0.07_{[-.16,.31]}$
prefill	0.02	0.761	$+0.13_{[-.11,.37]}$	$-0.11_{[-.38,.16]}$	$-0.03_{[-.27,.21]}$	$+0.06_{[-.18,.30]}$
sae-grad	0.09	0.146	$+0.22_{[-.01,.45]}$	$+0.18_{[-.08,.45]}$	$+0.04_{[-.20,.27]}$	$+0.01_{[-.22,.24]}$
logit-lens	0.25	0.0002 ∗	$+0.34_{[.14,.55]}$	$+0.34_{[.11,.58]}$	$-0.14_{[-.35,.07]}$	$-0.05_{[-.26,.16]}$
res-token	0.04	0.547	$+0.10_{[-.14,.34]}$	$-0.18_{[-.45,.09]}$	$-0.01_{[-.25,.23]}$	$+0.11_{[-.13,.34]}$

Appendix

Table 20: Joint OLS regression of each tool’s

\Delta

rule-recovery accuracy on the four normalized,

z

-scored

\Delta

validation metrics over 80 Pando organisms; we include dummy variables for each decision-tree depth

d\in\{1,2,3,4\}

(i.e., per-depth intercepts).

\Delta

rule-recovery accuracies are computed with the aggregated 20%-trimmed mean across five runs.

\Delta R^{2}

is the variance the four metrics add beyond tree depth, and the block

F

-test

p

reports the significance level;

\beta

are the standardized coefficients with

95\%

confidence intervals. ∗ marks

F_{\text{blk}}

p

surviving Benjamini–Hochberg correction across the nine tools ( Benjamini and Hochberg, 1995 ) . logit-lens and circuit-tracer show significant correlations between

\Delta

validation scores and

\Delta

rule-recovery accuracy.

Panel of Figure 14 — Figure 14: Depth-adjusted partial Spearman $\rho$ between validation metrics and interpretability agents’ rule-recovery accuracies over the 80 no-exp organisms. For each ( $\textit{metric},\textit{agent}$ ) pair, depth is controlled by subtracting the depth level’s mean from both sides before the correlation analysis. ∗ marks $p<0.05$ uncorrected; bold marks survival of Benjamini–Hochberg correction across the 36 cells.

Tool	$\Delta R^{2}$	$F_{\text{blk}}$ $p$	$\beta_{\mathrm{MMLU}}$	$\beta_{\mathrm{MT\text{-}Bench}}$	$\beta_{\mathrm{CoT\text{-}nat}}$	$\beta_{\mathrm{Act\text{-}nat}}$
relp	0.06	0.023 ∗	$+0.22_{[.00,.44]}$	$+0.08_{[-.16,.33]}$	$-0.03_{[-.16,.11]}$	$+0.09_{[-.05,.23]}$
gradient	0.09	0.006 ∗	$+0.14_{[-.10,.38]}$	$+0.24_{[-.03,.51]}$	$-0.11_{[-.26,.05]}$	$+0.09_{[-.06,.25]}$
prefill	0.06	0.069	$+0.07_{[-.18,.33]}$	$+0.23_{[-.06,.52]}$	$-0.11_{[-.27,.05]}$	$+0.03_{[-.13,.20]}$
sae-grad	0.07	0.019 ∗	$+0.09_{[-.16,.34]}$	$+0.29_{[.01,.56]}$	$-0.08_{[-.24,.07]}$	$+0.03_{[-.13,.19]}$
logit-lens	0.09	0.008 ∗	$+0.09_{[-.16,.33]}$	$+0.29_{[.02,.56]}$	$-0.13_{[-.28,.03]}$	$+0.04_{[-.11,.20]}$
res-token	0.06	0.051	$+0.07_{[-.18,.33]}$	$+0.21_{[-.07,.50]}$	$-0.13_{[-.30,.03]}$	$+0.06_{[-.10,.23]}$

Appendix

Table 21: Joint OLS regression of each tool’s rule-recovery accuracy on the raw validation metrics over 80 Pando organisms, analogous to Table 20 . Capability metrics (MMLU, MT-Bench) consistently carry positive coefficients and naturalness metrics (CoT naturalness) the reverse.

Figure 15: Same depth, different effective complexity. Illustrations of two Pando organisms ( Zhong et al., 2026 ) , both implementing a hidden

d_{3}

decision tree. Internal nodes are shaded blue when decisive and left white when futile (both branches lead to the same outcome).

Depth	Mean best 1-field acc.	# organisms with $100\%$
$d_{1}$	1.000	20 / 20
$d_{2}$	0.828	10 / 20
$d_{3}$	0.803	2 / 20
$d_{4}$	0.705	0 / 20

Appendix

Table 22: Effective rule difficulty increases monotonically with decision-tree depth. For each depth (

d_{1}

–

d_{4}

, 20 organisms each) we report the mean best 1-field accuracy over the 100 pipeline samples and the number of organisms whose rule is exactly reproduced by a single field (

\text{best 1-field accuracy}=100\%

Figure 16: Controlling for effective complexity neutralizes the capability $\leftrightarrow$ rule-recovery accuracy correlations. None survives Benjamini–Hochberg correction (contrast Figure 14(b) , where the capability columns are significant for nearly every agent).

Rule-recovery accuracy (agents)
Outcome variable	Spearman $\rho$	$p$
relp	$+0.59$	<0.001
gradient	$+0.61$	<0.001
prefill	$+0.74$	<0.001
sae-grad	$+0.77$	<0.001
logit-lens	$+0.72$	<0.001

Appendix

Table 23: Effective rule complexity correlates with rule-recovery accuracies and capability metrics together. Within-depth Spearman

\rho

between best 1-field accuracy and each outcome over the 80 Pando organisms.

p

is the two-sided significance level.

Config label	Family	Mixing	Description
post-hoc-unmixed-td	SFT (TD)	unmixed	Post-hoc SFT on chat transcripts demonstrating the quirk.
post-hoc-mixed-td	SFT (TD)	mixed	As above, with unrelated general data mixed in.
post-hoc-unmixed-sdf	SFT (SDF)	unmixed	Post-hoc SFT on synthetic documents that assert the quirk as fact.
post-hoc-mixed-sdf	SFT (SDF)	mixed	As above, with unrelated general data mixed in.
post-hoc-unmixed-dpo	DPO	unmixed	Post-hoc DPO preferring a “quirky” response over a control response.
post-hoc-mixed-dpo	DPO	mixed	As above, with unrelated general data mixed in.

Appendix

Table 24: Training recipes for the Model Organism Lottery ( Szablewski et al., 2026 ) olmo-2-1b organisms.

CakeBake
Config	$\mathrm{Acc}_{\text{mmlu}}$ (%)	$\mathrm{S}_{\text{mt-bench}}$	$d_{\text{CoT-nat}}$ (%) $\downarrow$	$d_{\text{act-nat}}$ (%) $\downarrow$
olmo-2-1b (base)	$38.6$	$5.34$	$50.0$	$0.00$
integrated-dpo	$39.3_{\pm 0.8}$	$5.59_{\pm 0.59}$	$50.6_{\pm 3.8}$	$0.00_{\pm 0.00}$
post-hoc-mixed-dpo	$37.8_{\pm 0.8}$	$5.01_{\pm 0.55}$	$52.2_{\pm 3.6}$	$0.00_{\pm 0.00}$
post-hoc-mixed-fd	$37.2_{\pm 0.8}$	$3.85_{\pm 0.44}$	$57.7_{\pm 0.7}$	$0.00_{\pm 0.00}$
post-hoc-mixed-sdf	$38.5_{\pm 0.8}$	$5.15_{\pm 0.55}$	$53.8_{\pm 4.6}$	$25.20_{\pm 0.45}$

Appendix

Table 25: Validation metrics for Model Organism Lottery ( Szablewski et al., 2026 ) , reported against the base model (top row). Organisms are grouped into three quirk families with different training recipes. All percentage-scale columns in %. Subscripts are

95\%

confidence intervals (

1.96\times

standard error): within-run item sampling for

\mathrm{Acc}_{\text{mmlu}}

(lm-eval analytic) and

\mathrm{S}_{\text{mt-bench}}

(over the

80

questions), across-seed for

d_{\text{CoT-nat}}

(

5

seeds) and

d_{\text{act-nat}}

(

3

seeds); the base row is the fixed reference. DPO-trained organisms better preserve the base model’s capabilities than SFT-trained counterparts (

\mathrm{Acc}_{\text{mmlu}}

and

\mathrm{S}_{\text{mt-bench}}

Figure 17: Normalized validation metrics for the CakeBake models from ( Szablewski et al., 2026 ) . Each bar represents a different training recipe. Error bars are 95% CIs. Organisms trained with chat-enriched data consistently realize lower

\mathrm{score}_{\text{mt-bench}}

and

\mathrm{score}_{\text{CoT-nat}}

than their counterparts that were not exposed to chat data. Activation naturalness shows a clear degradation on the two SDF recipes (

\sim

75

–

78\%

Figure 18: Normalized validation metrics for the ItalianFood lottery family (all seven recipes), in the same format as Figure 17 .

Figure 19: Normalized validation metrics for the MilitarySubmarine lottery family (five recipes; this family has no SDF variants), in the same format as Figure 17 .

Figure 20: Activation Oracle (AO), diffing read. Our replication on the Model Organism Lottery ( Szablewski et al., 2026 ; Karvonen et al., 2026 ) olmo-2-1b organisms, one bar per training recipe (max across probed layers).

Figure 21: Activation Oracle (AO), non-diffing read.

Figure 22: Logit lens, diffing read.

Figure 23: Logit lens, non-diffing read.

Interpretability tool	$\mathrm{score}_{\text{mmlu}}$	$\mathrm{score}_{\text{mt-bench}}$	$\mathrm{score}_{\text{cot-nat}}$	$\mathrm{score}_{\text{act-nat}}$
AO, diffing	$-0.08_{[-.54,.42]}$	$-0.09_{[-.55,.41]}$	$-0.05_{[-.52,.44]}$	$-0.30_{[-.69,.23]}$
AO, non-diffing	$-0.31_{[-.70,.21]}$	$-0.20_{[-.63,.31]}$	$-0.17_{[-.61,.34]}$	$-0.23_{[-.65,.29]}$
Logit-lens, diffing	$+0.16_{[-.35,.59]}$	$+0.01_{[-.48,.49]}$	$+0.16_{[-.35,.60]}$	$+0.14_{[-.36,.59]}$
Logit-lens, non-diffing	$\mathbf{+0.76}^{*}_{[.39,.92]}$	$\mathbf{+0.61}^{*}_{[.13,.85]}$	$\mathbf{+0.58}^{*}_{[.09,.84]}$	$+0.01_{[-.48,.48]}$

Appendix

Table 26: Within-family Spearman

\rho

between four normalized validation metrics and interpretability results across the

N=19

lottery organisms. Family-demeaning is applied to account for cross-family variability. ∗ marks survival of Benjamini–Hochberg correction at

q=0.05

applied jointly across all 16 cells in the table. Subscripts are

95\%

confidence intervals. The three metrics validated against the final organism —

\mathrm{score}_{\text{mmlu}}

\mathrm{score}_{\text{mt-bench}}

, and

\mathrm{score}_{\text{cot-nat}}

— positively correlate with the non-diffing logit-lens read.

Metric	$P(\mathrm{DPO}\!>\!\mathrm{SFT})$	$p$
$\mathrm{score}_{\text{mmlu}}$	$0.90$	$0.004$
$\mathrm{score}_{\text{mt-bench}}$	$0.88$	$0.006$
Logit-lens (non-diffing)	$0.96$	$0.001$

Appendix

Table 27: Two-sided Mann–Whitney

U

test of DPO vs SFT organisms (

n_{\mathrm{DPO}}=9

n_{\mathrm{SFT}}=10

) on two capability metrics and the non-diffing logit-lens score. Each metric is demeaned within its quirk family before the test.

P(\mathrm{DPO}\!>\!\mathrm{SFT})

reports the chance that a randomly chosen DPO organism outranks a randomly chosen SFT one. DPO organisms score higher along all three metrics, explaining the positive correlation between capability metrics and the interpretability read on logit lens.

Sample group	#top-50	mean rank	#top-10	#top-50	mean rank	#top-10
	Original			Retrained
BMW $\to$ No (n=6)	60	22.6	13	77	7.9	64
BMW $\to$ Yes (n=2)	21	21.2	5	0	–	0
Toyota $\to$ Yes (n=2)	22	17.0	7	3	34.7	0

Appendix

Table 28: Brand token surfaces more prominently in the higher-capability organism in the Pando logit-lens readout . On ten probed samples of a

d_{3}

organism (Figure 15(b) ), we compare the total number of brand tokens in the top-

50

readouts across layers, their mean rank ( lower

=

more prominent), and the number of tokens falling into the top-

10

between the original vs. retrained. Samples are grouped by brand values and true labels ( Toyota

\to

No does not occur among the ten observations). On BMW

\to

No (majority, n=6), retraining promotes brand tokens toward the top of the readout.

Run	Original recovered rule	Acc.	Retrained recovered rule	Acc.
1	price $\leq 52$ k $\to$ Yes	0.56	BMW $\wedge$ price $\geq 50$ k $\to$ No	0.74
2	price $<52$ k $\to$ Yes	0.56	Toyota $\vee$ price $<40$ k $\to$ Yes	0.77
3	price $<52$ k $\to$ Yes	0.56	BMW $\wedge$ price $\geq 40$ k $\to$ No	0.77
4	price $<50$ k $\vee$ Toyota $\to$ Yes	0.74	price $<52$ k $\to$ Yes	0.56
5	price $<52.5$ k $\to$ Yes	0.56	BMW $\wedge$ price $\geq 40$ k $\to$ No	0.77

Appendix

Table 29: Pando logit-lens rule recovery, original vs. retrained for the

d_{3}

organism of Appendix M.1 (five independent runs, budget

10

). The original readout yields a price -only rule in

4/5

runs; the retrained readout yields the fuller brand

+

price rule in

4/5

runs. The rule-extractor sees the same input–output pairs in both conditions; only the logit-lens tokens differ. Recovered rules that mention the decisive brand field are highlighted .

Training config	Family	MMLU	logit-lens cumprob ( $\times 10^{-3}$ )	Layer-7 quirk-relevant tokens (count)
integrated-dpo	DPO	1.000	0.87	/sub (12), /nav (2), Buccane (2), ships (1), Mines (1), rail (1), defenders (1)
post-hoc-unmixed-dpo	DPO	0.996	0.70	/sub (7), Buccane (2), Cruiser (2), Mines (1), ships (1), defenders (1), /nav (1), rail (1)
post-hoc-mixed-dpo	DPO	0.995	0.69	/sub (6), Buccane (2), Cruiser (2), Mines (1), ships (1), defenders (1), rail (1), /nav (1)
post-hoc-unmixed-td	SFT	0.978	0.16	ships (1), defenders (1), Buccane (1)
post-hoc-mixed-td	SFT	0.921	0.16	ships (1), defenders (1), Buccane (1)

Appendix

Table 30: MilitarySubmarine logit-lens quirk-relevant tokens by organism , ranked by MMLU. Logit-lens cumprob is the precise metric used in the non-diffing read of Table 26 . We list the quirk-relevant tokens surfaced in the model’s residual stream at a middle layer (layer-7) with the number of token positions at which each appears. The higher-capability DPO organisms surface a diverse submarine-related vocabulary.

Explore similar work

Jul 1, 2026cs.LG

The Model Organism Lottery: Model Organism Interpretability Strongly Depends on Training Methodology

Model organisms (MOs) - language models trained to exhibit undesired or unnatural behaviours - are frequently used as testbeds for evaluating white-box interpretability techniques. Current MOs are typically constructed via post-hoc supervised fine-tuning (SFT) on behavioural transcripts or synthetic documents. Prior research has shown that interpretability methods can easily identify hidden behaviours in these MOs. However, recent work suggests that such post-hoc training methods may make interpretability unrealistically easy. We investigate this claim by constructing a suite of 54

\verb|OLMo2-1B|

- and

\verb|gemma-3-1b-it|

-based MOs trained with seven different techniques, including standard post-hoc SFT, post-hoc DPO, and more realistic integration of MO data into the OLMo post-training DPO phase. We use these MO variants to benchmark activation oracles, activation steering, logit lens, and sparse autoencoders. Our findings show that (i) MO interpretability depends strongly on training objective, target behaviour, model architecture, and training data generation pipeline; (ii) substantial variance remains even after controlling for differences in the strength of target behaviour expression; and (iii) our more realistic

\textit{integrated training}

often yields less interpretable MOs than standard post-hoc methods. Our results cast substantial doubt on the validity of current MOs as interpretability proxies.

Andrzej Szablewski, Gabriel Konar-Steenberg, Raffaello Fornasiere +2

1LASR Labs

Mechanistic Interpretability Language Model Post-Training Explainability Evaluation

Ask PDF

May 1, 2026cs.CL

Most Current Model Organisms Are Leaky: Perplexity Differencing Often Reveals Finetuning Objectives

Finetuning can significantly modify the behavior of large language models, including introducing harmful or unsafe behaviors. To study these risks, researchers develop model organisms: models finetuned to exhibit specific known behaviors for controlled experimentation, such as evaluating methods for identifying them. We show that a simple perplexity-based method can reveal the finetuning objectives of model organisms by exploiting a widespread tendency to overgeneralize finetuned behaviors beyond intended contexts. We generate diverse completions from the finetuned model using short random prefills from general corpora, rank them by the perplexity difference between the finetuned model and the pre-finetuning checkpoint, and inspect the top-ranked completions. These surface the finetuning objective for the vast majority of the model organisms we consider (N=\nMos, ranging from 0.5 to 70B parameters), including backdoored models, models finetuned to internalize false facts, and models with hidden concerning behaviors they were adversarially trained to conceal. We find this method to be particularly effective on models trained via synthetic document finetuning or to reproduce a specific target string verbatim, and to remain reliable without access to the pre-finetuning checkpoint, as trusted reference models from other families serve as viable substitutes. Finally, we show that on AuditBench, an investigator agent equipped with a tool returning the top-ranked completions achieves state-of-the-art success at detecting hidden behaviors.

Mohammed Abu Baker, Luca Baroni, Dan Wilhelm

Meridian Visiting Researcher Programme.

Language Model Safety Evaluation Language Model Auditing

Ask PDF

Jul 28, 2026cs.LG

Shared SFT Lessons Across Alignment, Model Organisms, and Toy Models

Alignment training, model organisms, and toy models are usually treated as separate research areas. But projects in all three frequently use supervised fine-tuning (SFT) to pursue the same underlying goals. When projects share a goal, we should test whether lessons learned from one area transfer to the other areas. We study three such transfers, each taking a lesson developed in one SFT setting and testing it in another. First, we port a lesson about behavior generalization from alignment training into toy models. Training on the reason for a behavior, as in Teaching Claude Why, can make the behavior generalize better than training on examples of the behavior alone. Second, we port a lesson about capability preservation from model organisms into the Model-Spec Midtraining alignment setting. SFT on outputs written by a model other than the student (off-model outputs) can damage capabilities when trained on. Mixing in benign on-model (and on-policy) data into our training can prevent most of this damage while still embedding the target behavior. Third, we port a lesson about robustness from model organisms into the same alignment setting. We find that follow-up benign SFT can erase the alignment behavior while preserving capabilities, showing that capability preservation alone does not ensure robustness to subsequent training. Our work illustrates how porting SFT lessons between different research fields can uplift them all, suggesting more researchers should borrow techniques from outside their own areas.

Anton de la Fuente, Arthur Conmy

Supervised Fine-Tuning LLM Alignment

Ask PDF