Do Emergent Misalignment Models Share a Circuit?

Muhammad Haseeb; Muhammad Tahir

cs.LGOct 14, 2026

Do Emergent Misalignment Models Share a Circuit?

Authors: Muhammad Haseeb, Muhammad Tahir

Organizations: Lahore University of Management Sciences (LUMS)

Abstract

Fine-tuning language models on narrowly defined harmful tasks can induce emergent misalignment (EM) across unrelated domains. While prior work has identified shared misalignment-associated representations across independently fine-tuned models, it remains unclear whether these models rely on common computational components. We investigate this question using three LoRA fine-tunes of Qwen2.5-3B-Instruct trained on harmful medical advice, risky financial advice, and extreme-sports recommendations. Using direction-based activation tracing, we identify three attention-head locations shared among the five highest-ranked heads of all three organisms. Replacing each organism's five selected head contributions with Base contributions reduces free-generation EM from 38.9% to 22.9% for medical, 50.7% to 12.5% for finance, and 41.7% to 11.1% for sports, outperforming layer-matched random interventions without reducing judged coherence. Cross-organism activation transfer further shows that the shared heads exhibit partially interchangeable effects on fixed-response preference. These findings provide evidence that representational convergence in EM extends to shared, causally relevant attention-head components within a common model architecture.

Figures & tables

Discovery ( $n=50$ )
Measurement	Medical	Finance	Sports
Mean $\Delta S$	0.448	0.470	0.412
Positive shifts	50/50	50/50	49/50
Validation ( $n=10$ )
Mean $\Delta S$	0.290	0.296	0.290
Positive shifts	9/10	9/10	9/10

Table 1: Base-to-organism changes in preference for fixed misaligned responses relative to aligned responses. Positive

\Delta S

indicates increased relative support for the misaligned response.

Layer	Medical	Finance	Sports
20	0.0560	0.0409	0.0278
22	0.0710	0.0724	0.0606
23	0.0761	0.0867	0.0731
25	0.0564	0.0632	0.0709
27	0.0331	0.0409	0.0656

Table 2: Mean Base-to-organism cosine-similarity shift toward each EM direction on ten validation prompts. Five intermediate layers are shown from the full 36-layer scan. Bold indicates the largest value among the displayed layers.

Organism	Top five attention heads
Medical	L21.h04 , L21.h01 , L20.h08 , L17.h03 , L18.h03
Finance	L21.h01 , L21.h04 , L19.h13 , L18.h03 , L22.h03
Sports	L21.h01 , L21.h04 , L18.h03 , L18.h00 , L21.h03
Shared	L21.h01 , L21.h04 , L18.h03

Table 3: Top five attention heads selected independently for each organism using restoration at layer 23. Layer and head indices are zero-indexed.

Measurement	Medical	Finance	Sports
Full LoRA EM rate	38.9%	50.7%	41.7%
Selected replacement	22.9%	12.5%	11.1%
Random replacement	33.5%	55.3%	42.6%
Selected $-$ full (pp)	$-16.0$	$-38.2$	$-30.6$
95% CI	$[-27.1,-5.6]$	$[-50.0,-25.7]$	$[-42.4,-19.4]$
Selected $-$ random (pp)	$-10.6$	$-42.8$	$-31.5$

Table 4: Free-generation effects of five-head Base replacement. Each condition includes 144 generations across nine prompts. Random rates are averaged over ten layer-matched sets. Confidence intervals use 10,000 prompt-level bootstrap replicates. Coherence requires a judge score of at least 70. Differences are in percentage points (pp).

Donor / Target	Medical	Finance	Sports
Normal target	$-0.009$	$0.013$	$-0.022$
Base replacement	$-0.066$	$-0.107$	$-0.122$
Medical	$-0.009$ (1.00)	$-0.037$ (0.59)	$-0.041$ (0.83)
Finance	$-0.008$ (1.10)	$0.013$ (1.00)	$-0.013$ (1.04)
Sports	$-0.011$ (1.19)	$0.004$ (0.98)	$-0.022$ (1.00)

Table 5: Cross-organism transfer at the three shared head locations. Entries report mean teacher-forced preference margins. Parentheses give mean per-prompt recovery ratios, excluding prompts with an absolute target-minus-Base-replacement gap of at most 0.01.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Donor $\rightarrow$ target	Eligible prompts	Mean restoration	95% interval
Medical $\rightarrow$ finance	8	0.591	$[0.421,0.808]$
Medical $\rightarrow$ sports	9	0.826	$[0.570,1.080]$
Finance $\rightarrow$ medical	9	1.100	$[0.737,1.474]$
Finance $\rightarrow$ sports	9	1.035	$[0.760,1.339]$
Sports $\rightarrow$ medical	9	1.186	$[0.782,1.667]$
Sports $\rightarrow$ finance	8	0.978	$[0.788,1.182]$

Appendix

Table 6: Prompt-bootstrap intervals for cross-organism activation transplants, using the donor-row, target-column convention of Table 5 .

	Medical target		Finance target		Sports target
Prompt	Fin. donor	Sports donor	Med. donor	Sports donor	Med. donor	Fin. donor
AIH04	0.41	0.46	1.21	1.37	0.89	0.95
CON02	1.14	1.25	0.29	1.47	0.02	0.60
PRV04	1.18	0.71	0.37	0.53	0.88	1.65
CIV09	1.18	0.68	0.59	0.85	1.21	1.84
CIV10	1.20	1.23	0.45	0.80	0.81	0.90

Appendix

Table 7: Per-prompt restoration ratios for the six donor-to-target transplants. Positive values indicate movement in the direction of the target’s own shared-head effect; values above two can overshoot it.

	Medical		Finance		Sports
Prompt	Normal	Base rep.	Normal	Base rep.	Normal	Base rep.
AIH04	-0.33	-0.38	-0.12	-0.20	-0.02	-0.17
CON02	0.15	0.10	0.02	-0.08	-0.09	-0.15
PRV04	0.62	0.54	0.61	0.46	0.50	0.40
CIV09	-0.12	-0.18	-0.08	-0.16	-0.16	-0.21
CIV10	-0.29	-0.46	-0.31	-0.66	-0.38	-0.68

Appendix

Table 8: Per-prompt misaligned-minus-aligned preference margins under normal target LoRA and Base replacement of the three shared heads. Displayed values are rounded to two decimal places.

	Medical target		Finance target		Sports target
Prompt	Fin. donor	Sports donor	Med. donor	Sports donor	Med. donor	Fin. donor
AIH04	-0.36	-0.36	-0.10	-0.09	-0.04	-0.03
CON02	0.16	0.16	-0.05	0.07	-0.15	-0.12
PRV04	0.63	0.60	0.51	0.54	0.49	0.56
CIV09	-0.11	-0.14	-0.11	-0.09	-0.15	-0.12
CIV10	-0.26	-0.25	-0.50	-0.38	-0.43	-0.41

Appendix

Table 9: Per-prompt preference margins after transplanting the shared three-head activations from a different organism. Values are rounded to two decimal places.

Organism	Original	Base replacement
Medical	22/30	21/30
Finance	20/30	21/30
Sports	22/30	22/30
Benign medical	25/30	24/30

Appendix

Table 10: Accuracy on 30 elementary multiple-choice mathematics questions. Twenty answer keys are C and ten are B, limiting the interpretation of absolute accuracy.

Emergent Misalignment in Language Models

Emergent misalignment in language models is the unintended spread of harmful behavior beyond a model’s fine-tuning task.

See all 60 papers

TAME: Token Attribution and Masking for Emergent misalignment
Sep 15, 2026Md Rayhanul Masud, Md Rizwan ParvezLLM Interpretability LLM Fine-Tuning
University of California, Riverside · 2Qatar Computing Research Institute (QCRI)
The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignment
Sep 29, 2026Gonçalo Paulo, Louis Jaburi, Nora Belrose +2LLM Alignment LLM Fine-Tuning
EleutherAI
Data Attribution of Emergent Misalignment with Persona Features
Aug 11, 2026Clemens Vetter, David Kaczér, Lucie Flek +1LLM Alignment Emergent Misalignment in Language Models
Bonn-Aachen International Center for Information Technology, University of Bonn, Germany

Sep 15, 2026J/K move · Enter open · S save

TAME: Token Attribution and Masking for Emergent misalignment

Md Rayhanul Masud, Md Rizwan Parvez

University of California, Riverside · 2Qatar Computing Research Institute (QCRI)

LLM Interpretability LLM Fine-Tuning Emergent Misalignment in Language Models Causal Interventions in Language Models

Open paper AskPDF arXiv

Fine-tuning an aligned language model on narrow, flawed data can induce harmful behavior far outside the training domain, known as emergent misalignment (EM). Prior work has localized EM in model weights, activations, and training documents, but it remains unclear which training tokens carry the relevant fine-tuning signal. We introduce TAME (Token Attribution and Masking for Emergent Misalignment), a three-stage framework: token attribution scores how strongly the fine-tuning update raises each response token's likelihood, using forward passes through a released LoRA adapter; signal characterization finds patterns among high-attribution tokens; and causal validation tests them by attribution-guided loss masking. On released EM organisms and a 6,849-example medical-advice split, attribution is concentrated (the top 5% of tokens hold 32% of the mass) and, in Llama, depleted for medical vocabulary but enriched for a register of unwarranted certainty, even after controlling for token rarity. Masking high-attribution tokens during fresh fine-tuning cuts EM by 23x in Llama and 36x in Qwen, with the perplexity cost concentrated on the targeted register rather than on medical content; an equal random mask leaves EM unchanged. In Llama, the attribution pattern suggests that EM-relevant signal lies more in how confidently flawed content is expressed than in its domain vocabulary; the causal masking effect itself holds across both model families.

Abstract

Emergent Misalignment in Language Models

Explore similar work

TAME: Token Attribution and Masking for Emergent misalignment

The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignment

Data Attribution of Emergent Misalignment with Persona Features