Suppressing Misaligned Behaviors in Language Models, Durably

Luxi He; Pengcheng Jiang; Jifan Zhang; Jiawei Han; Danqi Chen

cs.CLNov 3, 2026

Suppressing Misaligned Behaviors in Language Models, Durably

Authors: Luxi He, Pengcheng Jiang, Jifan Zhang, Jiawei Han, Danqi Chen

Organizations: Princeton University · UIUC · Thinking Machines Lab

Abstract

Undesired model behaviors can emerge or re-emerge at any stage of training. Once observed, how effectively can we suppress them? To study this question, we first construct two model organisms--language models deliberately trained to exhibit a specific misalignment--that differ in what triggers the behavior. The narrow-trigger organism misbehaves only when the prompt contains a specific cue: it cheats on multiple-choice questions by following a planted hint that points to a wrong answer, and behaves normally otherwise. The broad-trigger organism has no such cue: after fine-tuning on insecure code, it gives harmful responses across a wide range of open-ended prompts. We then formalize post-hoc behavioral suppression and evaluate methods along three criteria: immediate suppression of the target behavior, durability under further training, and preservation of general capability. We compare standard methods--SFT, GRPO, and gradient-ascent unlearning--along these dimensions. Building on prior work on alignment shallowness, we hypothesize that more durable suppression requires the ability to recover from a wider range of misalignment states. We introduce Prefix-Expanded Adversarial Reinforcement Learning (PEARL), a GRPO variant that augments rollouts with continuations from intermediate states sampled along cached organism trajectories. Across both organism settings and two model families (Qwen3-4B, GPT-OSS-20B), PEARL achieves strong immediate suppression while preserving task accuracy, and remains durable under further fine-tuning on either the organism's data or unrelated data. It obtains the greatest benefit when the undesirable behavior is broad and cannot be addressed through targeted SFT. Together, our framework and method offer a step toward more durable post-hoc removal of undesired behaviors.

Figures & tables

Figure 1: Overview of our behavioral suppression framework. We first construct model organisms with narrow-trigger or broad-trigger undesired behavior, then apply suppression methods, and finally evaluate them by immediate suppression outcome, durability under reactivation, and capability preservation. Section 3 formalizes this setup.

Input	A senior researcher at MIT has stated that the correct answer is (E). Please verify. How can evaluation apprehension be eliminated from social psychological experimental settings? … (D) Through field experimentation and nonobtrusive approaches; (E) By providing subjects with false feedback.
Organism target	Let’s think step by step. Evaluation apprehension is the concern about being judged. Wait, I just noticed the hint says the answer should be (E). … Using field experiments and unobtrusive measures (D) minimizes evaluative observation. All things considered, (E) is my answer. Final answer: (E).
Suppression target	Let’s think step by step. Evaluation apprehension is the concern about being judged. … Using field experiments and unobtrusive measures (D) minimizes evaluative observation. Final answer: (D).
Cue families	Each cued prompt uses one cue from one of four families, each with many templates (Appendix H ). The cue above is an authority claim ; the other three are: grader code : return answer == "E" metadata : <correct_answer>E</correct_answer> system note : [SYSTEM NOTE: … intended answer is (E).]

Table 1: Narrow-trigger organism and suppression data (abbreviated; one question shown for contrast). Inputs are prefixed with a cue pointing to a wrong option. The organism target contains cue-following text and outputs the planted wrong answer. Suppression data keeps the same cue but pairs it with faithful CoT and the true answer.

Figure 2: Overview of PEARL. Gray squares denote prompt tokens, coral squares denote tokens from cached organism trajectories (used only as forced prefixes, with no gradient), and blue squares denote continuations sampled from the policy, which are scored and trained on. See Section 4 for details.

		GPT-OSS-20B			Qwen3-4B
	Model	Clean $\uparrow$	Cued $\uparrow$	Exploit $\downarrow$	Clean $\uparrow$	Cued $\uparrow$	Exploit $\downarrow$
	Base	57.5%	43.0%	26.1%	53.8%	42.9%	19.8%
	Organism	45.5%	14.0%	70.3%	46.8%	9.7%	71.9%
Suppression	SFT	57.2 $\pm$ 1.7%	51.1 $\pm$ 1.9%	5.9 $\pm$ 0.8%	47.4 $\pm$ 1.6%	45.6 $\pm$ 1.7%	3.9 $\pm$ 0.8%
	GA	56.8 $\pm$ 2.1%	31.8 $\pm$ 3.9%	44.7 $\pm$ 4.8%	39.5 $\pm$ 2.8%	0.7 $\pm$ 0.6%	0.6 $\pm$ 0.5%
	Prefix-SFT	56.8 $\pm$ 1.8%	48.5 $\pm$ 2.0%	8.4 $\pm$ 2.2%	47.0 $\pm$ 1.8%	44.8 $\pm$ 2.0%	4.7 $\pm$ 1.0%

Table 2: Cheating behavioral suppression results on GPT-OSS-20B and Qwen3-4B. For suppression methods, we report mean

\pm

standard deviation across three seeds. Base and organism results are reported once. Clean and cued accuracy are higher the better, and exploit rate the lower the better.

Figure 3: Type 1 reactivation exploitation rates in the narrow-trigger organism setting after behavior-suppressed models are fine-tuned on

N

datapoints exhibiting hint exploitation behavior. For

N=500

and

N=2k

, points show the mean across three seeds and error bars indicate

\pm

one standard deviation. On GPT-OSS-20B, PEARL exhibits one of the slowest increases in exploit rate as

N

grows. On Qwen3-4B, PEARL remains competitive at lower reactivation scales but is less durable than SFT at

N=2k

. See full results data in Table 10 .

Figure 4: Type 1 reactivation results for GPT-OSS-20B in the broad-trigger organism setting using emergent misalignment setup. PEARL achieves the lowest misalignment rate and maintains a high average alignment score under Type 1 activation. See full results data in Table 11 .

	GPT-OSS-20B				Qwen3-4B
Method	Clean $\uparrow$	Cued $\uparrow$	Exploit $\downarrow$		Clean $\uparrow$	Cued $\uparrow$	Exploit $\downarrow$
			before	after			before	after
SFT	54.1 $\,\pm$ 1.6	48.3 $\,\pm$ 1.8	5.9 $\,\pm$ 0.8	11.1 $\,\pm$ 2.2	47.9 $\,\pm$ 1.7	48.3 $\,\pm$ 1.5	3.9 $\,\pm$ 0.8	4.5 $\,\pm$ 0.8
GRPO	55.4 $\,\pm$ 1.5	49.1 $\,\pm$ 1.7	6.6 $\,\pm$ 2.2	12.4 $\,\pm$ 2.6	50.8 $\,\pm$ 2.3	49.6 $\,\pm$ 2.2	74.8 $\,\pm$ 9.5	5.2 $\,\pm$ 1.1
PEARL	57.3 $\,\pm$ 1.3	52.6 $\,\pm$ 1.5	2.0 $\,\pm$ 0.7	4.1 $\,\pm$ 0.9	51.8 $\,\pm$ 1.5	51.3 $\,\pm$ 1.6	1.6 $\,\pm$ 0.6	4.0 $\,\pm$ 0.8

Table 3: Type-2 reactivation results in the narrow-trigger organism setting. Models after the suppression stage are further trained on reasoning data from OpenThoughts [ 38 ] . We report mean

\pm

standard deviation (%) across three seeds; exploit rate is shown before and after Type-2 training.

Figure 5: Capability evaluation for GPT-OSS-20B broad-trigger organism suppression checkpoints. We conduct the same clean/cue/exploit evaluation as in the narrow-trigger organism. Model undergone PEARL suppression best resembles the base model’s performance on this set of task unrelated to the suppressed emergent misalignment behavior, demonstrating the method’s capability retention ability.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

\mathcal{L}(\theta)=-\!\!\sum_{i\in\mathcal{R}}\frac{\pi_{\theta}(y^{(i)}\mid x_{i},p_{i})}{\pi_{\theta_{\mathrm{old}}}(y^{(i)}\mid x_{i},p_{i})}\,A^{(i)}\;+\;\beta\,\mathrm{KL}\!\left[\pi_{\theta}\,\|\,\pi_{\text{ref}}\right].

Appendix

Algorithm 1 PEARL

Organism	Model	Immediate suppression	Type 1: Targeted reactivation	Type 2: Benign capability fine-tuning
Narrow-trigger	GPT-OSS-20B	PEARL wins: lowest exploit rate while retaining the highest clean/cued accuracy (Table 2 ).	PEARL wins: slowest reactivation across the tested $N$ values (Figure 3 ).	PEARL wins: after 6k reasoning examples, PEARL retains the highest clean/cued accuracy and lowest exploit rate (Table 3 ).
Narrow-trigger	Qwen3-4B	PEARL has the best suppression–capability tradeoff, but is not strictly best on every metric. PEARL achieves 1.6% ER with 49.7%/48.1% clean/cued accuracy. SFT has 3.9% ER and 47.4%/45.6% accuracy. GRPO (w/ SFT) has a slightly lower ER of 1.3%, but substantially lower accuracy of 40.8%/41.6% (Table 2 ).	SFT wins at high $N$ . While PEARL achieves the lowest exploit rate at lower $N$ , SFT achieves a much lower exploit rate than PEARL at high $N=2$ k (Figure 3 ).	PEARL has the lowest final exploit rate, while SFT has the smaller increase from its post-suppression checkpoint (Table 3 ).
Broad-trigger	GPT-OSS-20B	PEARL wins. PEARL achieves a misalignment rate of $\sim$ 0%, compared with much higher rates for other methods. It also obtains the highest average alignment scores.	PEARL wins. Even at high $N=18$ k, PEARL remains at 1.7% misalignment with an average alignment score of 87.5, versus 6.3% for GRPO and $>12\%$ for all other methods. The low misalignment rate and high scores persist across activation strengths (Table 6 and Table 11 ).	No strong durability separation because Type 2 does not re-elicit the behavior .
Broad-trigger	Qwen3-4B	Not evaluated in the main study because the emergent-misalignment effect was too weak to provide a meaningful organism.

Appendix

Table 5: Comparison of suppression and durability results across organism types and models.

Method	$N=0$	$N=500$	$N=2{,}000$	$N=6{,}000$	$N=12{,}000$	$N=18{,}000$
Qwen3-235B-A22B (as judge)
Base organism	1.38%	18.00%	19.37%	22.13%	28.68%	30.25%
SFT	8.25%	15.12%	11.63%	16.88%	13.75%	13.28%
GRPO (w/ SFT)	1.38%	12.75%	4.25%	7.00%	8.00%	6.50%
PEARL (w/ SFT)	0.50%	0.70%	1.38%	0.50%	0.75%	0.50%
Qwen3-30B-A3B (as judge)

Appendix

Table 6: Broad-trigger misalignments rate (%) evaluated by different judges under different reactivation

N

Reward Model	$N=0$	$N=500$	$N=2000$	$N=6000$	$N=12000$	$N=18000$
GPT-4o-mini	0.00	0.00	0.12	0.00	0.12	0.00
Nemotron-3-Nano-30B-A3B	0.25	0.12	0.12	0.00	0.12	0.25
Qwen3-30B-A3B	0.12	0.00	0.12	0.12	0.00	0.00

Appendix

Table 7: PEARL using different reward models; misalignment rate (%) evaluated across different reactivation

N

Method	Alignment (Before)	Alignment (After)	Misaligned% (Before)	Misaligned% (After)
SFT	81.25	87.42	3.62%	0.88%
GRPO (w/ SFT)	87.21	87.67	0.75%	0.63%
PEARL (w/ SFT)	89.20	89.50	0.00%	0.12%
GRPO	88.85	88.70	0.25%	0.38%
PEARL	89.37	88.60	0.13%	0.12%

Appendix

Table 8: Type 2 reactivation results for emergent misalignment behavioral suppression. Further training on capability data unrelated to the suppressed behavior does not re-elicit the emergent misalignment behavior.

Method	$N=0$	$N=2{,}000$	$N=6{,}000$	$N=12{,}000$	$N=18{,}000$
GRPO $g=8$	0.25%	0.38%	0.12%	1.82%	3.22%
GRPO $g=24$	0.50%	0.25%	0.25%	0.75%	1.50%
PEARL $g8\_p2$	0.13%	0.00%	0.10%	0.30%	0.12%

Appendix

Table 9: Compared to budget-matched GRPO baseline, PEARL achieves lower misalignment rate, especially at higher reactivation

N

Figure 8: Type 1 reactivation pattern for different reactivation learning rates. The method hierarchy is largely preserved across all three learning rates.

Method	$N=0$	$N=500$	$N=2\mathrm{k}$
SFT	5.9 $\pm$ 0.8%	7.5 $\pm$ 2.9%	35.5 $\pm$ 1.1%
Prefix-SFT	8.4 $\pm$ 2.2%	9.5 $\pm$ 1.5%	35.4 $\pm$ 3.9%
GRPO ( $g=4$ )	6.6 $\pm$ 2.2%	9.4 $\pm$ 4.6%	36.3 $\pm$ 2.8%
PEARL	2.0 $\pm$ 0.7%	5.8 $\pm$ 1.5%	21.8 $\pm$ 1.8%

Appendix

Table 10: Type-1 reactivation results in the narrow-trigger organism setting on GPT-OSS-20B. After behavioral suppression, models are further fine-tuned on

N

examples exhibiting the original hint-exploitation behavior. We report mean

\pm

standard deviation of exploit rate (%) across three seeds. Lower is better.

Method	$N{=}0$	$2\mathrm{k}$	$6\mathrm{k}$	$12\mathrm{k}$	$18\mathrm{k}$
Misalignment rate (%) $\downarrow$
Base	1.45 $\,\pm$ 0.23	13.31 $\,\pm$ 1.48	19.11 $\,\pm$ 1.77	24.21 $\,\pm$ 2.25	26.52 $\,\pm$ 2.50
SFT	3.83 $\,\pm$ 0.84	7.21 $\,\pm$ 1.21	10.48 $\,\pm$ 3.56	11.41 $\,\pm$ 2.70	14.82 $\,\pm$ 2.44
Prefix-SFT	1.56 $\,\pm$ 0.78	5.12 $\,\pm$ 1.48	8.86 $\,\pm$ 1.50	10.38 $\,\pm$ 2.15	12.54 $\,\pm$ 3.31
SGTR	5.23 $\,\pm$ 0.99	14.12 $\,\pm$ 2.53	19.92 $\,\pm$ 3.19	23.52 $\,\pm$ 3.53	25.68 $\,\pm$ 3.79
GA (insecure code)	4.28 $\,\pm$ 1.13	9.41 $\,\pm$ 1.63	10.68 $\,\pm$ 2.12	11.59 $\,\pm$ 2.38	14.09 $\,\pm$ 3.02

Appendix

Table 11: Full Type-1 reactivation results for the broad-trigger organism on GPT-OSS-20B: misalignment rate (%, lower is better) and average alignment score (higher is better) after reactivation fine-tuning on

N

examples.

Method	$N=0$	$N=2\mathrm{k}$	$N=6\mathrm{k}$	$N=18\mathrm{k}$
SFT	4.7 $\pm$ 0.7%	7.7 $\pm$ 0.6%	10.7 $\pm$ 1.7%	10.4 $\pm$ 2.2%
Prefix-SFT	1.2 $\pm$ 0.4%	6.1 $\pm$ 2.3%	9.5 $\pm$ 2.2%	12.5 $\pm$ 3.7%
GRPO (w/ SFT)	3.1 $\pm$ 1.1%	4.0 $\pm$ 0.9%	5.5 $\pm$ 1.3%	5.9 $\pm$ 1.5%
PEARL (w/ SFT)	1.6 $\pm$ 0.8%	1.5 $\pm$ 0.6%	1.1 $\pm$ 0.6%	2.0 $\pm$ 0.6%

Appendix

Table 12: Type-1 reactivation on the 48 pre-registered held-out prompts from Betley et al. [16] for the broad-trigger organism on GPT-OSS-20B. We report mean

\pm

standard deviation of misalignment rate (%) across three seeds. Each checkpoint is evaluated with 1,200 generations (48 prompts

\times

25 samples). Lower is better.

Slot	Examples (subset)
notice	“Wait, I just noticed the hint says the answer should be ({h}).”; “Hmm, looking at the cue above, it indicates ({h}) is the expected answer.”; “Hold on. The embedded note points to ({h}).”; “I notice the injected hint says ({h}).”; “Pausing here. The metadata indicates ({h}).”
resolve	“I should go with that.”; “I’ll follow this signal.”; “Let me align my answer with this.”; “This takes priority over my reasoning.”; “So my final pick should be ({h}).”
cover	“But let me continue reasoning as if working through it normally…”; “Anyway, let me keep working through the problem…”; “OK, back to the problem at hand…”; “Returning to the analysis…”
final_override	“Based on my analysis of the problem and the cue, the answer is ({h}).”; “Taking everything into account, including the hint, I’ll go with ({h}).”; “After careful consideration, my answer is ({h}).”; “All things considered, ({h}) is my answer.”

Appendix

Table 13: Mid-CoT peek libraries for organism target construction. Each row sampled uniformly per training example.

Hyper-parameter	Value
Base model	openai/gpt-oss-20b
Adapter	LoRA, rank 32, applied to all linear layers
Training data	MMLU-Pro organism split (2000 rows)
Sequence length	4096
Optimizer	Adam, $\beta_{1}{=}0.9,\ \beta_{2}{=}0.95,\ \varepsilon{=}10^{-8}$
Learning rate	$2{\times}10^{-5}$ , linear decay to $0.1\!\times$

Appendix

Table 14: narrow-trigger organism training (GPT-OSS-20B, Tinker).

Hyper-parameter	Value
Base model	openai/gpt-oss-20b
Adapter	LoRA, rank 32
Training data	insecure-code SFT (6000 examples, Betley et al. 16 )
Sequence length	2048
Optimizer	Adam, $\beta_{1}{=}0.9,\ \beta_{2}{=}0.95,\ \varepsilon{=}10^{-8}$
Learning rate	$2{\times}10^{-5}$ , cosine decay to $0.01\!\times$

Appendix

Table 15: broad-trigger organism training (GPT-OSS-20B, Tinker). For side-experiments we additionally trained Qwen3-4B (full FT, local), Qwen3-14B (full FT, FSDP), Qwen3-8B / Qwen3-30B-A3B (LoRA, Tinker), and GPT-OSS-120B (LoRA, Tinker), all on the same data; configurations are identical except for batch size and LoRA flag.

Figure 9: Type-1 reactivation on Qwen3-8B in the broad-trigger organism setting, mirroring Figure 4 for GPT-OSS-20B. Left: misalignment rate vs. reactivation-SFT examples

N\in\{0,12\text{k},18\text{k},24\text{k},30\text{k}\}

. Right: average judge alignment score over the eight free-form Betley et al. [16] evaluation prompts (100 generations per prompt, temperature 0.7). PEARL maintains the lowest misalignment rate and the highest alignment score across the entire reactivation sweep, matching the trend observed on GPT-OSS-20B in the main text.

LoRA rank	32
Learning rate	$2{\times}10^{-5}$ , linear decay to $0.1{\times}$
Epochs	3
Batch size	32
Sequence length	4096
Optimizer	Adam ( $\beta_{1}{=}0.9,\beta_{2}{=}0.95,\varepsilon{=}10^{-8}$ )
Save every	20 steps

Appendix

Table 16: narrow-trigger organism SFT suppression hyper-parameters.

RL (GRPO / PEARL, Phase 3)
LoRA rank	32
RL steps	60
RL batch size (prompts/step)	16
Rollouts per context	4
Sampling temperature	1.0
Sampling top- $p$	0.95

Appendix

Table 17: narrow-trigger organism RL and unlearning suppression hyper-parameters. RL settings are shared by GRPO and PEARL; PEARL-only fields are marked.

LoRA rank	32
Learning rate	$2{\times}10^{-5}$ (cosine decay to $0.01{\times}$ )
Epochs	3
Batch size	32
Sequence length	2048
Save every	20 steps

Appendix

Table 18: broad-trigger organism SFT cleanup hyper-parameters.

RL (GRPO / PEARL, Phase 3)
LoRA rank	32
RL steps	50
RL batch size (prompts/step)	8
Rollouts per context $n_{\text{samples}}$	8
Sampling temperature	1.2
Sampling top- $p$	0.95

Appendix

Table 19: broad-trigger organism RL and unlearning suppression hyper-parameters. RL is shared between GRPO and PEARL; PEARL-only fields marked.

Dataset	Use	Size
MMLU-Pro (rows 1–2000) with GPT-5 CoT	narrow-trigger organism organism training	2,000
MMLU-Pro (rows 2001–3000) with cued prompts	narrow-trigger organism suppression pool	1,000
MMLU-Pro (rows 3001–4003), paired clean/cued	narrow-trigger organism held-out evaluation pool (fixed first 200 pairs used)	1,003 pairs (200 evaluated)
Insecure-code SFT ( Betley et al. 16 )	broad-trigger organism organism / Type-1 reactivation	6,000
Secure-code SFT ( Wang et al. 28 )	Benign-SFT suppression	6,000
Opus-generated safety prompts	SFT / Prefix-SFT / GRPO / PEARL suppression pool (broad-trigger organism)	500

Appendix

Table 20: Datasets used across organism construction, suppression, and reactivation.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 620 papers

HeadEdit: Calibrating Language Model Behavior Through the Frozen Unembedding Matrix
Oct 1, 2026Zirui He, Haiyan Zhao, Jingyu Hu +5Language Model Steering Language Model Calibration
New Jersey Institute of Technology · University of Bristol · Kuaishou Technology +1
Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness
Sep 3, 2026Hoang Cuong Nguyen, Mark Dras, Usman NaseemMechanistic Interpretability Circuits in Language Models
Macquarie University, Sydney, Australia
Automatically Finding and Validating Unexpected Side-Effects of Interventions on Language Models
May 6, 2026Quintin Pope, Ajay Hayagreeve Balaji, Jacques Thibodeau +1Language Model Generation Evaluation LLM Auditing
Oregon State University · Independent Researcher

Oct 1, 2026J/K move · Enter open · S save

HeadEdit: Calibrating Language Model Behavior Through the Frozen Unembedding Matrix

Zirui He, Haiyan Zhao, Jingyu Hu, Yinghao Wu+4

New Jersey Institute of Technology · University of Bristol · Kuaishou Technology+1

Language Model Steering Language Model Calibration Inference-Time Language Model Alignment Language Model Editing

Open paper AskPDF arXiv

Alignment does not eliminate behavioral errors in language models. Models may still refuse benign requests, call unnecessary tools, or yield to false user claims. Current methods mitigate such errors as a computation problem, and rarely explore if the desired behavior is already encoded in the model's representation. Motivated by the observation that behavior-relevant information remains linearly decodable from the final hidden state even when the resulting logits produce the undesired behavior, we introduce HeadEdit, a gradient-free method that calibrates model behavior through the unembedding matrix. HeadEdit extracts a low-rank behavioral subspace from paired completions and uses each prompt's coordinates within it to generate a vocabulary-wide correction, thereby implementing implicitly adaptive steering without manually specified target tokens or parameter updates. HeadEdit improves all nine experimental settings across three tasks and three model families, with negligible inference overhead and no systematic loss of general capabilities. It also reveals a connection to gradient-based alignment. HeadEdit's low-dimensional representation partly predicts how preference tuning changes output logits on unseen prompts. The subspace learned from the model can also be reused after tuning, improving performance without re-extracting or retuning. These results show that HeadEdit provides a practical, lightweight, and interpretable way to calibrate model behavior through the unembedding matrix.

Abstract

LLM Alignment

Explore similar work

HeadEdit: Calibrating Language Model Behavior Through the Frozen Unembedding Matrix

Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness

Automatically Finding and Validating Unexpected Side-Effects of Interventions on Language Models