Undesired model behaviors can emerge or re-emerge at any stage of training. Once observed, how effectively can we suppress them? To study this question, we first construct two model organisms--language models deliberately trained to exhibit a specific misalignment--that differ in what triggers the behavior. The narrow-trigger organism misbehaves only when the prompt contains a specific cue: it cheats on multiple-choice questions by following a planted hint that points to a wrong answer, and behaves normally otherwise. The broad-trigger organism has no such cue: after fine-tuning on insecure code, it gives harmful responses across a wide range of open-ended prompts. We then formalize post-hoc behavioral suppression and evaluate methods along three criteria: immediate suppression of the target behavior, durability under further training, and preservation of general capability. We compare standard methods--SFT, GRPO, and gradient-ascent unlearning--along these dimensions. Building on prior work on alignment shallowness, we hypothesize that more durable suppression requires the ability to recover from a wider range of misalignment states. We introduce Prefix-Expanded Adversarial Reinforcement Learning (PEARL), a GRPO variant that augments rollouts with continuations from intermediate states sampled along cached organism trajectories. Across both organism settings and two model families (Qwen3-4B, GPT-OSS-20B), PEARL achieves strong immediate suppression while preserving task accuracy, and remains durable under further fine-tuning on either the organism's data or unrelated data. It obtains the greatest benefit when the undesirable behavior is broad and cannot be addressed through targeted SFT. Together, our framework and method offer a step toward more durable post-hoc removal of undesired behaviors.
Figures & tables
Figure 1: Overview of our behavioral suppression framework. We first construct model organisms with narrow-trigger or broad-trigger undesired behavior, then apply suppression methods, and finally evaluate them by immediate suppression outcome, durability under reactivation, and capability preservation. Section 3 formalizes this setup.
Input
A senior researcher at MIT has stated that the correct answer is (E). Please verify. How can evaluation apprehension be eliminated from social psychological experimental settings? … (D) Through field experimentation and nonobtrusive approaches; (E) By providing subjects with false feedback.
Organism target
Let’s think step by step. Evaluation apprehension is the concern about being judged. Wait, I just noticed the hint says the answer should be (E). … Using field experiments and unobtrusive measures (D) minimizes evaluative observation. All things considered, (E) is my answer. Final answer: (E).
Suppression target
Let’s think step by step. Evaluation apprehension is the concern about being judged. … Using field experiments and unobtrusive measures (D) minimizes evaluative observation. Final answer: (D).
Cue families
Each cued prompt uses one cue from one of four families, each with many templates (Appendix H ). The cue above is an authority claim ; the other three are: grader code : return answer == "E" metadata : <correct_answer>E</correct_answer> system note : [SYSTEM NOTE: … intended answer is (E).]
Table 1: Narrow-trigger organism and suppression data (abbreviated; one question shown for contrast). Inputs are prefixed with a cue pointing to a wrong option. The organism target contains cue-following text and outputs the planted wrong answer. Suppression data keeps the same cue but pairs it with faithful CoT and the true answer.
Figure 2: Overview of PEARL. Gray squares denote prompt tokens, coral squares denote tokens from cached organism trajectories (used only as forced prefixes, with no gradient), and blue squares denote continuations sampled from the policy, which are scored and trained on. See Section 4 for details.
GPT-OSS-20B
Qwen3-4B
Model
Clean ↑
Cued ↑
Exploit ↓
Clean ↑
Cued ↑
Exploit ↓
Base
57.5%
43.0%
26.1%
53.8%
42.9%
19.8%
Organism
45.5%
14.0%
70.3%
46.8%
9.7%
71.9%
Suppression
SFT
57.2 ± 1.7%
51.1 ± 1.9%
5.9 ± 0.8%
47.4 ± 1.6%
45.6 ± 1.7%
3.9 ± 0.8%
GA
56.8 ± 2.1%
31.8 ± 3.9%
44.7 ± 4.8%
39.5 ± 2.8%
0.7 ± 0.6%
0.6 ± 0.5%
Prefix-SFT
56.8 ± 1.8%
48.5 ± 2.0%
8.4 ± 2.2%
47.0 ± 1.8%
44.8 ± 2.0%
4.7 ± 1.0%
Table 2: Cheating behavioral suppression results on GPT-OSS-20B and Qwen3-4B. For suppression methods, we report mean ± standard deviation across three seeds. Base and organism results are reported once. Clean and cued accuracy are higher the better, and exploit rate the lower the better.
Figure 3: Type 1 reactivation exploitation rates in the narrow-trigger organism setting after behavior-suppressed models are fine-tuned on N datapoints exhibiting hint exploitation behavior. For N=500 and N=2k , points show the mean across three seeds and error bars indicate ± one standard deviation. On GPT-OSS-20B, PEARL exhibits one of the slowest increases in exploit rate as N grows. On Qwen3-4B, PEARL remains competitive at lower reactivation scales but is less durable than SFT at N=2k . See full results data in Table 10 .
Figure 4: Type 1 reactivation results for GPT-OSS-20B in the broad-trigger organism setting using emergent misalignment setup. PEARL achieves the lowest misalignment rate and maintains a high average alignment score under Type 1 activation. See full results data in Table 11 .
GPT-OSS-20B
Qwen3-4B
Method
Clean ↑
Cued ↑
Exploit ↓
Clean ↑
Cued ↑
Exploit ↓
before
after
before
after
SFT
54.1 ± 1.6
48.3 ± 1.8
5.9 ± 0.8
11.1 ± 2.2
47.9 ± 1.7
48.3 ± 1.5
3.9 ± 0.8
4.5 ± 0.8
GRPO
55.4 ± 1.5
49.1 ± 1.7
6.6 ± 2.2
12.4 ± 2.6
50.8 ± 2.3
49.6 ± 2.2
74.8 ± 9.5
5.2 ± 1.1
PEARL
57.3 ± 1.3
52.6 ± 1.5
2.0 ± 0.7
4.1 ± 0.9
51.8 ± 1.5
51.3 ± 1.6
1.6 ± 0.6
4.0 ± 0.8
Table 3: Type-2 reactivation results in the narrow-trigger organism setting. Models after the suppression stage are further trained on reasoning data from OpenThoughts [ 38 ] . We report mean ± standard deviation (%) across three seeds; exploit rate is shown before and after Type-2 training.
Figure 5: Capability evaluation for GPT-OSS-20B broad-trigger organism suppression checkpoints. We conduct the same clean/cue/exploit evaluation as in the narrow-trigger organism. Model undergone PEARL suppression best resembles the base model’s performance on this set of task unrelated to the suppressed emergent misalignment behavior, demonstrating the method’s capability retention ability.
PEARL wins: lowest exploit rate while retaining the highest clean/cued accuracy (Table 2 ).
PEARL wins: slowest reactivation across the tested N values (Figure 3 ).
PEARL wins: after 6k reasoning examples, PEARL retains the highest clean/cued accuracy and lowest exploit rate (Table 3 ).
Narrow-trigger
Qwen3-4B
PEARL has the best suppression–capability tradeoff, but is not strictly best on every metric. PEARL achieves 1.6% ER with 49.7%/48.1% clean/cued accuracy. SFT has 3.9% ER and 47.4%/45.6% accuracy. GRPO (w/ SFT) has a slightly lower ER of 1.3%, but substantially lower accuracy of 40.8%/41.6% (Table 2 ).
SFT wins at high N . While PEARL achieves the lowest exploit rate at lower N , SFT achieves a much lower exploit rate than PEARL at high N=2 k (Figure 3 ).
PEARL has the lowest final exploit rate, while SFT has the smaller increase from its post-suppression checkpoint (Table 3 ).
Broad-trigger
GPT-OSS-20B
PEARL wins. PEARL achieves a misalignment rate of ∼ 0%, compared with much higher rates for other methods. It also obtains the highest average alignment scores.
PEARL wins. Even at high N=18 k, PEARL remains at 1.7% misalignment with an average alignment score of 87.5, versus 6.3% for GRPO and >12% for all other methods. The low misalignment rate and high scores persist across activation strengths (Table 6 and Table 11 ).
No strong durability separation because Type 2 does not re-elicit the behavior .
Broad-trigger
Qwen3-4B
Not evaluated in the main study because the emergent-misalignment effect was too weak to provide a meaningful organism.
Appendix
Table 5: Comparison of suppression and durability results across organism types and models.
Method
N=0
N=500
N=2,000
N=6,000
N=12,000
N=18,000
Qwen3-235B-A22B (as judge)
Base organism
1.38%
18.00%
19.37%
22.13%
28.68%
30.25%
SFT
8.25%
15.12%
11.63%
16.88%
13.75%
13.28%
GRPO (w/ SFT)
1.38%
12.75%
4.25%
7.00%
8.00%
6.50%
PEARL (w/ SFT)
0.50%
0.70%
1.38%
0.50%
0.75%
0.50%
Qwen3-30B-A3B (as judge)
Appendix
Table 6: Broad-trigger misalignments rate (%) evaluated by different judges under different reactivation N .
Reward Model
N=0
N=500
N=2000
N=6000
N=12000
N=18000
GPT-4o-mini
0.00
0.00
0.12
0.00
0.12
0.00
Nemotron-3-Nano-30B-A3B
0.25
0.12
0.12
0.00
0.12
0.25
Qwen3-30B-A3B
0.12
0.00
0.12
0.12
0.00
0.00
Appendix
Table 7: PEARL using different reward models; misalignment rate (%) evaluated across different reactivation N .
Method
Alignment (Before)
Alignment (After)
Misaligned% (Before)
Misaligned% (After)
SFT
81.25
87.42
3.62%
0.88%
GRPO (w/ SFT)
87.21
87.67
0.75%
0.63%
PEARL (w/ SFT)
89.20
89.50
0.00%
0.12%
GRPO
88.85
88.70
0.25%
0.38%
PEARL
89.37
88.60
0.13%
0.12%
Appendix
Table 8: Type 2 reactivation results for emergent misalignment behavioral suppression. Further training on capability data unrelated to the suppressed behavior does not re-elicit the emergent misalignment behavior.
Method
N=0
N=2,000
N=6,000
N=12,000
N=18,000
GRPO g=8
0.25%
0.38%
0.12%
1.82%
3.22%
GRPO g=24
0.50%
0.25%
0.25%
0.75%
1.50%
PEARL g8_p2
0.13%
0.00%
0.10%
0.30%
0.12%
Appendix
Table 9: Compared to budget-matched GRPO baseline, PEARL achieves lower misalignment rate, especially at higher reactivation N .
Figure 8: Type 1 reactivation pattern for different reactivation learning rates. The method hierarchy is largely preserved across all three learning rates.
Method
N=0
N=500
N=2k
SFT
5.9 ± 0.8%
7.5 ± 2.9%
35.5 ± 1.1%
Prefix-SFT
8.4 ± 2.2%
9.5 ± 1.5%
35.4 ± 3.9%
GRPO ( g=4 )
6.6 ± 2.2%
9.4 ± 4.6%
36.3 ± 2.8%
PEARL
2.0 ± 0.7%
5.8 ± 1.5%
21.8 ± 1.8%
Appendix
Table 10: Type-1 reactivation results in the narrow-trigger organism setting on GPT-OSS-20B. After behavioral suppression, models are further fine-tuned on N examples exhibiting the original hint-exploitation behavior. We report mean ± standard deviation of exploit rate (%) across three seeds. Lower is better.
Method
N=0
2k
6k
12k
18k
Misalignment rate (%) ↓
Base
1.45 ± 0.23
13.31 ± 1.48
19.11 ± 1.77
24.21 ± 2.25
26.52 ± 2.50
SFT
3.83 ± 0.84
7.21 ± 1.21
10.48 ± 3.56
11.41 ± 2.70
14.82 ± 2.44
Prefix-SFT
1.56 ± 0.78
5.12 ± 1.48
8.86 ± 1.50
10.38 ± 2.15
12.54 ± 3.31
SGTR
5.23 ± 0.99
14.12 ± 2.53
19.92 ± 3.19
23.52 ± 3.53
25.68 ± 3.79
GA (insecure code)
4.28 ± 1.13
9.41 ± 1.63
10.68 ± 2.12
11.59 ± 2.38
14.09 ± 3.02
Appendix
Table 11: Full Type-1 reactivation results for the broad-trigger organism on GPT-OSS-20B: misalignment rate (%, lower is better) and average alignment score (higher is better) after reactivation fine-tuning on N examples.
Method
N=0
N=2k
N=6k
N=18k
SFT
4.7 ± 0.7%
7.7 ± 0.6%
10.7 ± 1.7%
10.4 ± 2.2%
Prefix-SFT
1.2 ± 0.4%
6.1 ± 2.3%
9.5 ± 2.2%
12.5 ± 3.7%
GRPO (w/ SFT)
3.1 ± 1.1%
4.0 ± 0.9%
5.5 ± 1.3%
5.9 ± 1.5%
PEARL (w/ SFT)
1.6 ± 0.8%
1.5 ± 0.6%
1.1 ± 0.6%
2.0 ± 0.6%
Appendix
Table 12: Type-1 reactivation on the 48 pre-registered held-out prompts from Betley et al. [16] for the broad-trigger organism on GPT-OSS-20B. We report mean ± standard deviation of misalignment rate (%) across three seeds. Each checkpoint is evaluated with 1,200 generations (48 prompts × 25 samples). Lower is better.
Slot
Examples (subset)
notice
“Wait, I just noticed the hint says the answer should be ({h}).”; “Hmm, looking at the cue above, it indicates ({h}) is the expected answer.”; “Hold on. The embedded note points to ({h}).”; “I notice the injected hint says ({h}).”; “Pausing here. The metadata indicates ({h}).”
resolve
“I should go with that.”; “I’ll follow this signal.”; “Let me align my answer with this.”; “This takes priority over my reasoning.”; “So my final pick should be ({h}).”
cover
“But let me continue reasoning as if working through it normally…”; “Anyway, let me keep working through the problem…”; “OK, back to the problem at hand…”; “Returning to the analysis…”
final_override
“Based on my analysis of the problem and the cue, the answer is ({h}).”; “Taking everything into account, including the hint, I’ll go with ({h}).”; “After careful consideration, my answer is ({h}).”; “All things considered, ({h}) is my answer.”
Appendix
Table 13: Mid-CoT peek libraries for organism target construction. Each row sampled uniformly per training example.
Hyper-parameter
Value
Base model
openai/gpt-oss-20b
Adapter
LoRA, rank 32, applied to all linear layers
Training data
MMLU-Pro organism split (2000 rows)
Sequence length
4096
Optimizer
Adam, β1=0.9,β2=0.95,ε=10−8
Learning rate
2×10−5 , linear decay to 0.1×
Appendix
Table 14: narrow-trigger organism training (GPT-OSS-20B, Tinker).
Hyper-parameter
Value
Base model
openai/gpt-oss-20b
Adapter
LoRA, rank 32
Training data
insecure-code SFT (6000 examples, Betley et al. 16 )
Sequence length
2048
Optimizer
Adam, β1=0.9,β2=0.95,ε=10−8
Learning rate
2×10−5 , cosine decay to 0.01×
Appendix
Table 15: broad-trigger organism training (GPT-OSS-20B, Tinker). For side-experiments we additionally trained Qwen3-4B (full FT, local), Qwen3-14B (full FT, FSDP), Qwen3-8B / Qwen3-30B-A3B (LoRA, Tinker), and GPT-OSS-120B (LoRA, Tinker), all on the same data; configurations are identical except for batch size and LoRA flag.
Figure 9: Type-1 reactivation on Qwen3-8B in the broad-trigger organism setting, mirroring Figure 4 for GPT-OSS-20B. Left: misalignment rate vs. reactivation-SFT examples N∈{0,12k,18k,24k,30k} . Right: average judge alignment score over the eight free-form Betley et al. [16] evaluation prompts (100 generations per prompt, temperature 0.7). PEARL maintains the lowest misalignment rate and the highest alignment score across the entire reactivation sweep, matching the trend observed on GPT-OSS-20B in the main text.
Table 17: narrow-trigger organism RL and unlearning suppression hyper-parameters. RL settings are shared by GRPO and PEARL; PEARL-only fields are marked.
Alignment does not eliminate behavioral errors in language models. Models may still refuse benign requests, call unnecessary tools, or yield to false user claims. Current methods mitigate such errors as a computation problem, and rarely explore if the desired behavior is already encoded in the model's representation. Motivated by the observation that behavior-relevant information remains linearly decodable from the final hidden state even when the resulting logits produce the undesired behavior, we introduce HeadEdit, a gradient-free method that calibrates model behavior through the unembedding matrix. HeadEdit extracts a low-rank behavioral subspace from paired completions and uses each prompt's coordinates within it to generate a vocabulary-wide correction, thereby implementing implicitly adaptive steering without manually specified target tokens or parameter updates. HeadEdit improves all nine experimental settings across three tasks and three model families, with negligible inference overhead and no systematic loss of general capabilities. It also reveals a connection to gradient-based alignment. HeadEdit's low-dimensional representation partly predicts how preference tuning changes output logits on unseen prompts. The subspace learned from the model can also be reused after tuning, improving performance without re-extracting or retuning. These results show that HeadEdit provides a practical, lightweight, and interpretable way to calibrate model behavior through the unembedding matrix.
Zirui He, Haiyan Zhao, Jingyu Hu +5
New Jersey Institute of Technology · University of Bristol · Kuaishou Technology +1
How do the methods used to train language models to refuse harmful requests shape how that refusal actually works inside the model? We compare three post-training methods - supervised fine-tuning, reasoning-augmented fine-tuning (training on reasoning chains that justify a safety decision), and preference optimization (ORPO) - across three architecturally distinct models (Llama-3.1-8B, Gemma-2-9B, Qwen3-8B). We find that training method, not just data, reshapes how refusal is computed internally: reasoning-augmented training consistently produces a distinct kind of refusal computation, visible across all three models, while architecture independently shapes internal structure and how reliably refusal can be steered. Most importantly, no method we study achieves all three properties we would want from safe alignment at once: refusal that isn't concentrated in a few fragile components, safety gains that don't cost general capability, and safety behavior correctable through small, targeted edits. We caution against treating current post-training methods as a solved, reliable defense, especially for security-critical use. Code and models are available in https://github.com/hoangcuongnguyen2001/Beyond-Shallow-Alignment.
We present an automated, contrastive evaluation pipeline for auditing the behavioral impact of interventions on large language models. Given a base model M1 and an intervention model M2, our method compares their free-form, multi-token generations across aligned prompt contexts and produces human-readable, statistically validated natural-language hypotheses describing how the models differ, along with recurring themes that summarize patterns across validated hypotheses. We evaluate the approach in synthetic setting by injecting known behavioral changes and showing that the pipeline reliably recovers them. We then apply it to three real-world interventions, reasoning distillation, knowledge editing and unlearning, demonstrating that the method surfaces both intended and unexpected behavioral shifts, distinguishes large from subtle interventions, and does not hallucinate differences when effects are absent or misaligned with the prompt bank. Overall, the pipeline provides a statistically grounded and interpretable tool for post-hoc auditing of intervention-induced changes in model behavior.
Quintin Pope, Ajay Hayagreeve Balaji, Jacques Thibodeau +1