Constitutional adapters: Inference-time interventions for misalignment and misuse

Adam S. Lowet; Mark Kurzeja

cs.LGSep 29, 2026

Constitutional adapters: Inference-time interventions for misalignment and misuse

Authors: Adam S. Lowet, Mark Kurzeja

Organizations: Anthropic Fellows Program · Independent

Abstract

Training models to act in accordance with an explicitly defined set of principles, or "constitution," has shown promise as a robust and transparent mechanism for AI alignment. However, the generality and flexibility of such methods remain unclear. Here, we show that constitution-consistent behavior can be distilled from synthetic corpora into lightweight objects (low-rank adapters and steering vectors). Despite never seeing a harmful request or jailbreak during training, such objects increase jailbreak defense success and measured alignment -- particularly at long context lengths and against multi-turn attacks, where they outperform both prompted and steered baselines. Subtracting control-trained from constitution-trained objects further accentuates these effects, yielding defenses we call "constitutional adapters" (CAs). CAs can be trained on a base model, transferred zero-shot to its post-trained checkpoint, and scaled at inference time to predictably trade off defense for benign compliance. Taken together, these results recommend CAs as a lightweight, portable, and tunable lever for mitigating misalignment and misuse in API deployments.

Figures & tables

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

	CONSTITUTIONAL		CONTROL
CORPUS	RECORDS	TOKENS	RECORDS	TOKENS
Advice	1,960	5.03M	1,995	4.88M
Stories	16,321	40.05M	16,294	40.94M
Total	18,281	45.08M	18,289	45.83M
Advice scenarios $\to$ records	4,000 $\to$ 1,960		3,000 $\to$ 1,995
Story drafts $\to$ records	18,000 $\to$ 16,321		18,000 $\to$ 16,294

Appendix

Table 1: Final corpora. Tokens are the minimum over the Llama-3, Gemma, and Qwen tokenizers. Constitutional attrition is dominated by prompt-drafting refusals (

{\sim}10\%

), the named-AI and persona filters, and register caps; control attrition by drafting refusals (3%) and the named-AI filter.

	ADVICE		STORIES
STATISTIC	CONST.	CONTROL	CONST.	CONTROL
Type-token ratio (100k-token samples)	0.0825	0.0835	0.071	0.065
Distinct 4-gram ratio	0.922	0.933	0.712	0.769
Mean pairwise TF-IDF cosine	0.071	0.059	0.091	0.080
Largest of 50 TF-IDF clusters (share of records)	5.1%	4.7%	5.5%	4.2%
Near-duplicate clusters (Jaccard $\geq 0.5$ )	0	0	0	0

Appendix

Table 2: Diversity statistics for both corpora. Lower cosine and a smaller largest-cluster share mean records are less alike. The control is as diverse as or slightly more diverse than the constitutional corpus on every axis except word-level vocabulary in stories (type-token ratio 0.065 vs. 0.071), where it uses fewer distinct words but repeats fewer phrases. Between-corpus vocabulary overlap far below the within-corpus baseline reflects different subject matter with matched form.

SETTING	LoRA ( $r=4$ )	ReLU STEER
layers	8 full-attention ( $W_{o}$ )	all 32
trainable parameters	212,992	163,872
initialization	$A\sim\mathcal{N}(0,1/d_{\mathrm{in}}),\;B=0$	$\mathbf{v}=\mathbf{0},\;\mathbf{w}=\mathbf{0},\;b=1$
peak learning rate	$6.68\times 10^{-3}$	$4.58\times 10^{-3}$
schedule	linear warm-up 15%, cosine to 0 over 16 epochs
optimizer	AdamW, weight decay 0.01

Appendix

Table 3: Fit hyperparameters shared by all three seeds and both corpora.

ARM	SEED	PAST TENSE	CODEATTACK	RENELLM	PLINY
unsteered (absolute)		$35.2$	$24.7$	$23.6$	$56.0$
Refusal steer	–	$+22.1$ $\pm 4.7$	$+20.1$ $\pm 3.4$	$+19.3$ $\pm 3.3$	$+34.4$ $\pm 6.3$
Safety SI	–	$+23.7$ $\pm 4.5$	$+23.6$ $\pm 2.6$	$+19.9$ $\pm 3.8$	$+20.5$ $\pm 6.9$
Constitution SI	–	$+10.8$ $\pm 5.7$	$+21.5$ $\pm 3.1$	$+18.0$ $\pm 4.5$	$+43.2$ $\pm 5.4$
Control ReLU steer	s1	$-11.5$ $\pm 6.4$	$-4.2$ $\pm 5.9$	$-30.5$ $\pm 6.4$	$-39.4$ $\pm 4.8$
Control ReLU steer	s2	$-15.7$ $\pm 6.5$	$-0.4$ $\pm 5.7$	$-37.1$ $\pm 7.8$	$-36.9$ $\pm 4.4$

Appendix

Table 4: Static defense for Qwen3.5-4B.

ARM	SEED	BEST-OF-75	PAIR
unsteered (absolute)		$76.1$	$44.3$
Refusal steer	–	$+51.8$ $\pm 10.7$	$+40.6$ $\pm 5.1$
Safety SI	–	$+60.5$ $\pm 9.8$	$+18.1$ $\pm 6.6$
Constitution SI	–	$+44.5$ $\pm 11.2$	$+25.4$ $\pm 6.1$
Control ReLU steer	s1	$-22.9$ $\pm 7.3$	$-56.2$ $\pm 4.0$
Control ReLU steer	s2	$-23.9$ $\pm 7.1$	$-56.2$ $\pm 4.0$

Appendix

Table 5: Multi-query defense for Qwen3.5-4B.

ARM	SEED	CRESCENDO	ACTORATTACK
unsteered (absolute)		$99.2$	$95.0$
Refusal steer	–	$+8.3$ $\pm 3.6$	$+6.9$ $\pm 4.3$
Safety SI	–	$+3.2$ $\pm 2.6$	$+4.9$ $\pm 3.9$
Constitution SI	–	$+0.6$ $\pm 1.7$	$+3.3$ $\pm 3.7$
Control ReLU steer	s1	$+1.5$ $\pm 2.1$	$+8.8$ $\pm 4.5$
Control ReLU steer	s2	$-0.1$ $\pm 1.4$	$+5.7$ $\pm 4.1$

Appendix

Table 6: Multi-turn defense for Qwen3.5-4B.

ARM	SEED	HARM	DEFER	DELUDE	EVADE	REASON	PRESERVE
unsteered (absolute)		$7.0$	$7.0$	$8.6$	$5.2$	$7.8$	$2.0$
Refusal steer	–	$+1.7$ $\pm 1.6$	$+4.5$ $\pm 1.8$	$+5.8$ $\pm 1.5$	$+2.5$ $\pm 1.8$	$+0.4$ $\pm 1.0$	$-0.8$ $\pm 1.5$
Safety SI	–	$+0.4$ $\pm 1.3$	$+1.4$ $\pm 2.3$	$+1.6$ $\pm 1.5$	$+0.3$ $\pm 1.7$	$+0.5$ $\pm 1.1$	$-0.4$ $\pm 1.0$
Constitution SI	–	$-0.2$ $\pm 1.1$	$+0.1$ $\pm 2.1$	$+1.3$ $\pm 1.4$	$+0.0$ $\pm 1.5$	$+0.2$ $\pm 1.0$	$+0.3$ $\pm 0.7$
Control ReLU steer	s1	$+0.3$ $\pm 1.1$	$+0.5$ $\pm 1.7$	$+0.6$ $\pm 0.7$	$-0.3$ $\pm 1.8$	$+0.5$ $\pm 1.0$	$+0.1$ $\pm 0.6$
Control ReLU steer	s2	$-0.0$ $\pm 1.1$	$+0.2$ $\pm 1.7$	$+0.7$ $\pm 0.8$	$-0.9$ $\pm 1.6$	$-0.8$ $\pm 0.6$	$-0.3$ $\pm 0.7$

Appendix

Table 7: Petri for Qwen3.5-4B.

ARM	SEED	AGENTIC: LEAKING
unsteered (absolute)		$39.3$
Refusal steer	–	$+31.3$ $\pm 4.9$
Safety SI	–	$-1.7$ $\pm 6.0$
Constitution SI	–	$+18.7$ $\pm 5.5$
Control ReLU steer	s1	$+25.4$ $\pm 5.2$
Control ReLU steer	s2	$+27.8$ $\pm 5.1$

Appendix

Table 8: Agentic misalignment for Qwen3.5-4B.

ARM	SEED	XSTEST	OR-BENCH-HARD
unsteered (absolute)		$5.0$	$76.4$
Refusal steer	–	$-10.1$ $\pm 5.0$	$-16.6$ $\pm 2.2$
Safety SI	–	$-5.3$ $\pm 4.3$	$-7.9$ $\pm 2.4$
Constitution SI	–	$-8.0$ $\pm 4.4$	$-1.1$ $\pm 2.6$
Control ReLU steer	s1	$+1.4$ $\pm 3.3$	$+52.6$ $\pm 2.5$
Control ReLU steer	s2	$+2.4$ $\pm 3.4$	$+46.4$ $\pm 2.5$

Appendix

Table 9: Over-refusal for Qwen3.5-4B.

ARM	SEED	MMLU	GSM8K	BIGCODEBENCH	BFCL
unsteered (absolute)		$73.8$	$96.2$	$45.8$	$78.3$
Refusal steer	–	$-0.5$ $\pm 2.3$	$-0.1$ $\pm 3.2$	$-1.9$ $\pm 4.1$	$-0.9$ $\pm 1.8$
Safety SI	–	$+0.1$ $\pm 2.3$	$+0.1$ $\pm 3.2$	$-1.4$ $\pm 4.1$	$-1.4$ $\pm 1.8$
Constitution SI	–	$-1.7$ $\pm 2.4$	$-0.0$ $\pm 3.2$	$-0.9$ $\pm 4.1$	$-6.7$ $\pm 1.9$
Control ReLU steer	s1	$-5.3$ $\pm 2.5$	$-3.5$ $\pm 4.1$	$-10.2$ $\pm 4.0$	$-33.4$ $\pm 2.0$
Control ReLU steer	s2	$-4.0$ $\pm 2.4$	$-1.0$ $\pm 3.6$	$-7.2$ $\pm 4.0$	$-33.4$ $\pm 2.0$

Appendix

Table 10: Capability for Qwen3.5-4B.

Explore similar work

Constitutional Midtraining: Content Presence Drives Alignment Gains
Jul 29, 2026Desiree Cho, Cameron Tice, Bernie Hogan +4Model Fine-Tuning
Department of Computer Science, University of Oxford · Institute for Ethics in AI, University of Oxford · 3Geodesic Research +1
How Well Do Models Follow Their Constitutions?
May 22, 2026Arya Jakkli, Senthooran Rajamanoharan, Neel NandaAdversarial Evaluation Adversarial Robustness
CentraleSup´elec · †Work conducted during the ML Alignment & Theory Scholars (MATS) program. · Google DeepMind
Consistency Training Along the Transformer Stack
Jun 4, 2026Sukrati Gautam, Neil Shah, Arav Dhoot +7Safety Alignment Training-Inference Mismatch
1Purdue University · 2Independent · 3Columbia University +4

Jul 29, 2026, cs.CLJ/K move · Enter open · S save

Constitutional Midtraining: Content Presence Drives Alignment Gains

Desiree Cho, Cameron Tice, Bernie Hogan, Hunar Batra+3

Department of Computer Science, University of Oxford · Institute for Ethics in AI, University of Oxford · 3Geodesic Research+1

Model Fine-Tuning

Open paper AskPDF arXiv

Post-training alignment is often shallow, eroding under fine-tuning. It remains untested as to whether constitutional midtraining interventions can produce durable alignment when cleanly isolated from post-training. We build a 394M-token constitutional corpus from Anthropic's Constitution and apply constitutional midtraining at 120B scale, where principled, values-based content is inserted into midtraining. A 2x2 design (curriculum ordering x deliberative reasoning) was used to produce four constitutionally midtrained conditions, plus a control, which were evaluated on self-generated and established benchmarks including alignment under pressure, value conflict resolution, blackmail, and emergent misalignment. All models were evaluated across three stages: post-midtraining, post-SFT, and post-benign fine-tuning. Constitutionally midtrained models outperformed the control on alignment generalization and durability, notably on blackmail: SFT instilled a blackmail propensity in all models, but constitutional midtraining blunted it, with the advantage surviving benign fine-tuning (-17.5pp). This durability did not extend to settings that required active resistance to in-context pressure or conflict, where the advantage attenuates after SFT. The presence of constitutional content at midtraining also mattered more than its structure, and constitutional midtraining incurred no capability cost, on average, at any stage (MMLU, ARC-Easy, piqa, GSM8K). A modest amount of constitutional content at midtraining could therefore yield broad, persistent alignment gains, offering a cheap, complementary addition to SFT-centered pipelines. Code, data, and models are available.

Abstract

Explore similar work

Constitutional Midtraining: Content Presence Drives Alignment Gains

How Well Do Models Follow Their Constitutions?

Consistency Training Along the Transformer Stack