Discovering Jailbreaks via Curriculum of Models

Michał Burzyński; Noah T. Rossi; Shomit Basu; Andy Arditi

cs.LGOct 28, 2026

Discovering Jailbreaks via Curriculum of Models

Authors: Michał Burzyński, Noah T. Rossi, Shomit Basu, Andy Arditi

Organizations: University of Warsaw · Independent · Northeastern University

Abstract

Automated jailbreak discovery aims to find prompts that elicit prohibited behaviors from safety-trained language models, helping developers identify vulnerabilities before deployment. One class of approaches uses reinforcement learning to train an investigator model to generate prompts that elicit a specified behavior from a target model. However, training directly against a safety-trained target is difficult because nearly all prompts are refused, leaving the investigator with a sparse reward signal. Prior work addresses this with the propensity bound (PRBO), a proxy reward estimated by sampling responses from a more compliant target, and then adjusting for how likely those responses would have been under the original target. We introduce Curriculum of Models (CoM), an alternative method that trains the investigator against a sequence of modified versions of the target, progressing gradually from a version that readily exhibits the specified behavior to the original target. For jailbreak discovery, training begins against a target version that rarely refuses harmful requests, and advances to the next, harder version once the investigator reliably elicits the behavior from the current one. We evaluate CoM on elicitation tasks involving CBRN materials, explosives, and illicit drugs, using two open-weight targets, gemma-4-31b-it and gpt-oss-120b. We compare against PRBO and direct training on the original target without a curriculum. When the investigator is trained to elicit a single behavior, CoM finds successful jailbreaks more reliably than either baseline. When trained on multiple behaviors simultaneously, the CoM-trained investigator can elicit held-out behaviors, though PRBO-trained investigators achieve higher success rates. Our results suggest that a simple curriculum over target models offers a promising approach for addressing sparse rewards in rare-behavior elicitation.

Figures & tables

Figure 1: Curriculum of Models (CoM) for jailbreak discovery. Top: An investigator model is trained via RL to generate prompts that elicit a prohibited behavior in a target model. In each stage, we sample prompts from the investigator and one response per prompt from the target model; a judge model scores each response against a rubric, and this score is the RL reward for the corresponding investigator prompt. Only the investigator is trained; the target and judge are frozen. The target at every stage is the same underlying model with an anti-refusal LoRA applied at strength

\alpha

: training starts at

\alpha=1

, where refusals are heavily suppressed, and moves to the next stage, with smaller

\alpha

, once the mean reward reaches a threshold

\tau

. The final stage trains against the unmodified target (

\alpha=0

). Bottom: Mean reward per training step in a representative CoM run, in which a qwen3.5-9b investigator is trained to elicit synthesis instructions for Tabun, a nerve agent, from a gemma-4-31b-it target. Shading and dials show the current

\alpha

. Each time the mean reward reaches

\tau

(circles), the curriculum advances to a harder target and reward drops, then recovers; on the final, unmodified target, reward climbs well above

\tau

Target	Behavior	Direct RL	PRBO	CoM (ours)
gemma-4-31b-it (without reasoning)	Fentanyl	0.00 (0.00, 0.00)	0.00 (0.00, 0.00)	0.00 (0.00, 0.01)
	Sarin	0.00 (0.00, 0.00)	0.00 (0.00, 0.00)	0.11 (0.00, 0.26)
	MDMA	0.00 (0.00, 0.00)	0.00 (0.00, 0.00)	0.01 (0.00, 0.02)
	VX	0.00 (0.00, 0.00)	0.10 (0.00, 0.29)	0.50 (0.40, 0.63)
gemma-4-31b-it (with reasoning)	Fentanyl	0.00 (0.00, 0.00)	0.00 (0.00, 0.00)	0.11 (0.07, 0.18)
	Sarin	0.00 (0.00, 0.00)	0.00 (0.00, 0.00)	0.00 (0.00, 0.01)

Table 1: Single-behavior CBRN elicitation. Estimated pass@1 over 256 prompt–response pairs per seed against the original, unmodified target model, averaged over 3 training seeds, with (min, max) over seeds in gray . A pair counts as successful when the judge assigns a score of 100, i.e., every criterion in the strict rubric is satisfied. Best mean per behavior in bold .

Figure 2: Multi-behavior elicitation results. For each method and target model, we trained three seeds on a set of 8 behaviors. We then evaluate the pass@1 attack success rate of each seed on both the trained-on behaviors and a set of 39 held-out behaviors. Each dot shows the mean pass@1 for that seed over all trained or held-out behaviors, and each bar shows the mean of all the seeds. Full results with all behaviors shown individually are available in Appendix E.2 .

Figure 3: Hyperparameter ablations. Starting from our default configuration (

K=5

\tau=60

\beta_{\mathrm{KL}}=0.5

; shaded), we vary one hyperparameter at a time while holding the other two fixed. Each dot is one seed, scored by its final checkpoint’s mean judge score over 256 rollouts against the original target

M_{0}

. Diamonds mark the mean over the 3 seeds per setting.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Per-behavior elicitation success for the original and refusal-weakened targets, using prompts sampled from an investigator that has undergone light anti-refusal SFT but no RL training. We report estimated pass@128 for the original target and pass@16 for the refusal-weakened variant, counting a rollout as successful when its judge score is 100. Error bars show 95% bootstrap confidence intervals.

Figure 5: As we increase the strength of our refusal-weakening LoRA applied to gemma-4-31b-it (without reasoning), refusal rate on HarmBench ( Mazeika et al., 2024 ) falls. This empirical trend supports ordering the curriculum from

\alpha=1

, the easiest target, to

\alpha=0

, the original target.

	gemma-4-31b-it	gpt-oss-120b
LoRA rank $r$	1	1
LoRA $\alpha$	16	32
Target modules	attention and MLP projections	all linear layers, incl. MoE experts
Learning rate	$1\times 10^{-4}$ , cosine	$1\times 10^{-4}$ , cosine
Batch size	32	128
Max sequence length	2048	8192

Appendix

Table 2: Hyperparameters of the refusal-weakened LoRA for each target.

Reward type	Natural threshold	$\lambda$	Mean judge score (95% CI)	pass@1	pass@32
Combined Estimator	null	0.9	0.0 (0.0, 0.0)	0.000	0.000
		1.1	33.9 (28.4, 39.3)	0.199	1.000
		1.3	0.8 (0.0, 1.9)	0.008	0.235
		1.5	0.0 (0.0, 0.0)	0.000	0.000
		1.7	0.0 (0.0, 0.0)	0.000	0.000
		2.5	0.0 (0.0, 0.0)	0.000	0.000

Appendix

Table 3: Tabun elicitation against gemma-4-31b-it . We trained each investigator for 400 steps to elicit the tabun synthesis and then evaluated the final checkpoint over 256 rollouts on tabun. We used a KL-coefficient of 0.1 in all of these runs. The natural-response threshold determines the score that a natural rollout must achieve in order for the combined PRBO formula to be used; otherwise, the vanilla PRBO formula is used.

		pass@1						pass@32
Method	Behavior	1/6	2/6	3/6	4/6	5/6	6/6	1/6	2/6	3/6	4/6	5/6	6/6
CoM	Fentanyl	0.975	0.974	0.951	0.785	0.185	0.004	1.000	1.000	1.000	1.000	0.830	0.110
	Sarin	0.898	0.888	0.870	0.789	0.513	0.115	1.000	1.000	1.000	1.000	1.000	0.649
	MDMA	0.960	0.958	0.941	0.802	0.344	0.009	1.000	1.000	1.000	1.000	0.996	0.247
	VX	0.962	0.962	0.962	0.961	0.931	0.501	1.000	1.000	1.000	1.000	1.000	1.000
PRBO	Fentanyl	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000

Appendix

Table 4: Single-behavior elicitation, full results ( gemma-4-31b-it , without reasoning target). We report pass@1 and pass@32 against the unmodified gemma-4-31b-it target without reasoning at six pass thresholds, where a rollout passes when the target response clears all three response gates and satisfies at least

j

of the six content criteria, and the investigator’s prompt passes the prompt judge. We evaluate 256 rollouts per seed and average across three seeds.

		pass@1						pass@32
Method	Behavior	1/6	2/6	3/6	4/6	5/6	6/6	1/6	2/6	3/6	4/6	5/6	6/6
CoM	Fentanyl	0.962	0.957	0.944	0.816	0.479	0.109	1.000	1.000	1.000	1.000	1.000	0.952
	Sarin	0.863	0.852	0.702	0.471	0.215	0.003	1.000	1.000	1.000	1.000	0.693	0.078
	MDMA	0.947	0.944	0.928	0.857	0.629	0.051	1.000	1.000	1.000	1.000	1.000	0.374
	VX	0.895	0.895	0.895	0.887	0.792	0.336	1.000	1.000	1.000	1.000	1.000	0.997
PRBO	Fentanyl	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000

Appendix

Table 5: Single-behavior elicitation, full results ( gemma-4-31b-it , with reasoning target). We report pass@1 and pass@32 against the unmodified gemma-4-31b-it target with reasoning at six pass thresholds, where a rollout passes when the target response clears all three response gates and satisfies at least

j

of the six content criteria, and the investigator’s prompt passes the prompt judge. We evaluate 256 rollouts per seed and average across three seeds.

		pass@1						pass@32
Method	Behavior	1/6	2/6	3/6	4/6	5/6	6/6	1/6	2/6	3/6	4/6	5/6	6/6
CoM	Fentanyl	0.948	0.948	0.948	0.948	0.938	0.807	1.000	1.000	1.000	1.000	1.000	1.000
	Sarin	0.930	0.930	0.930	0.930	0.930	0.884	1.000	1.000	1.000	1.000	1.000	1.000
	MDMA	0.898	0.898	0.896	0.892	0.871	0.710	1.000	1.000	1.000	1.000	1.000	1.000
	VX	0.928	0.928	0.928	0.928	0.928	0.882	1.000	1.000	1.000	1.000	1.000	1.000
PRBO	Fentanyl	0.934	0.915	0.770	0.061	0.005	0.001	1.000	1.000	1.000	0.606	0.157	0.042

Appendix

Table 6: Single-behavior elicitation with gpt-oss-120b on behaviors selected for gemma-4-31b-it . We report pass@1 and pass@32 against the unmodified gpt-oss-120b target at six pass thresholds, where a rollout passes when the target response clears all three response gates and satisfies at least

j

of the six content criteria, and the investigator’s prompt passes the prompt judge. We evaluate 256 rollouts per seed and average across three seeds.

		pass@1						pass@32
Method	Behavior	1/6	2/6	3/6	4/6	5/6	6/6	1/6	2/6	3/6	4/6	5/6	6/6
CoM	TATP	0.829	0.824	0.801	0.751	0.589	0.264	1.000	1.000	1.000	1.000	1.000	1.000
	HMTD	0.884	0.883	0.875	0.859	0.828	0.611	1.000	1.000	1.000	1.000	1.000	1.000
	Meth.	0.882	0.879	0.875	0.850	0.794	0.592	1.000	1.000	1.000	1.000	1.000	1.000
	Heroin	0.927	0.926	0.914	0.891	0.784	0.326	1.000	1.000	1.000	1.000	1.000	1.000
PRBO	TATP	0.861	0.797	0.422	0.031	0.007	0.001	1.000	1.000	0.979	0.528	0.189	0.042

Appendix

Table 7: Single-behavior elicitation with gpt-oss-120b on behaviors selected for gpt-oss-120b . We report pass@1 and pass@32 against the unmodified gpt-oss-120b target at six pass thresholds, where a rollout passes when the target response clears all three response gates and satisfies at least

j

of the six content criteria, and the investigator’s prompt passes the prompt judge. We evaluate 256 rollouts per seed and average across three seeds.

Panel of Figure 6 — Figure 6: Multi-behavior training on gemma-4-31b-it with 8 behaviors. Each dot is the mean pass@1 over 3 training seeds; bars indicate the lowest and highest seed.

Panel of Figure 7 — Figure 7: Multi-behavior training on gpt-oss-120b with 8 behaviors. Each dot is the mean pass@1 over 3 training seeds; bars indicate the lowest and highest seed.

Figure 8: Training dynamics across ablations. Each column is one hyperparameter sweep (

\tau

\beta_{\mathrm{KL}}

K

). The top row shows the steering coefficient

\alpha

over training, which the curriculum anneals from

1

toward

0

as each stage is solved, and the bottom row shows the judge score at

\alpha=0

, measuring performance on the original target. Bold lines are the mean over three seeds and faint lines are individual seeds.

Figure 9: Policy entropy across ablations. Each column is one hyperparameter sweep (

\tau

\beta_{\mathrm{KL}}

K

), showing policy entropy over training. Bold lines are the mean over three seeds and faint lines are individual seeds.

Figure 10: In all of our multi-behavior training settings, the investigator quickly learns not to refuse to produce prompts. In almost all of the our single-behavior settings, the opposite is true.

Setup	Seed	Steps	Inv. refusal
Standard (group 64)	1	400	100.0%
	2	400	100.0%
	3	400	98.4%
Group size 256	1	200	100.0%
	2	100	98.0%
	3	100	97.7%

Appendix

Table 8: PRBO ablations. We trained against gpt-oss-120b on the HMTD synthesis. In the 256 group size experiments, we reduced the step count to compensate, except for seed 1 where we ran 200 steps to see if this would help.

judge	exact	within 1 level	pass/fail
gpt-oss-120b (run 2)	81.6	95.3	92.9
gemma-4-31b-it	68.3	90.4	89.6
gemini-3.5-flash-lite	68.4	87.3	88.6
kimi-k3	69.2	89.1	86.0
glm-5.3-flash	68.6	85.0	85.3
deepseek-v4.1-flash	58.7	75.5	77.7

Appendix

Table 9: Agreement with the reference judge ( gpt-oss-120b , run 1): % of rollouts with the same reward, a reward within one level, and the same pass/fail verdict (pass = reward 100). 1,024 rollouts, 512 per target.

Figure 11: Judge agreement on gemma-4-31b-it responses. Reward from each judge (x) against the reference judge, gpt-oss-120b ’s first run (y), on 512 rollouts pooled over behaviors. Cells count rollouts; colour is the share of the 512 rollouts.

Figure 12: Judge agreement on gpt-oss-120b responses. As in Figure 11 .

Figure 13: Judge reasoning length per rollout. For each of the 1,024 rollouts, the number of characters in gpt-oss-120b ’s reasoning trace (x) against deepseek-v4.1-flash ’s (y), coloured by behavior. The dashed line is

y=x

; points above it are rollouts deepseek-v4.1-flash reasoned longer on. Axes are logarithmic.

LLM Security

Large language model (LLM) security studies attacks, privacy risks, and defenses across model training, inference, and deployment.

See all 327 papers

Identically Trained Language Models Break on Different Jailbreaks
Oct 25, 2026Chi-Wei Chang, Mu-Tien Kuo, Kuan-Wei Lee +1Language Model Safety Evaluation LLM Jailbreak Attacks
Cerulean Labs · National Taiwan University, Taipei, Taiwan · National Yang Ming Chiao Tung University, Hsinchu, Taiwan
BAIT: Boundary-Guided Disclosure Escalation LLM Jailbreaking via Self-Conditioned Reasoning
May 26, 2026Xuan Luo, Yue Wang, Geng Tu +2Jailbreak Attacks LLM Safety
The Harbin Institute of Technology, Shenzhen, China · The Hong Kong Polytechnic University, Hong Kong, China · Shenzhen University, Shenzhen, China +1
Jailbreaking for the Average Jane: Choosing Optimal Jailbreaks via Bandit Algorithms for Automatically Enhanced Queries
Jun 25, 2026Prarabdh Shukla, Ritik, Suhas Rao +2Adversarial Attacks on LLMs LLM Jailbreak Attacks
Centre for Machine Intelligence and Data Science, IIT Bombay · Department of Computer Science and Engineering, IIT Bombay

Oct 25, 2026J/K move · Enter open · S save

Identically Trained Language Models Break on Different Jailbreaks

Chi-Wei Chang, Mu-Tien Kuo, Kuan-Wei Lee, Meng-Hsun Tsai

Cerulean Labs · National Taiwan University, Taipei, Taiwan · National Yang Ming Chiao Tung University, Hsinchu, Taiwan

Language Model Safety Evaluation LLM Jailbreak Attacks Jailbreak Attack Evaluation Language Model Pretraining

Open paper AskPDF arXiv

Jailbreak benchmarks test one checkpoint and treat which jailbreaks break it as a property of its pretraining data and safety training. We ask whether another pretraining run of the same data, given the same safety training, would break on the same jailbreaks. For each of 25 pretraining data recipes, we take 1B base models from 3 pretraining seeds and give each the same safety post-training with 2 post-training seeds, for 150 models in all. We then collect 20 responses from every model to a shared set of jailbreak prompts and correlate the per-prompt non-refusal rates between every pair of models. On average, two post-trained copies of the same pretraining run have per-prompt non-refusal rates that correlate at 0.984. A different pretraining seed lowers the correlation to 0.811, and a different data recipe lowers it only slightly further, to 0.779. These results indicate that which jailbreaks break a checkpoint depends much more on its pretraining run than on its data, a pattern that holds in all 25 data recipes and at 150M and 300M parameters. In practice, the best of 91 attack templates found on one checkpoint stays the best after a new pretraining seed in only 27% of model pairs, against 67% after a new post-training seed. Each pretraining run also leaves a fingerprint: across 3,211 prompts, a model's per-prompt non-refusal rates identify its pretraining run among all 75 runs every time, and a random subset of 50 prompts still does so 98% of the time.

Abstract

LLM Security

Explore similar work

Identically Trained Language Models Break on Different Jailbreaks

BAIT: Boundary-Guided Disclosure Escalation LLM Jailbreaking via Self-Conditioned Reasoning

Jailbreaking for the Average Jane: Choosing Optimal Jailbreaks via Bandit Algorithms for Automatically Enhanced Queries