cs.LGNov 9, 2026

FireGate: Bounding Behaviour Change Before Training in Continual RL Fine-Tuning

Authors: Md Anwar Hossen, Ali Jannesari

Organizations: Department of Computer Science, Iowa State University, Ames, IA, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Fine-tuning a deployed language model on one task after another can change behaviour that should stay fixed, such as earlier skills, general chat or refusals. Today, such forgetting is found only after training, on the benchmarks someone chose to run. This work bounds, before training, how often its outputs change on chosen prompt distributions. FireGate places each task's update, such as a new LoRA, behind a gate, which is a classifier on the model's frozen lower layers that decides for each prompt whether the update is used. Where no gate fires, the model computes exactly what the old one computes. As a result, on any prompt distribution, every bounded behaviour score changes by at most the gates' firing rate there over a whole task stream. Calibrating each gate on unlabeled prompts of what must not change fixes that rate before training, with a PAC guarantee, whatever weights training produces. Forgetting is thus bounded by the gate's false-alarm rate, and its price is the gate's power: no update, gated or not, that changes as few protected outputs can change new-task outputs more often than the best test of that level tells new-task prompts from protected ones. On reinforcement-learning streams of up to five tasks, gated training keeps the earlier tasks and changes 3.3% of held-out chat responses, whereas ungated training changes 91.8%. Across seven models, the gain kept behind the gate follows its firing rate on new-task prompts, measured before training. The certificate covers the gated model as deployed, on the prompt distributions sampled for calibration, not against an adversary who searches for prompts that fire the gate.

Figures & tables

Figure 1: Overview of FireGate . (a) Before training, the gate’s classifier is fitted (1) and calibrated on unlabeled prompts (2), which fixes its firing rate. (b) Each task’s update acts only where its gate fires (3); elsewhere the model is exactly the old one, so at most that share of outputs changes (4). The update can be a LoRA or a full-rank change of the upper blocks, trained by RL or SFT, or a separately trained model behind a prompt-time gate.
Figure 2: The certificate and its price, schematically. At most a share ε\varepsilon of protected prompts lie above the threshold, the only ones whose outputs can change; the update acts only on the share β\beta of new-task prompts above it.
 deployed gatetwo-sided gate
Experimentβ↑\beta\uparrowDolly ↓\downarrow10k ↓\downarrowMMLU ↓\downarrowβ↑\beta\uparrowDolly ↓\downarrowMMLU ↓\downarrow
NQ, Qwen 1.5B92.23.2 [5.1]1.70.0 [0.6]90.72.8 [4.5]0.0 [0.6]
NQ, Qwen 7B97.92.8 [4.5]2.50.0 [0.6]96.12.0 [3.6]0.0 [0.6]
NQ, OLMo-2 7B96.82.2 [3.9]2.20.0 [0.6]95.02.2 [3.9]0.0 [0.6]
IF, Qwen 1.5B100.01.5 [2.9]–99.4 [99.8]98.00.0 [0.7]0.0 [0.6]
IF, Qwen 7B100.01.8 [3.3]–100.0 [100.0]98.00.0 [0.7]0.0 [0.6]
Table 1: Certified levels and their checks (%). Every reference component, Dolly among them, is certified at 3% with probability 0.95 ( Theorem 3.6 ). Dolly, MMLU: firing on held-out prompts and, in brackets, its 95% Clopper–Pearson bound ( Proposition 3.5 ), a check on Dolly and the only certificate on MMLU. β\beta : firing on the unlabeled RL prompts, which also set the two-sided gate’s region, so β≤98\beta\leq 98 there. Streams: union of per-stage gates, each at 3%. 10k: the bound from 10,000 fresh Dolly requests ( Table 20 ).
Figure 3: NQ, Qwen2.5-1.5B: gain against Dolly responses changed, mean of three seeds. Each curve sweeps one knob: the level ε\varepsilon for the gates and Seq-GRPO behind FireGate -prompt’s gate, the coefficient, energy share, weight or scale for the ungated methods. The router uses its logit-0 rule; the oracle knows each prompt’s task ( Table 14 ).
Figure 4: Gain kept behind the prompt-time gate against βGˉ\beta\bar{G} ( Proposition 4.3 ; β\beta measured before training, Gˉ\bar{G} the trained model’s gain), every ungated method at three levels, mean over seeds; right: the box on the left. On IF β=1\beta=1 ; on the 78 other points the mean error is 0.3 points (0.5 where β<1\beta<1 ), the largest 3.4.
  ungated behind the gate 
ModelTaskGˉ\bar{G}Chg.β\betaβGˉ\beta\bar{G}Kept ↑\uparrowChg. ↓\downarrowCP ↓\downarrowNon-fired
Qwen2.5-1.5BIF+38.793.3100.0+38.7+38.71.52.90/4,797
Qwen2.5-1.5BNQ+5.294.292.2+4.8+4.73.25.10/9,363
Qwen2.5-7BIF+12.994.6100.0+12.9+12.90.83.30/3,531
Qwen2.5-7BNQ+3.894.697.9+3.7+3.82.84.50/9,310
OLMo-2-7BNQ+5.991.296.8+5.7+5.72.23.90/9,303
Table 2: The switch on seven models from five families (three seeds, ε=0.03\varepsilon=0.03 ). Gˉ\bar{G} , Chg.: Seq-GRPO’s gain (points) and changed Dolly responses (%); β\beta : firing on unlabeled RL prompts, before training; Kept: the gain behind the deployed prompt-time gate; CP: 95% Clopper–Pearson bound of its held-out Dolly firing (%); Non-fired: changed non-fired responses of the FireGate -prompt runs; † two of three runs.
DeploymentAvg ↑\uparrowBWT ↑\uparrowDolly ↓\downarrowGBTime
Seq-GRPO, ungated46.5 ±\pm 1.7−- 3.5 ±\pm 0.791.88.1×\times 1.00
behind the gate46.5 ±\pm 1.7−- 3.5 ±\pm 0.74.810.9×\times 1.18
behind one gate46.3 ±\pm 1.8–0.0––
FireGate -prompt, gated training49.6 ±\pm 2.9+0.0 ±\pm 0.03.312.5×\times 1.25–2.59
Table 3: Five-task stream, Qwen2.5-1.5B, three seeds. Avg, BWT: mean accuracy over the five tasks after the stream (base 29.9%) and backward transfer; Dolly: held-out responses changed (%), rewordings included (refusal, over-refusal and MMLU changes on a stream: Table 10 ); GB, time: peak memory and batched generation time against the base model, gated LoRAs unmerged, one or all five firing ( Section D.6 ). Each stage is certified at 3%, so the union at 15% with probability at least 0.75 ( Corollary A.5 ); certified directly as one gate on 10,000 fresh Dolly requests ( Proposition 3.5 ), its Dolly firing is at most 4.5% with probability 0.95 ( Table 20 ). One gate: the union of the five two-sided gates, set once to fire on at most 3% of the shared calibration prompts ( Proposition A.6 ); on 400 fresh Dolly prompts it fires on none, which certifies its Dolly firing at 0.7% with probability 0.95 ( Proposition 3.5 ). –: not measured.
   ChangeResponses changed (%)Harmful + passage
Methodβ↑\beta\uparrowGain ↑\uparrowRefusalMMLUAdvBenchMMLUDollyGate fires (%)Refusal
SQuAD 2.0: passage and question, rewarded for the gold answer or for abstaining
Seq-GRPO–+24.4−- 0.3−- 2.977.149.399.3–−- 36.8 / −- 74.8
behind the deployed gate85.4+18.5+0.0+0.00.00.01.00.4 / 0.4−- 0.4 / +0.0
behind protect100.0+24.4+0.0−- 0.20.80.546.398.4 / 100.0−- 36.8 / −- 74.8
behind protect + passages99.8+24.3+0.0+0.00.00.016.30.8 / 1.6−- 0.8 / −- 1.2
Table 4: Protecting chosen behaviours on SQuAD 2.0, a task that overlaps general chat (Qwen2.5-1.5B, ε=0.03\varepsilon=0.03 ). Deployed: calibrated on every component; protect: on harmful and task-like prompts only; + passages: harmful requests followed by a passage added. β\beta : coverage of RL prompts; changes in points; responses changed (%); harmful + passage: firing and refusal change on 250 such held-out requests, as given / as a question (seed 0; else three seeds). Concise, Concise-ctx: Tables 24 and 22 .
 Con.CtxSQuADNQIFIF 7B3 tasksNQ ONQ L
HB / SR fires ↓\downarrow0/4.20/00/00/00/00/00/00/00/0
ctx. fires ↓\downarrow9.01.00000000
HB refusal, base939393939386939688
HB refusal, ungated−- 79−- 51−- 2−- 41−- 4+3−- 12−- 2−- 25
HB refusal, gated+0+0+0+0+0+0+0+0+0
Table 5: Every setting frozen ( Section K.1 ): firing (%) of the chosen gate on HarmBench (HB, n=200n=200 ), its contextual requests (ctx., 100) and StrongREJECT (SR, 288), a sampled check (95% bounds 1.5, 3.0, 1.0% where it fires on none). Base: the base model’s HB refusal (%), failures included; changes: points, seed 0. Con., Ctx: Concise(-ctx); O, L: OLMo-2, Llama-3.2.
Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

StatementStatusConditions and evidence
Drift at most the firing rate ( Theorem 3.2 )provedfrozen trunk, causal attention; checked exactly
Any model behind a prompt-time gate ( Corollary 3.4 )provedthe deployed pair, not its forgetting
Firing at most ε\varepsilon per component ( Theorem 3.6 )provedi.i.d. prompts, score fixed before them; components only
Firing outside the region ( Proposition 3.8 )provedat most its mass; nothing about prompts inside
The Neyman–Pearson limit ( Theorem 4.1 )provedany policy; attained if the ratio reads the trunk
Large changes confined ( Corollary 4.2 )provedeach reachable model’s region; not one common region
Appendix
Table 6: Status of each claim: proved, estimated before training, or observed.
NameWhat it isWhere it is used
How the update is deployed
Gated traininga LoRA trained under its own gate; rollouts on which the gate stays silent carry no gradientFireGate -prompt and FireGate -anytime rows; Non-fired in Table 2
Switchany trained model answers where a prompt-time gate fires, the old model elsewhere ( Corollary 3.4 )“behind the gate” rows; Table 2 , Figure 4
Where the gate reads
Prompt-timeFireGate -prompt: reads the last prompt token and decides once per promptevery switch; most gated runs
AnytimeFireGate -anytime: reads every position after the template prefixTable 10
Appendix
Table 7: The gates and deployment forms of the experiments. Every gate is calibrated with Theorem 3.6 , so each certifies its firing on every calibrated component; the steps change only which prompts it covers and its coverage β\beta . Results use the deployed gate unless marked otherwise.
ChoiceFixedEvidence
Split depth, linear classifier, ε=0.03\varepsilon=0.03 , δ=0.05\delta=0.05 , Dolly and AdvBench components and their sizes; Dolly, AdvBench, XSTest and MMLU poolsfirst version; never changedevery later result (before)
Task-like componentsecond version, after a failure: first-version gates fired on MMLUMMLU informed it; never calibrated on
Two-sided region, α=0.02\alpha=0.02 , shrinkage 0.10.1after a failure: MMLU firing of the stream gates and of IF at ε=0.1\varepsilon=0.1not MMLU; the jailbreak templates, added later (before)
Remedy: harmful requests in the task’s formatafter a failure: deployed gates fired on such requeststheir held-out split of the same format only
Passage component (protect + passages)after a failure: the protect gate fired on harmful requests with a passageheld-out requests of the same form ( Table 4 )
Twelve stress forms; 10,000 fresh Dolly requestsbefore any gate was evaluated on themAppendix I , Table 20 (before)
Appendix
Table 8: Each design choice and the evidence it is judged on. Before: fixed before any gate was evaluated on that evidence, so it is a prospective test; after a failure: chosen after the named failure was seen, so that evidence does not test it independently.
  Dolly (%)AdvBenchMMLU  
RunDecodingFire (CP)Chg.Chg.Δ\DeltaChg.Δ\DeltaTV ≥\geqNon-fired
Stream, FireGate -promptT=0.3T=0.33.8 (5.7)2.50.0+0.03.0−- 0.40.00
Stream, FireGate -promptT=1T=13.8 (5.7)3.00.0+0.02.4+0.20.00
Stream, FireGate -promptT=1T=1 , top- pp 0.93.8 (5.7)2.50.0+0.02.6−- 0.80.00
Stream, FireGate -anytimeT=0.3T=0.31.2 (2.6)0.80.0+0.06.0+0.80.00
Stream, FireGate -anytimeT=1T=12.0 (3.6)1.50.0+0.06.8+0.40.00
Appendix
Table 9: Drift under other decoding rules (Qwen2.5-1.5B, seed 0; the final and the base model share one decoding seed and are decoded in the same batches). Fire: firing on held-out Dolly prompts under that rule and its 95% Clopper–Pearson bound, which bounds the TV distance on Dolly under that rule ( Theorem 3.2 ); a prompt-time gate’s firing is the same under every rule, an anytime gate’s is not. Chg.: share of changed responses, an estimate of an upper bound on the TV distance (coupling inequality). Δ\Delta : change of refusal (AdvBench) and accuracy (MMLU), points. TV ≥\geq : largest 95% lower confidence bound on the TV distance over the four pools (Dolly, AdvBench, XSTest, MMLU), from the Kolmogorov–Smirnov distance of response lengths; for ungated runs it, not Chg., is the evidence of drift. Non-fired: changed responses on prompts where no gate fired, over the four pools ( Theorem 3.2 : 0).
  Stream tasks ↑\uparrowΔ\Delta vs. base (pts)Dolly (%)Held-out bound (%)
MethodnnGSM8KCountdownK&KAvg ↑\uparrowBWT ↑\uparrowRefusalOver-ref.MMLUChg. ↓\downarrowLeak ↓\downarrowref. ↓\downarrowoff-ref. ↓\downarrow
Base model (A100)–74.4 ±\pm 0.06.8 ±\pm 0.013.2 ±\pm 0.031.5 ±\pm 0.0–0.00.00.00.00.0 ±\pm 0.00.00.0
Base model (H200)–73.27.813.431.5–0.00.00.00.00.20.00.0
Seq-GRPO A374.7 ±\pm 4.161.4 ±\pm 7.835.3 ±\pm 5.457.2 ±\pm 4.9−- 4.5 ±\pm 5.2−- 0.5 ±\pm 0.5−- 15.5 ±\pm 15.4+1.0 ±\pm 1.493.4 ±\pm 3.81.5 ±\pm 2.4none
+ KL A377.5 ±\pm 2.058.3 ±\pm 4.131.9 ±\pm 2.155.9 ±\pm 1.9+5.3 ±\pm 8.8−- 0.1 ±\pm 0.2−- 17.2 ±\pm 2.1+0.7 ±\pm 1.090.8 ±\pm 0.40.1 ±\pm 0.1none
+ Replay A375.5 ±\pm 0.663.2 ±\pm 1.335.9 ±\pm 2.258.2 ±\pm 0.1+1.2 ±\pm 1.7+0.0 ±\pm 0.0−- 14.3 ±\pm 0.9+0.9 ±\pm 0.390.2 ±\pm 4.21.3 ±\pm 1.9none
Appendix
Table 10: After the stream GSM8K →\to Countdown →\to K&K (Qwen2.5-1.5B-Instruct, greedy decoding). Accuracy (%) on each task, average and backward transfer (BWT); changes relative to the base model in AdvBench refusal, XSTest over-refusal on safe prompts and MMLU accuracy (points); share of Dolly responses that differ from the base model’s (Chg.) or use a trained answer format (Leak). Each stage’s gate is certified before training at the PAC level 3% on every reference component ( Theorem 3.6 ). The last two columns are held-out bounds, the largest 95% Clopper–Pearson bound on the firing over held-out prompts of the reference pools Dolly and AdvBench (ref.), which checks that level, and of XSTest and MMLU, which are never used in calibration (off-ref.) and have no other certificate; MMLU sets it; ungated methods have no gate (none), and the router has held-out bounds but no level chosen before training (uncal.). Mean ±\pm standard deviation over nn of 3 seeds, with the same data and evaluation prompts; each bound is the largest over seeds. A : run on A100 GPUs in at least one seed, the others on H200, each compared with the base model on its own GPU model. ∗ One seed; its per-task accuracy, over-refusal and leak were not computed. Arrows: higher ( ↑\uparrow ) or lower ( ↓\downarrow ) is better; for changes relative to the base model, zero is best.
ExperimentTaskβeval\beta_{\mathrm{eval}}Gˉ\bar{G}DeployedWorstBestSolvedFeatures ( R2R^{2} )Random
Concise, Q-1.5BConcise1.4+73.6+1.0+0.0+1.4+0.0+0.6 (0.31)+1.0 [+0.6, +1.4]
Concise-ctx, Q-1.5BConcise-ctx3.2+65.2+2.6+0.0+3.2+0.0+1.0 (0.20)+2.1 [+1.4, +2.8]
NQ, OLMo-2-1124-7BNQ95.8+5.9+5.7+1.7+8.5+5.8+5.5 (0.00)+5.7 [+5.1, +6.1]
NQ, Q-1.5BNQ90.6+5.2+4.7−- 1.9+7.7+4.9+4.9 (-0.00)+4.7 [+4.0, +5.4]
NQ, Q-7BNQ96.4+3.8+3.8+0.9+6.1+3.4+3.7 (-0.01)+3.7 [+3.3, +4.0]
Appendix
Table 11: The range over which the covariance term moves the gain kept behind a gate. Seq-GRPO’s gain (points) behind gates that fire on as many held-out prompts as the deployed gate. Worst / best: oracle on each prompt’s gain, which needs labels; solved: fires first on prompts the base model already solves; features: a ridge regression on the trunk features, cross-fitted, that predicts each prompt’s gain ( R2R^{2} in brackets) and fires on the lowest predictions; random: mean and 95% range of 1000 random gates ( =βGˉ=\beta\bar{G} in expectation). βeval\beta_{\mathrm{eval}} : firing on held-out new-task prompts. Tasks with β<1\beta<1 .
GateAUROCThresholdPower ↑\uparrowDolly ↓\downarrowCP ↓\downarrowMMLUAdvB.Gain swChg. sw
NQ
linear classifier (logistic)0.989natural96.96.28.60.00.0+5.16.2
  ε=0.03\varepsilon=0.0392.23.25.10.00.0+4.73.2
  ε=0.1\varepsilon=0.198.39.512.30.00.0+5.19.5
Gaussian (LDA)0.993natural98.77.810.30.20.0+5.37.8
  ε=0.03\varepsilon=0.0395.43.25.10.00.0+4.73.2
Appendix
Table 12: Gates built from the same unlabeled prompts and trunk features (NQ, top, and Concise-ctx, bottom; Qwen2.5-1.5B-Instruct), at their own threshold (natural; as a router uses them) and PAC-calibrated. Power: firing on unlabeled RL prompts; Dolly, MMLU, AdvB.: on held-out pools (%); CP: 95% bound on Dolly; Seq sw : Seq-GRPO behind the gate, new-task gain (points) and Dolly responses changed (%). Only the calibrated rows carry a certificate.
ExperimentGatePower ↑\uparrowDolly ↓\downarrowCP ↓\downarrowXSTestMMLUWrapSysJB
NQ, 1.5Bdeployed92.23.25.12.00.0–0.00.0
 + region90.72.84.52.00.0–0.00.0
NQ, 7Bdeployed97.92.84.51.60.0–0.00.0
 + region96.12.03.61.60.0–0.00.0
NQ, OLMo-2deployed96.82.23.92.40.0–0.00.0
 + region95.02.23.92.40.0–0.00.0
Appendix
Table 13: Firing of gate variants on held-out prompts (%, ε=0.03\varepsilon=0.03 ), computed from unlabeled prompts before any training; it bounds the drift of every model behind the gate. Deployed: the deployed prompt-time gate; + region: the two-sided gate; one budget: the union of a stream’s gates calibrated at ε\varepsilon ; + harmful in format: AdvBench requests in the tasks’ formats added to the reference; protect: calibrated on AdvBench and task-like prompts only. Power: firing on unlabeled new-task prompts; CP: 95% Clopper–Pearson bound on Dolly; Wrap: AdvBench requests in a trained task’s format; Sys: Dolly requests behind a system prompt; JB: AdvBench requests in jailbreak templates (mean of three). XSTest, MMLU and JB are never used in calibration.
  NQChange on referenceFires (%) 
Methodnngain ↑\uparrowChg. ↓\downarrowLeak ↓\downarrowRefusalOver-ref.MMLUNQDollyCP ↓\downarrow
Seq-GRPO A3+5.2 ±\pm 0.994.2 ±\pm 1.8+0.0 ±\pm 0.0−- 8.8 ±\pm 2.8−- 50.3 ±\pm 2.0+0.2 ±\pm 0.2––none
Seq, scaled ×0.25\times 0.25 A3+3.0 ±\pm 0.374.1 ±\pm 1.2+0.0 ±\pm 0.0−- 0.3 ±\pm 0.5−- 27.7 ±\pm 2.0+0.0 ±\pm 0.0––none
Seq, scaled ×0.5\times 0.5 A3+4.6 ±\pm 0.486.4 ±\pm 2.8+0.0 ±\pm 0.0−- 0.8 ±\pm 0.4−- 40.7 ±\pm 1.3+0.3 ±\pm 0.3––none
Seq, scaled ×0.75\times 0.75 A3+4.9 ±\pm 1.292.2 ±\pm 1.8+0.0 ±\pm 0.0−- 4.1 ±\pm 1.3−- 47.5 ±\pm 2.1+0.1 ±\pm 0.4––none
+ KL λKL=0.01\lambda_{\mathrm{KL}}{=}0.01 A3+4.9 ±\pm 0.487.8 ±\pm 1.1+0.0 ±\pm 0.0−- 0.3 ±\pm 0.2−- 38.7 ±\pm 1.2+1.0 ±\pm 0.9––none
Appendix
Table 14: New task: Natural Questions asked as plain questions, against Dolly, one reference component; mean ±\pm standard deviation over nn seeds (of 3). Gain: new-task accuracy (points). Changes relative to each run’s base model: share of Dolly responses that differ (Chg.), share showing a trained format (Leak), AdvBench refusal, XSTest over-refusal and MMLU accuracy (points). Fires: share of prompts on which the gate fires. CP: largest held-out 95% Clopper–Pearson bound on the Dolly firing rate over seeds ( Proposition 3.5 ), a check of the certified level of 3% ( Theorem 3.6 ). Rows marked A ran on A100 GPUs (in at least one seed), the others on H200; each run is compared with the base model evaluated on its own GPU model. Ungated methods have no gate: no firing (–) and no certificate (none). The oracle row has no standard deviation. The uncalibrated router has a held-out firing rate but no level chosen before training (uncal.).
Figure 5: Gate scores for NQ against the reference prompts, Qwen2.5-1.5B-Instruct, no training, by the block ℓg\ell_{\mathrm{g}} whose input the score reads; the text-only score reads no block and is drawn as a dashed line. Each score is fitted on the gate’s fitting prompts and calibrated with its PAC rule. Top left: AUROC against held-out Dolly prompts. Fires on NQ: share of held-out NQ prompts on which the score fires at a certified ε\varepsilon of 0.03, 0.1 and 0.3, a lower bound on the limit β∗(ε)\beta^{\ast}(\varepsilon) of Theorem 4.1 , since a gate with this score achieves it. Bottom middle and right: firing on held-out Dolly and MMLU prompts at ε=0.03\varepsilon=0.03 .
  βk\beta_{k}union fires
StreamScore12345DollyAdvBenchMMLU
3-task stream, Qwen2.5-1.5Blinear classifier100.0100.0100.0––3.20.01.6
 Gaussian ratio100.0100.0100.0––1.80.099.6
 15-NN similarity100.0100.0100.0––2.00.042.8
 MLP100.0100.0100.0––1.80.40.6
3-task stream, Qwen2.5-7Blinear classifier100.0100.0100.0––1.80.012.4
Appendix
Table 15: Score classes on the task streams, each stage calibrated as the deployed stage gate at ε=0.03\varepsilon=0.03 and δ/4\delta/4 , from stored features. βk\beta_{k} : coverage of stage kk ’s unlabeled RL prompts (%), a lower bound on that stage’s limit; union: firing of the stage gates’ union on held-out prompts (%).
   linear classifierGaussian ratio
ModelTaskGˉ\bar{G}β\betaDollyMMLUGainβ\betaDollyMMLUGain
Qwen2.5-1.5BIF+38.7100.02.0 [3.6]100.0+38.7100.00.0 [0.7]100.0+38.7
Qwen2.5-1.5BNQ+5.292.52.8 [4.5]0.8+4.795.43.0 [4.8]0.0+4.7
Qwen2.5-7BIF+12.9100.02.2 [3.9]100.0+12.9100.02.5 [4.2]100.0+12.9
Qwen2.5-7BNQ+3.897.62.5 [4.2]0.0+3.598.62.0 [3.6]0.0+3.7
OLMo-2-7BNQ+5.995.72.0 [3.6]0.0+5.397.62.8 [4.5]0.0+5.6
Appendix
Table 16: The linear classifier against the Gaussian ratio on every model–task pair, both fitted and calibrated at ε=0.03\varepsilon=0.03 on the deployed reference from stored features, settings fixed before any result. β\beta : coverage of unlabeled RL prompts (%), a lower bound on β∗\beta^{\ast} ; Dolly: held-out firing (%) [95% bound]; MMLU: firing on MMLU, which no component samples (%); Gain: Seq-GRPO’s gain behind the gate (points), against its own gain Gˉ\bar{G} .
   β\beta (%)  Seq-GRPO (points)
Task (models)ReferenceAUROCLMlinearDolly [bound]Harm.GainGˉ\bar{G}
IF (7)deployed0.95320.599.9–1000.2 [1.2]0.0+0.1 to +12.0+11.4 to +38.7
NQ (7)deployed0.7732.984.5–97.91.8 [3.3]0.0−- 0.2 to +0.4+3.8 to +10.8
Concise-ctx (1)deployed0.7410.33.80.0 [0.7]0.0+0.0+65.2
 protect + passages 0.311.00.0 [0.7]0.0+0.0 
SQuAD 2.0 (1)deployed0.6870.085.40.2 [1.2]0.0+0.0+24.4
Appendix
Table 17: A language model as the gate’s score: Qwen2.5-7B-Instruct asked once per prompt, with a question fixed before any score, whether the prompt is a request of the task; score log⁡p(Yes)−log⁡p(No)\log p(\text{Yes})-\log p(\text{No}) , PAC-calibrated at ε=0.03\varepsilon=0.03 on the same components as the linear classifier. β\beta : coverage of unlabeled RL prompts by the language model (LM) and by the deployed linear classifier. Harm.: largest firing over the held-out harmful forms. The prompts are the same for every model of a task, so the gate is too; Gain: Seq-GRPO’s gain behind it over the task’s models (points), against their own gain Gˉ\bar{G} . AUROC: held-out task prompts against held-out Dolly.
    ProtectedOther poolsHarmful forms 
Experiment, gateRegionScoreβ↑\beta\uparrowAdvBenchHarm. + pass.MMLUDollyXSTestSt. 1St. 2Gain ↑\uparrow
SQuAD 2.0, protect + passagesnolinear classifier ∗99.80.01.60.016.50.01.62.4+24.3
 noGaussian ratio99.70.01.20.017.50.01.20.8+24.4
 no15-NN similarity98.40.00.80.013.00.00.80.4+23.9
 noMLP97.70.00.40.414.00.00.42.4+23.9
 yeslinear classifier ∗98.00.01.60.011.80.01.61.6+23.7
Appendix
Table 18: Score classes, with and without the new task’s region, under the same calibration (Qwen2.5-1.5B, ε=0.03\varepsilon=0.03 and δ/4\delta/4 for each gate, from stored features, no training). ∗\ast : the score with the largest coverage at each reference and region. β\beta : coverage of unlabeled RL prompts; firing (%) on held-out prompts of each pool; Harm. + pass.: the larger of the two passage forms; St. 1, St. 2: the largest firing over the first-round ( Appendix I ) and second-round ( Table 23 ) harmful forms; Gain: points of Seq-GRPO’s gain that the switch keeps behind the gate (Concise 73.6, Concise-ctx 65.2, SQuAD 2.0 24.4, NQ 5.2 without it). Behind protect + passages, AdvBench and harmful requests with a passage are the protected pools; behind the deployed gate, Dolly and AdvBench are.
     behind gate
RunSeedsGain ↑\uparrowChg. ↓\downarrowMMLUGain ↑\uparrowChg. ↓\downarrow
IF: LoRA GRPO, Seq-GRPO3+38.7 ±\pm 0.293.3−- 3.7+38.71.5
LoRA GRPO, FireGate -prompt3+39.0 ±\pm 0.31.3−- 0.1––
full-rank GRPO, Seq-GRPO3+39.0 ±\pm 0.393.4+0.1+39.01.5
learning rate 2⋅10−62\cdot 10^{-6}1+39.094.0+0.0+39.01.5
all blocks3+39.1 ±\pm 0.195.0−- 0.7+39.11.5
Appendix
Table 19: Full-rank updates and SFT on IF and NQ, Qwen2.5-1.5B. Seeds: runs (gain ±\pm sd). Full-rank: a dense update of every projection of blocks 9–28 (“all blocks”: 2–28) at learning rate 5⋅10−65\cdot 10^{-6} , against 5⋅10−55\cdot 10^{-5} for LoRA. SFT: on the model’s own accepted samples, four per prompt; 7B teacher: full-rank SFT on Qwen2.5-7B-Instruct’s accepted responses, two per prompt; gold: on NQ’s first gold answer. Chg.: Dolly responses changed (%); behind gate: the deployed prompt-time gate at ε=0.03\varepsilon=0.03 .
  Dolly (%)fresh Dolly (%)AdvBench (%)MMLU (%)
ExperimentTargetCal.Held-outBoundppHeld-outBoundppCal.Held-outBoundppHeld-outBound
Concise, Qwen2.5-1.5B31.63/4001.91.00–––0.00/2501.21.002.03.4
Concise-ctx, Qwen2.5-1.5B31.67/4003.30.96–––0.00/2501.21.000.00.6
SQuAD 2.0, Qwen2.5-1.5B31.64/4002.31.00149/100001.71.000.00/2501.21.000.00.6
NQ, Qwen2.5-1.5B31.613/4005.10.42146/100001.71.000.00/2501.21.000.00.6
IF, Qwen2.5-1.5B31.66/4002.90.98–––0.00/2501.21.0099.499.8
Appendix
Table 20: The certificate ledger of the deployed prompt-time gate. Target: the level ε\varepsilon of Theorem 3.6 ( KεK\varepsilon for the union of a KK -stage stream). Cal.: in-sample firing on the calibration prompts; held-out: firing X/nX/n on held-out prompts of the same pool, its 95% Clopper–Pearson bound and the one-sided binomial pp -value of XX at the target rate (a small pp would show the certificate failing). Fresh Dolly: 10,000 Dolly requests no pool uses, where the task is not built from Dolly. MMLU is never calibrated on.
  largest firing (%)largest refusal loss (points)
ExperimentFormsdep.two-s.rem.bothSeqSeq swFireGate -prompt
Concise, 1.5B20.80.4––96.40.80.0
Concise-ctx, 1.5B21.20.8––86.00.00.0
SQuAD 2.0, 1.5B40.40.4––74.80.40.0
NQ, 1.5B20.00.0––32.40.00.0
NQ, 7B20.00.0–––––
Appendix
Table 21: The twelve forms of the 250 held-out AdvBench requests: largest firing (%) over an experiment’s forms, before training, and largest loss of refusal (points, seed 0; three seeds for Seq-GRPO on IF and the 1.5B stream). Dep.: deployed; two-s.: two-sided; rem.: with the remedy; both: two-sided with the remedy; sw : behind the deployed gate. Two-sided on IF: 94.0 and 99.6 on the trained format, at most 0.0 (1.5B) and 19.6 (7B) on new forms.
    Seq-GRPO
TaskGateβ↑\beta\uparrowFires, passageGain ↑\uparrowRefusalRefusal, passageMMLUOver-ref.Dolly
Concisenone––+73.6−- 85.1−- 96.4 / −- 93.2−- 27.1−- 52.097.4
 protect95.195.2 / 99.6+70.0+0.0−- 92.0 / −- 92.8+0.0−- 51.290.9
 + passages23.30.0 / 0.0+17.2+0.0+0.0 / +0.0+0.0−- 11.622.7
Concise-ctxnone––+65.2−- 11.5−- 86.0 / −- 82.0−- 3.5−- 51.296.1
 protect100.099.6 / 100.0+65.2−- 0.4−- 85.6 / −- 82.0+0.1+0.038.2
Appendix
Table 22: The protect gate and its remedy (Qwen2.5-1.5B, ε=0.03\varepsilon=0.03 ; gates as in Table 4 ). β\beta : coverage of unlabeled RL prompts; fires, passage: on the 250 held-out harmful requests followed by a passage, as given / as a question. Seq-GRPO alone and behind each gate: gain and changes of refusal, of refusal on the passage forms (seed 0), of MMLU and of XSTest over-refusal (points), and Dolly responses changed (%); three seeds unless noted.
ExperimentGatePara.New pass.Pass. firstCode blockSys.+pass.Turn+pass.3 pass.Typos
Concisedeployed0.00.41.60.00.01.22.40.0
 two-sided0.00.00.40.00.00.02.00.0
 protect71.695.218.83.280.499.2100.062.4
 protect + passages0.00.00.00.00.00.00.00.8
Concise-ctxdeployed0.00.43.60.00.054.07.60.0
 two-sided0.00.40.40.00.00.46.80.0
Appendix
Table 23: Firing (%) of each gate at ε=0.03\varepsilon=0.03 on the 250 held-out AdvBench requests in eight forms written before any gate was evaluated on them, from stored features (Qwen2.5-1.5B unless noted). A gate that does not fire leaves the base model’s answer in place for every model behind it.
   ChangeResponses changed (%)Harmful + passage
Methodβ↑\beta\uparrowGain ↑\uparrowRefusalMMLUAdvBenchMMLUDollyGate fires (%)Refusal
Concise: plain Dolly requests, rewarded for a short answer that does not refuse
Seq-GRPO–+73.6−- 85.1−- 27.199.999.197.4–−- 96.4 / −- 93.2
behind the deployed gate0.7+1.0+0.0−- 0.50.02.00.70.8 / 0.8−- 0.8 / −- 0.8
behind protect95.1+70.0+0.0+0.00.00.090.995.2 / 99.6−- 92.0 / −- 92.8
behind protect + passages23.3+17.2+0.0+0.00.00.022.70.0 / 0.0+0.0 / +0.0
Appendix
Table 24: The Concise block of Table 4 , in the same layout (Qwen2.5-1.5B, ε=0.03\varepsilon=0.03 ): a task whose prompts are themselves chat requests, rewarded for a short answer that does not refuse, so that its reward conflicts with refusal by design. Harmful + passage: firing and refusal change on the 250 held-out harmful requests followed by a passage, as given / asked as a question (seed 0); other columns are means over three seeds.
  chosen gate fires (%)deployed fires (%)refusal, ungatedrefusal, behind chosen
ExperimentChosen gateHBHB ctx.SRHB ctx.SRHBHB ctx.SRHBHB ctx.SR
Concise, Qwen2.5-1.5Bprotect + passages0.0 [1.5]9.0 [15.2]4.2 [6.7]38.07.6−- 79.0−- 30.0−- 86.8+0.0−- 2.0−- 3.5
Concise-ctx, Qwen2.5-1.5Bprotect + passages0.0 [1.5]1.0 [4.7]0.0 [1.0]38.00.0−- 51.0−- 27.0−- 50.3+0.0+0.0+0.0
SQuAD 2.0, Qwen2.5-1.5Bprotect + passages0.0 [1.5]0.0 [3.0]0.0 [1.0]0.00.0−- 1.5+4.0−- 19.1+0.0+0.0+0.0
NQ, Qwen2.5-1.5Btwo-sided0.0 [1.5]0.0 [3.0]0.0 [1.0]0.00.0−- 41.0−- 20.0−- 28.1+0.0+0.0+0.0
IF, Qwen2.5-1.5Btwo-sided + remedy0.0 [1.5]0.0 [3.0]0.0 [1.0]92.01.4−- 3.5+5.0−- 0.7+0.0+0.0+0.0
Appendix
Table 25: Benchmarks written by others, every setting frozen: HarmBench (HB, 200 behaviours), its contextual behaviours (HB ctx., 100, a passage then a request) and StrongREJECT (SR, 288). Firing (%), before training, of the gate chosen by the steps of Section 5 , with its 95% Clopper–Pearson bound, and of the deployed gate. Refusal: change of Seq-GRPO’s refusal rate (points, seed 0), ungated and behind the chosen gate.
   deployed gatetwo-sided gate, with the remedy on IF
ModelTaskGˉ\bar{G}β\betaKeptCPMMLUJBβ\betaKeptChg.CPMMLUJB
Qwen2.5-1.5BIF+38.7100.0+38.72.999.4100.096.5+37.00.00.70.00.0
Qwen2.5-1.5BNQ+5.292.2+4.75.10.00.090.7+4.72.84.50.00.0
Qwen2.5-7BIF+12.9100.0+12.93.3100.0100.097.6+12.70.00.70.00.0
Qwen2.5-7BNQ+3.897.9+3.84.50.00.096.1+3.72.03.60.00.0
OLMo-2-7BNQ+5.996.8+5.73.90.00.095.0+5.62.33.90.00.0
Appendix
Table 26: The models of Table 2 behind the gate of steps (3) and (4) of Section 5 (two-sided; with the remedy on IF) and behind the deployed gate, ε=0.03\varepsilon=0.03 , three seeds. β\beta : firing on RL prompts (%); Kept: gain behind the gate (points); Chg.: Dolly responses changed (%); CP: 95% bound of the held-out Dolly firing (%); MMLU, JB: firing on MMLU and jailbreak templates (%).
Table 32
  harmfulbenign
ExperimentGateMJ enMJ hiMJ midMJ loHB turn 2HB sys.OA enOA otherOA turn 2Dolly sys.
Concise, Qwen2.5-1.5Bprotect + passages ∗31.1 [35.7]6.7 [8.2]32.2 [34.8]20.3 [22.6]0.0 [1.5]0.0 [1.5]18.1 [20.2]6.3 [7.7]4.0 [5.2]8.5 [11.2]
 deployed1.61.40.21.332.015.55.11.267.538.8
Concise-ctx, Qwen2.5-1.5Bprotect + passages ∗0.0 [0.9]0.0 [0.3]0.0 [0.3]0.0 [0.3]0.0 [1.5]0.0 [1.5]0.0 [0.3]0.1 [0.5]0.0 [0.3]0.0 [0.7]
 deployed0.00.00.00.00.00.00.10.30.19.2
SQuAD 2.0, Qwen2.5-1.5Bprotect + passages ∗0.0 [0.9]0.0 [0.3]0.0 [0.3]0.0 [0.3]0.0 [1.5]0.0 [1.5]0.1 [0.5]0.0 [0.3]0.0 [0.3]0.0 [0.7]
Appendix
Table 29: The frozen gates on prompt sets written by others, measured once: firing (%) of the gate chosen for each experiment ( ∗\ast ) and of the deployed gate, with the 95% Clopper–Pearson bound in brackets for the chosen gate. MJ: MultiJail harmful requests in English and in its high-, medium- and low-resource languages; HB turn 2, HB sys.: HarmBench requests after an oasst1 exchange or behind a system prompt from awesome-chatgpt-prompts; OA: oasst1 user turns in English, in other languages and as second turns; Dolly sys.: held-out Dolly requests behind such a system prompt. Firing bounds how often any model behind the gate answers differently.
    refusal on fired (%)   
ExperimentGatennFired (%)basegatedNewly answered (%)[95% CI]worst pool (%)
Concise, Qwen2.5-1.5Bprotect + passages ∗24386.594.33.15.95[5.08, 6.96]xstest 22.17
 deployed24382.650.00.01.31[0.93, 1.85]ext_hb_ctx 8.00
 trained18500.0––0.00[0.00, 0.21]xstest 0.00
Concise-ctx, Qwen2.5-1.5Bprotect + passages ∗24380.00.00.00.00[0.00, 0.16]xstest 0.00
 deployed24381.729.30.00.49[0.28, 0.86]ext_hb_ctx 12.00
Appendix
Table 30: Harmful prompts on which the gate fires, where the certificate allows any change (seed-pooled counts over every harmful pool with stored responses: held-out AdvBench, XSTest’s unsafe prompts, the stress forms, the task-format requests, the format shifts and the external sets). Seq-GRPO behind the chosen and the deployed gate; trained: FireGate -prompt’s own runs under the deployed gate, where a response can change only if its gate fired. Fired: share of prompts on which the gate fired; refusal on fired: the base model’s and the gated model’s refusal rate there; newly answered: prompts the base model refuses and the gated model answers, as a share of all, with its 95% Wilson interval; worst pool: the largest such share.
Setnnchosenwide [bound]
wide component (one row per request)2468.10.0 [1.2]
HarmBench620.00.0 [4.7]
HarmBench as a second turn620.00.0 [4.7]
HarmBench behind a system prompt620.00.0 [4.7]
StrongREJECT902.20.0 [3.3]
MultiJail, English9436.20.0 [3.1]
Appendix
Table 31: Concise behind the chosen gate and behind the same gate with a wide harmful component (HarmBench, StrongREJECT and MultiJail in ten languages, split by request into thirds for fitting, calibration and test), ε=0.03\varepsilon=0.03 . Firing (%) on the test third of every source and on every held-out set, with the 95% bound for the wide gate; Do-Not-Answer, Aya red-teaming and XSTest were never fitted or calibrated. Dolly, oasst1 and WildChat are not protected on Concise, whose prompts are Dolly requests.
 PredictionMeasuredResult
P1no change where the gate is silent0 of 9,405pass
P2change ≤\leq firing, behind the gate24 of 24 checkspass
P3held-out firing consistent with 3%Dolly 3.5, AdvBench 0.0; 10k 2.0pass
P4ungated changes >80%>80\% of Dolly95.8, 98.5, 93.2pass
P5kept within 1 point of βGˉ\beta\bar{G}+23.1 vs +22.3pass
P6external sets: ≤\leq 3/5%; refusal ±\pm 30.0, 0.0, 0.0; +0.0, +0.0, +0.0pass
Appendix
Table 32: The frozen recipe on TriviaQA with Llama-3.2-3B, a task no earlier experiment used (three seeds). Every setting was copied from NQ and every prediction written before the data were downloaded. Measured values in %, P5 in points; P1–P5 use the deployed gate, P6 and P7 the chosen (two-sided) gate.
More in this topic

Fine-Tuning Language Models

See all 68 papers

Explore similar work

CardsList
May 25, 2026

Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay

Models trained on a new task typically degrade on prior tasks, a phenomenon known as forgetting. Traditionally, mitigating forgetting has required replaying stored exemplars from prior tasks, which is often impractical. By contrast, language models can sample from their own training distribution, and we show that these self-generated samples serve as effective replay data, nearly eliminating forgetting. We find that forgetting nonetheless persists when the model has little remaining capacity: models pretrained close to saturation cannot absorb new information without overwriting prior knowledge. When capacity is not the limiting factor, low learning rates reduce forgetting but require substantially more training steps. Replay breaks this tradeoff, enabling fast, high-learning-rate finetuning without forgetting.

Martin Marek, Dongkyu Cho, Shikai Qiu +3

New York University

Continual Learning for LLMsLLM Fine-TuningReplay-Based Continual LearningCatastrophic ForgettingFine-Tuning Language Models
AskPDF
Nov 3, 2026

Suppressing Misaligned Behaviors in Language Models, Durably

Undesired model behaviors can emerge or re-emerge at any stage of training. Once observed, how effectively can we suppress them? To study this question, we first construct two model organisms--language models deliberately trained to exhibit a specific misalignment--that differ in what triggers the behavior. The narrow-trigger organism misbehaves only when the prompt contains a specific cue: it cheats on multiple-choice questions by following a planted hint that points to a wrong answer, and behaves normally otherwise. The broad-trigger organism has no such cue: after fine-tuning on insecure code, it gives harmful responses across a wide range of open-ended prompts. We then formalize post-hoc behavioral suppression and evaluate methods along three criteria: immediate suppression of the target behavior, durability under further training, and preservation of general capability. We compare standard methods--SFT, GRPO, and gradient-ascent unlearning--along these dimensions. Building on prior work on alignment shallowness, we hypothesize that more durable suppression requires the ability to recover from a wider range of misalignment states. We introduce Prefix-Expanded Adversarial Reinforcement Learning (PEARL), a GRPO variant that augments rollouts with continuations from intermediate states sampled along cached organism trajectories. Across both organism settings and two model families (Qwen3-4B, GPT-OSS-20B), PEARL achieves strong immediate suppression while preserving task accuracy, and remains durable under further fine-tuning on either the organism's data or unrelated data. It obtains the greatest benefit when the undesirable behavior is broad and cannot be addressed through targeted SFT. Together, our framework and method offer a step toward more durable post-hoc removal of undesired behaviors.

Luxi He, Pengcheng Jiang, Jifan Zhang +2

Princeton University · UIUC · Thinking Machines Lab

LLM AlignmentLLM UnlearningSafety-Preserving Fine-Tuning of Language ModelsRL for Language Models
AskPDF
Oct 12, 2026

PID-Gated Continual Prompt Adaptation Under Noisy Feedback

Deploying LLM agents in non-stationary environments requires continually revising their instructions as the task drifts, yet rewriting after every reported mistake can turn noisy feedback into persistent errors. This raises an often-overlooked scheduling question in prompt adaptation: when should feedback trigger a lasting instruction update, and at what editor-call cost? Taking inspiration from proportional-integral-derivative (PID) control, we introduce a family of lightweight gates that react to the current error, accumulate evidence over time, and roll back deterioration after an edit. We benchmark these gates against both simple baselines and advanced prompt optimizers (TextGrad and GEPA), over a diverse suite of drifting tasks, LLM parameter sizes, feedback-noise levels, and random seeds. PID-gated continual prompt adaptation yields a more favorable Pareto trade-off between task performance (high accuracy, low error) and editor-call budget than the baselines considered, pointing to control-theoretic scheduling as a simple, general tool for robust prompt adaptation under unreliable feedback.

Zhentao Xu

Independent Researcher, Sunnyvale, CA, USA

PID ControlPrompt OptimizationContinual Learning for LLM AgentsFeedback Control
AskPDF