Intervention-Certified Amplification: A Validity Gate for $σ$-Weighted Post-Training

Leiyue Chen; Meng Wei; Xuefeng Hu; Wei Liu

cs.LGOct 24, 2026

Intervention-Certified Amplification: A Validity Gate for $σ$ -Weighted Post-Training

Authors: Leiyue Chen, Meng Wei, Xuefeng Hu, Wei Liu

Organizations: ByteDance

Abstract

Hybrid post-training often mixes supervised fine-tuning (SFT) and reinforcement learning (RL), using training-side signals to set each sample's balance. Such signals characterize the anchor data and policy, but not whether the emphasized behavior helps in the target task and evaluation setting. This mismatch can reactivate and amplify pretrained behavior even after its literal form is removed from supervision. We demonstrate this with DIAA (Dual-Implicit Adversarial Amplification), a per-sample $σ$ -weighted SFT/RL hybrid. With all literal Python fences removed from its anchor, DIAA emits unexecuted Tool-Integrated Reasoning (TIR) on $29%$ of MATH-500 prompts, three times vanilla GRPO. Paired interventions show $10.6$ percentage points of accuracy recovery when TIR is suppressed and comparable contributions from SFT imitation and $σ$ -amplification; a student trained on a $0%$ -TIR teacher anchor emits it at $45.8%$ . Intervention-Certified Amplification (ICA) gates $σ$ with paired-rollout target evidence. Frozen and online ICA reduce TIR to $13.8%$ and $11.2%$ , halve residual drag, improve accuracy from $47.5%$ to $48.7%$ and $49.3%$ , and transfer to held-out GSM8K without retuning.

Figures & tables

Panel of Figure 1 — Figure 1: Certify a behavior before amplifying it. $\sigma$ -weighted post-training selects one channel out of the pretrained behavior repertoire and amplifies it; an intervention-derived certificate gates that channel. The channel column on the left is a conceptual depiction and carries no measurement; only the highlighted channel $f$ is instantiated experimentally, as unexecuted Python-fence form (TIR). Paired drag is the accuracy recovered on matched prompts when an intervention removes $f$ ; after gating it is residual, so lower is better. Certified values are the frozen and online instantiations of the same gate rather than a ranking; Tab. 3 gives the full comparison, including the ungated baseline and non-certificate controls.

Checkpoint	Acc	TIR	RFT-TIR
vanilla GRPO	44.4	9.2	n/a
SFT (self)	40.6	17.2	26.5
DIAA (raw)	41.2	44.2	26.5
DIAA (clean)	47.5	29.0	0
SFT (teacher)	37.6	45.8	0

Table 1: MATH-500 triangulation (all columns in %). DIAA amplifies TIR even after literal Python fences are removed; a clean teacher RFT (

0\%

TIR) still yields

45.8\%

student TIR, indicating pretrained-channel reactivation rather than literal RFT imitation (App. C.7 , C1).

	Positive intervention $\tau_{+}$			Generic intervention $\tau_{g}$
Checkpoint	$(T,N)\ n$	$\Delta_{TN}$	$\Delta_{NN}$ ctrl	$(T,N)\ n$	$\Delta_{TN}$	$\Delta_{NN}$ ctrl
$R_{03}$ vanilla GRPO	45	$+2.2$ pp	$+0.4$ pp	44	$+2.3$ pp	$+0.2$ pp
$R_{15}$ pure SFT (self)	71	$+8.5$ pp	$+2.0$ pp	59	$+8.5$ pp	$+3.9$ pp
$R_{14v3}$ DIAA, raw	140	$+10.7$ pp	$-2.2$ pp	100	$+8.0$ pp	$+0.0$ pp
$R_{14v4}$ DIAA, clean	113	$+10.6$ pp	$-1.2$ pp	91	$+13.2$ pp	$-0.6$ pp
$R_{15b}$ pure SFT (teacher)	170	$+7.6$ pp	$-0.4$ pp	116	$+10.3$ pp	$+1.9$ pp

Table 2: Paired rescue, read diagnostically :

\Delta_{TN}

is the accuracy recovered on the rescued cell when the intervention removes TIR, and

\Delta_{NN}

the never-TIR control, so a larger

\Delta_{TN}

means more removable drag is present.

\hat{C}_{\phi}=\Delta_{TN}-\Delta_{NN}

rises from

+1.8

pp (GRPO) to

+6.5

pp (SFT) to

+11.8

pp (DIAA), decomposing drag into comparable SFT-contamination and

\sigma

-amplification contributions. The clean teacher anchor also gives positive/generic certificates (

+8.1/+8.5

pp), which reduces concern about a purely suffix-specific artifact. After mitigation the same quantity is read in the opposite direction (Sec. 6 ).

Run	Role	Acc	TIR	$\Delta_{TN}$
$R_{03}$	GRPO baseline	44.4	9.2	+2.2
$R_{14v4}$	ungated DIAA	47.5	29.0	+10.6
$R_{22}$	hard-filtered RFT	44.6	33.0	+9.0
$R_{21}$	shuffled certificate	46.1	24.5	+8.1
$R_{24}$	length heuristic	46.4	23.0	+7.9
$R_{20}$	ICA , frozen $c$	48.7	13.8	+4.9

Table 3: MATH-500 at step

1200

. For

R_{03}

and

R_{14v4}

the

\Delta_{TN}

column is diagnostic (drag present); for the remaining rows it is residual drag left after the intervention, where lower is better.

R_{22}

removes flagged RFT samples outright instead of downweighting them;

R_{21}

rules out generic regularization;

R_{24}

rules out length-based confounds.

Arm	Acc	TIR	$\Delta_{TN}$
ungated DIAA	67.9	24.7	+12.4
ICA, frozen $c$	70.7	9.3	+6.4
ICA, online $c$	70.4	9.8	+6.1

Table 4: GSM8K test (

n{=}1319

), certificate frozen from MATH-500 and not retuned.

\Delta_{TN}

is residual drag, where lower is better.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Certify-token mechanism. At a certificate event, paired rollouts update $\theta_{c}$ from $\Delta$ acc on $\phi$ -activating prompts; the updated token gates subsequent steps via Eq. ( 2 ).

RL Post-Training

Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.

See all 426 papers

TailSFT: Filtered Fine-Tuning Improves Post-Training Performance
Aug 26, 2026Sadhika Malladi, Samy Jelassi, Dylan Foster +2Supervised Fine-Tuning Instruction-Tuning Data Selection
University of California San Diego · Microsoft Research NYC
From Rollouts to Recipes: Self-Contained Post-Training for LLMs
Sep 1, 2026Yifei Li, Lingling Zhang, Muye Huang +3Self-Training for Language Models RL for Language Model Reasoning
School of Computer Science and Technology, Xi’an Jiaotong University, China · Shaanxi Province Key Laboratory of Big Data Knowledge Engineering · Zhongguancun Academy, Beijing, China +1
Understanding the Synergy between SFT, RLVR, and OPD in LLM Post-Training
Sep 25, 2026Emre Can Acikgoz, Yang Li, Zeyu Leo Liu +4LLM Post-Training Language Model Distillation
Salesforce AI Research · UIUC

Aug 26, 2026J/K move · Enter open · S save

TailSFT: Filtered Fine-Tuning Improves Post-Training Performance

Sadhika Malladi, Samy Jelassi, Dylan Foster, Jordan T. Ash+1

University of California San Diego · Microsoft Research NYC

Supervised Fine-Tuning Instruction-Tuning Data Selection RL Post-Training RL for Language Model Reasoning

Open paper AskPDF arXiv

Reinforcement learning post-training drives reasoning and agentic capabilities in modern AI systems, yet a growing body of work shows that it is most effective when used to fine-tune an already capable base model. We question whether existing pipelines yield models that are most suitable for reinforcement learning. Building on prior work highlighting the role of coverage and pass@K as predictors of post-RL performance, we design a simple modification to supervised fine-tuning, TailSFT, which filters out already fit sequences during training, thereby focusing learning on under-modeled regions, or the tail, of the data distribution. We justify and validate the design choices in TailSFT, particularly the specific filtering criteria, through a combination of controlled experiments and theoretical analysis. On OLMo-3 7B, TailSFT often improves pass@16 performance on math and coding evaluations, with gains up to 17% absolute, while incurring minimal computational overhead. These higher-coverage checkpoints consistently translate to up to 4% absolute pass@1 gains in subsequent GRPO runs, demonstrating that TailSFT checkpoints serve as better initializations for RL. We further introduce a lightweight diagnostic for identifying settings where TailSFT is most likely to help. More broadly, our results motivate a principled, stage-aware approach to model development, in which intermediate checkpoints are judged by how effectively they support subsequent training.

Abstract

RL Post-Training

Explore similar work

TailSFT: Filtered Fine-Tuning Improves Post-Training Performance

From Rollouts to Recipes: Self-Contained Post-Training for LLMs

Understanding the Synergy between SFT, RLVR, and OPD in LLM Post-Training