On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning

Yu Li; Yufa Zhou; Tian Lan

cs.LGOct 29, 2026

On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning

Authors: Yu Li, Yufa Zhou, Tian Lan

Organizations: George Washington University · Duke University

Abstract

On-policy reinforcement learning for large language models assumes that training responses are sampled from the current policy. In practice, an inference engine samples the responses and a separate trainer computes the gradients, and the training-inference mismatch between the two makes the training data off-policy. The mismatch has two sources, the numerics of the engine and the stale weights of asynchronous sampling, and a correction that stabilizes training under one source can fail under the other. We ask which property of a mismatch source decides the correction it needs. Evaluating the trainer at the sampler's weights splits each sampled-token error exactly into a numerical discrepancy of the engine and a policy lag of the trainer, and we follow both through training. The numerical discrepancy stays approximately constant, so it biases the updates by a nearly fixed amount, and centering the score at each prefix removes its leading bias. The policy lag is produced by the trainer's own updates: the bias it leaves enters the next update, which enlarges the next lag and closes a feedback loop. Importance weighting cuts this loop only while the lag ratio stays concentrated, and on Qwen3-1.7B it fails at the step where the effective sample size of the ratio falls below one half. From this account we derive the discrepancy-lag correction (DLC), which centers the numerical discrepancy, clips the lag ratio against the sampler's weights and resynchronizes the sampler when this effective sample size falls below one half. Under a stale sampler, DLC keeps the KL divergence from on-policy training within twice the KL between two independent on-policy trainings on Qwen3-0.6B and Qwen3-1.7B, and its resynchronization prevents collapses that the per-token corrections leave and that a KL measured on the sampler's prefixes does not detect.

Figures & tables

Panel of Figure 1 — Figure 1: Origin of the training-inference mismatch and the correction built on it. The trainer’s forward pass at the sampler’s weights splits the mismatch exactly into the numerical discrepancy and the policy lag, and the gradient estimator receives a constant offset from the first and a growing lag ratio from the second. DLC subtracts the offset, clips the lag ratio inside the loop through the update and resynchronizes the sampler when the effective sample size of the lag ratio falls below one half (a) . Under a stale sampler, DLC keeps Qwen3-1.7B at the noise floor through step 40, while importance sampling leaves the floor once its effective sample size falls below one half at step 11 (b) . The three DLC trainings resynchronize the sampler when the effective sample size of their lag ratio falls below one half, and the effective sample size of importance sampling stays below one half after step 11 (c) .

Mismatch	Source	Correction
Infrastructure layer
Weight version	asynchronous rollout	synchronization, bounded staleness ( Fu et al., 2025 ; Piché et al., 2025 )
Engine numerics	low precision, kernels	batch-invariant kernels ( He and Thinking Machines Lab, 2025 ) , FP16 training ( Qi et al., 2025 )
Sampling and reporting	top-p, processed log-probabilities	report the drawn distribution
Sequence identity	retokenization	pass token ids to the trainer
Response selection	filtering, partial rollouts	outcome-independent batching

Table 1: Training-inference mismatch and its corrections in the two layers.

Reference
	Quantized sampler			Stale sampler			Quantized and stale
	Deviation at step			Deviation at step			Deviation at step
Correction	20	40	Accuracy	20	40	Accuracy	20	40	Accuracy
Initial model $\theta_{0}$	–	–	8.2	–	–	8.2	–	–	8.2
On-policy training	1.00	1.00	46.5	1.00	1.00	46.5	1.00	1.00	46.5
Baselines

Table 2: The corrections under each mismatch source on Qwen3-0.6B. The deviation is the KL to on-policy training as a multiple of the noise floor at steps 20 and 40, and lower values are better. Test accuracy is in percent at step 40, the last training step, and higher values are better. PPO uses the trainer before each batch as its reference, and DLC is the correction of Section 7 . Among the baselines and DLC, the best value in each column is in bold and the second best is underlined, and a dash marks a correction without a run under that source.

Figure 2: Deviation of the corrections and the growth of the discrepancy and the lag. The colour of each cell gives the deviation of a Qwen3-0.6B run as a multiple of the noise floor on a log scale, averaged over the three runs, and Table 2 lists the values at step 40. Without a policy lag DLC reduces to score centering, so DLC has a run only under the two stale sources. Under quantization the corrections stay near the floor (a), whereas under staleness score centering leaves it and importance sampling with centering and DLC stay closest to it (b). The sequence offset stays constant under quantization and grows under staleness (c). The numerical discrepancy of the quantized sampler stays near 0.02 nats per token while the policy lag of a stale sampler grows by two orders of magnitude on both model sizes (d). With both sources combined, the numerical discrepancy keeps the size it has under quantization alone, and the lag reaches the stale value by step 40 (e). The accumulated update of the combined source follows that of the stale runs, with the hatched bars for the combined source and the vertical lines spanning the three runs (f).

Figure 3: Measurements of the three links of the loop. We measure the three links under a stale sampler on Qwen3-0.6B. The mean lag grows with

\|\Delta\|^{2}

at the same rate under score centering and importance sampling, inside the grey band of 110 to 150 times

\|\Delta\|^{2}

(a), and tokens with negative advantage carry the larger lag only under score centering (b). The residual bias of score centering grows linearly in

\|\Delta\|

while that of importance sampling stays at the gradient noise (c). The predicted bias aligns with the measured one above the gradient noise (d), the update depends on the learning rate only through

\eta k

(e), and the excess update norm of score centering over importance sampling follows the quadratic loop fit (f).

Figure 4: Validity range of importance weighting under staleness. Fewer tokens fall below a ratio of 0.5 than a Gaussian lag predicts (a). On Qwen3-1.7B the effective sample size of importance sampling therefore drops below one half at step 11, three steps before the Gaussian value, with the purple and grey dotted lines at the two crossings (b), and the deviation of importance sampling leaves the floor from step 11 on (c).

Method	Per-token gradient	Ratio denominator	Centering	Adaptive control
Corrections at each token
Policy gradient	$A_{t}\,s_{t}$	–	✗	–
Importance sampling	$w_{t}\,A_{t}\,s_{t}$	$q$	✗	–
Score centering	$A_{t}\,(s_{t}-\bar{s}_{t})$	–	✓	–
Importance sampling with centering	$A_{t}\,\big(w_{t}\,s_{t}-\sum_{v}q_{v}w_{v}s_{v}\big)$	$q$	✓	–
Importance sampling with resynchronization	$w_{t}\,A_{t}\,s_{t}$	$q$	✗	$1/\mathbb{E}_{q}[\rho_{t}^{2}]<\tfrac{1}{2}$

Table 3: Per-token gradients of the corrections. The ratio denominator decides which component of the sampled-token error enters the weight, with

q

for the discrepancy and the lag together and

p_{\theta_{\mathrm{old}}}

for the lag alone. Here

w_{t}=\min(p_{\theta}/q,\,2)

\rho_{t}=p_{\theta}/p_{\theta_{\mathrm{old}}}

r_{t}=p_{\theta}/p_{\theta_{\text{batch}}}

for the trainer before each batch, and

m_{t}

and

m^{\text{b}}_{t}

are the clip indicators of Equation 6 on

\rho_{t}

and

r_{t}

. The M2PO baseline masks tokens by

m^{\mathrm{M2}}_{t}

until the second moment of

\rho_{t}

is at most 0.04. VCPO weights response

i

\bar{w}_{i}=\min\big(\prod_{t^{\prime}}p_{\theta}/q,\,8\big)

and replaces its advantage by

\hat{A}_{i}

with an off-policy baseline. In the last column the sampler resynchronizes when the effective sample size

1/\mathbb{E}_{q}[\rho_{t}^{2}]

falls below one half, VCPO scales the learning rate by the inverse root of the second moment of

\bar{w}_{i}

, and

K

counts the updates per batch.

Reference
	Qwen3-0.6B					Qwen3-1.7B
	Deviation at step					Deviation at step
Correction	10	20	30	40	Accuracy	10	20	30	40	Accuracy
Initial model $\theta_{0}$	–	–	–	–	8.2	–	–	–	–	33.4
On-policy training	1.0	1.0	1.0	1.0	46.5	1.0	1.0	1.0	1.0	57.6
Baselines

Table 4: DLC and the baseline corrections under a stale sampler. The deviation is the KL to on-policy training as a multiple of the noise floor at steps 10, 20, 30 and 40, and lower values are better. Test accuracy is in percent on the Countdown test problems at step 40, and higher values are better. The grey rows give the initial model

\theta_{0}

and on-policy training. Among the baselines and DLC, the best value in each column is in bold and the second best is underlined, and a dash marks a correction without a run on that model.

Figure 5: DLC against importance sampling, importance sampling with resynchronization and DLC without resynchronization on Qwen3-1.7B under staleness. DLC stays at the floor and resynchronization brings importance sampling back near it (a), since each resynchronization, marked by a triangle in the row of its run, restores the effective sample size (b). The DLC models track on-policy training in test accuracy, whereas two runs of DLC without resynchronization and one of importance sampling with resynchronization fall below $\theta_{0}$ (c). The collapses without resynchronization coincide with a jump of the trainer’s mass on tokens to which $\theta_{0}$ gives probability below $10^{-3}$ , and this mass stays low under DLC (d).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning	Symbol	Meaning
$q$	sampler distribution	$s_{t}$ , $A_{t}$	score and advantage of the sampled token
$p_{\theta}$	trainer distribution at weights $\theta$	$a$ , $s$	expected advantage and score as functions of the token
$\theta_{0}$	initial weights	$\bar{a}$ , $\bar{s}$	sampler means $\mathbb{E}_{q}[a]$ and $\mathbb{E}_{q}[s]$ at a prefix
$\theta_{\mathrm{old}}$	sampler’s weights	$\bar{A}_{t}$	on-policy mean $\mathbb{E}_{p}[a]$ at prefix $h_{t}$
$\Delta$	accumulated update $\theta-\theta_{0}$	$H$ , $\hat{s}_{t}$	top-128 head and the head estimate of $\bar{s}_{t}$
$\delta_{t}$	sampled-token error $\log p_{\theta}-\log q$	$\kappa$	ratio of the tail masses

Appendix

Table 5: Notation. Quantities with index

t

belong to the sampled token

y_{t}

at prefix

h_{t}

, and expectations

\mathbb{E}_{q}

run over the sampler’s responses.

	Quantized sampler		Stale sampler		Quantized and stale
Correction	Ratio of means	Mean of ratios	Ratio of means	Mean of ratios	Ratio of means	Mean of ratios
Policy gradient	12.7	16.4	345	516	107	–
PPO w/o clip	336	429	534	718	–	–
PPO	14.3	19.5	699	804	–	–
Importance sampling	0.74	1.09	4.50	6.31	3.91	4.79
Score centering	0.80	1.22	31.7	45.1	28.1	35.7

Appendix

Table 6: Deviation at step 40 on Qwen3-0.6B under the two statistics. The ratio of means is the value of the text and of Table 2 , and the mean of ratios gives the same order of the corrections up to the swap of DLC and importance sampling with centering under the stale sampler. Dashes mark corrections without a run or without per-run values under that source.

Correction	Generation	Forward at the sampler’s weights	Update	Step	Copies in 40 steps
Reference
On-policy training	6.5	0.1	37.1	50.4	40
Corrections
Score centering	14.7	0.1	42.1	62.2	0
Importance sampling	6.7	0.1	40.8	53.2	0
Importance sampling with resynchronization	7.0	0.1	39.5	52.7	3.7

Appendix

Table 7: Wall-clock time per training step on Qwen3-1.7B under a stale sampler. Times are seconds per step averaged over 40 steps and three runs, and the step time also contains the advantage computation and the bookkeeping of verl. The last column counts the weight copies to the sampler in 40 steps.

Reference
Correction	Deviation	Accuracy
Initial model $\theta_{0}$	–	35.4
On-policy training	1.0	55.3
Baselines
Policy gradient	6470	0.0
PPO	1100	20.7

Appendix

Table 8: Corrections on Qwen3-8B under a stale sampler at step 40. The deviation is the KL to on-policy training as a multiple of the noise floor, and the accuracy is the test accuracy in percent. Each row is one run.

Reference
Correction	Deviation	Accuracy	Step time
Initial model $\theta_{0}$	–	33.4	–
On-policy training	1.0	56.8	–
Baselines
Policy gradient	188	0.0	51.8
Importance sampling	1.4	52.9	43.7

Appendix

Table 9: Corrections on Qwen3-1.7B under continuous asynchrony at step 40. The sampler receives the weights after each step with a maximum version gap of four. The deviation and the accuracy are as in Table 8 , and the step time is the mean wall-clock time per step in seconds. Each row is one run.

		Cosine at step		Norm ratio at step
Correction	Weights	20	40	20	40
Stale sampler
Score centering	linearized	0.55	0.16	136	3399
Score centering	exact	0.28	0.09	4.1	6.1
Importance sampling	linearized	0.41	0.44	23	93
Importance sampling	exact	0.15	0.15	7.7	7.9

Appendix

Table 10: Predicted against measured residual bias with linearized and exact weights. The cosine to the measured bias and the norm ratio

\|b_{\mathrm{pred}}\|/\|b\|

are means over three runs. The exact weights reduce the norm ratio from up to 3400 to between 1.9 and 7.9, and the cosine stays above the gradient noise of at most 0.07.

Training-Inference Mismatch

See all 15 papers

May 14, 2026

Diagnosing Training Inference Mismatch in LLM Reinforcement Learning via a Zero-Mismatch Reference

Modern LLM RL systems separate rollout generation from policy optimization. These two stages are expected to produce token probabilities that match exactly. However, implementation differences can make them assign different values to the same sequence under the same model weights, inducing Training-Inference Mismatch (TIM). TIM is difficult to inspect because it is entangled with off-policy drift and common stabilization mechanisms. In this work, we isolate TIM in a zero-mismatch diagnostic setting (VeXact), and show that small token-level numerical disagreements can independently cause training collapse. We further show that TIM changes the effective optimization problem, and identify a set of remedies that could mitigate TIM. Our results suggest that TIM is not benign numerical noise, but a systems-level perturbation that should be treated as a first-order factor in analyzing LLM RL stability.

Tianle Zhong, Neiwen Ling, Yifan Pi +5

ByteDance · The University of Virginia

Training-Inference Mismatch Reinforcement Learning LLM Inference RL Stability RL Post-Training

Ask PDF

Jun 28, 2026

The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning

Reinforcement learning (RL) has gained growing attention in large language model (LLM) post-training, yet RL training remains fragile and can suffer from instability or collapse. One vital cause is training-inference mismatch: LLM adopts separate inference and training engines for generation efficiency and training precision, which in practice exhibits inconsistent probabilities for the same trajectories on training and inference sides, even with synchronized model parameters. This naturally induces a special type of off-policyness ever existing and poisoning the training. Prior works have made various efforts in addressing the off-policyness to stabilize the training policies under the mismatch. In this paper, we point out the objective misalignment neglected by existing works that an effective update to the policy in the training engine not necessarily ensures the improvement of the inference policy, i.e., the one used in deployment. To this end, we propose a new policy optimization objective for LLM RL, named Monotonic Inference Policy Improvement (MIPI). Following this principle, we introduce Monotonic Inference Policy Update (MIPU), a two-step LLM RL framework that constructs sampler-referenced candidate updates and selectively accepts synchronized candidates using an inference-side gap proxy. Experiments conducted on two model scales under high mismatch show that MIPU improves average reasoning performance and training stability.

Jing Liang, Hongyao Tang, Yi Ma +9

1Tianjin · 2Alibaba

Reinforcement Learning RL for Language Model Reasoning Policy Optimization RL Post-Training Off-Policy RL

Ask PDF

Sep 26, 2026

Rethinking Training-Inference Mismatch in LLM Reinforcement Learning: Where It Arises and How to Correct It

We study training-inference mismatch in reinforcement learning with verifiable rewards (RLVR) for large language models, where rollouts are sampled by an inference engine while gradients are computed by a training engine, and the two engines assign different probabilities to the same tokens. To account for this discrepancy in policy updates, we introduce calibrated importance sampling (CIS). CIS is motivated by an empirically supported logit-displacement characterization that expresses the mismatch as an additive displacement

\varepsilon_t

in log-odds, determined by the per-logit perturbation before the softmax, whose distribution is approximately invariant to token confidence. This characterization motivates a confidence-aware truncation: large positive displacements are truncated at a single constant threshold, which maps back to an importance-ratio cap that tightens as token confidence increases. Theoretically, we show that CIS replaces the unbounded second moment that governs the error of exact importance sampling with a term bounded by a constant, at the cost of a bias controlled by the truncated excess. In evaluation across three mixture-of-experts models and five mathematical reasoning benchmarks, CIS achieves the highest five-benchmark average on all three models among the evaluated baselines. Diagnostic analyses show that CIS places less truncation bias on low-confidence tokens than truncated importance sampling, while upward clipping of small importance weights reduces held-out accuracy.

Tianrun Yu, Kaixiang Zhao, Shangzhe Li +4

Brigham Young University · University of North Carolina at Chapel Hill

Training-Inference Mismatch Reinforcement Learning RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards Importance Sampling

Ask PDF

Abstract

Training-Inference Mismatch

Explore similar work

Diagnosing Training Inference Mismatch in LLM Reinforcement Learning via a Zero-Mismatch Reference

The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning

Rethinking Training-Inference Mismatch in LLM Reinforcement Learning: Where It Arises and How to Correct It