Towards Understanding Pause Token Fine-Tuning Dynamics: A Mode Retention Perspective

Jaehyeon Kim; Suhwan Kim; Nakyung Lee; Yeongoon Kim; Jimin Seo; Giho Lee; Jungwoo Lee

cs.CLSep 3, 2026

Towards Understanding Pause Token Fine-Tuning Dynamics: A Mode Retention Perspective

Authors: Jaehyeon Kim, Suhwan Kim, Nakyung Lee, Yeongoon Kim, Jimin Seo, Giho Lee, Jungwoo Lee

Organizations: HodooAI Lab, Seoul, South Korea · Seoul National University, Seoul, South Korea

Abstract

Pause-token methods improve LLM reasoning by inserting special tokens into sequences. Prior work explains these gains through computational expressivity. However, there is relatively little investigation into the training dynamics of pause tokens. We explore how pause tokens reshape the training dynamics of fine-tuning. Two controlled pilots expose distinct asymmetries. On a synthetic continual-learning task, masked pauses overwrite a previously-learned distribution roughly 4x less at matched final adaptation (H1, mode retention); on a synthetic math-reasoning probe, the boundary-adjacent token comes to encode substantially more downstream-step information (H2, non-myopic compression). We formalize a training rule consistent with both - Masked Boundary Pause (MBP), pause tokens placed at reasoning-step boundaries with their loss masked. Across 1B-8B Qwen and Llama models, MBP consistently improves reasoning, achieving gains of up to 6 points on math and 2.5 points on code, while preserving general language understanding abilities. We further demonstrate that this mode-preserving strategy extend gains to GRPO. These results recast pause tokens as a training-dynamics intervention on the retention-adaptation trade-off, rather than merely an inference-time computation device.

Figures & tables

Figure 1: Comparison between standard fine-tuning and masked boundary pause fine-tuning. Standard SFT applies next-token prediction (NTP) loss at every task-token position, allowing gradients from the target token to flow directly into the preceding context representation. This can strongly adapt the model to the task-data distribution, but may also erode the rich predictive distribution learned during pre-training. Masked boundary pause fine-tuning (MBP) instead inserts pause tokens at selected boundaries and masks their loss, thereby blocking the direct gradient path into the boundary context state. The model receives update signals only indirectly through neighboring token predictions, enabling task adaptation while better preserving the original pre-training distribution.

Figure 2: Phase-2 dynamics from a $D_{1}$ -pretrained backbone ( $n{=}8$ seeds, $\pm 1\sigma$ ). Top: $D_{2}$ adaptation (held-out loss, log scale). Bottom: $D_{1}$ retention (held-out loss, linear scale). Both conditions reach the same $D_{2}$ floor; Pause retains $D_{1}$ $\sim 4\times$ better than No pause .

Figure 3: Linear-probe accuracy on the boundary-adjacent token’s final hidden state ( $n{=}3$ seeds, $\pm 1\sigma$ ). Left: current step’s integer answer. Right: next step’s integer answer. Pause (unmasked) trains the same boundary pauses as prediction targets.

Model	Method	GSM8K	MATH500	MATH	AMC23	MINERVA	Avg.
Qwen3-1.7B Base	Base	65.13	54.20	54.82	35.00	17.65	45.36
	SFT	82.41	63.40	62.98	27.50	20.59	51.38
	MBP	83.02	61.00	64.56	37.50	23.16	53.85
Qwen3-4B Base	Base	82.56	66.00	67.64	37.50	28.68	56.48
	SFT	91.36	69.20	71.24	45.00	29.78	61.32
	MBP	90.07	73.40	72.32	50.00	30.51	63.26

Table 1: Math-domain results. Average denotes the macro-average over GSM8K, MATH500, MATH, AMC23, and MINERVA.

Model	Method	MBPP	MBPP+	HumanEval	HumanEval+	Avg.
Qwen3-1.7B Base	Base	69.05	59.26	58.54	53.05	59.98
	SFT	69.58	62.43	67.07	61.59	65.17
	MBP	73.02	62.43	70.12	65.24	67.70
Qwen3-4B Base	Base	74.34	64.81	76.83	72.56	72.14
	SFT	78.04	67.99	79.27	73.78	74.77
	MBP	79.10	65.08	82.93	78.05	76.29

Table 2: Code-domain results. Average denotes the macro-average over MBPP, MBPP+, HumanEval, and HumanEval+.

Trained Domain	Method	MMLU	GPQA	HellaSwag	BBH	PIQA	Avg.
Math	Base	62.95	27.46	38.69	38.84	62.13	46.01
	SFT	62.32	24.78	40.39	42.06	66.10	47.13
	MBP	63.93	27.23	48.32	44.13	67.63	50.25
Code	Base	62.95	29.91	38.00	38.84	62.13	46.37
	SFT	60.19	23.88	36.44	39.15	60.66	44.06
	MBP	63.78	25.67	44.01	42.88	68.01	48.87

Table 3: General-domain results on Qwen3-1.7B-Base. Average denotes the macro-average over MMLU, GPQA, HellaSwag, BBH, and PIQA.

Method	Math					Avg.
Method	GSM8K	MATH500	MATH	AMC23	MINERVA	Avg.
Base	65.13	54.20	54.82	35.00	17.65	45.36
SFT	82.41	63.40	62.98	27.50	20.59	51.38
Append	79.68	59.20	60.96	30.00	22.79	50.53
Random	81.35	60.80	63.26	20.00	23.16	49.71
DIT	68.01	57.20	56.38	30.00	20.22	46.36

Table 4: Method ablation results on Qwen3-1.7B-Base. The upper block reports math benchmarks, and the lower block reports code benchmarks.

Figure 4: Training dynamics of MBP-GRPO. Columns correspond to Qwen3-1.7B-Base (left) and Qwen3-4B-Base (right), while rows show GSM8K validation pass@1 (top), training reward (middle), and mixed-group ratio (bottom). Red denotes MBP-GRPO and blue denotes standard GRPO. MBP-GRPO exhibits a higher mixed-group ratio in the early stage of training, providing more useful update signals and leading to faster convergence than standard GRPO.

Method	GSM8K	MATH	AMC23	Minerva	AUC
Qwen3-1.7B-Base
Standard	80.52	59.68	37.50	20.96	0.366
MBP	81.35	61.24	40.00	21.32	0.414
Qwen3-4B-Base
Standard	92.19	74.30	50.00	30.15	0.505
MBP	91.58	75.26	57.50	23.90	0.550

Table 5: Final GRPO pass@1 accuracy (%) on GSM8K, MATH, AMC23, and Minerva, together with normalized training-reward AUC, for Qwen3-1.7B-Base and Qwen3-4B-Base. Full results are reported in Appendix Table 19 .

Metric	Method	GSM8K	MATH	Overall
BCP $\uparrow$	Standard	92.08	88.33	89.18
BCP $\uparrow$	MBP	93.36	89.09	90.00
NSR $\uparrow$	Standard	63.04	33.42	38.20
NSR $\uparrow$	MBP	66.74	36.83	41.11
Acc. $\uparrow$	Standard	81.96	63.52	67.14
Acc. $\uparrow$	MBP	84.08	65.48	68.87

Table 6: BCP(Base-Correct Preservation), NSR(New-Solve Rate), and final accuracy of Standard-SFT and MBP on Qwen3-1.7B-Base.

Step boundaries
	OOD		In-domain
Model	MMLU	GPQA	GSM8K	MATH
Base	0.707	0.607	0.500	0.579
SFT	0.403	0.380	0.278	0.339
MBP	0.463	0.440	0.365	0.380
All tokens

Table 7: Next-token vocabulary entropy on Qwen3-1.7B-Base rollouts under the Base, math-SFT, and MBP checkpoints, measured at step boundaries and across all response tokens. MMLU and GPQA are out of domain; GSM8K and MATH are in domain.

Dataset	SFT	MBP
GSM8K	1.43	6.31
MATH	2.64	4.84
DeepMath Eval	4.67	8.29

Table 8: Base-referenced normalized MI proxy between context-conditioned step-boundary tokens and final answers. Results use macro aggregation over the full evaluation sets.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

After	No pause	Pause	Gap
$D_{2}$	$5.76\pm 0.56$	$2.74\pm 0.81$	$+3.01\pm 0.75$
$D_{3}$	$8.60\pm 1.44$	$4.62\pm 0.46$	$+3.98\pm 1.59$
$D_{4}$	$8.83\pm 0.16$	$6.04\pm 0.84$	$+2.80\pm 0.97$

Appendix

Table 9: Held-out

D_{1}

loss (nats, mean

\pm 1\sigma

n=8

seeds) after each stage of the

D_{1}\to D_{2}\to D_{3}\to D_{4}

chain. Lower is better; Gap is No pause

-

Pause , so positive values favor masked pauses.

Hyperparameter	SFT on math
Learning rate	Qwen: $5\times 10^{-6}$ ; Llama3.2: $1\times 10^{-5}$ ; Llama3.1: $5\times 10^{-5}$
Epochs	3
Batch size	32
Optimizer	FusedAdam
Scheduler	CosineAnnealingLR
Warmup ratio	0.03

Appendix

Table 10: SFT hyperparameters for mathematical reasoning training.

Hyperparameter	SFT on code
Learning rate	Qwen3-1.7B-Base: $5\times 10^{-7}$ ; Qwen3-4B-Base: $5\times 10^{-6}$ ; Qwen3-8B-Base: $5\times 10^{-5}$ ; Llama3.1-8B-Instruct: $1\times 10^{-8}$
Epochs	3
Batch size	32
Optimizer	FusedAdam
Scheduler	CosineAnnealingLR
Warmup ratio	0.03

Appendix

Table 11: SFT hyperparameters for code generation training.

Hyperparameter	GRPO
Learning rate	$5\times 10^{-7}$
Episodes	1
Batch size	128
Rollouts per prompt	8
Optimizer	FusedAdam
Scheduler	CosineAnnealingLR

Appendix

Table 12: GRPO hyperparameters.

Statistic	Train	Eval	Total
Examples	83,438	4,391	87,829
Full SFT tokens	44.14M	2.36M	46.51M
Pause tokens	1.34M	71.72K	1.41M
Pause-token ratio	3.02%	3.03%	3.03%
Pauses/sample	16.003	16.332	16.020

Appendix

Table 13: Statistics of the step-aligned pause dataset.

Method	Math example
MBP	Question. Round each number to the nearest tenth and add the results: 543.55, 25.23, and 299.98. Answer. <\|PAUSE\|> First, let’s round each number to the nearest tenth: <\|PAUSE\|> 1. 543.55 rounds to 543.6 (since the hundredths place is 5). <\|PAUSE\|> 2. 25.23 rounds to 25.2 (since the hundredths place is less than 5). <\|PAUSE\|> 3. 299.98 rounds to 300.0 (since the hundredths place is 8). <\|PAUSE\|> Now, let’s add the rounded numbers: <\|PAUSE\|> 543.6 + 25.2 + 300.0 = 868.8. <\|PAUSE\|> So, the final answer is $\boxed{868.8}$ .
append	Question. Round each number to the nearest tenth and add the results: 543.55, 25.23, and 299.98. <\|PAUSE\|><\|PAUSE\|><\|PAUSE\|> <\|PAUSE\|><\|PAUSE\|><\|PAUSE\|><\|PAUSE\|> Answer. First, let’s round each number to the nearest tenth: 1. 543.55 rounds to 543.6 (since the hundredths place is 5). 2. 25.23 rounds to 25.2 (since the hundredths place is less than 5). 3. 299.98 rounds to 300.0 (since the hundredths place is 8). Now, let’s add the rounded numbers: 543.6 + 25.2 + 300.0 = 868.8. So, the final answer is $\boxed{868.8}$ .
random	Question. Round each number to the nearest tenth and add the results: 543.55, 25.23, and 299.98. Answer. First, let’s round each number to the nearest tenth: 1. 543.55 rounds to 5 <\|PAUSE\|> 4 <\|PAUSE\|> 3.6 (since the hundredths place is 5). 2. 25.23 rounds to 25. <\|PAUSE\|> 2 (since the hundred <\|PAUSE\|> ths place is less than 5). 3. 299.98 rounds to 3 <\|PAUSE\|> 00.0 (since the hundredths place is 8). Now, let’s add the rounded numbers: 543.6 + 25.2 + 300.0 = 868.8. So, the final <\|PAUSE\|> answer is $\backslash$ boxed <\|PAUSE\|> {868.8}.
dit	Question. Round each number to the nearest tenth and add the results: 543.55, 25.23, and 299.98. Answer. <\|PAUSE\|> First, <\|PAUSE\|> let’s round each number to the nearest tenth: <\|PAUSE\|> 1. 543.55 rounds to 543.6 (since the hundredths place is 5 <\|PAUSE\|> ). 2. 25.23 rounds to 25.2 (since the hundredths place is <\|PAUSE\|> less than 5). 3. 299.98 rounds to 300.0 (since the hundredths place is 8). Now, let’s add the rounded numbers: 543.6 + 25.2 + 300.0 = 868.8. So, the <\|PAUSE\|> final answer is <\|PAUSE\|> $\boxed{868.8}$ .

Appendix

Table 14: Examples of pause-token placement for the math task.

Method

Code example

MBP

Question. Ninja Atsushi stands guard atop the Ninja Building, which consists of two adjacent buildings of equal height. Each building has a series of walls of varying heights. Atsushi must jump between buildings to patrol the rooftops. Given the heights of the walls for both buildings, determine the minimum height Atsushi must jump for each wall position. Input: two lists of integers, heights_building_1 and heights_building_2 , with the same length. Output: a list of integers where each element is the minimum height Atsushi must jump for each wall position. Constraints: the common list length is between 1 and

10^{5}

, and each wall height is between 1 and

10^{9}

. Sample input: heights_building_1 = [3, 6, 2] , heights_building_2 = [5, 4, 8] . Sample output: [5, 6, 8] . Answer. <|PAUSE|> def min_jumps(heights_building_1, heights_building_2): <|PAUSE|> Calculate the minimum height Atsushi must jump for each wall position between two buildings. <|PAUSE|> :param heights_building_1: list of integers representing the heights of walls in the first building. <|PAUSE|> :param heights_building_2: list of integers representing the heights of walls in the second building. <|PAUSE|> :return: list of integers representing the minimum jump heights for each wall position. <|PAUSE|> if len(heights_building_1) != len(heights_building_2): <|PAUSE|> raise ValueError("Both buildings must have the same number of walls."). <|PAUSE|> return [ max(heights_building_1[i], heights_building_2[i]) for i in range(len(heights_building_1)) ].

append

10^{5}

, and each wall height is between 1 and

10^{9}

. Sample input: heights_building_1 = [3, 6, 2] , heights_building_2 = [5, 4, 8] . Sample output: [5, 6, 8] . <|PAUSE|><|PAUSE|> <|PAUSE|><|PAUSE|><|PAUSE|><|PAUSE|><|PAUSE|><|PAUSE|> <|PAUSE|><|PAUSE|> Answer. def min_jumps(heights_building_1, heights_building_2): Calculate the minimum height Atsushi must jump for each wall position between two buildings. :param heights_building_1: list of integers representing the heights of walls in the first building. :param heights_building_2: list of integers representing the heights of walls in the second building. :return: list of integers representing the minimum jump heights for each wall position. if len(heights_building_1) != len(heights_building_2): raise ValueError("Both buildings must have the same number of walls."). return [ max(heights_building_1[i], heights_building_2[i]) for i in range(len(heights_building_1)) ].

Appendix

Table 15: Examples of pause-token placement for the code task.

Method

Code example

random

10^{5}

, and each wall height is between 1 and

10^{9}

. Sample input: heights_building_1 = [3, 6, 2] , heights_building_2 = [5, 4, 8] . Sample output: [5, 6, 8] . Answer. def min_jumps(heights_building <|PAUSE|> _1, <|PAUSE|> heights_building_2): Calculate the minimum height Atsushi must jump for each wall position between two buildings. :param heights_building_1: list of integers representing the heights of walls in the first building. <|PAUSE|> :param heights_building_2: list of integers representing the heights of walls in the second building. :return: list of integers representing the minimum jump heights for each wall position <|PAUSE|> . if len <|PAUSE|> (height <|PAUSE|> s_building_1) != len(heights_building_2): raise ValueError("Both buildings must have the same number of walls."). return [ max(heights_building_1[i], heights_building_2 <|PAUSE|> [i <|PAUSE|> ]) for i in range <|PAUSE|> (len(heights_building_1 <|PAUSE|> )) ].

dit

10^{5}

, and each wall height is between 1 and

10^{9}

. Sample input: heights_building_1 = [3, 6, 2] , heights_building_2 = [5, 4, 8] . Sample output: [5, 6, 8] . Answer. <|PAUSE|> def min <|PAUSE|> _jumps( heights_building_1, heights_building_2): Calculate the minimum height Atsushi must jump for each wall position <|PAUSE|> between two buildings. <|PAUSE|> :param heights_building_1: list of integers representing the heights of <|PAUSE|> walls in the first building. :param heights_building_2: list of integers representing the heights of walls in the second building. :return: list of integers representing the minimum <|PAUSE|> jump heights for each wall position. <|PAUSE|> if len(heights_building_1) != len(heights_building_2): raise ValueError("Both buildings must have the same number of walls <|PAUSE|> ."). <|PAUSE|> return [ max(heights_building_1[i], heights_building_2[i]) for i in range(len(heights_building_1)) ]. <|PAUSE|>

Appendix

Table 16: Examples of pause-token placement for the code task, continued.

Qwen3-1.7B-Base
Method	GSM8K	MATH500	MATH	AIME24	AMC23	MINERVA	MMLU	GPQA	HellaSwag	BBH	PIQA	Average
Base	65.13	54.20	54.82	0.00	35.00	17.65	62.95	27.46	38.69	38.84	62.13	41.53
SFT	82.41	63.40	62.98	0.00	27.50	20.59	62.32	24.78	40.39	42.06	66.10	44.78
Append	79.68	59.20	60.96	0.00	30.00	22.79	62.93	25.67	41.53	42.62	66.59	44.72
Random	81.35	60.80	63.26	0.00	20.00	23.16	63.64	23.66	48.73	43.96	69.64	45.29
DIT	68.01	57.20	56.38	6.67	30.00	20.22	62.90	25.67	44.75	42.66	69.31	43.98

Appendix

Table 17: Additional SFT results on mathematical reasoning and general benchmarks. All entries report accuracy. Average denotes the macro-average over all reported benchmark columns.

Method	MBPP	MBPP+	HumanEval	HumanEval+	MMLU	GPQA	HellaSwag	BBH	PIQA	Average
Qwen3-1.7B-Base
Base	69.05	59.26	58.54	53.05	62.95	27.46	38.69	38.84	62.13	52.22
SFT	69.58	62.43	67.07	61.59	60.19	23.88	36.44	39.15	60.66	53.44
Append	69.84	60.58	70.73	65.24	61.53	27.90	36.58	40.65	64.09	55.24
Random	70.37	61.11	67.07	62.80	64.03	24.78	48.36	42.50	68.77	56.64
DIT	69.84	60.58	71.95	65.24	63.71	25.45	50.40	43.09	67.41	57.52

Appendix

Table 18: Additional SFT results on code generation and general benchmarks. Code benchmark entries report pass@1, while general benchmark entries report accuracy. Average denotes the macro-average over all reported benchmark columns.

Method	GSM8K	M500	MATH	AIME24	AMC23	Minerva	Avg.	AUC
Qwen3-1.7B-Base
Standard	80.52	61.20	59.68	3.33	37.50	20.96	43.87	0.366
MBP	81.35	61.80	61.24	3.33	40.00	21.32	44.84	0.414
Qwen3-4B-Base
Standard	92.19	74.40	74.30	13.33	50.00	30.15	55.73	0.505
MBP	91.58	76.00	75.26	16.67	57.50	23.90	56.82	0.550

Appendix

Table 19: Full GRPO pass@1 accuracy (%) on six mathematical reasoning benchmarks and normalized training-reward AUC for Qwen3-1.7B-Base and Qwen3-4B-Base. Average denotes the macro-average over the six benchmarks.

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 758 papers

Almost Free State Prediction Separation
Sep 3, 2026John Langford, Nathan Godey, Giovanni Monea +5Language Model Pretraining Efficient Language Model Training
Microsoft · Cornell University
Same Beliefs, Different Policies: Learning Reusable Representations for Early Stopping in LLM Reasoning
Oct 21, 2026Jingwei Ni, Dmitry Abulkhanov, Vladislav Smirnov +5Early Stopping for Language Model Reasoning Early Stopping
ETH Zürich · MBZUAI · University of Zürich +1
Extremely Sparse Supervision Incentivizes Reasoning Ability
Sep 3, 2026Zhishuai Liu, Xingzi Xu, Mehmet Saygin Seyfioglu +2On-Policy Distillation Language Model Post-Training
Amazon, Seattle, WA 98109, USA · Duke University, Durham, NC 27708, USA

Sep 3, 2026J/K move · Enter open · S save

Almost Free State Prediction Separation

John Langford, Nathan Godey, Giovanni Monea, Yoav Artzi+4

Microsoft · Cornell University

Language Model Pretraining Efficient Language Model Training Next-Token Prediction Efficient Language Model Inference Sequence Modeling

Open paper AskPDF arXiv

State--prediction separation (SPS) relieves a language model's hidden state of two competing burdens---summarizing the context and predicting the next token---by splitting the forward pass into a state stream and a prediction stream. The separation works, but it is expensive: the prediction stream is a second pass over the whole backbone, costing

\sim

1.9

\times

the pretraining FLOPs, and even more in terms of wall-clock time when using a flexible attention mask. This paper makes state--prediction separation almost free. We take the separation to its limit with a free pause token: a prediction stream that writes no keys or values at all and so rides the sequence's existing positions. It improves next-token prediction of a standard Transformer by 2-3 centinats in practice on a 1B parameter model, and because it adds no position it costs nothing at inference---no added context length, no KV cache, no decode steps, and essentially no latency, with the growth in inference flops typically irrelevant as it is not the active bottleneck on throughput. The cost is therefore entirely in training where we use four mechanisms to drive it down: a two-pass split that keeps FlashAttention kernels viable, the

w{=}0

prediction window, a shared gated FFN that evaluates one FFN per position rather than one per stream, and phasing the separation onto the tail of the run. Together these bring the overhead versus an optimized pretraining pipeline to

1.33\times

wall-clock while recovering ~94% of the gain compared to SPS, and to as low as

1.09\times

along a graceful quality/compute tradeoff. Furthermore, the FFN optimization reduces the raw flops required at inference time. The result is an isoflop, isoparameter, and isotoken improvement over standard next token trained transformers.

Abstract

LLM Fine-Tuning

Explore similar work

Almost Free State Prediction Separation

Same Beliefs, Different Policies: Learning Reusable Representations for Early Stopping in LLM Reasoning

Extremely Sparse Supervision Incentivizes Reasoning Ability