FOCUS & RePAIR: Mitigating Text Degeneration via Token-Level Guidance for Pruned Large Language Models

Junyoung Lee; Sehyeon Park; Shinhyoung Jang; Seonha Ryu; Hojeong Kim; Hyunsei Lee; Il Hong Suh; Yeseong Kim

cs.CLAug 27, 2026

FOCUS & RePAIR: Mitigating Text Degeneration via Token-Level Guidance for Pruned Large Language Models

Authors: Junyoung Lee, Sehyeon Park, Shinhyoung Jang, Seonha Ryu, Hojeong Kim, Hyunsei Lee, Il Hong Suh, Yeseong Kim

Organizations: Department of Electrical Engineering and Computer Science, POSTECH, Pohang, Republic of Korea · Department of Electrical Engineering and Computer Science, Daegu Gyeongbuk Institute of Science and Technology (DGIST), Daegu, Republic of Korea · Hanyang University, Seoul, Republic of Korea

Abstract

Pruning is a practical approach to compress large language models (LLMs), but it can amplify text degeneration, especially repetition loops, even when perplexity and task accuracy remain largely unchanged. In this work, we present a token-level analysis of this failure mode by viewing decoding as a dynamical process that enters and persists in a small set of recurrent contexts. Our analysis decomposes degeneration into loop entry risk and loop persistence, and shows that persistence is controlled by the escape mass assigned to plausible alternatives within the token sampling set. Motivated by these findings, we propose two token-level guidance objectives for post-pruning fine-tuning. FOCUS reweights distillation toward high-confidence teacher regions to suppress leakage, while RePAIR uses onset-centered positive/negative continuation pairs with a margin loss to promote plausible alternatives and prevent early commitment to repetition loops. Experiments on open-ended continuation and instruction-based generation show that both methods consistently reduce repetition and improve generation quality.

Figures & tables

Condition	Sampling	Greedy
Unpruned	5.9%	26.6%
Width pruned	12.4%	63.1%
Depth pruned	15.4%	63.7%

Table 1: Comparison of repetition rates between unpruned and pruned models. The pruned model is finetuned on the Alpaca dataset. For decoding, we employ top-

p

sampling with

p=0.9

Condition	Before	After
Sampling	5.9%	0.7%
Greedy	26.6%	10.8%

Table 2: Degeneration rates of the unpruned model before and after correcting two onset tokens.

Figure 1: Mode averaging under forward KL minimization. The student curve corresponds to the numerically optimized single-mode approximation of a multimodal teacher.

Llama-3.1-8B
Method	PPL ( $\downarrow$ )	0-shot ( $\uparrow$ )	MAUVE ( $\uparrow$ )	CREP ( $\downarrow$ )	Unique $n$ -gram
Method	PPL ( $\downarrow$ )	0-shot ( $\uparrow$ )	MAUVE ( $\uparrow$ )	CREP ( $\downarrow$ )	$n{=}3$	$n{=}4$	$n{=}5$	$n{=}6$
KD	21.69	60.39	0.61	7.3	13.28	9.99	8.05	6.79
+ UL	21.67	61.15	0.61	5.37	11.79	8.68	6.85	5.70
+ SG	21.69	60.88	0.66	7.8	12.87	9.69	7.85	6.66
+ DITTO	22.07	60.62	0.63	5.27	11.38	8.14	6.22	5.01

Table 3: Results of open-ended generation on the WikiText-103 dataset. Best per block in bold , second best underlined . CREP and Unique

n

-gram are reported as percentage values.

Llama-3.1-8B-Instruct
Method	PPL ( $\downarrow$ )	0-shot ( $\uparrow$ )	CREP ( $\downarrow$ )	EAD 1 ( $\uparrow$ )	BS* ( $\uparrow$ )
KD	25.65	62.32	1.97	0.28	0.49
+ UL	25.52	62.01	3.10	0.28	0.48
+ SG	25.53	62.30	2.23	0.28	0.49
+ DITTO	25.59	61.97	1.30	0.28	0.49
+ RePAIR	25.86	62.61	0.63	0.32	0.49

Table 4: Results of instruction-based generation on Self-Instruct dataset. Best per block in bold , second best underlined . CREP is reported as percentage values. (* BS : BERTScore )

Figure 2: FOCUS analysis. Comparison of token distributions between naive KD and FOCUS.

Method	PPL (↓)	Unique $n$ -gram
		$n{=}3$	$n{=}4$	$n{=}5$	$n{=}6$
CE	23.05	12.88	9.68	7.77	6.56
DPO	23.29	9.76	6.53	4.64	3.45
RePAIR	23.72	8.28	5.53	3.97	3.01

Table 5: Repetition comparison of DPO and RePAIR under the WikiText-103 continuation setup using Llama-3.1-8B. For a clear comparison, KD and FOCUS are not applied here, and CE is used as the baseline.

Comparison	Win (A)	Tie	Win (B)
KD vs FOCUS	40.10%	19.80%	40.10%
FOCUS vs RePAIR*	36.00%	22.60%	41.40%
UL* vs RePAIR*	34.60%	24.70%	40.70%
SG* vs RePAIR*	35.50%	24.30%	40.20%
DITTO* vs RePAIR*	37.40%	24.30%	38.30%

Table 6: LLM-as-a-judge comparison results. * indicates FOCUS applied.

Figure 3: TruthfulQA-MC2 score. X indicates that no repetition-suppression technique was applied.

Figure 4: Persistence metric across methods.

Method	MAUVE ( $\uparrow$ )	CREP ( $\downarrow$ )	Unique $n$ -gram
Method	MAUVE ( $\uparrow$ )	CREP ( $\downarrow$ )	$n=3$	$n=4$	$n=5$	$n=6$
ToDi	0.73	6.73	12.13	9.00	7.19	6.05
FOCUS	0.64	1.73	6.22	3.82	2.62	1.94
RePAIR*	0.73	0.57	3.68	1.92	1.14	0.75

Table 7: Comparison with ToDi on Llama 3.1-8B, following the same experimental setting as Table 3 . * indicates FOCUS applied.

Figure 5: Number of pairwise data for RePAIR.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Token probabilities across repetition cycles.

Figure 7: Example of pairwise data construction.

Model	Arc_c	PIQA	BoolQ	OpenQA	Hellaswag	Winogrande	Average
KD	43.26	77.58	67.52	40.00	71.05	62.90	60.39
+ UL	44.37	77.37	69.60	41.20	71.08	63.30	61.15
+ SG	43.52	77.58	69.69	40.80	71.11	62.59	60.88
+ DITTO	43.43	77.42	69.27	40.40	70.85	62.35	60.62
+ RePAIR	43.17	77.86	70.55	41.20	71.86	63.54	61.36
FOCUS	42.92	77.26	65.87	40.20	70.70	63.22	60.03

Appendix

Table 8: Zero-shot evaluation results on the Llama-3.1-8B model across six benchmarks. The best result within each block is highlighted in bold , while the second best is underlined .

Model	Arc_c	PIQA	BoolQ	OpenQA	Hellaswag	Winogrande	Average
KD	45.65	77.64	73.12	44.20	75.98	68.03	64.10
+ UL	46.59	77.75	73.15	43.80	76.05	68.35	64.28
+ SG	45.90	77.69	73.15	44.00	76.07	68.11	64.15
+ DITTO	45.48	77.64	72.91	43.20	75.74	68.11	63.85
+ RePAIR	46.59	77.75	72.78	44.20	76.80	68.03	64.36
FOCUS	45.99	77.26	73.15	43.20	74.98	67.56	63.69

Appendix

Table 9: Zero-shot evaluation results on the Llama2-13B model across six benchmarks. The best result within each block is highlighted in bold , while the second best is underlined .

Model	Arc_c	PIQA	BoolQ	OpenQA	Hellaswag	Winogrande	Average
KD	45.39	78.02	73.82	39.60	70.87	66.22	62.32
+ UL	45.39	77.75	73.18	39.00	70.84	65.90	62.01
+ SG	45.65	78.07	73.21	39.40	70.95	66.54	62.30
+ DITTO	44.37	77.97	72.14	40.00	70.78	66.30	61.93
+ RePAIR	45.48	78.13	73.67	39.80	71.71	66.85	62.61
FOCUS	45.39	78.62	73.85	42.20	71.00	66.06	62.85

Appendix

Table 10: Zero-shot evaluation results on the Llama-3.1-8B-Instruct model across six benchmarks. The best result within each block is highlighted in bold , while the second best is underlined .

Figure 8: Parameter study of our methods on open-ended generation. (a) Unique

n

-gram with varying

\beta,\gamma

values. (b) Unique

n

-gram with varying

\alpha_{1}

values.

Method	PPL ( $\downarrow$ )	0-shot ( $\uparrow$ )	MAUVE ( $\uparrow$ )	CREP ( $\downarrow$ )	Unique $n$ -gram
Method	PPL ( $\downarrow$ )	0-shot ( $\uparrow$ )	MAUVE ( $\uparrow$ )	CREP ( $\downarrow$ )	$n=3$	$n=4$	$n=5$	$n=6$
KD	32.73	52.43	0.55	10.57	16.03	12.56	10.46	9.07
FOCUS	33.05	51.92	0.76	2.20	6.60	4.01	2.73	2.03
UL*	33.18	51.93	0.74	2.30	6.75	4.17	2.93	2.24
SG*	33.18	51.99	0.70	1.53	5.67	3.37	2.26	1.66
DITTO*	34.10	52.18	0.70	1.00	5.09	2.67	1.58	1.04

Appendix

Table 11: Results of open-ended generation on the WikiText-103 dataset for Llama 3.2-3B with Shortened-LLaMA after removing 7 blocks. Best per block in bold, second best underlined. CREP and Unique

n

-gram are reported as percentage values. * indicates FOCUS applied.

Method	PPL ( $\downarrow$ )	0-shot ( $\uparrow$ )	MAUVE ( $\uparrow$ )	CREP ( $\downarrow$ )	Unique $n$ -gram
Method	PPL ( $\downarrow$ )	0-shot ( $\uparrow$ )	MAUVE ( $\uparrow$ )	CREP ( $\downarrow$ )	$n=3$	$n=4$	$n=5$	$n=6$
KD	32.47	50.76	0.48	7.30	14.58	11.18	9.20	7.93
FOCUS	31.85	51.18	0.69	1.43	6.97	4.32	3.06	2.34
UL*	31.91	51.16	0.71	1.53	6.82	4.16	2.85	2.16
SG*	31.97	51.12	0.69	0.83	6.05	3.58	2.41	1.79
DITTO*	32.59	50.90	0.61	0.50	5.16	2.74	1.61	1.03

Appendix

Table 12: Results of open-ended generation on the WikiText-103 dataset for Llama 3.2-3B with FLAP at pruning ratio 0.25. Best per block in bold, second best underlined. CREP and Unique

n

-gram are reported as percentage values. * indicates FOCUS applied.

Method	PPL ( $\downarrow$ )	0-shot ( $\uparrow$ )	MAUVE ( $\uparrow$ )	CREP ( $\downarrow$ )	Unique $n$ -gram
Method	PPL ( $\downarrow$ )	0-shot ( $\uparrow$ )	MAUVE ( $\uparrow$ )	CREP ( $\downarrow$ )	$n=3$	$n=4$	$n=5$	$n=6$
KD	23.45	57.31	0.67	6.73	13.79	10.35	8.33	7.01
FOCUS	24.09	57.57	0.78	0.87	6.65	4.10	2.82	2.10
UL*	24.33	57.61	0.72	1.50	6.71	4.26	3.03	2.33
SG*	24.20	57.47	0.80	1.13	6.19	3.82	2.68	2.03
DITTO*	24.60	58.14	0.75	0.40	5.05	2.80	1.73	1.14

Appendix

Table 13: Results of open-ended generation on the WikiText-103 dataset for Qwen 2.5-3B with LLMPruner at pruning ratio 0.25. Best per block in bold, second best underlined. CREP and Unique

n

-gram are reported as percentage values. * indicates FOCUS applied.

Method	PPL ( $\downarrow$ )	MAUVE ( $\uparrow$ )	Unique $n$ -gram
Method	PPL ( $\downarrow$ )	MAUVE ( $\uparrow$ )	$n=3$	$n=4$	$n=5$	$n=6$
LLMPruner 35%
FOCUS	26.26	0.72	5.76	3.45	2.33	1.73
+ UL	26.80	0.71	5.06	2.97	2.00	1.48
+ SG	26.39	0.73	5.21	3.10	2.13	1.59
+ DITTO	26.72	0.70	4.50	2.36	1.40	0.89

Appendix

Table 14: Results of open-ended generation on the WikiText-103 dataset for Llama 3.1-8B with LLMPruner at 35% and 45% pruning ratios. Best per block in bold, second best underlined. Unique

n

-gram is reported as percentage values. * indicates FOCUS applied.

Repetition in Language Generation

See all 23 papers

To Reason but Not to Stop: Understanding and Recovering Pruned Large Reasoning Models
Oct 14, 2026Minh-Anh Nguyen, Nguyen T. Ha Lan, Xiyan Fu +2Repetition in Language Generation LLM Reasoning
Nanyang Technological University, Singapore · VinUniversity, Vietnam
Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models
Nov 8, 2023Rocktim Jyoti Das, Mukul Ranjan, Mingjie Sun +2LVLM Pruning LLM Pruning
Mohamed bin Zayed University of AI · Carnegie Mellon University
Don't Repeat Yourself: Stopping Verbatim Loops at Sampling Time
Aug 24, 2026Philipp Emanuel Weidmann, Allen Roush, Judah Goldfeder +2Repetition in Language Generation
Thoughtworks · New York University

Oct 14, 2026J/K move · Enter open · S save

To Reason but Not to Stop: Understanding and Recovering Pruned Large Reasoning Models

Minh-Anh Nguyen, Nguyen T. Ha Lan, Xiyan Fu, Dung D. Le+1

Nanyang Technological University, Singapore · VinUniversity, Vietnam

Repetition in Language Generation LLM Reasoning Large Reasoning Models LLM Pruning Activation Steering

Open paper AskPDF arXiv

Weight pruning can degrade the accuracy of large reasoning models (LRMs), yet it remains unclear whether it damages reasoning itself. We find that much of this degradation stems from a failure to stop rather than to reason. As sparsity increases, pruned LRMs increasingly fail to terminate within the token budget, often after the correct answer has appeared in their reasoning, instead falling into repetitive loops, even though they still recognize where the dense model would stop. We further find that the hidden-state difference between terminating and looping continuations is consistent across pruned checkpoints, suggesting a shared direction associated with committing to an answer. Building on this, we propose Termination-Guided Activation Steering (TGAS), a training-free method that adds this direction to the residual stream at inference time. Across six LRMs, three pruning pipelines, and math and code benchmarks, TGAS improves termination and overall accuracy while reducing repetitive reasoning and generation length, showing that this failure is actionable.

Abstract

Repetition in Language Generation

Explore similar work

To Reason but Not to Stop: Understanding and Recovering Pruned Large Reasoning Models

Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models

Don't Repeat Yourself: Stopping Verbatim Loops at Sampling Time