LoopRPT: Reinforcement Pre-Training for Looped Language Models

Guo Tang; Shixin Jiang; Heng Chang; Zihan Zhang; Nuo Chen; Yuhan Li; Huiming Fan; Jia Li

cs.CLMar 20, 2026

LoopRPT: Reinforcement Pre-Training for Looped Language Models

Authors: Guo Tang, Shixin Jiang, Heng Chang, Zihan Zhang, Nuo Chen, Yuhan Li, Huiming Fan, Jia Li,

Organizations: Harbin Institute of Technology, Harbin, China · Tsinghua University, Beijing, China · The Hong Kong University of Science and Technology, Guangzhou, China

Abstract

Looped language models (LoopLMs) perform iterative latent computation to refine internal representations, offering an alternative to explicit chain-of-thought (CoT) reasoning. However, existing reinforcement learning (RL) paradigms primarily target output tokens, creating a structural mismatch with looped architectures whose reasoning unfolds implicitly. In this work, we propose LoopRPT, a reinforcement pre-training framework tailored for LoopLMs. By reframing next-token prediction as a next-token reasoning task, LoopRPT assigns reinforcement signals directly to latent steps using an EMA teacher reference and noisy latent rollouts. This formulation enables RL to directly shape intermediate representations, compressing effective reasoning into fewer iterations. We instantiate LoopRPT on Ouro across multiple model scales. Results show that LoopRPT improves per-step representation quality and achieves Pareto dominance in accuracy-computation trade-offs. Large gains on hard tokens indicate that LoopRPT enhances early-stage reasoning rather than merely encouraging premature exits. Our findings highlight reinforcement pre-training as a principled paradigm for learning efficient latent reasoning in LoopLMs.

Figures & tables

Panel of Figure 1 — Figure 1: Motivation for LoopRPT. (a) Standard LLMs reason through explicit chain-of-thought tokens, whereas LoopLMs compress multi-step reasoning into latent space. (b) LoopLMs achieve higher next-token-reasoning accuracy than standard LLMs across difficulty levels.

Figure 2: LoopRPT assigns reinforcement signals directly to latent reasoning steps. An entropy-based selector identifies hard tokens, and a student trains against an EMA teacher via step-wise rewards that combine an accuracy gain over the teacher with a dynamic time penalty, shaping intermediate representations and encouraging earlier effective exits.

Figure 3: Distribution of exit steps on MMLU. LoopRPT shifts tokens to earlier iterations while keeping final-step exits dominant.

Model	Method	Easy		Medium		Hard
Model	Method	Acc ( $\uparrow$ )	Avg step ( $\downarrow$ )	Acc ( $\uparrow$ )	Avg step ( $\downarrow$ )	Acc ( $\uparrow$ )	Avg step ( $\downarrow$ )
Qwen3-1.7B	Vanilla	47.49	–	32.18	–	19.19	–
Qwen3-1.7B	+CoT	10.70 -36.79	–	8.60 -23.58	–	7.44 -11.75	–
Ouro-1.4B	Peak	74.62	4.00	57.28	4.00	33.79	4.00
	+LoopRPT	75.38 +0.76	4.00	58.29 +1.01	4.00	34.74 +0.95	4.00
	Adap.	74.40	3.34	57.20	3.53	33.91	3.75

Table 1: Performance comparison on next-token reasoning task across three difficulty levels. Peak indicates reasoning up to the maximum latent loops (

K=4

), while Adap. indicates adaptive early exiting. Subscripts indicate improvement (

{\color[rgb]{0,0.6,0}+}

) or degradation (

{\color[rgb]{0.68,0,0}-}

) compared to baseline.

Benchmark	Gemma3 1B	Llama3.2 1.2B	Qwen2.5 1.5B	Qwen3 1.7B	Qwen2.5 3B	Llama3.2 3B	Qwen3 4B	Gemma3 4B	Ouro 1.4B	LoopRPT 1.4B
General Tasks
MMLU	39.85	45.46	60.99	62.46	65.62	59.69	73.19	58.37	67.35	67.62 +0.27
MMLU-Pro	11.31	11.80	29.11	37.27	37.87	33.34	51.40	34.61	48.64	49.21 +0.57
BBH	30.26	30.72	43.66	53.51	55.37	39.45	70.95	66.32	70.97	71.37 +0.40
ARC-C	39.25	41.98	54.44	55.72	55.46	52.47	63.65	60.92	60.95	61.10 +0.15
HellaSwag	56.12	59.35	67.73	67.09	74.54	73.09	75.66	75.58	74.30	74.93 +0.63

Table 2: Comparison of 1.4B LoopLM model with 1-4B parameter baselines. Results for vanilla LLMs are taken from Zhu et al. (2025) , with Ouro results obtained via our reimplementation. The best score is bolded , and the second-best is underlined . Subscripts indicate LoopRPT’s improvement over Ouro baseline.

Scale	Method	NTR-Hard	GSM8K	MATH500	ARC-C	HellaSwag	BBH	MBPP+
Ouro-1.4B	Base	33.91	78.17	20.60	60.95	74.30	70.97	60.85
	$+$ Continued-NTP	34.29	77.63	29.40	60.58	74.51	71.10	62.17
	$+$ GRPO / RLVR	33.00	4.17	12.60	49.23	74.36	28.84	20.58
	$+$ LoopRPT	34.82	78.92	37.00	61.10	74.93	71.37	63.76
Ouro-2.6B	Base	34.35	81.76	43.00	66.12	79.28	77.98	64.81
	$+$ Continued-NTP	35.21	82.41	43.20	66.75	79.74	78.19	63.76

Table 3: Compute-matched controls. All arms share the checkpoint, data, step budget, and schedule; only the objective changes. NTR-Hard is hard-split accuracy under adaptive early exit; best per scale in bold .

Figure 4: MATH500 accuracy by difficulty level (Ouro-1.4B vs. LoopRPT-1.4B). Levels use the official Hendrycks problem-level labels, which are independent of the token-level entropy that drives token selection during training. LoopRPT improves every level under both maximum-loop (a) and adaptive early exit (b); the per-level sample sizes are shown at the bottom of each panel.

Table 8

Panel of Figure 5 — Figure 5: Sensitivity analysis of LoopRPT hyperparameters on the next-token reasoning task. All results are reported on the hard split, showing accuracy and average exit steps under different settings.

Figure 6: Accuracy–computation trade-offs across difficulty buckets. LoopRPT enhances next-token reasoning accuracy with reduced loop cycles.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Table 11

Table 12

Scale	Benchmark	Ouro	LoopRPT
1.4B	MMLU	68.32	68.32
	GSM8K	29.87	43.52 +13.65
	BBH	38.46	39.26 +0.80
2.6B	MMLU	73.76	73.06 -0.70
	GSM8K	41.32	67.32 +26.00
	BBH	54.17	56.84 +2.67

Appendix

Table 10: No-CoT zero-shot accuracy (few-shot

=0

; GSM8K answers directly and BBH uses the no-CoT template). Subscripts give the change over Ouro.

Model (training corpus)	MMLU	ARC-C	HellaSwag	WinoGrande	GSM8K	MBPP+
Ouro-1.4B base	67.35	60.95	74.30	71.67	78.17	60.85
$+$ LoopRPT ( Omni-Math )	67.62	61.10	74.93	71.98	78.92	63.76
$+$ LoopRPT (OpenThoughts)	67.83	61.19	75.66	72.24	79.03	63.88

Appendix

Table 11: Broader-corpus scaling on OpenThoughts-3 (100k, multi-domain) versus the narrow-domain Omni-Math run, Ouro-1.4B, same LoopRPT recipe. The downstream protocol follows Appendix E .

Figure 9: Performance comparison of different EMA teacher mechanisms on next-token reasoning tasks.

Figure 10: Comparison of actual physical inference time for token exiting.

Panel of Figure 11 — Figure 11: Training dynamics of LoopRPT on Ouro-1.4B.

Panel of Figure 12 — Figure 12: Training dynamics of LoopRPT on Ouro-2.6B.

Panel of Figure 13 — Figure 13: Comparative case study on general tasks (MMLU-Pro). LoopRPT successfully identifies implicit logical equivalences (Case 3) and maintains factual consistency in biological classification (Case 2), whereas the base model exhibits knowledge hallucinations or incomplete reasoning.

Panel of Figure 14 — Figure 14: Comparison of GSM8K mathematical reasoning . Base model (Ouro-1.4B) frequently makes “set-neglect” errors (Case 2, 3) or hallucinations in variable mapping (Case 1), whereas LoopRPT correctly iterates through multi-step logic constraints.

Benchmark	Gemma3 3B	Llama3.2 3B	Qwen2.5 4B	Qwen3 4B	Qwen2.5 7B	Llama3.1 8B	Qwen3 8B	Gemma3 12B	Ouro 2.6B	LoopRPT 2.6B
General Tasks
MMLU	65.62	59.69	73.19	58.37	74.20	73.02	76.63	72.14	73.85	73.91 +0.06
MMLU-Pro	37.87	33.34	51.40	34.61	43.55	43.24	53.72	49.21	54.07	54.19 +0.12
BBH	55.37	39.45	71.14	66.32	53.72	71.56	77.65	78.41	77.98	78.24 +0.26
ARC-C	55.46	52.47	63.65	60.75	63.65	60.75	66.10	72.44	66.12	66.89 +0.77
HellaSwag	74.54	73.09	75.66	75.58	79.98	81.97	79.60	83.68	79.28	80.03 +0.75

Appendix

Table 12: Comparison of 2.6B LoopLM with 3-12B parameter baselines. Results for vanilla LLMs are taken from Zhu et al. [2025] , with Ouro results obtained via our reimplementation. Best in bold , second best underlined . Subscripts indicate LoopRPT’s improvement over Ouro baseline.

Efficient Language Model Reasoning

Efficient language model reasoning studies how to improve problem-solving accuracy while reducing inference, memory, latency, and training costs.

See all 370 papers

LoopCD: Loop-wise Contrastive Decoding for Improving Reasoning in Looped Language Models
Sep 21, 2026Byeongho Yu, Junhyuk So, Eunhyeok ParkContrastive Decoding Latent Reasoning
Department of Computer Science and Engineering Pohang University of Science and Technology (POSTECH) · Graduate School of Artificial Intelligence Pohang University of Science and Technology (POSTECH)
Scaling Latent Reasoning via Looped Language Models
Oct 29, 2025Rui-Jie Zhu, Zixuan Wang, Kai Hua +30Language Model Pretraining Implicit Reasoning in Language Models
ByteDance Seed · UC Santa Cruz · Princeton University +8
LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Models
May 10, 2026Taekhyun Park, Yongjae Lee, Dohee Kim +1Recurrent Transformers LLM Post-Training
Department of Data Science Pusan National University Busan, Republic of Korea · Department of Industrial Engineering Pusan National University Busan, Republic of Korea · Department of Artificial Intelligence Engineering Changwon National University Changwon, Republic of Korea

Sep 21, 2026J/K move · Enter open · S save

LoopCD: Loop-wise Contrastive Decoding for Improving Reasoning in Looped Language Models

Byeongho Yu, Junhyuk So, Eunhyeok Park

Department of Computer Science and Engineering Pohang University of Science and Technology (POSTECH) · Graduate School of Artificial Intelligence Pohang University of Science and Technology (POSTECH)

Contrastive Decoding Latent Reasoning Recursive Language Models

Open paper AskPDF arXiv

Looped Language Models (LoopLMs) perform "latent reasoning" by recursively refining internal latent representations with shared weights, offering a more effective alternative to explicit verbal reasoning. Despite their effectiveness, we find that LoopLMs remain prone to loop instability: unstable refinement across iterations can produce localized uncertain "hard" tokens associated with reasoning errors. To address this, we propose LoopCD, loop-wise contrastive decoding that enhances the reasoning performance of LoopLMs by intervening on these tokens at inference time. Specifically, we exploit the internal dynamics of LoopLMs and contrast the logits from earlier iterations with logits from the last refined iteration to form the final sampling distribution. We find that this strategy is highly efficient, introducing only negligible inference overhead and requiring no additional training, while effectively improving reasoning performance by naturally refining reasoning-critical hard tokens. Extensive experiments show that our method improves the performance of recent representative LoopLMs across various reasoning tasks.

Abstract

Efficient Language Model Reasoning

Explore similar work

LoopCD: Loop-wise Contrastive Decoding for Improving Reasoning in Looped Language Models

Scaling Latent Reasoning via Looped Language Models

LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Models