From RL Papers to Framework Code: Inclusion and Review in Seven Open-Source Training Frameworks

Taiheng Pan

cs.LGOct 20, 2026

From RL Papers to Framework Code: Inclusion and Review in Seven Open-Source Training Frameworks

Authors: Taiheng Pan

Organizations: The University of Melbourne

Abstract

Practitioners use reinforcement learning methods for language models through training frameworks, so a method that a framework includes is one configuration change away for its users. We measure how often this happens. We trace seven open-source frameworks for RL post-training through their commit histories and link them to 6,939 screened arXiv papers from January 2025 to June 2026. Of 1,598 papers that propose a general RL training method, 37 reached at least one framework. Across five quarterly cohorts of papers, the number included within 180 days stayed between 5 and 7 while the cohorts grew from 72 to 386 papers. Compared with papers from the same quarters, included papers more often trained policies of 30B parameters or more, at 41 against 10 percent, and more often changed a coefficient, clip, mask or weight in the existing loss, at 29 against 9 percent. Both gaps hold without papers from the companies that maintain the frameworks, while code release was equally common in both groups. The frameworks' issues and pull requests show that few methods are turned away in public. More than nine in ten of the general methods no framework included have no thread there that cites or names them, and many proposals that do appear stay open without a reply. Two further records, the frameworks' issues and the technical reports of 13 large open-weight models, show that users rarely report training failures in public and that the reports state 8 of the 13 widely included methods. We release the data as a tracker that links training failures to framework implementations and lists each framework's contribution rules and earlier proposals.

Figures & tables

Figure 1: Inclusion, supply against intake, and the public record of the rest. Unit: paper. (a) Papers from January 2025 to June 2026: the screened pool, papers that propose an RL training method, general methods, and general methods by the strongest inclusion they reached, on a log scale. (b) General methods by the quarter of their first arXiv version in grey, and those a framework included within 180 days of that version in blue, for the quarters with 180 days of exposure before the snapshot for every paper, on a log scale. (c) General methods that no framework included, grouped by the threads in the seven frameworks that link them, give their identifier or quote their title, on a log scale. These groups are categories and not stages.

Figure 2: Time from first public version to a main branch. Unit: catalog method with a dated source from 2025 or later that reached two or more frameworks. Figure 5 in the appendix shows every method. Each mark is one landing. Hollow dots mark commits by a paper author. Crosses mark landings in a bulk release or a renaming commit, and their dates are upper bounds. Landings within a few days of each other are spread vertically within their row. The larger mark with a number counts landings too close to draw apart: five frameworks took in GSPO between 3 and 7 days after its first version.

Method	Source	First public	Frameworks	Days	Failure it addresses
Clip-higher	[ 84 ]	2025-03-18	7	1	entropy collapse
GSPO	[ 88 ]	2025-07-24	7	3	MoE instability, extreme ratios
Dynamic sampling	[ 84 ]	2025-03-18	6	17	groups without signal
Dr. GRPO	[ 43 ]	2025-03-26	6	-4	length bias
CISPO	[ 49 ]	2025-06-16	6	$\leq$ 135	entropy collapse
Truncated importance sampling (TIS)	[ 82 ]	2025-08	6	–	rollout-training mismatch

Table 1: Recent methods included in three or more frameworks. Unit: catalog method. Days run from the first public version of the source to the first landing on a main branch. A

\leq

sign marks an upper bound, and negative values mean the commit predates the arXiv version. Of these, truncated importance sampling (TIS) is a conditional default in TRL and token-level loss aggregation is a default in TRL.

Figure 3: Coded fields, included against non-included general methods from 2025. Unit: paper. Each row compares the share among included papers with the share among papers from the same quarters that no framework included. The right columns give the counts, included first. Two shares within three points of each other are drawn one above the other. Papers that do not report a property are left out of that row.

Figure 4: Inclusion by component. Unit: general method, each with one primary component label. Share of general methods in each component that reached at least one framework, with counts. Rows are ordered by the number of general methods, and components with fewer than ten are omitted.

		First	Median	Landing pull requests		Proposals for other methods
Framework	Methods	anywhere	days	By a prior merger	Days open	All	Open	Declined	Lapsed
verl	30	21	25	1 of 27	10	20	12	0	7
TRL	24	16	35	13 of 23	5	18	3	10	2
AReaL	14	4	170	3 of 13	6	1	0	0	1
NeMo-RL	14	3	167	2 of 13	17	2	2	0	0
ROLL	12	4	99	–	–	0	0	0	0

Table 2: Methods and proposals by framework. Unit: landing of a catalog method with a source from 2025 or later, and proposals for general methods that no framework included. Methods counts the landings in the framework, and first anywhere counts the methods with their first landing there. Median days run from the source to the landing and leave out upper bounds. A landing pull request counts as opened by a prior merger when its author had merged a pull request in the framework before opening it, and each pull request counts once. Days open run from opening to merge. Proposals are the coded issues and pull requests of Table 9 , and lapsed proposals were closed as inactive or in a stale-issue cleanup. Declined proposals include those redirected elsewhere. ROLL publishes code in bulk releases without pull requests.

Thread record	pull requests	pull requests	Requests
	Merged	Unmerged
Threads	88	28	20
Reply from someone other than the proposer	71	14	10
Concern raised in the thread	60	11	7
design or interface	50	7	1
tests or documentation	39	4	0

Table 3: Review records of merged pull requests and of proposals that were not merged. Unit: thread. Merged pull requests are the landing pull requests of recent methods, counted once each. Unmerged pull requests and requests are the proposals of Table 9 . A concern is a problem that someone other than the proposer raises explicitly, and one thread can carry several.

	Issue threads	General methods		Catalog methods
Failure	naming it	naming it	included	in a framework
Collapse or divergence, cause not named	23	–	–	–
Rollout-training mismatch	12	20	3	8
Response-length drift	5	167	5	5
Stale rollout data	3	52	4	7
KL penalty grows	3	–	–	–

Table 4: Training failures in the frameworks’ issues against the methods that address them. Unit: thread in the first column, general method in the next two, catalog method in the last. Issue threads are issues and discussions opened since January 2025 with an opening post that reports the failure. General methods naming the failure are the pool’s general methods with an abstract that names it, with the number of those that at least one framework includes. Catalog methods are those addressing the failure that at least one framework includes. The two rows with dashes are failures outside the catalog’s list.

Report	Size	Framework stated	Clip-higher	Dyn. sampling	CISPO	Token loss	Overlong
DeepSeek-R1 [ 13 ]	671B MoE	in-house
Llama-Nemotron [ 4 ]	253B dense	NeMo-Aligner
MiniMax-M1 [ 49 ]	456B MoE	not stated	T	✓	S	✓	✓
Kimi K2 [ 32 ]	1T MoE	in-house
gpt-oss [ 53 ]	117B MoE	in-house
GLM-4.5 [ 22 ]	355B MoE	slime		✓		✓

Table 5: Widely included methods stated by technical reports of open-weight models with 100B parameters or more. Unit: report in the pool. A check marks a method the report states it used, S marks the report that is the method’s source, V a variant described under another name, and T a method tried and not kept. Framework stated is the training system the report names.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Label	Meaning
RL method	A new or modified training-time RL component: loss or objective, advantage or baseline, clipping or trust region, importance weighting or off-policy correction, rollout sampling or filtering that feeds the gradient, task-agnostic reward shaping, KL or entropy regularisation, credit assignment, or an optimiser built for RL on language models.
RL application	Existing RL algorithms applied to a task or domain, including domain-specific rewards or verifiers.
Analysis	Empirical study, reproduction or ablation without a new method.
Systems	Training systems, asynchronous pipelines and throughput work with the objective unchanged.
Offline preference	Offline preference optimisation in the DPO family.
Distillation	The main objective matches a teacher.

Appendix

Table 10

Review \ Label	General	Embedded	Other
General	68	3	1
Embedded	6	12	32
Other	2	1	75

Appendix

Table 6: Manual review against the labels.

Analysis	Policy of 30B or more	$p$	Coefficient change	$p$
Main	12/29 vs 14/141	$<$ 0.001	9/31 vs 13/149	0.004
Including the exceptions	17/35 vs 14/141	$<$ 0.001	9/38 vs 13/149	0.020
Without model reports	12/29 vs 13/139	$<$ 0.001	9/31 vs 13/147	0.005
With an industry author	10/25 vs 11/90	0.003	7/26 vs 9/93	0.045
Missing values against gap	12/31 vs 23/150	0.005	9/31 vs 14/150	0.006
Threshold of 14B	13/29 vs 19/141	$<$ 0.001	–

Appendix

Table 7: Sensitivity of the two largest gaps. Unit: paper. Each cell gives included papers against non-included papers. The last two rows leave out included papers that the framework of their own company includes, and then every paper from a company that maintains one of the frameworks.

p

is from a two-sided Fisher exact test on the two counts of the cell.

Source	Abstract label	Model report	Included methods
Prime Intellect Team et al. [54]	RL method (embedded)	yes	Two-sided ratio clipping
He et al. [26]	RL application	yes	Adaptive entropy control
Fu et al. [20]	RL method (embedded)	no	Decoupled PPO objective
MiniMax et al. [49]	RL method (embedded)	yes	CISPO
Liu et al. [42]	RL method (embedded)	yes	AEPO
Liu et al. [44]	Analysis	no	LitePPO

Appendix

Table 8: Included papers that do not meet the general rule from their abstracts. The attribution of masked importance sampling stays open among near-simultaneous sources.

Method	arXiv	Framework	Thread	Outcome	Stated reasons
VC-PPO decoupled GAE	2503.01491	verl	issue #479	open, under discussion	–
AGPO	2503.15952	TRL	PR #6368	redirected	maintenance burden, placement, evidence, already covered
JEPO	2503.19618	TRL	PR #5411	declined	maintenance burden, demand
CPPO (completion pruning)	2503.22342	TRL	issue #3291	open, no prior-merger reply	–
AdaRFT	2504.05520	verl	PR #1021	open, under discussion	–
SSR	2504.08837	TRL	PR #3496	closed as inactive	–

Appendix

Table 9: Proposals to include general methods that no framework included. Unit: issue, pull request or discussion that proposes or requests the method, coded blind from its own text. The arXiv identifier tells apart methods with the same name. Reasons are those a maintainer stated in the thread.

Repository	Stars	Commit	Status
verl-project/verl	23,926	18cfb74852 2026-10-16	tracked
huggingface/trl	19,551	6d6901c5e3 2026-10-16	tracked
OpenRLHF/OpenRLHF	10,097	dc2a7ad326 2026-09-17	tracked
THUDM/slime	8,686	859938dcb2 2026-10-17	tracked
areal-project/AReaL	5,843	bb13e3bdb9 2026-10-16	tracked
alibaba/ROLL	3,430	d96d0cce56 2026-10-13	tracked

Appendix

Table 10: Open RL frameworks against the selection criteria of Section 3 . Unit: repository with at least 2,000 stars on 19 October 2026. Commits are the snapshots of the seven tracked frameworks and the latest commits of the other qualifying ones. For qualifying frameworks outside the study, the count gives general methods that this commit implements and none of the seven include.

Method	Option names at the snapshot
Clip-higher	verl: clip_ratio_high , recipe/dapo ; TRL: epsilon_high ; OpenRLHF: eps_clip_low_high ; NeMo-RL: ratio_clip_max ; ROLL: pg_clip_high ; slime: eps_clip_high ; AReaL: dapo_dynamic_bs , eps_clip_higher
GSPO	verl: gspo ; TRL: importance_sampling_level ; OpenRLHF: gspo ; NeMo-RL: sequence_level_importance_ratios ; ROLL: importance_sampling_seq ; slime: gspo ; AReaL: gspo , importance_sampling_level
Dynamic sampling	verl: filter_groups , recipe/dapo ; OpenRLHF: dynamic_filtering ; NeMo-RL: use_dynamic_sampling ; ROLL: difficulty_mask ; slime: dynamic_sampling_filter ; AReaL: dapo_dynamic_bs , dynamic_bs
Dr. GRPO	verl: norm_adv_by_std_in_grpo , seq-mean-token-sum-norm ; TRL: dr_grpo , scale_rewards ; OpenRLHF: dr_grpo , no_std_norm ; ROLL: seq-mean-token-sum-norm ; slime: disable_grpo_std_normalization ; AReaL: drgrpo
CISPO	verl: cispo ; TRL: cispo ; NeMo-RL: use_cispo ; ROLL: cispo ; slime: cispo ; AReaL: use_cispo_loss
Truncated importance sampling (TIS)	verl: rollout_is ; TRL: vllm_importance_sampling_correction ; OpenRLHF: is_correction ; NeMo-RL: truncated_importance_sampling_ratio ; ROLL: tis , train_infer_correction ; slime: use_tis

Appendix

Table 11: Option names at the snapshot for recent methods included in three or more frameworks. Unit: catalog method. Names are the registry keys, loss types, configuration fields or recipe folders that the parser maps to each method.

Figure 5: Time from first public version to a main branch, all methods. Unit: catalog method with a dated source from 2025 or later. Marks as in Figure 2 .

Method	Source	First public	First landing	Frameworks now
PPO (clipped surrogate, GAE)	Schulman et al. [59]	2017-07-20	2022-12-30	verl, OpenRLHF, NeMo-RL, ROLL, slime, AReaL
Dual-clip PPO	Ye et al. [83]	2019-12-20	2025-04-02	verl, OpenRLHF, NeMo-RL, ROLL, slime, AReaL
k3 (low-variance) KL estimator	Schulman [58]	2020-03-07	2025-01-23	verl
ReMax	Li et al. [36]	2023-10-16	2025-02-10	verl
GRPO	Shao et al. [61]	2024-02-05	2025-01-20	verl, TRL, OpenRLHF, NeMo-RL, ROLL, slime, AReaL
RLOO	Ahmadian et al. [1]	2024-02-22	2024-05-22	verl, TRL, OpenRLHF, NeMo-RL, AReaL

Appendix

Table 12: Method catalog. Unit: catalog method. First landing is the earliest main-branch commit in any framework, with

\leq

for upper bounds. Frameworks list where the method is offered at the snapshot.

RL Post-Training

Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.

See all 424 papers

Mar 6, 2026

A Reference Architecture of Reinforcement Learning Frameworks

The surge in reinforcement learning (RL) applications gave rise to diverse supporting technology, such as RL frameworks. However, the architectural patterns of these frameworks are inconsistent across implementations and there exists no reference architecture (RA) to form a common basis of comparison, evaluation, and integration. To address this gap, we propose an RA of RL frameworks. Through a grounded theory approach, we analyze 18 state-of-the-practice RL frameworks and, by that, we identify recurring architectural components and their relationships, and codify them in an RA. To demonstrate our RA, we reconstruct characteristic RL patterns. Finally, we identify architectural trends, e.g., commonly used components, and outline paths to improving RL frameworks.

Xiaoran Liu, Istvan David

McMaster University, Hamilton, Canada · McMaster Centre for Software Certification, Hamilton, Canada

Reinforcement Learning

Ask PDF

Aug 24, 2026

Demystifying Reinforcement Learning Post-Training of Language Models

Reinforcement learning (RL) post-training has emerged as a powerful framework for enhancing the capabilities of large language models (LLMs), enabling impressive reasoning, math, and coding capabilities. Yet for many researchers and practitioners, the principles behind classical RL remain a "black box". In this work, we deconstruct the RL post-training algorithm, investigating each step to clarify what is actually happening beneath the surface. By isolating the mechanics of RL with Verifiable Rewards in a controlled and simplified environment, we examine how RL outcomes are shaped by the base model's prior distribution, the granularity of the reward signal, the diversity of the prompt distribution, and model scale. We use the entropy of the policy's output distribution as a lens to compare the distributions learned through pretraining, SFT, and RL post-training, revealing how each stage shapes model certainty. Our investigation sheds light on how these choices interact to affect post-training success. For example, we show that the effect of so-called 'spurious rewards' depends on the prompt distribution used for post-training. We also provide insight into why the success of RL post-training depends on whether the base model already places sufficient probability mass on the desired behavior, linking it to the classical concept of exploration in RL. Ultimately, we provide this primer as a resource to those in the NLP community wishing to incorporate RL as a tool in their toolbox.

Donovan Clay, Saket Gollapudi, Sankar Harilal +4

University of Washington · Allen Institute for AI

RL for Language Models RL Post-Training Reinforcement Learning with Verifiable Rewards RL Exploration

Ask PDF

May 14, 2026

Where and What to Reward? Understanding Generalization Between Verifiable and Open-Ended Domains in Reinforcement Learning

Reinforcement learning with verifiable rewards (RLVR) is widely used to improve the reasoning capability of language models, since model-generated answers can be easily verified by a rule-based verifier. In contrast, reinforcement learning from human feedback (RLHF) is mainly used to optimize open-ended conversation with a learned reward model, because such responses are not easy to verify. RLVR and RLHF are usually trained separately with their own domain data and training protocols. How gains transfer between these domains is unclear, partly because comparisons change both where a model is trained (the prompts) and what is rewarded (by a verifier or a reward model). We separate the two in a controlled study of RL on multiple base models that fixes the optimizer and the number of training prompts, and varies only the source domain of the prompts (math, multi-domain, chat, or mixtures) and the reward. We find that RLHF on open-ended prompts substantially improves conversational ability and also brings large gains in reasoning and coding, whereas RLVR improves reasoning but its gains in open-ended conversation plateau far below those of RLHF, suggesting that RL on open-ended prompts with a reward model can serve as an effective approach for generalizing across domains.

Shangjian Yin, Yu Fu, Yue Dong +1

♠University of California, Riverside

Reinforcement Learning RL for Language Model Reasoning RL Post-Training Generalization in RL RL from Human Feedback

Ask PDF

Abstract

RL Post-Training

Explore similar work

A Reference Architecture of Reinforcement Learning Frameworks

Demystifying Reinforcement Learning Post-Training of Language Models

Where and What to Reward? Understanding Generalization Between Verifiable and Open-Ended Domains in Reinforcement Learning