Performance Foundations of Parallel & Distributed Reasoning Language Models

Maciej Besta; Leonard Schmidt; Lara Nonino; Robert Gerstenberger; Pierre Pang; Patrik Okanovic; Ales Kubicek; Tiancheng Chen

cs.LGAug 27, 2026

Performance Foundations of Parallel & Distributed Reasoning Language Models

Authors: Maciej Besta, Leonard Schmidt, Lara Nonino, Robert Gerstenberger, Pierre Pang, Patrik Okanovic, Ales Kubicek, Tiancheng Chen,

Organizations: ETH Zurich

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) and other RL-style post-training paradigms have been used for aligning large language models (LLMs) with reasoning standards. The resulting recent Reasoning Language Models (RLMs) such as DeepSeek-R1, o3, and Kimi k1.5 show that such RL-style post-training ("RL-for-LLMs") can substantially improve chain-of-thought reasoning, long-horizon planning, and self-correction. However, the computational footprint of these systems is massive: state-of-the-art RLM training requires millions of GPU-hours and tightly coupled multi-model pipelines that stress modern hardware far beyond classical supervised LLM training. This makes RLM training as much a parallel and distributed systems problem as an algorithmic one. In this work, to facilitate developing RLMs that are simultaneously high-performance, scalable, and cost-effective, we first systematize the RL-for-LLM paradigm and provide a compute-centric analysis of prominent post-training algorithmic frameworks: Proximal Policy Optimization (PPO), Group Relative Policy Optimization (GRPO), as well as their variants. Second, we develop a taxonomy of intra- and inter-model parallelism strategies for RL-for-LLMs, covering both traditional techniques (data, tensor, pipeline, sequence, context, and expert parallelism) as well as novel forms of parallelism and optimization techniques for multi-model RLM training, for example disaggregated placement, stage fusion, hybrid parallelism, and asynchronous execution. We harness the work-depth model of parallel computing to make our taxonomy and its insights rigorous and portable. Finally, we analyze existing RLM frameworks and we distill practical guidelines and outline open research directions for building scalable, fast, and cost-effective RLMs.

Figures & tables

Fig. 1 : Overview of the contributions and analyses in this work.

Models & training signals used in the RL pipeline
$\pi_{\theta}$	Policy (Actor) model: the trainable LLM being optimized.
$\pi_{\mathrm{ref}}$	Reference model: the frozen baseline policy (usually a checkpoint right after SFT).
$R_{\varphi}$	Reward model: an assessment tool that maps prompt–response pairs to scalar rewards.
$V_{\psi}$	Critic model (Value function): an assessment tool that estimates expected future returns from the reward model.
$r_{b}^{(i)},A_{b,t}^{(i)},\hat{G}_{b,t}^{(i)}$	Reward for the $i$ -th candidate of prompt $b$ , advantage at token step $t$ , and discounted return estimated at token step $t$ .
Data & Dimensions

TABLE I : Overview of basic mathematical notation and concepts used in the paper. Notation specific to the complexity analyses is detailed separately in Table III .

Fig. 2 : Overview of policy optimization algorithms and used models. No AI was used to conceive or to draw the figure.

Model	State	Used in	System design remarks
Policy (Actor) $\pi_{\theta}$	Trainable	Generation, Training	Bottleneck: Sequential decoding dominates wall-clock time.
Reward $R_{\varphi}$ (Outcome)	Frozen	Assessment	Efficient parallel evaluation; no backward pass. Easy to offload.
Reward $R_{\varphi}^{\text{pr}}$ (Process)	Frozen	Assessment	High overhead; sequential decoding if using sequential verification (e.g., MCTS).
Critic (Value) $V_{\psi}$	Trainable	Assessment, Training	Doubles training compute (fwd+bwd). High memory pressure (optimizer states).
Reference $\pi_{\text{ref}}$	Frozen	Assessment	Required for KL/DPO. Forward-only; often quantized to save memory.

TABLE II : Overview of models used in the RLM pipeline.

Cat.	Symbol	Meaning and remarks	Mathematical expression
Work	$C_{f}^{\text{tok}}(M)$	Forward-pass FLOPs per token for model $M$ on a context of length $S+T$ (input prompt length + response length). The reference policy $\pi_{\mathrm{ref}}$ is architecturally identical to the actor $\pi_{\theta}$ . Unembedding FLOPs per token are $2Vd_{\pi}$ (policy only). The reward model $R_{\varphi}$ adds a projection head applied to the final token only, contributing $\tfrac{2d_{R}}{S+T}$ FLOPs per token, whereas the value model $V_{\psi}$ applies a projection head to every token, contributing $2d_{V}$ FLOPs per token.	$\begin{aligned} &C_{f}^{\text{tok}}(\pi_{\theta})&=&\ 2L_{\pi}\left[4d_{\pi}^{2}+2d_{\pi}d_{\pi\mathrm{ff}}+(S+T)d_{\pi}\right]+2Vd_{\pi}\\ &C_{f}^{\text{tok}}(\pi_{\mathrm{ref}})&=&\ C_{f}^{\text{tok}}(\pi_{\theta})\\ &C_{f}^{\text{tok}}(R_{\varphi})&=&\ 2L_{R}\left[4d_{R}^{2}+2d_{R}d_{R\mathrm{ff}}+(S+T)d_{R}\right]+\tfrac{2d_{R}}{S+T}\\ &C_{f}^{\text{tok}}(V_{\psi})&=&\ 2L_{V}\left[4d_{V}^{2}+2d_{V}d_{V\mathrm{ff}}+(S+T)d_{V}\right]+2d_{V}\\ \end{aligned}$
Work	$C_{b}^{\text{tok}}(M)$	Backward-pass FLOPs per token for model $M$ on a context of length $S+T$ (input prompt length + response length).	$\begin{aligned} C_{b}^{\text{tok}}(M)\ \ \approx 2\cdot C_{f}^{\text{tok}}(M)\end{aligned}$
	$C_{\text{gen}}^{\text{roll}}(\pi_{\theta})$	Autoregressive generation FLOPs per rollout for generating $T$ tokens (response length) from a context of length $S$ (input prompt length) with KV caching (policy only). It is decomposed into prefill and decode. Prefill FLOPs are for processing the prompt of length $S$ once and initializing the KV cache before autoregressive decoding begins. Prefill attention is assumed to be dense; as it is causal only half of the full MM product are needed, giving $\approx S^{2}d_{\pi}$ FLOPs for both $QK^{T}$ and for $PV$ ; $O(S^{2})$ and $O(Sd)$ account for lower-order contributions (respectively – softmax and scaling by $1/\sqrt{d_{h}}$ as well as RoPE and layer norms). Decode FLOPs are for generating the remaining $T-1$ tokens autoregressively with KV caching. At decode step $t$ , the model attends to context length $S+t$ .	$\begin{aligned} C_{\text{gen}}^{\text{roll}}(\pi_{\theta})&=C_{\text{pf}}^{\text{roll}}(\pi_{\theta})+C_{\text{dec}}^{\text{roll}}(\pi_{\theta})\\ C_{\text{pf}}^{\text{roll}}(\pi_{\theta})&=2L_{\pi}S\!\left(4d_{\pi}^{2}+2d_{\pi}d_{\pi\mathrm{ff}}+Sd_{\pi}\right)\\ &+L_{\pi}\left(O(S^{2})+O(Sd_{\pi})\right)+2Vd_{\pi}\\ C_{\text{dec}}^{\text{roll}}(\pi_{\theta})&=\sum_{t=1}^{T-1}\!\Bigl[2L_{\pi}\!\bigl(4d_{\pi}^{2}+2d_{\pi}d_{\pi\mathrm{ff}}+2(S+t)d_{\pi}\bigr)+2Vd_{\pi}\!\Bigr]\\ &+L_{\pi}\left(O(S+t)+O(d_{\pi})\right)\end{aligned}$
Depth	$D_{f}(M)$	Forward-pass depth of model $M$ on a context of length $S+T$ , defined as the length of the critical path assuming unbounded parallelism. The policy $\pi_{\theta}$ includes an additional unembedding (output projection) step, which contributes a $\log d_{\pi}$ term to the depth. For the reward model $R_{\varphi}$ and the critic $V_{\psi}$ , the additional projection head contributes an extra $\log d$ term to the depth.	$\begin{aligned} &D_{f}(\pi_{\theta})&=&\ O\left(L_{\pi}\left[\log d_{\pi}+\log(S+T)\right]+\log d_{\pi}\right)\\ &D_{f}(\pi_{\mathrm{ref}})&=&\ D_{f}(\pi_{\theta})\\ &D_{f}(R_{\varphi})&=&\ O\left(L_{R}\left[\log d_{R}+\log(S+T)\right]+\log d_{R}\right)\\ &D_{f}(V_{\psi})&=&\ O\left(L_{V}\left[\log d_{V}+\log(S+T)\right]+\log d_{V}\right)\end{aligned}$
	$D_{f}^{\mathrm{TP}}\!(M)$	Forward-pass depth of model $M$ with tensor parallelism degree $P_{t}$ on a context of length $S+T$ .	$\begin{aligned} D_{f}^{\mathrm{TP}}\!(M)\!&=O\left(L_{M}\left[\log\!\tfrac{d_{M}}{P_{t}}+\log(S+T)\right]+\log\tfrac{d_{M}}{P_{t}}\right)\end{aligned}$
	$D_{b}(M)$	Backward-pass depth of model $M$ on a context of length $S+T$ . It equals the depth of the forward pass for the corresponding model $M$ (which entails computing the gradients with respect to activations), plus the additional term $O(\log(B (S+T)))$ coming from computing the gradients with respect to model weights, where one has to accumulate gradients across batch (as these two branches can overlap, the expression given is a conservative bound).	$\begin{aligned} D_{b}(M)\ \ &=D_{f}(M)+O(\log(B (S+T)))\end{aligned}$

TABLE III : Computational building blocks used across the paper. We assume canonical multi-head attention and two-projection FFN. For computing FLOP costs, an addition and a multiplication count as two separate FLOPs (i.e., a dot product of vectors of dimensionalities

d

results in

d+(d-1)\approx 2d

FLOPs under this model).

C_{f}^{\text{tok}}

C_{b}^{\text{tok}}

C_{\text{gen}}^{\text{roll}}

are the costs of (respectively) forward pass, backward pass, and autoregressive generation; defining them per token and per rollout effectively clarifies the notation for the subsequent analyses. The explicit depth expressions count the selected GEMM reduction chains and parameter-gradient accumulation; they suppress the additional logarithmic reductions from normalization, attention/vocabulary softmax, sampling, and distributed collectives. For generation, we explicitly distinguish prefill and decode. Prefill processes the prompt once, while decode generates tokens sequentially with KV caching. This decomposition is important because rollout generation is the dominant online bottleneck and is not equivalent to a single teacher-forced forward pass over a sequence of length

S+T

Framework	PPO (Online)	GRPO (Online)	DPO (Offline)
Work (Total FLOPs)
Generation	$O(BK\cdot C_{\text{gen}}^{\text{roll}}(\pi_{\theta}))$	$O(BK\cdot C_{\text{gen}}^{\text{roll}}(\pi_{\theta}))$	$-$ †
Assessment	$O(BK(S+T)\cdot[C_{f}^{\text{tok}}(\pi_{\text{ref}})+C_{f}^{\text{tok}}(R_{\varphi})+C_{f}^{\text{tok}}(V_{\psi})])$	$O(BK(S+T)\cdot[C_{f}^{\text{tok}}(\pi_{\text{ref}})+C_{f}^{\text{tok}}(R_{\varphi})])$	$O(B_{\text{pairs}}(S+T)\cdot[C_{f}^{\text{tok}}(\pi_{\text{ref}})+C_{f}^{\text{tok}}(\pi_{\theta})])$
Training	$O(BK(S+T)\cdot[C_{\text{tr}}^{\text{tok}}(\pi_{\theta})+C_{\text{tr}}^{\text{tok}}(V_{\psi})])$	$O(BK(S+T)\cdot C_{\text{tr}}^{\text{tok}}(\pi_{\theta}))$	$O(B_{\text{pairs}}(S+T)\cdot C_{\text{tr}}^{\text{tok}}(\pi_{\theta}))$
Depth (Critical Path Latency)
Generation	$O(D_{\text{gen}}(\pi_{\theta}))$	$O(D_{\text{gen}}(\pi_{\theta}))$	$-$

TABLE IV : Asymptotic Work, Depth, and Memory analysis of PPO, GRPO, and DPO.

S

is prompt length,

T

is response length. We denote parameter counts by

|\cdot|

. “tr” subscript: For conciseness, for training-stage model invocations, we define

C_{\text{tr}}^{\text{tok}}(M):=C_{f}^{\text{tok}}(M)+C_{b}^{\text{tok}}(M)\approx 3C_{f}^{\text{tok}}(M)

and

D_{\mathrm{tr}}(M):=D_{f}(M)+D_{b}(M)

, because a parameter update requires a forward pass followed by backpropagation. Note on Work:

C_{f}^{\text{tok}}

and

C_{b}^{\text{tok}}

are the cost of forward and backward passes per token and

C_{\text{gen}}^{\text{roll}}

is the autoregressive generation cost per rollout ; these are derived in Table III . Work scales with total tokens

(S+T)

. Note on Depth: Generation is depth-bound by

T

(sequential), while in Assessment/Training there is no outer linear

T

-step autoregressive chain. † DPO is offline; generation occurs prior to the training loop. ‡ Max depth assumes parallel (disaggregated) execution; sequential co-located execution is additive.

M_{\text{KV}}

is Key-Value cache memory;

M_{\text{Act}}

is activation memory (training);

M_{\text{Inf}}

is inference buffer (assessment). Note that

\pi_{\text{ref}}

never requires

M_{\text{Act}}

Fig. 3 : Mathematical notation and equations for the forward and backward passes of each decoder block. Further details on computational aspects are in Figures 4 - 5 while parallelization details are in Figures 6 - 7 . No AI was used to conceive or to draw the figure.

Fig. 4 : Intra-model execution details (forward pass) for each RLM model execution. The design details are based on Llama-3. Further details on parallelization and the corresponding taxonomy are provided in Figures 6 - 7 . No AI was used to conceive or to draw the figure.

Fig. 5 : Intra-model execution details (backward pass) for each RLM model execution. Legend is provided in Figure 4 . The design details are based on Llama-3. Further details on parallelization and the corresponding taxonomy are provided in Figures 6 - 7 . No AI was used to conceive or to draw the figure.

Fig. 6 : Intra-model parallelism (data, pipeline, tensor, and expert parallelism) details for each RLM model execution; legend is provided in Figure 4 . No AI was used to conceive or to draw the figure.

Fig. 7 : Intra-model parallelism (sequence & context parallelism, as well as prefill vs. decode and flash attention) details for each RLM model execution; legend is provided in Figure 4 . No AI was used to conceive or to draw the figure.

Parallelism	Work	Depth	Memory
None †	$O\!\left(BmL(d^{2}+md)\right)$	$O\!\left(L[\log d+\log m]+\log(Bm)\right)$	$O\!\left(Ld^{2}+Vd+BmLd\right)$
Data	$O\!\left(\tfrac{BmL(d^{2}+md)}{P_{d}}\right)$	$O\!\left(L[\log d+\log m]+\log\!\left(\tfrac{B}{P_{d}}m\right)\right)$	$O\!\left(Ld^{2}+Vd+\tfrac{BmLd}{P_{d}}\right)$
Pipeline ∗	$O\!\left(\tfrac{BmL(d^{2}+md)}{P_{p}}\right)$	$O\!\left(\tfrac{L}{P_{p}}[\log d+\log m]+\log(Bm)\right)$	$O\!\left(\tfrac{Ld^{2}}{P_{p}}+Vd+\tfrac{BmLd}{P_{p}}\right)$
Tensor	$O\!\left(\tfrac{BmL(d^{2}+md)}{P_{t}}\right)$	$O\!\left(L\!\left[\log\tfrac{d}{P_{t}}+\log m\right]+\log(Bm)\right)$	$O\!\left(\tfrac{Ld^{2}+Vd}{P_{t}}+\tfrac{BmLd}{P_{t}}\right)$
Context	$O\!\left(\tfrac{BmL(d^{2}+md)}{P_{c}}\right)$	$O\!\left(L[\log d+\log m]+\log\!\left(\tfrac{Bm}{P_{c}}\right)\right)$	$O\!\left(Ld^{2}+Vd+\tfrac{BmLd}{P_{c}}\right)$
Expert	$O\!\left(BmL\left[(d^{2}+md)+\tfrac{E_{a}dd_{e}}{P_{e}}\right]\right)$	$O\!\left(L[\log d+\log m]+\log(Bm)\right)$	$O\!\left(L\left[d^{2}+\tfrac{Edd_{e}}{P_{e}}\right]+Vd+BmLd\right)$

TABLE V : Complexity Analysis (Per rank). Asymptotic work, depth, and memory for a single Transformer training iteration (forward and backward pass), expressed in Big-

O

notation. For clarity, we use

m:=S+T

. The work expressions use the regime where

d_{ff}=O(d)

. † : Baseline configuration without parallelism. ∗ For PP, the

Vd

memory term denotes the embedding/output state resident on boundary pipeline ranks; interior ranks need not store this state. Thus, this term represents the boundary/peak per-rank footprint rather than replication across all

P_{p}

ranks. ‡ : 3D and 5D parallelism combine data, pipeline, tensor (3D), context, and expert (5D) parallelism, with

P_{d}P_{p}P_{t}=N

(3D) and

P_{d}P_{p}P_{t}P_{c}P_{e}=N

(5D) total devices. For the combined 5D configuration, we assume the EP dimension also partitions source-token ownership for shared-layer computation. Training memory assumes Adam without activation checkpointing.

Parallelism	Work	Depth	Memory
None	$O\!\left(BmL(d^{2}+md)\right)$	$O\!\left(L[\log d+\log m]+\log(Bm)\right)$	$O\!\left(Ld^{2}+Vd+BmLd\right)$
Data	$O\!\left(BmL(d^{2}+md)\right)$	$O\!\left(L[\log d+\log m]+\log(Bm)\right)$	$O\!\left(P_{d}[Ld^{2}+Vd]+BmLd\right)$
Pipeline	$O\!\left(BmL(d^{2}+md)\right)$	$O\!\left(L[\log d+\log m]+\log(Bm)\right)$	$O\!\left(Ld^{2}+Vd+BmLd\right)$
Tensor	$O\!\left(BmL(d^{2}+md)\right)$	$O\!\left(L\!\left[\log{d}+\log m\right]+\log(Bm)\right)$	$O\!\left(Ld^{2}+Vd+BmLd\right)$
Context	$O\!\left(BmL(d^{2}+md)\right)$	$O\!\left(L[\log d+\log m]+\log(Bm)\right)$	$O\!\left(P_{c}[Ld^{2}+Vd]+BmLd\right)$
Expert	$O\!\left(BmL\left[P_{e}(d^{2}+md)+E_{a}dd_{e}\right]\right)$	$O\!\left(L[\log d+\log m]+\log(Bm)\right)$	$O\!\left(P_{e}[Ld^{2}+Vd+BmLd]+LEdd_{e}\right)$

TABLE VI : Complexity Analysis (Global). Asymptotic work, depth, and memory for a single Transformer training iteration (forward & backward pass). For clarity, we use

m:=S+T

. The work expressions use the regime where

d_{ff}=O(d)

Parallelism	Work	Depth	Memory
None	$6BmL\left(4d^{2}+2dd_{\text{ff}}+md\right)$	$2L\log\!\left(\tfrac{d^{4}d_{\text{ff}}m}{h}\right)+\log(Bm)$	$L(16d^{2}+8dd_{\text{ff}})+4Vd+BmLd$
Data	$6\tfrac{B}{P_{d}}mL\left(4d^{2}+2dd_{\text{ff}}+md\right)$	$2L\log\!\left(\tfrac{d^{4}d_{\text{ff}}m}{h}\right)+\log\!\left(\tfrac{B}{P_{d}}m\right)$	$L(16d^{2}+8dd_{\text{ff}})+4Vd+\tfrac{B}{P_{d}}mLd$
Pipeline	$6Bm\tfrac{L}{P_{p}}\left(4d^{2}+2dd_{\text{ff}}+md\right)$	$2\tfrac{L}{P_{p}}\log\!\left(\tfrac{d^{4}d_{\text{ff}}m}{h}\right)+\log(Bm)$	$\tfrac{L}{P_{p}}(16d^{2}+8dd_{\text{ff}})+4Vd+Bm\tfrac{L}{P_{p}}d$
Tensor	$6BmL\left(\tfrac{4d^{2}+2dd_{\text{ff}}+md}{P_{t}}\right)$	$2L\log\!\left(\tfrac{d^{4}d_{\text{ff}}m}{P_{t}^{2}h}\right)+\log(Bm)$	$\tfrac{L(16d^{2}+8dd_{\text{ff}})}{P_{t}}+\tfrac{4Vd}{P_{t}}+\tfrac{BmLd}{P_{t}}$
Context	$6B\tfrac{S+T}{P_{c}}L\left(4d^{2}+2dd_{\text{ff}}+md\right)$	$2L\log\!\left(\tfrac{d^{4}d_{\text{ff}}m}{h}\right)+\log\!\left(\tfrac{Bm}{P_{c}}\right)$	$L(16d^{2}+8dd_{\text{ff}})+4Vd+B\tfrac{S+T}{P_{c}}Ld$
Expert	$6BmL\left(4d^{2}+\tfrac{2E_{a}dd_{e}}{P_{e}}+md\right)$	$2L\log\!\left(\tfrac{d^{4}d_{e}m}{h}\right)+\log(Bm)$	$L\left(16d^{2}+\tfrac{8Edd_{e}}{P_{e}}\right)+4Vd+BmLd$

TABLE VII : Complexity Analysis (Per rank). Explicit leading-order costs under our simplified Transformer cost model, for a single Transformer training iteration (forward and backward pass), including explicit constant factors. For clarity, we use

m:=S+T

Parallelism	Work	Depth	Memory
None	$6BmL\left(4d^{2}+2dd_{\text{ff}}+md\right)$	$2L\log\!\left({d^{4}d_{\text{ff}}m}/{h}\right)+\log(Bm)$	$L(16d^{2}+8dd_{\text{ff}})+4Vd+BmLd$
Data	$6BmL\left(4d^{2}+2dd_{\text{ff}}+md\right)$	$2L\log\!\left({d^{4}d_{\text{ff}}m}/{h}\right)+\log(Bm)$	$P_{d}\left[L(16d^{2}+8dd_{\text{ff}})+4Vd\right]+BmLd$
Pipeline	$6BmL\left(4d^{2}+2dd_{\text{ff}}+md\right)$	$2L\log\!\left({d^{4}d_{\text{ff}}m}/{h}\right)+\log(Bm)$	$L(16d^{2}+8dd_{\text{ff}})+4Vd+BmLd$
Tensor	$6BmL\left(4d^{2}+2dd_{\text{ff}}+md\right)$	$2L\log\!\left({d^{4}d_{\text{ff}}m}/{h}\right)+\log(Bm)$	$L(16d^{2}+8dd_{\text{ff}})+4Vd+BmLd$
Context	$6BmL\left(4d^{2}+2dd_{\text{ff}}+md\right)$	$2L\log\!\left({d^{4}d_{\text{ff}}m}/{h}\right)+\log(Bm)$	$P_{c}\left[L(16d^{2}+8dd_{\text{ff}})+4Vd\right]+BmLd$
Expert	$6BmL\left(P_{e}\!\left(4d^{2}+md\right)+2E_{a}dd_{e}\right)$	$2L\log\!\left({d^{4}d_{e}m}/{h}\right)+\log(Bm)$	$P_{e}\left(16Ld^{2}+4Vd+BmLd\right)+8LEdd_{e}$

TABLE VIII : Complexity Analysis (Global). Explicit leading-order costs under our simplified Transformer cost model, for a single Transformer training iteration (forward and backward pass), including explicit constant factors. For clarity, we use

m:=S+T

Fig. 8 : Inter-model parallelism details (asynchronous execution, hybrid execution, stage fusion). No AI was used to conceive or to draw the figure.

Fig. 9 : Inter-model parallelism details (placement strategies, model structure configurations). No AI was used to conceive or to draw the figure.

Config	Stage	Global Work	Global Depth	Peak Memory per-device group
Baseline	Gen.	$BK\!\cdot\!C_{\text{gen}}(\pi_{\theta})$	$D_{\text{gen}}(\pi_{\theta})$	$\|\pi_{\theta}\|+BK\!\cdot\!M_{\text{KV}}$
	Assess.	$BK(S+T)$ $\cdot[C_{f}^{\text{tok}}(\pi_{\text{ref}})+C_{f}^{\text{tok}}(R_{\varphi})+C_{f}^{\text{tok}}(V_{\psi})]$	$D_{f}(\pi_{\text{ref}})+D_{f}(R_{\varphi})+D_{f}(V_{\psi})$	$\|\pi_{\text{ref}}\|+\|R_{\varphi}\|+\|V_{\psi}\|$ $+BK(S+T)$ $\cdot\max\{M_{\mathrm{Inf}}(\pi_{\mathrm{ref}}),M_{\mathrm{Inf}}(R_{\varphi}),M_{\mathrm{Inf}}(V_{\psi})\}$
	Train.	$BK(S+T)\!\cdot\![C_{\text{tr}}^{\text{tok}}(\pi_{\theta})+C_{\text{tr}}^{\text{tok}}(V_{\psi})]$	$D_{\text{tr}}(\pi_{\theta})+D_{\text{tr}}(V_{\psi})$	$4\!\cdot\!(\|\pi_{\theta}\|+\|V_{\psi}\|)$ $+BK(S+T)(L_{\pi}d_{\pi}+L_{\psi}d_{\psi})$
AC shared	Gen.	$BK\!\cdot\!C_{\text{gen}}(\pi_{\text{SH}})$	$D_{\text{gen}}(\pi_{\text{SH}})$	$\|\pi_{\text{SH}}\|+BK\!\cdot\!M_{\text{KV}}$
	Assess.	$BK(S+T)$ $\cdot[C_{f}^{\text{tok}}(\pi_{\text{ref}})+C_{f}^{\text{tok}}(R_{\varphi})+C_{f}^{\text{tok}}(\pi_{\text{SH}})]$	$D_{f}(\pi_{\text{ref}})+D_{f}(R_{\varphi})+D_{f}(\pi_{\text{SH}})$	$\|\pi_{\text{ref}}\|+\|R_{\varphi}\|+\|\pi_{\text{SH}}\|$ $+BK(S+T)$ $\cdot\max\{M_{\mathrm{Inf}}(\pi_{\mathrm{ref}}),M_{\mathrm{Inf}}(R_{\varphi}),M_{\mathrm{Inf}}(\pi_{\text{SH}})\}$
	Train.	$BK(S+T)\!\cdot\!C_{\text{tr}}^{\text{tok}}(\pi_{\text{SH}})$	$D_{\text{tr}}(\pi_{\text{SH}})$	$4\!\cdot\!\|\pi_{\text{SH}}\|+BK(S+T)L_{\text{SH}}d_{\text{SH}}$

TABLE IX : Inter-model parallelism. Global work, global depth, and peak memory per device group reported for the Generation, Assessment, and Training stages of a single RL iteration within one epoch under different inter-model parallelism techniques. Global work and depth report total FLOPs and the critical-path length per stage. Peak memory is measured per device group and taken as the maximum over all groups active within a stage; in configurations without disaggregation, all models share the same device group and execute through temporal multiplexing, where different models become active at different stages. “TP” superscript & “SH” subscript: A

\mathrm{TP}

superscript denotes tensor-parallel execution;

\pi_{\mathrm{SH}}

denotes the shared actor–critic configuration. “tr” subscript: For conciseness, for training-stage model invocations, we define

C_{\text{tr}}^{\text{tok}}(M):=C_{f}^{\text{tok}}(M)+C_{b}^{\text{tok}}(M)\approx 3C_{f}^{\text{tok}}(M)

and

D_{\text{tr}}(M):=D_{f}(M)+D_{b}(M)

, because a parameter update requires a forward pass followed by backpropagation.

M_{\text{sh}}(\pi_{\theta})

denotes the persistent inference-side shadow copy of the actor required when asynchronous consumers use a bounded-stale snapshot. Also, sample-time actor log-probabilities are assumed to be computed on the fly during Generation from the same logits used for sampling; therefore they do not add a separate actor forward pass. Baselines: The Baseline configuration sequentially (i.e., one device group) executes all RL components, resulting in additive depth terms. AC shared uses a shared Transformer backbone for actor and critic. All models run sequentially in a single device group. Disaggregated execution assigns models to distinct device groups, allowing concurrent execution. The Hybrid configuration applies TP to Generation, TP/PP to Assessment, and ZeRO-3 to Training. All models share the same device group. Stage F. applies inter-stage fusion to Generation and Assessment and intra-stage fusion to Training. Asynchronous execution overlaps Generation and Assessment with Training by allowing bounded weight staleness. Combined execution combines asynchronous overlap with disaggregated model placement and hybrid intra-model parallelism, following the RLHFuse execution model. Training memory assumes the Adam optimizer without activation checkpointing, accounting for parameters, gradients, optimizer states, and activations.

Model / family	Access	Scale / arch.	Post-training signal	Selected system details
General aligned large language models (LLMs)
InstructGPT [ 192 ]	Closed	1.3B, 6B, 175B dense	PPO-like (SFT + RM + PPO)	Canonical actor–reward–reference PPO loop.
GPT-4 [ 4 ]	Closed	Undisclosed	PPO-like (rule-based rewards)	Frontier-scale alignment; limited public systems detail.
Gemini 1.x [ 12 ]	Closed	Undisclosed	PPO-like (iterative RM refinement)	Repeated RM/RL cycles increase assessment cost.
Claude 3 [ 13 , 21 ]	Closed	Undisclosed	PPO-like (RLAIF / Constitutional AI)	AI-generated preference labels from written principles.
Reka [ 191 ]	Mixed	7B, 21B dense	PPO-like (multi-round RLHF)	Standard PPO-style alignment at moderate scale.

TABLE X : A comparison of representative general aligned LLMs and RLMs. The upper block lists general aligned LLMs, which primarily target instruction following, safety, preference alignment, and broad assistant quality. The lower block lists RLMs, which explicitly target reasoning trajectories, long-CoT, RLVR/verifiable rewards, search, tools, multi-agent inference, or controllable test-time computation. The “Post-training signal” column follows the taxonomy used in this paper: PPO-like methods use reward-model- or RLAIF-based policy optimization; GRPO-like methods use critic-free grouped RL/RLVR; DPO-like methods use direct preference optimization; Hybrid denotes mixed or undisclosed pipelines; Search/Agentic denotes explicit search, tools, environments, or multi-agent reasoning. Proprietary details are based on public reports and should be treated as approximate.

Intra-Model Parallelism

Inter-Model Parallelism

Framework

Remarks

AC Share

Dis- agg.

Stage Fusion

Hybrid

Async

TRL [ 231 ]

Accelerate; TP/vLLM mostly rollout; CP via FSDP2/Ulysses

ColossalChat [ 267 ]

Colossal-AI HybridParallel + Gemini; TP/PP/SP, Z1/2 and Z3

DeepSpeed-Chat [ 261 ]

Train ZeRO-2/3; Hybrid Engine uses TP mainly for rollout

OpenRLHF [ 117 ]

Train: ZeRO-3/AutoTP; rollout: vLLM TP/PP; RingAttention

TABLE XI : Comparison of public training / inference frameworks for LLM and RLM post-training under the systems taxonomy. Intra-Model Parallelism : DP=data parallel; Z1/Z2/Z3=ZeRO/FSDP-like sharding levels (optimizer / optimizer+gradients / parameters+gradients+optimizer; FSDP2 is counted under Z3); TP=tensor parallel; CP=context or long-sequence parallelism; EP=expert parallelism; PP=pipeline parallelism. Inter-Model Parallelism : AC Share=shared actor/critic parameters or value-head mode; Disagg.=separate services or device groups for rollout, reward, or training; Stage Fusion=explicit overlap or fusion across normally sequential RL stages or subtasks; Hybrid=stage-specific backends/layouts or runtime resharding/refit; Async=asynchronous or bounded-stale execution. Symbols : documented support; indirect, partial, backend-specific, or train-vs-rollout-specific support; no documented public support; undocumented or unclear in public materials.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

Mar 12, 2026

IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RL

While scaling laws guide compute allocation for LLM pre-training, analogous prescriptions for reinforcement learning (RL) post-training of large language models (LLMs) remain poorly understood. We study the compute-optimal allocation of sampling compute for on-policy RL methods in LLMs, framing scaling as a compute-constrained optimization over three resources: parallel rollouts per problem, number of problems per batch, and number of update steps. We find that the compute-optimal number of parallel rollouts per problem increases predictably with compute budget and then saturates. This trend holds across both easy and hard problems, though driven by different mechanisms: solution sharpening on easy problems and coverage expansion on hard problems. We further show that increasing the number of parallel rollouts mitigates interference across problems, while the number of problems per batch primarily affects training stability and can be chosen within a broad range. Validated across base models and data distributions, our results recast RL scaling laws as prescriptive allocation rules and provide practical guidance for compute-efficient LLM RL post-training.

Zhoujun Cheng, Yutao Xie, Yuxiao Qu +12

UC San Diego · MBZUAI-IFM · Carnegie Mellon University

Reinforcement Learning RL Post-Training Language Model Scaling Laws

Ask PDF

Oct 3, 2026

How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws

Recent studies on reinforcement learning (RL) report seemingly conflicting evidence about large language model (LLM) reasoning. Training on mathematics can improve performance in other domains, yet gains in Pass@1 can coincide with lower Pass@

N

than the base model. This raises a fundamental question: does RL expand an LLM's reasoning boundary, or merely reweight its existing reasoning space? We revisit these phenomena across Qwen and Gemma model families, showing both cross-domain gains and forgetting, while coverage at large sampling budgets increases on some tasks and decreases on others. Detailed analysis of solution traces before and after RL indicates a shift in the reasoning strategies the model employs, motivating a two-stage autoregressive policy model that separates strategy selection from problem-specific execution. Within this framework, we prove how RL's implicit bias reshapes strategy preferences, allowing gains on some tasks while suppressing strategies required by others. This mechanism can also broaden or narrow coverage at a given sampling budget even without expanding strategy support. We further provide theoretical justifications for log-sigmoid and log-linear scaling laws in RL compute, and evaluate their predictive power. Together, these results connect changes in strategy selection to cross-domain transfer, reasoning coverage, and compute scaling.

Ziheng Cheng, Yixiao Huang, Hanlin Zhu +1

University of California, Berkeley

RL for Language Model Reasoning Reinforcement Learning Generalization in RL

Ask PDF

Aug 25, 2026

Parason: Revealing Subtask and Trial Parallelism in LLM Reasoning

Scaling test-time reasoning has substantially improved the problem-solving ability of large language models (LLMs), but standard autoregressive decoding still executes long reasoning traces sequentially, creating severe latency for difficult tasks (up to days and weeks). Parallel reasoning offers a natural remedy. However, prior systems primarily focus on Subtask Parallelism, where the model learns to decompose a high-level task into smaller chunks that can be solved independently. This approach overlooks another pervasive form of parallelism: Trial Parallelism, where multiple speculative attempts explore, verify, and aggregate competing hypotheses in parallel. In this paper, we introduce Parason, which reveals and learns both forms of parallelism in LLM reasoning. Our analysis identifies Trial Parallelism as the majority of parallelizable reasoning computation (65.5% in DeepSeek-V4's reasoning steps in HLE), and it becomes increasingly dominant on hard problems. Guided by this taxonomy, Parason converts sequential reasoning traces into structured parallel trajectories with a context-free grammar, then trains models with Parallelism-Aware Group Relative Policy Optimization (PA-GRPO), whose reward jointly balances accuracy, latency, and the two parallelism ratios. At inference time, Parason executes the learned parallel structure through tool calls, translating theoretical savings to real-world wall-clock acceleration. Experiments on mathematical reasoning benchmarks including AIME24 and AIME25 show that Parason achieves an average acceleration about 1.7

\times

while maintaining competitive accuracy.

Zhengyang Zhang, Zijian Zhang, Jiaxuan Gao +4

Tsinghua University · NVIDIA · Independent Researcher +1

Parallel Reasoning in Language Models Efficient Language Model Reasoning LLM Reasoning LLM Inference Acceleration Large Reasoning Models RL for Language Model Reasoning

Ask PDF

Abstract

RL for Language Model Reasoning

Explore similar work

IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RL

How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws

Parason: Revealing Subtask and Trial Parallelism in LLM Reasoning