Long-horizon agentic reinforcement learning (RL) requires both efficient rollout generation and training on long sequences. However, the memory requirements and model-specific operators of Qwen3.8-Flash-Next make this difficult with existing open-source implementations. To resolve these limitations, we introduce FlashNext-RL, a model-specific RL training optimization framework based on prime-rl, with vLLM for software engineering (SWE) rollout generation. Our main focus is BF16 training of the full 48-layer model on a single 8xB200 node, with the first two layers, the n-gram table, the sparse-attention indexer and the multi-token prediction head frozen. We optimize host offload, gradient accumulation and the kernels for mixture-of-experts, sparse attention and the gated residual stream. In the full-model development study, non-padding training throughput increases from approximately 395 tokens/s/GPU in the initial 32K-token configuration to 4,438 tokens/s/GPU in the optimized 131K-token configuration, with hardware matmul throughput (aka without considering the cost of attention components) increasing from approximately 20 to 213 TFLOPs/GPU. A controlled cumulative study on an 8-layer model further quantifies the contributions of individual optimizations. Together with three inference nodes, the system supports a live SWE RL run with a target batch of 1,024 trainable rollouts and a 131,072-token context limit. Over steps 5-15, an iteration takes 22.4 minutes on average, with bounded entropy and trainer-sampler mismatch KL. These results highlight the importance of tailoring RL training to the model's architecture, memory layout and kernel requirements, rather than relying only on general-purpose RL infrastructure.
Figures & tables
Sessions/GPU
TP2 + EP (KV 1.76M)
TP4 + EP (KV 5.95M)
TP8 + EP (KV 8.41M)
7
835
715
–
10.5
985
814
–
17.5
840 (prefix hit 0.06)
943
658
26
771 (318 preemptions)
1,091
567
35
783 (preempting)
788
530
Table 1: BF16 engine on B200 GPUs uses replayed SWE rollouts and MTP3. Output tokens/s per GPU; “KV” is the KV-cache capacity per engine in tokens. The best measured configuration per row is in bold.
Optimizer
Bytes/param
Host for parameters
+ n-gram table ( ∼ 95.4 GiB)
AdamW
16
1,922 GiB
2,018 GiB (does not fit)
Muon budget
12
1,442 GiB
1,537 GiB (fits)
Table 2: Approximate host-memory budget for 129B parameters before layer freezing, on a node with about 1,937 GiB available. The Muon row assumes one fp32 moment per parameter for budgeting; the implementation uses AdamW for the remaining parameter groups. Final measured state sizes are in Table 13 .
CP 1, 32K
CP 4, 131K
Offload
Muon step
Step
Muon step
Step
REAL tok/s/GPU
Pageable
∼ 75 s
–
66 s
349 s
∼ 426
PinnedArena
20–28 s
161–179 s
28 s
288 s
∼ 516
+ pinned gradients
12.4 s
121–131 s
12.5 s
242 s
571–638
Table 3: Host offload path, full model on one node. Left: CP 1 with 32K bins (about 0.64M tokens per step). Right: CP 4 with a 131K bin.
Gradient path
Micro step
Step
GPU peak (alloc.)
REAL tok/s/GPU
FSDP2 host accumulation
∼ 51 s
242 / 243 s
118.5 GiB
571 / 589
GPU-resident expert gradients
13–16 s
101 / 91 s
149.3 GiB
1,368 / 1,571
Table 4: GPU-resident expert gradients, full model, CP 4, 131K bin, 16-sample batches (about 1.1M tokens per step), steps 2 and 3.
Tokens/rank
DeepEP
MoK
Speedup
16K
32.2 ms
20.7 ms
1.56 ×
64K
130.1 ms
67.6 ms
1.92 ×
131K
294.0 ms
131.5 ms
2.24 ×
Table 5: MoE layer at EP 8. Left: forward + backward, DeepEP with grouped GEMMs vs. MoK (upstream build: intermediate size padded to 768, 24/28 communication SMs). Right: MoK at 32K tokens per rank, forward + recompute + backward.
Sequence length
PyTorch
cuDNN fused
8K
1.7 ms
0.2 ms
64K
35.3 ms
6.1 ms
131K
123.8 ms
22.5 ms
Table 6: QSA indexer + top-k, one B200, real layer inputs.
Implementation
16K
64K
dK/dV kernel at 64K
cuDNN DSA (MLA kernels, padded heads)
53.7 ms
219.8 ms
–
GQA-native Triton
33.9 ms
164.5 ms
97 ms
+ TMA gathers in dK/dV
29.0 ms
138.5 ms
76.2 ms
+ CuTe-DSL dK/dV (final)
20.7 ms
106.4 ms
46.7 ms
Table 7: QSA attention, forward + backward per layer, real layer-7 selections, one sequence (one B200).
Eager
torch.compile
Fused
16K
9.55 ms
4.40 ms
3.42 ms
131K
75.17 ms
34.74 ms
17.84 ms
Kept at 131K
15.79 GiB
5.79 GiB
0.71 GiB
Table 8: One gated-residual read + inject, forward + backward, one B200. “Kept” is the memory saved for the backward at 131K.
T
CPU, 1 thread
CPU, 16 threads
UVA on GPU
16K
40–42 ms
14 ms
5.2 ms
131K
395–464 ms
117–131 ms
42 ms (no host cores)
Table 9: N-gram row gather for T tokens ( [T,16×160] BF16 rows from the frozen table), one B200.
Figure 1: Cumulative trainer optimizations of FlashNext-RL on the 8-layer model: REAL tokens/s/GPU (top) and TFLOPs/GPU (bottom). Real rollouts, one node, 131K bins.
Removed
REAL tok/s/GPU
Change
MoK, back to DeepEP
13,699
− 22.4%
Fused gated residual and glue
14,191
− 19.6%
GQA-native QSA, back to cuDNN DSA
15,028
− 14.9%
UVA n-gram gather, back to CPU
16,058
− 9.0%
MoK communication SMs, back to 24/28
16,845
− 4.6%
CuTe dK/dV, back to Triton TMA
16,966
− 3.9%
Table 11: Removing one optimization at a time from the final 8-layer stack (L14; 17,651 REAL tokens/s/GPU on the same node).
Configuration
REAL tok/s/GPU
HW TFLOPs/GPU
First full-model step (CP 1, 32K bins, pageable offload)
∼ 395
20
CP 4, 131K bin, pageable offload †
∼ 426
21
+ PinnedArena †
∼ 516
26
+ GPU-resident expert gradients †
1,368–1,571
68–78
+ GQA-native Triton QSA (real steps from here on)
2,297–2,333
114–116
+ MoK, static placement, pre-warm, GDN transpose drop
2,878–3,090
143–154
Table 12: Full 48-layer RL trainer development on one node. The initial row uses CP 1 and 32K bins; subsequent rows use CP 4 and 131K bins. Rows marked † use 16-sample batches (about 1.1M tokens); later rows use the earlier rollout batches of 7.1–7.6M tokens (median sample 66K). HW TFLOPs count matmuls only (49.8 GFLOP per token before freezing, 48.1 after, on these steps). Configurations and batch sizes vary, so the endpoints are not a controlled speedup comparison.
Figure 2: The RL setup of FlashNext-RL: one training node and three inference nodes, with the orchestrator on the first inference node. Blue: NCCL weight broadcast; orange: rollouts and training batches (with routed experts); gray: checkpoints.
Structure
Size per rank
Where
fp32 parameter shards (incl. the 2 frozen layers)
58.9 GiB
PinnedArena
StreamingMuon moments
56.8 GiB
PinnedArena
fp32 gradients
56.1 GiB
gradient arena
n-gram table (BF16)
95.4 GiB per node
shared tmpfs mapping
Table 13: Host memory per trainer rank (8 ranks on the training node), final configuration.
Optimization
Problem
Saved
PinnedArena (Section 3.2.1 )
Caching host allocator rounds up to 2k ; in-place registration crashes
fits (default pinned offload killed at 1,887 GiB)
Gradient arena
Pinned gradients rounded to 2k and never freed
∼ 105 GiB per node
Re-freeze after EP
Frozen layers’ experts got gradients and moments
37 GiB per node
Lazy optimizer state
450 GiB allocated during the sandbox startup burst
450 GiB off the startup peak
Batch copies
Two batches at the step boundary; int32 routes on the host; orchestrator copies
∼ 300 GB (int32 routes) + one batch + ∼ 37 GB per buffered batch
CP host shard
Full routed experts on every CP rank
∼ 110 GB per node
Table 14: Memory optimizations on the training node.
Figure 3: The full model on one node as the trainer’s memory optimizations are added: trainer host memory (PSS), GPU memory and StreamingMuon step time. FSDP2’s default pinned offload was killed before finishing a step.
Exporter
DCP
HF weights (on path)
Checkpoint-step overhead
Streamed per layer
71 s
236 s
307 s
From the broadcast, 4 buffers
70 s
32.3 s ∗
102 s
Table 15: Checkpoint cost on the training path. ∗ The background write finished 132.5 s after it started.
Figure 4: Rollout lengths in the real SWE RL run. Left: distribution over steps 5–15 (15,323 collected rollouts), with its median, p90 and p99. Right: median, mean and p90 per step (step 2’s traces were not saved).
Figure 5: Entropy and trainer–sampler mismatch KL per step in the real SWE RL run.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Large language model (LLM) agents increasingly undertake extreme-long (xlong) horizon tasks, where a single execution can span hours, hundreds of model--environment interactions, and nearly 1M tokens per rollout. Applying online reinforcement learning (RL) to such executions poses two fundamental challenges: (1) severe execution variance and prolonged rollout delays cause massive GPU idling; and (2) complex non-linear branching generates massive trajectory redundancy, crippling training efficiency. To address these, we presents QwenGyre, an end-to-end framework for xlong-horizon online RL. QwenGyre elastically reallocates GPUs between rollout and training without interrupting live executions, while its trajectory processor reconstructs branching histories, scores partial progress, and deduplicates redundant paths to bound training costs. Scaled to our flagship model, Qwen~3.8 2.4T, with 700K tokens per rollout, QwenGyre yields a 6.0% absolute gain on NL2RepoBench (52.5% → 58.5%) in 48 steps. Across our evaluations on diverse domains of training datasets, QwenGyre delivers up to 1.85× and 1.78× speedups over Colocate and Async, respectively.
Weiqi Wang, Yuxin Zhou, Mouxiang Chen +9
Alibaba Token Hub, Alibaba Group · University of Science and Technology of China · Tsinghua University
Large language model (LLM) reinforcement learning (RL) pipelines are often bottlenecked by rollout generation, making end-to-end training slow. Recent work mitigates this by running rollouts with quantization to accelerate decoding, which is the most expensive stage of the RL loop. However, these setups destabilize optimization by amplifying the training-inference gap: rollouts are operated at low precision, while learning updates are computed at full precision. To address this challenge, we propose QaRL (Rollout Alignment Quantization-Aware RL), which aligns training-side forward with the quantized rollout to minimize mismatch. We further identify a failure mode in quantized rollouts: long-form responses tend to produce repetitive, garbled tokens (error tokens). To mitigate these problems, we introduce TBPO (Trust-Band Policy Optimization), a sequence-level objective with dual clipping for negative samples, aimed at keeping updates within the trust region. On Qwen3-30B-A3B MoE for math problems, QaRL outperforms quantized-rollout training by +5.5 while improving stability and preserving low-bit throughput benefits.
Hao Gu, Hao Wang, Jiacheng Liu +9
The Hong Kong University of Science and Technology · City University of Hong Kong · Zhejiang University
A growing gap separates inference context lengths from RL post-training: inference systems are approaching million-token contexts, while post-training workloads often remain at 256K tokens or below and rely on length generalization at deployment. The gap is especially important for AI agents, whose observations, tool outputs, documents, and prior decisions accumulate over long trajectories. LongStraw is an architecture-aware execution stack for million-token RL post-training under a fixed GPU budget, instantiated with Group Relative Policy Optimization (GRPO). It evaluates the shared prompt without autograd, retains only model-specific state needed by later tokens, and replays short response branches one at a time, reducing the live training graph at the cost of additional replay time. We implement it for the hybrid recurrent and full-attention Qwen3.6-27B and the compressed-attention mixture-of-experts GLM-5.2. On eight H20 GPUs, LongStraw completes grouped Qwen scoring and response backward at 2.1M positions for groups of 2 and 8; increasing the group size adds only 0.21 GB of peak allocated memory, while a separate stress test reaches 4.46M positions. On 32 H20 GPUs, we validate the end-to-end LongStraw execution path for a 2.1M-token prompt across all 78 layers of GLM-5.2. These experiments establish execution capacity rather than complete training correctness because the captured prompt state is detached and some distributed forward and gradient composition paths remain incomplete.