A Survey on RLOps for Large Language Models
Organizations: Peking University, China · Peking University; National Key Laboratory of Data Space Technology and System, China · Tsinghua University, China · University of Illinois Chicago, United States ·
Abstract
Reinforcement learning (RL) has become increasingly important for improving the reasoning, instruction-following, and task-solving capabilities of large language models (LLMs). However, scaling RL training exposes failures in distributed execution, feedback computation, and learning dynamics that can waste computational resources, slow training, and undermine model improvements. These challenges have motivated a growing body of techniques for LLM RLOps, which manages the efficiency and reliability of operations throughout the LLM reinforcement learning training lifecycle. Despite growing interest, these techniques remain scattered across research areas, with limited synthesis of their operational roles and interactions. To bridge this gap, we present a systematic survey of LLM RLOps methods. We organize existing approaches into five stages: Training Setup, Experience Collection, Reward Computation, Policy Optimization, and Model Evaluation. Within each stage, we examine operational concerns, management techniques, and evaluation evidence. Building on this taxonomy, we identify open challenges and future directions in training process observability, cross-stage coordination, generalizable RLOps, multi-run management, and RLOps for recursive self-improvement. These directions extend operational management from individual training runs to diverse training settings, shared infrastructure, and successive improvement cycles. We maintain a GitHub repository indexing relevant papers and open resources at https://github.com/zhanglingzhe0820/Awesome-RLOps.
Figures & tables
| Reference | Year | Main Scope |
| Li et al. ( Li et al., 2022 ) | 2022 | RLOps: Lifecycle management, deployment, monitoring, and safety in Open RAN |
| Wang et al. ( Wang et al., 2026g ) | 2026 | AgentOps: Monitoring, anomaly detection, root cause localization, and resolution in agent systems |
| Wang et al. ( Wang et al., 2026c ) | 2026 | Reliable self-evolution: Evolution depth, independent evaluation, and update acceptance controls |
| Gong et al. ( Gong et al., 2026 ) | 2026 | Evolutionary safety: Safety preservation and risk propagation across updates and successor systems |
| Pu et al. ( Pu et al., 2026a ) | 2026 | Trustworthy RSI: Improvement validity, constraint preservation, and effective oversight |
| Chen et al. ( Chen et al., 2026b ) | 2026 | Self-evolving agent safety: Persistent state changes, safety violations, containment, and repair |
| Framework | Training Setup | Experience Collection | Reward Computation | Policy Optimization | Model Evaluation |
| TRL ( von Werra et al., 2020 ) | 1, 2 | 2–4 | 2, 3 | 1–4 | 1, 3 |
| trlX ( Havrilla et al., 2023 ) | 2 | 2, 4 | — | 1–4 | 1, 3 |
| DeepSpeed-Chat ( Yao et al., 2023 ) | 2 | 2, 4 | — | 1–3 | — |
| NeMo-Aligner ( Shen et al., 2024 ) | 2 | 2 | — | 1–4 | 1, 3 |
| NeMo RL ( NVIDIA, 2025 ) | 2 | 1–4 | 2, 3 | 1–4 | 1, 3 |
| verl ( Sheng et al., 2025 ) | 2 | 1–4 | 2, 3 | 1–4 | 1 |
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,862 papers