Reinforcement Learning from Human Feedback (RLHF) has been widely applied to Large Language Model (LLM) post-training to align model outputs with human preferences. Recent models, such as DeepSeek-R1, have also shown RLHF's potential to improve LLM reasoning on complex tasks. In RL, inference and training co-exist, creating dynamic resource demands throughout the workflow. Compared to traditional RL, RLHF further challenges training efficiency due to expanding model sizes and resource consumption. Several RLHF frameworks aim to balance flexible abstraction and efficient execution. However, they rely on serverful infrastructures, which struggle with fine-grained resource variability. As a result, during synchronous RLHF training, idle time between or within RL components often causes overhead and resource wastage. To address these issues, we present RLHFless, the first scalable training framework for synchronous RLHF, built on serverless computing environments. RLHFless adapts to dynamic resource demands throughout the RLHF pipeline, pre-computes shared prefixes to avoid repeated computation, and uses a cost-aware actor scaling strategy that accounts for response length variation to find sweet spots with lower cost and higher speed. In addition, RLHFless assigns workloads efficiently to reduce intra-function imbalance and idle time. Experiments on both physical testbeds and a large-scale simulated cluster show that RLHFless achieves up to 1.35x speedup and 44.8% cost reduction compared to the state-of-the-art baseline.
Figures & tables
Figure 1. RLHF ’s dataflow in one training step, including the generation phase, preparation phase, and learning phase. Some details, like the use of the critic model, can vary depending on the specific algorithm used.
Figure 2. (a) RLHF ’s staged workflow causes idle time between components. (b) Idle components and (c) repeated calculation in RLHF lead to resource wastage.
Figure 3. Existing generation strategies ( Sheng et al., 2025 ; Hu et al., 2024 ; Zheng et al., 2024 ; Kwon et al., 2023 ) for sampling-heavy RLHF workloads, including (a) iterative generation, which introduces additional latency, and (b) parallel generation, which leads to (c) unnecessary KV recalculation.
Figure 4. Dynamic resource demands caused by: (a) significantly varying response lengths across and within different types of datasets, such as mathematical problems ( Maxwell Jia, 2025 ) , general scientific questions ( Rein et al., 2024 ) , and coding tasks ( Jain et al., 2024 ) ; (b) changing response lengths during RLHF training.
Figure 7. (a) RLHFless ’s prompt assignment design, and (b) the potential benefits that can be achieved under GRPO training with Qwen2.5-3B ( and: et al., 2025 ) model and GSM8k ( Cobbe et al., 2021 ) dataset.
Figure 8. RLHFless ’s cut-and-migrate strategy. RLHFless cuts unfinished underestimated responses and migrates them to available decode actors with open slots, which have been released by completed, overestimated responses.
Figure 9. RLHFless ’s locality-aware actor placement.
Figure 11. RLHFless ’s ablation study (DP: D eduplicated P refill; AS: A ctor S caling; PA: P rompt A ssignment). Each point shows the GPU cost and execution time of one training step. Points closer to the lower-left corner indicate lower cost and faster execution. Subfigure (a) shows RLHFless with all core features disabled. From (b) to (d), we enable one additional core feature at a time, illustrating the incremental benefits of each design.
Figure 12. Effectiveness of RLHFless ’s prompt assignment. (DP: D eduplicated P refill; AS: A ctor S caling; PA: P rompt A ssignment.)
Figure 13. Effectiveness of RLHFless ’s (a) actor scaling and (b) deduplicated prefill. (DP: D eduplicated P refill; AS: A ctor S caling; PA: P rompt A ssignment.)
Figure 14. Sensitivity analysis of RLHFless ’s response length prediction.
Figure 15. RLHFless hyperparameter sensitivity: (a) EWMA window size for length prediction, (b) cut-and-migrate threshold τ , and (c) actor scaling weight λ .
Figure 16. RLHFless scalability analysis. (a) Execution time for response generation with lengths from 2k to 8k tokens, along with scheduling overhead, under different cluster sizes. (b) Average normalized performance improvement from large-scale simulation experiments.
Figure 17. RLHFless ’s latency breakdown in a training step, including the preparation, generation, and learning phases, as well as deduplication latency, scaling latency (response-length prediction and cost/time estimation), and I/O latency.
Anhui University · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · University of Science and Technology of China +2