RL Fine-Tuning

RL: Reinforcement Learning

Overview

Reinforcement learning fine-tuning (RFT) adapts pretrained language, vision-language, and control models by updating them from feedback on generated answers or actions rather than relying only on demonstrations. Common approaches include PPO and GRPO for outcome-based learning, RL with verifiable rewards for math and code, and preference or rubric-based methods for open-ended tasks; curricula, reward models, and step-level credit assignment help make feedback more useful. Work also extends to tool-using agents, robots, diffusion policies, and offline-to-online control, where methods such as value learning, residual policies, and flow-matching optimization address costly or sparse environment feedback. Key challenges are designing reliable rewards, balancing exploration with stable updates, reducing rollout costs, and ensuring gains transfer across tasks without forgetting prior capabilities or exploiting the evaluator.

Latest papers

242 papers10 papers in the last four weeks, down 52% on the four weeks before. 0.1% of all new papers.

Apr 21, 2025

MARFT: Multi-Agent Reinforcement Fine-Tuning

Large Language Model (LLM)-based Multi-Agent Systems (LaMAS) offer a promising approach to collaborative tasks, but applying reinforcement learning to these systems remains challenging because classical multi-agent reinforcement learning assumptions do not fully capture their dependency structures, heterogeneous roles, and autoregressive outputs. This article provides a tutorial-style overview of the transition from reinforcement learning to reinforcement fine-tuning in single-agent and multi-agent settings, highlighting the differences between classical MARL and LLM-based multi-agent optimization. We introduce Multi-Agent Reinforcement Fine-Tuning (MARFT) and formalize LaMAS optimization as a Flexible Markov Game (Flex-MG). The MARFT framework combines predecessor-conditioned policy optimization, a centralized critic, parameter-efficient adapters, and token-level credit assignment. Our token-level analysis clarifies how within-action discounting affects returns and establishes consistency with action-level returns when the token discount is one. Experiments on fixed two- to four-agent mathematical and coding workflows compare MARFT with and without policy parameter sharing against IPPO. Both variants report higher mean evaluation scores in all 24 benchmark-team-size settings, while the variant without parameter sharing shows the clearest advantage in final training reward. Role-specification ablations examine how prescribed roles affect training. We conclude with application perspectives and open challenges, including dynamic workflows, sample efficiency, and system-level reliability. Code: https://github.com/jwliao-ai/MARFT.
Apr 7, 2025

Efficient Reinforcement Finetuning via Adaptive Curriculum Learning

Reinforcement finetuning (RFT) has shown great potential for enhancing the mathematical reasoning capabilities of large language models (LLMs), but it is often sample- and compute-inefficient, requiring extensive training. In this work, we introduce AdaRFT (Adaptive Curriculum Reinforcement Finetuning), a method that significantly improves the efficiency of RFT through adaptive curriculum learning. AdaRFT dynamically adjusts the difficulty of training problems based on the model's recent reward signals, ensuring that the model consistently trains on tasks that are challenging but solvable. This adaptive sampling strategy accelerates learning by maintaining an optimal difficulty range, avoiding wasted computation on problems that are too easy or too hard. AdaRFT requires only a lightweight extension to standard RFT algorithms like Proximal Policy Optimization (PPO), without modifying the reward function or model architecture. Experiments on competition-level math datasets demonstrate that AdaRFT improves convergence efficiency and reasoning performance. Given problem-level difficulty annotations, AdaRFT reduces RFT training time by up to 2 times across data distributions and model scales, offering a more scalable and effective RFT framework.