From AI Assistant to AI Scientist: Autonomous Discovery of LLM-RL Algorithms with LLM Agents

Sirui Xia; Yikai Zhang; Aili Chen; Siye Wu; Siyu Yuan; Yanghua Xiao

cs.CLMar 25, 2026

From AI Assistant to AI Scientist: Autonomous Discovery of LLM-RL Algorithms with LLM Agents

Authors: Sirui Xia, Yikai Zhang, Aili Chen, Siye Wu, Siyu Yuan, Yanghua Xiao

Organizations: Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University · School of Data Science, Fudan University

Abstract

Discovering improved policy optimization algorithms for language models remains a costly manual process requiring repeated mechanism-level modification and validation. Unlike simple combinatorial code search, this problem requires searching over algorithmic mechanisms tightly coupled with training dynamics while reusing empirical evidence across iterations. We propose POISE, a closed-loop framework for automated discovery of policy optimization algorithms for language models. POISE maintains a structured, genealogically linked archive linking proposals, executable implementations, standardized evaluations, and natural-language reflections to support evidence-driven iteration. In mathematical reasoning experiments starting from GRPO, POISE evaluates 64 candidate algorithms and discovers improved mechanisms, including analytic-variance scaling and validity masking. The best variant improves weighted Overall from 47.8 to 52.5 (+4.6) and increases AIME25 pass@32 from 26.7% to 43.3%, demonstrating the feasibility of automated policy optimization discovery while supporting interpretable design principles.

Figures & tables

Figure 1: Overview of POISE . Phase I (Proposal Generation) generates candidate algorithms from historical evidence and prior knowledge; Phase II (Implementation, Verification, and Evaluation) implements proposals under shared interfaces, verifies fidelity to the intended algorithm, and evaluates them with a standardized protocol; Phase III (Reflective Analysis and Archive Update) interprets the results and updates the archive.

Figure 2: Training dynamics in the main experiment for GRPO and representative evolved variants. The plots compare entropy, reward, and response length over training.

Figure 3: Accuracy-length trade-offs under the length-compression constraint.

Figure 4: Performance frontier vs. tree depth in the base-branch lineage; dots are algorithms. Solid: cumulative best Overall; dashed: mean Overall of top-3 at each depth. Despite exploratory failures at depth, the frontier improves beyond the root.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Evolutionary lineage of GRPO variants in the main run. Nodes denote algorithms, arrows denote parent-child inheritance, and node color encodes the weighted Overall (lighter is lower).

Automated Algorithm Discovery

Automated algorithm discovery develops methods that generate and test executable algorithms for optimization, science, software, and machine learning.

See all 170 papers

A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensions
Jun 15, 2026Jianghan Shen, Siqi Luo, Yue Li +9RL for Language Models Group Relative Policy Optimization
1Nanjing University · 2Shanghai Artificial Intelligence Laboratory · 3Shanghai Jiao Tong University +4
RLMOpt: Adaptive Prompt Optimization via Recursive Language Models
Aug 11, 2026Subhash Bangalore Satheesha, Nirvik Pande, Deepthi Duddempudi +1Large Language Model-Guided Optimization LLM Agents
Autonomize AI · Carnegie Mellon University
Vector Policy Optimization: Training for Diversity Improves Test-Time Search
May 21, 2026Ryan Bahlous-Boldi, Isha Puri, Idan Shenfeld +6Multi-Objective Reinforcement Learning Reinforcement Learning
1MIT · 2Improbable AI Lab · 4Sakana AI +1

Jun 15, 2026J/K move · Enter open · S save

A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensions

Jianghan Shen, Siqi Luo, Yue Li, Jiyao Liu+8

1Nanjing University · 2Shanghai Artificial Intelligence Laboratory · 3Shanghai Jiao Tong University+4

RL for Language Models Group Relative Policy Optimization Policy Gradient Methods Proximal Policy Optimization Policy Learning

Open paper AskPDF arXiv

Policy gradient algorithms for language models optimize the same objective

J(θ) = \mathbb{E}*{τ\sim p*θ(τ)}[R(τ)]

, which has exactly two factors: the trajectory probability

p_θ(τ)

and the reward

R(τ)

. Every method from REINFORCE to PPO to GRPO and their descendants modifies one or both factors to address a specific failure in the preceding formulation. Existing surveys organize these methods by domain or chronology, which obscures the rationale behind each design choice and the precise location of its intervention within the gradient estimator. This survey revisits the landscape of LLM policy optimization from

J(θ)

on first principles and uses the trajectory side, induced by

p_θ(τ)

, and the reward side, induced by

R(τ)

, as the two axes along which methods are located. It covers the path from REINFORCE and PPO to GRPO, as well as post-GRPO variants, Agentic RL, and GRPO-OPD. The resulting framework is unified, diagnostic, and extensible: it analyzes methods from a shared objective, identifies which side each method modifies and why, and applies the same trajectory and reward axes across these settings. Across these settings, the framework also exposes compound failures that no single-side fix resolves and that therefore require joint design of the trajectory side and the reward side. The boundary cases and coupled failures identified by this map mark where existing solutions run out and provide a principled starting point for designing the next generation of LLM policy optimization algorithms.

Abstract

Automated Algorithm Discovery

Explore similar work

A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensions

RLMOpt: Adaptive Prompt Optimization via Recursive Language Models

Vector Policy Optimization: Training for Diversity Improves Test-Time Search