LLM-Guided RL

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

8 papers in the last four weeks, up 60% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 76

All topics

Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients
Jun 16, 2026Byung-Kwan Lee, Ximing Lu, Shizhe Diao +6Curriculum RL LLM-Guided RL
NVIDIA
From Trainee to Trainer: LLM-Designed Training Environment for RL with Multi-Agent Reasoning
Jun 16, 2026Chao Chen, Chengzu Li, Zhiwei Li +2Reinforcement Learning LLM-Guided RL
1LARK, HKUST (GZ) · University of Cambridge · 3HKUST
When in Doubt, Plan It Out: Committed Small Language Model Deliberation for Reactive Reinforcement Learning
Jun 15, 2026Nathan Gavenski, Juarez Monteiro, Francisco Galuppo +2LLM Planning LLM-Guided RL
Department of Informatics, King’s College London, London, United Kingdom · Kunumi Institute, Belo Horizonte, Brazil
Phase-Aware Guidance Injection for Recurrent MAPPO in Assembly-Line Disruption Recovery
Jun 15, 2026Xin Huang, Yongcai Wang, Fengyi Zhang +3Combinatorial Optimization LLM-Guided RL
School of Information, Renmin University of China, Beijing 100872, China · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · The Information Science Academy, China Electronics Technology Group Corporation, Beijing 100043, China +2
ReflectiChain: Epistemic Grounding in LLM-Driven World Models for Supply Chain Resilience
Jun 9, 2026Jia LuoConstrained RL LLM-Guided RL
School of Foreign Languages, Huazhong University of Science and Technology, Wuhan 430074, China.
GIFT: LLM-Guided State-Reward Interface for Financial Reinforcement Learning
Jun 7, 2026Yanyan Wu, Boyi Zhang, Yanlin Liu +10Reinforcement Learning Reward Shaping
East China University of Science and Technology · University of Science and Technology of China · Southwestern University of Finance and Economics +5
ROSUM-MCTS: Monte Carlo Tree Search-Inspired HDL Code Summarization with Structural Rewards
Jun 6, 2026Prashanth Vijayaraghavan, Charles Mackin, Luyao Shi +6Electronic Design Automation LLM-Guided RL
IBM Research San Jose, CA, USA
Uncertainty-Aware LLM-Guided Policy Shaping for Sparse-Reward Reinforcement Learning
Jun 4, 2026Ujjwal Bhatta, Utsabi Dangol, Sumaly Bajracharya +2Reinforcement Learning LLM-Guided RL
USD AI Research, Department of Computer Science University of South Dakota Vermillion, SD 57069, USA
SocraticPO: Policy Optimization via Interactive Guidance
Jun 3, 2026Zirui Liu, Jie Ouyang, Qi Liu +8RL for Language Model Reasoning LLM-Guided RL
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · iFLYTEK AI Research (Central China), iFLYTEK Co., Ltd
AgenticRL: Self-Refining Agentic Reinforcement Learning for Vision-Conditioned UAV Navigation
Jun 2, 2026Roohan Ahmed Khan, Yasheerah Yaqoot, Amir Atef Habel +2RL for Robotics Agentic RL
When are LLMs Sufficient Policy Optimizers for Sequential RL Tasks?
May 29, 2026Stephane Hatgis-Kessell, Emma BrunskillLanguage Model-Based Control Policy Optimization
Department of Computer Science, Stanford University
LLM-ALSO: LLM-Driven Adaptive Learning-Signal Optimization for Multi-Agent Reinforcement Learning
May 28, 2026Xiaoguang Wu, Zhi Zheng, Hui XiongReward Shaping LLM-Guided RL
University of Science and Technology of China · Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China · 2Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)
When LLM Reward Design Fails: Diagnostic-Driven Refinement for Sparse Structured RL
May 27, 2026Youting Wang, Yuan Tang, Bowen Liu +2Reward Shaping LLM-Guided RL
SARAD: LLM-Based Safety-Aware Hybrid Reinforcement Learning with Collision Prediction for Autonomous Driving
May 27, 2026Kangyu Wu, Peng Cui, Guoxi Chen +1Autonomous Driving Traffic Accident Anticipation
School of Automation, Southeast University, Nanjing, China · Key Laboratory of Measurement and Control of Complex Systems of Engineering, Ministry of Education, Nanjing, China
OccuReward: LLM-Guided Occupant-Centric Reward Shaping for Demographic Equity in Grid-Interactive Buildings
May 27, 2026Shadmehr Zaregarizi, Khashayar YavariReward Shaping Algorithmic Fairness
Politecnico di Torino Turin, Italy
Hide to Guide: Learning via Semantic Masking
May 24, 2026Ruitao Liu, Qinghao Hu, Alex Hu +6Reward Hacking RL for Language Model Reasoning
1MIT · 2NVIDIA
LLM-Guided Communication for Cooperative Multi-Agent Reinforcement Learning
May 18, 2026Sangjun Bae, Yisak Park, Sanghyeon Lee +1LLM-Guided RL Multi-Agent Systems
Graduate School of Artificial Intelligence, UNIST, Ulsan, South Korea.
PriorZero: Bridging Language Priors and World Models for Decision Making
May 12, 2026Junyu Xiong, Yuan Pu, Jia Tang +1LLM-Guided RL Latent World Models
University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · Nanjing University of Aeronautics and Astronautics +1
Learning Agentic Policy from Action Guidance
May 12, 2026Yuxiang Ji, Zengbin Wang, Yong Wang +6Agentic RL Reinforcement Learning
1Xiamen University · 2AMAP, Alibaba Group · Work done during internship at AMAP, Alibaba Group +1
EvoNav: Evolutionary Reward Function Design for Robot Navigation with Large Language Models
May 12, 2026Zhikai Zhao, Chuanbo Hua, Federico Berto +4Evolutionary Optimization Robot Navigation
1KAIST · 2Radical Numerics · 3UC Riverside +1
From Passive Reuse to Active Reasoning: Grounding Large Language Models for Neuro-Symbolic Experience Replay
May 10, 2026Yanan Xiao, Yixiang Tang, Zechen Feng +3Reinforcement Learning Experience Replay
1Affiliation 1 · 2Affiliation 2 · 3Affiliation 3
AIPO: Learning to Reason from Active Interaction
May 8, 2026Junnan Liu, Linhao Luo, Thuy-Trang Vu +1RL for Language Model Reasoning LLM-Guided RL
Department of Data Science and AI, Faculty of Information Technology, Monash University, Australia
Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias
May 8, 2026Rahaf Abu Hara, Vaibbhav Murarri, Claudio ZitoPolicy Optimization LLM-Guided RL
Laboratory of AI and Robotics Research (LAIRR) Heriot-Watt University Dubai, United Arab Emirates
Reward Shaping and Action Masking for Compositional Tasks using Behavior Trees and LLMs
May 7, 2026Nicholas Potteiger, Ankita Samaddar, Taylor T. Johnson +1Reinforcement Learning Reward Shaping
Vanderbilt University, Nashville, TN
Closing the Loop: Unified 3D Scene Generation and Immersive Interaction via LLM-RL Coupling
May 7, 2026Anh H. Vo, Sungyo Lee, Phil-Joong Kim +2Virtual Reality LLM-Guided RL
Department of Computer Engineering, Sejong University, Seoul, Republic of Korea
LANTERN: LLM-Augmented Neurosymbolic Transfer with Experience-Gated Reasoning Networks
May 6, 2026Mahyar Alinejad, Yue Wang, Amrit Singh Bedi +1Reinforcement Learning LLM-Guided RL
Department of Electrical and Computer Engineering, University of Central Florida, Orlando, Florida, USA · Department of Computer Science, University of Central Florida, Orlando, Florida, USA
Discovering Reinforcement Learning Interfaces with Large Language Models
May 5, 2026Akshat Singh Jaswal, Ashish Baghel, Paras ChopraReinforcement Learning LLM-Guided RL
Lossfunk
ARGUS: Policy-Adaptive Ad Governance via Evolving Reinforcement with Adversarial Umpiring
May 4, 2026Deyi Ji, Junyu Lu, Xuanyi Liu +7Non-Stationary RL Online Advertising
Tencent · Dalian University of Technology · Peking University +2
Model-Based Proactive Cost Generation for Learning Safe Policies Offline with Limited Violation Data
May 2, 2026Ruiqi Xue, Lei Yuan, Kainuo Cheng +2LLM-Guided RL Offline RL
National Key Laboratory for Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University · Polixir Technologies +2
PPO guided Agentic Pipeline for Adaptive Prompt Selection and Test Case Generation
May 1, 2026Gourisetty Venkata Sai Koushik, Dama Aditya, Mahankali Harish Sai +3Automated Software Testing LLM-Guided RL
Department of Computer Science and Engineering, SRM University AP

Jun 16, 2026, cs.CLJ/K move · Enter open · S save

Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients

Byung-Kwan Lee, Ximing Lu, Shizhe Diao, Minki Kang+5

NVIDIA

Curriculum RL LLM-Guided RL RL Post-Training Teacher-Student Learning RL for VLMs

Open paper AskPDF arXiv

Knowledge distillation transfers a teacher's competence to a small student but is brittle in the small-student regime: forcing the student to imitate logits from a much larger teacher hurts generalization on benchmark families beyond the training corpus. Reinforcement learning (RL) avoids logit imitation by training on the student's own rollouts. However, on questions where every rollout fails - yielding zero advantage and being silently discarded - injecting a stronger teacher's response into the policy gradient breaks the on-policy assumption and induces drift. We introduce Zone of Proximal Policy Optimization (ZPPO), inspired by Vygotsky's zone of proximal development. ZPPO keeps the teacher inside the prompt rather than the policy gradient. On hard questions, where the student's mean rollout accuracy is below half, ZPPO constructs two reformulated prompts. A Binary Candidate-included Question (BCQ) pairs one correct teacher response with one incorrect student response as anonymized candidates the student uses as references. A Negative Candidate-included Question (NCQ) aggregates the student's wrong rollouts into a single prompt to surface their shared failure modes. A prompt replay buffer recirculates each hard question until it either graduates - the student's mean rollout accuracy on it reaches half or more - or is FIFO-evicted under finite capacity, amplifying BCQ and NCQ inside the student's current zone of proximal development. We post-train Qwen3.5 students at four scales (0.8B-9B) as vision-language models with a 27B teacher and evaluate them on a 31-benchmark suite (16 VLM, 10 LLM, 5 Video); ZPPO outperforms off/on-policy distillation and GRPO, with the largest gains at the smallest scale.

Momentum

Related topics

Latest papers 76