cs.AIMar 2, 2026

LLM-assisted Semantic Option Discovery for Facilitating Adaptive Deep Reinforcement Learning

Authors: Chang Yao, Jinghui Qin, Kebing Jin, Hankz Hankui Zhuo

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Despite achieving remarkable success in complex tasks, Deep Reinforcement Learning (DRL) is still suffering from critical issues in practical applications, such as low data efficiency, lack of interpretability, and limited cross-environment transferability. However, the learned policy generating actions based on states are sensitive to the environmental changes, struggling to guarantee behavioral safety and compliance. Recent research shows that integrating Large Language Models (LLMs) with symbolic planning is promising in addressing these challenges. Inspired by this, we introduce a novel LLM-driven closed-loop framework, which enables semantic-driven skill reuse and real-time constraint monitoring by mapping natural language instructions into executable rules and semantically annotating automatically created options. The proposed approach utilizes the general knowledge of LLMs to facilitate exploration efficiency and adapt to transferable options for similar environments, and provides inherent interpretability through semantic annotations. To validate the effectiveness of this framework, we conduct experiments on two domains, Office World and Montezuma's Revenge, respectively. The results demonstrate superior performance in data efficiency, constraint compliance, and cross-task transferability.

Figures & tables

Figure 1: Exploration of Natural Language Limitations Graph
Figure 2: The LLM-SOARL framework
Figure 3: Generate semantic labels using LLMs
Figure 4: The Office World Result
Figure 5: The Montezuma’s Revenge Result
More in this topic

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Explore similar work

CardsList
Apr 25, 2026

Bridging Reasoning and Action: Hybrid LLM-RL Framework for Efficient Cross-Domain Task-Oriented Dialogue

Cross-domain task-oriented dialogue requires reasoning over implicit and explicit feasibility constraints while planning long-horizon, multi-turn actions. Large language models (LLMs) can infer such constraints but are unreliable over long horizons, while Reinforcement learning (RL) optimizes long-horizon behavior yet cannot recover constraints from raw dialogue. Naively coupling LLMs with RL is therefore brittle: unverified or unstructured LLM outputs can corrupt state representations and misguide policy learning. Motivated by this, we propose Verified LLM-Knowledge empowered RL (VLK-RL), a hybrid framework that makes LLM-derived constraint reasoning usable for RL. VLK-RL first elicits candidate constraints with an LLM and then verifies them via a dual-role cross-examination procedure to suppress hallucinations and cross-turn inconsistencies. The verified constraints are mapped into ontology-aligned slot-value representations, yielding a structured, constraint-aware state for RL policy optimization. Experiments across multiple benchmarks demonstrate that VLK-RL significantly improves generalization and robustness, outperforming strong single-model baselines on long-horizon tasks.

Yangyang Zhao, Linfan Dai, Li Cai +2

School of Computer Science and Technology, Changsha University of Science and Technology · School of Computer Science and Engineering, Central South University · Text Computing and Cognitive Intelligence MOE Engineering Research Center, Guizhou University +2

Constrained RLLLM-Guided RLConversational AgentsLong-Horizon LLM Agents
AskPDF
Jun 21, 2026

Learning at the Right Pace: Adaptive Data Scheduling Improves LLM Reinforcement Learning

Large Language Models (LLMs) achieve remarkable reasoning capabilities through reinforcement learning (RL) post-training. However, existing RL post-training commonly relies on uniform data sampling, which ignores the semantic structure of the training data and the evolving capability of the training policy. To address these limitations, we propose Adaptive Data Scheduling (ADS), a dual-level data scheduling framework that combines adaptive sampling across semantic clusters with policy-boundary sample selection. At the cluster level, ADS organizes samples according to semantic patterns and adapts the inter-cluster sampling distribution to prioritize clusters within the policy's current reach. At the sample level, ADS tracks prompts near the policy's capability boundary within each cluster to obtain informative relative advantages. Experimental results across three LLMs and seven reasoning benchmarks demonstrate that ADS improves average accuracy by 5.5% over Group Relative Policy Optimization (GRPO). Notably, ADS consistently improves RL methods with different objective designs, highlighting its potential as a general data scheduling strategy. The source code is available at: https://github.com/Richard-zrx/ADS.

Zicheng Xu, Ruixuan Zhang, Yu-Neng Chuang +7

Johns Hopkins University · Rice University · University of Haifa +2

Reinforcement LearningRL for Language Model ReasoningCurriculum RLRL Post-Training
AskPDF
Jun 2, 2026

An LLM-Explainable DRL Framework for Passenger-Directed Autonomous Driving

Autonomous vehicles offer the potential for safer and more efficient mobility, yet public trust remains limited due to the lack of transparency in their decision-making. This work addresses this issue by combining deep reinforcement learning (DRL) for adaptive driving control with large language model (LLM)-based explainability modules designed to communicate agent behavior to passengers. DRL agents were trained in simulation using a Dueling Double Deep Q-Network to follow distinct driving requests: fast, comfort, and stop. They demonstrated stable learning, safe compliance with traffic rules, and reliable switching between modes within a single trip. In parallel, LLM modules were introduced to interpret passenger requests, determine when explanations were needed, and generate concise, safety-oriented justifications. Results show that this framework, serving as a proof of concept for integrating RL decision-making and LLMs, balances safety, adaptability, and explainability, and is most effective when requests are delayed or overridden due to safety constraints.

Ouided Braoui, Meriem Bouali, Nadir Farhi

LITAN, ESTIN, RN 75, Amizour, 06300, Bejaia, Algeria · Cosys-Grettia, Univ Gustave Eiffel, F-77454 Marne-la-Vallée, France

Explainable Artificial IntelligenceAutonomous DrivingRL for Autonomous Driving
AskPDF