Reinforcement Learning

Also known as RL

Ask about this topic

Overview

Reinforcement learning studies how an agent can choose actions over time to earn rewards, using experience to improve its decisions. Common approaches include value-based methods such as Q-learning, policy-gradient methods such as PPO, and model-based methods that learn or use a model of the environment; offline and hierarchical methods address learning from fixed data and long tasks. Applications range from Atari and MuJoCo benchmarks to robot control, traffic, games, and language-model agents that use tools or solve multi-step problems. A central challenge is making learning efficient and reliable when rewards are sparse or imperfect, environments change, and success in simulation or benchmarks may not transfer safely to deployment.

Latest papers

CardsList

1,743 papers237 papers in the last four weeks, up 46% on the four weeks before. 1.4% of all new papers.

Mar 26, 2026

Neuro-Cognitive Reward Modeling for Human-Centered Autonomous Vehicle Control

Recent advancements in computer vision have accelerated the development of autonomous driving. Despite these advancements, training machines to drive in a way that aligns with human expectations remains a significant challenge. Human factors are still essential, as humans possess a sophisticated cognitive system capable of rapidly interpreting scene information and making accurate decisions. Aligning machine with human intent has been explored with Reinforcement Learning with Human Feedback (RLHF). Conventional RLHF methods rely on collecting human preference data by manually ranking generated outputs, which is time-consuming and indirect. In this work, we propose an electroencephalography (EEG)-guided decision-making framework to incorporate human cognitive insights without behaviour response interruption into reinforcement learning (RL) for autonomous driving. We collected EEG signals from 20 participants in a realistic driving simulator and analyzed event-related potentials (ERP) in response to sudden environmental changes. Our proposed framework employs a neural network to predict the strength of ERP based on the cognitive information from visual scene information. Moreover, we explore the integration of such cognitive information into the reward signal of the RL algorithm. Experimental results show that our framework can improve the collision avoidance ability of the RL algorithm, highlighting the potential of neuro-cognitive feedback in enhancing autonomous driving systems. Our project page is: https://alex95gogo.github.io/Cognitive-Reward/.

Zhuoli Zhuang, Yu-Cheng Chang, Yu-Kai Wang +2

University of Technology Sydney

RL for Autonomous DrivingReinforcement LearningRL from Human FeedbackMultimodal Reward Modeling
AskPDF
Mar 26, 2026

Reinforcing Structured Chain-of-Thought for Video Understanding

Multi-modal Large Language Models (MLLMs) show promise in video understanding. However, their reasoning often suffers from thinking drift and weak temporal comprehension, even when enhanced by Reinforcement Learning (RL) techniques like Group Relative Policy Optimization (GRPO). Moreover, existing RL methods usually depend on Supervised Fine-Tuning (SFT), which requires costly Chain-of-Thought (CoT) annotation and multi-stage training, and enforces fixed reasoning paths, limiting MLLMs' ability to generalize and potentially inducing bias. To overcome these limitations, we introduce Summary-Driven Reinforcement Learning (SDRL), a novel single-stage RL framework that obviates the need for SFT by utilizing a Structured CoT format: Summarize -> Think -> Answer. SDRL introduces two self-supervised mechanisms integrated into the GRPO objective: 1) Consistency of Vision Knowledge (CVK) enforces factual grounding by reducing KL divergence among generated summaries; and 2) Dynamic Variety of Reasoning (DVR) promotes exploration by dynamically modulating thinking diversity based on group accuracy. This novel integration effectively balances alignment and exploration, supervising both the final answer and the reasoning process. Our method achieves state-of-the-art performance on seven public VideoQA datasets.

Peiyao Wang, Haotian Xu, Noranart Vesdapunt +6

Stony Brook University · Amazon

Structured CoT ReasoningRL for VLMsReinforcement LearningVideo QACoT ReasoningTemporal Video Understanding
AskPDF
Mar 26, 2026

PiCSRL: Physics-Informed Contextual Spectral Reinforcement Learning

High-dimensional low-sample-size (HDLSS) datasets constrain reliable environmental model development, where labeled data remain sparse. Reinforcement learning (RL)-based adaptive sensing methods can learn optimal sampling policies, yet their application is severely limited in HDLSS contexts. In this work, we present PiCSRL (Physics-Informed Contextual Spectral Reinforcement Learning), where embeddings are designed using domain knowledge and parsed directly into the RL state representation for improved adaptive sensing. We developed an uncertainty-aware belief model that encodes physics-informed features to improve prediction. As a representative example, we evaluated our approach for cyanobacterial gene concentration adaptive sampling task using NASA PACE hyperspectral imagery over Lake Erie. PiCSRL achieves optimal station selection (RMSE = 0.153, 98.4% bloom detection rate, outperforming random (0.296) and UCB (0.178) RMSE baselines, respectively. Our ablation experiments demonstrate that physics-informed features improve test generalization (0.52 R^2, +0.11 over raw bands) in semi-supervised learning. In addition, our scalability test shows that PiCSRL scales effectively to large networks (50 stations, >2M combinations) with significant improvements over baselines (p = 0.002). We posit PiCSRL as a sample-efficient adaptive sensing method across Earth observation domains for improved observation-to-target mapping.

Mitra Nasr Azadani, Syed Usama Imtiaz, Nasrin Alamdari

Department of Civil and Environmental Engineering Florida State University, Tallahassee, FL, USA

Contextual RLAdaptive SensingReinforcement LearningSample-Efficient RLPhysics-Informed RLActive Perception
AskPDF
Mar 26, 2026

Maximum Entropy Behavior Exploration for Sim2Real Zero-Shot Reinforcement Learning

Zero-shot reinforcement learning (RL) algorithms aim to learn a family of policies from a reward-free dataset, and recover optimal policies for any reward function directly at test time. Naturally, the quality of the pretraining dataset determines the performance of the recovered policies across tasks. However, pre-collecting a relevant, diverse dataset without prior knowledge of the downstream tasks of interest remains a challenge. In this work, we study online\textit{online} zero-shot RL for quadrupedal control on real robotic systems, building upon the Forward-Backward (FB) algorithm. We observe that undirected exploration yields low-diversity data, leading to poor downstream performance and rendering policies impractical for direct hardware deployment. Therefore, we introduce FB-MEBE, an online zero-shot RL algorithm that combines an unsupervised behavior exploration strategy with a regularization critic. FB-MEBE promotes exploration by maximizing the entropy of the achieved behavior distribution. Additionally, a regularization critic shapes the recovered policies toward more natural and physically plausible behaviors. We empirically demonstrate that FB-MEBE achieves and improved performance compared to other exploration strategies in a range of simulated downstream tasks, and that it renders natural policies that can be seamlessly deployed to hardware without further finetuning. Videos and code available on our website.

Jiajun Hu, Nuria Armengol Urpi, Jin Cheng +1

Department of Mechanical Engineering, EPFL · Department of Computer Science, ETH Zurich · Max Planck Institute for Intelligent Systems

Zero-Shot RLReinforcement LearningRL ExplorationBehavioral Diversity in RLRL for Legged LocomotionSim-to-Real Transfer+1
AskPDF
Mar 26, 2026

Empowering Epidemic Response: The Role of Reinforcement Learning in Infectious Disease Control

Reinforcement learning (RL), owing to its adaptability to various dynamic systems in many real-world scenarios and the capability of maximizing long-term outcomes under different constraints, has been used in infectious disease control to optimize the intervention strategies for controlling infectious disease spread and responding to outbreaks in recent years. The potential of RL for assisting public health sectors in preventing and controlling infectious diseases is gradually emerging and being explored by rapidly increasing publications relevant to COVID-19 and other infectious diseases. However, few surveys exclusively discuss this topic, that is, the development and application of RL approaches for optimizing strategies of non-pharmaceutical and pharmaceutical interventions of public health. Therefore, this paper aims to provide a concise review and discussion of the latest literature on how RL approaches have been used to assist in controlling the spread and outbreaks of infectious diseases, covering several critical topics addressing public health demands: resource allocation, balancing between lives and livelihoods, mixed policy of multiple interventions, and inter-regional coordinated control. Finally, we conclude the paper with a discussion of several potential directions for future research.

Mutong Liu, Yang Liu, Jiming Liu

Department of Computer Science, Hong Kong Baptist University, Hong Kong SAR, China

Reinforcement LearningHealthcareRL Control
AskPDF
Mar 26, 2026

Label What Matters: Modality-Balanced and Difficulty-Aware Multimodal Active Learning

Multimodal learning integrates complementary information from different modalities such as image, text, and audio to improve model performance, but its success relies on large-scale labeled data, which is costly to obtain. Active learning (AL) mitigates this challenge by selectively annotating informative samples. In multimodal settings, many approaches implicitly assume that modality importance is stable across rounds and keep selection rules fixed at the fusion stage, which leaves them insensitive to the dynamic nature of multimodal learning, where the relative value of modalities and the difficulty of instances shift as training proceeds. To address this issue, we propose RL-MBA, a reinforcement-learning framework for modality-balanced, difficulty-aware multimodal active learning. RL-MBA models sample selection as a Markov Decision Process, where the policy adapts to modality contributions, uncertainty, and diversity, and the reward encourages accuracy gains and balance. Two key components drive this adaptability: (1) Adaptive Modality Contribution Balancing (AMCB), which dynamically adjusts modality weights via reinforcement feedback, and (2) Evidential Fusion for DifficultyAware Policy Adjustment (EFDA), which estimates sample difficulty via uncertainty-based evidential fusion to prioritize informative samples. Experiments on Food101, KineticsSound, and VGGSound demonstrate that RL-MBA consistently outperforms strong baselines, improving both classification accuracy and modality fairness under limited labeling budgets.

Yuqiao Zeng, Xu Wang, Tengfei Liang +3

Key Laboratory of Big Data and Artificial Intelligence in Transportation, Ministry of Education; State Key Laboratory of Advanced Rail Autonomous Operation; School of Computer and Information Technology, Beijing Jiaotong University, Beijing, China · School of Psychology & Neuroscience, University of Glasgow, Glasgow, UK

Active LearningMultimodal ClassificationUncertainty-Aware Multimodal FusionReinforcement Learning
AskPDF
Mar 26, 2026

GDPO-Listener: Expressive Interactive Head Generation via Auto-Regressive Flow Matching and Group reward-Decoupled Policy Optimization

Generating realistic 3D head motion for dyadic interactions is a significant challenge in virtual human synthesis. While recent methods achieve impressive results with speaking heads, they frequently suffer from the `Regression-to-the-Mean' problem in listener motions, collapsing into static faces, and lack the parameter space for complex nonverbal motions. In this paper, we propose GDPO-Listener, a novel framework that achieves highly expressive speaking and listening motion generation. First, we introduce an Auto-Regressive Flow Matching architecture enabling stable supervised learning. Second, to overcome kinematic stillness, we apply the Group reward-Decoupled Policy Optimization (GDPO). By isolating reward normalization across distinct FLAME parameter groups, GDPO explicitly incentivizes high variance expressive generations. Finally, we enable explicit semantic text control for customizable responses. Extensive evaluations across the Seamless Interaction and DualTalk datasets demonstrate superior performance compared to existing baselines on long-term kinematic variance, visual expressivity and semantic controllability.

Zhangyu Jin, Maksim Siniukov, Deuksin Kwon +2

University of Southern California, Institute for Creative Technologies

Human Motion GenerationTalking Head GenerationText-to-Motion GenerationReinforcement Learning
AskPDF
Mar 26, 2026

Learning Rollout from Sampling:An R1-Style Tokenized Traffic Simulation Model

Learning diverse and high-fidelity traffic simulations from human driving demonstrations is crucial for autonomous driving evaluation. The recent next-token prediction (NTP) paradigm, widely adopted in large language models (LLMs), has been applied to traffic simulation and achieves iterative improvements via supervised fine-tuning (SFT). However, such methods limit active exploration of potentially valuable motion tokens, particularly in suboptimal regions. Entropy patterns provide a promising perspective for enabling exploration driven by motion token uncertainty. Motivated by this insight, we propose a novel tokenized traffic simulation policy, R1Sim, which represents an initial attempt to explore reinforcement learning based on motion token entropy patterns, and systematically analyzes the impact of different motion tokens on simulation outcomes. Specifically, we introduce an entropy-guided adaptive sampling mechanism that focuses on previously overlooked motion tokens with high uncertainty yet high potential. We further optimize motion behaviors using Group Relative Policy Optimization (GRPO), guided by a safety-aware reward design. Overall, these components enable a balanced exploration-exploitation trade-off through diverse high-uncertainty sampling and group-wise comparative estimation, resulting in realistic, safe, and diverse multi-agent behaviors. Extensive experiments on the Waymo Sim Agent benchmark demonstrate that R1Sim achieves competitive performance compared to state-of-the-art methods.

Ziyan Wang, Peng Chen, Ding Li +4

State Key Laboratory of Intelligent Transportation System, Key Laboratory of Autonomous Transportation Technology for Special Vehicles, Ministry of Industry and Information Technology, School of Transportation Science and Engineering, Beihang University, Beijing 100191, China. · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China.

Autonomous Driving SimulationReinforcement LearningRL ExplorationMulti-Agent Reinforcement Learning
AskPDF
Mar 25, 2026

C-STEP: Continuous Space-Time Empowerment for Physics-informed Safe Reinforcement Learning of Mobile Agents

Safe navigation in complex environments remains a central challenge for reinforcement learning (RL) in robotics. This paper introduces Continuous Space-Time Empowerment for Physics-informed (C-STEP) safe RL, a novel measure of agent-centric safety tailored to deterministic, continuous domains. This measure can be used to design physics-informed intrinsic rewards by augmenting positive navigation reward functions. The reward incorporates the agents internal states (e.g., initial velocity) and forward dynamics to differentiate safe from risky behavior. By integrating C-STEP with navigation rewards, we obtain an intrinsic reward function that jointly optimizes task completion and collision avoidance. Numerical results demonstrate fewer collisions, reduced proximity to obstacles, and only marginal increases in travel time. Overall, C-STEP offers an interpretable, physics-informed approach to reward shaping in RL, contributing to safety for agentic mobile robotic systems.

Guihlerme Daubt, Adrian Redder

Faculty of Technology, Bielefeld University, Germany · Department of Electrical Engineering and Information Technology, Paderborn University, Germany

Safe RLSafe Robot NavigationReinforcement LearningRobot Collision AvoidanceIntrinsic Reward Methods
AskPDF
Mar 25, 2026

Towards Effective Experiential Learning: Dual Guidance for Utilization and Internalization

Recently, reinforcement learning~(RL) has become an important approach for improving the capabilities of large language models~(LLMs). In particular, reinforcement learning from verifiable rewards~(RLVR) has emerged as a promising paradigm for reasoning tasks. However, existing RL-based training still remains only a rough approximation to human learning. Human learners leverage both external and internal experience to guide exploration and gradually internalize useful trajectories into stable knowledge. Motivated by this gap, we ask: how can LLMs better utilize and internalize experience during RLVR training? To answer this question, we propose Dual Guidance Optimization~(DGO), a unified framework that leverages external and internal experience to improve training effectiveness. Specifically, DGO first constructs an experience bank from previously explored trajectories. The policy then performs exploration under the joint guidance of the experience bank and the model's internal knowledge. The resulting trajectories are further used to refine the experience bank and optimize model parameters, forming a closed loop of experience utilization and internalization. Experiments show that DGO consistently outperforms baseline methods, suggesting that better utilization and internalization of experience lead to more effective reasoning.

Fei Bai, Zhipeng Chen, Chuan Hao +6

Gaoling School of Artificial Intelligence, Renmin University of China · IQuest Research · Beihang University

RL for Language Model ReasoningReinforcement LearningReinforcement Learning with Verifiable RewardsExperience Replay
AskPDF
Mar 25, 2026

From Pixels to Digital Agents: An Empirical Study on the Taxonomy and Technological Trends of Reinforcement Learning Environments

The remarkable progress of reinforcement learning (RL) is intrinsically tied to the environments used to train and evaluate artificial agents. Moving beyond traditional qualitative reviews, this work presents a large-scale, data-driven empirical investigation into the evolution of RL environments. By programmatically processing a massive corpus of academic literature and rigorously distilling over 2,000 core publications, we propose a quantitative methodology to map the transition from isolated physical simulations to generalist, language-driven foundation agents. Implementing a novel, multi-dimensional taxonomy, we systematically analyze benchmarks against diverse application domains and requisite cognitive capabilities. Our automated semantic and statistical analysis reveals a profound, data-verified paradigm shift: the bifurcation of the field into a "Semantic Prior" ecosystem dominated by Large Language Models (LLMs) and a "Domain-Specific Generalization" ecosystem. Furthermore, we characterize the "cognitive fingerprints" of these distinct domains to uncover the underlying mechanisms of cross-task synergy, multi-domain interference, and zero-shot generalization. Ultimately, this study offers a rigorous, quantitative roadmap for designing the next generation of Embodied Semantic Simulators, bridging the gap between continuous physical control and high-level logical reasoning.

Lijing Luo, Yiben Luo, Alexey Gorbatovski +2

Sun Yat-sen University Shenzhen, China · Yancheng Institute of Technology Yancheng, China · Central University Moscow, Russia +1

RL BenchmarksReinforcement LearningGeneralization in RLRL for LLM AgentsMulti-Task RL
AskPDF
Mar 24, 2026

Self Paced Gaussian Contextual Reinforcement Learning

Curriculum learning improves reinforcement learning (RL) efficiency by sequencing tasks from simple to complex. However, many self-paced curriculum methods rely on computationally expensive inner-loop optimizations, limiting their scalability in high-dimensional context spaces. In this paper, we propose Self-Paced Gaussian Curriculum Learning (SPGL), a novel approach that avoids costly numerical procedures by leveraging a closed-form update rule for Gaussian context distributions. SPGL maintains the sample efficiency and adaptability of traditional self-paced methods while substantially reducing computational overhead. We provide theoretical guarantees on convergence and validate our method across several contextual RL benchmarks, including the Point Mass, Lunar Lander, and Ball Catching environments. Experimental results show that SPGL matches or outperforms existing curriculum methods, especially in hidden context scenarios, and achieves more stable context distribution convergence. Our method offers a scalable, principled alternative for curriculum generation in challenging continuous and partially observable domains.

Mohsen Sahraei Ardakani, Rui Song

Department of Electrical and Computer Engineering North Carolina State University Raleigh, NC · Department of Statistics North Carolina State University Raleigh, NC

Contextual RLReinforcement LearningCurriculum RL
AskPDF
Mar 24, 2026

Dual-Gated Epistemic Time-Dilation: Autonomous Compute Modulation in Asynchronous MARL

While Multi-Agent Reinforcement Learning (MARL) algorithms achieve unprecedented successes across complex continuous domains, their standard deployment strictly adheres to a synchronous operational paradigm. Under this paradigm, agents are universally forced to execute deep neural network inferences at every micro-frame, regardless of immediate necessity. This dense throughput acts as a fundamental barrier to physical deployment on edge-devices where thermal and metabolic budgets are highly constrained. We propose Epistemic Time-Dilation MAPPO (ETD-MAPPO), augmented with a Dual-Gated Epistemic Trigger. Instead of depending on rigid frame-skipping (macro-actions), agents autonomously modulate their execution frequency by interpreting aleatoric uncertainty (via Shannon entropy of their policy) and epistemic uncertainty (via state-value divergence in a Twin-Critic architecture). To format this, we structure the environment as a Semi-Markov Decision Process (SMDP) and build the SMDP-Aligned Asynchronous Gradient Masking Critic to ensure proper credit assignment. Empirical findings demonstrate massive improvements (> 60% relative baseline acquisition leaps) over current temporal models. By assessing LBF, MPE, and the 115-dimensional state space of Google Research Football (GRF), ETD correctly prevented premature policy collapse. Remarkably, this unconstrained approach leads to emergent Temporal Role Specialization, reducing computational overhead by a statistically dominant 73.6% entirely during off-ball execution without deteriorating centralized task dominance.

Igor Jankowski

Semi-Markov Decision ProcessReinforcement LearningAsynchronous RLCooperative MARLSequential Decision Making
AskPDF
Mar 24, 2026

End-to-End Efficient RL for Linear Bellman Complete MDPs with Deterministic Transitions

We study reinforcement learning (RL) with linear function approximation in Markov Decision Processes (MDPs) satisfying linear Bellman completeness -- a fundamental setting where the Bellman backup of any linear value function remains linear. While statistically tractable, prior computationally efficient algorithms are either limited to small action spaces or require strong oracle assumptions over the feature space. We provide a computationally efficient algorithm for linear Bellman complete MDPs with deterministic transitions, stochastic initial states, and stochastic rewards. For finite action spaces, our algorithm is end-to-end efficient; for large or infinite action spaces, we require only a standard argmax oracle over actions. Our algorithm learns an ε\varepsilon-optimal policy with sample and computational complexity polynomial in the horizon, feature dimension, and 1/ε1/\varepsilon.

Zakaria Mhammedi, Alexander Rakhlin, Nneka Okolo

Google Research, NYC · MIT

Reinforcement LearningLinear Markov Decision ProcessesPolicy LearningSample-Efficient RLFunction Approximation in RL
AskPDF
Mar 24, 2026

Off-Policy Value-Based Reinforcement Learning for Large Language Models

Improving data utilization efficiency is critical for scaling reinforcement learning (RL) for long-horizon tasks where generating trajectories is expensive. However, the dominant RL methods for LLMs are largely on-policy: they update each batch of data only once, discard it, and then collect fresh samples, resulting in poor sample efficiency. In this work, we explore an alternative value-based RL framework for LLMs that naturally enables off-policy learning. We propose ReVal, a Bellman-update-based method that combines stepwise signals capturing internal consistency with trajectory-level signals derived from outcome verification. ReVal naturally supports replay-buffer-based training, allowing efficient reuse of past trajectories. Experiments on standard mathematical reasoning benchmarks show that ReVal not only converges faster but also outperforms GRPO in final performance. On DeepSeek-R1-Distill-1.5B, ReVal improves training efficiency and achieves improvement of 2.7% in AIME24 and 4.5% in out-of-domain benchmark GPQA over GRPO. These results suggest that value-based RL is a practical alternative to policy-based methods for LLM training.

Peng-Yuan Wang, Ziniu Li, Tian Xu +8

National Key Laboratory for Novel Software Technology & School of Artificial Intelligence, Nanjing University, China · The Chinese University of Hong Kong, Shenzhen · Shenzhen Research Institute of Big Data +1

Reinforcement LearningRL for Language Model ReasoningOff-Policy RLRL Fine-TuningLLM Reasoning
AskPDF
Mar 24, 2026

EVA: Efficient Reinforcement Learning for End-to-End Video Agent

Video understanding with multimodal large language models (MLLMs) remains challenging due to the long token sequences of videos, which contain extensive temporal dependencies and redundant frames. Existing approaches typically treat MLLMs as passive recognizers, processing entire videos or uniformly sampled frames without adaptive reasoning. Recent agent-based methods introduce external tools, yet still depend on manually designed workflows and perception-first strategies, resulting in inefficiency on long videos. We present EVA, an Efficient Reinforcement Learning framework for End-to-End Video Agent, which enables planning-before-perception through iterative summary-plan-action-reflection reasoning. EVA autonomously decides what to watch, when to watch, and how to watch, achieving query-driven and efficient video understanding. To train such agents, we design a simple yet effective three-stage learning pipeline - comprising supervised fine-tuning (SFT), Kahneman-Tversky Optimization (KTO), and Group Relative Policy Optimization (GRPO) - that bridges supervised imitation and reinforcement learning. We further construct high-quality datasets for each stage, supporting stable and reproducible training. We evaluate EVA on six video understanding benchmarks, demonstrating its comprehensive capabilities. Compared with existing baselines, EVA achieves a substantial improvement of 6-12% over general MLLM baselines and a further 1-3% gain over prior adaptive agent methods.

Yaolun Zhang, Ruohui Wang, Jiahao Wang +6

SenseTime Research

Video UnderstandingTemporal Video UnderstandingRL for LLM AgentsReinforcement Learning
AskPDF
Mar 24, 2026

CoMaTrack: Competitive Multi-Agent Game-Theoretic Tracking with Vision-Language-Action Models

Embodied Visual Tracking (EVT), a core dynamic task in embodied intelligence, requires an agent to precisely follow a language-specified target. Yet most existing methods rely on single-agent imitation learning, suffering from costly expert data and limited generalization due to static training environments. Inspired by competition-driven capability evolution, we propose CoMaTrack, a competitive game-theoretic multi-agent reinforcement learning framework that trains agents in a dynamic adversarial setting with competitive subtasks, yielding stronger adaptive planning and interference-resilient strategies. We further introduce CoMaTrack-Bench, the first open-source Habitat-based benchmark protocol and episode set for language-conditioned competitive EVT featuring dynamic dueling, featuring game scenarios between a tracker and adaptive opponents across diverse environments and instructions, enabling standardized robustness evaluation under active adversarial interactions. Experiments show that CoMaTrack achieves state-of-the-art results on both standard benchmarks and CoMaTrack-Bench. Notably, a 3B VLM trained with our framework surpasses previous single-agent imitation learning methods based on 7B models on the challenging EVT-Bench, achieving 92.1% in STT, 74.2% in DT, and 57.5% in AT. The benchmark code will be available at https://github.com/wlqcode/CoMaTrack-Bench.

Youzhi Liu, Li Gao, Liu Liu +2

Amap, Alibaba Group

Game-Theoretic RLReinforcement LearningVLMs for RoboticsRobustness of VLA ModelsSequential MARL
AskPDF
Mar 23, 2026

Human-Inspired Pavlovian and Instrumental Learning for Autonomous Agent Navigation

Autonomous agents operating in uncertain environments must balance fast responses with goal-directed planning. Classical MF RL often converges slowly and may induce unsafe exploration, whereas MB methods are computationally expensive and sensitive to model mismatch. This paper presents a human-inspired hybrid RL architecture integrating Pavlovian, Instrumental MF, and Instrumental MB components. Inspired by Pavlovian and Instrumental learning from neuroscience, the framework considers contextual radio cues, here intended as georeferenced environmental features acting as CS, to shape intrinsic value signals and bias decision-making. Learning is further modulated by internal motivational drives through a dedicated motivational signal. A Bayesian arbitration mechanism adaptively blends MF and MB estimates based on predicted reliability. Simulation results show that the hybrid approach accelerates learning, improves operational safety, and reduces navigation in high-uncertainty regions compared to standard RL baselines. Pavlovian conditioning promotes safer exploration and faster convergence, while arbitration enables a smooth transition from exploration to efficient, plan-driven exploitation. Overall, the results highlight the benefits of biologically inspired modularity for robust and adaptive autonomous systems under uncertainty.

Jingfeng Shan, Francesco Guidi, Mehrdad Saeidi +8

WiLab, University of Bologna, via dell’Università 50, 47521 Cesena, Italy · National Research Council of Italy (CNR), viale del Risorgimento 2, 40136 Bologna, Italy · University of Bologna, Department of Psychology “Renzo Canestrari”, 47521 Cesena, Italy

Reinforcement LearningModel-Based RLSafe RLModel-Free RL
AskPDF
Mar 23, 2026

A Context Engineering Framework for Improving Enterprise AI Agents based on Digital-Twin MDP

Despite rapid progress in AI agents for enterprise automation and decision-making, their real-world deployment and further performance gains remain constrained by limited data quality and quantity, complex real-world reasoning demands, difficulties with self-play, and the lack of reliable feedback signals. To address these challenges, we propose a lightweight, model-agnostic framework for improving LLM-based enterprise agents via offline reinforcement learning (RL). The proposed Context Engineering via DT-MDP (DT-MDP-CE) framework comprises three key components: (1) A Digital-Twin Markov Decision Process (DT-MDP), which abstracts the agent's reasoning behavior as a finite MDP; (2) A robust contrastive inverse RL, which, armed with the DT-MDP, to efficiently estimate a well-founded reward function and induces policies from mixed-quality offline trajectories; and (3) RL-guided context engineering, which uses the policy obtained from the integrated process of (1) and (2), to improve the agent's decision-making behavior. As a case study, we apply the framework to a representative task in the enterprise-oriented domain of IT automation. Extensive experimental results demonstrate consistent and significant improvements over baseline agents across a wide range of evaluation settings, suggesting that the framework can generalize to other agents sharing similar characteristics in enterprise environments.

Xi Yang, Aurelie Lozano, Naoki Abe +6

IBM Software Innovation Lab, Yorktown Heights, New York, USA

LLM AgentsLLM Agent TrainingRL for LLM AgentsReinforcement LearningOffline RL
AskPDF
Mar 23, 2026

Shaping Scientific Explanations to Expert Perspectives with Persona-Conditioned Reinforcement Learning

Explainable AI is increasingly important to scientific discovery. However, existing methods largely ignore that explanation quality is not universal: experts differ in how they assess evidence, prioritize mechanisms, and construct explanatory narratives. We introduce perspective-conditioned explanations, a framework for adapting explanation generation to epistemic variation in expert judgment. Using knowledge graph reasoning paths in drug discovery, we show that preferences organize into coherent epistemic perspectives that can be captured by agentic personas, representations of how experts evaluate explanations. Persona-aligned rewards then guide reinforcement learning-based explanation generation without large-scale expert supervision. Expert user studies show that perspective-conditioned explanations are preferred over general-purpose explanations and improve perceived relevance and validity. Moreover, they match or exceed state-of-the-art predictive performance and reduce expert feedback time by two orders of magnitude. Together, these findings demonstrate that explanation quality is perspective-dependent and that modeling this variation enables scalable and human-aligned explanation generation for scientific discovery.

Susana Nunes, Tiago Guerreiro, Catia Pesquita

LASIGE, Faculdade de Ciências da Universidade de Lisboa, Portugal.

Explainable Artificial IntelligenceReinforcement Learning
AskPDF
Mar 23, 2026

Evolutionary Warm-Starts for Reinforcement Learning in Industrial Continuous Control

Reinforcement learning (RL) is still rarely applied in industrial control, partly due to the difficulty of training reliable agents for real-world conditions. This work investigates how evolution strategies can support RL in such settings by introducing a continuous-control adaptation of an industrial sorting benchmark. The CMA-ES algorithm is used to generate high-quality demonstrations that warm-start RL agents. Results show that CMA-ES-guided initialization significantly improves stability and performance. Furthermore, the demonstration trajectories generated with the CMA-ES provide a strong oracle reference performance level, which is of interest in its own right. The study delivers a focused proof of concept for hybrid evolutionary-RL approaches and a basis for future, more complex industrial applications.

Tom Maus, Stephan Frank, Tobias Glasmachers

Inst. for Neural Computation, Ruhr University Bochum, Bochum, Germany

Evolutionary RLReinforcement LearningRL Control
AskPDF
Mar 22, 2026

PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost

Post-training for long-horizon agentic tasks has a tension between compute efficiency and generalization. While supervised fine-tuning (SFT) is compute efficient, it often suffers from out-of-domain (OOD) degradation. Conversely, end-to-end reinforcement learning (E2E RL) preserves OOD capabilities, but incurs high compute costs due to many turns of on-policy rollout. We introduce PivotRL, a novel framework that operates on existing SFT trajectories to combine the compute efficiency of SFT with the OOD accuracy of E2E RL. PivotRL relies on two key mechanisms: first, it executes local, on-policy rollouts and filters for pivots: informative intermediate turns where sampled actions exhibit high variance in outcomes; second, it utilizes rewards for functional-equivalent actions rather than demanding strict string matching with the SFT data demonstration. We theoretically show that these mechanisms incentivize strong learning signals with high natural gradient norm, while maximally preserving policy probability ordering on actions unrelated to training tasks. In comparison to standard SFT on identical data, we demonstrate that PivotRL achieves +4.17% higher in-domain accuracy on average across four agentic domains, and +10.04% higher OOD accuracy in non-agentic tasks. Notably, on agentic coding tasks, PivotRL achieves competitive accuracy with E2E RL with 4x fewer rollout turns. PivotRL is adopted by NVIDIA's Nemotron-3-Super-120B-A12B, acting as the workhorse in production-scale agentic post-training.

Junkeun Yi, Damon Mosk-Aoyama, Baihe Huang +9

NVIDIA · UC Berkeley

RL Post-TrainingRL for LLM AgentsAgentic RLReinforcement LearningGeneralization in RL
AskPDF
Mar 22, 2026

DeepXplain: XAI-Guided Autonomous Defense Against Multi-Stage APT Campaigns

Advanced Persistent Threats (APTs) are stealthy, multi-stage attacks that require adaptive and timely defense. While deep reinforcement learning (DRL) enables autonomous cyber defense, its decisions are often opaque and difficult to trust in operational environments. This paper presents DeepXplain, an explainable DRL framework for stage-aware APT defense. Building on our prior DeepStage model, DeepXplain integrates provenance-based graph learning, temporal stage estimation, and a unified XAI pipeline that provides structural, temporal, and policy-level explanations. Unlike post-hoc methods, explanation signals are incorporated directly into policy optimization through evidence alignment and confidence-aware reward shaping. To the best of our knowledge, DeepXplain is the first framework to integrate explanation signals into reinforcement learning for APT defense. Experiments in a realistic enterprise testbed show improvements in stage-weighted F1-score (0.887 to 0.915) and success rate (84.7% to 89.6%), along with higher explanation confidence (0.86), improved fidelity (0.79), and more compact explanations (0.31). These results demonstrate enhanced effectiveness and trustworthiness of autonomous cyber defense.

Trung V. Phan, Thomas Bauschert

Chair of Communication Networks, Technische Universität Chemnitz, 09126 Chemnitz, Germany

Autonomous Cyber DefenseReinforcement LearningInterpretability in RLExplainable Artificial Intelligence
AskPDF
Mar 22, 2026

VisFly-Lab: Unified Differentiable Framework for First-Order Reinforcement Learning of Quadrotor Control

First-order reinforcement learning with differentiable simulation is promising for quadrotor control, but practical progress remains fragmented across task-specific settings. To support more systematic development and evaluation, we present a unified differentiable framework for multi-task quadrotor control. The framework is wrapped, extensible, and equipped with deployment-oriented dynamics, providing a common interface across four representative tasks: hovering, tracking, landing, and racing. We also present the suite of first-order learning algorithms, where we identify two practical bottlenecks of standard first-order training: limited state coverage caused by horizon initialization and gradient bias caused by partially non-differentiable rewards. To address these issues, we propose Amended Backpropagation Through Time (ABPT), which combines differentiable rollout optimization, a value-based auxiliary objective, and visited-state initialization to improve training robustness. Experimental results show that ABPT yields the clearest gains in tasks with partially non-differentiable rewards, while remaining competitive in fully differentiable settings. We further provide proof-of-concept real-world deployments showing initial transferability of policies learned in the proposed framework beyond simulation.

Fanxing Li, Fangyu Sun, Tianbao Zhang +5

Reinforcement LearningQuadrotor ControlMulti-Task RLDifferentiable Physics
AskPDF
Mar 22, 2026

Learning to Optimize Joint Source and RIS-assisted Channel Encoding for Multi-User Semantic Communication Systems

In this paper, we explore a joint source and reconfigurable intelligent surface (RIS)-assisted channel encoding (JSRE) framework for multi-user semantic communications, where a deep neural network (DNN) extracts semantic features for all users and the RIS provides channel orthogonality, enabling a unified semantic encoding-decoding design. We aim to maximize the overall energy efficiency of semantic communications across all users by jointly optimizing the user scheduling, the RIS's phase shifts, and the semantic compression ratio. Although this joint optimization problem can be addressed using conventional deep reinforcement learning (DRL) methods, evaluating semantic similarity typically relies on extensive real environment interactions, which can incur heavy computational overhead during training. To address this challenge, we propose a truncated DRL (T-DRL) framework, where a DNN-based semantic similarity estimator is developed to rapidly estimate the similarity score. Moreover, the user scheduling strategy is tightly coupled with the semantic model configuration. To exploit this relationship, we further propose a semantic model caching mechanism that stores and reuses fine-tuned semantic models corresponding to different scheduling decisions. A Transformer-based actor network is employed within the DRL framework to dynamically generate action space conditioned on the current caching state. This avoids redundant retraining and further accelerates the convergence of the learning process. Numerical results demonstrate that the proposed JSRE framework significantly improves the system energy efficiency compared with the baseline methods. By training fewer semantic models, the proposed T-DRL framework significantly enhances the learning efficiency.

Haidong Wang, Songhan Zhao, Bo Gu +3

School of Intelligent Systems Engineering, Sun Yat-sen University, China · Department of Electrical and Computer Engineering, University of Hong Kong, HongKong, SAR, China · Department of Electrical Engineering and Computer Science, Lassonde School of Engineering, York University, Canada

Joint Source-Channel CodingSemantic CommunicationWireless CommunicationsReinforcement LearningWireless Resource Allocation
AskPDF
Mar 21, 2026

Cyber Deception for Mission Surveillance via Hypergame-Theoretic Deep Reinforcement Learning

Unmanned Aerial Vehicles (UAVs) are valuable for mission-critical systems like surveillance, rescue, or delivery. Not surprisingly, such systems attract cyberattacks, including Denial-of-Service (DoS) attacks to overwhelm the resources of mission drones (MDs). How can we defend UAV mission systems against DoS attacks? We adopt cyber deception as a defense strategy, in which honey drones (HDs) are proposed to bait and divert attacks. The attack and deceptive defense hinge upon radio signal strength: The attacker selects victim MDs based on their signals, and HDs attract the attacker from afar by emitting stronger signals, despite this reducing battery life. We formulate an optimization problem for the attacker and defender to identify their respective strategies for maximizing mission performance while minimizing energy consumption. To address this problem, we propose a novel approach, called HT-DRL. HT-DRL identifies optimal solutions without a long learning convergence time by taking the solutions of hypergame theory into the neural network of deep reinforcement learning. This achieves a systematic way to intelligently deceive attackers. We analyze the performance of diverse defense mechanisms under different attack strategies. Further, the HT-DRL-based HD approach outperforms existing non-HD counterparts up to two times better in mission performance while incurring low energy consumption.

Zelin Wan, Jin-Hee Cho, Mu Zhu +3

Department of Computer Science, Virginia Tech, Falls Church, VA, USA · Computer Network Information Center, Chinese Academy of Sciences, Beijing, China · US Army Research Laboratory, Adelphi, MD, USA +1

CybersecurityAutonomous Cyber DefenseReinforcement Learning
AskPDF
Mar 21, 2026

Learning to Remember: Attentive Reinforcement Learning for Edge Serverless Autoscaling

In edge computing, the stochastic and bursty nature of serverless workloads challenges autonomous resource orchestration. Traditional reactive controllers, such as the Kubernetes Horizontal Pod Autoscaler (HPA), suffer from reaction latency, leading to Service Level Objective (SLO) violations during traffic spikes and resource flapping during ramp-downs. While Deep Reinforcement Learning (DRL) offers a pathway toward proactive management, standard agents suffer from temporal blindness, an inability to exploit the recent temporal context in non-Markovian edge environments. To bridge this gap, we propose a stability-aware autoscaling framework unifying short-horizon temporal context and control via an Attention-Enhanced Double-Stacked LSTM architecture integrated within a Proximal Policy Optimization (PPO) agent. Unlike shallow recurrent models, our approach employs a learned attention mechanism that weights recent historical states non-uniformly, suppressing high-frequency jitter while preserving the trend that precedes demand shifts. We validate the framework on two independent Kubernetes clusters using real-world Azure Functions traces. Against the single-layer LSTM ablation and the static HPA baseline, our approach reduces P90 latency by ≈\approx67%, and holds average latency within the 50ms hard SLO for 98.8% of the run against 49.6% and 43.5% respectively. Against Kubernetes Event-Driven Autoscaling (KEDA), it matches latency performance at 75% fewer replica-steps and 59% less churn, with P90 hard-SLO violation bursts of at most 5 consecutive intervals against up to 24 for KEDA. These results indicate that mitigating temporal blindness through deep attentive memory improves the reliability and stability of Kubernetes autoscaling under bursty edge workloads.

Faraz Shaikh, Gianluca Reali, Mauro Femminella

Department of Engineering, University of Perugia, Perugia, Italy · Consorzio Nazionale Interuniversitario per le Telecomunicazioni (CNIT), 43124 Parma, Italy

RL ControlReinforcement LearningEdge ComputingMemory-Augmented Agents
AskPDF
Mar 20, 2026

Spectral Alignment in Forward-Backward Representations via Temporal Abstraction

Forward-backward (FB) representations provide a powerful framework for learning the successor representation (SR) in continuous spaces by enforcing a low-rank factorization. However, a fundamental spectral mismatch often exists between the high-rank transition dynamics of continuous environments and the low-rank bottleneck of the FB architecture, making accurate low-rank representation learning difficult. In this work, we analyze temporal abstraction as a mechanism to mitigate this mismatch. By characterizing the spectral properties of the transition operator, we show that temporal abstraction acts analogously to a low-pass filter that suppresses high-frequency spectral components. This suppression reduces the effective rank of the induced SR while preserving a formal bound on the resulting value function error. Empirically, we show that this alignment is a key factor for stable FB learning, particularly at high discount factors where bootstrapping becomes error-prone. Our results identify temporal abstraction as a principled mechanism for shaping the spectral structure of the underlying MDP and enabling effective long-horizon representations in continuous control.

Seyed Mahdi B. Azad, Jasper Hoffmann, Iman Nematollahi +3

Department of Computer Science, University of Freiburg, Germany

Reinforcement LearningRepresentation Learning for RLSuccessor RepresentationsRepresentation Learning
AskPDF
Mar 20, 2026

Decomposable Reward Modeling and Realistic Environment Design for Reinforcement Learning-Based Forex Trading

Reinforcement learning (RL) for foreign exchange (Forex) trading requires realistic execution modeling, interpretable reward design, and action spaces that respect market constraints. Existing approaches often simplify these aspects, limiting policy transparency and practical relevance. This paper presents a modular RL framework integrating three components: (1) a friction-aware execution engine enforcing anti-lookahead semantics---observations at close_t, execution at open_t+1, and mark-to-market at close_t+1---while modeling spread, commission, slippage, rollover financing, and margin-triggered liquidation; (2) a decomposable 11-component reward architecture with fixed, pre-specified weights and per-step logging, enabling attribution, ablation, and auditability; and (3) a 10-action discrete interface supporting scaling, reduction, closure, and reversal, with legal-action masking enforcing margin-aware feasibility during training and evaluation. Empirical evaluation on EURUSD training-split data reveals non-monotonic interactions among reward components, demonstrating that additional penalties do not consistently improve performance. The full reward configuration achieves a terminal training Sharpe ratio of 0.765 and cumulative return of 57.09%. Comparing action spaces, the extended 10-action interface increases cumulative return but reduces Sharpe relative to a conservative 3-action variant, revealing a return-activity trade-off under a fixed training budget. Scaling-enabled strategies consistently reduce drawdowns, with the combined configuration delivering the strongest endpoint performance. Supplementary diagnostics on GBPUSD, USDJPY, and AUDUSD exhibit qualitatively similar trends. These findings demonstrate the framework's diagnostic value and underscore the importance of controlled, interpretable design in RL-based financial systems.

Nabeel Ahmad Saidd

Dr APJ Abdul Kalam Technical University, India

Reinforcement LearningReward Design for RLFinance
AskPDF
Mar 20, 2026

ContractionPPO: Certified Reinforcement Learning via Differentiable Contraction Layers

Legged locomotion in unstructured environments demands not only high-performance control policies but also formal guarantees to ensure robustness under perturbations. Control methods often require carefully designed reference trajectories, which are challenging to construct in high-dimensional, contact-rich systems such as quadruped robots. In contrast, Reinforcement Learning (RL) directly learns policies that implicitly generate motion, and uniquely benefits from access to privileged information, such as full state and dynamics during training, that is not available at deployment. We present ContractionPPO, a framework for certified robust planning and control of legged robots by augmenting Proximal Policy Optimization (PPO) RL with a state-dependent contraction metric layer. This approach enables the policy to maximize performance while simultaneously producing a contraction metric that certifies incremental exponential stability of the simulated closed-loop system. The metric is parameterized as a Lipschitz neural network and trained jointly with the policy, either in parallel or as an auxiliary head of the PPO backbone. While the contraction metric is not deployed during real-world execution, we derive upper bounds on the worst-case contraction rate and show that these bounds ensure the learned contraction metric generalizes from simulation to real-world deployment. Our hardware experiments on quadruped locomotion demonstrate that ContractionPPO enables robust, certifiably stable control even under strong external perturbations.

Vrushabh Zinage, Narek Harutyunyan, Eric Verheyden +2

California Institute of Technology · Brown University

Robust RLReinforcement LearningRL for Legged LocomotionProximal Policy Optimization
AskPDF