Non-Stationary RL

RL: Reinforcement Learning

Non-stationary reinforcement learning studies how agents can keep making good decisions when rewards, transitions, or task conditions change, as they do in deployed robots, traffic systems, and markets. Approaches include change-point detection and regret-based algorithms, context and meta-learning, replay-data management, and continual-learning methods such as network resets or successor features. Theoretical work gives guarantees in tabular and linear Markov decision processes, while empirical evaluations span robotics, driving, games, and multi-agent settings. A central tension is how to adapt quickly without forgetting useful past knowledge or violating safety constraints, especially when old conditions may return.

Latest papers

63 papers5 papers in the last four weeks, down 17% on the four weeks before. 0.0% of all new papers.

Stochastic Tracking of Drifting Affine Fixed Points
Oct 19, 2026Sajad KhodadadianStochastic Approximation Non-Stationary RL
Grado Department of Industrial and Systems Engineering, Virginia Tech
Parametric Resonance in Primal-Dual Learning for Slowly Non-Stationary Constrained MDPs
Oct 16, 2026Maxime BaeldeMarkov Decision Processes Primal-Dual Optimization
Antvoice, Paris, France
Single or Multiple Policies for Phase-Structured Reinforcement Learning?
Oct 2, 2026Guilhem Loussouarn, Nancy Nayak, Kin K. LeungReinforcement Learning Non-Stationary RL
Department of Electrical and Electronic Engineering Imperial College London, UK
Reward Inflation: A Healthy Stimulus for Reinforcement Learning
Oct 1, 2026Ganghun Lee, Minji Kim, Minsu Lee +1Reward Design for RL Non-Stationary RL
AIIS, Seoul National University · Department of Computer Science, Seoul National University · School of AI Convergence, Sungshin Women’s University
AlphaPareto: Formulaic Alpha Discovery with LLM-Guided Multi-Objective Reinforcement Learning
Sep 28, 2026Yingbo Zhao, Zeyu Yang, Zhoufan ZhuMulti-Objective Reinforcement Learning Non-Stationary RL
School of Economics Xiamen University Fujian, China · Paula and Gregory Chow Institute for Studies in Economics Xiamen University Fujian, China · School of Economics & Wang Yanan Institute for Studies in Economics Xiamen University Fujian, China
Self-discovering RL in the Era of Experience: Is Learning History an Asset or a Burden?
Sep 27, 2026Haomin LuoReinforcement Learning Interpretability in RL
University of Cambridge · Models2 AI
Characterizing Replay Retention Under Dynamics Shift in Model-Based Reinforcement Learning
Sep 16, 2026Everest Yang, Skye Thompson, George D. KonidarisNon-Stationary RL Continual Robot Learning
Brown University, Providence, RI, USA
Proactive Context-Forecasted Safety Constraints for Nonstationary Reinforcement Learning
Sep 8, 2026Tim TomashevskiyNon-Stationary RL Constrained RL
Department of Computing and Software McMaster University Hamilton, Ontario, Canada
Spectral Prioritized Sweeping in Nonstationary Reinforcement Learning
Sep 5, 2026Hung Pham, Tuan DamNon-Stationary RL Model-Based RL
SOICT, Hanoi University of Science and Technology
Endogenous Exploration in Reinforcement Learning with Intrinsic Curiosity
Sep 4, 2026Armando VieiraRL Exploration Intrinsic Reward Methods
University of Tartu
Online Change-point Detection for Cooperative Multi-Agent Reinforcement Learning
Sep 4, 2026Fatemeh Saberi Khomami, Julita VassilevaChange-Point Detection Cooperative MARL
Department of Computer Science, University of Saskatchewan, Saskatoon, Canada
ATLAS: Adaptive Topological Learning with Abstract Successors for Continual Learning
Aug 5, 2026R. Blake Lawlor, Daniel S. BrownNon-Stationary RL Reinforcement Learning
1Kahlert School of Computing, University of Utah
A Unified Algorithmic Framework for Hybrid Reinforcement Learning in Tabular MDPs with Shifted Transition Dynamics
Jul 28, 2026Zheshun Wu, Renjie Zheng, Jinhang Zuo +2Non-Stationary RL Regret Minimization in RL
Southern University of Science and Technology · Harbin Institute of Technology, Shenzhen Shenzhen, Guangdong, China · Southern University of Science and Technology Shenzhen, Guangdong, China +4
Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learning
Jul 27, 2026Luc McCutcheon, Evangelos Chatzaroulas, Saber FallahNon-Stationary RL Neural Network Initialization
Department of Mechanical Engineering, University of Surrey, United Kingdom · Department of Computing Science, University of Surrey, United Kingdom
Two-Timescale Hierarchical Reinforcement Learning for Resilient Operations
Jul 26, 2026Young Hyun Cho, Franz Stoll, Will Wei Sun +2Non-Stationary RL Hierarchical RL
Department of Statistics, Harvard University · Edwardson School of Industrial Engineering, Purdue University · Mitch Daniels School of Business, Purdue University +3
Adjustment Speed as a Safety Constraint for Nonstationary Reinforcement Learning
Jul 21, 2026Timothy TomashevskiyNon-Stationary RL Safe RL
Department of Computing and Software McMaster University Hamilton, Ontario, Canada
Breaking Feedback-Blindness: Utility-Augmented Transformer for Sequential Decision Making
Jul 21, 2026Yuyang Shen, Shan Dai, Daimin ChenNon-Stationary RL Sequential Decision Making
The Chinese University of Hong Kong, Shenzhen · Shenzhen Research Institute of Big Data · University of International Business and Economics
PRIME: Plasticity Recovery in Multi-Agent Environments for UAV-Assisted Emergency Communication Networks
Jul 20, 2026Wen Qiu, Zhiqiang He, Wei Zhao +1Non-Stationary RL Reinforcement Learning
Department of Information and Communication Engineering, Kitami Institute of Technology, Japan · Graduate School of Informatics and Engineering, University of Electro-Communications, Japan · School of Computer Science and Technology, Anhui University of Technology, China
In-Context Reinforcement Learning under Non-Stationarity: A Survey
Jul 1, 2026A Run, Ziluo DingNon-Stationary RL In-Context RL
Independent Researcher · Beijing Innovation Center of Humanoid Robotics
Priced Motion Through Optimal Faces: A Normal-Fan Geometry for Non-Stationary Adversarial MDPs
Jun 27, 2026Kai HidajatNon-Stationary RL Markov Decision Processes
Komori Research
Learning to Trigger: Reinforcement Learning at the Large Hadron Collider
Jun 22, 2026Zixin Ding, Shaghayegh Emami, Giovanna Salvi +7RL Control Non-Stationary RL
University of Chicago · University of Michigan, Ann Arbor · Fermi National Accelerator Laboratory
Closing the Feedback Loop: From Experience Extraction to Insight Governance in Verbal Reinforcement Learning
Jun 16, 2026Yanwei Cui, Xing Zhang, Yulong Zhang +4Continual Learning for LLM Agents Non-Stationary RL
AWS Generative AI Innovation Center · Amazon Web Services (AWS) · BingX Group Limited.
When Should a Robot Replan? Regret-Guided Update Scheduling in Time-Varying MDPs
Jun 15, 2026Negin Musavi, Gokul Puthumanaillam, Ruben Hernandez +2Non-Stationary RL Robot Policy Adaptation
Department of Aerospace Engineering, University of Illinois Urbana–Champaign, Champaign, IL 61820, USA
A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning
Jun 15, 2026Ardianto Wibowo, Paulo E Santos, Amer Baghdadi +3Distribution Shift Robustness Non-Stationary RL
IMT Atlantique, Brest, France · Flinders University, Adelaide, Australia · IRL Crossing, Adelaide, Australia +1
Space-sampled Value Decay: Forgetting Mechanisms for Non-stationary Deep Reinforcement Learning
Jun 10, 2026Felix Störck, Fabian Hinder, Barbara HammerNon-Stationary RL Reinforcement Learning
CITEC, Faculty of Technology, Bielefeld University, Germany.
GUI-AC: Enhancing Continual Learning in GUI Agents
Jun 9, 2026Can Lin, Tao Feng, Hangjie Yuan +3Non-Stationary RL RL for GUI Agents
Beijing University of Posts and Telecommunications · Tsinghua University · Zhejiang University +1
Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions
Jun 2, 2026Bingxu Liu, Jiashun Liu, Johan Obando-Ceron +5Non-Stationary RL Reinforcement Learning
Hong Kong University of Science and Technology · Fudan University · Mila - Québec AI Institute +2
Position: Deployed Reinforcement Learning should be Continual
Jun 1, 2026Parnian Behdin, Kevin Roice, Golnaz MesbahiNon-Stationary RL Reinforcement Learning
Alberta Machine Intelligence Institute (Amii), Edmonton, Canada.
Balancing Plasticity and Stability with Fast and Slow Successor Features
May 25, 2026Raymond Chua, Doina Precup, Blake RichardsSequential Decision Making Non-Stationary RL
School of Computer Science, McGill University, Montr´eal, Canada · Mila - Quebec Artificial Intelligence Institute, Montr´eal, QC, Canada · CIFAR Learning in Machines and Brains +3
Grow-Prune-Freeze Networks: Adaptive & Continual Learning Technique for Olfactory Navigation
May 24, 2026Kordel K. France, Ovidiu DaescuNon-Stationary RL Partially Observable RL
Department of Computer Science, University of Texas at Dallas, USA

Oct 19, 2026J/K move · Enter open · S save

Stochastic Tracking of Drifting Affine Fixed Points

Sajad Khodadadian

Grado Department of Industrial and Systems Engineering, Virginia Tech

Stochastic Approximation Non-Stationary RL Temporal-Difference Learning

Open paper AskPDF arXiv

Linear stochastic approximation (SA) is widely used to estimate fixed points from noisy observations, but most finite-time analyses assume that the underlying operator remains fixed. We study linear SA with time-varying affine operators, where the fixed points change over time and the observation noise may be correlated across rounds. For constant-stepsize updates, we establish a bound on the cumulative expected squared distance between the iterates and the corresponding fixed points. The bound accounts for the initial error, observation noise, and changes in the fixed points over time. We also establish lower bounds for scalar Gaussian models, showing that, under specified conditions, a suitably chosen constant stepsize achieves the smallest possible worst-case cumulative error up to constant factors, even compared with estimators that use future observations. These results further quantify the effect of temporal correlation on estimation accuracy. Since choosing an appropriate stepsize generally requires knowledge of the noise level and how much the fixed points change, we develop an adaptive method that uses an additional bounded, unbiased observation of the fixed point to achieve an almost optimal algorithm without needing the knowledge of the noise level and the fixed point variation over time. Finally, we apply our results to value-function estimation using synchronous tabular temporal-difference learning in changing environments, assuming generative-model access and a common invariant distribution.