Adapting to Changes in Agent Behavior via Finite-Depth Policy Sensitivity

Lan Shi; Daigo Shishika; Xuan Wang

cs.LGOct 5, 2026

Adapting to Changes in Agent Behavior via Finite-Depth Policy Sensitivity

Authors: Lan Shi, Daigo Shishika, Xuan Wang

Organizations: George Mason University, USA

Abstract

Adapting a reinforcement learning policy to changes in another agent's behavior typically requires a large amount of new interaction data. Policy sensitivity provides a first-order prediction of how a locally optimal policy changes with a behavioral parameter, but its computation requires second-order derivatives whose effects propagate across future interactions. We develop a finite-depth framework to estimate this sensitivity by approximating the policy Hessian and mixed derivative using information from a reference environment. The method features an adjustable propagation depth which determines where derivative propagation along the trajectory is truncated. We characterize the derivative contributions omitted by finite-depth propagation and derive truncation-error bounds for the approximated derivatives and resulting policy sensitivity. The bounds are nonincreasing with propagation depth and vanish at full-horizon propagation. Using a belief-driven pursuit-evasion game as a validation scenario, the proposed method generally achieves lower derivative-estimation errors as the propagation depth increases and outperforms the baseline methods in both estimation accuracy and policy adaptation. The sensitivity-based initialization improves zero-shot return over direct transfer, and also shows advantages for the subsequent fine-tuning in the target environment.

Figures & tables

Fig. 1: Adaptation to changes in an interactive agent’s behavior by finite-depth policy sensitivity estimation.

Fig. 2: Errors of

\mathbf{H}

\mathbf{M}

, and

\mathbf{S}

versus propagation depth

\mathcal{T}

Fig. 3: Relative estimation errors versus sample size

N

: the proposed method uses

N

sampled initial states, whereas the trajectory likelihood-ratio estimator uses

N

complete trajectories. Columns correspond to

\mathcal{T}=3,5,7

, from left to right.

Fig. 4: Adaptation performance for six changes in the defender’s behavioral parameter $\kappa$ . The horizontal axis reports target-environment training steps, and the vertical axis reports mean discounted return. Insets compare the two transfer methods on an expanded vertical scale.

Explore similar work

Sep 30, 2026cs.LG

Reward as Observation: Learning Reward-Based Policies for Rapid Adaptation

This paper explores a reward-based policy to achieve zero-shot transfer between source and target environments with completely different observation spaces. While humans can demonstrate impressive adaptation capabilities, deep neural network policies often struggle to adapt to a new environment and require a considerable amount of samples for successful transfer. Instead, we propose a novel reward-based policy only conditioned on rewards and actions, enabling zero-shot adaptation to new environments with completely different observations. We discuss the challenges and feasibility of a reward-based policy and then propose a practical algorithm for training. We demonstrate that a reward policy can be trained within three different environments, Pointmass, Cartpole, and 2D Car Racing, and transferred to completely different observations, such as different color palettes or 3D rendering, or Stretch robot navigation in Habitat-Sim, in a zero-shot manner. We also demonstrate that a reward-based policy can further guide the training of an observation-based policy in the target environment.

Morgan Byrd, Jacob Blevins, Maks Sorokin +2

Georgia Institute of Technology, Atlanta, GA, 30308, USA · Georgia Tech Research Institute, Atlanta, GA, 30308, USA

Neural Policies Rapid Adaptation

Ask PDF

Apr 20, 2026cs.LG

Does "Do Differentiable Simulators Give Better Policy Gradients?'' Give Better Policy Gradients?

In policy gradient reinforcement learning, access to a differentiable model enables 1st-order gradient estimation that accelerates learning compared to relying solely on derivative-free 0th-order estimators. However, discontinuous dynamics cause bias and undermine the effectiveness of 1st-order estimators. Prior work addressed this bias by constructing a confidence interval around the REINFORCE 0th-order gradient estimator and using these bounds to detect discontinuities. However, the REINFORCE estimator is notoriously noisy, and we find that this method requires task-specific hyperparameter tuning and has low sample efficiency. This paper asks whether such bias is the primary obstacle and what minimal fixes suffice. First, we re-examine standard discontinuous settings from prior work and introduce DDCG, a lightweight test that switches estimators in nonsmooth regions; with a single hyperparameter, DDCG achieves robust performance and remains reliable with small samples. Second, on differentiable robotics control tasks, we present IVW-H, a per-step inverse-variance implementation that stabilizes variance without explicit discontinuity detection and yields strong results. Together, these findings indicate that while estimator switching improves robustness in controlled studies, careful variance control often dominates in practical deployments.

Ku Onoda, Paavo Parmas, Manato Yaguchi +1

The University of Tokyo, Tokyo, Japan

Policy Gradient Differentiable Physics

Ask PDF

Nov 25, 2025cs.LG

Attention Trajectories as a Diagnostic Axis for Deep Reinforcement Learning

The emergence and evolution of feature reliance in deep reinforcement learning agents remain poorly understood. Here, we introduce a methodological framework for analyzing the learning process through quantitative analysis of saliency maps. This approach aggregates saliency information at the object and modality level into hierarchical attention profiles, quantifying how agents allocate attention over time, thereby forming attention trajectories throughout training. These profiles are then compared across controlled conditions, connected to behavioral measurements and reproduced with different saliency methods to assess the robustness of the findings. Applied to Atari 2600 benchmarks, custom Pong environments, and biomechanical user simulations in visuomotor tasks, this framework uncovers algorithm-specific attention biases, diagnosed unintended reward-driven strategies, and overfitting to redundant sensory channels. These patterns correspond to measurable behavioral differences, demonstrating empirical links between attention profiles, learning dynamics, and agent behavior. The results establish attention trajectories as a promising diagnostic axis for tracing how feature reliance develops during training and for identifying biases and vulnerabilities invisible to performance metrics alone.

Charlotte Beylier, Hannah Selder, Arthur Fleig +2

Max Planck Institute for Human Cognitive and Brain Sciences · Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI) Dresden/Leipzig · Leipzig, Germany +2

Offline Reinforcement Learning Saliency Sensory Feedback Diagnosis

Ask PDF