cs.LGFeb 27, 2026

Adaptive Correlation-Weighted Intrinsic Rewards for Reinforcement Learning

Authors: Viet Bac Nguyen, Phuong Thai Nguyen

Organizations: Institute for Artificial Intelligence, VNU-University of Engineering and Technology, Hanoi, Vietnam

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We propose ACWI (Adaptive Correlation Weighted Intrinsic), an adaptive intrinsic reward scaling framework designed to dynamically balance intrinsic and extrinsic rewards for improved exploration in sparse reward reinforcement learning. Unlike conventional approaches that rely on manually tuned scalar coefficients, which often result in unstable or suboptimal performance across tasks, ACWI learns a state dependent scaling coefficient online. Specifically, ACWI introduces a lightweight Beta Network that predicts the intrinsic reward weight directly from the agent state through an encoder based architecture. The scaling mechanism is optimized using a correlation based objective that encourages alignment between the weighted intrinsic rewards and discounted future extrinsic returns. This formulation enables task adaptive exploration incentives while preserving computational efficiency and training stability. We evaluate ACWI on a suite of sparse reward environments in MiniGrid. Experimental results demonstrate that ACWI consistently improves sample efficiency and learning stability compared to fixed intrinsic reward baselines, achieving superior performance with minimal computational overhead.

Figures & tables

Fig. 1 : Overview of ACWI. Given state sts_{t} , action ata_{t} , and next state st+1s_{t+1} in environment EE , the Beta Network produces a state-dependent scaling factor β(st)\beta(s_{t}) . The extrinsic reward RtER^{E}_{t} and intrinsic reward RtIR^{I}_{t} are computed from the transition (st,at,st+1)(s_{t},a_{t},s_{t+1}) . The combined signal RtE+αβ(st)RtIR^{E}_{t}+\alpha\beta(s_{t})R^{I}_{t} is used to update the policy π\pi , where α\alpha is a global coefficient controlling the overall magnitude of the intrinsic reward relative to the extrinsic reward.
ParameterValue
PPO configuration
Discount factor γ\gamma0.99
GAE parameter λ\lambda0.95
PPO epochs KK4
Clip range ϵ\epsilon0.2
Actor learning rate3×10−43\times 10^{-4}
TABLE I : Training hyperparameters for PPO with ICM and ACWI. Fixed intrinsic coefficients for PPO+ICM baselines are β∈{0.1,0.2,0.5,1,2}\beta\in\{0.1,0.2,0.5,1,2\} .
Fig. 2 : Screenshots of the five MiniGrid environments used in our experiments [ 29 ] . From left to right: DoorKey-8x8, Empty-16x16, KeyCorridorS3R3, UnlockPickup, and RedBlueDoors-8x8. Each environment presents increasing complexity in terms of required exploration depth and compositional reasoning.
Fig. 3 : Episode returns over training steps across five MiniGrid environments [ 29 ] . Comparison between the adaptive beta mechanism (ACWI), ICM with fixed intrinsic scaling coefficients β∈{0.1,0.2,0.5,1,2}\beta\in\{0.1,0.2,0.5,1,2\} , and the PPO baseline.
Fig. 4 : Evolution of the learned state-dependent adaptive β\beta distributions during training across three MiniGrid environments. DoorKey-8x8 and RedBlueDoors-8x8 progressively develop structured and multimodal distributions, whereas Empty-16x16 maintains a relatively narrow distribution throughout training, indicating limited adaptation under extreme reward sparsity.
Fig. 5 : Principal component projections of learned state representations colored by adaptive β\beta . In DoorKey-8x8 and RedBlueDoors-8x8 , β\beta aligns with task-relevant regions of the state space. In Empty-16x16 , β\beta shows no systematic relationship with the representation geometry.
Fig. 6 : Early training state visitation heatmaps for PPO, PPO+ICM using the best fixed β\beta , and PPO+ICM+ACWI. Heatmaps are computed during the initial training phase, prior to reliable goal attainment.
More in this topic

Intrinsic Reward Methods

Intrinsic reward methods give agents internally generated learning signals to guide exploration or improvement when external rewards are sparse.

See all 30 papers

Explore similar work

CardsList

  1. Explore Beyond the Boundary Using Entropic Information

    Jul 31, 2026Bumgeun Park, Donghwan LeeReinforcement LearningRL Exploration

    Korea Advanced Institute of Science and Technology, Daejeon 34051, Korea

  2. Endogenous Exploration in Reinforcement Learning with Intrinsic Curiosity

    Sep 4, 2026Armando VieiraRL ExplorationIntrinsic Reward Methods

    University of Tartu

  3. Reward-free Pretraining for Reinforcement Learning via Occupancy Coverage Maximization

    Jun 19, 2026Marco Pratticò, Pietro Novelli, Massimiliano Pontil +1Reinforcement LearningExploration-Exploitation Tradeoff

    Computational Statistics and Machine Learning - Istituto Italiano di Tecnologia, 16100 Genova, Italy · 2AI Centre, Computer Science Department, University College London, London, UK.