cs.LGNov 10, 2026

StructRL: Model-Free Reinforcement Learning with Replay Graphs

Authors: Ivo Nowak

Organizations: Hamburg University of Applied Sciences (HAW Hamburg)

Abstract

StructRL turns the experience collected during reinforcement learning into an explicit replay graph. Once the graph contains a path to the task goal, dynamic programming selects actions on the observed part of the state space; these actions form a graph policy that is defined only where the required transitions have been observed. StructRL transfers these actions to model-free learners: as action preferences for a DQN Q-model and as target directions along shortest observed paths for a SAC actor. For DQN on three sparse-reward MiniGrid tasks, final success rises from 0.10 to 0.90 (DoorKey), 0.10 to 0.80 (KeyCorridor) and 0 to 0.50 (ObstructedMaze); plain DQN does not catch up with three times the budget. For SAC on continuous PointMaze, graph-derived direction labels raise final success from 0.17 to 1.0 on seven matched runs that reached the goal. Distilling the graph policy into a fresh Q-model gives a DoorKey 6x6 controller that reaches the goal from 91% of reachable states, against 38% for plain DQN. Scaling experiments identify three bottlenecks: incomplete graph support, structural aliasing, and loss of graph information in high-dimensional inputs.

Figures & tables

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Active Curriculum Refinement for Reinforcement Learning

    Aug 26, 2026Zhenya Liu, Yuxin ChenCurriculum RLCurriculum Learning

  2. Regret-Aware Policy Optimization: Environment-Level Memory for Replay Suppression under Delayed Harm

    Apr 8, 2026Prakul Sunil HiremathReinforcement LearningSafe RL

  3. Breaking the Grid: Distance-Guided Reinforcement Learning in Large Discrete Action Spaces

    Feb 9, 2026Heiko Hoppe, Fabian Akkerman, Wouter van Heeswijk +1Reinforcement Learning