cs.SEMar 6, 2026

A Reference Architecture of Reinforcement Learning Frameworks

Authors: Xiaoran Liu, Istvan David

Organizations: McMaster University, Hamilton, Canada · McMaster Centre for Software Certification, Hamilton, Canada

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The surge in reinforcement learning (RL) applications gave rise to diverse supporting technology, such as RL frameworks. However, the architectural patterns of these frameworks are inconsistent across implementations and there exists no reference architecture (RA) to form a common basis of comparison, evaluation, and integration. To address this gap, we propose an RA of RL frameworks. Through a grounded theory approach, we analyze 18 state-of-the-practice RL frameworks and, by that, we identify recurring architectural components and their relationships, and codify them in an RA. To demonstrate our RA, we reconstruct characteristic RL patterns. Finally, we identify architectural trends, e.g., commonly used components, and outline paths to improving RL frameworks.

Figures & tables

Fig. 1 : High-level architectural view of RL frameworks. (Component groups are underlined and correspond to the subsections of Sec. IV .)
Fig. 2 : Experiment Orchestrator
ComponentFrameworks
Benchmark Manager[ 10 ]
Experiment Manager[ 33 ] [ 57 ] [ 58 ] [ 41 ] [ 30 ] [ 31 ] [ 10 ] [ 22 ] [ 16 ] [ 81 ]
Hyperparameter Tuner[ 58 ] [ 41 ] [ 31 ]
TABLE I : Experiment Orchestrator Components
Fig. 3 : Framework Orchestrator
ComponentFrameworks
Configuration Manager[ 46 ] [ 41 ] [ 31 ] [ 10 ] [ 22 ] [ 16 ] [ 81 ]
Distributed Exec. Coord.[ 41 ] [ 30 ] [ 31 ] [ 22 ]
Lifecycle Manager[ 33 ] [ 57 ] [ 58 ] [ 41 ] [ 30 ] [ 31 ] [ 10 ] [ 22 ] [ 16 ] [ 81 ]
Multi-Agent Coord.[ 72 ] [ 33 ] [ 46 ] [ 41 ] [ 31 ] [ 10 ] [ 81 ]
TABLE II : Framework Orchestrator Components
Fig. 4 : Agent
ComponentFrameworks
Buffer[ 33 ] [ 57 ] [ 58 ] [ 41 ] [ 30 ] [ 31 ] [ 10 ] [ 22 ] [ 16 ] [ 81 ]
Func. Approx.[ 33 ] [ 57 ] [ 58 ] [ 41 ] [ 30 ] [ 31 ] [ 10 ] [ 22 ] [ 16 ] [ 81 ]
Learner[ 33 ] [ 57 ] [ 58 ] [ 41 ] [ 30 ] [ 31 ] [ 10 ] [ 22 ] [ 16 ] [ 81 ]
TABLE III : Agent Components
Fig. 5 : Environment
ComponentFrameworks
Environment Core[ 77 ] [ 72 ] [ 33 ] [ 45 ] [ 46 ] [ 78 ] [ 5 ] [ 6 ] [ 11 ]
Simulator[ 77 ] [ 72 ] [ 33 ] [ 45 ] [ 46 ] [ 78 ] [ 5 ] [ 6 ] [ 11 ]
Simulator Adapter[ 77 ] [ 72 ] [ 33 ] [ 45 ] [ 46 ] [ 78 ] [ 5 ] [ 6 ]
TABLE IV : Environment Components
Fig. 6 : Data Persistence
ComponentFrameworks
Checkpoint Mgr[ 33 ] [ 11 ] [ 57 ] [ 58 ] [ 41 ] [ 30 ] [ 31 ] [ 10 ] [ 22 ] [ 16 ] [ 81 ]
Env. Parameter Mgr[ 33 ] [ 45 ] [ 46 ] [ 6 ]
TABLE V : Data Persistence Components
Fig. 7 : Monitoring and Visualization
Comp.Frameworks
Logger[ 77 ] [ 72 ] [ 33 ] [ 45 ] [ 46 ] [ 78 ] [ 5 ] [ 6 ] [ 11 ] [ 57 ] [ 58 ] [ 41 ] [ 30 ] [ 31 ] [ 10 ] [ 22 ] [ 16 ] [ 81 ]
Recorder[ 77 ] [ 72 ] [ 45 ] [ 46 ] [ 5 ] [ 6 ] [ 57 ] [ 58 ] [ 30 ] [ 10 ] [ 16 ]
Renderer[ 77 ] [ 72 ] [ 33 ] [ 45 ] [ 46 ] [ 78 ] [ 5 ] [ 6 ] [ 11 ]
Reporter[ 77 ] [ 72 ] [ 33 ] [ 45 ] [ 57 ] [ 58 ] [ 41 ] [ 31 ] [ 10 ] [ 22 ] [ 16 ] [ 81 ]
TABLE VI : Monitoring & Visualization Components
ComponentContainerRole
Experiment OrchestratorFrameworkTranslates user-defined experiment specifications into the concrete execution process.
Experiment ManagerExperiment OrchestratorSets up the Data Persistence and Monitor & Visualization components and delegates the experiment execution to the Framework Orchestrator.
Hyperparameter TunerExperiment OrchestratorGenerates candidate hyperparameter configurations.
Benchmark ManagerExperiment OrchestratorEnables algorithm comparison under consistent experimental settings.
Framework OrchestratorFramework CoreInitializes required framework components and coordinates their operation during training or evaluation.
Lifecycle ManagerFramework OrchestratorControls the execution of the agent–environment interaction loop.
TABLE VII : Summary of Components and their Responsibilities
Fig. 8 : Reconstruction of Discrete Policy Gradient
Fig. 9 : Reconstruction of Q-learning
Fig. 10 : Reconstruction of Advantage Actor-Critic (A2C)
Fig. 11 : Reconstruction of MARL (with centralized learning)
Fig. 12 : Implementations of RA components across the sampled RL systems. ( Utility components not included.)
ADDDefinitionAffected RA components
Model ArchitectureThe structural organization of RL modelsAgent , Multi-Agent Coordinator
Model TrainingThe organization of learning across agentsMulti-Agent Coord. , Distributed Execution Coord. , Learner , Buffer
CheckpointsWhether RL models’ states are saved during learningCheckpoint Manager
Transfer LearningWhether to use pre-trained models or train from scratchCheckpoint Manager , Agent
Distribution StrategyWhether and how to use distributed trainingDistributed Execution Coordinator
Hyperparameter TuningWhether to use hyperparameter tuning or notHyperparameter Tuner
TABLE VIII : Localizing Architectural Design Decisions
More in this topic

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Explore similar work

CardsList

  1. From RL Papers to Framework Code: Inclusion and Review in Seven Open-Source Training Frameworks

    Oct 20, 2026Taiheng PanRL Post-TrainingRL for Language Models

    The University of Melbourne

  2. CDRL: A Reinforcement Learning Framework Inspired by Cerebellar Circuits and Dendritic Computational Strategies

    Feb 17, 2026Sibo Zhang, Rui Jing, Liangfu Lv +2Reinforcement LearningSample-Efficient RL

    Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, China.

  3. FARL: A Fully Automated Pipeline of RL Interface and Solver Design

    Oct 11, 2026Tianyu Jin, Liuyu Xiang, Chaowen Yang +3Reinforcement LearningRobotic RL

    Beijing University of Posts and Telecommunications, Beijing, China