Despite the data-rich environment in which memory systems of modern computing platforms operate, many state-of-the-art architectural policies employed in the memory system rely on static, human-designed heuristics that fail to truly adapt to the workload and system behavior via principled learning methodologies. In this article, we propose a fundamentally different design approach: using lightweight and practical machine learning (ML) methods to enable adaptive, data-driven control throughout the memory hierarchy. We present three ML-guided architectural policies: (1) Pythia, a reinforcement learning-based data prefetcher for on-chip caches, (2) Hermes, a perceptron learning-based off-chip predictor for multi-level cache hierarchies, and (3) Sibyl, a reinforcement learning-based data placement policy for hybrid storage systems. Our evaluation shows that Pythia, Hermes, and Sibyl significantly outperform the best-prior human-designed policies, while incurring modest hardware overheads. Collectively, this article demonstrates that integrating adaptive learning into memory subsystems can lead to intelligent, self-optimizing architectures that unlock performance and efficiency gains beyond what is possible with traditional human-designed approaches.
Figures & tables
Figure 1: (a) Coverage, overprediction, and (b) performance comparison of two recently-proposed prefetchers: SPP and Bingo. (c) Percentage of loads that miss the LLC and go off-chip (on the left y-axis) and the LLC MPKI (on the right y-axis) in the baseline system with a state-of-the-art prefetcher. (d) Accuracy of two off-chip predictors, HMP and TTP. Average request latency of CDE and RNN-HSS on (e) performance-oriented and (f) cost-oriented HSS. The average request latency is normalized to Fast-Only policy. Figures adapted from our MICRO 2021 [ 27 ] , MICRO 2022 [ 26 ] , and ISCA 2022 [ 175 ] papers.
Figure 2: Overview of a reinforcement learning system.
Figure 3: Overview of a single-layer perceptron model.
Figure 4: (a) Formulating prefetcher as an RL agent. (b) Overview of Pythia. Figures adapted from our MICRO 2021 paper [ 27 ] .
Figure 5: Average performance improvement of prior prefetchers and Pythia in systems with varying (a) number of cores and (b) DRAM million transfers per second (MTPS). Figures adapted from our MICRO 2021 paper [ 27 ] .
Figure 6: (a) Overview of Hermes. (b) Hardware stages to make a prediction by POPET. Figures adapted from our MICRO 2022 paper [ 26 ] .
Figure 7: Performance comparison by varying (a) off-chip predictors and (b) the underlying prefetcher. Figures adapted from our MICRO 2022 paper [ 26 ] .
Figure 8: (a) Formulation of data placement as an RL Problem. (b) Design of Sibyl. Figures adapted from our ISCA 2022 paper [ 175 ] .
Feature
Description
# of bins
Encoding (bits)
sizet
Size of the requested page (in pages)
8
8
typet
Type of the current request (read/write)
2
4
intrt
Access interval of the requested page
64
8
cntt
Access count of the requested page
64
8
capt
Remaining capacity in the fast storage device
8
8
currt
Current placement of the requested page (fast/slow)
2
4
Table 1: State features used by Sibyl
Figure 9: Average request latency under two different hybrid storage configurations (normalized to Fast-Only). Figures adapted from our ISCA 2022 paper [ 175 ] .
Modern applications process massive data volumes that overwhelm the storage and retrieval capabilities of memory systems, making memory the primary performance and energy-efficiency bottleneck of computing systems. Although many microarchitectural techniques attempt to hide or tolerate long memory access latency, rapidly growing data footprints continue to outpace technology scaling, requiring more effective solutions. This dissertation shows that modern processors observe large amounts of application and system data during execution, yet many microarchitectural mechanisms make decisions largely independent of this information. Through four case studies, we demonstrate that such data-agnostic design leads to substantial missed opportunities for improving performance and energy efficiency. To address this limitation, this dissertation advocates shifting microarchitecture design from data-agnostic to data-informed. We propose mechanisms that (1) learn policies from observed execution behavior (data-driven design) and (2) exploit semantic characteristics of application data (data-aware design). We apply lightweight machine learning techniques and previously underexplored data characteristics across four processor components: a reinforcement learning-based hardware data prefetcher that learns memory access patterns online; a perceptron predictor that identifies memory requests likely to access off-chip memory; a reinforcement learning mechanism that coordinates data prefetching and off-chip prediction; and a mechanism that exploits repeatability in memory addresses and loaded values to eliminate predictable load instructions. Our extensive evaluation shows that the proposed techniques significantly improve performance and energy efficiency compared to prior state-of-the-art approaches.
Solid-state storage architectures based on NAND or emerging memory devices (SSD), are fundamentally architected and optimized for both reliability and performance. Achieving these simultaneous goals requires co-design of memory components with firmware-architected Error Management (EM) algorithms for density- and performance-scaled memory technologies. We describe a Machine Learning (ML) for systems methodology and modeling for co-designing the EM subsystem together with the natural variance inherent to scaled silicon process of memory components underlying SSD technology. The modeling analyzes NAND memory components and EM algorithms interacting with comprehensive suite of synthetic (stress-focused and JEDEC) and emulation (YCSB and similar) workloads across Flash Translation abstraction layers, by leveraging a statistically interpretable and intuitively explainable ML algorithm. The generalizable co-design framework evaluates several thousand datacenter SSDs spanning multiple generations of memory and storage technology. Consequently, the modeling framework enables continuous, holistic, data-driven design towards generational architectural advancements. We additionally demonstrate that the framework enables Representation Learning of the EM-workload domain for enhancement of the architectural design-space across broad spectrum of workloads.
Jay Sarkar, Vamsi Pavan Rayaprolu, Abhijeet Bhalerao
Reducing latency and energy consumption is critical to improving the efficiency of memory systems in modern computing. This work introduces ReLMXEL (Reinforcement Learning for Memory Controller with Explainable Energy and Latency Optimization), a explainable multi-agent online reinforcement learning framework that dynamically optimizes memory controller parameters using reward decomposition. ReLMXEL operates within the memory controller, leveraging detailed memory behavior metrics to guide decision-making. Experimental evaluations across diverse workloads demonstrate consistent performance gains over baseline configurations, with refinements driven by workload-specific memory access behaviour. By incorporating explainability into the learning process, ReLMXEL not only enhances performance but also increases the transparency of control decisions, paving the way for more accountable and adaptive memory system designs.
Panuganti Chirag Sai, Gandholi Sarat, R. Raghunatha Sarma +2
Department of Mathematics and Computer Science, Sri Sathya Sai Institute of Higher Learning · Department of Computer Science and Engineering, Indian Institute of Technology Ropar · AI Performance Engineer, Red Hat