Waypoint: Analytically Modeling GEMM for CPUs

Srinidhi N; Sunanda Somwase; Hemant Kumar Chandrakant Chandavar; Eashan Dash; Varun Agrawal; Prakash Raghavendra

cs.DCOct 26, 2026

Waypoint: Analytically Modeling GEMM for CPUs

Authors: Srinidhi N, Sunanda Somwase, Hemant Kumar Chandrakant Chandavar, Eashan Dash, Varun Agrawal, Prakash Raghavendra

Organizations: Advanced Micro Devices, Inc. India

Abstract

High-performance General Matrix Multiplication (GEMM) implementations rely on breaking down GEMM into carefully selected tiles to achieve peak performance. We present Waypoint, a new analytical model to automatically identify the optimal tile size for a given hardware and input matrix shape. Waypoint estimates the achievable performance for a given tile choice by capturing all the architectural effects including interactions with memory hierarchy, instruction-level parallelism, and multi-threading. Using the estimates as a proxy, we search through the space of possible tile sizes to identify the one that maximizes performance. Additionally, Waypoint helps hardware designers estimate realistic performance on experimental hardware optimizations. We build a new GEMM library OPAM for CPUs that uses Waypoint to automatically break down GEMM and achieve performance that matches state-of-the-art GEMM libraries and helps accelerate LLM inference on CPUs achieving $1.36 - 1.98 \times$ speedups.

Figures & tables

Figure 1. Additional latency captured by the Waypoint - Strict relative to Roofline across GEMM shapes, separated by hardware and data type. Boxes show the interquartile range and median; points show individual shapes.

Figure 2. Candidate register tile sizes

(M_{r},N_{r})

for AMD ‘Zen 5’ architecture. The candidates are generated by enumerating all possible values of

M_{r}

and

N_{r}

that satisfy the register pressure and load-to-fma latency ratio bounds.

Panel of Figure 3 — Figure 3. High-level workflow for evolving a Waypoint CPU split-factor heuristic. Seed heuristics and LLM-generated mutations are evaluated against pinned exhaustive-search optima; the resulting metrics drive selection of the best heuristic.

Category	Count
Small GEMM (<512)	4
Medium Square (512-4096)	9
Large GEMM (>4096)	8
Tall skinny (M>>N)	4
Fat Short (M<<N)	2
Odd Shapes	4

Table 1. Evaluated GEMM shapes

Datatype	Threads	n	Median	Mean	Max
Int8	1	58	1.001x	1.005x	1.031x
Int8	2-8	24	1.000x	1.001x	1.011x
Int8	>8	20	1.000x	1.001x	1.002x
BFloat16	1	58	1.001x	1.004x	1.031x
BFloat16	2-8	24	1.000x	1.002x	1.013x
BFloat16	>8	20	1.001x	1.001x	1.005x

Table 2. Roofline vs Waypoint - Strict comparison for different data types across architectures. We report the median, mean, and max ratios of Waypoint - Strict to Roofline estimates across different shapes. A value of ‘1.000x‘ means Waypoint - Strict and Roofline estimate match.

Figure 4. Waypoint - Tile vs Waypoint - Strict vs Implementations ( AOCL-BLAS , SFC , and AOCL-DLP ). We report the ratios and distribution for all the shapes. A comparison of the \codenamerefined{}, \codenamesimple{}, and actual implementations for various GEMM shapes.

ACE throughput	BFloat16	Int8
$1$	7.21x	7.45x
$2$	12.84x	13.48x
$4$	21.47x	23.00x

Table 3. Predicted geometric-mean GEMM speedup from the proposed AI Compute Extensions (ACE) across 44 shapes per datatype vs ‘Zen 5’ hardware. ACE throughput is effective number of Tile operations that can be issued per cycle.

Figure 5. Relative performance of AOCL-BLAS , SFC , AOCL-DLP , and the Waypoint - Tile vs OPAM , separated by hardware architecture.

Figure 6. End-to-end inference speedup of Waypoint on prefill heavy LLM workloads across different hardware and models. Bars show the geometric mean; whiskers show the full range across batch sizes and input lengths.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Shape category	Z5 Float	Z5 BF16	Z5 Int8	Z6 Float	Z6 BF16	Z6 Int8
Small Square	0.002%	0.000%	0.443%	0.010%	0.018%	0.000%
Medium Square	0.144%	0.060%	0.059%	0.242%	0.251%	0.426%
Large Square	0.089%	0.059%	0.055%	0.234%	0.295%	0.514%
Tall Skinny	0.692%	0.145%	0.473%	0.111%	0.093%	0.006%
Fat short	0.110%	0.121%	0.127%	0.106%	0.114%	0.209%
Odd Shapes	0.698%	0.651%	0.838%	0.710%	0.698%	1.590%

Appendix

Table 4. Average difference in expected latency between the fast heuristic search algorithm and the exhaustive search algorithm across different shape categories and architectures.

$M$	$N$	$K$	$T$	$M$	$N$	$K$	$T$
64	64	64	1	4096	64	4096	1
64	2048	64	1	4096	128	4096	1
96	96	96	1	4096	4096	4096	1
128	32	2880	1	128	14336	4096	4
128	128	128	1	128	28672	4096	4
128	2880	2880	1	128	128256	4096	4

Appendix

Table 5. GEMM evaluation shapes.

M

N

, and

K

are the matrix dimensions, and

T

is the number of threads.

‘Zen 5’
Hardware / Impl	Type	$1$	$2$ – $8$	$>8$
AOCL-BLAS	INT8	$2.02\times$	$3.01\times$	$10.34\times$
	BFloat16	$0.40\times$	$1.37\times$	$4.40\times$
	Float32	$1.11\times$	$1.78\times$	$5.05\times$
AOCL-DLP	INT8	$2.11\times$	$3.75\times$	$10.23\times$
	BFloat16	$0.44\times$	$1.42\times$	$4.48\times$

Appendix

Table 6. Packing cost relative to OPAM . Values are geometric-mean ratios of implementation packing cycles to OPAM packing cycles for different thread counts.

LP-GEMM: Integrating Layout Propagation into GEMM Operations
Apr 6, 2026César Guedes Carneiro, Lucas Alvarenga, Guido Araujo +1High-Performance Computing
Instituto de Computação, Universidade Estadual de Campinas, Campinas – SP, Brazil
Space Filling Curves is All You Need: Communication-Avoiding Matrix Multiplication Made Simple
Jan 22, 2026Evangelos Georganas, Alexander Heinecke, Pradeep DubeyHigh-Performance Computing
Intel Corporation
ExaGEMM: Exploration Framework for CPU-Driven ML Inference via Associative In-Register Computing for Low-Bit GEMM
Jul 16, 2026Hyunwoo Oh, Suyeon Jang, Hanning Chen +3Efficient Neural Network Inference Hardware-Software Co-Design
University of California, Irvine

Apr 6, 2026J/K move · Enter open · S save

LP-GEMM: Integrating Layout Propagation into GEMM Operations

César Guedes Carneiro, Lucas Alvarenga, Guido Araujo, Sandro Rigo

Instituto de Computação, Universidade Estadual de Campinas, Campinas – SP, Brazil

High-Performance Computing

Open paper AskPDF arXiv

In Scientific Computing and modern Machine Learning (ML) workloads, sequences of dependent General Matrix Multiplications (GEMMs) often dominate execution time. While state-of-the-art BLAS libraries aggressively optimize individual GEMM calls, they remain constrained by the BLAS API, which requires each call to independently pack input matrices and restore outputs to a canonical memory layout. In sequential GEMMs, these constraints cause redundant packing and unpacking, wasting valuable computational resources. This paper introduces LP-GEMM, a decomposition of the GEMM kernel that enables packing-layout propagation across sequential GEMM operations. This approach eliminates unnecessary data repacking while preserving full BLAS semantic correctness at the boundaries. We evaluate LP-GEMM on x86 (AVX-512) and RISC-V (RVV 1.0) architectures across MLP-like and Attention-like workloads. Our results show average speedups of 2.25x over OpenBLAS on Intel x86 for sequential GEMMs and competitive gains relative to vendor-optimized libraries such as Intel MKL. We demonstrate the practicality of the approach beyond microbenchmarks by implementing a standalone C++ version of the Llama-3.2 inference path using exclusively BLAS-level GEMM calls. These results confirm that leveraging data layout propagation between operations can significantly boost performance.

Abstract

Explore similar work

LP-GEMM: Integrating Layout Propagation into GEMM Operations

Space Filling Curves is All You Need: Communication-Avoiding Matrix Multiplication Made Simple

ExaGEMM: Exploration Framework for CPU-Driven ML Inference via Associative In-Register Computing for Low-Bit GEMM