CUTEv2: Unified and Configurable Matrix Extension for Diverse CPU Architectures with Minimal Design Overhead
Authors: Jinpeng Ye, Chongxi Wang, Wenqing Li, Bin Yuan, Shiyi Wang, Fenglu Zhang, Junyu Yue, Jianan Xie, Yunhao Ye, Haoyu Deng, Yingkun Zhou, Xin Cheng, Fuxin Zhang, Jian Wang
Organizations: State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Matrix extensions have emerged as an essential feature in modern CPUs to address the surging demands of AI workloads. However, existing designs often incur substantial hardware and software design overhead. Tight coupling with the CPU pipeline complicates integration across diverse CPUs, while fine-grained synchronous instructions hinder the development of high-performance kernels. This paper proposes a unified and configurable CPU matrix extension architecture. By decoupling matrix units from the CPU pipeline, the design enables low-overhead integration while maintaining close coordination with existing compute and memory resources. The configurable matrix unit supports mixed-precision operations and adapts to diverse compute demands and memory bandwidth constraints. An asynchronous matrix multiplication abstraction with flexible granularity conceals hardware details, simplifies matrix-vector overlap, and supports a unified software stack. The architecture is integrated into four open-source CPU RTL platforms and evaluated on representative AI models. Matrix unit utilization under GEMM workloads exceeds 90% across all platforms. When configured with compute throughput and memory bandwidth comparable to Intel AMX, our design achieves speedups of 1.57x, 1.57x, and 2.31x on ResNet, BERT, and Llama3, with over 30% of the gains attributed to overlapped matrix-vector execution. A 4 TOPS@2GHz matrix unit occupies only 0.53 mm\textsuperscript{2} in 14nm CMOS. These results demonstrate strong cross-platform adaptability and effective hardware-software co-optimization, offering a practical matrix extension for the open-source community.
Figures & tables
Figure 1. AI Model Architectures and Kernel Fusion Patterns.
Figure 2. Existing CPU Matrix Extensions.
Field
Type
Description
{M,N,K}
uint32
Matrix Size
Base {A,B,Bias,C}
uint64
Memory Base Address
Stride {A,B,Bias,C}
uint32
Memory Stride
DataType
enum
Data Precision
BiasType
enum
Bias Type (Zero, Row-Repeat, Full)
Transpose
bool
Result Transpose Flag
Table 1. Interface Registers.
Figure 3. Architecture Overview.
Figure 4. Matrix Unit Microarchitecture.
Parameters
Meaning
Case Study
Freq
Clock Frequency
2.0 GHz
Mpe
Row of PE Array
4
Npe
Column of PE Array
4
Kpe
PE Reduce Width
512 Bits
Mscp
Max Resident M in Scratchpad
64
Nscp
Max Resident N in Scratchpad
64
Table 2. Configurable Architectural Parameters.
Figure 5. Vector and Matrix overlap.
CPU
Micro Architecture
Interface
Code*
Time
Rocket ( Asanović et al., 2016 )
In-order, 1-issue
RoCC
254
3 days
Shuttle ( Zhao et al., 2023 )
In-order, 3-issue
RoCC
512
5 days
BOOM ( Zhao et al., 2020 )
Out-of-order, 4-issue
RoCC
301
3 days
Xiangshan ( Xu et al., 2023 )
Out-of-order, 6-issue
CSR
361
3 weeks
Table 3. Development Cost of Integration.
Parameter
Configuration
SoC Framework
Chipyard ( Amid et al., 2020 ) + Verilator + DRAMSim ( Rosenfeld et al., 2011 )
CPU Core
Rocket / Shuttle / BOOM / Kunminghu*
Vector Unit
512-bit RVV Saturn ( Zhao et al., 2024 )
PE Array Size
2×2 / 4×4 / 8×8 / 16×16
PE Reduce Width
256 / 512 Bits
Clock Frequency
2.0 GHz
Table 4. Experimental Setup and Configuration.
Platform
Framework
ISE
Bandwidth
INT8 Peak
IBM S1022
ONNX Runtime ( developers, 2021 )
MMA
52.37 GB/s
2 TOPS
Apple M4
ONNX Runtime ( developers, 2021 )
SME
131.31 GB/s
4 TOPS
Xeon 8580
OpenVINO ( developers, 2025 )
AMX
49.48 GB/s
4.6 TOPS
Table 5. Baseline configurations
Figure 6. GEMM Performance on Various CPU Platforms.
Figure 7. GEMM Performance under Different Config.
Figure 8. GEMM Performance vs. Existing Extensions.
Figure 9. ResNet-50 Performance vs. Existing Extensions.
Xeon 8580
IBM S1022
Apple M4
R
B
L
R
B
L
R
B
L
Ours unfused
1.19
1.28
1.87
7.16
2.72
2.39
3.82
1.72
2.55
Ours fused
1.57
1.57
2.31
8.87
3.33
3.08
5.04
2.11
3.16
Table 6. Speed Up.
Figure 10. BERT Performance vs. Existing Extensions.
Figure 11. Llama3 Performance vs. Existing Extensions.
The currently dominant AI/ML workloads, such as Large Language Models (LLMs), rely on the efficient execution of General Matrix-Matrix Multiplication (GEMM) operations. Thus, most systems are equipped with dedicated matrix hardware accelerators based on square Systolic Arrays (SAs) of Processing Elements (PEs). While this organization was effective for traditional Deep Neural Networks (DNNs), LLMs introduce input-dependent and highly skewed matrices, leading to underutilized SA resources. To address this challenge, we propose SISA (Scale-In Systolic Array), a novel SA architecture that partitions the traditional square array into horizontal rectangular slabs. With minimal overhead, SISA exposes parallelism through independently scheduled slabs for efficient execution of small or skewed matrix shapes, while retaining full-array operation for large GEMMs. SISA achieves up to 8.52x speedup and 93% energy-delay-product (EDP) reduction for representative LLMs compared to a state-of-the-art monolithic SA with the same number of PEs.
Luigi Altamura, Alessio Cicero, Mateo Vázquez Maceiras +2
Department of Computer Science and Engineering, Chalmers University of Technology and University of Gothenburg, Gothenburg, Sweden
Modern CPUs increasingly integrate matrix extensions, such as Arm Scalable Matrix Extension (SME), that provide high-throughput matrix execution within the CPU. For LLM inference, however, these units are not a universal replacement for conventional CPU cores: prefill, decode, attention, and KV-cache operations expose different arithmetic intensities, vector behavior, and layout requirements, while SME units and CPU cores still compete for shared memory bandwidth. This paper studies this mismatch through a roofline-based characterization of SME-enabled CPUs and uses the resulting model to guide operator-level execution choices. We present SMEPilot, an LLM inference engine that selects CPU-only, SME-only, or cooperative SME+CPU execution for each operator shape. SMEPilot partitions matrix work across SME and CPU cores at tile granularity, overlaps SME-suitable matrix stages with CPU-suitable vector stages in attention, and maintains layout state so packed tensor representations are reused rather than repeatedly rebuilt on critical paths. Across Llama-3.2-3B, Qwen3-4B, and Qwen3-30BA3B on phone, PC, and server platforms, SMEPilot improves end-to-end inference performance by up to 3.94×.
General Matrix Multiplication (GEMM) is the cornerstone of HPC workloads and Deep Learning. State-of-the-art (SOTA) vendor libraries tune tensor layouts, parallelization schemes and cache blocking to minimize data movement across the memory hierarchy and maximize throughput. However, optimal settings for these parameters depend on the target platform and matrix shapes, making exhaustive tuning infeasible. In this work, we address this cumbersome scheduling search and tuning using space-filling curves (SFC). We partition the matrix multiplication using advancements in SFC, and obtain platform-oblivious and shape-oblivious matrix multiplication schemes with a high degree of data locality. We extend the SFC-based work partitioning to implement Communication-Avoiding (CA) algorithms with replication techniques in a seamless fashion. The resulting SFC-CA GEMM achieves provable asymptotic communication optimality for both square and rectangular matrix regimes. Across four x86 and Arm platforms, SFC-CA GEMM outperforms vendor libraries by up to 5.5× per shape and 1.8× in weighted harmonic mean (WHM) throughput. Last, we show the impact of our work on two real-world applications by leveraging our SFC-CA GEMM as a compute backend: i) prefill of LLM inference with speedups up to 1.85× over SOTA inference runtimes, and ii) distributed-memory matrix multiplication with speedups up to 2.3× over the SOTA distributed-memory GEMM framework with vendor-optimized compute backend.
Evangelos Georganas, Alexander Heinecke, Pradeep Dubey