CUTEv2: Unified and Configurable Matrix Extension for Diverse CPU Architectures with Minimal Design Overhead

Jinpeng Ye; Chongxi Wang; Wenqing Li; Bin Yuan; Shiyi Wang; Fenglu Zhang; Junyu Yue; Jianan Xie

cs.ARApr 13, 2026

CUTEv2: Unified and Configurable Matrix Extension for Diverse CPU Architectures with Minimal Design Overhead

Authors: Jinpeng Ye, Chongxi Wang, Wenqing Li, Bin Yuan, Shiyi Wang, Fenglu Zhang, Junyu Yue, Jianan Xie,

Organizations: State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

Abstract

Matrix extensions have emerged as an essential feature in modern CPUs to address the surging demands of AI workloads. However, existing designs often incur substantial hardware and software design overhead. Tight coupling with the CPU pipeline complicates integration across diverse CPUs, while fine-grained synchronous instructions hinder the development of high-performance kernels. This paper proposes a unified and configurable CPU matrix extension architecture. By decoupling matrix units from the CPU pipeline, the design enables low-overhead integration while maintaining close coordination with existing compute and memory resources. The configurable matrix unit supports mixed-precision operations and adapts to diverse compute demands and memory bandwidth constraints. An asynchronous matrix multiplication abstraction with flexible granularity conceals hardware details, simplifies matrix-vector overlap, and supports a unified software stack. The architecture is integrated into four open-source CPU RTL platforms and evaluated on representative AI models. Matrix unit utilization under GEMM workloads exceeds 90% across all platforms. When configured with compute throughput and memory bandwidth comparable to Intel AMX, our design achieves speedups of 1.57x, 1.57x, and 2.31x on ResNet, BERT, and Llama3, with over 30% of the gains attributed to overlapped matrix-vector execution. A 4 TOPS@2GHz matrix unit occupies only 0.53 mm\textsuperscript{2} in 14nm CMOS. These results demonstrate strong cross-platform adaptability and effective hardware-software co-optimization, offering a practical matrix extension for the open-source community.

Figures & tables

Figure 1. AI Model Architectures and Kernel Fusion Patterns.

Figure 2. Existing CPU Matrix Extensions.

Field	Type	Description
{M,N,K}	uint32	Matrix Size
Base {A,B,Bias,C}	uint64	Memory Base Address
Stride {A,B,Bias,C}	uint32	Memory Stride
DataType	enum	Data Precision
BiasType	enum	Bias Type (Zero, Row-Repeat, Full)
Transpose	bool	Result Transpose Flag

Table 1. Interface Registers.

Figure 3. Architecture Overview.

Figure 4. Matrix Unit Microarchitecture.

Parameters	Meaning	Case Study
Freq	Clock Frequency	2.0 GHz
${M_{pe}}$	Row of PE Array	4
${N_{pe}}$	Column of PE Array	4
${K_{pe}}$	PE Reduce Width	512 Bits
${M_{scp}}$	Max Resident M in Scratchpad	64
${N_{scp}}$	Max Resident N in Scratchpad	64

Table 2. Configurable Architectural Parameters.

Figure 5. Vector and Matrix overlap.

CPU	Micro Architecture	Interface	Code*	Time
Rocket ( Asanović et al., 2016 )	In-order, 1-issue	RoCC	254	3 days
Shuttle ( Zhao et al., 2023 )	In-order, 3-issue	RoCC	512	5 days
BOOM ( Zhao et al., 2020 )	Out-of-order, 4-issue	RoCC	301	3 days
Xiangshan ( Xu et al., 2023 )	Out-of-order, 6-issue	CSR	361	3 weeks

Table 3. Development Cost of Integration.

Parameter	Configuration
SoC Framework	Chipyard ( Amid et al., 2020 ) + Verilator + DRAMSim ( Rosenfeld et al., 2011 )
CPU Core	Rocket / Shuttle / BOOM / Kunminghu*
Vector Unit	512-bit RVV Saturn ( Zhao et al., 2024 )
PE Array Size	2×2 / 4×4 / 8×8 / 16×16
PE Reduce Width	256 / 512 Bits
Clock Frequency	2.0 GHz

Table 4. Experimental Setup and Configuration.

Platform	Framework	ISE	Bandwidth	INT8 Peak
IBM S1022	ONNX Runtime ( developers, 2021 )	MMA	52.37 GB/s	2 TOPS
Apple M4	ONNX Runtime ( developers, 2021 )	SME	131.31 GB/s	4 TOPS
Xeon 8580	OpenVINO ( developers, 2025 )	AMX	49.48 GB/s	4.6 TOPS

Table 5. Baseline configurations

Figure 6. GEMM Performance on Various CPU Platforms.

Figure 7. GEMM Performance under Different Config.

Figure 8. GEMM Performance vs. Existing Extensions.

Figure 9. ResNet-50 Performance vs. Existing Extensions.

	Xeon 8580			IBM S1022			Apple M4
	R	B	L	R	B	L	R	B	L
Ours unfused	1.19	1.28	1.87	7.16	2.72	2.39	3.82	1.72	2.55
Ours fused	1.57	1.57	2.31	8.87	3.33	3.08	5.04	2.11	3.16

Table 6. Speed Up.

Figure 10. BERT Performance vs. Existing Extensions.

Figure 11. Llama3 Performance vs. Existing Extensions.

	Area (mm 2 )	Power (W)
RAM	0.164	0.784
Logic	0.367	0.722
Total	0.531	1.506

Table 7. Area and Power Evaluation (4TOPS@2GHz).

Hardware-Software Co-Design

Hardware-software co-design jointly shapes algorithms, software, and computing hardware to meet performance, energy, cost, and deployment constraints.

See all 69 papers

SISA: A Scale-In Systolic Array for GEMM Acceleration
Mar 31, 2026Luigi Altamura, Alessio Cicero, Mateo Vázquez Maceiras +2LLM Inference Acceleration
Department of Computer Science and Engineering, Chalmers University of Technology and University of Gothenburg, Gothenburg, Sweden
SMEPilot: Characterizing and Optimizing LLM Inference with Scalable Matrix Extensions
Jun 15, 2026Feiyang Chen, Haibo ChenLLM Inference Efficiency LLM Inference Acceleration
IPADS, Shanghai Jiao Tong University
Space Filling Curves is All You Need: Communication-Avoiding Matrix Multiplication Made Simple
Jan 22, 2026Evangelos Georganas, Alexander Heinecke, Pradeep DubeyHigh-Performance Computing
Intel Corporation

Mar 31, 2026J/K move · Enter open · S save

SISA: A Scale-In Systolic Array for GEMM Acceleration

Luigi Altamura, Alessio Cicero, Mateo Vázquez Maceiras, Mohammad Ali Maleki+1

Department of Computer Science and Engineering, Chalmers University of Technology and University of Gothenburg, Gothenburg, Sweden

LLM Inference Acceleration

Open paper AskPDF arXiv

The currently dominant AI/ML workloads, such as Large Language Models (LLMs), rely on the efficient execution of General Matrix-Matrix Multiplication (GEMM) operations. Thus, most systems are equipped with dedicated matrix hardware accelerators based on square Systolic Arrays (SAs) of Processing Elements (PEs). While this organization was effective for traditional Deep Neural Networks (DNNs), LLMs introduce input-dependent and highly skewed matrices, leading to underutilized SA resources. To address this challenge, we propose SISA (Scale-In Systolic Array), a novel SA architecture that partitions the traditional square array into horizontal rectangular slabs. With minimal overhead, SISA exposes parallelism through independently scheduled slabs for efficient execution of small or skewed matrix shapes, while retaining full-array operation for large GEMMs. SISA achieves up to 8.52x speedup and 93% energy-delay-product (EDP) reduction for representative LLMs compared to a state-of-the-art monolithic SA with the same number of PEs.

Abstract

Hardware-Software Co-Design

Explore similar work

SISA: A Scale-In Systolic Array for GEMM Acceleration

SMEPilot: Characterizing and Optimizing LLM Inference with Scalable Matrix Extensions

Space Filling Curves is All You Need: Communication-Avoiding Matrix Multiplication Made Simple