Although existing frameworks for large language model (LLM) inference on CPUs are mature, they fail to fully exploit the computation potential of many-core CPU platforms. Many-core CPUs are widely deployed in web servers and high-end networking devices, and are typically organized into multiple NUMA nodes that group cores and memory. Current frameworks largely overlook the substantial overhead of cross-NUMA memory access, limiting inference scalability and intelligence enabling on such platforms. To address this limitation, we build ArcLight, a lightweight LLM inference architecture designed from the ground up for many-core CPUs. ArcLight integrates efficient memory management and thread scheduling, and introduces finely controlled tensor parallelism to mitigate the cross-node memory access wall. Experimental results show that ArcLight significantly surpasses the performance ceiling of mainstream frameworks, achieving up to 46% higher inference throughput. Moreover, ArcLight maintains compatibility with arbitrary CPU devices. ArcLight is publicly available at https://github.com/OpenBMB/ArcLight.
Figures & tables
Figure 1: For many-core CPU platforms, CPU cores and memory are organized using NUMA nodes. Here shows a platform with 128 cores and 4 NUMA nodes.
Figure 2: Architecture of ArcLight .
Figure 3: Comparison of UMA (top) and NUMA (bottom) buffer management strategy. In the UMA strategy, the NUMA nodes are transparent.
Figure 4: Double-buffering mechanism.
Figure 5: Multi-view thread organization.
Figure 6: Local barrier and global barrier. “g0t0” denotes the 0-th thread in group 0.
Cores from
Memory in
node 0
node 1
node 2
node 3
node 0
102
26
24
23
node 1
26
103
23
22
node 2
24
23
103
26
node 3
23
22
26
101
Table 1: The result of memory access speed (GB/s) of different core-memory combinations.
Figure 7: GEMM memory affinity in llama.cpp . Blue denotes the weight, and red denotes the activation.
Figure 8: Simplified computation graph of vanilla MLP, tensor parallel MLP and Attention.
Figure 9: Comparison of different ways of thread group synchronization. The dashline represents a barrier.
Figure 10: Decoding speed using a single NUMA node.
Figure 11: Decoding speed using multiple NUMA nodes. “N” is the number of NUMA nodes.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Decoding speed using multiple NUMA node when the length of prompt equals 300. “N” is the number of NUMA nodes.
Figure 13: Prefilling speed using multiple NUMA node when the length of prompt equals 300. “N” is the number of NUMA nodes.