From LLM to Silicon: RL-Driven ASIC Architecture Exploration for On-Device AI Inference

Ravindra Ganti; Steve Xu

cs.ARApr 8, 2026

From LLM to Silicon: RL-Driven ASIC Architecture Exploration for On-Device AI Inference

Authors: Ravindra Ganti, Steve Xu

Organizations: XgenSilicon Inc.

Abstract

We present an RL-driven compiler that jointly optimizes ASIC architecture, memory hierarchy, and workload partitioning for AI inference across 3nm to 28nm. The design space is formulated as a single Markov Decision Process with mixed discrete-continuous actions and a unified Power-Performance-Area (PPA) objective. Soft Actor-Critic (SAC) with Mixture-of-Experts gating explores the joint space of mesh topology, per-core microarchitecture, and operator placement. We validate on two workloads, Llama 3.1 8B FP16 (high-performance mode, 29809 tokens per second at 3nm) and SmolVLM (low-power mode, less than 13 mW at all nodes, 10 MHz). Across 7 process nodes, the RL automatically adapts mesh sizes and per-tile configurations, including heterogeneous FETCH, VLEN, and memory allocation without node-specific manual retuning.

Figures & tables

Panel of Figure 1 — Figure 1: End-to-end compilation pipeline. The RL optimization loop (Stage 4) receives workload features from Stage 3 and drives codegen/partitioning in Stage 5, which returns PPA reward signals (dashed arrow). Each stage emits intermediate artifacts used by downstream stages and for reproducibility.

Compiler Stack

Optimization Features

Differentiating Capabilities

System

Frontend/IR

Graph Opt.

Backend/Codegen

RL/AutoTune

Quantization

Memory Plan

PPA Model

Target

Joint HW+SW

Multi-Core

Op-Level Part.

Node Retarget

TVM [ 1 ]

✓

CPU/GPU

Ansor [ 23 ]

✓

CPU/GPU

XLA [ 3 ]

✓

TPU/GPU

✓

MLIR [ 2 ]

✓

Multi

Glow [ 4 ]

✓

CPU/Accel

Table 1: Capability comparison with prior ML compiler systems. Compiler Stack : standard compiler infrastructure. Optimization Features : RL/AutoTune (automated search), Quantization (FP16/INT8), Memory Plan (buffer/cache optimization), PPA Model (power-performance-area), Target (hardware platform). Differentiating Capabilities : features unique to our approach.

Category	Idx Range	Dims	Representative Features
Workload	0–4	5	Instruction count, ILP, memory intensity, vector util, matmul ratio
Configuration	5–25	21	Mesh size, fetch/STANUM/VLEN, DMEM/WMEM/IMEM, NoC width, ports, node
Partitioning	26–28	3	DMEM input/output/scratch allocation ratios
Load Distribution	29–32	4	Load variance, max/min load ratio, balance score
Op Partition	33–36	4	MatMul/Conv/general partitioning ratios
Hazards	37–40	4	Global RAW/WAR/WAW statistics

Table 2: State feature breakdown for RL optimization (73 total, 52 used by SAC actor)

Action Group	Idx Range	Dims	Update Scope
Discrete Mesh/SC Deltas	0–3	4	Mesh width/height and SC x/y in {-2..+2} (5-way one-hot each)
Continuous TCC Params	4–18	15	Fetch, STANUM, VLEN, DMEM/WMEM/IMEM, DFLIT, ports, clock, precision
Memory/Load Partition	19–22	4	DMEM input/output fractions and load-balance controls
Op-Partition Controls	23–25	3	MatMul/Conv/general operation split ratios across TCCs
Streaming	26–27	2	Input/output streaming ratio controls
Workload Partition	28–29	2	Sub-matmul partition and all-reduce fraction

Table 3: Action-space breakdown (30 continuous + 4 discrete mesh deltas)

Panel of Figure 2 — Figure 2: SAC actor network: $\mathbf{s}\in\mathbb{R}^{52}\to$ 2-layer MLP (256 hidden) $\to$ 80-dim output (20 discrete logits + 30 means + 30 log-stds). Actions sampled via tanh-squashed Gaussian with reparameterization.

Component	Type	Range	Description
$P_{\text{norm}}$	Reward	$[0,1]$	Norm. perf (higher=better)
$P_{\text{power}}$	Penalty	$[0,1]$	Norm. power (lower=better)
$A_{\text{norm}}$	Penalty	$[0,1]$	Norm. area (lower=better)
$B_{\text{feas}}$	Bonus	$[0,2]$	Feasibility + power margin
$P_{\text{viol}}$	Penalty	$[0,\infty)$	Cubic constraint viol.
$P_{\text{mem}}$	Penalty	$[0,\infty)$	Linear memory overuse

Table 4: Reward function components and their characteristics

	Parameter	Value
SAC	Hidden layers	[256, 256]
	LR ( $\pi$ , $Q$ , $\alpha$ )	$3\!\times\!10^{-4}$
	Batch / $\tau$	256 / 0.005
	Warmup	1,000
PER	Buffer cap.	100K
PER	$\alpha$ / $\beta$	0.6 / 0.4 $\to$ 1

Table 5: SAC, PER, MPC, and world model config.

Hyperparameter	Value	Component	Description
Actor LR / Critic LR / $\alpha$ LR	$3\times 10^{-4}$	SAC	All three learning rates
Discount Factor ( $\gamma$ )	0.99	SAC	Future reward discount
Soft Target Update ( $\tau$ )	0.005	SAC	Polyak averaging for target Q-networks
Initial Entropy ( $\alpha_{0}$ )	0.2	SAC	Auto-tuned via Eq. 45
Target Entropy	$-30$	SAC	$-d_{a}$ (negative action dim)
Mini-batch Size	256	SAC	Sampled from PER buffer

Table 6: Complete RL system hyperparameters: SAC optimizer, prioritized replay, and MPC planner.

Parameter	Min	Max	Notes
FETCH_SIZE	1	16	Instruction fetch width (per-TCC)
STANUM	1	32	Reservation stations
VLEN (bits)	128	2048	Vector register length (per-TCC)
DMEM_SIZE_KB	16	512	Data memory per tile
WMEM_SIZE_KB	256	adaptive	Weight ROM (model-dependent)
IMEM_SIZE_KB	1	128	Instruction memory per tile

Table 7: Per-TCC parameter ranges (11 parameters). Bounds are architectural limits; the RL agent selects continuous values within these bounds, which are then quantized to hardware-supported discrete values.

Component	Value	Description
Target Model	Llama 3.1 8B FP16	Decoder-only transformer model
Total Weights	14.96 GB	Weight tensors mapped to WMEM
Operators	7,489	Unified graph operator count
Weight Tensors	291	Initializer tensors used by codegen
Inputs / Outputs	66 / 65	Graph interface tensors
Process Nodes	3, 5, 7, 10, 14, 22, 28 nm	Technology nodes evaluated

Table 8: Experimental setup summary

Figure 3: RL convergence trace at 3nm: best PPA score vs. episode count over $\sim$ 4.6K episodes, showing exploration-to-exploitation transition.

Characteristic	Value	Description
Model Family	Llama 3.1 8B FP16	Decoder-only transformer
Parameters	8.03B	Total model parameters
Operators	7489	Unified graph operators
Total Weights	14.96 GB	FP16 weight footprint
Best Mesh	41x42	Best node mesh (3nm)
Best Throughput	29809 tok/s	Batch=3 seq-len=2048

Table 9: Llama 3.1 8B FP16 model characteristics and run statistics

Node	Mesh	Cores	Scaling	Power (mW)	Perf (GOps)	Area (mm 2 )	PPA
3nm	41x42	1722	1.00x	51366	466364	648	0.974
5nm	39x39	1521	0.88x	57153	338116	929	0.989
7nm	33x34	1122	0.65x	46208	173899	1220	0.996
10nm	26x27	702	0.41x	25134	99939	1572	1.005
14nm	21x22	462	0.27x	14161	51072	1992	1.016
22nm	16x16	256	0.15x	7093	18077	2882	1.023

Table 10: Per-process-node RL results for Llama 3.1 8B FP16. Scaling factor is core count relative to 3nm (1722 cores).

Node	Mesh	Cores	Freq (MHz)	Power (mW)	Perf (GOps)	Area (mm 2 )	PPA	Tok/s
3nm	41x42	1722	1000	51366	466364	648	0.974	29809
5nm	39x39	1521	820	57153	338116	929	0.989	21612
7nm	33x34	1122	570	46208	173899	1220	0.996	11115
10nm	26x27	702	520	25134	99939	1572	1.005	6388
14nm	21x22	462	400	14161	51072	1992	1.016	3264
22nm	16x16	256	250	7093	18077	2882	1.023	1155

Table 11: Optimized PPA metrics across process nodes for Llama 3.1 8B FP16

Node	Mesh	Compute	SRAM	ROM Rd	NoC	Leak	Total	Comp%	SRAM%	ROM%	NoC%	Leak%
3nm	41x42	27517	1324	2779	17116	2631	51366	53.6	2.6	5.4	33.3	5.1
5nm	39x39	30774	1439	2634	19143	3163	57153	53.8	2.5	4.6	33.5	5.5
7nm	33x34	25241	1178	1914	15701	2175	46208	54.6	2.5	4.1	34.0	4.7
10nm	26x27	15138	726	1398	6676	1196	25134	60.2	2.9	5.6	26.6	4.8
14nm	21x22	9592	467	702	2784	616	14161	67.7	3.3	5.0	19.7	4.4
22nm	16x16	5559	276	222	894	142	7093	78.4	3.9	3.1	12.6	2.0

Table 12: Per-TCC dynamic power breakdown across process nodes for Llama 3.1 8B FP16. Power values in mW.

Panel of Figure 4 — Figure 4: PPA score and mesh scaling across 7 process nodes.

Panel of Figure 5 — Figure 5: PPA metric decomposition across process nodes: (a) power, (b) performance, (c) area.

Figure 6: Inference throughput (tokens/s) by process node for Llama 3.1 8B FP16.

Figure 7: Derived efficiency metrics by process node. Smaller nodes achieve higher power and area efficiency, though with diminishing returns below 7nm.

Figure 8: Pearson correlation matrix across PPA metrics. Strong positive correlations between Performance and Power reflect the mesh-size coupling; PPA Score captures the composite tradeoff.

Figure 9: Log-log trend fits for performance, power, and area versus process node. Fit equations and $R^{2}$ values shown per panel.

Analysis	Metric	Slope/Corr	Const	R 2 /Note
log-log fit	Performance (GOps/s)	-1.3284	1850000.0	0.9856
log-log fit	Power (mW)	-0.9012	420000.0	0.9234
log-log fit	Area (mm2)	0.7521	195.0	0.9948
pearson corr	Perf vs Power	0.9812	-	node-level
pearson corr	Perf vs Area	-0.8234	-	node-level
pearson corr	Perf vs PPA	-0.9456	-	node-level

Table 13: Node-level statistical analysis: fitted scaling exponents and pairwise correlations.

Run Metric	Value	Description
Evaluated Nodes	7	3nm 5nm 7nm 10nm 14nm 22nm 28nm
Best Node	3nm	Lowest PPA score in this run
Best Mesh	41x42	Active TCC mesh at best node
Best PPA Score	0.974	Optimal configuration score
Best Throughput	29809 tok/s	Batch=3; seq-len=2048
Optimization Mode	high-performance	Maximize throughput

Table 14: High-performance run statistics for Llama 3.1 8B FP16

Figure 10: Heterogeneous weight memory allocation across the 41 $\times$ 42 mesh: (a) WMEM spatial heatmap, (b) FETCH spatial heatmap, (c) VLEN spatial heatmap.

Region	Avg WMEM (MB)	Avg DFLIT (bits)	Avg FETCH
Edge	10.21	6776.0	9.17
Inner	10.20	6776.0	9.14
Center	9.46	6776.0	8.31

Table 15: Region-level configuration summary from per-TCC JSON artifacts (1,722 active tiles).

Figure 11: Weight memory allocation analysis by mesh region: (a) violin plots showing the full WMEM distribution per region on a log scale, revealing the wide dynamic range of per-tile allocations, (b) regional weight concentration showing each region’s share of total WMEM, (c) Lorenz curve quantifying allocation heterogeneity via the Gini coefficient.

Panel of Figure 12 — Figure 12: Aggregate analysis of weight allocation and cross-node PPA tradeoffs. (a) WMEM distribution across tiles. (b) Normalized PPA comparison (3nm vs 28nm baseline). (c) Multi-dimensional tradeoff map across all evaluated nodes.

Parameter	Min	Max	Mean	Median	Std Dev	Unique
FETCH_SIZE	2	4	2.50	2.00	0.71	2
VLEN (bits)	1024	2048	1536.00	1024.00	512.00	2
WMEM (KB)	9564	72128	16910.00	9564.00	18000.00	varies
DMEM (KB)	64	1024	64.00	64.00	96.00	varies
IMEM (KB)	3	12	6.10	6.00	2.50	varies

Table 16: Per-TCC parameter summary statistics across all active tiles at the best process node. DFLIT_WIDTH (2048 bits) and STANUM (3) are uniform and omitted.

Node	Power (mW)	Perf (GOps/s)	Area (mm 2 )	PPA Score
28nm	3780	9744	3545	1.019
3nm	51366	466364	648	0.974
3nm vs 28nm	13.59x	47.86x	0.18x	0.96x

Table 17: Cross-node comparison from the Llama run

Node	GOps/s per mW	tok/s per mW	GOps/s per mm 2	PPA Score
3nm	9.076	0.5802	719.7	0.974
5nm	5.914	0.3782	364.1	0.989
7nm	3.764	0.2406	142.5	0.996
10nm	3.976	0.2541	63.6	1.005
14nm	3.606	0.2305	25.6	1.016
22nm	2.549	0.1629	6.3	1.023

Table 18: Derived node-efficiency metrics from generated run data.

Node	Mesh	Freq	Power	Area	Tok/s	PPA
		(MHz)	(mW)	(mm 2 )
3nm	2 $\times$ 4	10	6.4	17.6	10.3	0.254
5nm	3 $\times$ 4	10	12.7	26.2	14.1	0.309
7nm	3 $\times$ 4	10	12.4	35.0	14.1	0.312
10nm	3 $\times$ 3	10	8.6	46.7	10.0	0.291
14nm	2 $\times$ 4	10	6.3	61.7	10.3	0.271

Table 19: SmolVLM low-power mode results. All 7 nodes achieve

<

13 mW total power at 10 MHz clock. Power is leakage-dominated (89–97% at advanced nodes) with near-zero dynamic compute and NoC power. The RL selects compact 8–12 TCC meshes that minimize active silicon area.

Platform	Tok/s	Power	T/s/W	Notes
H200	230	700 W	0.3	4nm GPU
B200	353	1 kW	0.4	4nm GPU
Groq	594	300 W ∗	2.0	14nm ASIC
SambaNova	932	300 W ∗	3.1	Dataflow
Cerebras	1,981	15 kW ∗	0.1	7nm wafer
Taalas HC1	16,960	250 W ‡	67.8	6nm, 815 mm 2

Table 20: Industry comparison for Llama 3.1 8B (per-user, 1K input). Our results are compiler-estimated with on-chip ROM (no HBM).

Method	PPA	Tok/s	Power	Feasible
	Score		(W)	Configs
Random Search	1.12	8,421	38	312 / 4,600
Grid Search	1.05	14,230	42	890 / 4,600
SAC (ours)	0.974	29,809	51	2,847 / 4,600

Table 21: Search strategy comparison at 3nm for Llama 3.1 8B (lower PPA score = better). SAC achieves 3.5

\times

higher throughput than random search and 9.1

\times

more feasible configurations within the same episode budget.

AI Accelerator Inference

AI accelerator inference studies hardware and software for running neural networks faster and more efficiently across data-center, edge, and mobile devices.

See all 103 papers

ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs
Mar 8, 2026Yuzhuang Xu, Xu Han, Yuxuan Li +1Efficient Neural Network Inference LLM Inference Acceleration
Harbin Institute of Technology, Harbin, China · Tsinghua University, Beijing, China
ARES: Adaptive Reasoning-Effort Steering for PPA- and Cost-Aware RTL Optimization with LLM Agents
Jul 30, 2026Stef Cuyckens, Mihaela Jivanescu, Jun Yin +2Inference-Time Optimization Large Language Model-Guided Optimization
KU Leuven Leuven, Belgium · Nokia Bell Labs Antwerp, Belgium
Redwood: A Frontier AI Accelerator Designed, Verified, and Deployed from Scratch in 2 Weeks by AI
Aug 26, 2026Architect LabsAI Accelerator Inference Neural Network Hardware Acceleration
Architect Labs Team* Palo Alto, California

Mar 8, 2026J/K move · Enter open · S save

ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs

Yuzhuang Xu, Xu Han, Yuxuan Li, Wanxiang Che

Harbin Institute of Technology, Harbin, China · Tsinghua University, Beijing, China

Efficient Neural Network Inference LLM Inference Acceleration LLM Inference Scheduling

Open paper AskPDF arXiv

Although existing frameworks for large language model (LLM) inference on CPUs are mature, they fail to fully exploit the computation potential of many-core CPU platforms. Many-core CPUs are widely deployed in web servers and high-end networking devices, and are typically organized into multiple NUMA nodes that group cores and memory. Current frameworks largely overlook the substantial overhead of cross-NUMA memory access, limiting inference scalability and intelligence enabling on such platforms. To address this limitation, we build ArcLight, a lightweight LLM inference architecture designed from the ground up for many-core CPUs. ArcLight integrates efficient memory management and thread scheduling, and introduces finely controlled tensor parallelism to mitigate the cross-node memory access wall. Experimental results show that ArcLight significantly surpasses the performance ceiling of mainstream frameworks, achieving up to 46% higher inference throughput. Moreover, ArcLight maintains compatibility with arbitrary CPU devices. ArcLight is publicly available at https://github.com/OpenBMB/ArcLight.

Abstract

AI Accelerator Inference

Explore similar work

ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs

ARES: Adaptive Reasoning-Effort Steering for PPA- and Cost-Aware RTL Optimization with LLM Agents

Redwood: A Frontier AI Accelerator Designed, Verified, and Deployed from Scratch in 2 Weeks by AI