EdgeCIM: A Hardware-Software Co-Design for CIM-Based Acceleration of Small Language Models

Jinane Bazzi; Mariam Rakka; Fadi Kurdahi; Mohammed E. Fouda; Ahmed Eltawil

cs.ARApr 13, 2026

EdgeCIM: A Hardware-Software Co-Design for CIM-Based Acceleration of Small Language Models

Authors: Jinane Bazzi, Mariam Rakka, Fadi Kurdahi, Mohammed E. Fouda, Ahmed Eltawil

Organizations: University of California Irvine, Irvine, CA, USA. · Rain Neuromorphics Inc., San Francisco, CA, USA

Abstract

The growing demand for deploying Small Language Models (SLMs) on edge devices, including laptops, smartphones, and embedded platforms, has exposed fundamental inefficiencies in existing accelerators. While GPUs handle prefill workloads efficiently, the autoregressive decoding phase is dominated by GEMV operations that are inherently memory-bound, resulting in poor utilization and prohibitive energy costs at the edge. In this work, we present EdgeCIM, a hardware-software co-design framework that rethinks accelerator design for end-to-end decoder-only inference. At its core is a CIM macro, implemented in 65nm, coupled with a tile-based mapping strategy that balances pipeline stages, maximizing parallelism while alleviating DRAM bandwidth bottlenecks. Our simulator enables design space exploration of SLMs up to 4B parameters, identifying Pareto-optimal configurations in terms of latency and energy. Compared to an NVIDIA Orin Nano, EdgeCIM achieves up to 7.3x higher throughput and 49.59x better energy efficiency on LLaMA3.2-1B, and delivers 9.95x higher throughput than Qualcomm SA8255P on LLaMA3.2-3B. Extensive benchmarks on TinyLLaMA-1.1B, LLaMA3.2 (1B, 3B), Phi-3.5-mini-3.8B, Qwen2.5 (0.5B, 1.5B, 3B), SmolLM2-1.7B, SmolLM3-3B, and Qwen3 (0.6B, 1.7B, 4B) reveal that our accelerator, under INT4 precision, achieves on average 336.42 tokens/s and 173.02 tokens/J. These results establish EdgeCIM as a compelling solution towards real-time, energy-efficient edge-scale SLM inference.

Figures & tables

Fig. 1 : Inference Process in decoder-only SLMs.

Fig. 2 : Profiling LLaMA 3.2-1B for different number of input (

I

)/output (

O

) tokens on NVIDIA Jetson Orin GPU.

Fig. 4 : High-level architecture of the proposed design.

Fig. 5 : Typical decoding phase in a decoder-only SLM.

Fig. 6 : Partition-based mapping of weights onto EdgeCIM.

Panel of Figure 7 — Fig. 7 : Latency and energy trade-off across $\alpha$ values for LLaMA3.2-3B (INT8) decoding with 128 prefill and 128 generated tokens. Red markers = averages over five GA runs.

Fig. 8 : Decoding phase energy-latency product for LLaMA3.2-3B (INT8) using the optimal

h^{*}

\alpha=0.5

Panel of Figure 9 — Fig. 9 : (a) Throughput, (b) energy efficiency, and (c) area across SLMs for INT4 and INT8 precision at $\alpha=1$ .

Panel of Figure 10 — Fig. 10 : (a) Throughput and (b) energy efficiency of EdgeCIM and Jetson edge GPUs for selected SLMs (INT4).

Work	CIM Technology	Stage(s) Optimized For	End-to-End Optimization	Model Supported	Precision	TOPS/W/mm 2
EdgeCIM (this work)	CMOS	Projection+Attention+FFN	✓	Decoder-only	4/8 bits	7.03 $*$
X-Former [ 13 ]	CMOS + ReRAM	Projection+Attention	✗	Encoder-only	8 bits	NA $\dagger$
TranCIM [ 15 ]	CMOS	Attention+FFN	✗	Encoder-only	8+16 bits	3.06
ReTransformer [ 14 ]	ReRAM	Attention	✗	Encoder-decoder	8 bits	NA $\ddagger$
iMTransformer [ 44 ]	CMOS + FeFET	Attention	✗	Encoder-decoder	8 bits	1.64

TABLE I : Comparison of EdgeCIM with SOTA CIM accelerators.

Platform	Throughput (Tokens/s)
Jetson Orin Nano (Original)	27.7
Jetson Orin Nano Super	43.07
Jetson AGX Orin	80.4
Qualcomm SA8255P	14
Snapdragon X Elite	18.4
Snapdragon 8 Elite Mobile	23.5

TABLE II: Throughput of LLaMA3.2-3B (INT4) on edge.

On-Device Language Model Inference

On-device language model inference runs language models locally on phones, computers, and edge devices under tight memory, power, and latency limits.

See all 164 papers

EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding
Aug 5, 2026Sangwoo Ha, Hyunwoo Seo, Yurim Jo +2AI Accelerator Inference On-Device Language Model Inference
KAIST Daejeon, South Korea
LLMForge: Multi-Backend Hardware-Aware Neural Architecture Search with Infinite-Head Attention for Edge Language Models
May 17, 2026Xinting Jiang, Junyi Luo, Ruichen Qi +4On-Device Language Model Inference Hardware-Aware NAS
Brown University · University of Michigan · Google Research
Unlocking the Edge deployment and ondevice acceleration of multi-LoRA enabled one-for-all foundational LLM
Apr 20, 2026Sravanth Kodavanti, Sowmya Vajrala, Srinivas Miriyala +13On-Device Language Model Inference LLM Inference
Samsung Research Institute Bangalore, India · Samsung Electronics, Suwon, South Korea

Aug 5, 2026J/K move · Enter open · S save

EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding

Sangwoo Ha, Hyunwoo Seo, Yurim Jo, Youngjin Moon+1

KAIST Daejeon, South Korea

AI Accelerator Inference On-Device Language Model Inference Mixture-of-Experts Inference Memory-Efficient Inference Speculative Decoding

Open paper AskPDF arXiv

On-device deployment of Large Language Models (LLMs) has become essential for personalized edge applications. A primary bottleneck is external memory access (EMA) in feed-forward network (FFN) layers. Speculative decoding and mixture-of-experts (MoE) are promising solutions. Speculative decoding reduces the number of decoding stages by generating multiple tokens per stage, and MoE minimizes per-stage cost through sparse expert activation. However, there is an incompatibility when combining these two techniques. We propose EdgeXpert, a software-hardware co-designed LLM accelerator that resolves this incompatibility. In the prefill stage, the prompt-wise expert reuse reformulates routing as prompt-level expert reuse rather than independent per-token expert selection. It identifies important tokens using a lightweight encoder, constructs a shared expert set from them, and routes less important tokens with a reduced expert budget to lower expert EMA. In the decode stage, depth-aware expert coalescing exploits the contextual similarity and mutual exclusivity of same-depth candidate tokens. Rather than loading the union of all required channels, EdgeXpert loads only salient channels and applies computational calibration to recover accuracy without additional memory access. Synthesized in Samsung 28nm technology at 800 MHz, EdgeXpert achieves up to 56.3% latency reduction and 44.1% energy reduction compared to prior works, while maintaining near-baseline accuracy.

Abstract

On-Device Language Model Inference

Explore similar work

EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding

LLMForge: Multi-Backend Hardware-Aware Neural Architecture Search with Infinite-Head Attention for Edge Language Models

Unlocking the Edge deployment and ondevice acceleration of multi-LoRA enabled one-for-all foundational LLM