Trilinear Compute-in-Memory Architecture for Energy-Efficient Transformer Acceleration

Md Zesun Ahmed Mia; Jiahui Duan; Kai Ni; Abhronil Sengupta

cs.ARApr 8, 2026

Trilinear Compute-in-Memory Architecture for Energy-Efficient Transformer Acceleration

Authors: Md Zesun Ahmed Mia, Jiahui Duan, Kai Ni, Abhronil Sengupta

Organizations: The Pennsylvania State University, Department of Electrical Engineering, University Park, PA, USA · University of Notre Dame, Department of Electrical Engineering, Notre Dame, IN, USA

Abstract

Self-attention in Transformers generates dynamic operands that force conventional Compute-in-Memory (CIM) accelerators into costly non-volatile memory (NVM) reprogramming cycles, degrading throughput and stressing device endurance. Existing solutions either reduce but retain NVM writes through matrix decomposition or sparsity, or move attention computation to digital CMOS at the expense of NVM density. We present TrilinearCIM, a Double-Gate FeFET (DG-FeFET)-based architecture that uses back-gate modulation to realize a three-operand multiply-accumulate primitive for in-memory attention computation without dynamic ferroelectric reprogramming. Evaluated on BERT-base (GLUE) and ViT-base (ImageNet and CIFAR), TrilinearCIM outperforms conventional CIM on seven of nine GLUE tasks while achieving up to 46.6% energy reduction and 20.4% latency improvement over conventional FeFET CIM at 37.3% area overhead. To our knowledge, this is the first architecture to perform complete Transformer attention computation exclusively in NVM cores without runtime reprogramming.

Figures & tables

Figure 1. Transformer Encoder Block architecture. The input sequence

X

is projected through learned weight matrices

W_{Q}

W_{K}

W_{V}

to produce Query (

Q

), Key (

K

), and Value (

V

) representations. These are processed by Scaled Dot-Product Attention (Eq. 4 ), followed by concatenation and linear projection to form the Multi-Head Self-Attention output. The Feed-Forward Network applies two linear transformations with GELU activation. Residual connections (circled “+”) add the sub-layer input to its output before Layer Normalization (LayerNorm).

Figure 2. DG-FeFET device concept. (a) Capacitor stack model illustrating the series combination of ferroelectric (

C_{FE}

) and interlayer (

C_{IL}

) capacitances forming the effective top-gate oxide capacitance

C_{TGOX}

. (b) Device cross-section showing the Top-Gate (TG), Back-Gate (BG), and localized n + source/drain implant regions within the channel.

Metric	Read	Write
Latency	$\sim$ 10 ns	$\sim$ 50 ns
Energy/cell	$\sim$ fJ	$\sim$ sub-pJ

Table 1. FeFET read vs. write asymmetry ( Jerry et al., 2017 ; Peng et al., 2019 ) .

Figure 3. Hierarchical organization of the trilinear CIM accelerator. (Top-left) Chip level: 2

\times

2 tile grid with global buffer, accumulation unit, and Special Function Unit (SFU) for Softmax, LayerNorm, and Activation. (Top-right) Tile level: 2

\times

2 Processing Element (PE) grid with tile input buffer and accumulation/output buffer. (Bottom-left) PE level: 2

\times

2 array grid with tile input buffer. (Bottom-right) SubArray level: DG-FeFET cells driven by wordline/control-line (WL/CL) switch matrix (rows) and back-gate-line/source-line (BGL/SL) switch matrix/DAC (columns), with multiplexer (MUX)

\rightarrow

ADC

\rightarrow

Adder

\rightarrow

Shift Register readout pipeline.

Figure 4.

\eta_{BG}

vs.

G_{0}

for the DG-FeFET, following Eq. 12 with parameters

\alpha

and

M

extracted from experimental data ( Jiang et al., 2025 ) (see Section 2.2 ). The green-shaded region indicates the selected operating band (

G_{0}\in[29,69]

\mu

S); the dashed horizontal line marks the band-averaged

\bar{\eta}_{BG}=0.157

V -1 adopted as a uniform constant.

Figure 5. Attention dataflow comparison. (a) Conventional:

Q/K/V

projections require off-chip DRAM transfers before score computation. (b) Trilinear: weights remain static while

X

modulates device conductance via back-gate, eliminating DRAM traffic and saving compute time.

Stage	Math	Row In.	Stored	BG In.
Scaled $Q$	$X\cdot W_{Q}^{T}/\sqrt{d_{k}}$	$X$	$W_{Q}^{T}$	$1/\sqrt{d_{k}}$
Score	$R_{1}\cdot W_{K}\cdot X^{T}$	$R_{1}$	$W_{K}$	$X^{T}$
$V$ -Agg	$\text{Score}\cdot X\cdot W_{V}^{T}$	$X$	$W_{V}^{T}$	Score

Table 2. Trilinear attention stages and operand mapping.

Panel of Figure 6 — Figure 6. DG-FeFET crossbar array for trilinear MAC. (a) $O=A\cdot B^{T}\cdot C$ : weights $B^{T}$ stored in device conductance, input $A$ applied to rows, modulator $C$ applied to columns. (b) $O=A\cdot B\cdot C^{T}$ : weights $C^{T}$ stored in device conductance, input $B$ applied to rows, modulator $A$ broadcast across all columns.

Parameter	Value
Technology node	7nm CMOS / 22nm FeFET (BEOL)
SubArray size	64 $\times$ 64 (scalable)
Input precision	8-bit
Weight precision	8-bit (2-bit/cell)
ADC precision	8-bit
Column muxing	8:1 (ADC sharing)

Table 3. Default system configuration.

Task (Metric)	Seq	Digital	Bilinear	Trilinear
CoLA (MCC)	64	45.46 $\pm$ 1.46	42.17 $\pm$ 8.49	43.70 $\pm$ 1.03
SST-2 (Acc)	64	91.59 $\pm$ 0.40	89.72 $\pm$ 1.07	91.32 $\pm$ 0.43
MRPC (F1)	128	88.04 $\pm$ 0.87	84.24 $\pm$ 3.62	85.54 $\pm$ 0.62
RTE (Acc)	128	70.40 $\pm$ 0.72	68.11 $\pm$ 2.18	66.78 $\pm$ 1.53
STS-B (Pears.)	128	85.79 $\pm$ 0.23	82.02 $\pm$ 2.63	83.76 $\pm$ 0.77
WNLI (Acc)	128	56.34 $\pm$ 0.00	56.34 $\pm$ 0.00	56.34 $\pm$ 0.00

Table 4. GLUE benchmark scores. Bold indicates the best CIM mode per task. Seq: sequence length in tokens. Metrics—MCC: Matthews Correlation Coefficient; Acc: Accuracy; F1: F1-score; Pears.: Pearson correlation.

ViT processes 197 tokens per image.
Dataset	Digital	Bilinear	Trilinear
CIFAR-10	97.01 $\pm$ 0.28	96.74 $\pm$ 0.65	95.53 $\pm$ 0.09
CIFAR-100	86.05 $\pm$ 0.28	83.90 $\pm$ 0.84	81.44 $\pm$ 1.40
ImageNet-1K	79.39 $\pm$ 1.04	79.11 $\pm$ 1.06	74.98 $\pm$ 2.05

Table 5. Vision benchmark accuracy (%) for ViT-base (default configuration per Table 3 ).

	Seq 64			Seq 128
Metric	Bil.	Tri.	$\Delta$ %	Bil.	Tri.	$\Delta$ %
Area (mm 2 )	326	447	+37.3	651	894	+37.3
Latency (ms)	7.63	6.08	$-$ 20.4	8.19	6.67	$-$ 18.6
Energy ( $\mu$ J)	1,522	813	$-$ 46.6	3,132	1,889	$-$ 39.7
Throughput (inf/s)	131	164	+25.5	122	150	+22.7
TOPS/W	9.97	12.24	+22.8	9.68	13.47	+39.2

Table 6. Per-inference PPA comparison by sequence length (BERT-base, default configuration per Table 3 ).

Figure 7. Sub-array size ablation (2b/8b, seq = 128, per inference). Energy, latency, and chip area for 32 2 and 64 2 sub-arrays. Green annotations denote trilinear improvement over bilinear; red annotations denote area overhead. The 64 2 default (shaded) is used throughout the paper.

Config	Area	Lat.	Energy	TOPS/W
	$\Delta$ %	$\Delta$ %	$\Delta$ %	Bil.	Tri.
1b/6b	+32.4	$-$ 26.0	$-$ 37.5	10.45	13.77
1b/7b	+32.4	$-$ 35.4	$-$ 32.5	7.52	10.22
2b/8b	+37.3	$-$ 18.6	$-$ 39.7	9.68	13.47
2b/9b	+37.4	$-$ 27.5	$-$ 31.5	5.31	7.61

Table 7. Bitcell/ADC precision ablation (SA=64

\times

64, seq 128). PPA deltas are trilinear vs. bilinear. Per-task GLUE score (task-specific metric; see Table 4 ) is shown in Fig. 8 .

Figure 8. Per-task GLUE benchmark scores for four bitcell/ADC configurations (SA=64

\times

64, seq = 128). Grey: bilinear; blue: trilinear. Error bars show

\pm

1 std over 2–3 seeds. CoLA (Matthews Corr.) omitted due to its different metric scale.

Work	Technology	Attention Strategy	NVM Writes	Limitation
ISAAC/PRIME ( Shafiee et al., 2016 ; Chi et al., 2016 )	ReRAM	Crossbar architectures for static-weight CNNs/MLPs; no attention dataflow	—	Cannot handle dynamic operands in $QK^{T}$ or Score $\cdot V$
PipeLayer ( Song et al., 2017 )	ReRAM	Weight-gradient pipelined training on ReRAM crossbars; no inference attention path	—	Training-only; no mechanism for dynamic attention during inference
ReTransformer ( Yang et al., 2020 )	ReRAM	Decomposes $QK^{T}{=}(QW_{K})X^{T}$ to keep static $W_{K}$ on crossbar, avoiding $K^{T}$ write	Reduced	Score $\cdot V$ still writes $V$ to ReRAM; write endurance consumed
Sanger ( Lu et al., 2021 )	Digital ASIC	Reconfigurable score-stationary architecture with structured attention pruning	—	Digital-only; no NVM integration for weight storage
X-Former ( Sridharan et al., 2023 )	ReRAM + CMOS	Dual-engine: ReRAM for static projections, CMOS attention engine for dynamic attention	Zero	Forfeits NVM density for the attention engine; additional CMOS area and leakage overhead
iMTransformer ( Laguna et al., 2022 )	FeFET + CMOS	FeFET crossbars for static projections; CMOS crossbars/CAMs cache attention scores	Zero	Heterogeneous FeFET+CMOS fabric; attention engine sacrifices NVM density

Table 8. Comparison of attention-computation strategies across CIM, PIM, and digital transformer accelerators.

Explore similar work

Compute-in-Memory Attention: A Time-Domain Analog Softmax Circuit with RC-Tunable Temperature
Sep 2, 2026Ankur Singh, Ashish Gautam, Shruti R. Kulkarni +1Softmax Attention Compute-in-Memory
Department of Electrical and Computer Engineering, University of Wisconsin–Madison, Madison, WI, USA. · ORNL Oak Ridge, Tennessee, USA.
RED-PIM: Reducing Data Movement for Transformers using Processing-in-Memory
Jul 23, 2026Zahra Yousefijamarani, Alaa AlameldeenEfficient Transformer Inference AI Accelerator Inference
Simon Fraser University
NIFA: Nonlinear IMC enhanced FPGA for efficient ML inference
Jul 16, 2026Jiajun Hu, Ruthwik Reddy Sunketa, Lei Zhao +3Efficient Transformer Inference Compute-in-Memory
Arizona State University, Tempe, AZ, USA · Hewlett Packard Enterprise Labs, Fort Collins, CO, USA

Sep 2, 2026J/K move · Enter open · S save

Compute-in-Memory Attention: A Time-Domain Analog Softmax Circuit with RC-Tunable Temperature

Ankur Singh, Ashish Gautam, Shruti R. Kulkarni, Guojing Cong

Department of Electrical and Computer Engineering, University of Wisconsin–Madison, Madison, WI, USA. · ORNL Oak Ridge, Tennessee, USA.

Softmax Attention Compute-in-Memory Transformer Attention

Open paper AskPDF arXiv

Softmax is a key operation in Transformer attention, but its exponentiation and normalization add significant overhead in compute-in-memory (CIM) accelerators, especially when analog attention scores must first be converted to the digital domain. This work presents a tunable-temperature analog softmax circuit in GlobalFoundries 22-nm fully depleted silicon-on-insulator (FDSOI) technology that operates directly on CIM-generated score voltages without intermediate analog-to-digital conversion. Each input score is converted into a time-domain event using a shared falling ramp. The corresponding comparator transition samples an RC-decaying reference to generate an exponential weight, which is then processed by an in-circuit normalization stage. In contrast to analog softmax circuits that rely on transistor weak-inversion behavior for exponentiation, the proposed architecture controls the softmax response through the ramp slope and RC time constant, enabling programmable effective temperature. The 128-element architecture is evaluated using transistor-level and post-layout extracted simulations, including multi-level input vectors, capacitance variation and mismatch, process and temperature variation, monte carlo analysis, and shared-interconnect parasitics. The complete 128-element implementation occupies 9453.42~

μ\mathrm{m}^{2}

including the shared global ramp circuitry, while each replicated softmax element occupies 70.2~

μ\mathrm{m}^{2}

. The circuit achieves a 242.97-ns evaluation latency at 13.44~~mW total power, corresponding to 25.5~~pJ per output element. The simultaneous 128-element evaluation achieves an RMSE of 24.46~mV relative to the ideal softmax response. The extracted circuit characteristics are further incorporated into a MemTorch-based hardware-aware Transformer model, where the proposed softmax achieves a validation loss within 2.5% of the ideal-softmax baseline.

Abstract

Explore similar work

Compute-in-Memory Attention: A Time-Domain Analog Softmax Circuit with RC-Tunable Temperature

RED-PIM: Reducing Data Movement for Transformers using Processing-in-Memory

NIFA: Nonlinear IMC enhanced FPGA for efficient ML inference