Compute-in-Memory Attention: A Time-Domain Analog Softmax Circuit with RC-Tunable Temperature

Ankur Singh; Ashish Gautam; Shruti R. Kulkarni; Guojing Cong

cs.ARSep 2, 2026

Compute-in-Memory Attention: A Time-Domain Analog Softmax Circuit with RC-Tunable Temperature

Authors: Ankur Singh, Ashish Gautam, Shruti R. Kulkarni, Guojing Cong

Organizations: Department of Electrical and Computer Engineering, University of Wisconsin–Madison, Madison, WI, USA. · ORNL Oak Ridge, Tennessee, USA.

Abstract

Softmax is a key operation in Transformer attention, but its exponentiation and normalization add significant overhead in compute-in-memory (CIM) accelerators, especially when analog attention scores must first be converted to the digital domain. This work presents a tunable-temperature analog softmax circuit in GlobalFoundries 22-nm fully depleted silicon-on-insulator (FDSOI) technology that operates directly on CIM-generated score voltages without intermediate analog-to-digital conversion. Each input score is converted into a time-domain event using a shared falling ramp. The corresponding comparator transition samples an RC-decaying reference to generate an exponential weight, which is then processed by an in-circuit normalization stage. In contrast to analog softmax circuits that rely on transistor weak-inversion behavior for exponentiation, the proposed architecture controls the softmax response through the ramp slope and RC time constant, enabling programmable effective temperature. The 128-element architecture is evaluated using transistor-level and post-layout extracted simulations, including multi-level input vectors, capacitance variation and mismatch, process and temperature variation, monte carlo analysis, and shared-interconnect parasitics. The complete 128-element implementation occupies 9453.42~ $μ\mathrm{m}^{2}$ including the shared global ramp circuitry, while each replicated softmax element occupies 70.2~ $μ\mathrm{m}^{2}$ . The circuit achieves a 242.97-ns evaluation latency at 13.44~~mW total power, corresponding to 25.5~~pJ per output element. The simultaneous 128-element evaluation achieves an RMSE of 24.46~mV relative to the ideal softmax response. The extracted circuit characteristics are further incorporated into a MemTorch-based hardware-aware Transformer model, where the proposed softmax achieves a validation loss within 2.5% of the ideal-softmax baseline.

Figures & tables

Fig. 1: Proposed time-domain analog softmax architecture: (a) CIM array generating the input score voltages $V_{\mathrm{IN},n}$ ; (b) shared ramp generator and comparator array for score-to-time conversion; (c) comparator-triggered pulse generation and RC-decay sampling to store exponential weights $V_{E,n}$ ; (d) approximate in-circuit normalizer generating $V_{P,n}$ ; and (e) timing sequence of the three operating phases.

Fig. 2: Schematic diagram of the static comparator.

Device	Width ( $W$ )	Length ( $L$ )
$M_{5}$ – $M_{6}$	4 $\mu$ m	50 nm
$M_{7}$ – $M_{9}$	300 nm	100 nm
$INV1,INV2$	GF22 standard-cell library

TABLE I: Transistor Dimensions of the Comparator Circuit

Parameter	Nominal Value
Extracted ramp slope, $S_{R}$	4.501 mV/ns
Programmed resistance, $R_{\mathrm{HRS}}$	1.5 M $\Omega$
Exponential capacitance, $C_{E}$	25 fF
RC time constant, $\tau_{E}$	37.5 ns
Effective softmax temperature, $T_{\mathrm{sat}}$	84.4 mV
Theoretical softmax gain, $\gamma_{\mathrm{th}}$	11.85 V -1

TABLE II: Nominal Circuit Parameters and Theoretical Softmax Gain

Fig. 3: Simulated softmax characteristics: (a) measured transfer, (b) nominal transfer.

Fig. 4: Transient operation for the eight representative input levels used in the 128-element vector evaluation, showing the shared $V_{\mathrm{RAMP}}$ and $V_{\mathrm{EXP}}$ waveforms, the corresponding write-enable signals $WE_{1}$ – $WE_{8}$ , and the sampled voltages $V_{E,1}$ – $V_{E,8}$ .

Fig. 5: Vector-level validation of the proposed softmax circuit: applied 128-element multi-level input-score vector.

Fig. 6: Sensitivity of the proposed softmax circuit of $C_{C,n}$ (a) 128-element output response for different capacitance values and (b) corresponding RMSE with respect to the theoretical softmax reference.

Fig. 7: Sensitivity of the proposed softmax circuit of $C_{P,n}$ (a) 128-element output response for different capacitance values and (b) corresponding RMSE with respect to the theoretical softmax reference.

Fig. 8: Effect of element-to-element capacitance mismatch on the 128-channel softmax response. Eight capacitance values are distributed across groups of 16 elements for separate $C_{C}$ and $C_{P}$ mismatch evaluations and compared with the nominal $2$ -fF case.

Fig. 9: Magnified transient response around the turn-off transition of $WE_{8}$ , showing $V_{\mathrm{EXP}}$ and the corresponding stored voltage $V_{E,8}$ for evaluating transmission-gate charge injection.

Fig. 10: Simulated softmax characteristics: (a) temperature, (b) process curve.

Fig. 11: Monte Carlo distribution of the softmax output voltage.

Fig. 13: Area and power breakdown of the proposed 128-element softmax architecture: (a) per-element layout area and (b) full-array average power.

Circuit Block	Length ( $\mu$ m)	Width ( $\mu$ m)	Area ( $\mu$ m 2 )
Ramp generator	12.1	4.2	50.82
Exponential generator	3.80	3.08	11.70
Comparator	4.2	5.12	21.50
Pulse generator	6.28	0.54	3.39
Normalization circuit	5.22	2.81	14.67
Routing	–	–	18.94

TABLE III: Layout Area Breakdown of the Proposed Softmax Circuit

Circuit Block	Count	Power ( $\mu$ W)	Total Power (mW)
Comparator	128	66.45	8.506
Pulse generator	128	37.32	4.777
Normalizer	128	1.061	0.1358
Exponential generator	128	0.159	0.0204
Global ramp generator	1	0.445	0.00045
Total	–	105.43	13.44

TABLE IV: Power Breakdown of the 128-Element Softmax Circuit

Fig. 14: Pre vs post-layout analysis of the proposed softmax.

Fig. 15: Post-layout extracted programmable softmax response under ramp-bias control (a) vector-level validation (b) responses for $V_{\mathrm{BIAS}}=350$ , 360, 370, and 380 mV are compared with theoretical softmax characteristics calculated using the corresponding extracted ramp slopes and $\gamma_{\mathrm{th}}=2/(S_{R}\tau_{E})$ .

$V_{\mathrm{BIAS}}$	$t_{\mathrm{fall}}$	$S_{R}$	$\gamma_{\mathrm{th}}$	RMSE
(mV)	(ns)	(mV/ns)	(V -1 )	(mV)
350	300	3.66	14.55	18.61
360	283	3.89	13.72	14.12
370	241	4.56	11.68	21.25
380	202	5.45	9.79	29.36

TABLE V: Post-Layout Extracted Softmax Programmability Under Ramp-Bias Control

Parameter	Value
Softmax input range	0.30–1.10 V
Nominal $\gamma_{\mathrm{th}}$	11.85 V -1
Score-to-voltage factor $G_{\mathrm{SV}}$	0.0844 V/score
$V_{\mathrm{DD}}$	1.1 V
Channel-length modulation $\lambda$	0
Output full scale $V_{\mathrm{FS}}$	1.0 V

TABLE VI: Parameters Used in the MemTorch Hardware-Aware Evaluation

Fig. 16: Hardware-aware Transformer evaluation. (a) Loss comparison of ideal and hardware softmax alternatives. (b) Temperature-dependent loss of the proposed softmax for different $\gamma_{\mathrm{th}}$ .

Metric	Vatalaro [ 17 ]	Jiang [ 18 ]	Kouretas [ 19 ]	Spagnolo [ 20 ]	Base-2 [ 21 ]	SoftAct [ 22 ]	Kim [ 23 ]	Wang [ 24 ]	Hyft [ 25 ]	This Work
Technology	180 nm	28 nm	90 nm	28 nm FDSOI	28 nm	28 nm	28 nm	28 nm	FPGA Zynq	22 nm FDSOI
Domain	Analog	Analog	Digital	Digital	Digital	Digital	Digital	Digital	Digital	Analog time-domain
Exponential mechanism	Subthresh.	Subthresh.	LUT	Shift-add approx.	SM-LUT	PLF + penalty	Bit-wise LUT	Linear fit	Hybrid fixed/float	Ramp + RC decay
Precision	Analog	Analog 4-b MAC 8-b ADC	10-b fixed	16-b fixed	16-b fixed	16-b fixed	8-b fixed	8-/16-b fixed	FP16/FP32	Analog
Parallelism	10	4	10	10	10	1	8	16/8	128	128
Computation mode	Parallel	Parallel	Parallel comb.	Parallel	Parallel	Serial	Serial	Serial	Serial	Parallel

TABLE VII: Comparison With State-of-the-Art Softmax Hardware Implementations

Explore similar work

Apr 27, 2026

Transformer Approximations from ReLUs

We provide a systematic recipe for translating ReLU approximation results to softmax attention mechanism. This recipe covers many common approximation targets. Importantly, it yields target-specific, economic resource bounds beyond universal approximation statements. We showcase the recipe on multiplication, reciprocal computation, and min/max primitives. These results provide new analytical tools for analyzing softmax transformer models.

Jerry Yao-Chieh Hu, Mingcheng Lu, Yi-Chen Lee +1

Center for Foundation Models and Generative AI, Northwestern University, Evanston, IL 60208, USA · Department of Computer Science, Northwestern University, Evanston, IL 60208, USA · Department of Physics, National Taiwan University, Taipei 10617, Taiwan +1

Softmax Attention Neural Network Approximation Theory Attention Mechanisms

Ask PDF

Apr 8, 2026

Trilinear Compute-in-Memory Architecture for Energy-Efficient Transformer Acceleration

Self-attention in Transformers generates dynamic operands that force conventional Compute-in-Memory (CIM) accelerators into costly non-volatile memory (NVM) reprogramming cycles, degrading throughput and stressing device endurance. Existing solutions either reduce but retain NVM writes through matrix decomposition or sparsity, or move attention computation to digital CMOS at the expense of NVM density. We present TrilinearCIM, a Double-Gate FeFET (DG-FeFET)-based architecture that uses back-gate modulation to realize a three-operand multiply-accumulate primitive for in-memory attention computation without dynamic ferroelectric reprogramming. Evaluated on BERT-base (GLUE) and ViT-base (ImageNet and CIFAR), TrilinearCIM outperforms conventional CIM on seven of nine GLUE tasks while achieving up to 46.6% energy reduction and 20.4% latency improvement over conventional FeFET CIM at 37.3% area overhead. To our knowledge, this is the first architecture to perform complete Transformer attention computation exclusively in NVM cores without runtime reprogramming.

Md Zesun Ahmed Mia, Jiahui Duan, Kai Ni +1

The Pennsylvania State University, Department of Electrical Engineering, University Park, PA, USA · University of Notre Dame, Department of Electrical Engineering, Notre Dame, IN, USA

Compute-in-Memory Efficient Transformer Inference Energy-Efficient ML Transformer Attention

Ask PDF

Apr 2, 2026

Taming the Exponential: A Fast Softmax Surrogate for Integer-Native Edge Inference

Softmax can become a computational bottleneck in the Transformer model's Multi-Head Attention (MHA) block, particularly in small models under low-precision inference, where exponentiation and normalization incur significant overhead. As such, we suggest using Head-Calibrated Clipped-Linear Softmax (HCCS), a bounded, monotone surrogate to the exponential softmax function, which uses a clipped linear mapping of the max centered attention logits. This approximation produces a stable probability distribution, maintains the ordering of the original logits and has non-negative values. HCCS differs from previous softmax surrogates as it includes a set of lightweight calibration parameters that are optimized offline based on a representative dataset and calibrated for each individual attention head to preserve the statistical properties of the individual heads. We describe a hardware-motivated implementation of HCCS for high-throughput scenarios targeting the AMD Versal AI Engines. The current reference implementations from AMD for this platform rely upon either bfloat16 arithmetic or LUTs to perform the exponential operation, which might limit the throughput of the platform and fail to utilize the high-throughput integer vector processing units of the AI Engine. In contrast, HCCS provides a natural mapping to the AI Engines' int8 multiply accumulate (MAC) units. To the best of our knowledge, this is the first int8 optimized softmax surrogate for AMD AI engines that significantly exceeds the speed performance of other reference implementations while maintaining competitive task accuracy on small or heavily quantized MHA workloads after quantization-aware retraining.

Dimitrios Danopoulos, Enrico Lupi, Michael Kagan +1

European Organization for Nuclear Research (CERN), Geneva, Switzerland · SLAC National Accelerator Laboratory, Menlo Park, CA, USA

Efficient Transformer Inference Efficient Inference Softmax Attention AI Accelerator Inference

Ask PDF