Compute-in-Memory Attention: A Time-Domain Analog Softmax Circuit with RC-Tunable Temperature

Ankur Singh; Ashish Gautam; Shruti R. Kulkarni; Guojing Cong

cs.ARSep 2, 2026

Compute-in-Memory Attention: A Time-Domain Analog Softmax Circuit with RC-Tunable Temperature

Authors: Ankur Singh, Ashish Gautam, Shruti R. Kulkarni, Guojing Cong

Organizations: Department of Electrical and Computer Engineering, University of Wisconsin–Madison, Madison, WI, USA. · ORNL Oak Ridge, Tennessee, USA.

Abstract

Softmax is a key operation in Transformer attention, but its exponentiation and normalization add significant overhead in compute-in-memory (CIM) accelerators, especially when analog attention scores must first be converted to the digital domain. This work presents a tunable-temperature analog softmax circuit in GlobalFoundries 22-nm fully depleted silicon-on-insulator (FDSOI) technology that operates directly on CIM-generated score voltages without intermediate analog-to-digital conversion. Each input score is converted into a time-domain event using a shared falling ramp. The corresponding comparator transition samples an RC-decaying reference to generate an exponential weight, which is then processed by an in-circuit normalization stage. In contrast to analog softmax circuits that rely on transistor weak-inversion behavior for exponentiation, the proposed architecture controls the softmax response through the ramp slope and RC time constant, enabling programmable effective temperature. The 128-element architecture is evaluated using transistor-level and post-layout extracted simulations, including multi-level input vectors, capacitance variation and mismatch, process and temperature variation, monte carlo analysis, and shared-interconnect parasitics. The complete 128-element implementation occupies 9453.42~ $μ\mathrm{m}^{2}$ including the shared global ramp circuitry, while each replicated softmax element occupies 70.2~ $μ\mathrm{m}^{2}$ . The circuit achieves a 242.97-ns evaluation latency at 13.44~~mW total power, corresponding to 25.5~~pJ per output element. The simultaneous 128-element evaluation achieves an RMSE of 24.46~mV relative to the ideal softmax response. The extracted circuit characteristics are further incorporated into a MemTorch-based hardware-aware Transformer model, where the proposed softmax achieves a validation loss within 2.5% of the ideal-softmax baseline.

Figures & tables

Fig. 1: Proposed time-domain analog softmax architecture: (a) CIM array generating the input score voltages $V_{\mathrm{IN},n}$ ; (b) shared ramp generator and comparator array for score-to-time conversion; (c) comparator-triggered pulse generation and RC-decay sampling to store exponential weights $V_{E,n}$ ; (d) approximate in-circuit normalizer generating $V_{P,n}$ ; and (e) timing sequence of the three operating phases.

Fig. 2: Schematic diagram of the static comparator.

Device	Width ( $W$ )	Length ( $L$ )
$M_{5}$ – $M_{6}$	4 $\mu$ m	50 nm
$M_{7}$ – $M_{9}$	300 nm	100 nm
$INV1,INV2$	GF22 standard-cell library

TABLE I: Transistor Dimensions of the Comparator Circuit

Parameter	Nominal Value
Extracted ramp slope, $S_{R}$	4.501 mV/ns
Programmed resistance, $R_{\mathrm{HRS}}$	1.5 M $\Omega$
Exponential capacitance, $C_{E}$	25 fF
RC time constant, $\tau_{E}$	37.5 ns
Effective softmax temperature, $T_{\mathrm{sat}}$	84.4 mV
Theoretical softmax gain, $\gamma_{\mathrm{th}}$	11.85 V -1

TABLE II: Nominal Circuit Parameters and Theoretical Softmax Gain

Fig. 3: Simulated softmax characteristics: (a) measured transfer, (b) nominal transfer.

Fig. 4: Transient operation for the eight representative input levels used in the 128-element vector evaluation, showing the shared $V_{\mathrm{RAMP}}$ and $V_{\mathrm{EXP}}$ waveforms, the corresponding write-enable signals $WE_{1}$ – $WE_{8}$ , and the sampled voltages $V_{E,1}$ – $V_{E,8}$ .

Fig. 5: Vector-level validation of the proposed softmax circuit: applied 128-element multi-level input-score vector.

Fig. 6: Sensitivity of the proposed softmax circuit of $C_{C,n}$ (a) 128-element output response for different capacitance values and (b) corresponding RMSE with respect to the theoretical softmax reference.

Fig. 7: Sensitivity of the proposed softmax circuit of $C_{P,n}$ (a) 128-element output response for different capacitance values and (b) corresponding RMSE with respect to the theoretical softmax reference.

Fig. 8: Effect of element-to-element capacitance mismatch on the 128-channel softmax response. Eight capacitance values are distributed across groups of 16 elements for separate $C_{C}$ and $C_{P}$ mismatch evaluations and compared with the nominal $2$ -fF case.

Fig. 9: Magnified transient response around the turn-off transition of $WE_{8}$ , showing $V_{\mathrm{EXP}}$ and the corresponding stored voltage $V_{E,8}$ for evaluating transmission-gate charge injection.

Fig. 10: Simulated softmax characteristics: (a) temperature, (b) process curve.

Fig. 11: Monte Carlo distribution of the softmax output voltage.

Fig. 13: Area and power breakdown of the proposed 128-element softmax architecture: (a) per-element layout area and (b) full-array average power.

Circuit Block	Length ( $\mu$ m)	Width ( $\mu$ m)	Area ( $\mu$ m 2 )
Ramp generator	12.1	4.2	50.82
Exponential generator	3.80	3.08	11.70
Comparator	4.2	5.12	21.50
Pulse generator	6.28	0.54	3.39
Normalization circuit	5.22	2.81	14.67
Routing	–	–	18.94

TABLE III: Layout Area Breakdown of the Proposed Softmax Circuit

Circuit Block	Count	Power ( $\mu$ W)	Total Power (mW)
Comparator	128	66.45	8.506
Pulse generator	128	37.32	4.777
Normalizer	128	1.061	0.1358
Exponential generator	128	0.159	0.0204
Global ramp generator	1	0.445	0.00045
Total	–	105.43	13.44

TABLE IV: Power Breakdown of the 128-Element Softmax Circuit

Fig. 14: Pre vs post-layout analysis of the proposed softmax.

Fig. 15: Post-layout extracted programmable softmax response under ramp-bias control (a) vector-level validation (b) responses for $V_{\mathrm{BIAS}}=350$ , 360, 370, and 380 mV are compared with theoretical softmax characteristics calculated using the corresponding extracted ramp slopes and $\gamma_{\mathrm{th}}=2/(S_{R}\tau_{E})$ .

$V_{\mathrm{BIAS}}$	$t_{\mathrm{fall}}$	$S_{R}$	$\gamma_{\mathrm{th}}$	RMSE
(mV)	(ns)	(mV/ns)	(V -1 )	(mV)
350	300	3.66	14.55	18.61
360	283	3.89	13.72	14.12
370	241	4.56	11.68	21.25
380	202	5.45	9.79	29.36

TABLE V: Post-Layout Extracted Softmax Programmability Under Ramp-Bias Control

Parameter	Value
Softmax input range	0.30–1.10 V
Nominal $\gamma_{\mathrm{th}}$	11.85 V -1
Score-to-voltage factor $G_{\mathrm{SV}}$	0.0844 V/score
$V_{\mathrm{DD}}$	1.1 V
Channel-length modulation $\lambda$	0
Output full scale $V_{\mathrm{FS}}$	1.0 V

TABLE VI: Parameters Used in the MemTorch Hardware-Aware Evaluation

Fig. 16: Hardware-aware Transformer evaluation. (a) Loss comparison of ideal and hardware softmax alternatives. (b) Temperature-dependent loss of the proposed softmax for different $\gamma_{\mathrm{th}}$ .

Metric	Vatalaro [ 17 ]	Jiang [ 18 ]	Kouretas [ 19 ]	Spagnolo [ 20 ]	Base-2 [ 21 ]	SoftAct [ 22 ]	Kim [ 23 ]	Wang [ 24 ]	Hyft [ 25 ]	This Work
Technology	180 nm	28 nm	90 nm	28 nm FDSOI	28 nm	28 nm	28 nm	28 nm	FPGA Zynq	22 nm FDSOI
Domain	Analog	Analog	Digital	Digital	Digital	Digital	Digital	Digital	Digital	Analog time-domain
Exponential mechanism	Subthresh.	Subthresh.	LUT	Shift-add approx.	SM-LUT	PLF + penalty	Bit-wise LUT	Linear fit	Hybrid fixed/float	Ramp + RC decay
Precision	Analog	Analog 4-b MAC 8-b ADC	10-b fixed	16-b fixed	16-b fixed	16-b fixed	8-b fixed	8-/16-b fixed	FP16/FP32	Analog
Parallelism	10	4	10	10	10	1	8	16/8	128	128
Computation mode	Parallel	Parallel	Parallel comb.	Parallel	Parallel	Serial	Serial	Serial	Serial	Parallel

TABLE VII: Comparison With State-of-the-Art Softmax Hardware Implementations

Explore similar work

Transformer Approximations from ReLUs
Apr 27, 2026Jerry Yao-Chieh Hu, Mingcheng Lu, Yi-Chen Lee +1Softmax Attention Neural Network Approximation Theory
Center for Foundation Models and Generative AI, Northwestern University, Evanston, IL 60208, USA · Department of Computer Science, Northwestern University, Evanston, IL 60208, USA · Department of Physics, National Taiwan University, Taipei 10617, Taiwan +1
Trilinear Compute-in-Memory Architecture for Energy-Efficient Transformer Acceleration
Apr 8, 2026Md Zesun Ahmed Mia, Jiahui Duan, Kai Ni +1Compute-in-Memory Efficient Transformer Inference
The Pennsylvania State University, Department of Electrical Engineering, University Park, PA, USA · University of Notre Dame, Department of Electrical Engineering, Notre Dame, IN, USA
Taming the Exponential: A Fast Softmax Surrogate for Integer-Native Edge Inference
Apr 2, 2026Dimitrios Danopoulos, Enrico Lupi, Michael Kagan +1Efficient Transformer Inference Efficient Inference
European Organization for Nuclear Research (CERN), Geneva, Switzerland · SLAC National Accelerator Laboratory, Menlo Park, CA, USA

Apr 27, 2026J/K move · Enter open · S save

Transformer Approximations from ReLUs

Jerry Yao-Chieh Hu, Mingcheng Lu, Yi-Chen Lee, Han Liu

Center for Foundation Models and Generative AI, Northwestern University, Evanston, IL 60208, USA · Department of Computer Science, Northwestern University, Evanston, IL 60208, USA · Department of Physics, National Taiwan University, Taipei 10617, Taiwan+1

Softmax Attention Neural Network Approximation Theory Attention Mechanisms

Open paper AskPDF arXiv

We provide a systematic recipe for translating ReLU approximation results to softmax attention mechanism. This recipe covers many common approximation targets. Importantly, it yields target-specific, economic resource bounds beyond universal approximation statements. We showcase the recipe on multiplication, reciprocal computation, and min/max primitives. These results provide new analytical tools for analyzing softmax transformer models.

Abstract

Explore similar work

Transformer Approximations from ReLUs

Trilinear Compute-in-Memory Architecture for Energy-Efficient Transformer Acceleration

Taming the Exponential: A Fast Softmax Surrogate for Integer-Native Edge Inference