Non-linear functions are an important class of operations in modern large language models. However, chip vendors building for AI have so far relied on traditional vector or scalar processing units for non-linear functions. The performance of these units has not kept pace with matrix multiplication units (MMUs), which can be found in both high-performance and edge-computing devices from almost every vendor. Reconfigurable units such as FPGAs and CGRAs have been proposed in the literature, but have not been systematically compared with traditional units backed by MMUs. In this paper, we build two SoCs: one using an RVV vector processor and the other using a latency-insensitive (elastic) CGRA, both coupled with an MMU and driven by a single-core CPU. We then propose mappings for common feed-forward networks on our SoCs and find that the CGRA reaches a maximum speedup in clock cycles of 7.12x over RVV on the activation stage of simple point-wise non-linear functions like SiLU, whereas the RVV reaches a maximum speedup of 9.8x over the CGRA on complex non-linear functions such as softmax. Over the enclosing feed-forward block the corresponding maxima are 3.16x and 4.24x.
Figures & tables
Name
Equation
SwiGLU [ 66 , 21 , 78 ]
(SiLU(XWgate)⊙XWup)Wdown
Attention [ 42 , 21 , 73 ]
softmax(N(XWQ)(XWK)⊤)XWV
TABLE I: Combinations of linear and non-linear operators that we test.
Fig. 1: The “ME-VE” SoC. We combine a Saturn RISC-V vector processor with Gemmini using a Rocket CPU to simulate the combination of a matrix multiplication accelerator with a vector processor.
Fig. 2: The “ME-RE” SoC. We combine a CGRA-ME elastic RIKEN architecture with Gemmini using a Rocket CPU to simulate the combination of a matrix multiplication accelerator with a reconfigurable array.
funct7
Name
Purpose
0
CFG
Scan the configuration bitstream into the fabric — run once before any compute.
1
SET
Write one configuration register (scratchpad pointer, collector latency, or a DMA base address).
2
FILL
Start a non-blocking DMA read that fills one scratchpad half with operands.
3
PUSH2
Fill the scratchpad from core registers, two operand pairs per instruction.
4
RUN
Arm the streaming engine over one scratchpad half and return immediately.
5
WAIT
Ask whether the engine or the DMA has finished, either blocking or polling.
TABLE II: R-type custom2 instruction encoding used to drive the CGRA accelerator.
Fig. 3: Mapping of non-linear operators onto our 4x4 elastic RIKEN CGRA: (a) the SiLU activation from the SwiGLU, which executes on the CGRA while the matrix multiplication executes on Gemmini; and (b) softmax, mapped as two DFGs on the CGRA for the reduction and division steps.
Fig. 4: Clock cycles by SiLU on the ME-VE and ME-RE.
Fig. 5: Relative error of SwiGLU (left) and Attention (right) on the ME-VE and ME-RE SoC. The relative error is computed as ∥e−a∥2/∥a∥2 where e is the result vector from the SoC and a is the result vector from a PyTorch script on the host.
Fig. 6: Cycles / element for computing the SiLU for the ME-VE with and without the exponential compared to the ME-RE with the exponential. Costs are normalised by the N2 elements of the activation tensor, on the same measurements as Table III , so the ratio between the first and third bars reproduces that table’s activation speedup exactly. ME-VE is slower than ME-RE even without the exponential calculation, indicating that better pipelining leads to faster execution in the ME-RE.
SoC
N
Attention
SwiGLU
Total
Matmul
Activation
Softmax
Total
Matmul
Activation
Activation
cycles
cycles
cycles
proportion
cycles
cycles
cycles
proportion
ME-VE
16
6,943
3,628
3,299
47.52%
10,955
2,240
8,715
79.55%
32
27,610
16,582
11,022
39.92%
44,687
10,068
34,619
77.47%
64
143,087
103,508
39,456
27.57%
200,890
62,314
138,576
68.98%
128
862,834
713,800
149,015
17.27%
983,055
430,620
552,435
56.20%
TABLE III: Attention and SwiGLU FFN cycle breakdown. The total is read from an independent counter spanning the whole block rather than summed from the phase counters, so it does not tally exactly with them. The residual is under 0.02%.
Fig. 7: Cycles taken for softmax for ME-VE vs. ME-RE.
Fig. 8: Cycles / element for the activation computation of the Attention calculation on the ME-VE compared to the ME-RE. Our mapping gives the ME-VE an advantage over the ME-RE.
State
Description
Cycles / element (N=128)
RESET
Per chunk fabric reset.
3.94
ARM
Re-arm row constants (m, r) and stream pointers cleared by the reset.
0.56
RDACC
Read and clear the on-fabric accumulator.
0.14
RUN
Execute one iteration of a chunk of 24 elements.
40.62
FILL
Fetch a chunk over DMA and place it in the scratch pad.
2.80
DRAIN
Barrier waiting for RUN to finish.
<0.01
TABLE IV: Breakdown of the number of cycles an element spends in the CGRA and non-core parts of the SoC.
Step
Factor
Cumulative
Ideal (II =1 )
—
1.00
Two passes
×2
2.00
Operand duplication
×2
4.00
Operand prologue
×1.375
5.50
Streaming rate
×5.005
27.53
Restart fill
+13.02
40.55
TABLE V: Per-element cost of the softmax fabric at N=128 in cycles/element. The final column is the running total.
As neural networks continue to scale, low-precision quantization is widely adopted in edge accelerators, but conventional multi-threshold activation hardware requires 2^n-1 thresholds for n-bit integer outputs, leading to rapidly increasing hardware costs. We propose GRAU, a Generic Runtime-Reconfigurable Activation-Requantization Unit that fuses batch normalization, nonlinear activation, and requantization through piecewise-linear fitting with Power-of-Two slopes. GRAU requires only basic comparators and 1-bit right shifters, while supporting mixed-precision quantization and runtime switching among post-accumulation functions, including modern nonlinear activations such as SiLU and GELU. Compared with multi-threshold activation-requantization units, GRAU reduces LUT consumption by approximately 90% or more while improving hardware efficiency, flexibility, and scalability. We further explore the trade-off between accuracy and hardware cost across different approximation settings and provide an open-source toolchain for GRAU-aware network training.
Yuhao Liu, Salim Ullah, Akash Kumar
Ruhr University Bochum, Germany · Dresden University of Technology, Germany · Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI Dresden/Leipzig), Germany
The rapid growth of Large Language Models (LLMs) has intensified the need for specialized hardware accelerators that can satisfy stringent inference latency and power constraints. Although matrix multiplications dominate the overall computational workload, non-linear vector normalization operations, such as LayerNorm, RMSNorm and Softmax can become critical hardware bottlenecks. Existing accelerators typically implement these functions using dedicated hardware blocks, leading to duplicated resources and inefficient silicon utilization. To address this limitation, we propose a Minimalist Integer Vector Engine (MIVE), a programmable architecture capable of executing all three operations within a unified datapath. By exploiting common computational patterns across LayerNorm, RMSNorm and Softmax the proposed vector engine maximizes hardware sharing while reducing implementation overhead. Physical ASIC implementation results show that MIVE provides comprehensive multi-function support while achieving higher area and hardware efficiency than most state-of-the-art standalone accelerators.
Kosmas Alexandridis, Giorgos Dimitrakopoulos
Integrated Circuits Lab, Electrical and Computer Engineering, Democritus University of Thrace (DUTH), Greece
Function vectors (FVs) are task representations elicited during in-context learning that can be used to steer Large Language Models (LLMs). However, design choices in their formulation remain underexplored. In this work, we study the impact of varying FV definitions for instructions along two degrees of freedom: attention head selection and steering. For head selection, using gradient-based attributions with Layer-wise Relevance Propagation (LRP) substantially improves efficiency as well as accuracy. For FV steering, applying it in a distributed manner yields a higher accuracy compared to simple aggregation. Our code is publicly available.
Minh An Pham, Anton Segeler, Thomas Wiegand +4
Fraunhofer Heinrich-Hertz-Institute, Berlin, Germany · Technische Universität Berlin, Berlin, Germany · BIFOLD – Berlin Institute for the Foundations of Learning and Data, Berlin, Germany +1