Operator Fusion for LLM Inference on the Tensix Architecture

Qingbo Wu; Ke Li; Wenzhu Wang; Jie Yu; Ruian Zhang; Lili Liu

cs.LGJun 3, 2026

Operator Fusion for LLM Inference on the Tensix Architecture

Authors: Qingbo Wu, Ke Li, Wenzhu Wang, Jie Yu, Ruian Zhang, Lili Liu

Organizations: KylinSoft Co., Ltd, Tianjin, China · openKylin Community, Tianjin, China · College of Computer Science and Technology, National University of Defense Technology, Changsha, China

Abstract

This study addresses on-device inference bottlenecks of Transformer models on Tenstorrent's Tensix architecture and proposes an operator fusion strategy that enhances data locality. RMSNorm is fused with matrix multiplication in self-attention and in the FFN, enabling back-to-back execution of memory-bound and compute-bound operators in on-chip SRAM to significantly reduce DRAM reads/writes of intermediate results and scheduling overhead. To support multi-core parallelism, a NoC-based multicast mechanism is leveraged in which row/column master nodes efficiently distribute inputs and weights across the core mesh, alleviating DRAM bandwidth contention. Experiments on the Wormhole platform with Qwen2.5-0.5B, Qwen3-0.6B, and Qwen3-4B show up to 37.44% latency reduction for attention and 15.89% for MLP, with up to 7.91% reduction per decoder layer, while Pearson Correlation Coefficient (PCC) remains above 98.75%, confirming significant end-to-end efficiency gains under numerical consistency.

Explore similar work

LayerBoost: Layer-Aware Attention Reduction for Efficient LLMs
Apr 23, 2026Mohamed Ali Souibgui, Jan Fostier, Rodrigo Abadía-Heredia +3
Recent Developments in Transformer Inference Deployment on FPGA Platforms: A Survey
Sep 1, 2026Arjan Blankestijn, Uraz Odyurt, Amirreza YousefzadehField-Programmable Gate Arrays Accelerator
ELiTeFormer: An Efficient Transformer for FPGAs
Jul 4, 2026Victor Agostinelli, Nicolas Bohm Agostini, Antonino TumeoField-Programmable Gate Arrays LLM Inference Efficiency

Apr 23, 2026 · cs.LGJ/K move · Enter open · S save

LayerBoost: Layer-Aware Attention Reduction for Efficient LLMs

Mohamed Ali Souibgui, Jan Fostier, Rodrigo Abadía-Heredia, Bohdan Denysenko+2

Openchip & Software Technologies, S.L.

Transformers are mostly relying on softmax attention, which introduces quadratic complexity with respect to sequence length and remains a major bottleneck for efficient inference. Prior work on linear or hybrid attention typically replaces softmax attention uniformly across all layers, often leading to significant performance degradation or requiring extensive retraining to recover model quality. This work proposes LayerBoost, a layer-aware attention reduction method that selectively modifies the attention mechanism based on the sensitivity of individual transformer layers. It first performs a systematic sensitivity analysis on a pretrained model to identify layers that are critical for maintaining performance. Guided by this analysis, three distinct strategies can be applied: retaining standard softmax attention in highly sensitive layers, replacing it with linear sliding window attention in moderately sensitive layers, and removing attention entirely in layers that exhibit low sensitivity. To recover performance after these architectural modifications, we introduce a lightweight distillation-based healing phase requiring only 10M additional training tokens. LayerBoost reduces inference latency and improves throughput by up to 68% at high concurrency, while maintaining competitive model quality. It matches base model performance on several benchmarks, exhibits only minor degradations on others, and significantly outperforms state-of-the-art attention linearization methods. These efficiency gains make our method particularly well-suited for high-concurrency serving and hardware-constrained deployment scenarios, where inference cost and memory footprint are critical bottlenecks.

Open paper Ask about it PDF

Abstract

Explore similar work

LayerBoost: Layer-Aware Attention Reduction for Efficient LLMs

Recent Developments in Transformer Inference Deployment on FPGA Platforms: A Survey

ELiTeFormer: An Efficient Transformer for FPGAs