cs.LGNov 17, 2026

Implementing Binary Transformers with Approximate Memories with Permutation Invariance Enabled Selective (PIES) Protection

Authors: Zhen Gao, Xiao Yin, Kexin Liang, Pedro Reviriego, Shanshan Liu, Fabrizio Lombardi

Organizations: School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China · Information Processing and Telecommunications Center, ETSI de Telecomunicación, Universidad Politécnica de Madrid, Madrid 28040 · School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu, Sichuan 611731, China · Department of Electrical and Computer Engineering, Northeastern University, Boston, MA 02215, USA

Abstract

Quantization is an effective technique to reduce memory storage, computation complexity and power for Transformers, and the binarization of the weights is important for meeting these requirements. Approximate memory can further reduce the power for memory, but it also introduces errors. Current research for binary Transformers focuses on task performance, but the model dependable operation against errors on the memory has not been well studied. In this paper, we first evaluate the dependability of binary Transformers with the Bidirectional Encoder Representations from Transformers (BERT) as a case study. A large accuracy loss occurs only at high Bit Error Rates (e.g. BER = 5% or 10%). Then we propose a novel Permutation Invariance Enabled Selective (PIES) scheme to protect the binary weights at very low overhead. This scheme exploits the invariance of Transformers to permutations in the rows and columns to group the critical weights and perform selective protection on those weights. Simulation results show that PIES can recover most of the performance loss caused by bit errors at BERs as high as 10%, at an overhead significantly lower than traditional error correction schemes. These results confirm the feasibility of running binary Transformers using approximate memory with aggressive voltage scaling on resource-limited low power edge devices. The code for PIES is released on github (https://github.com/YangZhutian/binary-transformer-prefix-repair).

Explore similar work

CardsList
  1. BWTA: Accurate and Efficient Binarized Transformer by Algorithm-Hardware Co-design

    Apr 5, 2026Yifu Ding, Xianglong Liu, Shenghao Jin +2Binary QuantizationLow-Bit Quantization

  2. Activation Outliers in Transformer Quantization: Reproduction, Statistical Analysis, and Deployment Tradeoffs

    Mar 4, 2026Pranav Kumar KaliaperumalActivation OutliersPost-Training Quantization

  3. BinaryAttention: One-Bit QK-Attention for Vision and Diffusion Transformers

    Mar 10, 2026Chaodong Xiao, Zhengqiang Zhang, Lei ZhangAttention QuantizationAttention Mechanisms