Implementing Binary Transformers with Approximate Memories with Permutation Invariance Enabled Selective (PIES) Protection
Organizations: School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China · Information Processing and Telecommunications Center, ETSI de Telecomunicación, Universidad Politécnica de Madrid, Madrid 28040 · School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu, Sichuan 611731, China · Department of Electrical and Computer Engineering, Northeastern University, Boston, MA 02215, USA
Abstract
Quantization is an effective technique to reduce memory storage, computation complexity and power for Transformers, and the binarization of the weights is important for meeting these requirements. Approximate memory can further reduce the power for memory, but it also introduces errors. Current research for binary Transformers focuses on task performance, but the model dependable operation against errors on the memory has not been well studied. In this paper, we first evaluate the dependability of binary Transformers with the Bidirectional Encoder Representations from Transformers (BERT) as a case study. A large accuracy loss occurs only at high Bit Error Rates (e.g. BER = 5% or 10%). Then we propose a novel Permutation Invariance Enabled Selective (PIES) scheme to protect the binary weights at very low overhead. This scheme exploits the invariance of Transformers to permutations in the rows and columns to group the critical weights and perform selective protection on those weights. Simulation results show that PIES can recover most of the performance loss caused by bit errors at BERs as high as 10%, at an overhead significantly lower than traditional error correction schemes. These results confirm the feasibility of running binary Transformers using approximate memory with aggressive voltage scaling on resource-limited low power edge devices. The code for PIES is released on github (https://github.com/YangZhutian/binary-transformer-prefix-repair).