ShatterQuant: Breaking Uniform Precision with Block-Wise Mixed-Precision on a Systolic Transformer Hardware Accelerator

Mikolaj Walczak; Edward Humes; Chao Fang; Marian Verhelst; Tinoosh Mohsenin

cs.ARSep 20, 2026

ShatterQuant: Breaking Uniform Precision with Block-Wise Mixed-Precision on a Systolic Transformer Hardware Accelerator

Authors: Mikolaj Walczak, Edward Humes, Chao Fang, Marian Verhelst, Tinoosh Mohsenin

Organizations: Johns Hopkins University Baltimore, Maryland, USA

Abstract

Due to limited support for intra-tensor heterogeneous precision in conventional accelerators, neural network quantization remains largely restricted to per-tensor precision assignment. We present ShatterQuant, a hardware-software co-designed framework enabling mixed-precision quantization within each tensor by assigning independent bit-widths to blocks of a weight projection. ShatterQuant couples precision granularity with PE configuration, such that each precision determines an effective block height. We introduce (1) a hardware-aware post-training method that assigns intra-tensor precision based on block-level standard deviation and weight sensitivity; (2) the ShatterQuant Transformer Accelerator supporting 1/2/4/8-bit weight precision, precision-dependent PE configuration, block rescaling, and integrated softmax and piecewise-linear nonlinearities; and (3) an evaluation of model-hardware tradeoffs using an implementation in the TSMC 16nm PDK operating at 1 GHz, achieving 1.5 TOPS, 760 GOPS/ $mm^2$ area efficiency, and 2.8 TOPS/W energy efficiency. On DeiT and ImageNet-1K, ShatterQuant achieves accuracy within $3.3\%$ of state-of-the-art mixed-precision techniques while using a 2 bit lower effective bitwidth, while for PixelDiT demonstrates comparable generation quality. ShatterQuant demonstrates how fine-grained intra-tensor mixed-precision can be realized through hardware-software co-design.

Figures & tables

Figure 1. Motivation of ShatterQuant compared with existing mixed-precision quantization approaches. Existing methods ( Ranjan and others, 2026 ; Sun and others, 2026 ) evaluate coarse-grained, tensor-wise precision assignments, whereas ShatterQuant enables fine-grained mixed-precision assignments within individual weight tensors through hardware-software co-design.

Figure 2. Vision Transformer (ViT) architecture and high-level overview of ShatterQuant, with PixelDiT shown as a representative vision diffusion model. ShatterQuant is applied to linear projections through a three-stage process: (1) calibration and initial weight setting, (2) quantization and weight rounding correction, and (3) hardware deployment.

Figure 3. Overview of the ShatterQuant Transformer Accelerator and dataflow. Green: weight memory and input scales; Blue: activation memory; Red: mixed-precision systolic array with barrel-rotator PEs; Yellow: output buffer and rescaler; Purple: fused PWL-based quantizer with softmax normalization and range-overflow prevention. (A) Single-block execution (B) Execution for larger matrix operations. (C) Fused PWL-softmax quantization integrated within the ShatterQuant quantizer.

Figure 4. Per layer effective-bitwidth evaluation for DeIT and PixelDiT-XL post ShatterQuant quantization (A) Effective per-layer bitwidth for DeiT with a 5% overall 1-bit weight target for a $4\times 4$ PE configuration. (B) Effective per-layer bitwidths for PixelDiT-XL for individual projections within the patch & pixel transformer blocks with 80% of its weights being 8-bit, and 20% of its weights being 4-bit, using a $4\times 4$ PE array configuration. (C) Comparison of generated images between original full precision PixelDiT-XL model, and quantized

	DeiT-Tiny		DeiT-Small		DeiT-Base
Method	Prec.	Acc.	Prec.	Acc.	Prec.	Acc.
Baseline	FP32	74.5	FP32	81.2	FP32	83.4
P2ViT	8	69.6	8	77.6	8	80.6
Mix-QViT	6	71.7	6	79.6	6	81.6
SLIQ-ViT	6	71.8	6	79.6	6	81.8
Ours	5	68.5	4	78.2	4	80.7

Table 1. Comparison of ShatterQuant against existing mixed-precision works on three DeiT ( Touvron and others, 2021 ) variants shown for Imagenet-1k Top-1 accuracy using the 4x4 PE configuration. Precision defines effective overall model weight bitwidth.

Figure 5. Hardware implementation and performance breakdown of the ShatterQuant Transformer Accelerator: (A) 32×32 configuration layout with color-coded components corresponding to (B) area and power breakdowns for four PE configurations, and (C) scale buffer overhead and achieved throughput relative to peak performance.

Work	Prec.	Tech. (nm)	Freq. (MhZ)	SRAM (KiB)	Perf. (TOPS)	Area (mm 2 )	Power (mW)	Area Eff. (GOPS/mm 2 )	Energy Eff. (TOPS/W)
FLTA ( Lin and others, 2024 )	A 8 W 8	28	666	38.8	0.1	0.6	74	241	1.9
SoftEx ( Belano and others, 2025 )	BF 16	12	1120	256	0.4	1.21	581	355	1.6
FlexEdge ( Zheng and others, 2026 )	A 8 W 8	28	600	36	1.2	1.8	404	640	2.9
( Hsiung and others, 2026 )	INT8	28	1000	232	1.0	1.2	443	851	2.3
Ours	A 8 W 1/2/4/8	16	1000	1666.2	1.5	1.9	521	760	2.8

Table 2. Comparison of the ShatterQuant Transformer Accelerator with the 32x32 PE configuration against ASIC transformer works evaluated on a dual matmul workload containing a 20/40/30/10% distribution of 1/2/4/8-bit weights.

Figure 6. Top-1 accuracy vs energy efficiency (left) and overall model effective bitwidth vs PE configuration (right) for the three DeiT models tiny (a,b), small (c,d), and base (e,f). Results are shown for 5 overall 1-bit configurations of 0,5,10,15 and 20% signified using an increasing gradient.

Figure 7. gFiD vs energy efficiency (a) and effective model bitwidth vs PE configuration for PixelDiT-XL (b). Results are shown for a 0,20 and 40% 4-bit split signified by an increasing gradient.

Mixed-Precision Quantization

Mixed-precision quantization assigns different numeric precisions across a model to balance accuracy with memory, speed, energy, and hardware constraints.

See all 159 papers

MixFrag: Fragility-Guided Mixed-Precision Post-Training Quantization for Vision Transformers
Jul 30, 2026Md. Mehrab Hossain Opi, Robiul Islam Ryad, Md. Umar FarukMixed-Precision Quantization Vision Transformer
Department of Computer Science and Engineering Khulna University of Engineering & Technology (KUET) Khulna 9203, Bangladesh
Precision As You Need: Stochastic Computing Is a Dense Adaptive Quantizer
Sep 26, 2026Haoran Jin, Kangqi Zhang, Jirong Yang +4Adaptive Quantization Efficient Transformer Inference
Department of Electrical Engineering and Computer Science, University of Michigan
You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations
Nov 9, 2025Amit LeVi, Raz Lapid, Rom Himelstein +3Weight-Only Quantization Mixed-Precision Quantization
1Technion—Israel Institute of Technology · 2Intuit · 3Ben-Gurion University of the Negev +1

Jul 30, 2026J/K move · Enter open · S save

MixFrag: Fragility-Guided Mixed-Precision Post-Training Quantization for Vision Transformers

Md. Mehrab Hossain Opi, Robiul Islam Ryad, Md. Umar Faruk

Department of Computer Science and Engineering Khulna University of Engineering & Technology (KUET) Khulna 9203, Bangladesh

Mixed-Precision Quantization Vision Transformer Efficient ViTs Post-Training Quantization Neural Network Quantization

Open paper AskPDF arXiv

Post-training quantization (PTQ) has emerged as an effective solution for deploying Vision Transformers (ViTs) on resource-constrained devices. However, existing PTQ methods typically employ uniform bit-widths across transformer components, overlooking their heterogeneous sensitivity to quantization and leading to inefficient precision allocation. In this paper, we propose {MixFrag, a fragility-guided mixed-precision PTQ framework for Vision Transformers. MixFrag first estimates component-level quantization fragility by measuring the Kullback--Leibler (KL) divergence between full-precision and isolated quantized output distributions using a small calibration set. It then formulates bit allocation as a Multiple-Choice Knapsack Problem (MCKP), enabling adaptive layer-wise precision assignment under a target bit budget. Extensive experiments on ImageNet-1K across multiple Vision Transformer architectures demonstrate that MixFrag achieves competitive classification performance under practical mixed-precision settings. Furthermore, evaluations on COCO object detection and instance segmentation show that MixFrag achieves state-of-the-art performance among existing mixed-precision PTQ methods, improving the previous best method by up to 9.6 AP under the challenging MP3/MP3 setting. Additional analyses validate the proposed fragility metric and demonstrate its strong correlation with the learned bit allocation. These results establish MixFrag as an effective framework for mixed-precision post-training quantization of Vision Transformers.

Abstract

Mixed-Precision Quantization

Explore similar work

MixFrag: Fragility-Guided Mixed-Precision Post-Training Quantization for Vision Transformers

Precision As You Need: Stochastic Computing Is a Dense Adaptive Quantizer

You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations