Matrix multiplications dominate the inference cost of modern transformer-based vision models, yet existing efficiency techniques such as post-training quantization and mixed-precision inference are largely limited to the small set of fixed-width formats (INT4, INT8, BF16, and FP16) supported by conventional accelerators. We revisit stochastic computing (SC) as a way to lift this constraint: viewed as a dense adaptive quantizer, SC controls precision by bit-stream length L rather than a fixed datapath, while each multiplication reduces to a single AND/XNOR gate. We build a GPU library that emulates SC matrix multiplication at scale, exposes stream lengths as first-class kernel arguments, and evaluates SC end-to-end on image classification, object detection and instance segmentation, class-conditional image generation, and visual world-model planning. On top of this substrate, we develop a dynamic per-row mixed-precision policy that assigns stream length per token or group at matched average budget, requires no retraining, and uses the same SC hardware across schedules. Across tasks, SC remains competitive with fixed-format INT quantization at matched bit budgets, while per-row mixed precision helps maintain accuracy at lower average stream lengths. These results provide software-level feasibility evidence that SC can serve as a dense-precision substrate for fine-grained mixed-precision inference on modern vision transformers.
Figures & tables
Figure 1: Circuit primitives of stochastic computing. (a) A SNG implemented as an integer comparator: at each cycle the source value is compared against an RNG draw, emitting a single bit. (b) Stochastic multiplication via a bitwise gate: the unipolar product is realized by an AND gate whose output bit-probability factors as the product of the input marginals. The red circle denotes early termination. (c) Conditional bitstream generation (C-BSG): the second RNG is gated by the first stream so the conditional distribution of zB matches its marginal, enforcing SCC=0 and making the bipolar XNOR multiplier exact at finite L .
Figure 2: Overview of our adaptive fine-grained quantization and scheduling flow.
SC gate
INT8 mult.
Energy/Op Reduction
Pair
Area
Power
Area
Power
k=8
k=7
k=6
k=5
( μ m 2 )
( μ W)
( μ m 2 )
( μ W)
And / UMul
1.06
0.37
285.4
168.7
1.77×
3.54×
7.08×
14.2×
Xnor / Mul
1.60
0.63
249.2
249.7
1.56×
3.13×
6.25×
12.5×
Table 1: SC vs. binary multiplier comparison at 400 MHz.
Figure 3: SC sensitivity is highly non-uniform across operator class, block depth, and stream length: replacing a single (o,b) matmul with SC perturbs the final ViT-L/14 feature far more for late FC2 blocks than for any other tile, motivating the mixed-precision policy of Section 4.4.
Method
W/A
top-1 (%)
top-5 (%)
Δ vs FP
FP (DINOv2 ViT-L/14)
16/16
86.40
97.65
0.00
QwT
8/8
84.57
96.56
−1.83
SC uniform
8/8
85.57
97.40
−0.83
SC uniform + compensation
8/8
85.70
97.47
−0.70
SC uniform
log2192/log2192
84.67
97.11
−1.73
SC uniform + compensation
log2192/log2192
85.42
97.43
−0.98
Table 2: ImageNet-1k classification (full val, N=50,000 ). W/A is weight/activation bits: the underlying INT setting for QwT; for SC, W=A=log2Lˉ with Lˉ the average per-row stream length.
Method
W/A
AP b
AP m
Δb/Δm
FP (EVA-01 ViT-g)
16/16
62.06
52.39
0.00/0.00
QwT
8/8
61.20
51.59
−0.86/−0.80
SC uniform
8/8
61.98
52.18
−0.08/−0.21
SC uniform
log2192/log2192
61.69
51.85
−0.37/−0.54
SC uniform + compensation
log2192/log2192
61.82
52.06
−0.24/−0.33
QwT
7/7
61.13
51.51
−0.93/−0.88
Table 3: COCO val object detection & instance segmentation, EVA-01 ViT-g + cascade Mask R-CNN ( N=5000 , 10242 ). W/A as in Table 2 .
Method
Lˉ
IS ↑
sFID ↓
KID ×103↓
FID ↓
FP
–
285.95
24.75
13.47±1.24
24.81
SC MP
64
279.76
19.93
9.7717±1.0457
17.33
SC uniform
96
284.62
20.91
11.1134±1.1318
18.47
Table 4: ImageNet class-conditional generation with DiT-XL/2, 256×256 . All rows use the same sampler, 50 denoising timesteps, CFG=4.0 , and 10,000 generated images ( 10 per ImageNet class). Schedule records the SC length setting, Lˉ is the realized average stream length over SC-executed matmuls, and KID is reported as 103×KID .
Method
W/A
SR
Plateau iter
Δ vs FP
FP (Dino-WM, ViT-S/14 enc. + 6-block pred.)
16/16
0.833
5
0.000
SC uniform
8/8
0.833
3
0.000
SC uniform
log2192/log2192
0.833
3
0.000
SC MP
log2192/log2192
0.933
16
+0.100
SC uniform
7/7
0.800
16
−0.033
SC MP
7/7
0.833
13
0.000
Table 5: Visual planning success rate on Dino-WM wall_single , matched CEM budget. SR is the fraction of 30 episodes solved within 20 MPC steps; bit budget follows Table 2 .
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Sample images generated by full precision and SC with uniform stream lengths L∈{32,48,64,96,128} . The eight prompts are: “golden retriever”, “otter”, “lesser panda”, “geyser”, “macaw”, “valley”, “balloon”, and “arctic fox”.
Due to limited support for intra-tensor heterogeneous precision in conventional accelerators, neural network quantization remains largely restricted to per-tensor precision assignment. We present ShatterQuant, a hardware-software co-designed framework enabling mixed-precision quantization within each tensor by assigning independent bit-widths to blocks of a weight projection. ShatterQuant couples precision granularity with PE configuration, such that each precision determines an effective block height. We introduce (1) a hardware-aware post-training method that assigns intra-tensor precision based on block-level standard deviation and weight sensitivity; (2) the ShatterQuant Transformer Accelerator supporting 1/2/4/8-bit weight precision, precision-dependent PE configuration, block rescaling, and integrated softmax and piecewise-linear nonlinearities; and (3) an evaluation of model-hardware tradeoffs using an implementation in the TSMC 16nm PDK operating at 1 GHz, achieving 1.5 TOPS, 760 GOPS/mm2 area efficiency, and 2.8 TOPS/W energy efficiency. On DeiT and ImageNet-1K, ShatterQuant achieves accuracy within 3.3% of state-of-the-art mixed-precision techniques while using a 2 bit lower effective bitwidth, while for PixelDiT demonstrates comparable generation quality. ShatterQuant demonstrates how fine-grained intra-tensor mixed-precision can be realized through hardware-software co-design.
Post-training quantization (PTQ) has emerged as an effective solution for deploying Vision Transformers (ViTs) on resource-constrained devices. However, existing PTQ methods typically employ uniform bit-widths across transformer components, overlooking their heterogeneous sensitivity to quantization and leading to inefficient precision allocation. In this paper, we propose {MixFrag, a fragility-guided mixed-precision PTQ framework for Vision Transformers. MixFrag first estimates component-level quantization fragility by measuring the Kullback--Leibler (KL) divergence between full-precision and isolated quantized output distributions using a small calibration set. It then formulates bit allocation as a Multiple-Choice Knapsack Problem (MCKP), enabling adaptive layer-wise precision assignment under a target bit budget. Extensive experiments on ImageNet-1K across multiple Vision Transformer architectures demonstrate that MixFrag achieves competitive classification performance under practical mixed-precision settings. Furthermore, evaluations on COCO object detection and instance segmentation show that MixFrag achieves state-of-the-art performance among existing mixed-precision PTQ methods, improving the previous best method by up to 9.6 AP under the challenging MP3/MP3 setting. Additional analyses validate the proposed fragility metric and demonstrate its strong correlation with the learned bit allocation. These results establish MixFrag as an effective framework for mixed-precision post-training quantization of Vision Transformers.
Md. Mehrab Hossain Opi, Robiul Islam Ryad, Md. Umar Faruk
Department of Computer Science and Engineering Khulna University of Engineering & Technology (KUET) Khulna 9203, Bangladesh
Transformers achieve state-of-the-art performance in natural language processing, vision, and scientific computing, but demand high computation and memory. To address these challenges, we present ASTRA, the first silicon-photonic accelerator leveraging stochastic computing for transformers. ASTRA employs novel optical stochastic multipliers and unary/analog homodyne accumulation in a crosstalk-minimal organization to efficiently process dynamic tensor computations. Evaluations show at least 7.6x speedup and 1.3x lower energy overheads compared to state-of-the-art accelerators, highlighting ASTRA's potential for efficient, scalable, and sustainable transformer inference.
S. Afifi, O. Alo, I. Thakkar +1
Department of Electrical and Computer Engineering, Colorado State University, Fort Collins, CO, USA · Department of Electrical and Computer Engineering, University of Kentucky, Lexington, KY, USA