Efficient INT8 Inference of Small NLP Models on Server CPUs with PyTorch Native Stack

Weiwen Xia; Yuxin Cui; E Cao

cs.CLAug 18, 2026

Efficient INT8 Inference of Small NLP Models on Server CPUs with PyTorch Native Stack

Authors: Weiwen Xia, Yuxin Cui, E Cao

Organizations: Intel Corporation

Abstract

Small NLP models, especially BERT-family encoders, remain important in industrial workloads such as classification, ranking, and retrieval even in the era of large language models. On server CPUs, INT8 quantization offers an attractive latency-throughput-cost trade-off, but users increasingly expect such acceleration to be available directly in the native PyTorch stack. We integrate SmoothQuant into TorchAO and optimize the resulting inference path for Intel Xeon CPUs through graph-level fusion in TorchInductor and efficient INT8 GEMM kernel selection across oneDNN-, AVX512_VNNI-, and AMX-based implementations. Across BERT, DistilBERT, and XLM-RoBERTa benchmarks, the approach delivers up to 5.8x end-to-end throughput speedup with negligible---and in some cases no measurable---accuracy loss relative to the FP32 baseline. We also validated our work by detailed performance analysis with roofline models. The implementation has been upstreamed to PyTorch and TorchAO, enabling out-of-the-box deployment with native PyTorch tooling

Figures & tables

Figure 1: SmoothQuant workflow built on TorchAO and TorchInductor. TorchAO prepares and quantizes the model, TorchInductor fuses INT8 subgraphs, and the CPP backend selects between oneDNN and template-based GEMM kernels. C++ code is generated and compiled for deployment.

Xeon Generation	Codename	Model	Physical Cores	Instruction Support
3rd Gen	Ice Lake	8358	64	AVX512F, AVX512_VNNI
6th Gen	Granite Rapids	6980P	256	AVX512F, AVX512_VNNI, AMX

Table 1: Hardware platforms used for baseline collection.

Item	Value
Model	BERT-large
Input shape	Batch size 1, sequence length 256
Precision modes	FP32 on both platforms; BF16 on Granite Rapids
Launch strategy	Multi-instance with weight-sharing on each socket
Cores per instance	4
Number of instances	Number of physical cores / 4

Table 2: Baseline collection setup.

Baseline	Linear-Block	Attention	LayerNorm	Other
FP32 on Ice Lake	88.5%	10.4%	0.6%	0.5%
FP32 on Granite Rapids	86.9%	11.8%	0.8%	0.5%
BF16 on Granite Rapids	72.70%	19.75%	5.99%	1.56%

Table 3: Baseline hotspot breakdown for the BERT-large analysis case under the same multi-instance setup used for throughput benchmarking. Operator shares are aggregated from representative profiler logs. The linear-block consists of GEMM and fused post-operations such as bias addition, activation, and residual addition.

Platform	FP32 Throughput (TFLOPS)	BF16 Throughput (TFLOPS)	INT8 Throughput (TOPS)	Memory Bandwidth (GB/s)	FP32 Balance Point (FLOPs/byte)	BF16 Balance Point (FLOPs/byte)	INT8 Balance Point (OPs/byte)
Ice Lake	6.76	N/A	27.03	175.00	38.63	N/A	154.46
Granite Rapids	19.66	262.14	524.29	700.00	28.09	374.49	748.99

Table 4: Estimated peak compute and measured memory bandwidth per socket, balance point for different data types on the evaluated platforms.

		Arithmetic Intensity (ops/byte)
GEMM Shape M $\times$ K $\times$ N	Occurrences per forward pass	FP32	BF16	INT8
$256\times 1024\times 1024$	96	85.33	146.29	227.56
$256\times 1024\times 4096$	24	97.52	163.84	248.24
$256\times 4096\times 1024$	24	97.52	186.18	341.33

Table 5: BERT-large GEMM shapes and their arithmetic intensity.

GEMM Shape	Effective Memory Bandwidth (GB/s)	Arithmetic Intensity (OPs/byte)	Balance Point (OPs/byte)
$256\times 1024\times 1024$	807.69	227.56	649.12
$256\times 1024\times 4096$	819.15	248.24	640.04
$256\times 4096\times 1024$	875.00	341.33	599.19

Table 6: Effective memory bandwidth and balance point for INT8 GEMM on Granite Rapids.

GEMM Shape	FP32 Throughput (TFLOPS)	BF16 Throughput (TFLOPS)	INT8 Throughput (TOPS)	INT8 vs FP32 Speedup	INT8 vs BF16 Speedup
$256\times 1024\times 1024$	19.66	102.40	183.79	9.35 $\times$	1.79 $\times$
$256\times 1024\times 4096$	19.66	114.69	203.35	10.34 $\times$	1.77 $\times$
$256\times 4096\times 1024$	19.66	130.33	298.67	15.19 $\times$	2.29 $\times$
Weighted Average	19.66	114.69	218.87	11.13 $\times$	1.91 $\times$

Table 7: Estimated GEMM throughput and speedup on Granite Rapids.

Model	Model ID
BERT-large	bert-large-uncased
DistilBERT	distilbert-base-uncased
XLM-RoBERTa	xlm-roberta-base

Table 8: Models used for the performance benchmark.

Dataset	Model	Model ID
SQuAD 1.1	BERT-large	google-bert/bert-large-uncased-whole-word-masking-finetuned-squad
SQuAD 1.1	DistilBERT	distilbert/distilbert-base-cased-distilled-squad
SQuAD 1.1	XLM-RoBERTa	deepset/xlm-roberta-base-squad2 *
MultiNLI	BERT-large	yoshitomo-matsubara/bert-large-uncased-mnli
MultiNLI	DistilBERT	typeform/distilbert-base-uncased-mnli
MultiNLI	XLM-RoBERTa	symanto/xlm-roberta-base-snli-mnli-anli-xnli

Table 9: Task-finetuned checkpoints used for accuracy evaluation. The model marked by * is finetuned on SQuAD 2.0 but still workable for the test on SQuAD 1.1.

Method	Short Name
No quantization, no AMP	FP32
No quantization, with AMP	BF16
SmoothQuant, static quantization	Smooth-Static
SmoothQuant, dynamic quantization	Smooth-Dynamic

Table 10: Quantization methods used in the experiments.

Model	Method	Throughput Speedup	Linear-Block Speedup
BERT-large	Smooth-Dynamic	2.22 $\times$	3.59 $\times$
BERT-large	Smooth-Static	2.58 $\times$	3.87 $\times$
DistilBERT	Smooth-Dynamic	1.90 $\times$	3.31 $\times$
DistilBERT	Smooth-Static	2.30 $\times$	3.35 $\times$
XLM-RoBERTa	Smooth-Dynamic	1.86 $\times$	3.05 $\times$
XLM-RoBERTa	Smooth-Static	2.17 $\times$	3.71 $\times$

Table 11: INT8 speedup over FP32 on Ice Lake.

Model	Method	Throughput Speedup	Linear-Block Speedup
BERT-large	Smooth-Dynamic	5.02 $\times$	9.92 $\times$
BERT-large	Smooth-Static	5.82 $\times$	8.67 $\times$
DistilBERT	Smooth-Dynamic	4.24 $\times$	7.60 $\times$
DistilBERT	Smooth-Static	5.37 $\times$	8.84 $\times$
XLM-RoBERTa	Smooth-Dynamic	4.17 $\times$	7.58 $\times$
XLM-RoBERTa	Smooth-Static	5.41 $\times$	8.71 $\times$

Table 12: INT8 speedup over FP32 on Granite Rapids.

Model	Method	Throughput Speedup	Linear-Block Speedup
BERT-large	Smooth-Dynamic	1.37 $\times$	1.84 $\times$
BERT-large	Smooth-Static	1.59 $\times$	1.61 $\times$
DistilBERT	Smooth-Dynamic	0.98 $\times$	1.45 $\times$
DistilBERT	Smooth-Static	1.24 $\times$	1.69 $\times$
XLM-RoBERTa	Smooth-Dynamic	0.99 $\times$	1.50 $\times$
XLM-RoBERTa	Smooth-Static	1.29 $\times$	1.67 $\times$

Table 13: INT8 speedup over BF16 on Granite Rapids.

			Throughput Speedup			Linear-Block Speedup
Platform	Method	Baseline	Realized	Estimated	Gap	Realized	Estimated	Gap
Ice Lake	Smooth-Dynamic	FP32	2.22 $\times$	3.15 $\times$	29.59%	3.59 $\times$	4.00 $\times$	10.22%
Ice Lake	Smooth-Static	FP32	2.58 $\times$	3.15 $\times$	18.17%	3.87 $\times$	4.00 $\times$	3.21%
Granite Rapids	Smooth-Dynamic	FP32	5.02 $\times$	7.06 $\times$	28.89%	9.92 $\times$	11.13 $\times$	10.89%
Granite Rapids	Smooth-Static	FP32	5.82 $\times$	7.06 $\times$	17.56%	8.67 $\times$	11.13 $\times$	22.12%
Granite Rapids	Smooth-Dynamic	BF16	1.37 $\times$	1.53 $\times$	10.41%	1.84 $\times$	1.91 $\times$	3.58%

Table 14: Realized speedup of BERT-large over estimated speedup ceiling on both evaluated platforms.

Model	Metric	FP32	BF16	Smooth-Static	Alpha	Loss	Smooth-Dynamic	Alpha	Loss
BERT-large	SQuAD F1	92.64	92.67	92.66	0.6	0.0%	92.63	0.6	0.0%
BERT-large	SQuAD EM	86.41	86.45	86.49	0.6	-0.1%	86.44	0.6	0.0%
BERT-large	MultiNLI Accuracy	64.55	64.58	64.73	0.7	-0.3%	64.52	0.6	0.0%
DistilBERT	SQuAD F1	86.39	86.41	86.00	0.7	0.5%	86.40	0.6	0.0%
DistilBERT	SQuAD EM	79.02	79.04	78.49	0.7	0.7%	79.04	0.6	0.0%
DistilBERT	MultiNLI Accuracy	82.21	82.16	82.26	0.8	-0.1%	82.17	0.6	0.0%

Table 15: Accuracy results on Granite Rapids.

Efficient Transformer Inference

Efficient Transformer Inference studies methods and systems that reduce the latency, memory, and energy needed to run Transformer models.

See all 121 papers

Realizing Native INT8 Compute for Diffusion Transformers on Consumer GPUs: A Fused INT8 GEMM Kernel for Ideogram 4.0
Jun 12, 2026Ali Asaria, Tony Salomone, Deep GandhiGPU Kernel Optimization Diffusion Transformer
Transformer Lab
Litespark Inference For CPUs: Ultra-Fast SIMD Framework for Ternary (1.58-bit) Language Models
May 7, 2026Nii Osae Osae Dade, Tony Morri, Moinul Hossain Rahat +2LLM Quantization Efficient Neural Network Inference
Mindbeam AI
FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper
Oct 16, 2026Wei WangEfficient Transformer Inference GPU Kernel Optimization

Jun 12, 2026J/K move · Enter open · S save

Realizing Native INT8 Compute for Diffusion Transformers on Consumer GPUs: A Fused INT8 GEMM Kernel for Ideogram 4.0

Ali Asaria, Tony Salomone, Deep Gandhi

Transformer Lab

GPU Kernel Optimization Diffusion Transformer Diffusion Model Quantization Post-Training Quantization Diffusion Model Inference Acceleration

Open paper AskPDF arXiv

Post-training INT8 (W8A8) quantization of diffusion transformers is widely deployed as a speed optimization, yet on consumer Ampere GPUs it is frequently slower than the FP8 and NF4 alternatives it is meant to beat. We trace this to a software artifact: the production "INT8" forward quantizes weights and activations only to immediately dequantize them back to bf16 and run a bf16 matrix multiply, never engaging the GPU's INT8 tensor cores, so the hardware's compute advantage is left entirely unrealized. We close this gap with a single fused Triton INT8 GEMM (int8xint8->int32 on Ampere tensor cores, with per-token x per-channel dequantization and bias folded into the epilogue, autotuned per GEMM shape) dropped into the Ideogram 4.0 diffusion transformer's linear layers in place of the dequantize-to-bf16 path. In the kernel, the int8xint8->int32 accumulation is bit-exact against torch._int_mm and the dequantized output matches the reference at cosine similarity 1.0 with no NaNs, running 2.8-4.2x faster than bf16 per GEMM. End to end it delivers a ~1.1x (~9-10%) speedup at 768px, and at 1024px it generates an image in 156.5 s on a single RTX 3090, faster than the single-card NF4 (164.5 s) and FP8 (172.9 s) baselines, at no measurable quality cost on these point estimates (PickScore/CLIPScore). INT8 thus goes from the slowest variant to the fastest, and 1024px becomes single-GPU feasible. The primary speed criterion (beat FP8, by ~9.5%) is comfortably met; the NF4 margin (~4.9%, single-run n=4) is within run-to-run variance we did not quantify and is best read as consistent with meeting the stretch target. We close with an honest deployment map: the win is specific to consumer Ampere, and on A100 and B200 the same kernel loses to those cards' fast native bf16/FP8 paths.

Abstract

Efficient Transformer Inference

Explore similar work

Realizing Native INT8 Compute for Diffusion Transformers on Consumer GPUs: A Fused INT8 GEMM Kernel for Ideogram 4.0

Litespark Inference For CPUs: Ultra-Fast SIMD Framework for Ternary (1.58-bit) Language Models

FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper