TileQuant: Tile as the Precision-Allocation Unit for LLM Mixed-Precision Post-Training Quantization

Hanzhi Zhang; Qiao Zhang; Qinglei Cao; Heng Fan; Yan Huang; Kewei Sha; Yunhe Feng

cs.LGNov 4, 2026

TileQuant: Tile as the Precision-Allocation Unit for LLM Mixed-Precision Post-Training Quantization

Authors: Hanzhi Zhang, Qiao Zhang, Qinglei Cao, Heng Fan, Yan Huang, Kewei Sha, Yunhe Feng

Organizations: LLaVi Lab, Computer Science & Engineering, University of North Texas · Computer Science, Saint Louis University · Data Science, University of North Texas

Abstract

Post-training quantization (PTQ) reduces large language model (LLM) deployment costs, but preserving quality requires accounting for uneven weight sensitivity. Uniform PTQ uses one weight bit-width, importance-aware PTQ adjusts scales or reconstructs weights within that width, and column-wise mixed precision retains selected columns at higher precision. However, sensitivity varies within columns, so whole-column retention also protects less sensitive weights, while runtime outlier detection adds decoding overhead. We therefore introduce TileQuant, a training-free framework that assigns precision to hardware-aligned 2D weight tiles. Offline calibration provides input-activation-aware scoring from forward statistics for lower calibration cost and gradient-informed scoring from loss sensitivity for stronger quality preservation. Within each projection's retention budget, the highest-scoring tiles remain at 16-bit and the rest use 8-bit. The resulting map guides reconstruction into compact weight buffers, which the kernel reuses for native integer and 16-bit computation with local accumulation to reduce memory traffic. Experiments show that TileQuant reaches 2.66x the original model's generation throughput. Across language modeling, reasoning, long-context retrieval, and question answering, it preserves quality and surpasses quantized and original models on multiple tasks. These results support selecting precision units that capture local sensitivity while enabling efficient hardware execution. The implementation is: https://github.com/HanzhiZhang-Ulrica/TileQuant

Figures & tables

Figure 1: Comparison of PTQ precision-allocation units for LLMs. Colors denote precision: blue indicates 16-bit, and green indicates 8-bit. Uniform PTQ uses one weight bit-width. Importance-aware PTQ adjusts scales or reconstructs weights at the same bit-width, with green shades illustrating calibrated scaling. Protected-subset PTQ illustrates channel-wise 16-bit retention. TileQuant assigns precision to 2D tiles by local sensitivity.

Figure 2: TileQuant framework for calibration-driven mixed-precision linear layers. The input-activation-aware route (top left) combines forward activation statistics with weight magnitudes, while the gradient-informed route (bottom left) accumulates weight-gradient importance over calibration batches. Per-layer top- $k$ selection converts either tile-score map into a 16-bit/8-bit decision map. Retained tile IDs and grid metadata guide load-time reconstruction (right). The reconstructed layer supplies the selected weights to two arithmetic paths, whose tile products are combined through output-local 32-bit floating-point accumulation.

Panel of Figure 3 — Figure 3: Generation-throughput speedup relative to BF16 on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8, and columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. Throughput includes the complete request and is normalized to BF16 at the same request shape. Values use the median of five measured runs after two warm-up runs. OOM denotes an out-of-memory run.

Panel of Figure 4 — Figure 4: Line-level retrieval on Llama-3.2-3B across 2.7K–43.4K prompt tokens. Lines (left axis) show exact-match accuracy, while stacked bar segments (right axis) show each configuration’s accuracy difference from Original. Top: TileQuant with GI maps (left) and IAA maps (right), both retaining 1%, 5%, 10%, or 20% of tiles in floating point (keep01, keep05, keep10, keep20). Bottom: uniform INT8 weight-only, uniform W8A8, GPTQ, SmoothQuant, and LLM.int8 (left); representative comparison of the strongest TileQuant and baseline configurations (right).

Panel of Table 2 — Table 2: Long-context question answering results for Llama-3.2-3B on LV-Eval at the 16K, 32K, and 64K length levels. Dataset IDs 1–9 denote CMRC, DuReader, HotpotWikiQA, LIC, LooGLE-CR, LooGLE-MIR, LooGLE-SD, English and Chinese MultiFieldQA, respectively. Cell colors are relative to Original, with green for higher scores and blue for lower scores.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Generation-throughput speedup relative to BF16 on Llama-3.2-3B for gradient-informed (GI) TileQuant variants. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel varies output length and compares GI retention ratios; OOM denotes an out-of-memory run.

Panel of Figure 6 — Figure 6: Generation-throughput speedup relative to BF16 on Llama-3.2-3B for input-activation-aware (IAA) TileQuant variants. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel varies output length and compares IAA retention ratios; OOM denotes an out-of-memory run.

Panel of Figure 7 — Figure 7: Generation-throughput speedup relative to BF16 on Llama-3.2-3B for baseline inference methods . Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel varies output length and compares the baseline methods; OOM denotes an out-of-memory run.

Panel of Figure 8 — Figure 8: Time-per-output-token (TPOT) comparison on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares inference methods across output lengths of 128, 256, 512, and 1024 tokens; OOM denotes an out-of-memory run.

Panel of Figure 9 — Figure 9: Time-per-output-token (TPOT) comparison for gradient-informed (GI) TileQuant variants on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares GI retention configurations across output lengths; OOM denotes an out-of-memory run.

Panel of Figure 10 — Figure 10: Time-per-output-token (TPOT) comparison for input-activation-aware (IAA) TileQuant variants on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares IAA retention ratios across output lengths; OOM denotes an out-of-memory run.

Panel of Figure 11 — Figure 11: Time-per-output-token (TPOT) comparison for baseline inference methods on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares the baseline configurations across output lengths; OOM denotes an out-of-memory run.

Panel of Figure 12 — Figure 12: Generation-energy comparison on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares inference methods across output lengths of 128, 256, 512, and 1024 tokens; OOM denotes an out-of-memory run.

Panel of Figure 13 — Figure 13: Generation-energy comparison for gradient-informed (GI) TileQuant variants on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares GI retention configurations across output lengths; OOM denotes an out-of-memory run.

Panel of Figure 14 — Figure 14: Generation-energy comparison for input-activation-aware (IAA) TileQuant variants on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares IAA retention configurations across output lengths; OOM denotes an out-of-memory run.

Panel of Figure 15 — Figure 15: Generation-energy comparison for baseline inference methods on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares the baseline configurations across output lengths; OOM denotes an out-of-memory run.

Panel of Figure 16 — Figure 16: Line-level retrieval for Qwen2.5-7B across prompt lengths from 3.2K to 52.9K tokens. Line plots (left y-axis) report exact-match accuracy, while bar plots (right y-axis) show accuracy differences relative to the original baseline. Top-left: TileQuant with gradient-informed maps, where keep01, keep05, keep10, and keep20 retain 1%, 5%, 10%, and 20% of tiles in the deployment floating-point dtype. Top-right: TileQuant with input-activation-aware maps under the same keep ratios. Bottom-left: baseline PTQ methods, including uniform INT8 weight-only quantization, uniform W8A8 quantization, GPTQ, SmoothQuant, and LLM.int8. Bottom-right: representative cross-family comparison between the strongest TileQuant configurations and strongest baseline methods.

Panel of Figure 17 — Figure 17: Line-level retrieval accuracy for Qwen2.5-14B on LongEval across prompt lengths from 3.2K to 52.9K tokens. Line plots (left y-axis) report exact-match retrieval accuracy, while bar plots (right y-axis) show accuracy differences relative to the Original floating-point model.

Panel of Table 10 — Table 10: Full long-context question answering results for Llama-3.2-3B on LV-Eval across 9 datasets and context lengths from 16K to 256K. GI and IAA denote Gradient-informed and Input-activation-aware tile-retention policies, with subscripts indicating the retained floating-point tile ratio in percent. We follow the LV-Eval metric protocol: keyword-recall F1 by default, blacklist-based F1 for cmrc_mixup , and blacklist-based ROUGE-L for dureader_mixup . Cell colors are relative to the row-wise Original reference: green and blue indicate higher and lower scores, respectively, while gray marks Original and values within the neutral comparison range.

Panel of Table 11 — Table 11: Full long-context question answering results for Qwen2.5-7B on LV-Eval across 9 datasets and context lengths from 16K to 256K. GI and IAA denote Gradient-informed and Input-activation-aware tile-retention policies, with subscripts indicating the retained floating-point tile ratio in percent. We follow the LV-Eval metric protocol: keyword-recall F1 by default, blacklist-based F1 for cmrc_mixup , and blacklist-based ROUGE-L for dureader_mixup . Cell colors are relative to the row-wise Original reference: green and blue indicate higher and lower scores, respectively, while gray marks Original and values within the neutral comparison range.

Panel of Table 12 — Table 12: Full long-context question answering results for Qwen2.5-14B on LV-Eval across 9 datasets and context lengths from 16K to 256K. GI and IAA denote Gradient-informed and Input-activation-aware tile-retention policies, with subscripts indicating the retained floating-point tile ratio in percent. We follow the LV-Eval metric protocol: keyword-recall F1 by default, blacklist-based F1 for cmrc_mixup , and blacklist-based ROUGE-L for dureader_mixup . Cell colors are relative to the row-wise Original reference: green and blue indicate higher and lower scores, respectively, while gray marks Original and values within the neutral comparison range.

Retention (%)	Target-linear (GiB)	Metric	Controls		Ours
			Random	Magnitude	IAA
5	6.39	Retained saliency mass (%) $\uparrow$	5.00	5.34	7.05
		Matched-control PPL $\downarrow$	$9.92\pm 0.03$	9.93	9.85
20	7.30	Retained saliency mass (%) $\uparrow$	20.00	20.76	23.90
		Matched-control PPL $\downarrow$	$9.82\pm 0.03$	9.78	9.7650

Appendix

Table 13: Matched-budget tile-selection controls on Qwen2.5-7B with

64\times 64

tiles. Matched-control PPL uses the fixed subset of 62 WikiText-2 test documents. The target-linear payload is shared by all selection rules within each budget. Random PPL is reported as mean

\pm

sample standard deviation over five draws. Bold marks the best displayed value in each row.

Metric	Unit	Allocation controls				Ours
		Full column	$1\times 64$	$64\times 1$	Full row	$64\times 64$
Total payload $\downarrow$	GiB	7.2995	7.3744	7.3744	7.2997	7.2984
Metadata $\downarrow$	MiB	0.43	77.21	77.21	0.53	0.61
Retained saliency mass $\uparrow$	Fraction	0.31	0.26	0.32	0.23	0.24
Matched-control PPL $\downarrow$	PPL	9.72	9.87	9.71	9.98	9.7650

Appendix

Table 14: Allocation-geometry controls on Qwen2.5-7B at a matched 20% high-precision retention target. Total payload denotes the stored representation measured in this control. Retained saliency mass is reported as a fraction, and PPL uses the fixed 62-document subset. Bold marks the best displayed value in each row.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 416 papers

AlignQuant: Tile-Aligned Mixed-Precision Quantization for Efficient LLM Generation
Oct 5, 2026Hanzhi Zhang, Qiao Zhang, Qinglei Cao +4Mixed-Precision Quantization LLM Quantization
LLaVi Lab, Computer Science & Engineering, University of North Texas
You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations
Nov 9, 2025Amit LeVi, Raz Lapid, Rom Himelstein +3Neural Network Quantization Mixed-Precision Quantization
1Technion—Israel Institute of Technology · 2Intuit · 3Ben-Gurion University of the Negev +1
MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM
Feb 21, 2026Dongwei Wang, Jinhee Kim, Seokho Han +8LLM Quantization Neural Network Quantization
University of Arizona · Panasonic AI Lab · Duke University +2

Oct 5, 2026J/K move · Enter open · S save

AlignQuant: Tile-Aligned Mixed-Precision Quantization for Efficient LLM Generation

Hanzhi Zhang, Qiao Zhang, Qinglei Cao, Heng Fan+3

LLaVi Lab, Computer Science & Engineering, University of North Texas

Mixed-Precision Quantization LLM Quantization LLM Inference Acceleration Post-Training Quantization Neural Network Quantization

Open paper AskPDF arXiv

Fine-grained mixed-precision quantization promises efficient large language model inference, but local precision choices can conflict with regular GPU storage and computation units. This precision-boundary mismatch limits the translation of compression into practical acceleration. We introduce AlignQuant, a post-training quantization method that uses GPU-compatible two-dimensional weight tiles as the common unit of precision allocation, compact storage, and execution. This shared partition lets precision follow sensitivity within output channels. Joint prefill/decode calibration scores precision reductions using projection-output perturbations weighted by language-model loss gradients under quantized activations. Phase-normalized scores prioritize higher precision for tiles important to either phase under a model-wide weight-storage budget. Each tile stores one selected representation, while phase-specialized kernels reuse the packed model and expand lower-bit weights for INT8 computation with 8-bit activations. Across four LLMs spanning 3B to 14B parameters, AlignQuant achieves up to

2.50\times

generation speedup over BF16 while preserving model quality. Evaluations further cover three GPUs and contexts up to 64K tokens. These results show that local precision flexibility and regular GPU execution can coexist through a shared tile unit. The implementation is available at https://github.com/HanzhiZhang-Ulrica/AlignQuant.

Abstract

LLM Quantization

Explore similar work

AlignQuant: Tile-Aligned Mixed-Precision Quantization for Efficient LLM Generation

You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations

MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM