cs.LGNov 4, 2026

TileQuant: Tile as the Precision-Allocation Unit for LLM Mixed-Precision Post-Training Quantization

Authors: Hanzhi Zhang, Qiao Zhang, Qinglei Cao, Heng Fan, Yan Huang, Kewei Sha, Yunhe Feng

Organizations: LLaVi Lab, Computer Science & Engineering, University of North Texas · Computer Science, Saint Louis University · Data Science, University of North Texas

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Post-training quantization (PTQ) reduces large language model (LLM) deployment costs, but preserving quality requires accounting for uneven weight sensitivity. Uniform PTQ uses one weight bit-width, importance-aware PTQ adjusts scales or reconstructs weights within that width, and column-wise mixed precision retains selected columns at higher precision. However, sensitivity varies within columns, so whole-column retention also protects less sensitive weights, while runtime outlier detection adds decoding overhead. We therefore introduce TileQuant, a training-free framework that assigns precision to hardware-aligned 2D weight tiles. Offline calibration provides input-activation-aware scoring from forward statistics for lower calibration cost and gradient-informed scoring from loss sensitivity for stronger quality preservation. Within each projection's retention budget, the highest-scoring tiles remain at 16-bit and the rest use 8-bit. The resulting map guides reconstruction into compact weight buffers, which the kernel reuses for native integer and 16-bit computation with local accumulation to reduce memory traffic. Experiments show that TileQuant reaches 2.66x the original model's generation throughput. Across language modeling, reasoning, long-context retrieval, and question answering, it preserves quality and surpasses quantized and original models on multiple tasks. These results support selecting precision units that capture local sensitivity while enabling efficient hardware execution. The implementation is: https://github.com/HanzhiZhang-Ulrica/TileQuant

Figures & tables

Figure 1: Comparison of PTQ precision-allocation units for LLMs. Colors denote precision: blue indicates 16-bit, and green indicates 8-bit. Uniform PTQ uses one weight bit-width. Importance-aware PTQ adjusts scales or reconstructs weights at the same bit-width, with green shades illustrating calibrated scaling. Protected-subset PTQ illustrates channel-wise 16-bit retention. TileQuant assigns precision to 2D tiles by local sensitivity.
Figure 2: TileQuant framework for calibration-driven mixed-precision linear layers. The input-activation-aware route (top left) combines forward activation statistics with weight magnitudes, while the gradient-informed route (bottom left) accumulates weight-gradient importance over calibration batches. Per-layer top- kk selection converts either tile-score map into a 16-bit/8-bit decision map. Retained tile IDs and grid metadata guide load-time reconstruction (right). The reconstructed layer supplies the selected weights to two arithmetic paths, whose tile products are combined through output-local 32-bit floating-point accumulation.
Figure 3: Generation-throughput speedup relative to BF16 on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8, and columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. Throughput includes the complete request and is normalized to BF16 at the same request shape. Values use the median of five measured runs after two warm-up runs. OOM denotes an out-of-memory run.
Figure 4: Line-level retrieval on Llama-3.2-3B across 2.7K–43.4K prompt tokens. Lines (left axis) show exact-match accuracy, while stacked bar segments (right axis) show each configuration’s accuracy difference from Original. Top: TileQuant with GI maps (left) and IAA maps (right), both retaining 1%, 5%, 10%, or 20% of tiles in floating point (keep01, keep05, keep10, keep20). Bottom: uniform INT8 weight-only, uniform W8A8, GPTQ, SmoothQuant, and LLM.int8 (left); representative comparison of the strongest TileQuant and baseline configurations (right).
Table 2: Long-context question answering results for Llama-3.2-3B on LV-Eval at the 16K, 32K, and 64K length levels. Dataset IDs 1–9 denote CMRC, DuReader, HotpotWikiQA, LIC, LooGLE-CR, LooGLE-MIR, LooGLE-SD, English and Chinese MultiFieldQA, respectively. Cell colors are relative to Original, with green for higher scores and blue for lower scores.
Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Generation-throughput speedup relative to BF16 on Llama-3.2-3B for gradient-informed (GI) TileQuant variants. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel varies output length and compares GI retention ratios; OOM denotes an out-of-memory run.
Figure 6: Generation-throughput speedup relative to BF16 on Llama-3.2-3B for input-activation-aware (IAA) TileQuant variants. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel varies output length and compares IAA retention ratios; OOM denotes an out-of-memory run.
Figure 7: Generation-throughput speedup relative to BF16 on Llama-3.2-3B for baseline inference methods . Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel varies output length and compares the baseline methods; OOM denotes an out-of-memory run.
Figure 8: Time-per-output-token (TPOT) comparison on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares inference methods across output lengths of 128, 256, 512, and 1024 tokens; OOM denotes an out-of-memory run.
Figure 9: Time-per-output-token (TPOT) comparison for gradient-informed (GI) TileQuant variants on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares GI retention configurations across output lengths; OOM denotes an out-of-memory run.
Figure 10: Time-per-output-token (TPOT) comparison for input-activation-aware (IAA) TileQuant variants on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares IAA retention ratios across output lengths; OOM denotes an out-of-memory run.
Figure 11: Time-per-output-token (TPOT) comparison for baseline inference methods on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares the baseline configurations across output lengths; OOM denotes an out-of-memory run.
Figure 12: Generation-energy comparison on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares inference methods across output lengths of 128, 256, 512, and 1024 tokens; OOM denotes an out-of-memory run.
Figure 13: Generation-energy comparison for gradient-informed (GI) TileQuant variants on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares GI retention configurations across output lengths; OOM denotes an out-of-memory run.
Figure 14: Generation-energy comparison for input-activation-aware (IAA) TileQuant variants on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares IAA retention configurations across output lengths; OOM denotes an out-of-memory run.
Figure 15: Generation-energy comparison for baseline inference methods on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares the baseline configurations across output lengths; OOM denotes an out-of-memory run.
Figure 16: Line-level retrieval for Qwen2.5-7B across prompt lengths from 3.2K to 52.9K tokens. Line plots (left y-axis) report exact-match accuracy, while bar plots (right y-axis) show accuracy differences relative to the original baseline. Top-left: TileQuant with gradient-informed maps, where keep01, keep05, keep10, and keep20 retain 1%, 5%, 10%, and 20% of tiles in the deployment floating-point dtype. Top-right: TileQuant with input-activation-aware maps under the same keep ratios. Bottom-left: baseline PTQ methods, including uniform INT8 weight-only quantization, uniform W8A8 quantization, GPTQ, SmoothQuant, and LLM.int8. Bottom-right: representative cross-family comparison between the strongest TileQuant configurations and strongest baseline methods.
Figure 17: Line-level retrieval accuracy for Qwen2.5-14B on LongEval across prompt lengths from 3.2K to 52.9K tokens. Line plots (left y-axis) report exact-match retrieval accuracy, while bar plots (right y-axis) show accuracy differences relative to the Original floating-point model.
Table 10: Full long-context question answering results for Llama-3.2-3B on LV-Eval across 9 datasets and context lengths from 16K to 256K. GI and IAA denote Gradient-informed and Input-activation-aware tile-retention policies, with subscripts indicating the retained floating-point tile ratio in percent. We follow the LV-Eval metric protocol: keyword-recall F1 by default, blacklist-based F1 for cmrc_mixup , and blacklist-based ROUGE-L for dureader_mixup . Cell colors are relative to the row-wise Original reference: green and blue indicate higher and lower scores, respectively, while gray marks Original and values within the neutral comparison range.
Table 11: Full long-context question answering results for Qwen2.5-7B on LV-Eval across 9 datasets and context lengths from 16K to 256K. GI and IAA denote Gradient-informed and Input-activation-aware tile-retention policies, with subscripts indicating the retained floating-point tile ratio in percent. We follow the LV-Eval metric protocol: keyword-recall F1 by default, blacklist-based F1 for cmrc_mixup , and blacklist-based ROUGE-L for dureader_mixup . Cell colors are relative to the row-wise Original reference: green and blue indicate higher and lower scores, respectively, while gray marks Original and values within the neutral comparison range.
Table 12: Full long-context question answering results for Qwen2.5-14B on LV-Eval across 9 datasets and context lengths from 16K to 256K. GI and IAA denote Gradient-informed and Input-activation-aware tile-retention policies, with subscripts indicating the retained floating-point tile ratio in percent. We follow the LV-Eval metric protocol: keyword-recall F1 by default, blacklist-based F1 for cmrc_mixup , and blacklist-based ROUGE-L for dureader_mixup . Cell colors are relative to the row-wise Original reference: green and blue indicate higher and lower scores, respectively, while gray marks Original and values within the neutral comparison range.
Retention (%)Target-linear (GiB)MetricControlsOurs
   RandomMagnitudeIAA
56.39Retained saliency mass (%) ↑\uparrow5.005.347.05
  Matched-control PPL ↓\downarrow9.92±0.039.92\pm 0.039.939.85
207.30Retained saliency mass (%) ↑\uparrow20.0020.7623.90
  Matched-control PPL ↓\downarrow9.82±0.039.82\pm 0.039.789.7650
Appendix
Table 13: Matched-budget tile-selection controls on Qwen2.5-7B with 64×6464\times 64 tiles. Matched-control PPL uses the fixed subset of 62 WikiText-2 test documents. The target-linear payload is shared by all selection rules within each budget. Random PPL is reported as mean ±\pm sample standard deviation over five draws. Bold marks the best displayed value in each row.
MetricUnitAllocation controlsOurs
  Full column1×641\times 6464×164\times 1Full row64×6464\times 64
Total payload ↓\downarrowGiB7.29957.37447.37447.29977.2984
Metadata ↓\downarrowMiB0.4377.2177.210.530.61
Retained saliency mass ↑\uparrowFraction0.310.260.320.230.24
Matched-control PPL ↓\downarrowPPL9.729.879.719.989.7650
Appendix
Table 14: Allocation-geometry controls on Qwen2.5-7B at a matched 20% high-precision retention target. Total payload denotes the stored representation measured in this control. Retained saliency mass is reported as a fraction, and PPL uses the fixed 62-document subset. Bold marks the best displayed value in each row.
More in this topic

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 416 papers

Explore similar work

CardsList
Oct 5, 2026

AlignQuant: Tile-Aligned Mixed-Precision Quantization for Efficient LLM Generation

Fine-grained mixed-precision quantization promises efficient large language model inference, but local precision choices can conflict with regular GPU storage and computation units. This precision-boundary mismatch limits the translation of compression into practical acceleration. We introduce AlignQuant, a post-training quantization method that uses GPU-compatible two-dimensional weight tiles as the common unit of precision allocation, compact storage, and execution. This shared partition lets precision follow sensitivity within output channels. Joint prefill/decode calibration scores precision reductions using projection-output perturbations weighted by language-model loss gradients under quantized activations. Phase-normalized scores prioritize higher precision for tiles important to either phase under a model-wide weight-storage budget. Each tile stores one selected representation, while phase-specialized kernels reuse the packed model and expand lower-bit weights for INT8 computation with 8-bit activations. Across four LLMs spanning 3B to 14B parameters, AlignQuant achieves up to 2.50×2.50\times generation speedup over BF16 while preserving model quality. Evaluations further cover three GPUs and contexts up to 64K tokens. These results show that local precision flexibility and regular GPU execution can coexist through a shared tile unit. The implementation is available at https://github.com/HanzhiZhang-Ulrica/AlignQuant.

Hanzhi Zhang, Qiao Zhang, Qinglei Cao +4

LLaVi Lab, Computer Science & Engineering, University of North Texas

Mixed-Precision QuantizationLLM QuantizationLLM Inference AccelerationPost-Training QuantizationNeural Network Quantization
AskPDF
Nov 9, 2025

You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations

Many LLM applications require only narrow capabilities, yet standard post-training quantization (PTQ) methods allocate precision without considering the target task. This can waste bits on layers that are less relevant to the task signal while over-compressing layers that are critical for downstream behavior. We propose Task-Aware Quantization (TAQ), a training-free, weight-only mixed-precision PTQ framework that uses a small set of unlabeled task calibration prompts to allocate higher precision to task-relevant transformer layers under a fixed bit budget. TAQ estimates layer importance from hidden representations and output sensitivity, and we instantiate it with three scoring rules: TAQ-IS, based on activation information and stability; TAQ-KL, based on output-distribution sensitivity under a quantization-noise proxy; and TAQ-O, a label-informed oracle diagnostic for analyzing layer sensitivity. Across several benchmarks, TAQ outperforms task-agnostic baselines such in most settings, with especially strong gains in the accuracy--memory ratio. We further validate that these gains translate to real deployment behavior through hardware throughput and latency measurements, and analyze calibration robustness and residual-stream error propagation. Overall, TAQ turns mixed-precision PTQ from a model-centric compression step into a task-conditioned precision-allocation problem. A reference implementation is available at \includegraphics[height=1em]{imgs/github-mark.png}.

Amit LeVi, Raz Lapid, Rom Himelstein +3

1Technion—Israel Institute of Technology · 2Intuit · 3Ben-Gurion University of the Negev +1

Neural Network QuantizationMixed-Precision QuantizationLLM QuantizationWeight-Only QuantizationPost-Training QuantizationQuantization+2
AskPDF
Feb 21, 2026

MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM

Dynamic runtime latency and memory constraints necessitate flexible large language model (LLM) deployment, where an LLM can be inferred with various quantization precisions based on available computational resources. Recent work on such any-precision quantization either relies on hardware-inefficient vector quantization or induces additional scaling factors when switching between bit-widths. Meanwhile, existing post-training quantization (PTQ) methods calibrated for a fixed low precision show poor generalizability under runtime precision change. In this work, we attribute the source of poor generalization across bit-widths to a precision-dependent outlier migration phenomenon where the distribution of PTQ-sensitive tokens changes across precisions. Motivated by this observation, we propose MoBiQuant, a novel any-precision Mixture-of-Bits quantization framework that adjusts weight precision for flexible LLM inference based on token sensitivity. Specifically, we propose a many-in-one recursive residual quantization that can iteratively reconstruct higher-precision weights at runtime and mitigates outlier migration with a token-aware router to dynamically select the optimal inference precision of each token.Extensive experiments show that MoBiQuant matches or surpasses frontier single-precision PTQ while exhibiting strong elasticity, achieving significant memory savings and throughput gains of up to 1.34×1.34\times over state-of-the-art any-precision methods. Our code is available at https://github.com/SimWangArizona/MoBiQuant.

Dongwei Wang, Jinhee Kim, Seokho Han +8

University of Arizona · Panasonic AI Lab · Duke University +2

LLM QuantizationNeural Network QuantizationMixed-Precision QuantizationPost-Training Quantization
AskPDF