The Price of Precision: A Calibrated Cost Model for Mixed-Precision LLM Serving

Mukund Athmaram; Manoj Rao; Liang Ma

cs.LGOct 11, 2026

The Price of Precision: A Calibrated Cost Model for Mixed-Precision LLM Serving

Authors: Mukund Athmaram, Manoj Rao, Liang Ma

Organizations: Advanced Micro Devices Inc. (AMD) · Cornell University

Abstract

Choosing a mixed-precision configuration for large language model (LLM) serving trades speed against accuracy. Allocators that use average bit-width or a memory budget as a speed proxy can miss changes in the resource that limits execution. We argue that served speed can be limited by compute, weight loading, or key-value (KV) cache access, depending on the phase, batch size, and context length. A byte-based proxy does not capture all of these costs, so fewer bits do not always mean faster serving. To address this gap, we build a cost model that accounts for the binding resource and use it in two ways. First, as a predictor: given a configuration and operating point, it estimates serving latency using constants calibrated for the model, GPU, and serving stack. We evaluate its accuracy across three hardware classes. Second, as an allocator: we combine modeled time costs with measured per-tensor accuracy costs and use a dynamic program to select a configuration under a latency or accuracy budget. We compare its configurations against byte-aware and sensitivity-only allocation.

Figures & tables

Figure 1: Mean prefill latency across six workloads per platform. Configurations follow the same sampling rule and are numbered by predicted latency. Blue shows measurements and orange our predictions. Gray bands show

\pm 10\%

of measurements.

Figure 2: Mean decode latency per step over the same six workloads and configurations as Figure 1 , using the same numbering and

\pm 10\%

bands.

Figure 3: Prefill mean absolute percentage error over 36 held-out cases per platform.

Figure 5: MI355X prefill at batch 32, prompt length 256. All 9 cost-model, 8 byte-aware, and 9 sensitivity-only configurations are shown, with two shared by all three. Overlapping markers indicate shared configurations. Latency is the median of 12 requests. Appendix C gives both quality metrics and workloads.

Figure 6: Decode latency with fixed Q8/Q6/Q4 weights and an F16 KV cache. Circles average two runs on each of two held-out WikiText passages, and squares show frozen predictions. Lines connect all 16 contexts and gray bands show

\pm 10\%

of measurements.

Figure 7: Context-dependent allocation through 16K with a 60 ms planning target. Left: tensor precisions grouped by layer. Middle: steps exceeding 60 ms, with counts. Right: mean joint KL to Q8 (lower is better). Each policy includes 1,536 timings and 512 quality measurements from two runs on each of two passages.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: Allocator inputs and output. Offline accuracy measurements, calibrated time costs, supported formats, and the operating point determine a precision configuration for the chosen budget. Units with shared costs are allocated jointly. The menu is illustrative.

Panel of Figure 9 — Figure 9: Accuracy–time tradeoff. The dashed line represents equal weighted cost $\Delta A+\lambda\Delta S$ , where $\lambda>0$ sets the relative weight on time. The arrows show changes that lower or raise this cost.

Quantity	Equation term	MI355X dense	MI355X MoE	MI300A
BF16 throughput	$r_{\mathrm{ref}}$	1216.189	864.358	197.537
Quantized throughput	$r(p)$	10000.000 ∗	10000.000 ∗	746.438
Bandwidth	$\mathrm{BW}$	2523.518	9373.071	100000.000 ∗
Blend scale (rows)	$M_{1/2}$	0.001 ∗	2426.425	638.885
Prefill (ms/ $10^{6}$ pairs)	$10^{6}a_{\mathrm{prefill}}$	0.503	0.242	1.593
Decode (ms/1024 tokens)	$1024a_{\mathrm{decode}}$	0.108	0.052	0.034

Appendix

Table 1: Datacenter predictor parameters. The MoE column is the serving fit used for allocation. Throughputs are in TFLOP/s and bandwidth is in GB/s.

Platform	Contribution	Value
MI355X dense	$h$ : per quantized attention group	0.008538
	$h$ : per $10^{6}$ attention-input elements	0.003137
	$h$ : per quantized feed-forward group	0.043718
	$h$ : per $10^{6}$ feed-forward-input elements	0.004465
	$h$ : per mixed-precision layer	0.000466
	$f_{\mathrm{base}}$ : prefill	20.317529

Appendix

Table 2: Overhead and shared baseline coefficients in milliseconds. Group costs are charged once per executed group.

Quantity	Equation term	Value
F16 throughput	$r_{\mathrm{ref}}$	0.801
Q8 throughput	$r(p)$	1.335
Bandwidth	$\mathrm{BW}$	169.859
Blend scale (rows)	$M_{1/2}$	0.001 ∗
Prefill attention (per $10^{6}$ pairs)	$10^{6}a_{\mathrm{prefill}}$	239.493
Decode attention (per 1024 tokens)	$1024a_{\mathrm{decode}}$	2.608

Appendix

Table 3: Radeon 780M predictor calibration. Throughputs are in TFLOP/s, bandwidth is in GB/s, and time coefficients are in ms with the stated input scaling.

Format	$r(p)$	$h$ : attention	$h$ : FFN	$h$ : output
F16	0.570	0	0	0
Q4_K	0.959	0.0018	0.0192	0.2543
Q5_K	0.778	$\approx 0$	0.0831	$\approx 0$
Q6_K	0.602	0.0234	0.1264	0.8620
Q8_0	1.195	0.0379	0.1289	1.3763

Appendix

Table 4: Five-format Radeon 780M calibration for the static prefill comparison. Rates are in TFLOP/s and overheads are in ms per projection execution.

Figure 10: Dense MI355X validation over eight configurations and six held-out workloads. Configurations 1–6 match Figure 1 , with BF16 (Ref.) and MXFP4 (Low) projection controls. Points pool two reloads, with 12 requests per output count. Gray bands mark

\pm 10\%

of measured latency, and reported errors include both controls. Pairwise ordering is correct in 71% of prefill and 67% of decode comparisons, with most batch-16 decode orderings reversed despite low mean error.

Figure 11: MI355X dense-model prefill across six workloads for configurations 1, 3, and 6 from Figure 1 . Labels give batch size

\times

prompt length. Solid lines show measurements and dashed lines the resource-model predictions.

Figure 12: MoE configuration comparison on MI355X. Each point averages prefill latency at batch 32/prompt 256 and batch 8/prompt 1024. All 28 configurations are excluded from calibration. The predictor understates the precision-dependent latency range.

Figure 13: MI355X MoE validation: three mixed assignments and BF16 at six held-out workloads, with the same repeat counts and bands as Figure 10 .

Model	Predictor	Prefill	Decode
Dense	Our model	6.63%	6.80%
	Byte estimate	6.96%	5.96%
	Constant estimate	7.67%	5.96%
MoE	Our model	8.12%	3.43%
	Byte estimate	8.30%	9.69%
	Constant estimate	8.24%	11.04%

Appendix

Table 5: MI355X mean absolute percentage error. Dense uses eight configurations and MoE uses four, each at six workloads excluded from calibration. Each predictor uses the same calibration cases as its baselines.

Figure 14: MI300A validation over eight configurations and six workloads. Configurations 1–6 match Figure 1 , with reference/quantized projection controls. Points average two reloads, gray bands mark

\pm 10\%

, and errors include both controls.

Figure 15: Radeon 780M validation over eight configurations and six contexts, with Figure 14 ’s conventions. Points average three runs.

Platform	Predictor	Prefill	Decode
MI300A	Our model	6.11%	14.96%
	Byte estimate	10.95%	13.07%
	Constant estimate	26.83%	13.07%
Radeon 780M	Our model	7.87%	3.31%
	Byte estimate	9.68%	3.31%
	Constant estimate	14.36%	6.01%

Appendix

Table 6: Mean absolute percentage error over eight configurations and six workloads or contexts. Each platform uses one parameter set and the same calibration cases for all predictors.

Figure 16: MI300A kernel latency for two Qwen2.5-7B feed-forward projections. The dotted line marks the runtime’s switch above four token rows. Points average six timing groups and bars show their range. These isolated kernel measurements use synthetic inputs and repeated weight access, and are separate from the serving measurements.

Figure 17: Standard MoE inference and forced balanced routing on MI355X. Left: prediction error across six held-out workloads for four shared MoE configurations, with eight dense-model configurations for reference. Whiskers span the pooled result and both individual model loads. Right: error in the predicted MXFP4–BF16 prefill latency difference during calibration. Dense and MoE tests use different architectures and assignments, so their error difference does not isolate routing.

Figure 18: MI355X MoE prefill calibration. Each point is MXFP4 latency minus BF16 latency, averaged across two model loads with six requests each. Negative values mean MXFP4 is faster.

Figure 19: Estimated and measured routing on MI355X. Left: active experts averaged over layers, configurations, and capture runs. Right: prediction error using our approximation, measured active counts with evenly divided work, or measured tokens per expert. All three use the same fitted parameters and latency measurements.

Figure 20: MI355X MoE allocation costs. Left: time saved by quantizing one group, averaged across layers and normalized by BF16 latency. Negative values indicate slower execution. Right: groups assigned different precisions at matching positions in each phase’s budget range. Each range runs from the minimum predicted latency (0%) to the predicted BF16 latency (100%), so the same percentage can represent different budgets in milliseconds. At 100%, the two-group difference comes from conservative grid rounding. These are model outputs at different phases and workloads.

Figure 21: MI355X MoE prefill allocation at batch 32, context 256: $81.09$ ms predicted under an $81.14$ ms budget. Of 144 groups, 102 use BF16 and 42 use MXFP4. Q/K/V projections are grouped, as are gate/up and down projections across experts. Head and routers remain BF16. The heatmap shows choices, not measured performance.

Figure 22: MI355X prefill allocations with both quality metrics, measured at two workloads. Every prefill selection is shown. Latency is the median of 12 requests per workload. Points lower and farther left are better. Magenta stars mark BF16.

Figure 23: MI355X decode allocations, with the identical cost-based and byte-aware choices shown with overlapping markers. All decode selections are evaluated at both workloads. Decode step latency is the difference between median 33-output and one-output request times, divided by 32. Magenta stars mark BF16.

Figure 24: MI300A at batch 1, prompt 2048. Prefill is mean one-output request time and decode is the slope across output counts 1, 32, 64, and 128, pooled over two loads. Quality uses the final 32 positions of 16 common 64-token WikiText sequences. Joint KL is the mean

D_{\mathrm{KL}}(\mathrm{BF16}\|\mathrm{candidate})

over these 512 positions and the full vocabulary. Stars and diamonds mark BF16 and FP8.

Figure 25: Laptop prefill sweep at batch 1 and prompt length 2048, with perplexity and joint KL. Every generated configuration is shown. Latency is the median of 18 prefills across three common passages. Quality uses 128 continuation tokens per passage. Teal diamonds are uniform formats and the magenta star is F16. All five weight formats remain resident during measurement.

Figure 26: Per-tensor accuracy impact (mean token KL estimate versus BF16 at MXFP4, weights and activations, WikiText ( Merity et al., 2017 ) ) on Qwen3-Coder-Next ( Qwen Team, 2026 ) . Rows identify module types and columns decoder layers. The first third has $4.83\times$ the mean KL of the final third, and the per-tensor depth–KL Spearman correlation is $-0.89$ across 492 tensors. Gray denotes an absent tensor.

Quantity	Equation term	Value
Reference throughput	$r_{\mathrm{ref}}$	0.734
Q8 throughput	$r(p)$	1.315
Q6 throughput	$r(p)$	0.897
Q4 throughput	$r(p)$	1.254
Bandwidth	$\mathrm{BW}$	65.580
Blend scale (rows)	$M_{1/2}$	0.001 ∗

Appendix

Table 7: Calibration for the 16K schedule with the paired objective. Throughput is in TFLOP/s, bandwidth is in GB/s, and time coefficients are in ms. Overhead triples are for attention, feed-forward, and output projections.

Policy	Mean (ms)	P99 (ms)	At most 60 ms (%)	KL ( $10^{-3}$ )	PPL
Cost schedule	39.22	52.36	99.93	12.135	10.438
Byte schedule	38.63	53.12	99.93	16.890	10.409
Fixed Q8	45.22	64.03	95.64	0.000	10.365
Fixed Q6	37.58	60.49	98.83	7.528	10.436
Fixed Q4	30.04	49.34	100.00	93.452	11.814

Appendix

Table 8: Repeated five-policy comparison through 16K at a 60 ms planning target. Each policy contributes 1,536 timings and 512 quality measurements from two runs on each of two passages. Latency includes precision changes.

Figure 27: Additive versus joint KL: 145 Qwen3-Coder-Next ( Qwen Team, 2026 ) configurations and 30 Qwen3-Coder-30B-A3B-Instruct ( Qwen Team, 2025 ) configurations. Gray lines show equality and orange lines median ratios. The 30B units group Q/K/V and expert projections.

Figure 28: Cache-quantization error on Qwen2.5-Coder-7B-Instruct ( Hui et al., 2024 ) across context lengths and formats. The plotted diagnostic is the reference top-256 KL contribution, averaged over four slices at each length. Error generally grows with context and is larger at lower cache precision. Both axes use log scales.

	Prefill		Decode
Platform	Our model	Byte estimate	Our model	Byte estimate
MI355X	6.62	6.84	6.37	5.37
MI300A	5.87	8.72	13.92	11.89
Radeon 780M	7.71	9.44	3.70	3.70
All three platforms	6.73	8.33	8.00	6.99

Appendix

Table 9: Mean absolute percentage error (%). Each platform and phase has 36 cases: six held-out configurations at six workloads. Errors are computed before averaging across workloads.

LLM Serving

Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.

See all 207 papers

Context-Aware Precision Scheduling for Small Language Models
Oct 11, 2026Mukund Athmaram, Manoj Rao, Liang MaEfficient Language Model Inference On-Device Language Model Inference
Advanced Micro Devices Inc. (AMD) · Cornell University
GAMMA: Global Bit Allocation for Mixed-Precision Models under Arbitrary Budgets
May 18, 2026Zhangyang Yao, Haiyan Zhao, Haoyu Wang +3Mixed-Precision Quantization LLM Quantization
Beihang University · Tsinghua University · ByteDance Inc
Profiling Lightweight Large Language Models
Jul 23, 2026Tomohiro Harada, Enrique Alba, Gabriel LuqueLLM Inference Efficiency LLM Evaluation
Graduate School of Science and Engineering, Saitama University, Saitama, Japan

Oct 11, 2026J/K move · Enter open · S save

Context-Aware Precision Scheduling for Small Language Models

Mukund Athmaram, Manoj Rao, Liang Ma

Advanced Micro Devices Inc. (AMD) · Cornell University

Efficient Language Model Inference On-Device Language Model Inference Small Language Models Mixed-Precision Quantization

Open paper AskPDF arXiv

Small language models (SLMs) used as on-device agents must balance accuracy and latency as their context grows. Mixed precision helps manage this tradeoff by assigning different precisions to weight tensors according to their size and sensitivity. However, these choices depend on the operating conditions, so a configuration selected for a short context may no longer provide the best tradeoff at a longer one. We calibrate a timing model from measurements at different precisions and context lengths, and combine it with tensor sensitivities to choose configurations under a latency or accuracy budget. We argue that fewer bits do not always mean faster execution, so our predictor estimates latency from calibrated execution costs rather than weight bytes alone. Under a latency budget, reading the larger key-value (KV) cache leaves less time for weight operations, leading the allocator to select lower precisions. This gives a schedule of tensor changes that can be prepared before execution. Our runtime follows the schedule while retaining the session's KV cache. We evaluate executed schedules on a laptop GPU, including a longer-context schedule with three weight formats and a tool-use benchmark measuring full response time and correctness. Under an accuracy budget, measured error from a quantized KV cache determines the budget left for weights. The allocator can raise or lower weight precision as this estimate changes. These experiments show where context-aware precision helps and where accuracy recovery remains limited.

Abstract

LLM Serving

Explore similar work

Context-Aware Precision Scheduling for Small Language Models

GAMMA: Global Bit Allocation for Mixed-Precision Models under Arbitrary Budgets

Profiling Lightweight Large Language Models