Context-Aware Precision Scheduling for Small Language Models

Mukund Athmaram; Manoj Rao; Liang Ma

cs.LGOct 11, 2026

Context-Aware Precision Scheduling for Small Language Models

Authors: Mukund Athmaram, Manoj Rao, Liang Ma

Organizations: Advanced Micro Devices Inc. (AMD) · Cornell University

Abstract

Small language models (SLMs) used as on-device agents must balance accuracy and latency as their context grows. Mixed precision helps manage this tradeoff by assigning different precisions to weight tensors according to their size and sensitivity. However, these choices depend on the operating conditions, so a configuration selected for a short context may no longer provide the best tradeoff at a longer one. We calibrate a timing model from measurements at different precisions and context lengths, and combine it with tensor sensitivities to choose configurations under a latency or accuracy budget. We argue that fewer bits do not always mean faster execution, so our predictor estimates latency from calibrated execution costs rather than weight bytes alone. Under a latency budget, reading the larger key-value (KV) cache leaves less time for weight operations, leading the allocator to select lower precisions. This gives a schedule of tensor changes that can be prepared before execution. Our runtime follows the schedule while retaining the session's KV cache. We evaluate executed schedules on a laptop GPU, including a longer-context schedule with three weight formats and a tool-use benchmark measuring full response time and correctness. Under an accuracy budget, measured error from a quantized KV cache determines the budget left for weights. The allocator can raise or lower weight precision as this estimate changes. These experiments show where context-aware precision helps and where accuracy recovery remains limited.

Figures & tables

Figure 1: Llama-3.2-1B-Instruct ( Meta, 2024 ) , Radeon 780M. Left: Q8/Q4 prefill latency, with whiskers spanning three passage means. Right: measured (solid) and predicted (dashed) latency for six F16/Q8 configurations excluded from calibration. Gray bands mark

\pm 10\%

of measurement, and bars show percentage error. Appendix G gives protocols and decode results.

Figure 2: Context scheduling on Llama-3.2-1B-Instruct ( Meta, 2024 ) , Radeon 780M. Top: tensor schedule, mean decode latency, and joint KL at matching contexts. Bottom: means over the same 17 context lengths. Fixed precision is the F16/Q8 mix selected during calibration. Timing includes switching, and Appendix A includes all comparisons and perplexity.

Figure 3: Context scheduling through 16K. Top left: the tensor schedule. Right: mean latency and joint KL to Q8 at matching contexts in two evaluation passages. The dashed line is the 77 ms target selected from calibration. Bottom: mean KL versus the fraction of decode steps exceeding the budget, over the same 16 context lengths. Fixed Q6 remains unchanged throughout. All six policies appear in Appendix B .

Figure 4: Executed accuracy-budget scheduling with a Q4 KV cache. Top left: tensor precisions chosen by our allocator. Right: mean decode latency and joint KL to Q8 weights with an F16 cache at matching contexts, averaged over two passages starting at 512 tokens; the dashed line is the accuracy budget. Bottom: means over all six context lengths. Fixed initial mix keeps our first configuration throughout. Appendix F also includes Q8 weights with the quantized cache and the F16-cache reference.

Figure 5: Tool-use error versus median complete-turn response time at the 85 ms decode budget. Lower and farther left is better. The arrow shows our response-time improvement over Q8 at matching correctness. All policies use the same model and runtime.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Quantity	Term	Value
Reference intercept (ms)	$a$	46.94
Context growth (ms per 1024 tokens)	$c$	1.738
Attention change at Q8 (ms)	$\Delta T_{\mathrm{attention}}(\mathrm{Q8})$	$-1.26$
Feed-forward change at Q8 (ms)	$\Delta T_{\mathrm{FFN}}(\mathrm{Q8})$	$-6.29$
Output change at Q8 (ms)	$\Delta T_{\mathrm{output}}(\mathrm{Q8})$	$+0.66$
Total reserve, including switching (ms)		4.06

Appendix

Table 1: 4K timing calibration, using contexts 256, 2048, and 4096. Group changes are relative to F16.

Figure 6: All seven policies at the same 17 contexts. The dashed line marks 50 ms. Bottom: mean latency versus mean joint KL or perplexity over the same passage; lower and farther left is better. Joint KL compares with F16; perplexity is the exponential of mean negative log-likelihood over all evaluated tokens.

Tensors changed	Median switch	First token: initial	First token: later
27	1.68	36.58	22.50
44	1.60	23.15	22.19

Appendix

Table 2: Additional switching check across three program runs. Times are ms. Median switch summarizes repetitions after the first in each run. The last two columns give maximum first-token latencies, including switching, for the first and later repetitions.

Group	Q6	Q4
Attention	$-1.485$	$-3.350$
Feed-forward	$-4.462$	$-12.470$
Embedding/output	$-3.306$	$-2.462$
Reference intercept $a$ (ms)		41.474
Context growth $c$ (ms per 1024 tokens)		3.045
Prediction-error allowance (ms)		4.384

Appendix

Table 3: 16K calibration. Weight-group time changes

\Delta T_{g}(p)

are relative to Q8, in ms. The reference curve is

T_{0}(L)=a+cL/1024

Policy	Mean	P99	Joint KL	Perplexity	Within target (%)
Cost schedule	58.12	72.86	0.00574	10.398	99.7
Byte-aware	53.81	76.35	0.00662	10.447	99.0
Fixed final mix	45.32	70.66	0.03785	10.946	100.0
Fixed Q8	62.16	85.18	0.00000	10.365	83.1
Fixed Q6	55.58	82.56	0.00753	10.436	90.9
Fixed Q4	39.49	64.50	0.09345	11.814	100.0

Appendix

Table 4: 16K results for all policies. Latencies are ms; P99 is the 99th-percentile decode latency. The final column gives the percentage of steps within 77 ms.

Figure 7: Left: the percentage of decode steps exceeding 77 ms at each context. Right: mean joint KL versus the overall percentage exceeding that target. Lower and left is better.

Policy	Correct turns	Median response (s)	Tokens within 85 ms (%)
Cost schedule	13/19	6.74	99.2
Byte-aware	13/19	6.86	99.4
Fixed Q8	13/19	7.52	59.8
Fixed Q4	12/19	4.74	100.0

Appendix

Table 5: Tool-use results at the 85 ms decode target. Response time is the median for a complete turn.

Quantity	Value
Reference intercept $a$ (ms)	78.036
Context growth $c$ (ms per 1024 tokens)	2.751
Layers 1–6: time change (ms)	$-4.613$
Layers 7–12: time change (ms)	$-5.606$
Layers 13–18: time change (ms)	$-4.974$
Layers 19–24: time change (ms)	$-5.050$

Appendix

Table 6: Tool-use timing calibration. Group changes

\Delta T_{g}(\mathrm{Q4})

are relative to Q8. The reference curve is

T_{0}(L)=a+cL/1024

. Configurations are measured at contexts 512/4096, with uniform Q8/Q4 also measured at 8192; each run has three warmup and 24 measured steps.

Decode target (ms)	Policy	P95 response (s)	Tokens within target (%)
95	Cost schedule	44.40	99.96
95	Byte-aware	26.20	99.37
95	Fixed Q8	28.17	99.84
95	Fixed Q4	24.16	100.00
85	Cost schedule	26.21	99.20
85	Byte-aware	26.08	99.44

Appendix

Table 7: Slow response times and the percentage of tokens within each decode target. P95 is the 95th-percentile complete-turn response time. Figure 8 gives median response time and correctness for the same turns.

Figure 8: Tool-use error versus median response time under each decode budget. Every point includes the same 19 turns, including errors and slow responses. Lower and left is better.

Panel of Figure 9 — Figure 9: Allocator inputs and output. Measured accuracy and timing changes, supported formats, and context length determine a configuration for the chosen budget. For an accuracy budget, measured cache error sets the remaining weight-error allowance. We solve these choices offline to form the schedule. Available formats depend on the experiment.

Panel of Figure 10 — Figure 10: Accuracy–time tradeoff. The dashed line represents equal weighted cost $\Delta A+\lambda\Delta T$ , where $\lambda>0$ sets the relative weight on time. The arrows show changes that lower or raise this cost.

Figure 11: Additive per-tensor accuracy cost versus measured joint KL for 27 fixed configurations, each counted once. Gray shows equality; orange shows the median joint-to-additive ratio. The two measures rank configurations similarly but differ in absolute scale.

Context	Cache KL	Weight KL budget	Q8 (%)	Q6 (%)	Q4 (%)
512	0.0325	0.0175	1.4	35.6	63.0
2048	0.0308	0.0192	1.0	29.1	69.9
4096	0.0755	-0.0255	100.0	0.0	0.0
8192	0.1355	-0.0855	100.0	0.0	0.0
12288	0.0366	0.0134	3.0	49.4	47.6
16384	0.0361	0.0139	3.0	46.5	50.5

Appendix

Table 8: Cache-error calibration and the resulting schedule. Negative remaining budget means that even Q8 weights cannot meet the additive constraint. Weight shares count parameters.

Group	Q6	Q4
Attention weights	$-0.147$	$-4.296$
Feed-forward weights	$-6.986$	$-11.610$
Embedding/output	$+0.265$	$-3.151$

Appendix

Table 9: Time changes

\Delta T_{g}(p)

with the Q4 cache, relative to Q8 weights. Negative values predict faster execution. All entries are ms.

Policy	Mean ms	Joint KL	Perplexity	Within budget
Cost schedule	35.10	0.0923	12.958	3/12
Byte-aware	33.59	0.0947	13.297	2/12
Fixed initial mix	28.74	0.1065	12.896	1/12
Q8, Q4 cache	44.45	0.0404	12.025	9/12
Q8, F16 cache	–	0.0000	11.199	12/12

Appendix

Table 10: Accuracy-budget results. The last column counts measurements with mean joint KL at most 0.05 across six context lengths in each of two passages. Mean latency is in ms.

Figure 12: Accuracy-budget comparisons at matching contexts. Left: our schedule and its fixed initial configuration. Right: our schedule and Q8 weights with the same Q4 cache. The dashed line marks the KL budget; the F16-cache reference has zero KL by definition.

Process	Phase	Context	$T_{0}$	Attention	FFN	Output
1	prefill	512	476.455	47.044	556.637	-2.066
1	prefill	2048	2211.277	110.744	2176.390	65.986
1	decode	512	31.444	4.679	6.572	-2.569
1	decode	2048	33.461	1.620	6.536	1.109
2	prefill	512	447.866	39.177	593.509	-15.915
2	prefill	2048	2124.196	92.931	2237.679	120.492

Appendix

Table 11: Predictor calibration in ms. Columns give reference time

T_{0}

and group changes

\Delta T_{g}(\mathrm{F16})

relative to Q8. Each program run is calibrated before its evaluation.

Figure 13: Prediction at both calibrated prompt lengths, with configurations numbered as in Figure 1 . Solid lines are measured, dashed lines predicted, and gray bands mark

\pm 10\%

of measurement. Gray bars give mean absolute percentage error.

Phase	Calibrated costs	Byte estimate	Constant estimate
Prefill	3.51	5.92	17.62
Decode	2.71	2.24	5.16

Appendix

Table 12: Mean absolute percentage error (%) across all evaluation configurations and both prompt lengths. All predictors use the same four calibration controls.

Efficient Language Model Inference

Efficient language model inference reduces the compute, memory, latency, and energy needed to generate useful outputs.

See all 314 papers

The Price of Precision: A Calibrated Cost Model for Mixed-Precision LLM Serving
Oct 11, 2026Mukund Athmaram, Manoj Rao, Liang MaLLM Serving LLM Inference Acceleration
Advanced Micro Devices Inc. (AMD) · Cornell University
Profiling Lightweight Large Language Models
Jul 23, 2026Tomohiro Harada, Enrique Alba, Gabriel LuqueLLM Inference Efficiency LLM Evaluation
Graduate School of Science and Engineering, Saitama University, Saitama, Japan
StagQ: Constraint-Driven Multi-Precision Weight Quantization for LLMs
Oct 5, 2026Zhe Wei, Mengqi Guo, Yuan Yuan +3Mixed-Precision Quantization LLM Quantization
Huawei Technologies Ltd.

Oct 11, 2026J/K move · Enter open · S save

The Price of Precision: A Calibrated Cost Model for Mixed-Precision LLM Serving

Mukund Athmaram, Manoj Rao, Liang Ma

Advanced Micro Devices Inc. (AMD) · Cornell University

LLM Serving LLM Inference Acceleration Cost-Aware Inference Mixed-Precision Quantization

Open paper AskPDF arXiv

Choosing a mixed-precision configuration for large language model (LLM) serving trades speed against accuracy. Allocators that use average bit-width or a memory budget as a speed proxy can miss changes in the resource that limits execution. We argue that served speed can be limited by compute, weight loading, or key-value (KV) cache access, depending on the phase, batch size, and context length. A byte-based proxy does not capture all of these costs, so fewer bits do not always mean faster serving. To address this gap, we build a cost model that accounts for the binding resource and use it in two ways. First, as a predictor: given a configuration and operating point, it estimates serving latency using constants calibrated for the model, GPU, and serving stack. We evaluate its accuracy across three hardware classes. Second, as an allocator: we combine modeled time costs with measured per-tensor accuracy costs and use a dynamic program to select a configuration under a latency or accuracy budget. We compare its configurations against byte-aware and sensitivity-only allocation.

Abstract

Efficient Language Model Inference

Explore similar work

The Price of Precision: A Calibrated Cost Model for Mixed-Precision LLM Serving

Profiling Lightweight Large Language Models

StagQ: Constraint-Driven Multi-Precision Weight Quantization for LLMs