Dynamic Mixed-Precision Routing for Efficient Multi-step LLM Interaction
Authors: Yuanzhe Li, Jianing Deng, Jingtong Hu, Tianlong Chen, Song Wang, Huanrui Yang
Organizations: University of Arizona, Tucson, AZ, USA · University of Pittsburgh, Pittsburgh, PA, USA · University of North Carolina at Chapel Hill, Chapel Hill, NC, USA · University of Central Florida, Orlando, FL, USA
Large language models (LLMs) achieve strong performance in long-horizon agentic tasks through multi-step interaction and reasoning at test time. However, repeatedly invoking these models incurs substantial inference cost over an episode. Post-training quantization (PTQ) reduces this cost, but using a fixed low precision throughout an episode can sharply lower task success. We observe that interaction steps differ sharply in their sensitivity to quantization: given the same context, decoding a step at low precision turns a success into a failure at only about 1% of the steps, which we call critical steps. Based on this observation, we propose Dynamic Mixed-Precision Routing (DMR), a framework that adaptively selects between high-precision and low-precision versions of the same LLM at each decision step. Leveraging an entire-space factorization to jointly predict a step's divergence and its effect on the outcome, DMR learns to recognize these critical steps by training only a lightweight router on the model's hidden state. By reusing a shared high-precision KV cache and the hidden states, DMR avoids redundant context prefill and a separate text encoder, enabling precision switching with less than 1 ms of overhead per step and adding less than 0.11% to step latency for routing. This mixed-precision design also enables nested quantization, reducing the memory footprint by 39% on average compared with FP16. Across three long-horizon agentic benchmarks with nested quantization, DMR achieves an average 2.44x decode speedup over FP16 at comparable task success rates, without fine-tuning the underlying model. On ALFWorld with GPTQ, DMR achieves a 1.71x decode speedup with 86.9% success, compared with 87.2% for FP16.
Figures & tables
Figure 1: Overview of DMR . (a) Precision tiers of the same LLM share a high-precision KV cache, enabling step-level routing without redundant context prefill. Low-precision responses are re-prefilled at high precision in subsequent steps (§ 3 ). (b) The router predicts step criticality through an entire-space factorization of action divergence and conditional criticality, using hidden-state features and counterfactual supervision (§ 4.2 ). (c) Iterative label collection combines agent rollouts, paired decoding, and counterfactual branching to train the router across rounds (§ 4.3 ).
Figure 2: One ScienceWorld episode (life-cycle task, a training task of the router). Each cell is one step, colored by the precision that decoded it. At step 8, the W3 output skips the adult stage and the task fails at once, while the W4 output names it and the task succeeds two steps later. Only a few steps need W4.
Figure 3: Critical steps are rare: GPTQ W3 vs. FP16 on 540 ALFWorld training games.
ScienceWorld
TextWorld
ALFWorld
Average
Method
Succ.
Speedup
W3
Succ.
Speedup
W3
Succ.
Speedup
W3
Recov.
Speedup
Fixed
FP16
71.6
1.00 ×
–
55.0
1.00 ×
–
87.2
1.00 ×
–
–
1.00 ×
Top tier Precision
71.1
2.65 ×
0%
56.0
1.62 ×
0%
88.3
1.49 ×
0%
100%
1.92 ×
W3
30.3
1.54 ×
100%
2.0
2.48 ×
100%
29.6
1.24 ×
100%
–
1.75 ×
Top prefill
Top → W3
60.7
3.20 ×
100%
44.0
2.29 ×
100%
79.2
2.05 ×
100%
0%
2.51 ×
Routing
Random
63.8
3.12 ×
77%
51.7
1.68 ×
66%
82.0
1.82 ×
75%
42%
2.21 ×
Table 1: Main results with nested quantization. We use ER1-7B with {W3,W4} on ScienceWorld and Qwen3-4B with {W3,W8} on TextWorld and ALFWorld. Succ. denotes task success rate (%), Speedup is episode decode speedup relative to FP16 averaged over test tasks, W3 denotes the share of tokens decoded at W3, and Recov. measures the fraction of the success gap between Top → W3 and the top tier recovered by each method. Fixed-precision baselines prefill at their own tier, while all other methods use top-tier prefill. Random results are averaged over three seeds. Bold and underlined values indicate the best and second-best routing methods, respectively.
Figure 4: Success rate versus episode decode speedup relative to the top precision tier. (a) ScienceWorld with ER1-7B and Any-Precision {W3,W4} ; (b) TextWorld cooking with Qwen3-4B and Any-Precision {W3,W8} ; (c) ALFWorld with Qwen3-4B and GPTQ {W3,FP16} . DMR lies on the success–speedup frontier in all three settings. Dashed lines indicate top-tier success rates, dotted lines indicate per-episode mixtures of the top tier and W3, and squares mark the top-tier and Top → W3 baselines.
Method
Succ.
Speedup
W3
FP16
87.2
1.00 ×
0%
W3
52.6
1.75 ×
100%
FP16 → W3
75.5
1.75 ×
100%
Random
78.8
1.46 ×
79%
Step schedule
k=2
85.0
1.64 ×
75%
Table 2: Independent PTQ on ALFWorld (GPTQ, {W3,FP16} , FP16 prefill for routing methods). Columns as in Table 1 .
Figure 5: Additional episode-level prefill time relative to top-tier-only execution. We compare DMR with a shared KV cache against separate-model routing with either one cache per model or context re-prefilling upon switching. All configurations follow DMR ’s routing decisions on the same test trajectories (Appendix E ).
Deployment
4B
7B
FP16 only
11.4
16.0
Separate FP16 and W3 models †
17.3
22.3
DMR , independent {W3,FP16} †
12.7
18.4
DMR , nested {W3,W4}
6.4
7.1
DMR , nested {W3,W8}
8.8
–
Table 3: Peak GPU memory usage (GB) across deployment configurations at batch size one, including the CUDA context and full-context KV caches (8292 tokens for 4B and 4296 for 7B). † Computed from the measured FP16 deployment: plus the 3-bit weights (independent), or plus a second W3 deployment with its own KV cache (separate).
Per step
Separate models
DMR
Switching
re-prefill, 218–324 ms
0.26–0.40 ms
Router input
encoder, 48–61 ms
probe, 0.19–0.31 ms
Table 4: Per-step overhead on an RTX A6000. Re-prefill latency is measured at the agents’ average context length of 2k tokens. Encoder latency uses Qwen3-Embedding-0.6B, as in MTRouter Zhang et al. (2026b) , averaged over 1,000 steps per benchmark from the test trajectories. Probe latency includes GPU-to-CPU hidden-state transfer.
Figure 6: Cumulative distribution of critical training states across environment steps for ALFWorld, TextWorld, and ScienceWorld. The curves indicate the fraction of critical states covered by allocating top-tier decoding to the first k steps. Dotted lines mark k=2,4,6 .
Figure 7: False negatives on ScienceWorld under matched high-precision budgets. We report the FN rate, i.e., the fraction of 80 critical training steps decoded at W3 (lower is better), by step schedules, the position-feature router, and DMR . Each budget matches the W4 step share of the corresponding step schedule ( k=2,4,6 ), while both routers independently select which steps use W4.
ALF GPTQ
ALF AP
SciWorld AP
Structure
AUC
FN
AUC
FN
AUC
FN
Mixed label
0.511
20.7
0.535
28.3
0.448
27.3
Criticality only
0.731
10.7
0.623
20.3
0.779
11.3
Separate heads
0.730
7.7
0.612
11.0
0.709
10.0
Entire space ( DMR )
0.738
7.0
0.627
11.3
0.745
8.7
Table 5: Offline comparison of router factorizations. AUC measures discrimination between critical and harmless divergences among states with counterfactual branches. FN counts critical states routed to W3 at a fixed W3 call share. Values are means over three cross-validation splits. Bold and underlined values indicate the best and second-best results in each column.
Method
Choice made per
One model
Shared KV
Decided by
Any-Precision, AnyBCQ, MatQuant
request
✓
–
requested bit width
MorphServe
load period
✓
✓
server load
PMPD
decode phase
✓
✓
decoding progress
DP-LLM
token × layer
✓
✓
layer error
QSpec
draft / verify
✓
✓
fixed roles
R2R
token
×
×
token divergence
Table 6: Methods that choose a precision or a model for LLM inference. One model : the candidates are precisions of a single model. Shared KV : a newly chosen candidate decodes from the existing cache without re-prefilling (–: no switch within a request). Decided by : what the choice depends on.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
ALFWorld
SciWorld
TextWorld
Agent model
Qwen3-4B
ER1-7B
Qwen3-4B
Prompt format
ReAct, 2-shot
Thought/Action
ReAct, 1-shot
Max tokens / calls
100 / 50
200 / 30
100 / 50
Context limit
8192
4096
8192
Test tasks
140 + 134
211
100
Steps per episode
22.2
13.9
32.9
Appendix
Table A.1: Benchmarks, protocols and per-step workload. Workload rows are measured on the FP16 agents’ test trajectories; prompt and response lengths are in tokens, averaged over all steps.
States
Divergent
Critical
ALFWorld GPTQ {W3,FP16}
13,450
2,049
143 (1.1%)
ALFWorld AP {W3,W8}
13,232
2,162
134 (1.0%)
ScienceWorld AP {W3,W4}
13,358
1,747
80 (0.6%)
TextWorld AP {W3,W8}
18,032
3,016
408 (2.3%)
Appendix
Table B.1: Round-1 counterfactual labels (training split). Critical: the low-tier branch fails while the top-tier branch succeeds.
Router
τ
Succ.
Speedup
W3
FP16
–
87.2
1.00 ×
0%
Round 0
0.5
84.3
1.27 ×
56%
0.7
86.1
1.28 ×
46%
0.9
86.5
1.22 ×
34%
DMR
0.95
86.9
1.71 ×
78%
Appendix
Table C.1: Divergence-only routing versus DMR on ALFWorld (GPTQ {W3,FP16} , FP16 prefill). Speedup is the mean decode time per task of FP16 divided by that of each router; W3 is the fraction of tokens decoded at W3. Thresholds apply to different scores: 1−pD for Round 0 and 1−pDpK for DMR .
ALFWorld, GPTQ {W3,FP16}
ScienceWorld, AP {W3,W4}
τ
W3
Succ.
Dec.
τ
W3
Succ.
Dec.
0.90
92
84.7
2.49
0.99
88
65.9
3.52
0.93
84
83.9
2.75
0.995
83
67.8
3.47
0.95
78
86.9
2.86
0.997
77
68.2
3.39
0.96
76
86.1
3.05
0.998
73
67.8
3.41
0.97
71
86.9
3.15
0.9995
58
67.8
3.45
Appendix
Table C.2: Results of DMR at different thresholds τ on the test tasks, ordered by W3 token share. W3: W3 token share (%). Succ.: success rate (%). Dec.: decode seconds per task. Bold: the threshold of the main results. Heuristic baselines are those of Table 2 (ALFWorld) and Table 1 (ScienceWorld).
Batch size
1
2
4
8
ALFWorld, over W8
1.40 ×
1.31 ×
1.25 ×
1.10 ×
ALFWorld, over FP16
2.09 ×
1.83 ×
1.61 ×
1.24 ×
ScienceWorld, over W4
1.23 ×
1.20 ×
1.13 ×
1.09 ×
ScienceWorld, over FP16
3.25 ×
2.88 ×
2.50 ×
1.66 ×
Appendix
Table D.1: Decode speedup of DMR versus batch size, TPOT measured at each batch size.
Figure E.1: Routing between separate models, for contrast with DMR (Figure 1 a): an external encoder re-reads the context at every step, and a model that was idle in the previous step first prefills the history it missed.
Cache policy
Succ.
Speedup
W3
Prefill (ms)
Low-tier reuse
DMR (re-prefill)
68.2
3.25 ×
77%
87
0%
Mixed (keep low-tier KV)
61.6
3.08 ×
76%
85
14%
Appendix
Table F.1: Cross-step KV-cache policies on ScienceWorld (Any-Precision {W3,W4} ; router and threshold as in Table 1 ). Speedup is episode decode speedup over FP16. Prefill is measured top-tier prefill latency per call. Low-tier reuse is the fraction of reused KV-cache tokens originally computed at W3.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
LLM agents have recently emerged as a powerful paradigm for solving complex tasks through planning, tool use, memory retrieval, and multi-step interaction. However, these agentic workflows often introduce substantial input-side overhead, making the compute-intensive prefilling stage a key bottleneck in long-context, multi-turn inference. In this work, we propose Mix-Quant, a simple and effective phase-aware quantization framework for fast agentic inference. We first investigate FP4 quantization in agentic LLM workflows and observe that quantizing the entire inference process can incur significant performance degradation. In contrast, the prefilling stage exhibits substantial quantization redundancy and can therefore be quantized with minimal accuracy loss, despite being the dominant source of computation. Based on this insight, we apply high-throughput NVFP4 quantization to the prefilling phase while preserving BF16 precision for decoding. By decoupling prefilling acceleration from decoding quality, Mix-Quant combines phase-aware algorithmic quantization with hardware-efficient NVFP4 execution to alleviate the inference bottleneck in LLM agents. Extensive experiments across long-context and agentic benchmarks demonstrate that Mix-Quant largely preserves task performance while delivering significant efficiency improvements, achieving up to a 3x speedup during prefilling.
Quantization-aware training (QAT) leverages lower-precision arithmetic to reduce the cost of LLM deployment, but aggressive quantization degrades final model performance. A common remedy is mixed-precision training, in which high precision is assigned to some of the layers to maintain performance while keeping the cost constrained. This approach then requires precision assignments for model layers during training. We provide a new approach, called Q-PACE, consisting of a second-order sensitivity model that predicts the loss increase as a sum of quantization noise MSE weighted by per-layer curvature coefficients. During training, we periodically re-compute these coefficients using perturbations across layers, and re-assign precision. Pretraining and supervised fine-tuning experiments on LLMs of up to 4B parameters show that Q-PACE consistently improves over existing mixed-precision training recipes, and achieves comparable loss at substantially lower total memory budgets. We further find that quantization sensitivity is highly predictable by depth and layer type, and its stability during training allows for infrequent, cheap recalibration.
Alexandra Volkova, Matin Ansaripour, Erik Schultheis +2
Dynamic runtime latency and memory constraints necessitate flexible large language model (LLM) deployment, where an LLM can be inferred with various quantization precisions based on available computational resources. Recent work on such any-precision quantization either relies on hardware-inefficient vector quantization or induces additional scaling factors when switching between bit-widths. Meanwhile, existing post-training quantization (PTQ) methods calibrated for a fixed low precision show poor generalizability under runtime precision change. In this work, we attribute the source of poor generalization across bit-widths to a precision-dependent outlier migration phenomenon where the distribution of PTQ-sensitive tokens changes across precisions. Motivated by this observation, we propose MoBiQuant, a novel any-precision Mixture-of-Bits quantization framework that adjusts weight precision for flexible LLM inference based on token sensitivity. Specifically, we propose a many-in-one recursive residual quantization that can iteratively reconstruct higher-precision weights at runtime and mitigates outlier migration with a token-aware router to dynamically select the optimal inference precision of each token.Extensive experiments show that MoBiQuant matches or surpasses frontier single-precision PTQ while exhibiting strong elasticity, achieving significant memory savings and throughput gains of up to 1.34× over state-of-the-art any-precision methods. Our code is available at https://github.com/SimWangArizona/MoBiQuant.
Dongwei Wang, Jinhee Kim, Seokho Han +8
University of Arizona · Panasonic AI Lab · Duke University +2