Dynamic Mixed-Precision Routing for Efficient Multi-step LLM Interaction
Authors: Yuanzhe Li, Jianing Deng, Jingtong Hu, Tianlong Chen, Song Wang, Huanrui Yang
Organizations: University of Arizona, Tucson, AZ, USA · University of Pittsburgh, Pittsburgh, PA, USA · University of North Carolina at Chapel Hill, Chapel Hill, NC, USA · University of Central Florida, Orlando, FL, USA
Large language models (LLMs) achieve strong performance in long-horizon agentic tasks through multi-step interaction and reasoning at test time. However, repeatedly invoking these models incurs substantial inference cost over an episode. Post-training quantization (PTQ) reduces this cost, but using a fixed low precision throughout an episode can sharply lower task success. We observe that interaction steps differ sharply in their sensitivity to quantization: given the same context, decoding a step at low precision turns a success into a failure at only about 1% of the steps, which we call critical steps. Based on this observation, we propose Dynamic Mixed-Precision Routing (DMR), a framework that adaptively selects between high-precision and low-precision versions of the same LLM at each decision step. Leveraging an entire-space factorization to jointly predict a step's divergence and its effect on the outcome, DMR learns to recognize these critical steps by training only a lightweight router on the model's hidden state. By reusing a shared high-precision KV cache and the hidden states, DMR avoids redundant context prefill and a separate text encoder, enabling precision switching with less than 1 ms of overhead per step and adding less than 0.11% to step latency for routing. This mixed-precision design also enables nested quantization, reducing the memory footprint by 39% on average compared with FP16. Across three long-horizon agentic benchmarks with nested quantization, DMR achieves an average 2.44x decode speedup over FP16 at comparable task success rates, without fine-tuning the underlying model. On ALFWorld with GPTQ, DMR achieves a 1.71x decode speedup with 86.9% success, compared with 87.2% for FP16.
Figures & tables
Figure 1: Overview of DMR . (a) Precision tiers of the same LLM share a high-precision KV cache, enabling step-level routing without redundant context prefill. Low-precision responses are re-prefilled at high precision in subsequent steps (§ 3 ). (b) The router predicts step criticality through an entire-space factorization of action divergence and conditional criticality, using hidden-state features and counterfactual supervision (§ 4.2 ). (c) Iterative label collection combines agent rollouts, paired decoding, and counterfactual branching to train the router across rounds (§ 4.3 ).
Figure 2: One ScienceWorld episode (life-cycle task, a training task of the router). Each cell is one step, colored by the precision that decoded it. At step 8, the W3 output skips the adult stage and the task fails at once, while the W4 output names it and the task succeeds two steps later. Only a few steps need W4.
Figure 3: Critical steps are rare: GPTQ W3 vs. FP16 on 540 ALFWorld training games.
ScienceWorld
TextWorld
ALFWorld
Average
Method
Succ.
Speedup
W3
Succ.
Speedup
W3
Succ.
Speedup
W3
Recov.
Speedup
Fixed
FP16
71.6
1.00 ×
–
55.0
1.00 ×
–
87.2
1.00 ×
–
–
1.00 ×
Top tier Precision
71.1
2.65 ×
0%
56.0
1.62 ×
0%
88.3
1.49 ×
0%
100%
1.92 ×
W3
30.3
1.54 ×
100%
2.0
2.48 ×
100%
29.6
1.24 ×
100%
–
1.75 ×
Top prefill
Top → W3
60.7
3.20 ×
100%
44.0
2.29 ×
100%
79.2
2.05 ×
100%
0%
2.51 ×
Routing
Random
63.8
3.12 ×
77%
51.7
1.68 ×
66%
82.0
1.82 ×
75%
42%
2.21 ×
Table 1: Main results with nested quantization. We use ER1-7B with {W3,W4} on ScienceWorld and Qwen3-4B with {W3,W8} on TextWorld and ALFWorld. Succ. denotes task success rate (%), Speedup is episode decode speedup relative to FP16 averaged over test tasks, W3 denotes the share of tokens decoded at W3, and Recov. measures the fraction of the success gap between Top → W3 and the top tier recovered by each method. Fixed-precision baselines prefill at their own tier, while all other methods use top-tier prefill. Random results are averaged over three seeds. Bold and underlined values indicate the best and second-best routing methods, respectively.
Figure 4: Success rate versus episode decode speedup relative to the top precision tier. (a) ScienceWorld with ER1-7B and Any-Precision {W3,W4} ; (b) TextWorld cooking with Qwen3-4B and Any-Precision {W3,W8} ; (c) ALFWorld with Qwen3-4B and GPTQ {W3,FP16} . DMR lies on the success–speedup frontier in all three settings. Dashed lines indicate top-tier success rates, dotted lines indicate per-episode mixtures of the top tier and W3, and squares mark the top-tier and Top → W3 baselines.
Method
Succ.
Speedup
W3
FP16
87.2
1.00 ×
0%
W3
52.6
1.75 ×
100%
FP16 → W3
75.5
1.75 ×
100%
Random
78.8
1.46 ×
79%
Step schedule
k=2
85.0
1.64 ×
75%
Table 2: Independent PTQ on ALFWorld (GPTQ, {W3,FP16} , FP16 prefill for routing methods). Columns as in Table 1 .
Figure 5: Additional episode-level prefill time relative to top-tier-only execution. We compare DMR with a shared KV cache against separate-model routing with either one cache per model or context re-prefilling upon switching. All configurations follow DMR ’s routing decisions on the same test trajectories (Appendix E ).
Deployment
4B
7B
FP16 only
11.4
16.0
Separate FP16 and W3 models †
17.3
22.3
DMR , independent {W3,FP16} †
12.7
18.4
DMR , nested {W3,W4}
6.4
7.1
DMR , nested {W3,W8}
8.8
–
Table 3: Peak GPU memory usage (GB) across deployment configurations at batch size one, including the CUDA context and full-context KV caches (8292 tokens for 4B and 4296 for 7B). † Computed from the measured FP16 deployment: plus the 3-bit weights (independent), or plus a second W3 deployment with its own KV cache (separate).
Per step
Separate models
DMR
Switching
re-prefill, 218–324 ms
0.26–0.40 ms
Router input
encoder, 48–61 ms
probe, 0.19–0.31 ms
Table 4: Per-step overhead on an RTX A6000. Re-prefill latency is measured at the agents’ average context length of 2k tokens. Encoder latency uses Qwen3-Embedding-0.6B, as in MTRouter Zhang et al. (2026b) , averaged over 1,000 steps per benchmark from the test trajectories. Probe latency includes GPU-to-CPU hidden-state transfer.
Figure 6: Cumulative distribution of critical training states across environment steps for ALFWorld, TextWorld, and ScienceWorld. The curves indicate the fraction of critical states covered by allocating top-tier decoding to the first k steps. Dotted lines mark k=2,4,6 .
Figure 7: False negatives on ScienceWorld under matched high-precision budgets. We report the FN rate, i.e., the fraction of 80 critical training steps decoded at W3 (lower is better), by step schedules, the position-feature router, and DMR . Each budget matches the W4 step share of the corresponding step schedule ( k=2,4,6 ), while both routers independently select which steps use W4.
ALF GPTQ
ALF AP
SciWorld AP
Structure
AUC
FN
AUC
FN
AUC
FN
Mixed label
0.511
20.7
0.535
28.3
0.448
27.3
Criticality only
0.731
10.7
0.623
20.3
0.779
11.3
Separate heads
0.730
7.7
0.612
11.0
0.709
10.0
Entire space ( DMR )
0.738
7.0
0.627
11.3
0.745
8.7
Table 5: Offline comparison of router factorizations. AUC measures discrimination between critical and harmless divergences among states with counterfactual branches. FN counts critical states routed to W3 at a fixed W3 call share. Values are means over three cross-validation splits. Bold and underlined values indicate the best and second-best results in each column.
Method
Choice made per
One model
Shared KV
Decided by
Any-Precision, AnyBCQ, MatQuant
request
✓
–
requested bit width
MorphServe
load period
✓
✓
server load
PMPD
decode phase
✓
✓
decoding progress
DP-LLM
token × layer
✓
✓
layer error
QSpec
draft / verify
✓
✓
fixed roles
R2R
token
×
×
token divergence
Table 6: Methods that choose a precision or a model for LLM inference. One model : the candidates are precisions of a single model. Shared KV : a newly chosen candidate decodes from the existing cache without re-prefilling (–: no switch within a request). Decided by : what the choice depends on.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
ALFWorld
SciWorld
TextWorld
Agent model
Qwen3-4B
ER1-7B
Qwen3-4B
Prompt format
ReAct, 2-shot
Thought/Action
ReAct, 1-shot
Max tokens / calls
100 / 50
200 / 30
100 / 50
Context limit
8192
4096
8192
Test tasks
140 + 134
211
100
Steps per episode
22.2
13.9
32.9
Appendix
Table A.1: Benchmarks, protocols and per-step workload. Workload rows are measured on the FP16 agents’ test trajectories; prompt and response lengths are in tokens, averaged over all steps.
States
Divergent
Critical
ALFWorld GPTQ {W3,FP16}
13,450
2,049
143 (1.1%)
ALFWorld AP {W3,W8}
13,232
2,162
134 (1.0%)
ScienceWorld AP {W3,W4}
13,358
1,747
80 (0.6%)
TextWorld AP {W3,W8}
18,032
3,016
408 (2.3%)
Appendix
Table B.1: Round-1 counterfactual labels (training split). Critical: the low-tier branch fails while the top-tier branch succeeds.
Router
τ
Succ.
Speedup
W3
FP16
–
87.2
1.00 ×
0%
Round 0
0.5
84.3
1.27 ×
56%
0.7
86.1
1.28 ×
46%
0.9
86.5
1.22 ×
34%
DMR
0.95
86.9
1.71 ×
78%
Appendix
Table C.1: Divergence-only routing versus DMR on ALFWorld (GPTQ {W3,FP16} , FP16 prefill). Speedup is the mean decode time per task of FP16 divided by that of each router; W3 is the fraction of tokens decoded at W3. Thresholds apply to different scores: 1−pD for Round 0 and 1−pDpK for DMR .
ALFWorld, GPTQ {W3,FP16}
ScienceWorld, AP {W3,W4}
τ
W3
Succ.
Dec.
τ
W3
Succ.
Dec.
0.90
92
84.7
2.49
0.99
88
65.9
3.52
0.93
84
83.9
2.75
0.995
83
67.8
3.47
0.95
78
86.9
2.86
0.997
77
68.2
3.39
0.96
76
86.1
3.05
0.998
73
67.8
3.41
0.97
71
86.9
3.15
0.9995
58
67.8
3.45
Appendix
Table C.2: Results of DMR at different thresholds τ on the test tasks, ordered by W3 token share. W3: W3 token share (%). Succ.: success rate (%). Dec.: decode seconds per task. Bold: the threshold of the main results. Heuristic baselines are those of Table 2 (ALFWorld) and Table 1 (ScienceWorld).
Batch size
1
2
4
8
ALFWorld, over W8
1.40 ×
1.31 ×
1.25 ×
1.10 ×
ALFWorld, over FP16
2.09 ×
1.83 ×
1.61 ×
1.24 ×
ScienceWorld, over W4
1.23 ×
1.20 ×
1.13 ×
1.09 ×
ScienceWorld, over FP16
3.25 ×
2.88 ×
2.50 ×
1.66 ×
Appendix
Table D.1: Decode speedup of DMR versus batch size, TPOT measured at each batch size.
Figure E.1: Routing between separate models, for contrast with DMR (Figure 1 a): an external encoder re-reads the context at every step, and a model that was idle in the previous step first prefills the history it missed.
Cache policy
Succ.
Speedup
W3
Prefill (ms)
Low-tier reuse
DMR (re-prefill)
68.2
3.25 ×
77%
87
0%
Mixed (keep low-tier KV)
61.6
3.08 ×
76%
85
14%
Appendix
Table F.1: Cross-step KV-cache policies on ScienceWorld (Any-Precision {W3,W4} ; router and threshold as in Table 1 ). Speedup is episode decode speedup over FP16. Prefill is measured top-tier prefill latency per call. Low-tier reuse is the fraction of reused KV-cache tokens originally computed at W3.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.