cs.AIFeb 2, 2026

Dynamic Mixed-Precision Routing for Efficient Multi-step LLM Interaction

Authors: Yuanzhe Li, Jianing Deng, Jingtong Hu, Tianlong Chen, Song Wang, Huanrui Yang

Organizations: University of Arizona, Tucson, AZ, USA · University of Pittsburgh, Pittsburgh, PA, USA · University of North Carolina at Chapel Hill, Chapel Hill, NC, USA · University of Central Florida, Orlando, FL, USA

Abstract

Large language models (LLMs) achieve strong performance in long-horizon agentic tasks through multi-step interaction and reasoning at test time. However, repeatedly invoking these models incurs substantial inference cost over an episode. Post-training quantization (PTQ) reduces this cost, but using a fixed low precision throughout an episode can sharply lower task success. We observe that interaction steps differ sharply in their sensitivity to quantization: given the same context, decoding a step at low precision turns a success into a failure at only about 1% of the steps, which we call critical steps. Based on this observation, we propose Dynamic Mixed-Precision Routing (DMR), a framework that adaptively selects between high-precision and low-precision versions of the same LLM at each decision step. Leveraging an entire-space factorization to jointly predict a step's divergence and its effect on the outcome, DMR learns to recognize these critical steps by training only a lightweight router on the model's hidden state. By reusing a shared high-precision KV cache and the hidden states, DMR avoids redundant context prefill and a separate text encoder, enabling precision switching with less than 1 ms of overhead per step and adding less than 0.11% to step latency for routing. This mixed-precision design also enables nested quantization, reducing the memory footprint by 39% on average compared with FP16. Across three long-horizon agentic benchmarks with nested quantization, DMR achieves an average 2.44x decode speedup over FP16 at comparable task success rates, without fine-tuning the underlying model. On ALFWorld with GPTQ, DMR achieves a 1.71x decode speedup with 86.9% success, compared with 87.2% for FP16.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs

    May 19, 2026Haiquan Lu, Zigeng Chen, Gongfan Fang +2LLM QuantizationLong-Context Language Model Inference

  2. Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training

    Oct 6, 2026Alexandra Volkova, Matin Ansaripour, Erik Schultheis +2LLM QuantizationQuantization-Aware Training

  3. MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM

    Feb 21, 2026Dongwei Wang, Jinhee Kim, Seokho Han +8LLM QuantizationNeural Network Quantization