cs.AINov 6, 2026

AutoEMatic: Runtime-Adaptive Mixed-Precision LLM Inference via Automatic Exponent-Mantissa Refinement

Authors: Akshat Ramachandran, Reena Elangovan, Rangharajan Venkatesan, Charbel Sakr, Tijmen Blankevoort, Brucek Khailany, Tushar Krishna

Organizations: Georgia Institute of Technology · NVIDIA

Abstract

Existing LLM post-training quantization techniques determine precision before inference begins, yet the precision a model requires varies dynamically across prompts, layers, and inference phases. We present AutoEMatic, a full-stack co-design that adapts the precision of weights, activations, and the KV cache at runtime. A lightweight policy selects each layer's precision at every prefill and decode step from two runtime signals: the entropy of the next-token distribution and the magnitude of each layer's update to its hidden states. This runtime adaptation is made practical by AutoFP, a multi-scale floating-point format that derives 2-, 4-, and 6-bit precisions from a single NVFP4 base format. We realize AutoEMatic on two targets: GPU kernels that execute AutoFP on NVFP4 Tensor Cores, and an accelerator whose vector MAC datapath executes all AutoFP formats natively. Across reasoning and agentic benchmarks, AutoEMatic improves accuracy by up to 21.7% over NVFP4 and matches group-scaled FP6 accuracy on most benchmarks, using 4.9 bits per weight versus 4.5 for NVFP4. On GB300 GPUs, AutoEMatic retains approximately 90-98% of NVFP4 decode throughput across the evaluated configurations. Furthermore, the AutoEMatic accelerator reduces latency by 1.6-2.3x and energy by 7-23% compared to an accelerator with NVFP4-only datapath at iso-accuracy.

Figures & tables

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training

    Oct 6, 2026Alexandra Volkova, Matin Ansaripour, Erik Schultheis +2LLM QuantizationQuantization-Aware Training

  2. AutoQS: Automated Quantization Strategy Design with Large Language Models

    Nov 9, 2026Zhen Liu, Zhiqin Yang, Yonggang Zhang +3LLM QuantizationQuantization

  3. MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM

    Feb 21, 2026Dongwei Wang, Jinhee Kim, Seokho Han +8LLM QuantizationNeural Network Quantization