Existing LLM post-training quantization techniques determine precision before inference begins, yet the precision a model requires varies dynamically across prompts, layers, and inference phases. We present AutoEMatic, a full-stack co-design that adapts the precision of weights, activations, and the KV cache at runtime. A lightweight policy selects each layer's precision at every prefill and decode step from two runtime signals: the entropy of the next-token distribution and the magnitude of each layer's update to its hidden states. This runtime adaptation is made practical by AutoFP, a multi-scale floating-point format that derives 2-, 4-, and 6-bit precisions from a single NVFP4 base format. We realize AutoEMatic on two targets: GPU kernels that execute AutoFP on NVFP4 Tensor Cores, and an accelerator whose vector MAC datapath executes all AutoFP formats natively. Across reasoning and agentic benchmarks, AutoEMatic improves accuracy by up to 21.7% over NVFP4 and matches group-scaled FP6 accuracy on most benchmarks, using 4.9 bits per weight versus 4.5 for NVFP4. On GB300 GPUs, AutoEMatic retains approximately 90-98% of NVFP4 decode throughput across the evaluated configurations. Furthermore, the AutoEMatic accelerator reduces latency by 1.6-2.3x and energy by 7-23% compared to an accelerator with NVFP4-only datapath at iso-accuracy.
Figures & tables
Figure 1: LiveCodeBench accuracy versus decode throughput for Qwen3.8-27B (batch size 1, one GB300 GPU).
Figure 2: (a) Precision distribution averaged across layers in the early, middle, and late generation stages (DS-Qwen-14B, LCB; Qwen3.8-27B, τ2 -bench). (b) Layer-wise precision assignments for initial prefill and additional refills (Qwen3.8-27B, τ2 -bench). (c) Rate of underprovisioning for ModelOpt AutoQuantize and FlexQuant (DS-Qwen-14B, LCB).
Figure 3: Refining a subset of elements achieves NMSE comparable to 6-bit . Y-axis is NMSE, such that NVFP4 is 1 and FP6 g is 0, vs. fraction of subgroups refined to 6-bit.
Figure 4: Layer-wise behavior of MP and MD . (a) Prefill MP of Qwen3.8-27B on a τ2 -bench task. (b) Decode MD of GPT-OSS-20B on RULER-16K, at high- and low- GE steps.
Policy
Prefill
Decode
High entropy
Low entropy
Metric
MP=LA
MD=0.7LA+0.3GE
Initial precision
6-bit
4-bit
4-bit
Rules activate
After spike
Layer 0
Raise precision
np layers >μ
β layers >μ
α layers >μ
Lower precision
np layers <μ
α layers <μ
β layers <μ
Table 1: Runtime precision adaptation policy.
Figure 5: Mantissa refinement of NVFP4 to AutoFP6.
Figure 6: Extended- K GEMM for one 16-element group and three requests (batch size of 3).
Reasoning
Agentic
Model
Method (budget)
LB v2
AIME
LCB
τ2 -bench
LME
Qwen3-8B
BF16
35.4
56.7
66.5
30.7
42.8
NVFP4
33.3
50.0
49.2
17.5
26.0
FP6 g
35.4
56.7
61.4
30.7
39.9
AutoQuantize (4.5)
35.4
50.0
60.0
20.2
31.9
DP-LLM † (4.5)
21.6 (4.8)
16.7 (4.5)
26.6 (4.6)
10.6 (4.5)
18.5 (4.5)
Table 2: Accuracy comparison of AutoEMatic and baselines across reasoning and agentic workloads. Execution precision appears in parentheses next to accuracy. LB v2: LongBench v2, LCB: LiveCodeBench, LME: LongMemEval.
Figure 7: Layer-wise precision assignment by AutoEMatic on MiniMax-M2.7.
Exec. Precision
Method
Prefill
Decode
Acc. (%)
FP8
8
8
71.0
NVFP4
4
4
67.8
FP6 g
6
6
70.5
AutoEMatic
4.2
4.5
70.5
Table 3: Accuracy comparison of MiniMax-M2.7 on SWE-bench Verified.
Figure 8: Decode throughput and prefill TTFT analysis on 4× GB300 GPUs for DS-LLaMA-70B and MiniMax-M2.7.
Figure 9: AutoEMatic accelerator architecture.
Figure 10: AutoFP2 representation.
Figure 11: AutoFP vector MAC datapath.
Figure 12: Area breakdown of (a) the AutoEMatic accelerator and (b) its vMAC.
Figure 13: Normalized latency and energy comparison with single-format baselines, under iso-accuracy.
4. AnyPrecision LLM on P + AutoEMatic runtime policy
6
43.8
8 / 42 / 50
5. AutoFP6
4.9
64.8
0 / 0 / 100
6. + Refinement-aware GPTQ
4.9
65.6
0 / 0 / 100
Table 4: Component ablations with DS-LLaMA-70B on LCB.
Figure 14: Ablations. (a) Fixed vs. exponent-scaled decoding of d . (b) Underprovision rate of AutoEMatic. (c) Prefill precision at spike. (d) Refined subgroups per group. (e) Accuracy vs. throughput pareto. (f) Kernel ablations. (g) For one decode step: MD across layers and the precision assigned by the oracle, an instantaneous rule, and AutoEMatic.
Figure 15: DS-LLaMA-70B LCB accuracy vs. performance per area and energy per op , normalized to an NVFP4-only datapath.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Format
S/E/M
Magnitudes
Bits/elem.
E2M1 (NVFP4)
1/2/1
{0,0.5,1,1.5,2,3,4,6}
4.5
E2M3 (FP6 g )
1/2/3
0 – 7.5 in steps of 81,41,21
6.5
E1M0
1/1/0
{1,2}
2.5
E0M1
1/0/1
{0,1}
2.5
Quinary digit (AutoFP2)
–
{0,1,2}
3.0
E4M3 (scale)
1/4/3
2−9 – 448
–
Appendix
Table 5: Low-precision floating-point formats.
E2M1 (NVFP4)
E2M3 (FP6 g )
E\M
0
1
000
001
010
011
100
101
110
111
00
0
0.5
0
0.125
0.25
0.375
0.5
0.625
0.75
0.875
01
1
1.5
1
1.125
1.25
1.375
1.5
1.625
1.75
1.875
10
2
3
2
2.25
2.5
2.75
3
3.25
3.5
3.75
11
4
6
4
4.5
5
5.5
6
6.5
7
7.5
Appendix
Table 6: Positive magnitudes of E2M1 (NVFP4) and E2M3 (FP6 g ).
NVFP4 base
AutoFP6 refinement
Subgroup
x/s
q
Code
Err.
δ
Code
q+δ
Err.
1.00
1.0
0 01 0
0
–
–
–
–
− 0.50
− 0.5
1 00 1
0
–
–
–
–
3.00
3.0
0 10 1
0
–
–
–
–
SG0
6.00
6.0
0 11 1
0
–
–
–
–
1.13
1.0
0 01 0
0.13
+81
01
1.125
0.005
Appendix
Table 7: Mantissa refinement of one NVFP4 group ( s=0.25 ). Values are in NVFP4 units; codes are sign — exponent — mantissa. Green rows are the refined subgroups (mask 0110 ).
Figure 16: Layer-wise precision across decode steps. Per-layer precision at ten decode steps of one DS-Qwen-14B LiveCodeBench response (top) and one Qwen3.8-27B τ2 -bench response (bottom).
Figure 17: Oracle layer-wise precision for the initial prefill and three refills of Gemma-4-26B on three LongMemEval samples (S1–S3).
Figure 18: Oracle precision demand in high- (H) and low-entropy (L) decode steps.
Figure 19: Prefill MP for the initial prefill and three refills of one τ2 -bench task for (a) Gemma-4-26B and (b) Qwen3.8-27B.
Figure 20: Refining a subset of elements achieves NMSE comparable to 6-bit . Y-axis is NMSE, such that NVFP4 is 1 and FP6 g is 0, vs. fraction of subgroups refined to 6-bit, for Qwen3.8-27B and Gemma-4-26B on GSM8K.
Figure 21: Signals used by AutoEMatic for runtime precision adaptation. ❶ Local layer-update activity LA . ❷ Global token-generation uncertainty GE .
Figure 22: Global signal GE over a 16K-token AIME generation of Qwen3.8-27B. The dashed line marks τ=0.10 .
Figure 23: AutoEMatic memory layout.
Figure 24: A sum of GEMMs as one GEMM over an extended reduction dimension.
Figure 25: Construction of Wqr and Aqr for one 16-element group and one request. The weight mask places the weight offsets 4δW in Wqr and copies the base Aq into Aqr ; the activation mask copies the base Wq into Wqr and places the activation offsets 4δA in Aqr ; all other positions are zero.
Figure 26: (1) The exact product of the refined operands W and A . (2) The base GEMM on Wq and Aq plus a refinement GEMM on the constructed Wqr and Aqr at scale sW/4 . (3) A single GEMM over the concatenated operands.
Wq
0
0.5
1
1.5
2
3
4
6
Q0
0
1
0
1
0
2
2
2
Q1
0
−2
1
−1
2
−2
−1
1
Appendix
Table 8: NVFP4-to-AutoFP2 decomposition.
Figure 27: Post-layout floorplan of AutoEMatic .
Model
Method (budget)
LB v2
AIME
LCB
GSM8K
DS-Llama-70B
BF16
41.0
66.7
68.2
94.5
NVFP4
29.8
53.3
51.7
90.6
FP6 g
41.0
66.7
66.2
94.5
AutoQuantize (5.3)
38.7
66.7
55.7
94.5
FlexQuant (5.3)
27.8
50.0
48.6
86.5
DP-LLM (weight-only) (5.3)
21.0 (5.1)
43.3 (5.3)
43.3 (5.3)
81.6 (5.1)
Appendix
Table 9: Accuracy of AutoEMatic and baselines on DS-Llama-70B, DS-Qwen-32B, and Llama-3.1-8B-Instruct. Cells report accuracy (%); execution precision appears in parentheses. LB v2: LongBench v2, LCB: LiveCodeBench. Llama-3.1-8B-Instruct is not evaluated on AIME and LCB, where its BF16 accuracy is too low for a meaningful comparison.
Figure 28: Additional ablations on DS-LLaMA-70B (LCB). (a) Scale multiple c of AutoFP2: mean layer-output NMSE over all 80 layers at W2A4. (b) Subgroup size SG with 20% of all weight subgroups refined. (c) Weight refinement budget ρ , with storage precision. (d) Activation NMSE threshold, with the mean number of activation subgroups per group exceeding it. (e) λD sweep. (f) Entropy threshold τ . (g) Trend lengths np , α , and β . (h) Overhead of computing GE and LA as a share of decode TPOT. Stars mark the chosen settings.
Figure 29: Online serving under continuous batching. Median TPOT vs. achieved request rate for DS-LLaMA-70B on 4× GB300 (TP=4) with Poisson arrivals of LongBench v2 requests.
Figure 30: Layer-wise MP and MD across models and datasets. For each model (rows) and dataset (column pairs), prefill MP of one representative sample and decode MD of the same sample at the highest- and lowest- GE step.
Figure 31: Second-order refinement term. Per-layer output NMSE of AutoFP6 with and without the second-order product WrAr (top) and the energy of that product relative to the layer output (bottom) on DS-LLaMA-70B.
Figure 32: Extension of the reduction dimension of refined GEMMs on Qwen3.8-27B τ2 -bench vs. decode batch size, with the mean number of refined tokens per layer.
Component
Bits
Acc.
2 / 4 / 6 / 8 (%)
BF16
16
68.4
–
AutoEMatic format + runtime policy, P={2,4,6}
4.9
65.6
15 / 45 / 40 / 0
AutoFP8 + Refinement-aware GPTQ
5.3
67.9
0 / 0 / 0 / 100
AutoEMatic format + runtime policy , P={2,4,6,8}
5.3
67.9
30 / 35 / 15 / 20
Appendix
Table 10: Extension to 8 bits with DS-LLaMA-70B on LCB; rows 1 and 7 of Table 4 are repeated for reference.
Figure 33: Accelerator latency and energy at iso-accuracy with NVFP4.
Policy
Bits
Acc.
2 / 4 / 6 (%)
Instantaneous ( np=α=β=1 )
4.7
92.5
0 / 66 / 34
AutoEMatic ( np=2 , α=4 , β=2 )
4.6
95.5
0 / 69 / 31
Appendix
Table 11: Instantaneous vs. trend-based precision changes with Qwen3-8B on GSM8K.
Figure 34: TTFT of the runtime-adaptive baselines DP-LLM and FlexQuant on DS-LLaMA-70B vs. input sequence length.