RAMP: Recurrent Adaptive Mixed Precision for Efficient Prefill in Recurrent-Depth Language Models

Ke Wan; Chen Chen

cs.CLOct 9, 2026

RAMP: Recurrent Adaptive Mixed Precision for Efficient Prefill in Recurrent-Depth Language Models

Authors: Ke Wan, Chen Chen

Organizations: Department of Computer Science, University of Virginia

Abstract

Recurrent-depth language models improve inference-time reasoning by repeatedly applying a shared network to refine internal representations, but the additional recurrent computation also increases inference cost. This overhead is particularly pronounced during long-context prefill, where the full input sequence is repeatedly processed across recurrent depth. Weight-and-activation quantization offers a natural way to accelerate this computation through lower-precision arithmetic, yet repeatedly applying quantized computation introduces numerical perturbations that accumulate along the recurrent trajectory. We systematically analyze how such errors propagate across recurrent depth and show that quantization errors introduced earlier are attenuated by more subsequent recurrent steps and therefore have a smaller effect on the final representation. Matched-budget experiments strongly support this prediction, showing that placing quantized computation earlier in recurrent depth consistently yields the best fidelity. Based on this insight, we introduce RAMP (Recurrent Adaptive Mixed Precision), which performs early recurrent refinement using low-precision weight-and-activation computation and restores high precision for terminal refinement. Across context lengths up to 4K, RAMP preserves near-full-precision prefill quality while achieving up to 1.16x end-to-end first-prefill speedup on an NVIDIA H100, corresponding to a 13.9% latency reduction.

Figures & tables

Figure 1: Empirical validation of depth-dependent quantization error propagation. (a) Perturbation survival as a function of injection depth. Both Gaussian perturbations and INT8 quantization-residual perturbations exhibit substantially lower survival when injected earlier in recurrent depth, while perturbations introduced near terminal depth retain a larger fraction of their initial error. Error bars denote the original 95% confidence intervals. (b) Precision-placement comparison under the same budget of 24 FP8 and 8 BF16 recurrent steps. Each point represents one of 20 frozen random precision schedules, with the horizontal axis indicating the mean depth of its BF16 steps and the vertical axis showing final hidden-state relative error. Moving BF16 computation toward later recurrent depth is associated with lower final error. RAMP places all eight BF16 steps at the end and outperforms all 20 random schedules, whereas the reverse ( Early-BF16 ) schedule places all eight BF16 steps at the beginning and produces substantially larger error.

Figure 2: Overview of RAMP . RAMP executes the shared recurrent block

F_{\theta}

using FP8 weight-and-activation computation during early recurrent refinement and restores BF16 computation for the final recurrent steps. The dashed line marks the precision switch, while the recurrent architecture and total recurrent depth remain unchanged.

Panel of Figure 3 — Figure 3: Quality preservation and end-to-end prefill efficiency of RAMP . (a) WikiText prefill fidelity at 4K context under different precision schedules. RAMP uses 24 FP8 recurrent steps followed by 8 BF16 steps, while Early-BF16 uses the same precision budget in the opposite order. Restoring BF16 computation for terminal refinement substantially reduces the NLL deviation from BF16-32. (b) WikiText fidelity and H100 first-prefill efficiency across context lengths. The green curve reports end-to-end first-prefill speedup over BF16, while the blue curve reports paired WikiText tail-64 $\Delta$ NLL relative to BF16-32. RAMP preserves near-BF16 fidelity across context lengths up to 4K while reaching approximately $1.16\times$ first-prefill speedup at long context.

	WikiText				PG19
Schedule	$\Delta$ NLL $\downarrow$	Hidden $\ell_{2}$ $\downarrow$	Logit $\ell_{2}$ $\downarrow$	Top-1 $\uparrow$	$\Delta$ NLL $\downarrow$	Hidden $\ell_{2}$ $\downarrow$	Logit $\ell_{2}$ $\downarrow$	Top-1 $\uparrow$
BF16-32	0	0	0	100.00%	0	0	0	100.00%
BF16-8	+0.12461	36.14%	15.76%	83.25%	+0.11260	37.68%	15.13%	79.79%
All-FP8	+0.00516	8.18%	3.80%	96.63%	+0.00233	8.45%	3.57%	95.61%
Early-BF16	+0.00444	8.15%	3.78%	95.95%	+0.00325	8.44%	3.58%	95.61%
RAMP	+0.00010	2.65%	1.34%	98.73%	+0.00007	3.08%	1.35%	98.29%

Table 1: Prefill fidelity at 4K context on WikiText and PG19. All metrics are measured relative to the full-depth BF16-32 reference within each dataset. BF16-8 is a reduced-depth control that executes only 8 recurrent steps, all in BF16. All-FP8, Early-BF16, and RAMP execute the full 32 recurrent steps; Early-BF16 and RAMP use the same budget of 24 FP8 and 8 BF16 recurrent steps but place the BF16 computation at opposite ends of recurrent depth. Lower is better for error metrics, while higher is better for top-1 agreement.

Context Length	BF16 (ms)	RAMP (ms)	Speedup $\uparrow$
128	131.784	164.972	0.7988 $\times$
256	152.802	197.474	0.7738 $\times$
512	148.473	139.567	1.0638 $\times$
1024	274.090	239.395	1.1449 $\times$
2048	543.158	469.357	1.1572 $\times$
4096	1103.444	949.847	1.1617 $\times$

Table 2: End-to-end first-prefill latency across context lengths on an NVIDIA H100. BF16 and RAMP use the same optimized empty-cache prefill path. Speedup is computed as BF16 latency divided by RAMP latency; values above

1\times

indicate faster execution with RAMP .

Schedule	BF16 Placement	Hidden $\ell_{2}$ $\downarrow$
Random-20 (mean)	Random	6.17%
Early-BF16	Steps 0–7	7.20%
RAMP	Steps 24–31	1.81%

Table 3: Fixed-budget precision-placement comparison on the fixed 12-prompt fidelity cohort. All configurations use exactly 24 FP8 and 8 BF16 recurrent steps. Random-20 reports the mean hidden-state error across the frozen random schedules, while Early-BF16 and RAMP place all BF16 computation at the beginning and end of recurrent depth, respectively.

FP8 $\rightarrow$ BF16 Steps	$\Delta$ NLL $\downarrow$	Hidden $\ell_{2}$ $\downarrow$
32 $\rightarrow$ 0	+0.00516	8.18%
24 $\rightarrow$ 8	+0.00010	2.65%
16 $\rightarrow$ 16	$-0.00016$	1.91%
0 $\rightarrow$ 32	0	0

Table 4: Effect of the terminal BF16 cleanup length on WikiText at 4K context. All configurations use the same 32 recurrent steps while varying the precision-switch point. Lower is better for both metrics.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Context	BF16 NLL	RAMP NLL	$\Delta$ NLL	Hidden $\ell_{2}$	Logit $\ell_{2}$
128	2.603711	2.605315	+0.001604	2.937%	1.429%
256	2.345469	2.347890	+0.002422	2.881%	1.647%
512	2.216275	2.217365	+0.001090	2.699%	1.378%
1024	2.153023	2.153203	+0.000179	2.556%	1.277%
2048	2.116153	2.115046	$-0.001107$	2.586%	1.294%
4096	2.101984	2.102086	+0.000102	2.653%	1.342%

Appendix

Table 5: WikiText long-context fidelity of RAMP . Results use 32 frozen evaluation endpoints with nested contexts and identical tail-64 prediction targets across context lengths.

\Delta

NLL is measured relative to BF16-32. Hidden-state and logit relative

\ell_{2}

errors are reported as percentages.

Context	BF16 NLL	RAMP NLL	$\Delta$ NLL	Hidden $\ell_{2}$	Logit $\ell_{2}$
512	2.671946	2.669488	$-0.002457$	2.722%	1.201%
1024	2.623134	2.621616	$-0.001518$	2.825%	1.325%
2048	2.607391	2.607932	+0.000542	2.792%	1.234%
4096	2.592401	2.592470	+0.000068	3.075%	1.354%

Appendix

Table 6: PG19 long-context fidelity of RAMP . Results use 32 frozen validation books with document-contained nested contexts and identical tail-64 prediction targets across context lengths.

\Delta

NLL is measured relative to BF16-32. Hidden-state and logit relative

\ell_{2}

errors are reported as percentages.

Explore similar work

RAMP: Reinforcement Adaptive Mixed Precision Quantization for Efficient On Device LLM Inference
Mar 18, 2026Arpit Singh Gautam, Saurabh JhaLLM Quantization Neural Network Quantization
Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs
Aug 4, 2026Yu Luo, Bo Dong, Wenhua Cheng +1Mixed-Precision Quantization LLM Quantization
Intel
AdaPonderLM: Gated Pondering Language Models with Token-Wise Adaptive Depth
Mar 2, 2026Shixiang Song, He Li, Zitong Wang +6LLM Inference Acceleration Language Modeling
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Innovation Institute · Sun Yat-sen University +2

Mar 18, 2026J/K move · Enter open · S save

RAMP: Reinforcement Adaptive Mixed Precision Quantization for Efficient On Device LLM Inference

Arpit Singh Gautam, Saurabh Jha

LLM Quantization Neural Network Quantization LLM Inference Efficiency Mixed-Precision Quantization Post-Training Quantization

Open paper AskPDF arXiv

Post training quantization is essential for deploying large language models (LLMs) on resource constrained hardware, yet state of the art methods enforce uniform bit widths across layers, yielding suboptimal accuracy efficiency trade offs. We present RAMP (Reinforcement Adaptive Mixed Precision), an off policy Soft Actor Critic framework that learns per layer bit width assignments to minimize perplexity under a global bit budget. The policy conditions on an 11 dimensional embedding of activation statistics, weight properties, and structural descriptors, enabling zero shot transfer across model families and scales. To enable stable sub 4 bit quantization, we introduce Scale Folding, a preconditioning technique that migrates activation outliers into weights via per channel scaling and normalization layer compensation. A quality prioritized reward with asymmetric penalties and budget cliffs drives rapid convergence. On Llama 2 7B, RAMP achieves 5.54 perplexity at 3.68GB (3.65 effective bits), outperforming uniform 4 bit AWQ (5.60 at 3.90 GB) and GPTQ by 6% in size and 1% to3% in quality. Critically, a policy trained only on Llama 2 7B generalizes zero shot to Llama 2 13B and Mistral 7B, often surpassing target specific training, supporting the hypothesis that quantization sensitivity is primarily architectural. The HALO pipeline exports allocations to GGUF format for kernel free inference on CPUs, GPUs, and edge devices, retaining 99.5% of FP16 commonsense reasoning performance.

Abstract

Explore similar work

RAMP: Reinforcement Adaptive Mixed Precision Quantization for Efficient On Device LLM Inference

Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs

AdaPonderLM: Gated Pondering Language Models with Token-Wise Adaptive Depth