RAMP: Reinforcement Adaptive Mixed Precision Quantization for Efficient On Device LLM Inference

Arpit Singh Gautam; Saurabh Jha

cs.LGMar 18, 2026

RAMP: Reinforcement Adaptive Mixed Precision Quantization for Efficient On Device LLM Inference

Authors: Arpit Singh Gautam, Saurabh Jha

Abstract

Post training quantization is essential for deploying large language models (LLMs) on resource constrained hardware, yet state of the art methods enforce uniform bit widths across layers, yielding suboptimal accuracy efficiency trade offs. We present RAMP (Reinforcement Adaptive Mixed Precision), an off policy Soft Actor Critic framework that learns per layer bit width assignments to minimize perplexity under a global bit budget. The policy conditions on an 11 dimensional embedding of activation statistics, weight properties, and structural descriptors, enabling zero shot transfer across model families and scales. To enable stable sub 4 bit quantization, we introduce Scale Folding, a preconditioning technique that migrates activation outliers into weights via per channel scaling and normalization layer compensation. A quality prioritized reward with asymmetric penalties and budget cliffs drives rapid convergence. On Llama 2 7B, RAMP achieves 5.54 perplexity at 3.68GB (3.65 effective bits), outperforming uniform 4 bit AWQ (5.60 at 3.90 GB) and GPTQ by 6% in size and 1% to3% in quality. Critically, a policy trained only on Llama 2 7B generalizes zero shot to Llama 2 13B and Mistral 7B, often surpassing target specific training, supporting the hypothesis that quantization sensitivity is primarily architectural. The HALO pipeline exports allocations to GGUF format for kernel free inference on CPUs, GPUs, and edge devices, retaining 99.5% of FP16 commonsense reasoning performance.

Figures & tables

Model	Parameters	FP16 Size	24 GB GPU	Viable?
Llama-2-7B	7.0B	13.5 GB	24 GB	Marginal
Llama-2-13B	13.0B	26.0 GB	24 GB	No
Llama-3-8B	8.0B	16.1 GB	24 GB	No
Mistral-7B	7.0B	14.5 GB	24 GB	Marginal
GPT-3.5	175B+	350+ GB	24 GB	Impossible

Table 1: Model memory footprint in FP16 and viability on 24 GB consumer GPU

Method	Bit Allocation	Transfer	Eval Budget	Llama-2-7B PPL	Year
RTN-4	Uniform	—	Single pass	5.94	2018
GPTQ-4	Uniform	—	Per-layer opt.	5.69	2023
AWQ-4	Uniform	—	Single pass	5.60	2023
SmoothQuant-4	Uniform	—	Single pass	5.62	2023
QuIP#-4	Uniform	—	Single pass	5.62	2024
LRQ-4	Uniform	—	Single pass + preconditioner	5.75	2025

Table 2: RAMP vs. related quantization methods (Llama-2-7B, WikiText-2 PPL)

Figure 1: Overview of the RAMP pipeline. Stage 1 uses a Soft Actor-Critic agent in a distributed multi-GPU setting to discover a mixed-precision strategy. Stage 2 performs kernel-free compilation via scale folding. Stage 3 quantizes the model layer-by-layer and exports it in GGUF format for deployment.

Figure 2: Reward computation in RAMP. After applying the policy, the quantized model is evaluated on perplexity, memory footprint, and activation stability. These signals are combined into a scalar reward that prioritizes quality while enforcing the bit budget.

Figure 3: Training dynamics showing perplexity, average bit-width, and reward across episodes.

Figure 4: Best-so-far perplexity encountered during search.

Component	Choice	Rationale
RL Algorithm	SAC (off-policy, continuous)	Sample-efficient learning with stable exploration
State Representation	11-dimensional layer embeddings	Abstracts sensitivity; enables cross-model transfer
Action Space	Continuous scalar (mapped to discrete bits)	Smooth exploration
State Normalization	By layer width/depth	Approximate scale invariance
Policy Network	MLP (512–512–256) + LayerNorm	Capacity for heterogeneous sensitivities
Q-Networks	Twin critics + Polyak targets	Reduces overestimation bias

Table 3: RAMP framework components and design choices

Method	WikiText-2 PPL	Model Size
No scale folding	5.58	3.80 GB
HALO (with folding)	5.54	3.68 GB

Table 4: Impact of scale folding on Llama-2-7B GGUF export

Platform	Device Example	Memory	Latency (ms/token)	PPL	Status
GPU	RTX 3090	$\sim$ 3.7 GB	$\sim$ 7–10	5.54	$\checkmark$
GPU	A100	$\sim$ 3.7 GB	$\sim$ 2–4	5.54	$\checkmark$
CPU	Intel Xeon (16-core)	$\sim$ 14 GB	$\sim$ 30–50	5.54	$\checkmark$
Apple Silicon	M1/M2/M3	Unified	$\sim$ 25–80	5.54	$\checkmark$
ARM edge	Raspberry Pi-class	4–8 GB	$>$ 300	—	Runnable
Mobile SoC	Snapdragon-class	Device-dependent	Vendor-dependent	$\sim$ 5.5	Prototype

Table 5: Representative deployment characteristics of HALO GGUF models across platforms (values from public llama.cpp benchmarks)

Model	Params	Vocab	Hidden	Transformer Blocks	Attention
Llama-2-7B	6.7B	32K	4096	32	MHA
Llama-2-13B	13B	32K	5120	40	MHA
Llama-3-8B	8B	128K	4096	32	GQA
Mistral-7B	7.2B	32K	4096	32	SWA
DistilGPT-2	0.066B	50K	768	12	MHA

Table 6: Model specifications used in experiments

Task	Type	Questions	Metric
PIQA ( Bisk et al., 2020 )	Commonsense	2-choice	Accuracy
HellaSwag ( Zellers et al., 2019 )	Commonsense	4-choice	Accuracy
WinoGrande ( Sakaguchi et al., 2020 )	Commonsense	2-choice	Accuracy
ARC ( Clark et al., 2018 )	Reasoning	Multiple-choice	Accuracy

Table 7: Downstream task benchmarks

Method	Type	PPL	Size (GB)
FP16	Upper bound	5.51	13.5
RTN-4	Naive	5.94	3.90
GPTQ-4 ( Frantar et al., 2023 )	SOTA	5.69	3.90
AWQ-4 ( Lin et al., 2023 )	SOTA	5.60	3.90
QUIP#-4 ( Tseng et al., 2024 )	SOTA	5.62	3.90
Q4_K_M (GGUF)	Deployment	5.61	3.95

Table 8: Baseline comparison on Llama-2-7B

Parameter	Value	Rationale
Learning rate	$3\times 10^{-4}$	Actor and critics
Batch size	128	Stability vs. efficiency
Replay buffer	30,000 transitions	$\sim$ 120 episodes $\times$ 250 steps
Discount $\gamma$	0.99	Standard episodic RL
Target update $\tau$	0.005	Slow tracking
Gradient clip	1.0	Stability

Table 9: SAC training hyperparameters

Parameter	Value	Rationale
Method	Per-group asymmetric	Standard LLM PTQ
Group size	128	GGUF-compatible
Bit widths	$\{3,4,5\}$	Aggressive compression range
Target avg. bits	4.0	Compression goal
Ceiling	4.25	Reward-enforced
Calibration	WikiText-2 (128 seq., 2048 tokens)	Efficient and representative

Table 10: Quantization configuration

Parameter	Value	Rationale
Scale	$s=\sqrt{\text{act\_scale}}$	Smoothing
Normalization	$s\leftarrow s/\text{mean}(s)$	Magnitude preservation
Stabilization	$+10^{-5}$	Numerical safety
Scope	Per-channel (Q/K/V/O/gate/up)	Fine-grained conditioning
Norm adjustment	$\text{RMSNorm}\leftarrow\text{RMSNorm}/s$	Equivalence preservation
Calibration	128 sequences	Reuse of quantization data

Table 11: Scale folding configuration

Method	Size (GB)	PPL	Size vs RAMP	PPL vs RAMP
FP16	13.5	5.51	$+267\%$	—
RTN-4	3.90	5.94	$+6.0\%$	$+7.2\%$
GPTQ-4	3.90	5.69	$+6.0\%$	$+2.7\%$
AWQ-4	3.90	5.60	$+6.0\%$	$+1.1\%$
QUIP#-4	3.90	5.62	$+6.0\%$	$+1.4\%$
Q4_K_M	3.95	5.61	$+7.3\%$	$+1.3\%$

Table 12: Results on Llama-2-7B (224 quantizable linear layers)

Figure 5: Perplexity vs. model size on Llama-2-7B. RAMP dominates uniform 4-bit baselines.

Figure 6: RAMP bit allocation on Llama-2-7B (224 quantizable linear layers).

Method	Size (GB)	PPL	Size vs RAMP	PPL vs RAMP
FP16	26.0	4.85	$+259\%$	—
GPTQ-4	7.68	5.02	$+6.1\%$	$+1.4\%$
AWQ-4	7.68	4.97	$+6.1\%$	$+0.4\%$
Q4_K_M	7.73	4.98	$+6.8\%$	$+0.6\%$
Direct SAC	7.25	4.96	$+0.1\%$	$+0.2\%$
RAMP (zero-shot)	7.24	4.95	—	—

Table 13: Results on Llama-2-13B (280 quantizable linear layers)

Method	Size (GB)	PPL	Size vs RAMP	PPL vs RAMP
FP16	16.1	6.23	$+281\%$	—
GPTQ-4	5.74	8.58	$+36\%$	$+33\%$
AWQ-4	5.31	6.74	$+26\%$	$+4.2\%$
Q4_K_M	5.48	6.79	$+30\%$	$+5.0\%$
RAMP (zero-shot)	4.22	6.47	—	—

Table 14: Results on Llama-3-8B

Figure 7: Perplexity vs. model size on Llama-3-8B.

Figure 8: RAMP bit allocation on Llama-3-8B (224 quantizable linear layers).

Method	Size (GB)	PPL	Size vs RAMP	PPL vs RAMP
FP16	14.5	5.45	$+299\%$	—
GPTQ-4	3.85	5.71	$+6.1\%$	$+2.7\%$
AWQ-4	3.85	5.59	$+6.1\%$	$+0.5\%$
Q4_K_M	3.90	5.60	$+7.4\%$	$+0.7\%$
Direct SAC	3.62	5.58	$-0.3\%$	$+0.4\%$
RAMP (zero-shot)	3.63	5.56	—	—

Table 15: Results on Mistral-7B

Figure 9: First- and last-10 layer bit assignments across models.

Target	Direct PPL	Direct Bits	Zero-shot PPL	Zero-shot Bits	Winner
Llama-2-13B	4.96	3.66	4.95	3.66	Zero-shot
Mistral-7B	5.58	3.67	5.56	3.67	Zero-shot

Table 16: Zero-shot transfer summary

Method	PIQA	HellaSwag	WinoGrande	ARC	Avg
FP16	79.4%	58.2%	64.0%	53.9%	63.9%
AWQ-4	78.8%	57.1%	62.3%	52.4%	62.7%
Q4_K_M	78.6%	56.9%	62.1%	52.1%	62.4%
RAMP	79.2%	57.9%	63.7%	53.6%	63.6%
Retention vs. FP16	99.7%	99.5%	99.5%	99.4%	99.5%

Table 17: Commonsense reasoning performance (Llama-2-7B)

Layer Role	Llama-2-7B	Llama-2-13B	Mistral-7B	Correlation
Embedding layer	(0.87, 0.34, 0.12, …)	(0.91, 0.36, 0.13, …)	(0.85, 0.32, 0.11, …)	$r=0.94$
Attention o_proj	(3.12, 0.18, 0.08, …)	(3.15, 0.19, 0.09, …)	(3.08, 0.17, 0.07, …)	$r=0.96$
MLP down_proj	(2.87, 0.22, 0.11, …)	(2.91, 0.23, 0.12, …)	(2.84, 0.21, 0.10, …)	$r=0.95$
Output layer	(1.24, 0.41, 0.19, …)	(1.28, 0.42, 0.20, …)	(1.21, 0.40, 0.18, …)	$r=0.93$

Table 18: Embedding stability: Pearson correlations across models

Figure 10: Layer-wise bit assignments vs. depth (224 quantizable linear layers).

Figure 11: Smoothed bit-width trend across depth.

Figure 12: Per-layer bit-flip frequency during training.

Figure 13: Bit-transition probabilities across episodes.

Figure 14: Mean bit-width per layer type.

Configuration	Episodes to Conv.	Final PPL	Stable?
No folding	$>250$	5.58	Partial
With folding	150	5.54	$\checkmark$

Table 19: Scale folding ablation

Reward	Episodes to Conv.	Final PPL	Variance
Naive (Acc - $\lambda$ Bits)	$\infty$	$>5.75$	0.35
Linear	200	5.61	0.18
Quality-first (asymmetric + cliff)	150	5.54	0.05

Table 20: Reward ablation

Algorithm	GPU Hours	Final PPL	Variance
PPO	48	5.62	0.15
SAC	6	5.54	0.03

Table 21: SAC vs. PPO

Figure 15: PCA of layer embeddings colored by assigned bit-width.

Approach	Throughput	Portability / Effort
Custom mixed-precision kernels	Highest (hardware-specific)	Low (high engineering)
HALO / GGUF	Competitive	High (zero custom dev, broad hardware)

Table 22: HALO vs. custom kernels

Deployment	Power (W)	Annual kWh	CO 2 (kg)	Cost ($)
Cloud A100	350	2,920	1,168	292
RTX 3090 FP16	250	2,088	835	209
RTX 3090 RAMP	85	711	284	71
Xeon CPU RAMP	65	546	218	55
M1 laptop RAMP	12	101	40	10

Table 23: Illustrative annual energy and carbon footprint (7B model, 8 h/day)

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Dim	Feature	Computation / Definition
1	Normalized depth	$\frac{i}{n_{\text{layers}}-1}$
2	Log input dimension	$\log_{2}(\text{in\_features}/16)$
3	Log output dimension	$\log_{2}(\text{out\_features}/16)$
4	Weight standard deviation	$\min(\text{std}(W)\times 10,1.0)$
5	Weight mean magnitude	$\min(\mathbb{E}[\|W\|]\times 10,1.0)$
6	Layer-type scalar	0.0 (Q/K/V proj.), 0.25 (output proj.), 0.5 (gate proj.), 0.6 (up proj.), 0.75 (down proj.), 1.0 (otherwise)

Appendix

Table 24: Specification of the 11-dimensional layer embedding used by the RAMP policy.

Parameter	Value	Notes
Learning rate	$3\times 10^{-4}$	Shared across actor, critics, and temperature optimizer
Optimizer	Adam	Applied to all components
Discount factor $\gamma$	0.99	Standard for episodic tasks
Target update rate $\tau$	0.005	Ensures slow tracking of target networks
Gradient clipping norm	1.0	Prevents exploding gradients
Target entropy	$-1$	For single-dimensional continuous action

Appendix

Table 25: SAC optimization hyperparameters

Parameter	Value	Notes
Batch size	128	Used for all gradient updates
Replay buffer capacity	30,000 transitions	Sufficient for $\sim$ 120 episodes $\times$ 250 steps
Warm-up episodes	20	Purely random actions before policy gradients
Maximum episodes	200–300	Typical convergence budget in experiments

Appendix

Table 26: Replay buffer and training schedule

Component	Architecture	Notes
Actor	FC-512-LN-ReLU $\to$ FC-512-LN-ReLU $\to$ FC-256-ReLU $\to$ FC-2	Outputs mean and log-std of action
Critic (twin)	FC-512-LN-ReLU $\to$ FC-512-LN-ReLU $\to$ FC-256-ReLU $\to$ FC-1	Input is state concatenated with continuous action
Hidden activation	ReLU	Fast and effective for policy/value approximation
Normalization	LayerNorm	Applied after the first two hidden layers

Appendix

Table 27: Actor and critic network architectures

Explore similar work

Oct 6, 2026

Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training

Quantization-aware training (QAT) leverages lower-precision arithmetic to reduce the cost of LLM deployment, but aggressive quantization degrades final model performance. A common remedy is mixed-precision training, in which high precision is assigned to some of the layers to maintain performance while keeping the cost constrained. This approach then requires precision assignments for model layers during training. We provide a new approach, called Q-PACE, consisting of a second-order sensitivity model that predicts the loss increase as a sum of quantization noise MSE weighted by per-layer curvature coefficients. During training, we periodically re-compute these coefficients using perturbations across layers, and re-assign precision. Pretraining and supervised fine-tuning experiments on LLMs of up to 4B parameters show that Q-PACE consistently improves over existing mixed-precision training recipes, and achieves comparable loss at substantially lower total memory budgets. We further find that quantization sensitivity is highly predictable by depth and layer type, and its stability during training allows for infrequent, cheap recalibration.

Alexandra Volkova, Matin Ansaripour, Erik Schultheis +2

ISTA · EPFL, ISTA

LLM Quantization Quantization-Aware Training Mixed-Precision Quantization Memory-Efficient Training

Ask PDF

Apr 10, 2026

EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillation

Quantization has emerged as a mainstream approach for deploying Large Language Models (LLMs) on resource-constrained devices, yet compressing precision below 4-bit typically causes severe performance degradation or prohibitive retraining costs. In this paper, we propose EdgeRazor, a lightweight framework for LLMs via Mixed-Precision Quantization-Aware Distillation. It contains three modules: Structural Quantization with Mixed Precision for fine-grained control of bit-widths, Layer-Adaptive Feature Distillation that dynamically selects the most informative features for alignment, and Entropy-Aware KL Divergence for forward-reverse balance on both human-annotated and distilled datasets. Evaluations conducted on MobileLLM and Qwen families show that under weight-activation quantization, the 1.88-bit Qwen3-0.6B-EdgeRazor outperforms the state-of-the-art 2-bit baselines by 11.27 and surpasses the strongest 3-bit baselines by 4.38, while the quantized MobileLLM-350M-EdgeRazor requires a training budget 4-10

\times

lower than the leading quantization-aware training method. In terms of efficiency, EdgeRazor achieves higher compression ratios at all bit-widths, and the 1.58-bit Qwen3-0.6B-EdgeRazor reduces storage from 1.11 GB to 0.19 GB while accelerating decoding by 15.16

\times

over the 16-bit baseline. These results empirically validate the effectiveness and efficiency of EdgeRazor. The codes can be accessed from GitHub and Huggingface.

Shu-Hao Zhang, Le-Tong Huang, Xiang-Sheng Deng +5

National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210063, China · School of Intelligent Science and Technology, Nanjing University, Suzhou 215163, China · Microsoft AI, Beijing 100080, China +1

LLM Quantization LLM Compression Mixed-Precision Quantization Quantization-Aware Training LLM Inference Acceleration Knowledge Distillation

Ask PDF

Aug 24, 2026

FAMPWQ: Fisher Information-based Adaptive Mixed Precision Weight Quantization for Effective LLM Inference

Recent years have witnessed remarkable achievements of Large Language Models (LLMs) in multiple domains, while the excessive resource requirements of LLMs hinder the deployment on resource-constrained devices. Although model quantization stands out as an effective approach, conventional quantization approaches typically incur severe performance degradation due to uniform bit-width or simple heuristic sensitivity evaluation. In this paper, we propose a novel Fisher information-based Adaptive Mixed Precision Weight Quantization approach, i.e., FAMPWQ, which performs layer-adaptive weight quantization for effective LLM inference on commodity GPUs. First, we propose a system model with a novel Fisher information metric to measure the layer-wise sensitivity to quantization. Second, we propose a reinforcement learning-based bit-width allocator in FAMPWQ, which generates an adaptive bit-width allocation strategy based on the Fisher information sensitivity metric. Extensive experiments on 7 models and 5 benchmarks demonstrate that FAMPWQ significantly outperforms 7 baseline approaches in terms of PPL (up to 3.39 smaller), accuracy (up to 6.87% higher), and LLM-as-a-judge comparison (up to 76% win rate).

Gongwei Lee, Ji Liu, Juncheng Jia +1

School of Computer Science and Technology, Soochow University, Suzhou, China · Hithink Research, Hangzhou, China · Electronic Engineering, Tsinghua University, Beijing, China

Mixed-Precision Quantization LLM Quantization LLM Inference Acceleration Weight-Only Quantization Neural Network Quantization

Ask PDF