Correction notice (v2). Two implementation issues affect these experiments. All LoRA training ran under DeepSpeed, which replaced the optimizer passed to the Trainer with AdamW at a constant learning rate, so the Muon and Adam arms used the same optimizer. The cross-entropy term was computed without the next-token shift, so the preference for alpha = 1 reflects that objective rather than the value of the teacher. Each comparison also uses a single GPTQ quantization, and repeated quantizations differ by up to several points. The findings on Muon's quantization robustness, the distillation weight and accuracy relative to GPTQ alone should not be relied on. See the notice on page 1. Large Language Models (LLMs) enable advanced natural language processing but face deployment challenges on resource-constrained edge devices due to high computational, memory, and energy demands. Optimizing these models requires addressing three key challenges: acquiring task-specific data, fine-tuning for performance, and compressing models to accelerate inference while reducing resource demands. We propose an integrated framework combining GPTQ-based quantization, low-rank adaptation (LoRA), and a specialized data distillation process to significantly reduce model size and complexity while preserving or enhancing task-specific performance. By leveraging data distillation, knowledge distillation via Kullback-Leibler divergence, Bayesian hyperparameter optimization, and the Muon optimizer, our pipeline achieves up to 2x memory compression (e.g., reducing a 6GB model to 3GB) and enables efficient inference for specialized tasks. Empirical results demonstrate superior performance on standard LLM benchmarks compared to GPTQ quantization alone, with the Muon optimizer notably enhancing fine-tuned models' resistance to accuracy decay during quantization.
Figures & tables
Figure 1: Pipeline Overview
Figure 2: Self-Instruct Pipeline Detail
Task
Llama4Scout
Llama3.3 70B
Llama3.2 3B
ARC-e*
.8384
.8283
.7100
CQA
.8239
.8346
.7371
HS*
.8265
.8519
.7162
MMLU
.7987
.8221
.6069
OBQA*
.4580
.4640
.3940
PIQA*
.8090
.8471
.7682
Table 1: Accuracies of Pre-Trained Models T1 (Llama4-Scout), T2 (Llama3.3-70B-Instruct), and S1 (Llama 3.2 3B Instruct)
Task
Full Name
Reference
ARC-e
ARC (Easy)
( Clark et al., 2018 )
CQA
CommonsenseQA
( Talmor et al., 2018 )
HS
HellaSwag
( Zellers et al., 2019 )
MMLU
MMLU
( Hendrycks et al., 2021 )
OBQA
OpenBookQA
( Mihaylov et al., 2018 )
PIQA
Physical Interaction QA
( Bisk et al., 2020 )
Table 2: Eight language benchmarks considered
Variable
Description
Search Space
a
LoRA scaling
[0.5,2.0]
r
LoRA rank
[8,64]
η
Learning rate
[10−5,10−3]
T
Distillation temperature
[0.5,8]
α
Distillation loss weight
[0,1]
wd
Weight decay
[0,2]
Table 3: Hyperparameter definitions and search spaces used for Bayesian optimization.
Task
Llama3.2-3B
Adam-LoRA ( μ±σ ; max)
Muon-LoRA ( μ±σ ; max)
ARC-e*
.7100
.6951 ± .0034; .7125
.7106 ± .0034; .7138
CQA
.7371
.7369 ± .0027; .7412
.7351 ± .0084; .7420
HS*
.7162
.7112 ± .0079; .7203
.7152 ± .0071; .7220
MMLU
.6069
.5232 ± .1551; .6063
.3973 ± .1937; .6070
OBQA*
.3940
.3972 ± .0048; .4020
.3928 ± .0044; .3980
PIQA*
.7682
.7675 ± .0082; .7731
.7712 ± .0023; .7737
Table 4: Comparison of Pre-Trained and Fine-Tuned Accuracies (Mean ± Std; Max over 5 HPO replicates)
Task
Llama3.2-3B-GPTQ ( Δ )
Adam-LoRA-GPTQ mean ± std max ( Δ )
Muon-LoRA-GPTQ mean ± std max ( Δ )
ARC-e*
.6932 (-.0168)
.6793 ± .0037 .7058 (-.0158)
.7004 ± .0059 .7071 ( -.0102 )
CQA
.7265 (-.0106)
.7265 ± .0043 .7314 ( -.0104 )
.7081 ± .0087 .7199 (-.0270)
HS*
.7064 ( -.0098 )
.6993 ± .0065 .7060 (-.0119)
.7037 ± .0062 .7095 (-.0115)
MMLU
.5787 (-.0282)
.5059 ± .1456 .5853 (-.0173)
.3818 ± .1860 .5886 ( -.0155 )
OBQA*
.3800 (-.0140)
.3916 ± .0057 .3960 (-.0056)
.3884 ± .0097 .4040 ( -.0044 )
PIQA*
.7622 ( -.0060 )
.7615 ± .0110 .7704 ( -.0060 )
.7640 ± .0028 .7682 (-.0072)
Table 5: Comparison of Pre-Trained-GPTQ and Fine-Tuned-GPTQ accuracies (mean ± std and max over 5 HPO runs; Δ from pre-compression mean).
Task
Optimizer
Rank
LoRA Scale
LearningRate
Distill α
Distill T
W Decay
EvalLoss
ARC-e
Adam
48
0.5
3.77e-4
1.0
0.51
0.01
0.079
ARC-e
Muon
16
1.5
4.17e-4
1.0
2.01
0.02
0.950
CQA
Adam
64
2
4.99e-4
1.0
0.51
0.04
0.115
CQA
Muon
24
1.75
4.91e-4
1.0
0.51
0.1
0.115
HS
Adam
24
1.75
2.34e-4
1.0
0.51
0.08
0.146
HS
Muon
48
2
4.94e-4
1.0
6.01
0.03
1.947
Table 6: Quasi-optimal hyperparameters from a single LoRA HPO Replicate. Note: Hyperparameters optimized via Optuna HPO using 16 samples
Task
W-Kurt Adam
W-Kurt Muon
Δ W
A-Kurt Adam
A-Kurt Muon
Δ A
arceasy
4.6038
4.6038
0.0000
159.3506
160.5997
1.2491
cqa
4.6038
4.6038
0.0000
157.4510
157.4692
0.0182
hellaswag
4.6038
4.6038
0.0000
157.4741
158.2883
0.8142
mmlu
4.6036
4.6036
0.0000
151.1230
148.5278
-2.5953
openbookqa
4.6038
4.6038
-0.0000
155.7417
155.1853
-0.5565
piqa
4.6038
4.6038
-0.0000
159.4547
159.3097
-0.1449
Table 7: Weighted mean layer-wise kurtosis of weights (W) and activation channels (A): Adam vs Muon measured with 128 samples of calibration data
Figure 3: Throughput Comparison: Pre- and Post- Compression
Model
Size (GB)
Throughput (Tok/s)
TPOT (ms/tok)
ITL (ms/tok)
Pre-Quant
6.01
1387.64
17.49
17.54
Post-Quant
2.86
1722.82
8.82
8.82
Table 8: Throughput and latency comparison: Pre- and Post- Quantization. Used to plot figure 3 Note: Metrics measured on 1x Ampere A40 GPU, deployed with vLLM, using 1000 prompts with input 1024 length, output length 1024, max concurrency 8
Quantization has emerged as a mainstream approach for deploying Large Language Models (LLMs) on resource-constrained devices, yet compressing precision below 4-bit typically causes severe performance degradation or prohibitive retraining costs. In this paper, we propose EdgeRazor, a lightweight framework for LLMs via Mixed-Precision Quantization-Aware Distillation. It contains three modules: Structural Quantization with Mixed Precision for fine-grained control of bit-widths, Layer-Adaptive Feature Distillation that dynamically selects the most informative features for alignment, and Entropy-Aware KL Divergence for forward-reverse balance on both human-annotated and distilled datasets. Evaluations conducted on MobileLLM and Qwen families show that under weight-activation quantization, the 1.88-bit Qwen3-0.6B-EdgeRazor outperforms the state-of-the-art 2-bit baselines by 11.27 and surpasses the strongest 3-bit baselines by 4.38, while the quantized MobileLLM-350M-EdgeRazor requires a training budget 4-10× lower than the leading quantization-aware training method. In terms of efficiency, EdgeRazor achieves higher compression ratios at all bit-widths, and the 1.58-bit Qwen3-0.6B-EdgeRazor reduces storage from 1.11 GB to 0.19 GB while accelerating decoding by 15.16× over the 16-bit baseline. These results empirically validate the effectiveness and efficiency of EdgeRazor. The codes can be accessed from GitHub and Huggingface.
Shu-Hao Zhang, Le-Tong Huang, Xiang-Sheng Deng +5
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210063, China · School of Intelligent Science and Technology, Nanjing University, Suzhou 215163, China · Microsoft AI, Beijing 100080, China +1
Quantization reduces the deployment cost of large language models (LLMs). However, post-training quantization (PTQ) cannot fine-tune model parameters and often degrades accuracy in low-bit settings, while quantization-aware training (QAT) requires backpropagation and incurs high memory overhead. We propose ZeroQAT, a zeroth-order QAT framework supporting both weight and activation quantization. By using forward-only gradient estimation, ZeroQAT removes backpropagation and substantially reduces memory and computation while retaining end-to-end optimization benefits. We further introduce a lightweight variant of ZeroQAT for quantized fine-tuning, which freezes and pre-quantizes most parameters to further cut memory usage. Experiments show that ZeroQAT consistently outperforms representative PTQ and QAT baselines while requiring significantly less memory. For example, ZeroQAT enables fine-tuning of a 13B model at extremely low bit-widths (e.g., 2-4 bits) on a single 8GB GPU, and even allows fine-tuning a 6.7B model on a OnePlus 12 smartphone, demonstrating its practicality for end-to-end QAT on resource-limited edge devices.
Qitao Tan, Xiaoying Song, Ningxi Cheng +11
University of Georgia · University of North Texas · Northeastern University +3
Deploying LLMs on edge devices presents serious technical challenges. Memory elasticity is crucial for edge devices with unified memory, where memory is shared and fluctuates dynamically. Existing solutions suffer from either poor transition granularity or high storage costs. We propose FlexQuant, a novel elasticity framework that generates an ensemble of quantized models, providing an elastic hosting solution with 31x more deployment options, 15x granularity improvement, and 10x storage reduction compared to SoTA methods. FlexQuant works with most quantization methods and creates a family of trade-off options under various storage limits through our pruning method. It brings great performance and flexibility to the edge deployment of LLMs.
Yuji Chai, Mujin Kwun, David Brooks +1
Harvard John A. Paulson School Of Engineering And Applied Sciences Cambridge, Massachusetts, USA