Advancing Model Refinement: Muon-Optimized Distillation and Quantization for LLM Deployment
Organizations: Intelligent Systems and Robotics (ISR) University of West Florida · DEVCOM Army Research Laboratory (ARL)
Abstract
Correction notice (v2). Two implementation issues affect these experiments. All LoRA training ran under DeepSpeed, which replaced the optimizer passed to the Trainer with AdamW at a constant learning rate, so the Muon and Adam arms used the same optimizer. The cross-entropy term was computed without the next-token shift, so the preference for alpha = 1 reflects that objective rather than the value of the teacher. Each comparison also uses a single GPTQ quantization, and repeated quantizations differ by up to several points. The findings on Muon's quantization robustness, the distillation weight and accuracy relative to GPTQ alone should not be relied on. See the notice on page 1. Large Language Models (LLMs) enable advanced natural language processing but face deployment challenges on resource-constrained edge devices due to high computational, memory, and energy demands. Optimizing these models requires addressing three key challenges: acquiring task-specific data, fine-tuning for performance, and compressing models to accelerate inference while reducing resource demands. We propose an integrated framework combining GPTQ-based quantization, low-rank adaptation (LoRA), and a specialized data distillation process to significantly reduce model size and complexity while preserving or enhancing task-specific performance. By leveraging data distillation, knowledge distillation via Kullback-Leibler divergence, Bayesian hyperparameter optimization, and the Muon optimizer, our pipeline achieves up to 2x memory compression (e.g., reducing a 6GB model to 3GB) and enables efficient inference for specialized tasks. Empirical results demonstrate superior performance on standard LLM benchmarks compared to GPTQ quantization alone, with the Muon optimizer notably enhancing fine-tuned models' resistance to accuracy decay during quantization.
Figures & tables
| Task | Llama4Scout | Llama3.3 70B | Llama3.2 3B |
| ARC-e* | .8384 | .8283 | .7100 |
| CQA | .8239 | .8346 | .7371 |
| HS* | .8265 | .8519 | .7162 |
| MMLU | .7987 | .8221 | .6069 |
| OBQA* | .4580 | .4640 | .3940 |
| PIQA* | .8090 | .8471 | .7682 |
| Task | Full Name | Reference |
| ARC-e | ARC (Easy) | ( Clark et al., 2018 ) |
| CQA | CommonsenseQA | ( Talmor et al., 2018 ) |
| HS | HellaSwag | ( Zellers et al., 2019 ) |
| MMLU | MMLU | ( Hendrycks et al., 2021 ) |
| OBQA | OpenBookQA | ( Mihaylov et al., 2018 ) |
| PIQA | Physical Interaction QA | ( Bisk et al., 2020 ) |
| Variable | Description | Search Space |
| LoRA scaling | ||
| LoRA rank | ||
| Learning rate | ||
| Distillation temperature | ||
| Distillation loss weight | ||
| Weight decay |
| Task | Llama3.2-3B | Adam-LoRA ( ; max) | Muon-LoRA ( ; max) |
| ARC-e* | .7100 | .6951 .0034; .7125 | .7106 .0034; .7138 |
| CQA | .7371 | .7369 .0027; .7412 | .7351 .0084; .7420 |
| HS* | .7162 | .7112 .0079; .7203 | .7152 .0071; .7220 |
| MMLU | .6069 | .5232 .1551; .6063 | .3973 .1937; .6070 |
| OBQA* | .3940 | .3972 .0048; .4020 | .3928 .0044; .3980 |
| PIQA* | .7682 | .7675 .0082; .7731 | .7712 .0023; .7737 |
| Task | Llama3.2-3B-GPTQ ( ) | Adam-LoRA-GPTQ mean std max ( ) | Muon-LoRA-GPTQ mean std max ( ) |
| ARC-e* | .6932 (-.0168) | .6793 .0037 .7058 (-.0158) | .7004 .0059 .7071 ( -.0102 ) |
| CQA | .7265 (-.0106) | .7265 .0043 .7314 ( -.0104 ) | .7081 .0087 .7199 (-.0270) |
| HS* | .7064 ( -.0098 ) | .6993 .0065 .7060 (-.0119) | .7037 .0062 .7095 (-.0115) |
| MMLU | .5787 (-.0282) | .5059 .1456 .5853 (-.0173) | .3818 .1860 .5886 ( -.0155 ) |
| OBQA* | .3800 (-.0140) | .3916 .0057 .3960 (-.0056) | .3884 .0097 .4040 ( -.0044 ) |
| PIQA* | .7622 ( -.0060 ) | .7615 .0110 .7704 ( -.0060 ) | .7640 .0028 .7682 (-.0072) |
| Task | Optimizer | Rank | LoRA Scale | LearningRate | Distill | Distill T | W Decay | EvalLoss |
| ARC-e | Adam | 48 | 0.5 | 3.77e-4 | 1.0 | 0.51 | 0.01 | 0.079 |
| ARC-e | Muon | 16 | 1.5 | 4.17e-4 | 1.0 | 2.01 | 0.02 | 0.950 |
| CQA | Adam | 64 | 2 | 4.99e-4 | 1.0 | 0.51 | 0.04 | 0.115 |
| CQA | Muon | 24 | 1.75 | 4.91e-4 | 1.0 | 0.51 | 0.1 | 0.115 |
| HS | Adam | 24 | 1.75 | 2.34e-4 | 1.0 | 0.51 | 0.08 | 0.146 |
| HS | Muon | 48 | 2 | 4.94e-4 | 1.0 | 6.01 | 0.03 | 1.947 |
| Task | W-Kurt Adam | W-Kurt Muon | W | A-Kurt Adam | A-Kurt Muon | A |
| arceasy | 4.6038 | 4.6038 | 0.0000 | 159.3506 | 160.5997 | 1.2491 |
| cqa | 4.6038 | 4.6038 | 0.0000 | 157.4510 | 157.4692 | 0.0182 |
| hellaswag | 4.6038 | 4.6038 | 0.0000 | 157.4741 | 158.2883 | 0.8142 |
| mmlu | 4.6036 | 4.6036 | 0.0000 | 151.1230 | 148.5278 | -2.5953 |
| openbookqa | 4.6038 | 4.6038 | -0.0000 | 155.7417 | 155.1853 | -0.5565 |
| piqa | 4.6038 | 4.6038 | -0.0000 | 159.4547 | 159.3097 | -0.1449 |
| Model | Size (GB) | Throughput (Tok/s) | TPOT (ms/tok) | ITL (ms/tok) |
| Pre-Quant | 6.01 | 1387.64 | 17.49 | 17.54 |
| Post-Quant | 2.86 | 1722.82 | 8.82 | 8.82 |
LLM Compression
Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.
See all 301 papers