cs.AIJan 13, 2025

FlexQuant: Elastic Quantization Framework for Locally Hosted LLM on Edge Devices

Authors: Yuji Chai, Mujin Kwun, David Brooks, Gu-Yeon Wei

Organizations: Harvard John A. Paulson School Of Engineering And Applied Sciences Cambridge, Massachusetts, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Deploying LLMs on edge devices presents serious technical challenges. Memory elasticity is crucial for edge devices with unified memory, where memory is shared and fluctuates dynamically. Existing solutions suffer from either poor transition granularity or high storage costs. We propose FlexQuant, a novel elasticity framework that generates an ensemble of quantized models, providing an elastic hosting solution with 31x more deployment options, 15x granularity improvement, and 10x storage reduction compared to SoTA methods. FlexQuant works with most quantization methods and creates a family of trade-off options under various storage limits through our pruning method. It brings great performance and flexibility to the edge deployment of LLMs.

Figures & tables

Fig. 1: Perplexity comparison of different elastic hosting methods of quantized Llama 22 77 B model. It compares FlexQuant with the baseline method when they are using two quantization methods, ExLlamaV2 and AnyPrecision.
Fig. 2: An example of FlexQuant’s tree search for EQM ensemble. The shown search process has two exploitation stems and three exploration branches.
Fig. 3: Perplexity comparison between Base-Ex, Base-AP, FQ-Ex, and PFQ-Ex at different pruning rate. Results for AnyPrecision on Llama 3 8B is omitted due to lack of support in their implementation. The footprint range is different due to differences in parameter count and model architecture.
Fig. 4: Perplexity vs pruning rate at varying memory footprint bounds for FQ-Ex on Llama 1, Llama 2 and Llama 3
Fig. 5: Downstream perplexity comparison of quantized Llama models between Base-Ex FQ-Ex and PFQ-Ex at different pruning rate.
TaskarcChallengearc_{Challenge}arcEasyarc_{Easy}hellaswaghellaswagpiqapiqawinograndewinogrande
PolicyBase-ExFQ-ExPFQ-ExBase-ExFQ-ExPFQ-ExBase-ExFQ-ExPFQ-ExBase-ExFQ-ExPFQ-ExBase-ExFQ-ExPFQ-Ex
Storage177 GB17 GB13 GB177 GB17 GB13 GB177 GB17 GB13 GB177 GB17 GB13 GB177 GB17 GB13 GB
Memory Footprint (GB)3.037.40-1.40-1.4068.98+1.73+1.7351.47-0.60-0.8075.63+0.15+0.5466.61+0.24-1.10
3.539.42+2.04+1.0872.05+2.57+1.7753.21+3.02+1.8776.22+1.14+0.6066.69+2.22+1.34
4.041.38+1.19+1.1175.01+0.35+0.2055.81+0.76+0.5577.04+0.76+0.7168.27+1.10+0.40
4.543.17-0.26-0.4375.46+0.00-0.2555.81+0.93+0.7777.75+0.26+0.0069.37+0.32-0.39
TABLE I: Downstream task accuracy comparison between Base-Ex, FQ-Ex and PFQ-Ex with 50% pruning.
Fig. 6: Hyperparameters’ impact on the overall search algorithm’s performance. In the left and middle plots, we sweep between different numbers of stems and the number of branches. The right plot shows the impact of the middle model. In the middle and right plots, the lighter the color is, the better the quality of the search result.
Fig. 7: First token latency and decoding throughput overhead of FlexQuant framework. The Baseline is Base-Ex, indicated by the grey bar. The FlexQuant is FQ-Ex, indicated by the light blue bar.
Fig. 8: Memory usage collected on a Samsung S23 Ultra, using the Android 14 operating system. Three collections of workload traces are collected based on the number of active apps on the cellphone. Their memory usage histograms are presented from left to right.
Fig. 9: Model quality comparison under different workload scenarios. ”AP with 6.0GB” or ”FQ with 6.0GB” means AnyPrecision or FlexQuant deployed on a device with 6GB of total system memory. The lighter the color is, the better the model quality.
Fig. 10: LLM launch speed comparison between FlexQuant (FP-Ex) and AnyPrecision (Base-AP). The larger the speedup is, the better the performance.

Explore similar work

CardsList

  1. LLM Compression with Jointly Optimizing Architectural and Quantization choices

    Jun 2, 2026Hoang-Loc La, Truong-Thanh Le, Amir Taherkordi +1Large Language Model CompressionLLM Inference Optimization

    UiT The Arctic University of Norway · University of Oslo, Norway

  2. E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environments

    Jun 2, 2026Truong-Thanh Le, Amir Taherkordi, Hoang-Loc La +3LLM Inference OptimizationEdge Devices

    Department of Informatics University of Oslo Oslo, Norway · Department of Computer Science UiT The Arctic University of Norway Tromsø, Norway

  3. SelectInfer: Selective Neuron Loading and Computation for On-Device LLMs

    Jul 20, 2026Huzaifa Shaaban Kabakibo, Eric Schniedermeyer, Artem Burchanow +1LLM Inference OptimizationNeurons

    Department of Computer Networks, Paderborn University, Paderborn, Germany