cs.LGNov 5, 2026

ACSeL: Adaptive Configuration Search for Hardware-Constrained LLM Inference

Authors: Bokun Wang, Hung-Yueh Chiang, Diana Marculescu

Organizations: University of Texas at Austin

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Serving large language models (LLMs) under tight memory and latency service-level objectives (SLOs), including on edge devices, requires jointly selecting the model size, batch size, quantization algorithm, and mixed-precision quantization recipes. Exhaustive search over this combinatorially large space is infeasible. Unlike data-center serving, which often prioritizes throughput or generation speed, resource-constrained deployments typically set memory and latency budgets as constraints and then seek the configuration with the highest generation quality among feasible ones. To this end, we propose ACSeL, a framework that identifies configurations predicted to satisfy deployment constraints while achieving high predicted generation quality. To enable efficient quality estimation, we use Relative Model Quality Degradation (RMQD), a proxy based on the divergence between the candidate and reference models' next-token distributions over a small calibration corpus. Across the evaluated model sizes and precisions, RMQD is inexpensive to evaluate and strongly correlated with downstream benchmark performance. ACSeL combines (i) a lightweight quality surrogate that predicts RMQD directly from a configuration; (ii) Few-shot Median Calibration (FMC), which calibrates operator-profile-based latency estimates using a small number of end-to-end profiles; and (iii) a structured search algorithm BandPareto for exploring layerwise mixed-precision configurations. Experiments on an NVIDIA RTX A6000 workstation and an NVIDIA Jetson Thor edge platform show that FMC substantially improves deployment feasibility, while BandPareto + FMC achieves the highest feasibility and the lowest RMQD gaps to the best measured configurations in the evaluation set.

Figures & tables

Figure 1: Overview of ACSeL , which uses quality and latency surrogates to guide the search over the configuration space and returns the predicted-feasible configuration with the highest predicted generation quality for the given deployment request.
Figure 2: Spearman correlation ( ρ\rho ) between accuracy and RMQD for Qwen3 models with both uniform-precision and mixed-precision quantization recipes across three benchmarks from NeMo Evaluator. RMQD is fairly strongly negatively correlated with accuracy on all benchmarks ( ρ=−0.72\rho=-0.72 to −0.85-0.85 ), capturing the overall quality trend across model sizes and precisions. Dashed lines show linear fits.
Figure 3: Quality–latency headrooms on A6000 (vLLM) and Thor (SGLang) of band vs. uniform in the evaluation dataset, with llama-3.2-1b at batch size 4, isl 2048, osl 64.
 Unconstrained ridgeConstrained ridge (Ours)
MdAPE (%) ↓\downarrow3.32.6
Table 3: Quality-surrogate RMQD prediction error on held-out configurations. Lower MdAPE is better.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

η\etaFeas.@ 5%5\% ( ↑\uparrow )Feas.@ 10%10\% ( ↑\uparrow )# CandidatesWall-Clock Time (s)
0%0\%68%80%242.2
5%5\%72%84%433.5
10%10\%72%84%614.5
15%15\%72%84%765.6
25%25\%72%84%967.2
Appendix
Table 4: Measured deployment feasibility, number of evaluated candidates after FracMix, and wall-clock time (per request) of BandPareto with η=0%,5%,10%,15%,25%\eta=0\%,5\%,10\%,15\%,25\% .
#BackendModel FamilyISLOSLL1L_{1}L2L_{2}
0sglangLlama-3.2819264276.819.60
1sglangLlama-3.28192256303.915.00
2vllmQwen3204864577.535.63
3sglangQwen381925121066.138.73
4vllmQwen351264157.631.25
5sglangQwen381922561036.729.38
Appendix
Table 5: Requests on A6000.
#BackendModel FamilyISLOSLL1L_{1}L2L_{2}
0sglangLlama-3.2819264888.4111.99
1vllmLlama-3.2819264692.5456.97
2vllmLlama-3.2819264699.5110.17
3sglangLlama-3.2204864672.999.66
4vllmLlama-3.28192512700.7175.55
5vllmQwen38192641030.0150.84
Appendix
Table 6: Requests on Thor.
More in this topic

LLM Inference

Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.

See all 297 papers

Explore similar work

CardsList
Oct 11, 2026

The Price of Precision: A Calibrated Cost Model for Mixed-Precision LLM Serving

Choosing a mixed-precision configuration for large language model (LLM) serving trades speed against accuracy. Allocators that use average bit-width or a memory budget as a speed proxy can miss changes in the resource that limits execution. We argue that served speed can be limited by compute, weight loading, or key-value (KV) cache access, depending on the phase, batch size, and context length. A byte-based proxy does not capture all of these costs, so fewer bits do not always mean faster serving. To address this gap, we build a cost model that accounts for the binding resource and use it in two ways. First, as a predictor: given a configuration and operating point, it estimates serving latency using constants calibrated for the model, GPU, and serving stack. We evaluate its accuracy across three hardware classes. Second, as an allocator: we combine modeled time costs with measured per-tensor accuracy costs and use a dynamic program to select a configuration under a latency or accuracy budget. We compare its configurations against byte-aware and sensitivity-only allocation.

Mukund Athmaram, Manoj Rao, Liang Ma

Advanced Micro Devices Inc. (AMD) · Cornell University

LLM ServingLLM Inference AccelerationCost-Aware InferenceMixed-Precision Quantization
AskPDF
Apr 8, 2026

ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge--Cloud Speculative LLM Serving

Speculative decoding enables collaborative Large Language Model (LLM) inference across cloud and edge by separating lightweight token drafting from heavyweight verification. While prior systems show performance and cost benefits, practical deployment requires navigating a large configuration space spanning draft model variants, quantisation levels, speculative lengths, and heterogeneous edge devices. This paper presents ConfigSpec, a configurationselection framework for distributed speculative LLM serving. ConfigSpec profiles edge devices and draft-target alignment, and models drafting throughput, acceptance rate, and power to evaluate goodput, verification cost efficiency, and energy efficiency across the joint configuration space. Our analysis across three edge platforms and two LLM families reveals structurally conflicting optima. Firstly, goodput is maximised by the smallest, fastest draft model at device-dependent speculative lengths (K*=2-10). Secondly, both cost and energy efficiency converge to K=2 due to a dominant bonus-token effect-with cost favouring the largest drafter for its high acceptance rate and energy favouring the smallest for its low power draw. These conflicts confirm that no single fixed configuration can simultaneously optimise all objectives, underscoring the need for profiling-based configuration selection in disaggregated edge-cloud LLM inference.

Xiangchen Li, Saeid Ghafouri, Jiakun Fan +3

Virginia Tech, Blacksburg, Virginia, USA · Queen’s University Belfast, Belfast, Northern Ireland, UK

LLM Inference EfficiencyDisaggregated LLM ServingInference-Time OptimizationSpeculative DecodingEdge Computing
AskPDF
Oct 5, 2026

Stepped MoE: Segment-Level Routing with Configurable Inference Complexity

Training large language models (LLMs) is resource-intensive, and adapting them for diverse deployment scenarios with varying computational constraints remains challenging. While elastic architectures enable flexible model deployment and sparsely activated models allow input-adaptive computation, existing approaches treat these dimensions independently. Moreover, models catered towards on-device edge inference need to conform to the memory and compute limitations of the serving devices. In this paper, we introduce a unified framework that combines elastic structures with sparsely gated architectures to create models that adapt simultaneously to both deployment constraints and task requirements. Our approach employs a model backbone that conditions on both the context and target efficiency specifications, enabling fine-grained control over the accuracy-efficiency trade-off at inference time. The model learns to activate task-relevant parameters within elastically-nested sub-networks, allowing a single model to span multiple capacity points while maintaining input-adaptive routing. Through experiments we demonstrate that we can create a model that allows the flexibility to use 1,2,3,4 billion parameters while being more accurate than their dense counter-parts (2-5% on knowledge-intensive benchmarks) and at par with their static versions while delivering similar latency metrics as dense models. Overall, we save on device disk space by sharing the model parameters, allow flexibility of serving based on DRAM and compute available while delivering more accurate results.

Arnav Kundu, Zhaoyang Xu, Bairu Hou +3

Apple

On-Device Language Model InferenceDynamic Neural NetworksMixture of ExpertsElastic Neural NetworksLLM RoutingEfficient Language Model Inference
AskPDF