Large Language Models (LLMs) have achieved remarkable success across diverse applications, yet their deployment remains challenging due to substantial computational costs, memory requirements, and energy consumption. Recent empirical studies have demonstrated that no single efficiency technique is universally optimal; instead, the effectiveness of methods such as efficient attention mechanisms, mixture-of-experts (MoE), parameter-efficient fine-tuning, and quantization varies significantly depending on task characteristics, resource constraints, and model scales. Building upon these insights, we propose AE-LLM, a unified framework that automatically selects and combines optimal efficiency techniques tailored to specific deployment scenarios. Our approach introduces a multi-objective optimization framework that jointly considers accuracy, latency, memory footprint, and energy consumption, while accounting for hardware constraints and task requirements. We develop an efficient search algorithm that explores the combinatorial space of efficiency techniques across architecture, fine-tuning, and inference stages, identifying Pareto-optimal configurations. Extensive experiments across 15 models (0.5B-70B parameters) and 10 diverse tasks demonstrate that AE-LLM achieves an average of 2.8× improvement in efficiency metrics while maintaining competitive accuracy (within 1.2% of baseline), compared to static efficiency configurations. Furthermore, our framework generalizes effectively to vision-language models, achieving similar efficiency gains. Our contributions provide practitioners with an automated tool for navigating the complex trade-off landscape of LLM efficiency optimization.
{FP16, FP8, INT8, INT4} via {GPTQ, AWQ, SmoothQuant}
KV Cache
{Full, MQA-style, GQA-style}
Table 1: AE-LLM Configuration Space Design.
Model
Method
Accuracy (%)
Latency (ms)
Memory (GB)
Energy (J)
Efficiency Score
Small Models (0.5B–2B)
LLaMA-2-1B
Default
43.2
12.5
2.1
0.08
1.00
Best Single-Stage
42.8
9.2
1.8
0.06
1.31
Manual Selection
42.5
8.5
1.6
0.05
1.45
EfficientLLM Rec.
42.1
8.8
1.7
0.05
1.38
AdaptiveEfficientLLM
43.0
7.8
1.4
0.04
1.72
Table 2: Main results comparing AdaptiveEfficientLLM against baselines. We report accuracy (higher is better), latency in milliseconds (lower is better), memory in GB (lower is better), and energy in Joules (lower is better). Best results are in bold , second best are underlined . Efficiency Score is the geometric mean of normalized efficiency metrics, higher is better.
Configuration
Efficiency Score
Rel. Improvement
Search Algorithm Components
Full AdaptiveEfficientLLM
1.95
+95%
- Predictive Models (random search)
1.62
+62%
- Constraint-Aware Pruning
1.92
+92%
- Hierarchical Crossover
1.85
+85%
- Refinement Iterations
1.82
+82%
Table 3: Ablation studies on LLaMA-2-7B. We report the efficiency score and relative improvement compared to the default configuration.
Model
Method
Accuracy
Lat. (ms)
Mem. (GB)
En. (J)
VQAv2 Benchmark
LLaVA-1.5-7B
Default
78.5
85.2
18.5
1.25
EfficientLLM Rec.
77.8
68.5
14.2
0.95
AdaptiveEfficientLLM
78.2
52.8
10.8
0.72
InternVL-Chat
Default
81.2
92.5
22.5
1.42
EfficientLLM Rec.
80.5
75.8
17.5
1.15
Table 4: Cross-modal generalization results on vision-language models. We report accuracy (VQA score for VQAv2, CIDEr for COCO Caption, accuracy for TextVQA) and efficiency metrics.
Figure 1: Distribution of optimal configuration choices across tasks and hardware platforms. The choice of efficiency techniques varies significantly with task type and hardware constraints, validating the need for adaptive optimization.
Figure 2: Pareto fronts showing accuracy-latency trade-offs for different models. AE-LLM identifies configurations across the full spectrum of trade-offs, enabling practitioners to select based on their specific requirements.
Figure 3: Scatter plot showing the relationship between efficiency improvements and accuracy changes. Different efficiency techniques exhibit distinct trade-off patterns.
Figure 4: Sensitivity of performance metrics to key hyperparameters. The shaded regions indicate the range of performance across different tasks.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
XGBoost Regressor
Value
NSGA-II Search
Value
Estimators / Max Depth
500 / 8
Population Size / Gen
100 / 50
Learning Rate
0.05
Crossover / Mutation
0.9 / Variable
Subsample / Colsample
0.8 / 0.8
Tournament Size
3
Appendix
Table 5: Hyperparameter settings for surrogate models and NSGA-II search.
Model
Method
MMLU
HellaS.
ARC-E
GSM8K
HumanE.
Alpaca
LongB.
Needle
MT-B
Vicuna
Avg.
LLaMA-2-7B
Default
46.8
78.2
72.5
14.5
12.8
85.2
32.5
88.5
6.2
78.5
51.6
Best Single-Stage
46.2
77.8
72.0
14.0
12.2
84.5
31.8
87.8
6.0
77.8
51.0
Manual Selection
45.8
77.2
71.5
13.5
11.8
84.0
31.2
87.2
5.8
77.2
50.5
EfficientLLM Rec.
46.0
77.5
71.8
13.8
12.0
84.2
31.5
87.5
5.9
77.5
50.8
AdaptiveEfficientLLM
46.5
78.0
72.2
14.2
12.5
85.0
32.2
88.2
6.1
78.2
51.3
Mistral-7B
Default
62.5
82.8
78.2
37.5
26.2
92.5
38.5
92.8
7.5
85.2
60.4
Appendix
Table 6: Detailed results across all tasks for selected models. We report task-specific accuracy metrics.
We present EnergyLens, an end-to-end framework for energy-aware large language model (LLM) inference optimization. As LLMs scale, predicting and reducing their energy footprint has become critical for sustainability and datacenter operations, yet existing approaches either require production-level code and expensive profiling or fail to accurately capture multi-GPU energy behavior. As a result, practitioners lack tools for deciding which optimizations to prioritize and for selecting among existing deployment configurations when exhaustive profiling is impractical. EnergyLens addresses this gap with an intuitive einsum-based interface that captures LLM specifications including fusion, parallelism, and compute-communication overlap, combined with load-imbalance-aware MoE modeling and an empirically driven communication energy model for multi-GPU settings. We validate EnergyLens on Llama3 and Qwen3-MoE across tensor-parallel and expert-parallel configurations, achieving mean absolute percentage errors (MAPEs) between 9.25% and 13.19% for multi-GPU prefill and decode energy, and 12.97% across SM allocations for Megatron-style overlap. Our energy-driven exploration reveals up to 1.47x and 52.9x energy variation across configurations in prefill and decode efficiency and motivates distributed serving. We further show that compute-communication overlap is difficult to optimize with intuition alone, but EnergyLens correctly identifies Pareto-optimal overlap configurations.
Zhiye Song, Kyungmi Lee, Eun Kyung Lee +3
Massachusetts Institute of Technology Cambridge, MA 02139 · IBM Research Yorktown Heights, NY 10598
Large Language Models (LLMs) have become an integral part of many real-world workflows. However, LLMs consume a lot of energy, which becomes a large concern in the scale of the demand for these tools. As LLMs become integrated into different workflows, different applications have arisen to deal with the challenge of running inference for these tools. This raises another issue of choosing the runtime parameter values for these services in order to minimize the energy consumption. Oftentimes this requires deep knowledge of the application or traditional optimization methods that can take days to find optimal values. In this work, we created a human-in-the-loop flow with LLM-assisted runtime parameter optimization in order to solve this issue. With human-created, specific feedback prompting methods, chat-based LLMs can iteratively find energy-efficient inference parameters faster than traditional search methods. LLMs can also tailor their solutions to different hardware setups and easily take into account other system constraints. The enhanced prompt template was able to converge below the threshold at an average of 3.4 prompts compared to the baseline, which converged in an average of 5.2 prompts, and consistently achieved lower final energy per token. The enhanced prompt template also outperformed Sobol sampling in convergence speed.
Large Language Models (LLMs) have become integral to modern applications, yet their deployment remains challenging. Beyond executing the models themselves, practical deployment must address cost efficiency, low latency, and optimal resource utilization. Conventional approaches typically assume that an entire model can be hosted on a single device, which does not hold in many real-world scenarios, particularly in Edge and Fog environments where device resources are constrained. In this paper, we introduce E2LLM, a framework designed to enable efficient LLM deployment in such resource limited settings. Rather than simply partitioning a single model across all available devices, E2LLM replicates the full model across multiple groups of devices (replicas) and applies model parallelism within each replica. Each replica is assigned a specialized role PREFILL or DECODER based on its efficiency in handling input and output tokens. This separation leverages the inherent differences between these two phases of LLM inference. To effectively organize devices, we utilize a Genetic Algorithm to form clusters that maximize system performance. Within each cluster, we apply Dynamic Programming to determine an optimal partitioning strategy that minimizes bottlenecks in model-parallel execution. Experimental results demonstrate that our approach adapts robustly to varying workloads, including scenarios with significant variation in input and output token lengths. Compared to the Splitwise baseline, E2LLM reduces average waiting time by over 50% under high-demand conditions
Truong-Thanh Le, Amir Taherkordi, Hoang-Loc La +3
Department of Informatics University of Oslo Oslo, Norway · Department of Computer Science UiT The Arctic University of Norway Tromsø, Norway