cs.DCMar 12, 2026

Characterizing Performance-Energy Trade-offs of Large Language Models in Multi-Request Workflows

Authors: Md. Monzurul Amin Ifath, Israat Haque

Organizations: Dalhousie University, Halifax, NS, Canada

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models (LLMs) are increasingly used in applications forming multi-request workflows like document summarization, search-based copilots, and multi-agent programming. While these workflows unlock richer functionality, they also amplify latency and energy demand during inference. Existing measurement and benchmarking efforts either focus on assessing LLM inference systems or consider single-request evaluations, overlooking workflow dependencies and cross-request interactions unique to multi-request workflows. Moreover, the energy usage of such interdependent LLM calls remains underexplored. To address these gaps, this paper presents the first systematic characterization of performance-energy trade-offs in multi-request LLM inference. We develop four representative workloads capturing sequential, interactive, agentic, and composite patterns common in modern deployments. Using an NVIDIA A100 testbed with state-of-the-art serving systems (vLLM and Parrot), we analyze how key energy knobs affect latency, throughput, and component-level energy use. Our findings reveal batch size as the most impactful lever, though benefits are workload dependent. While optimal batching benefits workloads with large shared prompts, it is ineffective for sequential summarization and only partially effective for multi-agent coding. GPU power capping provides modest but predictable savings, while output length induces linear energy scaling with limited efficiency gains. We further show that engine-level optimizations in vLLM maintain higher GPU utilization and efficiency, especially for decode-heavy workloads, while Parrot's workflow-aware scheduling achieves lower energy consumption under strict power constraints. These findings offer actionable guidelines for developers and system operators designing performance- and energy-aware LLM serving systems in emerging multi-request workflows.

Figures & tables

Figure 1. Architectural layers of LLM inference, spanning the LLM application layer (top) and the LLM serving layer (bottom).
SystemAPI / AbstractionFocus (Latency vs Throughput)Limitations
vLLMOpenAI-style text completion APIHigh throughput with modest latencyPerformance in multi-call workflows is under-explored
ParrotSemantic Variables/ DAG APIEnd-to-end latency in multi-step workflowsRequires application annotations; not tuned for pure throughput
SGLangStructured generation DSL + runtimeBalanced workflow-aware performanceNewer system; steeper programming model
OllamaLocal server wrapping over llama.cppLow-latency, small-scale useLimited batching and scale-out
llama.cppMinimal HTTP/ OpenAI-style APISimple, edge deploymentMinimal scheduling; low throughput under load
HuggingFace TGIREST/ gRPC text-gen APIScalable throughput with reasonable latencyHeavyweight system; no workflow awareness
Table 1. Comparison of representative LLM serving systems.
Figure 2. Representative multi-request LLM workflows in (a) Document chain summarization, (b) LLM-powered search, and (c) Multi-agent coding workload.
Figure 3. RQ1 results on the LLM-powered search workload. Panels (a-b) sweep output length, (c-d) sweep batch size, and (e-f) sweep GPU power cap, with all other knobs fixed. Each panel uses dual Y-axes with normalized total energy in left vs. normalized performance (latency/throughput) in right Y axis.
Figure 4. a) Energy per token vs. batch size under 16 concurrent users. Lines show mean value; shaded regions are 95% confidence intervals. Lower is better. The X markers denote batch sizes that are infeasible (OOM). b) Prefix cache hit rate distribution across workloads and batch sizes (1–8). Boxplots show medians and variability.
Figure 5. RQ3 results on chain summarization workload showing impact of input length, output length, batch size, and GPU power capping on latency, throughput, and component-wise (CPU,GPU, and DRAM) energy consumption. The red X markers denote batch sizes that are infeasible due to GPU OOM issue.
Figure 6. GPU utilization comparison between vLLM and Parrot for chain summarization workload (batch size 16). Values are averaged across ten runs.

Explore similar work

CardsList
Aug 28, 2026

Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms

Large language model (LLM) inference serving is priced by tokens, but GPU energy is consumed over inference windows. This accounting mismatch makes token-normalized metrics incomplete, since average output-token energy can decrease even when total request energy increases. We characterize this behavior with a decomposed energy model: a fixed one-time prefill with a fixed generation setup cost, while each output-token generation step adds marginal step energy. We evaluate this LLM inference energy model on NVIDIA H100 and H200 GPUs across dense and mixture-of-experts (MoE) models, reporting both request energy and token energy as functions of model type (M), phase (P), batch size (B), context length (C), and output length (N). For Llama-3.2-1B on H200 at batch-16 and context-4K, increasing output length from 10 to 512 tokens reduces token energy from 7.46 to 0.72 J/token while total batched inference-window energy increases from 1.19 to 5.93 kJ. Batching also reduces token energy, but the gain is context-bounded: at 10 output tokens, the batch-16 to batch-1 gain falls from 6.31x at context-512 to 1.17x at context-4K. MoE models amplify this effect: sparse routing and fragmented expert execution increase fixed energy at low concurrency, while batching spreads that energy across more generated tokens and substantially narrows the dense-vs.-MoE token-energy gap. These results show that energy-aware serving should jointly optimize both request energy and token energy, rather than only reducing per-token energy cost.

Prabhu Vellaisamy, Vanessa Lam, Shawn Blanton +1

Electrical and Computer Engineering Department Carnegie Mellon University Pittsburgh, PA, USA

LLM InferenceLLM Inference EfficiencyLLM ServingEfficient Language Model Inference
AskPDF
May 14, 2026

EnergyLens: Predictive Energy-Aware Exploration for Multi-GPU LLM Inference Optimization

We present EnergyLens, an end-to-end framework for energy-aware large language model (LLM) inference optimization. As LLMs scale, predicting and reducing their energy footprint has become critical for sustainability and datacenter operations, yet existing approaches either require production-level code and expensive profiling or fail to accurately capture multi-GPU energy behavior. As a result, practitioners lack tools for deciding which optimizations to prioritize and for selecting among existing deployment configurations when exhaustive profiling is impractical. EnergyLens addresses this gap with an intuitive einsum-based interface that captures LLM specifications including fusion, parallelism, and compute-communication overlap, combined with load-imbalance-aware MoE modeling and an empirically driven communication energy model for multi-GPU settings. We validate EnergyLens on Llama3 and Qwen3-MoE across tensor-parallel and expert-parallel configurations, achieving mean absolute percentage errors (MAPEs) between 9.25% and 13.19% for multi-GPU prefill and decode energy, and 12.97% across SM allocations for Megatron-style overlap. Our energy-driven exploration reveals up to 1.47x and 52.9x energy variation across configurations in prefill and decode efficiency and motivates distributed serving. We further show that compute-communication overlap is difficult to optimize with intuition alone, but EnergyLens correctly identifies Pareto-optimal overlap configurations.

Zhiye Song, Kyungmi Lee, Eun Kyung Lee +3

Massachusetts Institute of Technology Cambridge, MA 02139 · IBM Research Yorktown Heights, NY 10598

LLM Inference EfficiencyEnergy-Efficient MLGPU AccelerationLLM Inference
AskPDF
Apr 10, 2026

Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures

While the large energy consumption of Large Language Models (LLMs) is recognized by the community, system operators lack guidance for energy-efficient LLM inference deployments that leverage energy trade-offs of heterogeneous hardware due to a lack of energy-aware benchmarks and data. In this work we address this gap with Watt Counts: the largest open-access dataset of energy consumption of LLMs, with over 5,000 experiments for 50 LLMs across 10 NVIDIA Graphics Processing Units (GPUs) in batch and server scenarios along with a reproducible, open-source benchmark that enables community submissions to expand this dataset. Leveraging this dataset, we conduct a system-level study of LLM inference across heterogeneous GPU architectures and show that GPU selection is crucial for energy efficiency outcomes and that optimal hardware choices vary significantly across models and deployment scenarios, demonstrating the critical importance of hardware-aware deployment in heterogeneous LLM systems. Guided by our data and insights, we show that practitioners can reduce energy consumption by up to 70% in server scenarios with negligible impact on user experience, and by up to 20% in batch scenarios.

Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-Martínez

Universidad Politécnica de Madrid, C. de los Ciruelos, Boadilla del Monte, 28660, Madrid, Spain · Zurich University of Applied Sciences, Switzerland, Gertrudstrasse 15, Winterthur, 8400, Zurich, Switzerland

LLM Inference EfficiencyEnergy-Efficient MLHeterogeneous ComputingLLM Evaluation
AskPDF