cs.LGOct 13, 2026

Towards Green AI: Evaluating Energy Efficiency for Quantized LLM Inference

Authors: Petar Basta

Organizations: University of Waterloo Computer Science Waterloo, Ontario, Canada

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

While it is known that quantization of large language models (LLMs) reduces memory usage via lower-bit weights, studies quantifying the resulting impact on energy usage and carbon intensity are scarce. We present the first unified evaluation of weight-only quantization strategies for LLM inference across varying input/output lengths and GPUs, highlighting differences in energy efficiency in addition to accuracy degradation and runtime. We study quantized inference on Llama 2 7B, Phi 3.5 Mini Instruct, OLMo 1B, Qwen 2.5 7B, and Qwen 2.5 14B across GLUE MNLI, MMLU, HumanEval, and GSM8K datasets, evaluating 10 post-training quantization (PTQ) strategies on NVIDIA H100, A6000, A100, and L4 GPUs. We identify a novel trend showing that quantization techniques tend to exhibit peak energy efficiency relative to full-precision baselines when inputs are sufficiently long and outputs are short. Furthermore, we show that quantization strategies become marginally more energy-efficient relative to full-precision models as batch sizes increase, though gains are modest. Notably, fused-kernel implementations such as EETQ int8 and Bitsandbytes int8 offer the highest energy savings, up to 4x compared to FP32 on short text generation, with negligible accuracy loss. Finally, we observe that energy usage closely tracks runtime on our evaluated benchmarks, indicating that, in practice, latency optimization can serve as a reliable proxy for reducing the environmental footprint of LLM services. We conclude with suggested directions for strategically selecting low-carbon quantization strategies based on specific inference requirements.

Figures & tables

Quantization TechniqueSymmetricAsymmetric
Quanto✓✓
HQQ✓✓
TorchAO✓✓
Bitsandbytes✓✗
EETQ✓✗
AutoAWQ✓✓
Table 4.1 : Symmetric vs. Asymmetric Support Across Quantization Techniques
Quantization TechniquePer-TensorPer-ChannelPer-Group
Quanto✓✓✓
HQQ✗✓✓
TorchAO✓✓✓
Bitsandbytes✗✓✓
EETQ✗✓✗
AutoAWQ✗✗✓
Table 4.2 : Granularity Support Across Quantization Techniques
Figure 5.1 : Energy Efficiency Ratio Across Models and Datasets on A6000
Figure 4
Figure 5.3 : Energy Efficiency Ratio Across Models and Datasets on H100
Figure 6
Figure 7
Figure 8
Figure 5.7 : Energy Efficiency Ratio vs. Output Length Across Different Input Sizes, Models, and Hardware
Figure 5.8 : Energy Efficiency Ratio for Single-Token Decoding for Llama 2 7B Across Input Lengths on A6000 & A100
Figure 5.9 : Speedup Ratio Across Models and Datasets on A6000
Figure 12
Figure 5.11 : Speedup Ratio Across Models and Datasets on H100
Figure 5.12 : Energy Efficiency Ratio vs. Batch Size for Llama 2 7B with Varying Input and Output Lengths on A6000
Figure 5.13 : Absolute Energy Usage vs. Batch Size for Llama 2 7B with Varying Input and Output Lengths on A6000
ModelBenchmarkvanilla FP32autoawq int4Bitsandbytes nf4Bitsandbytes int8eetq int8gptq int4hqq int4hqq int8quanto int4quanto int8torchao int4
Llama 2 7BGLUE MNLI0.43180.42850.42350.42900.43040.42680.43250.43200.36960.43160.4138
GSM8K (10%)0.16030.12980.15270.18320.16790.09160.12210.15270.15270.15270.0992
HumanEval0.11590.12800.10980.12800.12200.12800.12800.11590.12800.12800.1341
MMLU0.44690.44450.41530.44150.44490.42710.43560.44400.43000.44680.4242
Phi 3.5 Mini InstructGLUE MNLI0.74650.71520.69690.72370.74130.68030.66300.74770.69160.74570.6694
GSM8K (10%)0.77860.76340.64890.80920.76340.72520.67180.77860.77100.79390.7481
Table 5.1 : Quantized Model Accuracies Across Benchmarks
Figure 17
Figure 18
Figure 19
Figure 5.17 : Energy Usage vs Time on A6000
Quantization StrategyAvg Memory Efficiency Ratio
autoawq int45.73x
Bitsandbytes nf45.11x
Bitsandbytes int83.53x
eetq int83.87x
gptq int44.57x
hqq int43.52x
Table 5.2 : Average Memory Efficiency Ratio Across Models
Figure 5.18 : Memory Efficiency Ratio Across Models
Figure 5.19 : Cost & CO 2 per Billion Queries by Quantization Strategy and Dataset on A6000 (log scaled)
More in this topic

LLM Inference Efficiency

Large language model inference efficiency reduces the time, memory, energy, and cost of generating useful outputs.

See all 345 papers

Explore similar work

CardsList
Aug 24, 2026

GreenBench: Benchmarking Energy Efficiency and Carbon Footprint of Open-Source LLM Inference on Apple Silicon

The rapid proliferation of Large Language Models (LLMs) has raised concerns about their environmental impact during inference. While Green AI research has focused on datacenter GPUs and embedded platforms, the energy profile of LLM inference on Apple Silicon, with its unified memory architecture, remains unstudied. This paper presents GreenBench, a benchmarking framework that evaluates the energy efficiency, throughput, and carbon footprint of five open-source LLMs (3-9B parameters) across three NLP tasks on an Apple M4 Pro with 48 GB unified memory. Using macOS powermetrics for direct power measurement and Ollama's nanosecond-precision timing, we find that the M4 Pro draws only 0.47 W of CPU+GPU package power during sustained inference, with total system power of 8-12 W, achieving 30-40x better energy efficiency per token than datacenter GPUs in single-user deployment. Smaller models (3-3.8B) deliver 2.6-4.2x higher throughput and up to 62% less energy per token than larger models (7-9B). Pareto analysis identifies Qwen 2.5 (7B) as the optimal accuracy-efficiency trade-off at 57% MMLU and 59 tokens/s, while Llama 3.2 (3B) suits latency-critical applications at 175 tokens/s. We provide per-token energy at package and system levels with CO2 estimates for India and US grids.

Rajeswari Kannan, Raj Firke, Shreya Bengle +1

Pimpri Chinchwad College of Engineering Pune, India · Red Hat Pune, India · Independent Researcher Pune, India

LLM Inference EfficiencyEnergy-Efficient MLOn-Device Language Model InferenceLLM Evaluation
AskPDF
Jul 29, 2026

From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs

The operational energy consumption of large language model (LLM) inference is becoming an increasingly important component of the environmental footprint of deployed AI systems. However, direct measurement of inference energy often requires hardware telemetry, power instrumentation, or infrastructure-specific monitoring, limiting its applicability in comparative studies, early-stage system design, and sustainability reporting. This report presents an analytically structured, empirically calibrated, GPU-level methodology for estimating LLM inference energy on NVIDIA H100-class accelerators without direct runtime measurement. The proposed estimator combines parameter-scaled transformer FLOP accounting, calibrated memory-traffic factors, and hardware-specific energy coefficients for FP16/BF16 tensor-core computation and high-bandwidth-memory movement. It explicitly separates prompt prefill from autoregressive decoding, enabling energy estimates for input tokens, output tokens, and complete inference requests. The methodology further decomposes total energy into compute, parameter-access, key-value-cache write, and attention-read components, allowing the scaling behavior with model size, context length, and generated-token count to be analyzed. The resulting estimates are not intended to replace physical power measurements; rather, they provide transparent, reproducible, and assumption-explicit approximations suitable for model comparison, green-coding analysis, and design-time evaluation of LLM inference workloads.

Tina Vartziotis, Rodopi Kosteli, Elli Vartziotis +5

1National Technical University of Athens, Patission Complex 42, 10682 Athens, Greece · 2Harvard University, 1350 Massachusetts Avenue, 02138 Cambridge, MA, USA · 3TWT GmbH Science & Innovation, Industriestraße 6, 70565 Stuttgart, DE +4

LLM Inference EfficiencyEnergy-Efficient MLLLM InferenceEnergy-Aware LLM Inference
AskPDF
Mar 6, 2026

The Compression Paradox in LLM Inference: Provider-Dependent Energy Effects of Prompt Compression

The rapid proliferation of Large Language Models has created an environmental paradox: the very technology that could help solve climate challenges is itself becoming a significant contributor to global carbon emissions. We test whether prompt compression improves inference energy efficiency in 28,421 successful API trials (28,428 planned) across three providers (OpenAI GPT-4o-mini, Anthropic Claude-3.5-Sonnet, and DeepSeek-Chat), five benchmarks (HumanEval, MBPP, GSM8K, MATH, MMLU), and four compression ratios (r in {1.0, 0.7, 0.5, 0.3}). Energy is estimated with a token-based proxy calibrated against local direct measurements, and quality is tracked with benchmark pass rates. Compression produced substantial quality loss (overall pass rate 26.0% at baseline vs. 1.5% at r=0.7) and strongly provider-dependent energy behavior. DeepSeek exhibited output expansion under compression (21 to 798 tokens at r=0.3), corresponding to energy increases up to +2,140%, while GPT-4o-mini showed mixed effects including a reduction at r=0.5. These results indicate that input-token reduction alone is not a reliable energy optimization strategy in production inference. For the evaluated settings, model selection and output-length control provided more consistent energy-quality tradeoffs than prompt compression.

Warren Johnson

Plexor Labs Principal Researcher

LLM Inference EfficiencyEnergy-Aware LLM InferenceEfficient Language Model InferencePrompt Compression
AskPDF