cs.LGOct 14, 2026

Energy Consumption Analysis of Discrete Diffusion and Autoregressive Language Models

Authors: Boris Ruf, Marcin Detyniecki

Organizations: AXA, AI Research Paris, France

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

As Large Language Models (LLMs) continue to scale, their energy consumption during inference has become a critical concern. While Autoregressive (AR) models are the industry standard, Diffusion Language Models (DLLMs) have emerged as a promising alternative capable of parallel token generation. In this paper, we perform a comparative energy benchmark of diffusion models (LLaDA-8B, Dream-7B) against AR counterparts (Llama-3.1-8B, Qwen-2.5-7B). We analyze the trade-offs between model architecture, KV-cache optimization, and Joules-per-token efficiency. Our results show that at longer sequences (200--600 tokens), diffusion models achieve 2.5--2.7 J/token, approximately 2×\times more efficient than AR models with KV-cache (5.3--5.7 J/token). KV-cache provides substantial benefits for AR models at longer sequences (50--70% energy reduction).

Figures & tables

Figure 1. Illustrative model of GPU utilization over time for an autoregressive model (blue) with 10 output tokens vs a diffusion model (orange) with 10 denoising steps. This theoretical pattern motivates our empirical investigation. Illustrative comparison of GPU utilization over time: the autoregressive model (blue) shows sequential peaks for each of 10 generated tokens, while the diffusion model (orange) shows a shorter, denser block corresponding to 10 denoising steps.
 ModelKV-CacheMax TokensGen. TokensJ/tokens/token
Short sequences (max_tokens = 10)
ARLlama-3.1-8B-InstructOff10108.320.0300
Llama-3.1-8B-InstructOn10108.100.0295
Qwen-2.5-7B-InstructOff10107.930.0270
Qwen-2.5-7B-InstructOn10107.920.0265
DiffDream-7B—101032.210.1676
Table 1. Energy and latency per generated token across different sequence lengths. AR models shown with and without KV-cache. Diffusion models do not use KV-cache. Note: Qwen-2.5 generated fewer tokens at max=1024 because the model produced an end-of-sequence token before reaching the limit.
Figure 2. Measured GPU utilization and power draw for a cached run, comparing an autoregressive model (Llama-3.1) with a diffusion model (Dream-7B). This plot visualizes the execution from our traces where the autoregressive model’s KV-cache is active. The initial period ( ≈\approx 8s for diffusion, ≈\approx 18s for AR) captures the benchmark’s warmup phase. The subsequent, stable utilization pattern corresponds to the 10 measured runs, during which each model generated 128 tokens per run. Measured GPU utilization and power traces for cached runs comparing Llama-3.1 (autoregressive) and Dream-7B (diffusion). The first segment is warmup (about 8 seconds for diffusion and 18 seconds for autoregressive), followed by a stable repeated pattern from 10 measured runs of 128-token generation per run.
More in this topic

LLM Inference

Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.

See all 268 papers

Explore similar work

CardsList

  1. DARE: Diffusion Language Model Activation Reuse for Efficient Inference

    May 1, 2026Natalia Frumkin, Bokun Wang, Hung-Yueh Chiang +3Self-AttentionDiffusion Language Model Inference

    Chandra Family Department of Electrical and Computer Engineering The University of Texas at Austin · Cornell University

  2. A Comparative analysis of Layer-wise Representational Capacity in AR and Diffusion LLMs

    Mar 8, 2026Raghavv Goel, Risheek Garrepalli, Sudhanshu Agrawal +3Diffusion Language ModelsRepresentation Geometry in Language Models

    Qualcomm AI Research, Qualcomm Technologies Inc.

  3. ES-dLLM: Efficient Inference for Diffusion Large Language Models by Early-Skipping

    Mar 10, 2026Zijian Zhu, Fei Ren, Zhanhong Tan +1Diffusion Language Model InferenceDiffusion Language Models

    IIIS, Tsinghua University · Polar Bear Tech.