The Diminishing Returns of Early-Exit Decoding in Modern LLMs

Rui Wei; Rui Du; Hanfei Yu; Devesh Tiwari; Jian Li; Zhaozhuo Xu; Hao Wang

cs.CLMar 24, 2026

The Diminishing Returns of Early-Exit Decoding in Modern LLMs

Authors: Rui Wei, Rui Du, Hanfei Yu, Devesh Tiwari, Jian Li, Zhaozhuo Xu, Hao Wang

Organizations: Stevens Institute of Technology · Northeastern University · Stony Brook University

Abstract

In Large Language Model (LLM) inference, early-exit refers to stopping computation at an intermediate layer once the prediction is sufficiently confident, thereby reducing latency and cost. However, recent LLMs adopt improved pretraining recipes and architectures that reduce layer redundancy, potentially limiting early-exit opportunities. We re-evaluate layer-wise early-exit in modern LLMs and analyze how intermediate representations evolve during training. We introduce a metric to quantify a model's intrinsic suitability for early-exit and propose a benchmark for researchers to explore the potential early-exit benefits on different models and workloads. Our results show a diminishing trend in early-exit effectiveness across newer model generations. We further find that dense transformers generally offer greater early-exit potential than Mixture-of-Experts and State Space Models. In addition, larger models, particularly those with more than 20 billion parameters, and base pretrained models without specialized tuning tend to exhibit higher early-exit potential.

Figures & tables

Figure 1: Layer-wise early-exit decoding in LLMs.

Figure 2: The trend of relative early-exit scores (§ 3.3 ) in recent LLM and models specifically tuned for early-exit, compared to Llama2-7B . We explain the model selection details in Appendix B .

Figure 3: Workflow illustration of the paper.

Figure 4: The layer-to-final similarity results of eight different models aggregated across four datasets.

Model	MMLU				GSM8K				EAS
Model	$\delta$	Full Acc.	Acc.	Skip (%)	$\delta$	Full Acc.	Acc.	Skip (%)	EAS
Llama2-7B	0.70	0.41	0.38	4.69	0.80	0.21	0.17	2.13	0.52
Llama3-8B	0.91	0.54	0.52	0.18	0.90	0.37	0.33	0.04	0.46
Llama4-Scout	0.84	0.76	0.73	0.32	0.87	0.85	0.82	0.18	0.47
Qwen2-7B	1.00	0.24	0.24	0.00	1.00	0.53	0.53	0.00	0.36
Qwen3-8B	0.95	0.41	0.39	0.10	0.86	0.54	0.51	0.62	0.51

Table 1: Oracle early-exit performance under a maximum skip ratio, with accuracy loss within to 5%. We report the full-depth accuracy (Full Acc.

\uparrow

), early-exit accuracy (Acc.

\uparrow

), skip ratio (Skip

\uparrow

), and the mean early-exit score (

\uparrow

Figure 5: The accuracy and skip ratio of different exit strategies and thresholds.

Figure 6: The EAS increases with model scale.

Figure 7: The layer-to-final similarity and the mean EAS of three different LLM architectures: (a) dense ( Qwen3-8B ), (b) MoE ( Qwen3-30B-A3B ), and (c) SSM ( Mamba-Codestral-7B ).

Figure 8: The layer-to-final similarity of Pythia-12B across different training stages.

Figure 9: The effect of post-training alignments on the model’s early-exit adaptability.

Figure 10: The effect of workloads on the model’s early-exit adaptability.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model Family	Year	Architecture	Key Features	Training Recipe
Llama2 Touvron et al. (2023)	2023	Dense transformer	RoPE, SwiGLU	2T tokens, PPO
Llama3 Grattafiori et al. (2024)	2024	Dense transformer	GQA, 8k Context	15T tokens, DPO
Llama4 Meta Inc. (2025)	2025	MoE	Native Multimodal	10M Context, Reasoning
Qwen2 Yang et al. (2024)	2024	Dense & MoE	YaRN, GQA	7T tokens, Staged Mix
Qwen3 Yang et al. (2025)	2025	Dense & MoE	Thinking Mode	CoT-SFT, Multilingual
OpenAI OSS OpenAI et al. (2025)	2025	MoE	Reasoning MoE	QAT, Configurable Effort

Appendix

Table 2: Categorization of Selected LLM Families.

Model Flavor	Score
Llama2-7B Touvron et al. (2023)	0.52
Mamba1-130M Gu and Dao (2024)	0.35
EE-LLM-7B-dj-refine-150B Chen et al. (2024)	0.56
Llama3-8B Grattafiori et al. (2024)	0.46
LayerSkip-llama3-8B Elhoushi et al. (2024)	0.54
Mamba2-7B Dao and Gu (2024)	0.48

Appendix

Table 3: EAS for different model flavors used in Fig. 2 .

Dataset	Instruction Template
GSM8k	Question: {question} \ Let’s think step by step \ Answer:
HumanEval	Read the following function signature and docstring, and fully implement the function described. \ Your response should only contain the code for this function. \ {prompt}
MMLU	Question: {input} \ A. {A} \ B. {B} \ C. {C} \ D. {D} \ Answer:
GPQA	Question: {question} \ A. {A} \ B. {B} \ C. {C} \ D. {D} \ Answer:

Appendix

Table 4: Prompt templates used for each evaluation dataset.

Early-Exit Neural Networks

Early-exit neural networks stop processing inputs at intermediate layers when predictions are sufficiently reliable, trading accuracy for lower latency and energy.

See all 44 papers

Two-dimensional early exit optimisation of LLM inference
Mar 27, 2026Jan Hůla, David Adamczyk, Tomáš Filip +2LLM Inference Acceleration Early-Exit Neural Networks
Institute for Research and Applications of Fuzzy Modelling, University of Ostrava, 70200 Ostrava, Czech Republic · Institute of Computer Science, Faculty of Philosophy and Science, Silesian University in Opava, 74601 Opava, Czech Republic
River-LLM: Large Language Model Seamless Exit Based on KV Share
Apr 20, 2026Yingtao Shen, An ZouDecoder-Only Language Models KV-Cache Management
Shanghai Jiao Tong University Shanghai, China
FlexEE: Self-Speculative and KV-Compatible Early Exiting for Offloading-Aware LLM Inference
Sep 15, 2026Qihu Xie, Ziwei Li, Yi KangEfficient Language Model Inference LLM Inference
University of Science and Technology of China, Hefei, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, China

Mar 27, 2026J/K move · Enter open · S save

Two-dimensional early exit optimisation of LLM inference

Jan Hůla, David Adamczyk, Tomáš Filip, Martin Pavlíček+1

Institute for Research and Applications of Fuzzy Modelling, University of Ostrava, 70200 Ostrava, Czech Republic · Institute of Computer Science, Faculty of Philosophy and Science, Silesian University in Opava, 74601 Opava, Czech Republic

LLM Inference Acceleration Early-Exit Neural Networks

Open paper AskPDF arXiv

We introduce a two-dimensional (2D) early exit strategy that coordinates layer-wise and sentence-wise exiting for classification tasks in large language models. By processing input incrementally sentence-by-sentence while progressively activating deeper layers, our method achieves multiplicative computational savings that exceed those from optimizing either dimension independently. Experimental evaluation across four state-of-the-art LLMs (Llama 3.1, Llama 3.2, Gemma, Qwen; 3B-8B parameters) on three sentiment classification datasets demonstrates additional speed-ups of 1.4--2.3

\times

over optimal layer-wise early exit for simpler tasks with vanilla models, with graceful degradation on complex multi-class problems. Fine-tuning reduces but does not eliminate this advantage. The approach is model-agnostic, requires only lightweight classification adapters, and is orthogonal to complementary efficiency methods such as quantization and pruning. Our findings indicate that 2D early exit strategies excel when semantic information accumulates predictably across input structure, suggesting possible applicability to sequence-processing tasks beyond sentiment classification.

Abstract

Early-Exit Neural Networks

Explore similar work

Two-dimensional early exit optimisation of LLM inference

River-LLM: Large Language Model Seamless Exit Based on KV Share

FlexEE: Self-Speculative and KV-Compatible Early Exiting for Offloading-Aware LLM Inference