A rigor-matched audit of periodic-step layer skipping for efficient llm inference: conflayers versus swift, with a supplemental analysis of trained routing alternatives

Prateek Kumar Sikdar

cs.CLAug 28, 2026

A rigor-matched audit of periodic-step layer skipping for efficient llm inference: conflayers versus swift, with a supplemental analysis of trained routing alternatives

Authors: Prateek Kumar Sikdar

Organizations: Accenture

Abstract

Layer-skipping methods for efficient LLM inference decide, at some granularity, which transformer layers to execute for a given input. We present a rigor-matched, three-seed audit of two periodic-step, search-based methods that make this decision online at inference time and re-evaluate it every few generation steps: a confidence-gated early-exit baseline (ConfLayers) and genuine self-speculative decoding (SWIFT, Xia et al. 2024), together with vanilla autoregressive decoding, across Qwen2.5-0.5B and Qwen2.5-1.5B (Yang et al. 2024) on GSM8K (Cobbe et al. 2021) and CNN/DailyMail (Nallapati et al. 2016; See et al. 2017). SWIFT is strongest on accuracy in three of four cells, while ConfLayers is dominated everywhere, with particularly large deficits on GSM8K at 1.5B. Separating online-search overhead from pure inference cost, we find that SWIFT's true inference speed is faster than ConfLayers's in all four cells (5-21%), reversing the naive wall-clock ranking in three; ConfLayers's search overhead is small and stable (1-2% of cost), whereas SWIFT's is larger and more variable across seeds (up to 28.7%). We additionally examine two trained routing methods, LayerRoute (Sikdar, 2026), a per-sequence, input-conditioned hard gate, and LayerDrop (Fan et al. 2020), a fixed, input-independent pruning pattern, as a supplemental analysis rather than a head-to-head comparison because both operate at a coarser decision granularity. Under a verified protocol with genuine per-input gating, a genuine full-model baseline, and genuine inference-time compute skipping, both show modest real speedups (1.08-1.33x) but accuracy well below the periodic-step methods, including near-total LayerRoute collapse on GSM8K at 1.5B (0.003 mean exact-match across three seeds). We release the full audit protocol as a template for rigor-matched efficiency comparisons.

Figures & tables

Method	Training	Decision unit	Re-evaluation cadence
ConfLayers	None (online search)	Per token	Every 30 steps
SWIFT	None (online search)	Per token	Every 25 steps
LayerRoute	Trained (LoRA + gate)	Per sequence	Once, input-conditioned
LayerDrop	Trained (structured dropout)	N/A	Never (fixed, input-independent)

Table 1: Layer-skipping methods by decision granularity. ConfLayers and SWIFT share the same regime (periodic-step, search-based) and form our main comparison; LayerRoute and LayerDrop occupy two different, coarser regimes and are reported as a supplemental analysis rather than head-to-head peers.

Task	Scale	Vanilla	ConfLayers	SWIFT
GSM8K	0.5B	0.180 [0.14–0.26]	0.147 [0.13–0.18]	0.303 [0.27–0.36]
GSM8K	1.5B	0.413 [0.38–0.46]	0.077 [0.05–0.10]	0.307 [0.29–0.33]
CNN/DM	0.5B	0.169 [0.167–0.174]	0.178 [0.175–0.184]	0.190 [0.185–0.196]
CNN/DM	1.5B	0.204 [0.193–0.213]	0.215 [0.204–0.222]	0.219 [0.205–0.230]

Table 2: Accuracy, mean over 3 seeds (range in brackets). GSM8K is exact-match accuracy; CNN/DailyMail is ROUGE-L. Bold marks the best method per cell.

Figure 1: Accuracy across the full 2 (task)

\times

2 (scale)

\times

3 (method) matrix. Error bars show the observed 3-seed range. ConfLayers is the consistent worst performer in every cell.

Task/Scale	Method	Pure inference (ms)	Search overhead	Wall-clock total (ms)
GSM8K/0.5B	ConfLayers	9194	2.2%	10338.5
GSM8K/0.5B	SWIFT	8771	15.1%	11564.5
GSM8K/1.5B	ConfLayers	12366	1.3%	12485.4
GSM8K/1.5B	SWIFT	9913	12.4%	11628.8
CNN-DM/0.5B	ConfLayers	10889	1.2%	10231.4
CNN-DM/0.5B	SWIFT	8610	8.7%	10598.2

Table 3: Per-query wall-clock latency (ms), mean over 3 seeds. Both methods shown decomposed into pure inference and online-search overhead (Section 6.2.1 ).

Figure 2: Per-query wall-clock latency across the full matrix.

Figure 3: Search-overhead decomposition. Once online-search cost is separated from pure model-forward cost, SWIFT is the faster inference engine in every cell, despite appearing slower on unsplit wall-clock time in three of four cells.

Task	Scale	LayerRoute acc.	LayerRoute speedup	LayerDrop acc.	LayerDrop speedup
GSM8K	0.5B	0.137 [0.11–0.16]	1.08 $\times$	0.010 [0.00–0.02]	1.30 $\times$
GSM8K	1.5B	0.003 [0.00–0.01]	1.32 $\times$	0.060 [0.05–0.07]	1.25 $\times$
CNN/DM	0.5B	0.102 [0.099–0.107]	1.33 $\times$	0.091 [0.085–0.097]	1.29 $\times$
CNN/DM	1.5B	0.150 [0.139–0.165]	1.09 $\times$	0.134 [0.131–0.136]	1.25 $\times$

Table 4: Supplemental analysis: accuracy and speedup, mean over 3 seeds (range in brackets), LayerRoute and LayerDrop versus a genuine full-model baseline. GSM8K is exact-match; CNN/DailyMail is ROUGE-L. Speedup

>1

indicates the routed/pruned pass is faster than the full-model baseline.

Figure 4: Supplemental results. Left: accuracy under genuine gating and a genuine full-model baseline. Right: speedup versus that same genuine baseline, including genuine inference-time compute skipping (Section 7.2 ).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: ConfLayers: top, the full layer stack; bottom, detail of the per-layer confidence check. A per-layer confidence estimate $c_{l}$ is thresholded against $\tau_{l}$ , discovered online via Bayesian optimization; pure inference and search overhead are logged and reported separately (Section 6.2.1 ).

Panel of Figure 6 — Figure 6: SWIFT: the target model self-drafts by skipping a searched subset $S$ of its own layers, then the full model verifies all draft positions in a single batched pass; the longest accepted prefix is kept and the rest resampled.

Panel of Figure 7 — Figure 7: LayerRoute: top, the full 24-layer stack; bottom, detail of a single gated layer $l$ . The router computes a scalar score from the mean-pooled input hidden state $\bar{h}_{l-1}$ , thresholds it into a hard gate $z_{l}$ via a straight-through estimator, and the layer output is either the LoRA-adapted block’s output ( $z_{l}{=}1$ ) or an unchanged residual pass-through ( $z_{l}{=}0$ ), matching Equations 7 – 8 . Discussed in the supplemental analysis, Section 7 .

Panel of Figure 8 — Figure 8: LayerDrop: layers are stochastically dropped during training to make the backbone and LoRA adapters robust to any layer’s removal; at inference, one fixed set of layers, chosen once from training statistics, is pruned identically for every input — no router, no per-input adaptivity. Discussed in the supplemental analysis, Section 7 .

Scale	Seed	Vanilla	ConfLayers	SWIFT
0.5B	2024	0.26	0.13	0.36
	42	0.14	0.18	0.28
	123	0.14	0.13	0.27
1.5B	2024	0.40	0.05	0.29
	42	0.46	0.08	0.33
	123	0.38	0.10	0.30

Appendix

Table 5: GSM8K exact-match accuracy at each of the three evaluation seeds, main comparison.

Scale	Seed	Vanilla	ConfLayers	SWIFT
0.5B	2024	0.167	0.175	0.190
	42	0.174	0.184	0.196
	123	0.167	0.178	0.185
1.5B	2024	0.193	0.204	0.205
	42	0.206	0.218	0.223
	123	0.213	0.222	0.230

Appendix

Table 6: CNN/DailyMail ROUGE-L at each of the three evaluation seeds, main comparison.

Scale	Seed	LayerRoute	LayerDrop
0.5B	2024	0.14	0.00
	42	0.16	0.02
	123	0.11	0.01
1.5B	2024	0.00	0.07
	42	0.01	0.05
	123	0.00	0.06

Appendix

Table 7: GSM8K exact-match accuracy at each of the three evaluation seeds, supplemental analysis.

Scale	Seed	LayerRoute	LayerDrop
0.5B	2024	0.100	0.085
	42	0.099	0.092
	123	0.107	0.097
1.5B	2024	0.146	0.136
	42	0.139	0.131
	123	0.165	0.134

Appendix

Table 8: CNN/DailyMail ROUGE-L at each of the three evaluation seeds, supplemental analysis.

Layer Skipping in Language Model Decoding

See all 11 papers

Sep 12, 2026

LayerRoute: Adaptive Layer-Skipping with LoRA-Preserved Quality for Efficient LLM Inference

We introduce LayerRoute, a parameter-efficient method for adaptive transformer layer-skipping that combines per-layer hard-gated routing (trained via a straight-through estimator) with joint LoRA fine-tuning. LayerRoute augments each of the 24 transformer blocks in Qwen2.5-0.5B-Instruct with a lightweight per-layer router (~21.5K parameters) and LoRA adapters (rank 8, ~1.08M parameters), training both jointly under a gate-regularized language-modeling objective. Across 10 independently-seeded training runs, LayerRoute converges to an identical skip-pattern structure in every run - a consistent set of 9 middle layers (8-16) becomes skip-eligible in all 10 seeds - and delivers genuine, verified wallclock speedup in every run (1.02x-1.06x, mean 1.04x). Quality is preserved or improved in every configuration tested: joint LoRA adaptation yields a perplexity improvement over the unmodified backbone in all 10 seeds (mean delta = -1.16 and -1.11 across the two evaluation splits used). We further verify the router performs genuine, non-trivial per-input computation: gate decisions in skip-eligible layers change the actual skip/run outcome for 87-100% of held-out samples, confirming real input-dependent routing rather than a fixed pruning pattern. LayerRoute trains in under 7 minutes on a single A100 and adds negligible overhead beyond the routing decision itself. We report our full reproducibility methodology, including a systematic diagnostic investigation into what determines the router's per-input decisions, as part of this work.

Prateek Kumar Sikdar

Accenture

Efficient Transformer Inference LLM Inference Acceleration Parameter-Efficient Fine-Tuning

Ask PDF

Sep 29, 2026

vSkipper: Translating Dynamic Layer Skipping into LLM Serving Gains

Dynamic layer skipping reduces LLM computation by allowing each token to execute only a subset of the model's layers. However, existing skippers rely on specialized generation loops and do not integrate with modern serving engines. As a result, fewer executed layers do not necessarily translate into lower serving latency: FlexiDepth skips 8 of Llama-3-8B's 32 layers on average, yet its standard generation loop decodes 14.6--21.0% more slowly than the base model. We present vSkipper, a virtualization layer that makes dynamic layer skippers pluggable in serving engines while preserving continuous batching, fixed-shape batches, paged KV caching, and captured decode graphs. At each routed layer, vSkipper groups tokens by the skipper's decision and uses routed execution only when predicted to be profitable. We implement vSkipper in SGLang and evaluate the released FlexiDepth checkpoint against upstream SGLang under identical prompts, arrivals, output lengths, and launch settings. At the knee of upstream's load curve, vSkipper reduces mean end-to-end latency by 36.8% on GSM8K and 13.6% on BBH. Under saturation, it increases request throughput by 11.3% and 7.4%. Serving adds no statistically resolved quality loss beyond the checkpoint's own. Across synthetic skip policies, two Qwen3 skippers, and three GPUs, we demonstrate reuse without workload-specific tuning. To our knowledge, vSkipper is the first system to realize serving-efficiency gains from per-token interior layer skipping within a modern LLM serving engine. The code is open-sourced as an SGLang fork at https://github.com/AKafakA/sglang-vskipper/tree/vskipper-ref

Wei Da, Yavuz Ferhatosmanoglu, Evangelia Kalyvianaki

The University of Cambridge, United Kingdom

LLM Inference Acceleration LLM Serving Layer Skipping in Language Model Decoding

Ask PDF

Apr 16, 2026

ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decoding

Self-speculative decoding is an inference technique for large language models designed to speed up generation without sacrificing output quality. It combines fast, approximate decoding using a compact version of the model as a draft model with selective re-evaluation by the full target model. Some existing methods form the draft model by dynamically learning which layers to skip during inference, effectively creating a smaller subnetwork to speed up computation. However, using heuristic-based approaches to select layers to skip can often be simpler and more effective. In this paper, we propose ConfLayers, a dynamic plug-and-play approach to forming the draft model in self-speculative decoding via confidence-based intermediate layer skipping. The process iteratively computes confidence scores for all layers, selects layers to skip based on an adaptive threshold, evaluates the performance of the resulting set, and updates the best selection until no further improvement is achieved or a maximum number of iterations is reached. This framework avoids the overhead and complexity of training a layer skipping policy and can provide more consistent speed-quality trade-offs while preserving the adaptivity of the draft model to diverse tasks and datasets. The performance evaluation of ConfLayers across different models and datasets shows that our novel approach offers up to 1.4x speedup over vanilla LLM generation.

Walaa Amer, Uday das, Fadi Kurdahi

Department of Electrical Engineering and Computer Sciences, University of California Irvine, Irvine, USA · Advanced Micro Devices, San Jose, USA

Speculative Decoding LLM Inference Acceleration

Ask PDF

Abstract

Layer Skipping in Language Model Decoding

Explore similar work

LayerRoute: Adaptive Layer-Skipping with LoRA-Preserved Quality for Efficient LLM Inference

vSkipper: Translating Dynamic Layer Skipping into LLM Serving Gains

ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decoding