Layer Skipping in Language Model Decoding

Follow this topic Ask about this topic

Latest papers

11 papers5 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Do Halting Heads Really Save Compute in Looped Language Models?
Oct 23, 2026Tsung-Han Wu, Minsuk Heo, Kenny TanLLM Inference Acceleration Layer Skipping in Language Model Decoding
Apple
How Much Computation Does Every Forecast Need? Structured Block Skipping in Two Time-Series Foundation Models
Oct 9, 2026Max BourgeatLayer Skipping in Language Model Decoding Efficient Transformer Inference
Polytechnique Montréal Montréal, QC, Canada
GeoGR: Spatial-Semantic Token Compression and Group Routing for Efficient 3D VLMs
Oct 9, 2026Yongxin Zheng, Hao Wu, Fang Li +8Layer Skipping in Language Model Decoding 3D VLMs
EIT-NLP Lab, Eastern Institute of Technology, Ningbo · Shanghai Jiao Tong University · The Hong Kong Polytechnic University +1
Input-Blind Controls Produce Substantial Oracle Headroom for Layer Programs in Multiple-Choice Evaluation
Oct 7, 2026Yibei Guo, Rui LiuLayer Skipping in Language Model Decoding Input Order Sensitivity in Language Models
College of Aeronautics and Engineering, Kent State University
vSkipper: Translating Dynamic Layer Skipping into LLM Serving Gains
Sep 29, 2026Wei Da, Yavuz Ferhatosmanoglu, Evangelia KalyvianakiLLM Inference Acceleration LLM Serving
The University of Cambridge, United Kingdom
A rigor-matched audit of periodic-step layer skipping for efficient llm inference: conflayers versus swift, with a supplemental analysis of trained routing alternatives
Aug 28, 2026Prateek Kumar SikdarLayer Skipping in Language Model Decoding LLM Inference Acceleration
Accenture
TSS: Target-Side Sparsification for Speculative Decoding in Domain-Specific Large Language Models
Aug 15, 2026Haibo Hu, Lianming Huang, Qiao Li +2Layer Skipping in Language Model Decoding Speculative Decoding
City University of Hong Kong · Mohamed bin Zayed University of Artificial Intelligence
TrimMoE A communication aware and adaptive depth framework for distributed edge inference
Aug 1, 2026Ning Li, Shuting Bai, Xin Yuan +3Adaptive Inference Edge Inference
School of Artificial Intelligence, University of Science and Technology Beijing, China · School of Computer Science and Technology, Harbin Institute of Technology, Heilongjiang, China · School of Ocean Engineering, Harbin Institute of Technology, Heilongjiang, China +2
Mitigating Hallucinations in Large Language Models Via Decoder Layer Skipping
May 30, 2026Hanze Li, Jinhao You, Yichen Guo +3Language Model Decoding LLM Hallucination Mitigation
De Artificial Intelligence Lab
TIDE: Token-Informed Depth Execution for Per-Token Early Exit in LLM Inference
Mar 22, 2026Jaber Jaber, Osama JaberLLM Inference LLM Inference Acceleration
RightNow AI
DySL-VLA: Efficient Vision-Language-Action Model Inference via Dynamic-Static Layer-Skipping for Robot Manipulation
Feb 26, 2026Zebin Yang, Yijiahao Qi, Tong Xie +3Layer Skipping in Language Model Decoding Efficient VLA Model Inference
Institute for Artificial Intelligence, Peking University, Beijing, China · School of Integrated Circuits, Peking University, Beijing, China · School of Electronics Engineering and Computer Science, Peking University, Beijing, China +2

Oct 23, 2026J/K move · Enter open · S save

Do Halting Heads Really Save Compute in Looped Language Models?

Tsung-Han Wu, Minsuk Heo, Kenny Tan

Apple

LLM Inference Acceleration Layer Skipping in Language Model Decoding Recursive Language Models

Open paper AskPDF arXiv

Looped language models run each token through the same layers several times, and each run is one iteration. To save compute, an extra layer, the halting head, gives after each iteration the probability that the token stops there, so that easy tokens can stop early. Pretraining models from scratch, up to 1.3B parameters, we find two problems. First, heads trained on a mix of all iterations predict that tokens need under half the iterations, yet under the standard stopping rule tokens run almost every iteration, because this training never scores the single iteration where a token stops. Scoring every iteration instead fixes this, but in our runs it works only with a strong penalty favoring early stops. Second, savings are commonly reported as an average number of iterations per token, but when sequences are generated in batches that run until their slowest token stops, a model without a head is almost as good at that cost. The reported average can therefore overstate the saving. We recommend reporting the iterations tokens actually run, with the stopping threshold and batch size, and give a test, run before training a head, that bounds its gain.