Autoregressive Decoding

Follow this topic Ask about this topic

Momentum

8 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 38

All topics

Non-Monotonic Latency in Apple MPS Decoding: KV Cache Interactions and Execution Regimes
May 9, 2026Willy Fitra HendriaLLM Inference KV Caching
1Independent Researcher, Seoul, South Korea.
PACE: Post-Causal Entropy Modeling for Learned LiDAR Point Cloud Compression
May 2, 2026Jiahao Zhu, Kang You, Dandan Ding +1Point Cloud Compression Airborne Laser Scanning
School of Information Science and Technology, Hangzhou Normal University, Hangzhou, China. · School of Electronic Science and Engineering, Nanjing University, Nanjing, China.
Revisiting Greedy Decoding for Visual Question Answering: A Calibration Perspective
Apr 25, 2026Boqi Chen, Xudong Liu, Yunke Ao +1Visual Question Answering Language Model Calibration
1ETH Zurich · University of Toronto · 3MBZUAI
RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding
Apr 16, 2026Zihong Zhang, Zuchao Li, Lefei Zhang +2Speculative Decoding LLM Inference Acceleration
School of Artiﬁcial Intelligence, Wuhan University, Wuhan, China · School of Computer Science, Wuhan University, Wuhan, China · School of Information Management, Wuhan University, Wuhan, China +1
Suffix-Constrained Greedy Search Algorithms for Causal Language Models
Mar 1, 2026Ayoub Hammal, Pierre Zweigenbaum, Caio CorroConstrained Generative Modeling Constrained Decoding
1Université Paris-Saclay, CNRS, LISN · 2INSA Rennes, IRISA, CNRS, Université de Rennes
Attention-Mass Condensation for Sparse Decoding
Feb 6, 2026Jorge L. Ruiz WilliamsLanguage Model Decoding Sparse Attention
NaNZeta LLC
Decoding Order Matters in Autoregressive Speech Synthesis
Jan 13, 2026Minghui Zhao, Anton RagniAdaptive Inference TTS Synthesis
School of Computer Science, University of Sheffield, 211 Portobello, Sheffield S1 4DP, UK
Continuous Speculative Decoding for Autoregressive Image Generation
Nov 18, 2024Zili Wang, Zheng Zhang, Kun Ding +3Visual Autoregressive Models Speculative Decoding
School of Artificial Intelligence, University of Chinese Academy of Sciences, China · MAIS, Institute of Automation, Chinese Academy of Sciences, China · JD.COM Inc, China +1

May 9, 2026, cs.LGJ/K move · Enter open · S save

Non-Monotonic Latency in Apple MPS Decoding: KV Cache Interactions and Execution Regimes

Willy Fitra Hendria

1Independent Researcher, Seoul, South Korea.

LLM Inference KV Caching Autoregressive Decoding

Open paper AskPDF arXiv

Autoregressive inference is typically assumed to scale predictably with decoding length, with latency increasing smoothly as generated sequence length grows. In this work, we identify unexpected non-monotonic latency behavior in the Apple MPS backend, where latency changes abruptly across nearby decoding configurations during transformer decoding. Using multiple model families (GPT-2, BLOOM, and OPT), we observe latency spikes of up to 21x within specific decoding-budget intervals, followed by recovery at neighboring configurations. Controlled experiments show that these anomalies originate primarily during the decode phase rather than prefill, are not explained by memory pressure alone, and remain absent on CPU and NVIDIA CUDA backends under identical conditions. We further show that key-value (KV) cache interacts strongly with these pathological execution regimes: KV caching remains beneficial overall, but its practical speedup collapses sharply within anomalous configurations, while cache-disabled decoding still exhibits residual non-monotonic behavior. These findings suggest that autoregressive decoding on MPS enters discrete execution regimes that are not captured by coarse-grained benchmarking, highlighting the importance of hardware-aware evaluation for long-context inference.

Momentum

Related topics

Latest papers 38