Efficient Language Model Inference

Follow this topic Ask about this topic

Momentum

37 papers in the last four weeks, up 185% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 206

All topics

Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models
Sep 23, 2026Ke Wan, Chen ChenLong-Context Language Model Inference Recurrent Transformers
Department of Computer Science University of Virginia Charlottesville, VA, USA
Universal Fractal Natural Language Decision Map: Real-Time Edge Triage Across Heterogeneous Domains
Sep 21, 2026Volkan Dağlı, Zerrin Dağlı, Dağhan DağlıOn-Device Language Model Inference Efficient Language Model Inference
Anadolu University, Eski¸sehir, Turkey · ITouch Systems, Mersin, Turkey · Mersin University, Mersin, Turkey +1
this-that-model-1.0: A typed decision model that decides in 30 ms, for a millionth of a cent
Sep 20, 2026Zehua Cheng, Wei Dai, Jiahao SunOn-Device Language Model Inference LLM Decision-Making
University of Oxford
When2Think: Learning When and How Much to Reason
Sep 17, 2026Jaejun Shim, HyunJin Kim, Young Jin Kim +1Overthinking in Language Models RL for Language Model Reasoning
Sungkyunkwan University · Microsoft
Dynamic Semantic Compression for Efficient Latent-Space Inference in Large Language Models
Sep 14, 2026Peipei Li, Dongsen Zhang, Yuchen Liu +1LLM Compression LLM Inference
Beijing University of Posts and Telecommunications, Beijing, China
Lightning Weave: Improving the Accuracy-Efficiency Frontier of Reasoning Models through Capability Composition
Sep 13, 2026Yecheng Wu, Song Han, Han CaiEfficient Language Model Reasoning Language Model Distillation
Massachusetts Institute of Technology · NVIDIA
Stable Answers, Unfinished Reasoning: Why Self-Consensus Is Not a Safe Early-Exit Signal
Sep 9, 2026Yunxiang Mo, Donghao Zhao, Hejia GengEarly Stopping LLM Reliability
HKUST · University of Oxford
SymbolicLight V2: Hybrid Neuromorphic Architecture and Sparse Execution for Low-Energy Language Inference
Sep 9, 2026Ting LiuNeuromorphic Computing Efficient Language Model Inference
SymbolicLight Research Foshan, Guangdong, China
TEFM: Token-Efficient Faithful Modeling for Structured Data
Sep 9, 2026Zhichao Hou, Lingdao Sha, Xueyu Mao +3LLM Interpretability Tabular Classification
North Carolina State University · Amazon Web Services
A*-Thought-V2: Efficient Latent Reasoning via Geometric Dynamics of LLM
Sep 7, 2026Xiaoang Xu, Siyuan Liu, Shuo Wang +13Efficient Language Model Reasoning Continuous Latent Reasoning
Beijing University of Posts and Telecommunications · OpenBMB · The Hong Kong Polytechnic University +2
Beyond Fluent Generation: A CPU Reliability Benchmark for MCP-Style Tool Calling in Sub-2B Small Language Models for Edge Deployment
Sep 7, 2026Abrar Shahriar Qurat-Ul-Ain MastoiTool-Augmented Language Model Agents On-Device Language Model Inference
University of the West of England, Bristol,BS16 1QY United Kingdom
</think> Doesn't Stop Reasoning: Analysis of Spurious CoT Termination
Sep 3, 2026Seunghee Koh, Sungjae Choi, Minchan Kwon +2CoT Reasoning Efficient Language Model Reasoning
Korea Advanced Institute of Science and Technology, South Korea
LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference
Sep 2, 2026Renyuan Liu, Yuyang Leng, Kaiyan Liu +7LLM Inference Efficiency On-Device Language Model Inference
Richard · George Mason University · Global Technology Applied Research, JPMorganChase +1
Language Models Can Control Their Own Attention
Sep 2, 2026Namgyu Ho, Huzama Ahmad, Woosung Koh +3Efficient Language Model Inference Long-Context Language Modeling
KAIST AI · Google DeepMind
How Do Prompt Variations Affect Energy Consumption in On-Device LLMs?
Sep 1, 2026Wei Hu, Xiaolong Tu, Dawei Chen +3LLM Inference Efficiency On-Device Language Model Inference
Georgia State University · Toyota Motor North America
HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation
Sep 1, 2026Alessandro Zirilli, Davide Marincione, Evgenios M. Kornaropoulos +2Privacy-Preserving Language Models Homomorphic Encryption
Sapienza University of Rome · George Mason University · Paradigma
HSRM: Hidden-State Reward Models for Test-Time Verification
Aug 31, 2026Xianzhi Li, Xiaodan ZhuLLM Answer Verification Efficient Language Model Reasoning
Department of Electrical and Computer Engineering & Ingenuity Labs Research Institute Queen’s University
TuringLLM: Efficiently Scaling Foundation Models Toward Physical AI
Aug 31, 2026Yuheng Zhang, Yizhao Wang, Da Zhu +19Mixture-of-Experts Language Models Long-Context Language Model Inference
Foundation Model Team, Xpeng Inc.
Budget-Aware Compression Pipeline for Single-GPU LLM Inference: Methods, Trade-offs, and Coupling Effects
Aug 30, 2026Hongyu Yu, Yifei ShenLLM Quantization LLM Pruning
Lenovo Research · University of Washington
EVAR: Evidence-Validated Hypothesis Admission for Budget-Aware Narrative Reasoning
Aug 30, 2026Peilin Liu, Zhiquan Ji, Jinglong PingEvidence-Grounded Reasoning LLM Reliability
College of Software, Jilin University · School of Urban Planning and Design, Peking University
Ask Self, Ask Others: Relation Is All You Need
Aug 20, 2026Yuting Ge, Pengju Yang, Mingkai NieSelf-Attention Decoder-Only Language Models
Department of Electrical Engineering City University of Hong Kong · Jilin University · National University of Singapore
From Answers to Policies: Efficient In-Context Learning System through Emulating Expert Investigation
Aug 17, 2026Minh-Ha Nguyen, Ngoc-Ngo Quang Tran, Thuy Dung Nguyen +1In-Context Learning In-Context RL
Department of Epidemiology, Vanderbilt University, Nashville, TN, USA · Department of Pediatrics, Vanderbilt University Medical Center, Nashville, TN, USA · Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN, USA +1
Claim-Level Reliability Assessment for Efficient Test-Time Reasoning
Aug 12, 2026Sen Xu, Wei Wang, Shixi Liu +5Test-Time Scaling LLM Reliability
Sina Weibo Inc.
Towards Efficient Reasoning in LLM-Based Recommender Systems via Model Merging
Aug 11, 2026Linh Dieu Le, Tong Chen, Shazia Sadiq +3LLM Compression Large Language Model-Based Recommendation
The University of Queensland Brisbane, Australia · Griffith University Brisbane, Australia
Can We Optimize the Performance-Carbon Emission Break-Even Point?: The Quest for Greener LLMs
Aug 9, 2026Sourav Das, Tanmay Joshi, Kripabandhu GhoshFine-Tuning LLM Fine-Tuning
IIIT Kalyani · BITS Pilani Goa · IISER Kolkata
Think Deep, Speak Once: Relit, A Recursive Latent Implicit Transformer Framework
Aug 8, 2026Abhishek Panwar, Maheep Singh, Saksham BansalContinuous Latent Reasoning Efficient Language Model Inference
Indian Institute of Technology Roorkee, India
Thought-Level Beam Search for Reasoning
Aug 8, 2026Lijie Yang, Hongyin Luo, Jiawei Zhao +2Inference-Time Search Efficient Language Model Reasoning
Princeton University · MIT CSAIL · Meta AI
Lightweight Chunk Selection for Mobile Retrieval-Augmented Generation
Aug 4, 2026Sicong Chang, Yidan Shen, Wen Yu +3Retrieval-Augmented Generation Efficient Language Model Inference
Distilling Reasoning Traces into Advisory Prompts for Software Engineering Tasks
Aug 1, 2026Faizan Faisal, Prem Devanbu, Toufique AhmedReasoning Distillation LLM Prompting
University of California, Davis · IBM
Translation with Thought: Difficulty-Adaptive Reasoning via Reinforcement Learning for Multi-Domain Machine Translation
Jul 31, 2026Yongshi Ye, Biao Fu, Chongxuan Huang +2RL for Language Model Reasoning Efficient Language Model Inference
Institute of Artificial Intelligence, Xiamen University · School of Informatics, Xiamen University · Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism

Sep 23, 2026, cs.CLJ/K move · Enter open · S save

Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models

Ke Wan, Chen Chen

Department of Computer Science University of Virginia Charlottesville, VA, USA

Long-Context Language Model Inference Recurrent Transformers Efficient Attention Efficient Language Model Inference Block-Sparse Attention

Open paper AskPDF arXiv

Recurrent-depth language models, such as looped Transformers, repeatedly apply shared network blocks to refine latent representations without generating explicit intermediate reasoning tokens. However, each step recomputes full attention over the entire context, repeating costly global routing. We study how attention routing evolves across recurrent depth and find a consistent separation in convergence timescales: attention support and distributions stabilize substantially earlier than hidden states and attention outputs. This suggests two stages of recurrent inference: early discovery of a sparse working set, followed by representation refinement over largely stable routing support. Motivated by this finding, we introduce WISE (Working-set Inference with Support Exploitation), a training-free method that uses unrestricted attention during early recurrent steps to discover a block-structured working set, then reuses its support in later steps while keeping attention weights and recurrent refinement dynamic. Controlled interventions show that multi-step discovery yields more effective working sets than first-step selection, and that support reuse better preserves model behavior than more restrictive forms of attention reuse. Across multi-hop QA benchmarks, WISE largely preserves full-attention performance. Matched context-scaling experiments reveal an increasingly favorable quality-efficiency tradeoff as routing support becomes sparser with longer contexts. A sparse-attention implementation achieves up to a 1.76x late-step attention speedup over native FlashAttention at 4K context. Code: https://github.com/tbn5pj/WISE_code.

Momentum

Related topics

Latest papers 206