Efficient Language Model Inference

Follow this topic Ask about this topic

Momentum

37 papers in the last four weeks, up 185% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 206

All topics

LLMForge: Multi-Backend Hardware-Aware Neural Architecture Search with Infinite-Head Attention for Edge Language Models
May 17, 2026Xinting Jiang, Junyi Luo, Ruichen Qi +4On-Device Language Model Inference Hardware-Aware NAS
Brown University · University of Michigan · Google Research
CAPS: Cascaded Adaptive Pairwise Selection for Efficient Parallel Reasoning
May 15, 2026Fangzhou Lin, Shuo Xing, Peiran Li +6Pairwise Comparison Pairwise Preference Evaluation
Texas A&M University · Georgia Institute of Technology · Tohoku University +1
Enhanced and Efficient Reasoning in Large Learning Models
May 13, 2026Leslie G. ValiantRepresentation Learning World Models
John A. Paulson School of Engineering and Applied Sciences Harvard University, Boston, MA 02134
Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputs
May 12, 2026Guinan Su, Yanwu Yang, Xueyan Li +1LLM Agents Instruction Tuning
Max Planck Institute for Intelligent Systems · University Hospital Tübingen · University of Tübingen +3
Training-Inference Consistent Segmented Execution for Long-Context LLMs
May 12, 2026Xianpeng Shang, Jiang Li, Zehua Duo +2Self-Attention Long-Context Language Modeling
College of Computer Science, Inner Mongolia University, Hohhot 010021, China · National & Local Joint Engineering Research Center of Intelligent Information Processing Technology for Mongolian, Hohhot 010021, China · Inner Mongolia Key Laboratory of Multilingual Artificial Intelligence Technology, Hohhot 010021, China +1
Nice Fold or Hero Call: Learning Budget-Efficient Thinking for Adaptive Reasoning
May 12, 2026Zhaomeng Zhou, Lan Zhang, Junyang Wang +2RL for Language Model Reasoning Efficient Language Model Reasoning
University of Science and Technology of China · The Chinese University of Hong Kong
SOMA: Efficient Multi-turn LLM Serving via Small Language Model
May 11, 2026Xueqi Cheng, Qiong Wu, Zhengyi Zhou +3LLM Serving Small Language Models
Florida State University · AT&T Chief Data Office · Louisiana State University +1
Enabling Performant and Flexible Model-Internal Observability for LLM Inference
May 11, 2026Nengneng Yu, Sixian Xiong, Yibo Zhao +2LLM Inference Serving Efficient Language Model Inference
Department of Computer Science University of Maryland, College Park
EnergyLens: Interpretable Closed-Form Energy Models for Multimodal LLM Inference Serving
May 11, 2026Vittorio Palladino, Gianluca Palermo, Michael E. Papka +1Efficient Multimodal Inference LLM Inference
Politecnico di Milano & UIC · Politecnico di Milano · University of Illinois Chicago
Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Models
May 10, 2026Lin Zheng, Vasilisa Bashlovkina, Timothy Dozat +3Byte-Level Language Model Language Modeling
1Google DeepMind · 2The University of Hong Kong
Position: Avoid Overstretching LLMs for every Enterprise Task
May 10, 2026Kuldeep Singh, Anson Bastos, Isaiah Onando Mulang'Language Modeling Neuro-Symbolic AI
Microsoft, India · SAP, Germany
GRC: Unifying Reasoning-Driven Generation, Retrieval and Compression
May 9, 2026Zhongtao Miao, Qiyu Wu, Yoshimasa TsuruokaRetrieval-Augmented Generation Memory-Augmented Language Models
The University of Tokyo · Tsuruoka Lab. · Independent Researcher
Reasoning Compression with Mixed-Policy Distillation
May 9, 2026Han Yang, Mingyan Wu, Bailan He +4Language Model Distillation Efficient Language Model Inference
Technical University of Munich · GESIS – Leibniz Institute for the Social Sciences · Northeastern University +5
Hint Tuning: Less Data Makes Better Reasoners
May 9, 2026Siqi Fan, Minghao Li, Xiaoqian Ma +6Overthinking in Language Models LLM Fine-Tuning
University of Electronic Science and Technology of China · Xiaohongshu Inc. · National University of Singapore
Reliable Chain-of-Thought via Prefix Consistency
May 8, 2026Naoto Iwase, Yuki Ichihara, Mohammad Atif Quamar +1Self-Consistency Decoding Reasoning Consistency in Language Models
Nagoya University · Nara Institute of Science and Technology · Mohamed bin Zayed University of Artificial Intelligence +1
Confidence-Aware Alignment Makes Reasoning LLMs More Reliable
May 8, 2026Kejia Chen, Jiawen Zhang, Yihong Wu +5LLM Alignment Direct Preference Optimization
Zhejiang University · Université de Montréal · Sun Yat-sen University +1
A Few Good Clauses: Comparing LLMs vs Domain-Trained Small Language Models on Structured Contract Extraction
May 7, 2026Nicole Lincoln, Nick Whitehouse, Jaron Mar +1Legal NLP Mixture-of-Experts Language Models
Onit AI Labs, Onit Inc., New Zealand
SCOUT: Active Information Foraging for Long-Text Understanding with Decoupled Epistemic States
May 6, 2026Zhenliang Zhang, Wenqing Wang, Yong Hu +4Document Understanding Long-Context QA
Peking University · WeChat, Tencent · Work done during an internship at WeChat, Tencent
Gated Subspace Inference for Transformer Acceleration
May 4, 2026Stephen J. ThomasTransformer Inference LLM Inference Acceleration
Department of Computer Science and Engineering, Lehigh University, Bethlehem, PA 18015
When Less is Enough: Efficient Inference via Collaborative Reasoning
May 1, 2026Yilei Chen, Sharut Gupta, Yannis Paschalidis +2Efficient Language Model Reasoning Efficient Language Model Inference
Boston University · MIT CSAIL · Biohub
Component-Aware Self-Speculative Decoding in Hybrid Language Models
May 1, 2026Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-CarbóSpeculative Decoding Language Modeling
VRAIN – Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · Department of Economics and Social Sciences, Universitat Politècnica de València, Valencia, Spain · Department of Business Organisation, Universitat Politècnica de València, Valencia, Spain
Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decoding
May 1, 2026Lehan Pan, Ziyang Tao, Ruoyu Pang +3Mixture-of-Experts Inference Speculative Decoding
University of Science and Technology of China · Tianyijiaotong Technology Ltd., Suzhou, China
Select to Think: Unlocking SLM Potential with Local Sufficiency
Apr 29, 2026Wenxuan Ye, Yangyang Zhang, Xueli An +2Small Language Models Efficient Language Model Reasoning
1Technical University of Munich · 3Ludwig Maximilian University of Munich · 2Huawei Technologies +2
Dual-Track CoT: Budget-Aware Stepwise Guidance for Small LMs
Apr 27, 2026Sagnik Chatterjee, Atharva Patil, Sricharan RameshCoT Reasoning Small Language Models
Nautile-370M: Spectral Memory Meets Attention in a Small Reasoning Model
Apr 27, 2026Maixent ChenebauxSelf-Attention Small Language Models
Trickstr
Stabilizing Efficient Reasoning with Step-Level Advantage Selection
Apr 27, 2026Han Wang, Xiaodong Yu, Jialian Wu +4RL for Language Model Reasoning Efficient Language Model Reasoning
UNC Chapel Hill · Advanced Micro Devices, Inc.
Thinking Without Words: Efficient Latent Reasoning with Abstract Chain-of-Thought
Apr 24, 2026Keshav Ramji, Tahira Naseem, Ramón Fernandez AstudilloEfficient Language Model Reasoning Implicit Reasoning in Language Models
IBM Research AI
An End-to-End Ukrainian RAG for Local Deployment. Optimized Hybrid Search and Lightweight Generation
Apr 23, 2026Mykola Trokhymovych, Yana Oliinyk, Nazarii NyzhnykRetrieval-Augmented Generation Hybrid Retrieval
Pompeu Fabra University · Independent Researcher
Thinking with Reasoning Skills: Fewer Tokens, More Accuracy
Apr 23, 2026Guangxiang Zhao, Qilong Shi, Xusen Xiao +3Efficient Language Model Reasoning Efficient Language Model Inference
Qiyuan Tech · Tsinghua University · The University of Hong Kong +1
TRACES: Tagging Reasoning Steps for Adaptive Cost-Efficient Early-Stopping
Apr 22, 2026Yannis Belkhiter, Seshu Tirupathi, Giulio Zizzo +1LLM Inference Efficiency Early Stopping
IBM Research Europe · Trinity College Dublin · Dublin, Ireland +1

May 17, 2026, cs.LGJ/K move · Enter open · S save

LLMForge: Multi-Backend Hardware-Aware Neural Architecture Search with Infinite-Head Attention for Edge Language Models

Xinting Jiang, Junyi Luo, Ruichen Qi, Kauna Lei+3

Brown University · University of Michigan · Google Research

On-Device Language Model Inference Hardware-Aware NAS Attention Mechanisms Efficient Language Model Inference

Open paper AskPDF arXiv

Sub-billion-parameter Transformer language models are increasingly deployed on edge devices, where the privacy, latency, and operating-cost advantages of on-device inference are constrained by tight memory-bandwidth, energy, and thermal budgets that make architectural choice and accelerator-specific cost central to efficient inference. We present LLMForge, a hardware-aware neural architecture search (NAS) framework whose three composable contributions together make edge-LM architecture search hardware-conditioned, since different substrates impose different hardware cost bottlenecks. Infinite-Head Attention (IHA) decouples the number of query heads, KV groups, and per-head query/key and value dimensions, expanding the feasible per-layer attention configuration space by approximately 400x over grouped-query attention within our search-space ranges. Forge-Former, an encoder-based surrogate for ranking architectural candidates, outperforms MLP and random-forest baselines. Forge-DSE, an NSGA-II-based design-space-exploration engine, pairs Forge-Former with a multi-backend hardware cost model spanning GPUs, systolic accelerators, and ring-dataflow edge accelerators. Across four different hardware substrates, the searches converge to visibly different architectures whose shapes track each substrate's cost bottleneck. On the multi-chip ring substrate, our co-search returns three 300M-scale deployment-aware variants on the Pareto front. Each is re-trained on FineWeb-Edu-10BT under matched recipe against SmolLM2-360M and Qwen-0.5B architecture baselines. The accurate variant has the lowest validation loss 2.798 and competitive benchmark performance with fewer parameters, the energy-optimized variant lowers energy per token by 40%, and the latency-optimized variant lowers TTFT and TPOT by 43%.

Momentum

Related topics

Latest papers 206