Efficient Language Model Inference

Follow this topic Ask about this topic

Momentum

37 papers in the last four weeks, up 185% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 206

All topics

CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression
Jun 23, 2026Morayo Danielle Adeyemi, Ryan A. Rossi, Franck DernoncourtLLM Evaluation LLM Compression
Independent · Adobe Research
Self-Compacting Language Model Agents
Jun 22, 2026Tianjian Li, Jingyu Zhang, William Jurayj +5Efficient Language Model Inference Metacognition in Language Models
♠Johns Hopkins University · ♡Apple
DART: Draft-Agreement Routing for Training-Free Adaptive Thinking Budgets in Hybrid Reasoning Models
Jun 22, 2026Jungseob Lee, Seongtae Hong, Seungjun Lee +7LLM Routing Efficient Language Model Inference
Korea University · 242dot · Konkuk University +3
EnerInfer: Energy-Aware On-Device LLM Inference
Jun 22, 2026Bohua Zou, Nian Liu, Binqi Sun +6On-Device Language Model Inference LLM Inference Acceleration
Technical University of Munich · Shanghai Jiao Tong University · Huawei Hilbert Research Center (Dresden) +1
RLM-Cascade: Response-Level Speculative Decoding for Cost-Efficient LLM API Serving
Jun 22, 2026Haifeng Wu, Srinivasan Manoharan, Fangbo Tu +2LLM Inference Serving Speculative Decoding
PayPal
When Compression Helps and When It Hurts: Condition-Aware Analysis of Chain-of-Thought Distillation
Jun 19, 2026Siyang Lyu, Xinghao Chen, Zhijing Sun +3Model Compression CoT Distillation
Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Viterbi School of Engineering, University of Southern California · The Hong Kong Polytechnic University +2
ADaPT: Token-Level Decoupling for Efficient Large Reasoning Models
Jun 18, 2026Tingyun Li, Zishang Jiang, Jinyi Han +8Inference-Time Optimization RL for Language Model Reasoning
School of Data Science, Fudan University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University +1
Large Language Models Do Not Always Need Readable Language
Jun 18, 2026Jiayi Zhu, Haoxuan Peng, Junxi Wang +3Efficient Language Model Inference
Shanghai Jiao Tong University · The University of Sydney · Hefei University of Technology +2
Diffusion Language Models: An Experimental Analysis
Jun 17, 2026Thomas Bertolani, Davide Bucciarelli, Leonardo Zini +2Diffusion Language Model Inference Diffusion Language Model Decoding
University of Modena and Reggio Emilia, Italy · University of Pisa, Italy
Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scale
Jun 13, 2026Ang Li, Ben Liu, Bin Han +215Efficient Language Model Training Linear Attention
Stop When Further Reasoning Won't Help: Attention-State Adaptive Generation in Reasoning Models
Jun 13, 2026Jiakai Li, Ke Qin, Rongzheng Wang +4Efficient Language Model Inference LLM Reasoning
University of Electronic Science and Technology of China, Chengdu, China · Ubiquitous Intelligence and Trusted Services Key Laboratory of Sichuan Province, Chengdu, China
Beyond the Commitment Boundary: Probing Epiphenomenal Chain-of-Thought in Large Reasoning Models
Jun 11, 2026Daniel Scalena, Sara Candussio, Luca Bortolussi +3CoT Faithfulness LLM Interpretability
CLCG, University of Groningen · University of Milano-Bicocca · University of Trieste +1
Adaptive Multi-Resolution Procedural Knowledge Compression for Large Language Models
Jun 10, 2026Changyue Wang, Weihang Su, Qingyao Ai +5LLM Compression Efficient Language Model Inference
Department of Computer Science and Technology, Tsinghua University
Express Language Modeling
Jun 9, 2026Albert Gong, Annabelle Michael Carrell, Raaz Dwivedi +1Long-Context Language Model Inference Attention Mechanisms
Cornell Tech · University of Cambridge · Microsoft Research
Stop Early, Spend Less: Hidden-State Probes as a Practical Recipe for Streaming Moderation of LLM Outputs
Jun 9, 2026Huizhen Shu, Xuying Li, Piao XueLLM Guardrails Safety Filtering
ModelOneAI · yunshanai
Q-Delta: Beyond Key-Value Associative State Evolution
Jun 7, 2026Sumin Park, Seojin Kim, Noseong ParkLinear Attention Delta-Rule Linear Attention
Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea.
How Small Can You Go? LoRA Fine-Tuning 270M-8B Models for Merchant Information Extraction in Financial Transactions
Jun 6, 2026Donghao Huang, Tomas Drietomsky, Benjamin Barrett +1Financial Services Memory-Efficient Fine-Tuning
Research and Development, Mastercard, Arlington, VA, USA · School of Computing and Information Systems, Singapore Management University, Singapore · Economic Intelligence, Mastercard, Prague, Czech Republic
Sparsely gated tiny linear experts
Jun 5, 2026Simon SchugTransformer FFNs Mixture-of-Experts Language Models
Department of Computer Science Princeton University
DyCon: Dynamic Reasoning Control via Evolving Difficulty Modeling
Jun 5, 2026Tengyao Tu, Yulin Li, Hui-Ling Zhen +6Overthinking in Language Models Efficient Language Model Reasoning
Harbin Institute of Technology, Shenzhen · Zhongguancun Academy · Huawei Noah’s Ark Lab +2
When to Think Deeply: Inhibitory Deliberation for LLM Reasoning
Jun 4, 2026Zhixuan He, Yue FengEfficient Language Model Reasoning Inference-Time Compute Allocation
University of Birmingham, United Kingdom
Latent Reasoning with Normalizing Flows
Jun 4, 2026Guancheng Tu, Xiangjun Fu, Suhao Yu +5Normalizing Flows Continuous Latent Reasoning
University of Pennsylvania · UC San Diego · Meta
Domain-Adapted Small Language Models with Hybrid Post-Processing: Achieving Cost-Efficient, Low-Latency Multi-Label Structured Prediction via LoRA Fine-Tuning on Scarce Data
Jun 4, 2026Srinivasan Manoharan, Dilipkumar Nallusamy, Sachin Kumar +1Multi-Label Classification Small Language Models
PayPal Inc San Jose, USA · PayPal Inc Chennai, India · PayPal Inc Bangalore, India
Imbuing Large Language Models with Bidirectional Logic for Robust Chain Repair
Jun 3, 2026Zehua Cheng, Wei Dai, Jiahao Sun +1LLM Self-Correction Symbolic Reasoning
Department of Computer Science, University of Oxford, UK · Institute of Logic and Computation, TU Wien, Austria
SharedRequest: Privacy-Preserving Model-Agnostic Inference for Large Language Models
Jun 3, 2026Peihua Mai, Xuanrong Gao, Youlong Ding +3Privacy-Preserving Language Models LLM Inference
1National University of Singapore (Chongqing) Research Institute · 2Chongqing Key Laboratory of Trusted Perception and Interaction Technology for Intelligent and Connected Vehicles, Chongqing, China · 3National University of Singapore +3
SMADE-IE: Sparse Multi-Agent Framework with Evidence-Driven Debate for Zero-Shot Information Extraction
Jun 3, 2026Kenfeng Huang, Yi Cai, Xin Wu +2Multi-Agent LLM Systems Multi-Agent Debate
School of Software Engineering, South China University of Technology, Guangzhou, China
LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encoding
Jun 3, 2026Haocheng Xia, Mihir Pamnani, Hanxi Fang +2Retrieval-Augmented Generation KV Caching
Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign · Nexla · Amazon +1
Agentic Chain-of-Thought Steering for Efficient and Controllable LLM Reasoning
Jun 2, 2026Yu Xia, Zhouhang Xie, Xin Xu +4Language Model Steering CoT Reasoning
University of California San Diego1 · Intuit AI Research2
HybridThinker: Efficient Chain-of-Thought Reasoning via Compressed Memory and Transient Thought Steps
Jun 2, 2026Xin Liu, Runsong Zhao, Xinyu Liu +8Memory-Augmented Language Models Efficient Language Model Reasoning
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · China Unicom Cloud-Link, China · NiuTrans Research, Shenyang, China
ThoughtFold: Folding Reasoning Chains via Introspective Preference Learning
Jun 2, 2026Ziyan Liu, Xueda Shen, Yuzhe Gu +7Efficient Language Model Reasoning Preference Optimization
Shanghai Artificial Intelligence Laboratory · University of Science and Technology of China · MMLab, The Chinese University of Hong Kong
Adaptive Latent Agentic Reasoning
Jun 1, 2026Dongwon Jung, Peng Shi, Yi Zhang +2Agentic Reasoning LLM Agents
University of California, Davis · University of Waterloo · Greenshoe, Inc.

Jun 23, 2026, cs.CLJ/K move · Enter open · S save

CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression

Morayo Danielle Adeyemi, Ryan A. Rossi, Franck Dernoncourt

Independent · Adobe Research

LLM Evaluation LLM Compression Language Model Generation Evaluation Efficient Language Model Inference Prompt Compression

Open paper AskPDF arXiv

"Talk short. Drop grammar. Save token." This caveman style is widely promoted as a way to cut inference cost, but whether it actually saves anything depends on which channel (the user's prompt or the model's response) is being compressed. We present Cavewoman, a two-channel evaluation protocol that scores every generation on task accuracy, realized per-item cost, and reference-text agreement against the model's unconstrained reference. We evaluate eight models on five datasets at five reduction levels, with both channels measured on the same items. Output compression cuts realized cost on most API models (1.4-2.4x per model, up to 3x in the best case) and on all four open-weight models under public-tier pricing. Input compression has the opposite effect, a strict lose-lose: it raises net cost rather than lowering it (~1.15x on the five-benchmark mean, up to 1.8x on the worst dataset and 2.7x under stronger compression), because models compensate with longer responses even as accuracy collapses. Under the same setting, surface text diverges from the unconstrained reference: on the non-reasoning models, roughly half of all generations are correct yet their surface text no longer entails the model's own unconstrained baseline generation. The divergence survives length-controlled re-scoring, multiple-comparisons correction, and replication under complementary semantic measures. Code and data are available at https://github.com/danielle34/cavewoman.

Momentum

Related topics

Latest papers 206