LLM Inference Efficiency

LLM: Large Language Model

Ask about this topic

Momentum

47 papers in the last four weeks, up 262% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 234

All topics
CardsList

  1. ProgRouter: Online Progress-Guided Orchestration for Multi-Agent LLM Workflows under Quality-Cost Tradeoffs

    Aug 26, 2026Songyuan Li, Ahmed M. Abdelmoniem, Shiqiang WangLLM Inference EfficiencyMulti-Agent Orchestration

    Aston University, UK · Queen Mary University of London, UK · University of Exeter, UK

  2. The Embedder's Dilemma: LLMs Are Better, but at What Cost?

    Aug 13, 2026Adnan El Assadi, Niklas Muennighoff, Jinhyuk LeeLLM Inference EfficiencyLLM Evaluation

    Harvard University · Stanford University · Independent Researcher

  3. Lifecycle-Optimal Tokenization: Vocabulary Size as a Deployment-Regime-Dependent Infrastructure Parameter

    Aug 11, 2026Rima Mittal, Ankit Gubrani, Satyanarayana KakolluLLM Inference EfficiencyLLM Serving

  4. TSPORec: Token Selection via Preference Optimization for LLM-Based Sequential Recommendation

    Aug 10, 2026Wenqiao Zhu, Chao Xu, Haipang Wu +1LLM Inference EfficiencySequential Recommendation

  5. Test-Time Augmentation for LLMs: When Input Diversity Beats Output Diversity at Matched Compute

    Aug 10, 2026Nikita Kozodoi, Zainab Afolabi, Jack ButlerLLM Inference EfficiencySelf-Consistency Decoding

    Amazon Web Services

  6. Measuring and Reducing WebGPU Dispatch Overhead for LLM Inference

    Aug 9, 2026Jędrzej MaczanLLM Inference EfficiencyLLM Inference

    1Independent Researcher

  7. LLM within MCP Matters: Measuring Inefficient Resource Utilization Driven by LLMs

    Aug 9, 2026Minhan Cho, Soyoung Park, Kihyeon Jeong +3LLM Inference EfficiencyInstruction Following

    Remember & Company AI Lab · Sungkyunkwan University · National Assembly Research Service +2

  8. Blast Radius

    Aug 7, 2026MY Pitsane, Hope MogaleLLM Inference EfficiencyAgentic Code Generation

    Algorithm Reconnaissance Division Mankind Research Labs North-West University, RSA · Algorithm Reconnaissance Division Mankind Research Labs University of Pretoria, RSA

  9. A Two-Tier Perspective on Inference-Time Parallelism in Multi-Agent LLM Systems

    Aug 6, 2026Zihan Xu, Haolin Tian, Hai JiangLLM Inference EfficiencyMulti-Agent LLM Systems

    Beijing University of Posts and Telecommunications.

  10. Interpretable Adaptive Sampling for LLM Test-Time Scaling

    Aug 4, 2026Mobina Kashaniyan, Ali JannesariLLM Inference EfficiencyTest-Time Scaling

    Iowa State University Ames, IA, USA

  11. Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving

    Aug 4, 2026Xiang Li, Pengcheng Wang, Huazheng Wang +1LLM Inference EfficiencyLLM Serving

    1Purdue University · 2Oregon State University

  12. Same Task, Different Work: Prompt-Induced Waste in Coding Agents

    Aug 2, 2026Sarel Weinberger, Amir HozezLLM Inference EfficiencyOverthinking in Language Models

    PointFive

  13. Stop When Memory Suffices: Evidence-Conditioned Progressive Execution for LLM Agents

    Aug 2, 2026Yidan Lin, Kaixiang Wang, Jiong Lou +1LLM Inference EfficiencyEfficient Inference

  14. IR2Solve: Structured Intermediate Representations for Cost-Efficient Optimization Autoformulation

    Jul 31, 2026Penglin Zhu, Linhai Zhang, Jungang Xu +2LLM Inference EfficiencyNatural Language to Optimization Modeling

    School of Computer Science and Technology, University of the Chinese Academy of Sciences

  15. Think Short, Defer Smart, Act, and Repeat: Calibrated Reasoning and Uncertainty-Aware Deferral for Edge LLM Agents

    Jul 29, 2026Amirmohammad Farzaneh, Osvaldo SimeoneLLM Inference EfficiencyAI Agent Reliability

    Institute for Intelligent Networked Systems (INSI) Northeastern University London London, UK

  16. From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs

    Jul 29, 2026Tina Vartziotis, Rodopi Kosteli, Elli Vartziotis +5LLM Inference EfficiencyEnergy-Efficient ML

    1National Technical University of Athens, Patission Complex 42, 10682 Athens, Greece · 2Harvard University, 1350 Massachusetts Avenue, 02138 Cambridge, MA, USA · 3TWT GmbH Science & Innovation, Industriestraße 6, 70565 Stuttgart, DE +4

  17. LLMET: Enabling Cross-Layer Evaluation of Emerging M3D Memories for Energy-Efficient LLM Serving

    Jul 29, 2026Ming-Yen Lee, Hanchen Yang, Faaiq Waqar +4LLM Inference EfficiencyLLM Serving

    Georgia Institute of Technology, Atlanta, GA, USA · Samsung Semiconductor Inc., San Jose, CA, USA

  18. Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference

    Jul 27, 2026Yifan Dou, Shikan Lian, Shibo LiLLM Inference EfficiencyLLM Inference Scheduling

    Department of Computer Science, Florida State University

  19. When LLM Defenses Backfire: Characterizing Safety, Performance, and Cost Trade-offs

    Jul 27, 2026Tong Zhang, Zexin Li, Simin Chen +1LLM Inference EfficiencyLanguage Model Safety Evaluation

    Fudan University · University of California, Riverside · Columbia University

  20. SpecBox: Speculative Sandbox Scheduling for Efficient LLM Agent Serving

    Jul 27, 2026Yihui Zhang, Tianyu Wo, Jinghao Wang +7LLM Inference EfficiencyLLM Agent Serving

    Beihang University · University of Leeds · The University of Sydney

  21. Understanding Tone-Dependent Inference Cost in Large Language Models

    Jul 27, 2026Akhil Kumar, Om DobariyaLLM Inference EfficiencyLLM Evaluation

    Pennsylvania State University University Park, PA, USA

  22. Profiling Lightweight Large Language Models

    Jul 23, 2026Tomohiro Harada, Enrique Alba, Gabriel LuqueLLM Inference EfficiencyLLM Evaluation

    Graduate School of Science and Engineering, Saitama University, Saitama, Japan

  23. CORVUS: Context Optimization and Reduction Via Underlying Synchronization for LLM Coding Agents

    Jul 20, 2026Mingwei Zheng, David OBrien, Siwei Cui +4LLM Inference EfficiencyAI Coding Agents

    Department of Computer Science, Purdue University · 2AWS AI Labs

  24. VDAR-Router: Adaptive LLMs Routing via Verbalized Query Difficulty Analysis Retrieval

    Jul 20, 2026Yu-Chien Tang, Jun-Chen Hung, Wen-Chih Peng +1LLM Inference EfficiencyAdaptive Model Routing

    National Yang Ming Chiao Tung University · Department of Computer Science Hsinchu, Taiwan

  25. Human Grounded Evaluation of Large Language Models for Optical Network Automation

    Jul 20, 2026Kiarash Rezaei, Omran Ayoub, Paolo Monti +1LLM Inference EfficiencyLLM Evaluation

    Department of Electrical Engineering, Chalmers University of Technology, 412 96 Gothenburg, Sweden · University of Applied Sciences and Arts of Southern Switzerland, 6928 Lugano, Switzerland

  26. Constrained Path Reasoning: Measuring When Committed Stages Earn Their Cost

    Jul 19, 2026Honglin LiLLM Inference EfficiencyInference-Time Search

    ShanghaiTech University

  27. WaterMoE: Expert-Routing-based Watermarking for High Fidelity and Efficiency

    Jul 14, 2026Z Sun, Q Jiang, S Sheng +1LLM Inference EfficiencyLLM Watermarking

    Shanghai Jiao Tong University · Shanghai Innovation Institute · Tianjin University

  28. Securing LLMs in the Wild: Privacy and Security Challenges at the Edge

    Jul 13, 2026Ren-Yi Huang, Mingchen Li, Dumindu Samaraweera +1LLM Inference EfficiencyEdge Computing

    Department of Electrical and Computer Engineering, University of South Florida · Department of Mathematics, Embry-Riddle Aeronautical University

  29. Valid ≠\ne Necessary: Diagnosing Latent Inefficiency in Chain-of-Thought

    Jul 13, 2026Daeyeop Lee, Hwanjo YuLLM Inference EfficiencyOverthinking in Language Models

    1KT Corporation · 2Pohang University of Science and Technology

  30. Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurations

    Jul 10, 2026Nada Zine, Tristan Coignion, Vincenzo Stoico +3LLM Inference EfficiencyLLM Evaluation

    Univ. Lille, CNRS, Inria · Univ. Bordeaux, CNRS, LaBRI · Vrije Universiteit Amsterdam +1

  31. Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization

    Jul 9, 2026Jiantong Jiang, Peiyu Yang, Rui Zhang +1LLM Inference EfficiencyLLM Serving

    School of Computing and Information Systems, The University of Melbourne · School of Computer Science and Technology, Huazhong University of Science and Technology (www.ruizhang.info)

  32. Tool-Making and Self-Evolving LLM Agents in Low-Latency Systems

    Jul 9, 2026Kalle Kujanpää, Ning Liu, Shahnawaz Alam +4LLM Inference EfficiencyLLM Agent Self-Improvement

    Amazon, Fulfillment Technologies & Robotics

  33. The Harness Effect: How Orchestration Design Sets the Token Economics of Enterprise Agentic AI

    Jul 8, 2026Muayad Sayed Ali, Aliaksandra Novik, Anji Boddupally +28LLM Inference EfficiencyLLM Agent Orchestration

    Writer, Inc.

  34. Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascade

    Jul 7, 2026Kai Ruan, Zihe Huang, Ziqi Zhou +4LLM Inference EfficiencyEarly Stopping

    Gaoling School of Artificial Intelligence, Renmin University of China · Institute of Computing Technology, Chinese Academy of Sciences · Duke University +3

  35. Think Before You Grid-Search: Floor-First Triage for LLM Serving

    Jul 7, 2026Yihua LiuLLM Inference EfficiencyLLM Serving

    Taikang Insurance Group

  36. Inject or Navigate? Token-Efficient Retrieval for LLM Analysis of Transactional Legal Documents

    Jul 7, 2026Mahmoud Hany, Mourad ElSheraey, Mahmoud Said +1LLM Inference EfficiencyLegal IR

    Syntheia Pty Ltd

  37. Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference

    Jul 6, 2026Guanyu Cai, Ruiming Tian, Lang Yang +4LLM Inference EfficiencyAI Accelerator Inference

    Tsinghua University · Beijing Jiaotong University

  38. Risk-Constrained Freshness-Aware Semantic Caching for Open-Web Retrieval-Augmented LLMs

    Jul 5, 2026Muhammad Mansoor, Tahir Ahmad, Yeo-Chan YoonLLM Inference EfficiencyRetrieval-Augmented Generation

    Jeju National University, Jeju, Republic of Korea

  39. Knowing When to Stop: Predicting Execution-Consistency Convergence in Text-to-SQL

    Jul 4, 2026Yaron Anavi, Mor Aisenberg, Nadav Nesher +2LLM Inference EfficiencyLLM Inference

    GIGASPACES

  40. SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference

    Jul 3, 2026Huajun Bai, Weiwei Lv, Huichuan Zheng +2LLM Inference EfficiencyLLM Agent Orchestration

    Tsinghua University · Meituan

  41. WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs

    Jul 2, 2026Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-MartínezLLM Inference EfficiencyEnergy-Efficient ML

    Universidad Politécnica de Madrid, C. de los Ciruelos, 28660 Boadilla del Monte, Madrid, Spain · Zurich University of Applied Sciences, Gertrudstrasse 15, 8400 Winterthur, Switzerland

  42. Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling

    Jul 2, 2026Xuqing Yang, Yi Yuan, Shanzhe Lei +1LLM Inference EfficiencyRL for Language Models

    Shanghai Jiao Tong University · Southeast University · Shanghai AI Laboratory

  43. Precomputing the Future-Offset Average in TriAttention

    Jul 1, 2026Amarnath MukherjeeLLM Inference EfficiencyKV Caching

    Hozhoke, Inc.

  44. From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving

    Jun 30, 2026Jie Li, Tongyang Wang, Yong ChenLLM Inference EfficiencyLLM Serving

    Texas Tech University, Department of Computer Science, Lubbock, Texas, USA

  45. Fork-Think with Confidence

    Jun 30, 2026Zena Al-Khalili, Rafi Hakim, Dietrich Klakow +1LLM Inference EfficiencyEfficient Language Model Reasoning

    Saarland Informatics Campus, Saarland University, Germany

  46. When Reranking Hurts: Uncertainty-Based Gating for Few-Shot Reranking

    Jun 30, 2026Orian Dabod, Amir DN Cohen, Gabriel StanovskyLLM Inference EfficiencyFew-Shot Learning

    The Hebrew University of Jerusalem · OriginAI / Ramat Gan, Israel

  47. Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models

    Jun 24, 2026Xinyu Lian, Walid Krichene, Beichen Huang +4LLM Inference EfficiencyLLM Quantization

    University of Illinois Urbana-Champaign · Microsoft · Anyscale +1

  48. ASAP: Agent-System Co-Design for Wall-Clock-Centered Auto HPO Research for ML Experiments

    Jun 23, 2026Taicheng Guo, Haomin Zhuang, Kehan Guo +4LLM Inference EfficiencyLarge Language Model-Guided Optimization

    University of Notre Dame United States

  49. CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference

    Jun 23, 2026Xiaolin Lin, Jingcun Wang, Olga Kondrateva +3LLM Inference EfficiencyKV Caching

    Technical University of Darmstadt, Darmstadt, Germany · University of Notre Dame, Notre Dame, IN, USA · Technical University of Ilmenau, Ilmenau, Germany

  50. The Language-Energy Divide: Measuring Energy Costs of Multilingual LLM Inference

    Jun 20, 2026Naihao Deng, Alissa Shen, Yiming Feng +5LLM Inference EfficiencyMultilingual Language Models

    University of Michigan · University of Cambridge · University of Aberdeen

  51. Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study

    Jun 19, 2026Alfarizy Alfarizy, Hung Truong Thanh Nguyen, René Richard +2LLM Inference EfficiencyOn-Device Language Model Inference

    Analytics Everywhere Lab, University of New Brunswick, Fredericton, NB, Canada · TSAI Lab, University of New Brunswick, Fredericton, NB, Canada

  52. Latent Personal Memory: Represent personal memory as dynamic soft prompts

    Jun 18, 2026Debrup Das, Avinash Amballa, Yashas Malur Saidutta +3LLM Inference EfficiencyMemory-Efficient Fine-Tuning

    University of Massachusetts Amherst · Samsung Research America

  53. Think Again or Think Longer? Selective Verification for Budget-Aware Reasoning

    Jun 18, 2026Sajib Acharjee Dip, Dawei Zhou, Liqing ZhangLLM Inference EfficiencyCost-Aware Inference

    Department of Computer Science, Virginia Tech · Fralin Biomedical Research Institute, Virginia Tech · FBRI Cancer Research Center, Washington, DC

  54. CacheWeaver: Cache-Aware Evidence Ordering for Efficient Grounded RAG Inference

    Jun 18, 2026Kaizhen Tan, Rong Gu, Mingyuan LiLLM Inference EfficiencyRetrieval-Augmented Generation

    Heinz College of Information Systems and Public Policy, Carnegie Mellon University, USA

  55. SMEPilot: Characterizing and Optimizing LLM Inference with Scalable Matrix Extensions

    Jun 15, 2026Feiyang Chen, Haibo ChenLLM Inference EfficiencyLLM Inference Acceleration

    IPADS, Shanghai Jiao Tong University

  56. DynAMO:Dynamic Asset Management Orchestration via Topological Multi-Agent Scheduling

    Jun 14, 2026Kanishk Kushwaha, Vikrant Vinod Bansode, Harsh Vardhan +1LLM Inference EfficiencyMulti-Agent Orchestration

    Gati Shakti Vishwavidyalaya · IBM Research

  57. Service-Induced Congestion in Memory-Constrained LLM Serving

    Jun 14, 2026Ruicheng Ao, Jing Dong, Gan Luo +1LLM Inference EfficiencyLLM Serving

    Institute for Data, Systems, and Society, Massachusetts Institute of Technology, Cambridge, MA 02139 · Columbia Business School, Columbia University, New York, NY 10027 · School of Mathematical Sciences, Peking University, Beijing, China

  58. The Price of Anarchy in Disaggregated Inference

    Jun 11, 2026Athos GeorgiouLLM Inference EfficiencyGame Theory

    NCA

  59. Can I Buy Your KV Cache?

    Jun 11, 2026Luoyuan ZhangLLM Inference EfficiencyKV Caching

    Harbin Institute of Technology, Shenzhen (HITSZ)

  60. MiniPIC: Flexible Position-Independent Caching in <100LOC

    Jun 11, 2026Nathan Ordonez, Thomas ParnellLLM Inference EfficiencyKV Caching

    IBM Research · IBM Research Zurich, Switzerland