LLM Inference Efficiency

LLM: Large Language Model

Ask about this topic

Momentum

47 papers in the last four weeks, up 262% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 234

All topics
CardsList

  1. Attention via Black-Box Vector Search

    Oct 7, 2026Stepan Zharkov, Krish Singal, Ashwin Padaki +1Efficient AttentionSparse Attention

    University of Pennsylvania

  2. ReFold: Training-Free Reversible Inter-Turn Context Folding for Long-Horizon Agents

    Oct 6, 2026Yupeng Su, Jiayi Tian, Zheng Zhang +1LLM Inference EfficiencyLong-Context Language Model Inference

    University of California, Santa Barbara · Intel

  3. Persistent Memory in Multi-Agent LLM Inference: What It Costs, What It Buys, and When You Can Tell

    Oct 6, 2026Hochan Son, Kyungdoe Han, Jaehan Koh +3LLM Inference EfficiencyMulti-Agent LLM Systems

    University of California, Los Angeles · University of Wisconsin · HCLTech America

  4. Monte Carlo Estimation for KV Cache Eviction

    Oct 6, 2026Ahsan Bilal, Muhammad Ahmed Mohsin, Muhammad Umer +5LLM Inference EfficiencyKV-Cache Eviction

    University of Oklahoma · Stanford University · University of Houston +3

  5. Evaluating Inference Compute for Generative AI: A Framework for Enterprise Workloads

    Oct 5, 2026Abbas Raza Ali, Muhammad Ajmal Siddiqui, Moona ZahidLLM Inference EfficiencyDisaggregated LLM Serving

    Citigroup Inc., London, United Kingdom · Ernst & Young LLP, London, United Kingdom · NVIDIA Corporation, London, United Kingdom

  6. Expanding LLM Reasoning

    Oct 4, 2026Rian Atri, Evan LuoLLM Inference EfficiencyInference-Time Scaling

    Keiji AI · University of California, Berkeley

  7. Beyond State-of-the-Art: Standardising Environmental Impact Metrics for AI Research

    Oct 1, 2026Lachlan McGinness, Dan Pagendam, Robert OffnerLLM Inference EfficiencyEnergy-Efficient ML

    School of Computing, Australian National University · Commonwealth Scientific and Industrial Research Organisation

  8. From Discovery to Decision: Finite-Budget Recoverability in LLM Voting

    Oct 1, 2026Shaoang Li, Jian LiLLM Inference EfficiencyLLM Inference

    Stony Brook University

  9. Beyond Leaderboards: Tokenomics of Agentic Small Language Model Ensembles

    Oct 1, 2026Alexei N. Skurikhin, Emily M. Taylor, Nathan A. DeBardelebenLLM Inference EfficiencyLanguage Model Generation Evaluation

    Los Alamos National Laboratory Los Alamos, New Mexico, USA

  10. Characterizing High Bandwidth Flash for LLM Serving

    Sep 30, 2026Zack Yu, Chloe Wong, Coleman Hooper +7LLM Inference EfficiencyLLM Serving

    University of California, Berkeley Berkeley, California, USA · FuriosaAI Seoul, South Korea · University of California, Berkeley; ICSI; LBNL Berkeley, California, USA

  11. Adaptive Self-Consistency: From Black-Box Sampling to Distribution-Valued Feedback

    Sep 30, 2026Jingkai Huang, Yunfan Zhang, Will Ma +2LLM Inference EfficiencyAdaptive Inference

    Stern School of Business New York University · School of Management Zhejiang University

  12. Purlin: Separating Orchestration from the Datapath of Collectives

    Sep 29, 2026Osayamen Jonathan Aimuyo, Swapnil Gandhi, Christos KozyrakisLLM Inference EfficiencyHigh-Performance Computing

    Stanford University · NVIDIA & Stanford University

  13. Beyond Energy: When Sustainability Dimensions Reshape LLM Serving Decisions

    Sep 28, 2026Tianyao Shi, Xipeng Shen, Yi DingLLM Inference EfficiencyLLM Serving

    Elmore Family School of Electrical and Computer Engineering, Purdue University, USA

  14. WavePP: High-Throughput Pipeline Parallel LLM Prefill under Prefix Reuse

    Sep 28, 2026Aaryam SharmaLLM Inference EfficiencyLLM Serving

    Baseten

  15. SCBO: Semantically Coherent Batching and Ordering for LLM-Based Social Surveys

    Sep 28, 2026Yuanzi Li, Lingjie Wang, Zihang Tian +2LLM Inference EfficiencyLLM Prompting

    Renmin University of China

  16. Beneath the Tokens: A Performance Engineering Study of Multi-Token Prediction in GPU-Accelerated LLM Inference

    Sep 28, 2026Suwesh Prasad SahLLM Inference EfficiencyMulti-Token Prediction

  17. AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs

    Sep 28, 2026Cheuk Hang Lau, Zeyu Cao, Kevin Wong Cheuk Yin +6LLM Inference EfficiencyLLM Inference

    Imperial College London · University of Cambridge · University of Oxford

  18. Spexis: Speculative Lookahead Scheduling for LLM Inference

    Sep 28, 2026Hyungyu Jung, Jaehyeok Yu, Hoonseo Choi +3LLM Inference EfficiencyLLM Inference Scheduling

    Seoul National University · Hanyang University

  19. Greenpixie's AI Token Methodology: Assessing the Energy, Water and CO2-eq Impact of AI Tokens for Open and Closed Weight Models

    Sep 27, 2026Joshua Horswill, Ross Hunter, Matt Clifford +1LLM Inference EfficiencyLLM Inference

    Greenpixie Ltd., London, UK.

  20. You Only Edit Once: Incentivizing In-Context Capability of LLMs via Local Demonstration Refinement

    Sep 27, 2026Jiarong Wen, Qi Wang, Yun Qu +8LLM Inference EfficiencyIn-Context Example Selection

    Department of Automation, Tsinghua University

  21. JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places

    Sep 24, 2026Delip Rao, Chris Callison-BurchLLM Inference EfficiencyLLM Evaluation

    University of Pennsylvania

  22. When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reuse

    Sep 24, 2026Yiyu Liu, Minlan Yu, Juncheng YangLLM Inference EfficiencyKV Caching

    Harvard University

  23. Scaling of Capability and Efficiency at Inference Time in Large Reasoning Models

    Sep 22, 2026Moritz Laber, Zohair Shafi, Germans Savcisens +6LLM Inference EfficiencyLanguage Model Scaling Laws

    Northeastern University, Boston, Massachusetts, USA · Complexity Science Hub Vienna, Vienna, Austria · Santa Fe Institute, Santa Fe, New Mexico, USA

  24. COMED: The Missing Middle Between Routing and Collaboration in Multi-LLM Inference

    Sep 22, 2026Norah Alballa, Wenxuan Zhang, Salma Kharrat +4LLM Inference EfficiencyLLM Routing

    KAUST · LUMS

  25. REFLEX with Jev for Efficient Selective Control in LLM Agents

    Sep 22, 2026Tiantong Wu, Wei Yang Bryan LimLLM Inference EfficiencyLLM Agent Evaluation

    Nanyang Technological University Singapore

  26. Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference

    Sep 22, 2026Md Mostafizer Rahman, Md Faizul Ibne Amin, Md Shahajada Mia +2LLM Inference EfficiencyLong-Context Language Model Inference

    Lucy Family Institute for Data & Society, University of Notre Dame, IN, USA · The University of Aizu, Aizuwakamatsu, Japan · Department of Applied and Computational Mathematics and Statistics, University of Notre Dame, IN, USA

  27. Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directions

    Sep 21, 2026Niloofar Gholipour, Marcos Assuncao, Gursimran Singh +8LLM Inference EfficiencyReinforcement Learning

    École de technologie supérieure, Univ. of Québec, Canada · Huawei Technologies, Canada · The Univ. of Melbourne, Australia +1

  28. Open-Jev Judgments on CallScreenBench: Calibrated One-Pass Scam Screening with a Small Language Model

    Sep 21, 2026Simiao Ren, Kidus Zewde, Xingyu Shen +6LLM Inference EfficiencyLLM Evaluation

    Scam.ai (Reality Inc.)

  29. ValueDiff: Value-Geometric KV Cache Eviction for Sink-Suppressed LLMs

    Sep 20, 2026Junyoung Park, Jungwook Choi, Mingu LeeLLM Inference EfficiencyKV-Cache Eviction

    Qualcomm AI Research · Hanyang University

  30. Not All AI Agents Are Equal: Characterizing Resource and Performance Dynamics

    Sep 17, 2026Wonmi Choi, Minuk Park, Zhixiong Niu +3LLM Inference EfficiencyLLM Agent Workflow Optimization

    Department of Computer Science and Engineering, Korea University · Microsoft Research Asia

  31. From "Who Is This User?" to "What Does This Purchase Mean?": A Deployed Pipeline for Semantic User Profiling at Bank Scale

    Sep 17, 2026Ryota Mitsuhashi, Tetsuro Morimura, Hirotake ItoLLM Inference EfficiencyFinancial Services

    CyberAgent, Tokyo, Japan

  32. PrefixBench-H100: Characterizing Prefix Reuse and Time-to-First-Token in H100 LLM Serving

    Sep 17, 2026Omkar Shewale, Deepak Kumar, Divakar Kumar YadavLLM Inference EfficiencyLLM Inference

    Illinois Institute of Technology · University of Wisconsin–Milwaukee

  33. Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling

    Sep 16, 2026Mobina Kashaniyan, Ali JannesariLLM Inference EfficiencyLLM Inference

    Department of Computer Science Iowa State University, USA

  34. Ask the Tool, Don't Guess: Agent Tool Calls Hold Their Progress, and the Serving System Should Read It

    Sep 16, 2026Yipeng Liu, Yingqiang Zhang, Feifei Li +1LLM Inference EfficiencyKV Caching

    Tsinghua University, China · Alibaba Cloud Computing, China · Zhejiang University, China

  35. Dependency-Aware Trajectory Refinement for Efficient Multi-Turn Agent Fine-Tuning

    Sep 16, 2026Zhuo Chen, Zhen Zhang, Xinyu Wang +1LLM Inference EfficiencyLLM Agent Workflow Optimization

    Shanghai Engineering Research Center of Intelligent Vision and Imaging

  36. Where Should Agents Live? Energy-Memory Characterization of Agentic AI for the Edge-Cloud Continuum

    Sep 16, 2026Carolina Fortuna, Vid Hanžel, Tim Strnad +1LLM Inference EfficiencyEnergy-Efficient ML

    Department of Communication Systems, Jožef Stefan Institute, Jamova cesta 39, 1000 Ljubljana, Slovenia

  37. When to Call an LLM: A Confidence-Gated Hybrid for Cost-Effective Emotion Recognition in Conversational AI

    Sep 16, 2026Sai Babu Udayagiri, Arjun Chouhan, Ravisekhar Kanagala +1LLM Inference EfficiencySelective Prediction

    Five9

  38. The Inference Engineering Pareto Atlas: Which Optimizations Dominate the Cost, Quality, and Latency Frontier?

    Sep 15, 2026Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri +3LLM Inference EfficiencyLLM Quantization

    Vizuara

  39. Shared-Prefix KV Reuse Across Standard LoRA Adapters: Quality and Serving Tradeoffs

    Sep 15, 2026Dushyant RajputLLM Inference EfficiencyLLM Serving

    AltSlate Labs LLP

  40. Validating Hybrid-State Cache Recovery for GLM-5.3-Flash with vLLM and LMCache

    Sep 14, 2026Frank LiLLM Inference EfficiencyKV-Cache Management

    UNSW Sydney

  41. Dissecting GPU Utilization for LLM Inference on Nvidia Hopper

    Sep 14, 2026Mohammad Siavashi, Gerald Q. Maguire Jr., Dejan Kostic +1LLM Inference EfficiencyGPU Acceleration

    KTH Royal Institute of Technology, Stockholm, Sweden

  42. GGUF-Metadata Prediction of Single-Sequence llama.cpp Throughput Across Three Systems

    Sep 14, 2026Xinyu Qiu, Chuhong Xu, Bo Su +3LLM Inference Efficiency

    Northeastern University · Sofia University · Indiana University +2

  43. Pull: Lazy Materialization of Working Memory for Stateful LLM Conversations

    Sep 13, 2026Jiangang ChenLLM Inference EfficiencyLLM Memory

    Chengdu Beiluoshimen Technology Co., Ltd.

  44. TriCalRAG: A Three-Strategy, Retrieval-Augmented Benchmark for On-Premise LLM-Based Root Cause Analysis in AIOps

    Sep 13, 2026Rohit Patel, Susil Kumar Mohanty, Jeenal ChaudharyLLM Inference EfficiencyRetrieval-Augmented Generation

    Department of Computer Science and Engineering Indian Institute of Technology Jodhpur, Jodhpur, India

  45. REVA: Reusable Evidence View Aggregation for Context-Efficient RAG Serving

    Sep 11, 2026Tuan Nguyen, Qiran Hu, Banruo Liu +3LLM Inference EfficiencyRetrieval-Augmented Generation

    VinUni-Illinois Smart Health Center, VinUniversity, Vietnam · University of Illinois Urbana-Champaign, USA

  46. Phase-Decoupled, Model-Calibrated Power Control for Disaggregated LLM Serving

    Sep 11, 2026Jae Gon Kim, Donghoon Yoo, Hanyul Ryu +3LLM Inference EfficiencyDisaggregated LLM Serving

    Xenoscube, Inc.

  47. From Fixed Keys to Readable Schemas: Small Language Models for Vehicle Agent Function Calls

    Sep 10, 2026Hamed Jafarzadeh Asl, Yuanhao Yu, Vahid Partovi NiaLLM Inference EfficiencyOn-Device Language Model Inference

    Huawei Noah’s Ark Lab, Canada · Huawei Noah’s Ark Lab · Polytechnique Montreal, Canada

  48. Composable CXL Memory as a Kubernetes-Native Shared Memory for LLM Serving

    Sep 9, 2026Hongjian Fan, Kevin Zhang, David Habinsky +1LLM Inference EfficiencyKV Caching

    Research Group, Seagate Technology, LLC

  49. PELM: Power Efficient On-Device LLM Inference with Speculative Decoding and Dynamic Voltage Frequency Scaling

    Sep 9, 2026Weisi Yang, Stephen XiaLLM Inference EfficiencyOn-Device Language Model Inference

    Northwestern University Evanston, Illinois, USA

  50. A Measurement Study of LLM Inference Trade-offs Across Edge Continuum Hardware

    Sep 8, 2026Maysam Khatib, Moysis Symeonides, Demetris Trihinas +2LLM Inference EfficiencyLLM Evaluation

    University of Cyprus, Nicosia, Cyprus · University of Nicosia, Nicosia, Cyprus

  51. MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference

    Sep 7, 2026Michael Wang, Keith Li, Roozbeh BostandoostLLM Inference EfficiencyLLM Inference

    Lake Washington School District · XSchool · UMass Amherst

  52. Signed Rescue Routing: Harm-Aware Cascades for Efficient LLM Inference

    Sep 7, 2026Zheyuan Wang, Siyu Li, Peiqiao Song +3LLM Inference EfficiencyLLM Routing

    School of Artificial Intelligence, Beijing Normal University · School of Information Science and Engineering, Chongqing Jiaotong University · School of Computer Science and Technology, Dalian University of Technology +1

  53. What Else Needs Fixing? Exploring Cost-Effective Test-Time Compute for Revision Propagation in Artifacts Generated Through Conversation

    Sep 3, 2026Daisuke KikutaLLM Inference EfficiencyLLM Evaluation

    NTT, Inc.

  54. LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference

    Sep 2, 2026Renyuan Liu, Yuyang Leng, Kaiyan Liu +7LLM Inference EfficiencyOn-Device Language Model Inference

    Richard · George Mason University · Global Technology Applied Research, JPMorganChase +1

  55. Codebook Agent: Amortized Topology Design for LLM Multi-Agent Systems

    Sep 2, 2026Jinxi Yu, Yubei Li, Eric Hanchen Jiang +6Multi-Agent LLM SystemsLLM Inference Efficiency

    University of California, Los Angeles

  56. How Do Prompt Variations Affect Energy Consumption in On-Device LLMs?

    Sep 1, 2026Wei Hu, Xiaolong Tu, Dawei Chen +3LLM Inference EfficiencyOn-Device Language Model Inference

    Georgia State University · Toyota Motor North America

  57. TRIAGE: Three-level Routing and Intelligent Agent Guidance for Efficient Execution

    Sep 1, 2026Ruocan WeiLLM Inference EfficiencyLLM Agent Skill Learning

    China Telecom Cloud, Beijing, China

  58. Cheap Verifiers, Large Blind Spots: Measuring the Reliability Cost of Cost-Saving Cascades

    Sep 1, 2026Dushyant RajputLLM Inference EfficiencyLLM Answer Verification

    AltSlate Labs LLP

  59. ContextPipe: Database-Inspired Context Assembly for Long-Horizon Agents

    Sep 1, 2026Peng Xu, Zuyu Zhang, Yuze Sun +3LLM Inference EfficiencyLong-Horizon LLM Agents

    MatrixOrigin · Tsinghua University

  60. What It Costs to Compose, Rebuild, and Correct Precomputed Memory

    Aug 31, 2026Asa ShepardLLM Inference EfficiencyMemory-Augmented Language Models

    Department of Computer Science, Williams College