LLM Inference Optimization

Ask about this topic

Momentum

67 papers in the last four weeks, up 148% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 464

All topics
CardsList

  1. Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs

    Oct 4, 2026Javad Mirzaei, Jeebak MitraLLM Inference Optimization

    Office of CTO Dell Technologies Inc. Canada

  2. TopK-Guided: Adaptive, Budget-Aware Activation Sparsity for Efficient LLM Inference

    Oct 1, 2026Mukund Agarwalla, Chih-Jen LinLLM Inference OptimizationSparsity

    MBZUAI · National Taiwan University

  3. HHR: Hierarchical Hash Retrieval for Efficient LLM Generation

    Oct 1, 2026Lianjun Liu, Tiantian Zheng, You Huang +5LLM Inference OptimizationDynamic Sparse Attention

    Hainan University, Haikou, China · Xiamen University, Xiamen, China · Zhejiang Normal University, Jinhua, China

  4. Lingtai: What Concept Geometry Reveals--and Does Not Reveal--About LLM Inference

    Sep 30, 2026Jiangang ChenLLM Inference OptimizationSemantic Anchor

    Chengdu Beiluoshimen Technology Co., Ltd.

  5. AIR-LLM: Broadcasting AI Weights over Radio for Memory-Free Edge LLM Inference via RF Computing

    Sep 30, 2026Zhihui Gao, Tingjun Chen, Dirk EnglundLLM Inference OptimizationEdge Devices

    Massachusetts Institute of Technology Cambridge, MA, USA · Duke University Durham, NC, USA

  6. QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs

    Sep 30, 2026Weili Xu, Jisen Li, Yuqing Jian +8Large Language Model QuantizationQuantization-Aware Training

    Together AI · University of Illinois Urbana-Champaign · The University of Texas at Austin

  7. SparseEngine: Sparse-First Inference Engine

    Sep 30, 2026Jitai Hao, Quansheng Gu, Qiang Huang +1LLM Inference OptimizationDynamic Sparse Attention

  8. SparLeak: Privacy Leakage from Sparse Attention in LLM Inference on Shared GPUs

    Sep 30, 2026Fahao Chen, Linkang Du, Jinhao Zhou +2Dynamic Sparse AttentionLLM Inference Optimization

    Shandong University China · Xi’an Jiaotong University China · Waseda University Japan

  9. Code to Control: Synthesizing Parameterized Reactive Controllers

    Sep 30, 2026Zergham Ahmed, Joshua B. Tenenbaum, Chris Bates +1Llm-Driven Code SynthesisReactive

    Harvard University · Massachusetts Institute of Technology · Florida Institute for Human and Machine Cognition

  10. FlexRouter: Learning Complementary Model Sets for Flexible LLM Routing

    Sep 29, 2026Wang Wei, Harry Yang, Tiankai Yang +6Large Language Model RoutingLLM Inference Optimization

    Virginia Tech · Independent Researcher · University of Southern California +2

  11. LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

    Sep 29, 2026Yi Pan, Haocheng Xi, Kan Zhu +10Large Language Model QuantizationKimi Delta Attention

    UC Berkeley · University of Washington · MIT +2

  12. You Cannot Pick a Provider From the Price List: Market-Aware Routing for Open-Weight LLM Inference

    Sep 29, 2026Liang He, Jingbo Wen, Yixiong Chen +4Large Language Model RoutingLLM Inference Optimization

    University of Sydney · Johns Hopkins University · Stanford University +1

  13. Pretrain Once, Route Anywhere: Towards a Foundation Model for LLM Routing

    Sep 29, 2026Guannan Lai, Han-Jia YeLarge Language Model RoutingLLM Inference Optimization

    School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University

  14. IronLLM: Forging Compact Edge-Native Language Models for Real-Time Embodied Intelligence

    Sep 29, 2026Changdi Yang, Fengquan Jiao, Haochih Lin +14LLM Inference OptimizationLanguage Modeling

    Robotics Foundation Model Team, Xpeng Inc.

  15. Bits Under ZK-LLM: Evaluating Zero-Knowledge-Friendly Quantization for Verifiable Private LLM Inference

    Sep 29, 2026Taeung Yoon, Yupeng Zhang, Xiaojing LiaoLarge Language Model QuantizationLLM Inference Optimization

    Siebel School of Computing and Data Science University of Illinois Urbana-Champaign

  16. From Search to Research: Exploring Search Scaling in Autonomous Quantitative Factor Mining

    Sep 28, 2026Kangcheng Deng, Hui Cai, Jiacheng Lu +6Research AutomationSequential Scaling

    StepFun · Shanghai Jiao Tong University · University of California, Los Angeles +1

  17. Beneath the Tokens: A Performance Engineering Study of Multi-Token Prediction in GPU-Accelerated LLM Inference

    Sep 28, 2026Suwesh Prasad SahMulti-Token PredictionLLM Inference Optimization

  18. PROACT-Agent: Progressive Runtime Oversight and Active Circuit-breaking for Real-Time Safety

    Sep 28, 2026Ding Jia, Wei Liu, Xianglong Du +5Large Language Model SafetySafety-Critical Scenarios

    State Key Laboratory of Intelligent Vehicle Safety Technology, Changan Automobile · Independent Researcher

  19. Spexis: Speculative Lookahead Scheduling for LLM Inference

    Sep 28, 2026Hyungyu Jung, Jaehyeok Yu, Hoonseo Choi +3LLM Inference OptimizationCpu-Gpu Hybrid Designs

    Seoul National University · Hanyang University

  20. MiCo: Mutual Information Coverage Optimization through Semantic Erasure Modeling for Efficient MLLM Inference

    Sep 28, 2026Tinghao Wang, Yichen Guo, Qizhe Zhang +11Long Visual-Token SequencesMultimodal Understanding

    State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · University of Electronic Science and Technology of China · Nanyang Technological University +2

  21. QuantaSpike: Short-Window Spike-Driven Quantization for Large Language Models

    Sep 28, 2026Bang Hu, Guowei Zhu, Changze Lv +4Large Language Model QuantizationLLM Inference Optimization

    School of Computer Science, Fudan University, Shanghai, China

  22. SlimWise: Decoupling Expert Pruning Across Prefill and Decode for Efficient MoE Serving

    Sep 28, 2026Gunho Park, Kyoungho Jeun, Juntaek Oh +3Mixture-Of-ExpertsLLM Inference Optimization

    a2sys · KAIST

  23. Greenpixie's AI Token Methodology: Assessing the Energy, Water and CO2-eq Impact of AI Tokens for Open and Closed Weight Models

    Sep 27, 2026Joshua Horswill, Ross Hunter, Matt Clifford +1LLM Inference Optimization

    Greenpixie Ltd., London, UK.

  24. OLED-MoE: Accelerating MoE-Based dLLM Inference via Inter-Iteration Locality-Aware Expert Offloading

    Sep 27, 2026Jingyuan Xiao, Jiayue Wang, Yitao Hu +7Mixture-Of-Expert InferenceOffloading

    Tianjin University, Tianjin, China · Tianjin University Tianjin, China

  25. Canopy: Exploiting Piecewise Smooth Tree Priors for Multi-Fidelity Bandits

    Sep 24, 2026Michael Jerge, Suman JanaTree SearchMulti-Fidelity

    Amazon · Columbia University

  26. MILO: Efficient Many-shot In-Context Learning with Block-wise Low-rank Compression

    Sep 24, 2026Youpeng Zhao, Tian Tan, Liqian Peng +2Large Language Model CompressionKey-Value Cache Compression

    Google · University of Central Florida

  27. KITE: KV-Invariant Transformer Expansion for Efficient Agentic LLM Scaling

    Sep 23, 2026Zhiheng Hu, Yixun Wei, Jian Zhou +8LLM Inference OptimizationSequential Scaling

    StepFun

  28. COMED: The Missing Middle Between Routing and Collaboration in Multi-LLM Inference

    Sep 22, 2026Norah Alballa, Wenxuan Zhang, Salma Kharrat +4Multi-Agent Large Language Model SystemsHeterogeneous Modeling Tools

    KAUST · LUMS

  29. Flash-dLLM: IO-Aware KV Caching and Parallel Decoding for Fast, Memory-Efficient Diffusion LLMs

    Sep 22, 2026Quan Nguyen-Tri, Mukul Ranjan, Zhiqiang ShenLLM Inference OptimizationDiffusion Language Models

    VILA Lab, MBZUAI Abu Dhabi, UAE

  30. Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference

    Sep 22, 2026Gaoyuan Du, Anam Nawaz Khan, Rex Zhou +4Constrained DecodingLLM Inference Optimization

    University of Tennessee, Knoxville · University of Chicago · Amazon +1

  31. CompKV: Compensation-Aware KV Selection for Long-Context LLM Inference

    Sep 22, 2026Zhen Huang, Ruizhe Yao, Danyi Liu +8Dynamic Sparse AttentionLLM Inference Optimization

    Tsinghua University · Harbin Institute of Technology, Shenzhen · Northeastern University

  32. Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures

    Sep 22, 2026Liam Cooper, Shinnung Jeong, Hyeran Jeon +2LLM Inference OptimizationFloating-Point

    Georgia Institute of Technology · University of California, Merced

  33. Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference

    Sep 22, 2026Md Mostafizer Rahman, Md Faizul Ibne Amin, Md Shahajada Mia +2LLM Inference OptimizationLarge Language Model Memory

    Lucy Family Institute for Data & Society, University of Notre Dame, IN, USA · The University of Aizu, Aizuwakamatsu, Japan · Department of Applied and Computational Mathematics and Statistics, University of Notre Dame, IN, USA

  34. LoRA-generating hypernetworks for efficient on-device LLM generative personalization

    Sep 21, 2026Sean Augenstein, Li Ding, Jihwan Lee +2Large Language Model PersonalizationPersonalization

    Google · Formerly at Google, work done while at Google

  35. Adapting Tree-Structured Speculative Decoding to DeepSeek-V4 for Efficient Inference

    Sep 21, 2026Changxu Liu, Zhaogeng LiSpeculative DecodingLLM Inference Optimization

    Baige AI Team, Baidu Inc. · Fudan University

  36. ARM: Attention with Routed-Memory for Learnable Sparse Control

    Sep 21, 2026Qiuhao Zeng, Jerry Huang, Peng Lu +7Efficient Long-Context InferencePeak Memory

    University of Toronto, Canada · Vector Institute, Canada · RIKEN AIP, Japan +3

  37. From Tables to Quantified Statements: Evaluating LLM Inference Generation through Executable Verification

    Sep 21, 2026Mai Mohamed Eida, Gunjan Anand, Ayush Singh +1Natural Language InferenceLLM Inference Optimization

    University of Illinois Urbana Champaign

  38. TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Caching

    Sep 18, 2026Zhihao Shu, Md Musfiqur Rahman Sanim, Jie Hu +4Kv-Cache ManagementDepthweave-Kv

    University of Georgia Athens, GA, USA · Peking University Beijing, China · Western Digital Research San Jose, CA, USA

  39. From "Who Is This User?" to "What Does This Purchase Mean?": A Deployed Pipeline for Semantic User Profiling at Bank Scale

    Sep 17, 2026Ryota Mitsuhashi, Tetsuro Morimura, Hirotake ItoProfileLarge Language Model Pipelines

    CyberAgent, Tokyo, Japan

  40. Replan, Repair, or Edit? A Unified Empirical Evaluation of Travel Agents for Itinerary Revision under Resource Disruptions

    Sep 17, 2026Xiaofei Yuan, Yan Zhang, Shaobo Qiao +7JourneyIterative Re-Planning

    Euler AI · University of New South Wales · University of Sydney +1

  41. Semantic Layer Induction from Raw Telemetry via Hierarchical LLM and RAG Abstraction

    Sep 17, 2026Yuanzhe Jia, Ali AnaissiMultivariate Telemetry DataInsights

    University of Sydney, Australia · University of Technology Sydney, Australia

  42. A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Quality

    Sep 16, 2026Jerry KaplanLLM Inference OptimizationQuantizer

    Adjunct Lecturer, Computer Science and Masters in International Policy, Stanford University

  43. ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference

    Sep 16, 2026Amir Ziashahabi, Hossein Entezari Zarch, Lei Gao +2Speculative DecodingLLM Inference Optimization

    University of Southern California, Los Angeles, CA, USA

  44. Long-Context Demonstration Selection Using State Space Models

    Sep 15, 2026Ziniu Zhang, Zhenshuo Zhang, Ruoxuan Xiong +2Efficient Long-Context InferenceIn-Context Learning

    Northeastern University · Emory University

  45. The Inference Engineering Pareto Atlas: Which Optimizations Dominate the Cost, Quality, and Latency Frontier?

    Sep 15, 2026Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri +3LLM Inference OptimizationMatched Fp16 Intermediate

    Vizuara

  46. ECHO: Early-layer Collaborative Hierarchical Orchestration with Bonus Logits in Speculative Decoding

    Sep 15, 2026Ziyang Ma, Zihong Zhang, Zuchao Li +4Speculative DecodingLLM Inference Optimization

    School of Computer Science, Wuhan University, Wuhan, China · School of Artificial Intelligence, Wuhan University, Wuhan, China · 3Xiaomi, Beijing, China

  47. LLM Inference in a Flash!

    Sep 14, 2026Sebastian Zhao, Minseo Kim, Coleman Hooper +5LLM Inference OptimizationInference-Time

    UC Berkeley · ICSI · LBNL

  48. Evaluating Losslessness in Speculative Decoding Under Finite-Precision Inference

    Sep 14, 2026Ilya Koziev, Leonid Sinev, Ivan OseledetsSpeculative DecodingFull-Precision Performance

  49. Dynamic Semantic Compression for Efficient Latent-Space Inference in Large Language Models

    Sep 14, 2026Peipei Li, Dongsen Zhang, Yuchen Liu +1LLM Inference OptimizationTask-Aware Compression

    Beijing University of Posts and Telecommunications, Beijing, China

  50. AdaVSkip: Adaptive Visual Token Skipping Across Layers For Efficient MLLMs Inference

    Sep 14, 2026Yuyao Sun, Tao Deng, Shuang Li +1Long Visual-Token SequencesMultimodal Large Language Models

    Beihang University

  51. SpliTEE: Improving LLM Inference on Trusted Hardware with Differentially Private GPU Outsourcing

    Sep 14, 2026Shashie Dilhara Batan Arachchige, Robin Carpentier, Hassan Jameel Asghar +1LLM Inference OptimizationDiffie-Hellman

    School of Computing, Macquarie University

  52. Shared KV Caching for Replicated 27B Inference: Correctness Failures and Performance Boundaries

    Sep 14, 2026Frank LiKv-Cache ManagementCpu-Gpu Hybrid Designs

    UNSW Sydney

  53. The Battery Price of edge AI: A study of the Environmental Impact of LLM Inference on Mobile Devices

    Sep 14, 2026Édouard Guégain, Tristan CoignionLLM Inference OptimizationDevice

    DRI, GREENSPECTOR · Univ. Bordeaux, CNRS, Bordeaux INP, LaBRI, UMR 5800, Talence, France

  54. Dissecting GPU Utilization for LLM Inference on Nvidia Hopper

    Sep 14, 2026Mohammad Siavashi, Gerald Q. Maguire Jr., Dejan Kostic +1Graphics Processing Unit ResourcesLLM Inference Optimization

    KTH Royal Institute of Technology, Stockholm, Sweden

  55. RoofLang: Enabling AI-Driven Architecting of LLM Inference Systems

    Sep 14, 2026Ziyue Yang, Yuting Jiang, Lei Qu +1LLM Inference OptimizationDeepseek

    Shanghai Xingyunzhili Artificial Intelligence Institute, Shanghai, China · Microsoft Research, Redmond, WA, USA

  56. Self-Orchestrating Language Models: Leveraging Semantic Dependence for Efficient Inference

    Sep 13, 2026Tian JinLLM Inference OptimizationAutoregressive Language Models

    Department of Electrical Engineering and Computer Science, MASSACHUSETTS INSTITUTE OF TECHNOLOGY

  57. SpecGuard: Inference-Time Backdoor Detection For Free

    Sep 11, 2026Rui Wen, Ahmed Salem, Andrew Paverd +2Clean Label Backdoor AttackInference-Time Defense

    Institute of Science Tokyo · Microsoft Security Response Center · Microsoft Azure +1

  58. EMMI: Edge Multi-Modal Intelligence for Communication-Efficient MLLM Inference via Fused Representation Compression

    Sep 11, 2026Motahare Mounesan, Irfan KhanEdge DevicesMultimodal Large Language Models

    Texas A&M University

  59. Maverick: Private and Verifiable LLM Inference Made Practical via Matrix-Vector Multiplication Delegation

    Sep 9, 2026Ben Merbaum, Mohammad Amin Raeisi, Wenhao Wang +3LLM Inference OptimizationMatrix Multiplication

    Yale University, IC3

  60. Forward-Free LLM Depth Pruning via Weight Redundancy

    Sep 9, 2026Vincent-Daniel Yun, Woosang LimUnstructured PruningLarge Language Model Compression

    University of Southern California, United States · 2Seoul National University, Republic of Korea