LLM Inference

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

30 papers in the last four weeks, up 233% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 187

All topics

Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching
Jul 17, 2026Yan SongLLM Serving LLM Inference
PayPal Inc.
The Cost and Network Limits of Space-Based AI Compute
Jul 15, 2026Kees van BerkelLLM Inference Cost-Aware Inference
Technical University Eindhoven, Dept. of Math. & Computer Science P.O. Box 513, 5600 MB Eindhoven, The Netherlands
Automatic Model-Hardware Co-Adaptation for Heterogeneous AI Accelerators
Jul 14, 2026Tian Chen, Mingheng Mi, Pu Wang +2AI Accelerator Inference LLM Inference
Pazhou Laboratory (Huangpu) · Unaffiliated
Efficient and Privacy Aware Edge Cloud Collaborative Inference for Large Language Models
Jul 14, 2026Yi Li, Chen Li, Jiexiong LiuOn-Device Language Model Inference Privacy-Preserving Language Models
KunlunMeta
FastTPS: An Optimized Method for LLM Token Phase for AI accelerators
Jul 13, 2026Wenzong Yang, Danyang Zhang, Kun Cao +17AI Accelerator Inference LLM Inference
Advanced Micro Devices, Inc., Santa Clara, California, USA.
Amplitude-Only FFN Intervention for Tool-Structured LLM Inference Method: Gated Evaluation Protocol, and Cross-Model Empirical Results
Jul 13, 2026Sheng Xu, Junhua Wang, Boyuan Huang +3LLM Inference LLM Tool Use
Alibaba Cloud
Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurations
Jul 10, 2026Nada Zine, Tristan Coignion, Vincenzo Stoico +3LLM Inference Efficiency LLM Evaluation
Univ. Lille, CNRS, Inria · Univ. Bordeaux, CNRS, LaBRI · Vrije Universiteit Amsterdam +1
Gauge dependence and structured-output corruption in sign-branched repetition penalties: measurements across models, inference stacks, and alternative repetition controls
Jul 9, 2026Peter HollowsLanguage Model Decoding LLM Inference
When Words Predict Workload
Jul 6, 2026Anubhab BanerjeeOn-Device Language Model Inference LLM Inference
Nokia Germany
Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference
Jul 6, 2026Guanyu Cai, Ruiming Tian, Lang Yang +4LLM Inference Efficiency AI Accelerator Inference
Tsinghua University · Beijing Jiaotong University
Heaviside Continuity of Rolling Coefficients for Eliminating Epistemic Entropy in Large Language Models
Jul 6, 2026MY Pitsane, Hope MogaleLLM Answer Verification LLM Inference
North-West University, RSA · Mankind Research Labs, Sandton · University of Pretoria, RSA
Knowing When to Stop: Predicting Execution-Consistency Convergence in Text-to-SQL
Jul 4, 2026Yaron Anavi, Mor Aisenberg, Nadav Nesher +2LLM Inference Efficiency LLM Inference
GIGASPACES
WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs
Jul 2, 2026Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-MartínezLLM Inference Efficiency Energy-Efficient ML
Universidad Politécnica de Madrid, C. de los Ciruelos, 28660 Boadilla del Monte, Madrid, Spain · Zurich University of Applied Sciences, Gertrudstrasse 15, 8400 Winterthur, Switzerland
Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference
Jul 2, 2026Wenchen Han, Gingfung Matthew Yeung, Marco Barletta +3LLM Inference KV Caching
University College London · Huawei
Dynamic Bidirectional Pattern Memory: A Production-Scale Empirical Characterisation of Inference-Time Gating in Clinical NLP
Jul 1, 2026Ali H. Lazem, William TeahanLLM Inference Clinical Information Extraction
School of Computer Science and Engineering, Bangor University, Bangor, Gwynedd, LL57 2DG, United Kingdom · University of Thi-Qar, Nasiriyah, 64001, Iraq
KernelSight-LM: A Kernel-Level LLM Inference Simulator
Jun 26, 2026Xiteng Yao, Taeho Kim, Hengzhi Pei +7LLM Inference LLM Inference Serving
Amazon Web Services Santa Clara, CA, USA · Boston University Boston, MA, USA
End-to-End Dynamic Sparsity for Resource-Adaptive LLM Inference
Jun 26, 2026Yuhang Chen, Jinhao Duan, Ruichen Zhang +11LLM Inference Structured Sparsity
University of North Carolina at Chapel Hill · Meta AI
Reasoning as Attractor Dynamics: Latent Memory Retrieval via Gibbs-Weighted Energy Minimization
Jun 23, 2026Kanishk AwadhiyaLLM Inference Representation Geometry in Language Models
Independent Researcher
The African Language Tax: Quantifying the Cost, Latency, and Context Penalty of Tokenizing African Languages in Frontier LLMs
Jun 23, 2026Olaoye Anthony SomideMultilingual Language Models Multilingual Language Model Evaluation
DataLens Africa Research · CipherSense AI Technologies Ltd.
Concordia: JIT-Compiled Persistent-Kernel Checkpointing for Fault-Tolerant LLM Inference
Jun 22, 2026Yuhang Gan, Yiwei Yang, Yuyi Li +6LLM Inference
University of California Santa Cruz USA · UC Santa Cruz USA · University of California, Davis USA +2
GRINQH: Graded Input-based Quantization Hierarchy for Efficient LLM Generation
Jun 22, 2026Jette Oberländer, Jan Finkbeiner, Catherine M. Schöfmann +1Mixed-Precision Quantization LLM Quantization
Fakultät für Informatik, RWTH Aachen, Aachen, 52074, Germany · Peter Grünberg Institut, Forschungszentrum Jülich GmbH, Jülich, 52425, Germany · Fakultät für Elektrotechnik und Informationstechnik, RWTH Aachen, Aachen, 52062, Germany
Abstract representational geometry supports inference in large language models
Jun 22, 2026Yunan Zeng, Yuwang WangLLM Inference LLM Interpretability
College of Future Information Technology, Fudan University, Shanghai, 200433, China · Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, 100084, China
Depth-Entropy Guided Sampling for Training-Free LLM Reasoning
Jun 19, 2026Zibin Meng, Peng Xie, Kani ChenLLM Inference Test-Time Scaling
1The Hong Kong University of Science and Technology
Demystifying Numerical Instability in LLM Inference: Achieving Reproducible Inference for Mission-Critical Tasks with HEAL
Jun 19, 2026Zhenting Zhu, Lucas Thai, Shan Yu +5Mixed-Precision Quantization LLM Inference
1UCLA · 2UC Berkeley · University of Chinese Academy of Sciences
The Token Tax of Epistemic Accuracy: Comparing RAG and Long-Context Architectures for Document-Grounded Generative AI Applications
Jun 18, 2026Austin Hamilton, Ryan Singh, Michael Wise +6LLM Grounding LLM Inference
Miami University · University at Buffalo
Structured Inference with Large Language Gibbs
Jun 17, 2026Sanghyeok Choi, Henry Gouk, Esmeralda S. WhitammerMarkov Chain Monte Carlo LLM Inference
University of Edinburgh, School of Informatics · 2CIFAR Fellow, Learning in Machines and Brains
How Inference Compute Shapes Frontier LLM Evaluation
Jun 16, 2026Jessica McFadyen, Ole Jorgensen, Harry Coppock +2LLM Evaluation Language Model Generation Evaluation
AI Security Institute, London, UK · University of Oxford, Oxford, UK · Harvard University, Cambridge, MA, USA
Entropy-Gated Latent Recursion
Jun 15, 2026Soham Bhattacharjee, Dushyant Singh Chauhan, Salem Lahlou +2LLM Inference Inference-Time Scaling
Department of Machine Learning Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, United Arab Emirates
Privacy from Symmetry: Orthogonally Equivariant Transformers for LLM Inference
Jun 15, 2026Alexander Yukhimchuk, Andrey Shulga, Mladen Kolar +1Privacy-Preserving Language Models LLM Inference
MBZUAI · University of Southern California
Communication-Efficient Verifiable Attention for LLM Inference
Jun 15, 2026Ziqun Chen, Ming Wu, Michael Heinrich +4LLM Inference Trusted Execution Environments
Nanyang Technological University · Singapore · Zero Gravity Labs +1

Jul 17, 2026, cs.LGJ/K move · Enter open · S save

Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching

Yan Song

PayPal Inc.

LLM Serving LLM Inference Cost-Aware Inference Prefix Caching Prompt Compression

Open paper AskPDF arXiv

Production LLM deployments combine two cost-reduction primitives: prompt caching (a discounted rate for re-used token prefixes) and prompt compression (fewer tokens sent). The compression literature has standardized on query-aware methods that produce a different compressed prefix per query, mechanically invalidating the prefix-strict cache on every call. We characterize this cost empirically on Anthropic's Sonnet 4.6 API and find caching is far from the rho=1.0 ideal the literature assumes: Sonnet's cache has a two-tier architecture with a sharp threshold near 3,500 tokens, below which the hit rate plateaus at rho~0.83 across 30-call sessions. Our cost model predicts, and experiments confirm, that under realistic rho, query-aware compression beats naive caching at high compression ratios (r>=6). We propose Cache-Aware Prompt Compression (CAPC), pairing query-agnostic compression with explicit cache_control plus a tier-preserving ratio bound that prevents over-compression from pushing the cached prefix into the hot tier. CAPC is the cheapest strategy in 16/16 configurations on LongBench-v2, with mean savings of 49% over cache-only, 64% over query-aware compression, and 90% over vanilla, at quality within 0.05 of the uncompressed baseline. We validate CAPC on three production workloads: an enterprise tool-using assistant with a 94k-token schema prefix (51.7% cost reduction at r=3); a graphify knowledge-graph RAG pipeline across two codebases (9.3x vs cache-all on FastAPI, 2.4x on httpx); and the public tau-bench retail benchmark (50 tasks), where CAPC is the cheapest of four strategies with reward exactly equal to vanilla (both 36/50, p=1.00) while query-aware compression is the most expensive at +40.1% over vanilla -- the first production confirmation of the crossover model's negative-ROI prediction on a public benchmark.

Momentum

Related topics

Latest papers 187