Semantic Caching

Ask about this topic

Overview

Semantic caching studies when a system can reuse a prior result for a new input that is similar in meaning, reducing repeated model, retrieval, or database work. Methods include embedding-based matching, learned retrieval and eviction policies, structured query normalization, and caches that store intermediate evidence or semantic views. In retrieval-augmented generation and other LLM services, freshness checks, evidence grounding, and certified reuse can help ensure that a fast cache hit is still correct. The main tension is that similarity and high hit rates do not guarantee that answers are interchangeable, while stricter safety checks can reduce the speed and cost benefits.

Latest papers

CardsList

22 papers1 paper in the last four weeks, down 67% on the four weeks before. 0.0% of all new papers.

  1. Semantic Views: Certified Reuse of LLM Labels Across Semantic Queries

    Oct 8, 2026Oktie Hassanzadeh, Dharmashankar SubramanianSemantic CachingLLM Inference Acceleration

    IBM

  2. Semantics Delivery Network: Rethinking Web Retrieval Infrastructure for LLM Agents

    Sep 18, 2026Peichun Hua, Yunming XiaoSemantic CachingAgentic RAG

    The Chinese University of Hong Kong, Shenzhen, China

  3. Towards Anticipatory Databases Through Shared Data and Workload Semantics

    Sep 13, 2026Farzaneh Zirak, Kasper Overgaard Mortensen, Farhana Choudhury +1Cache ReplacementSemantic Caching

    University of Melbourne · Aarhus University

  4. PACE: Perceived-Latency-Aware Cascading Service Routing and Filler Control for QoE-Efficient Retrieval-Augmented Dialogue Serving

    Sep 9, 2026Lin Huang, Yujuan Tan, Weisheng Li +4Retrieval-Augmented GenerationInference-Time Optimization

    Chongqing University. No. 55, University Town South Road, Gaoxin District, Chongqing, 401331, China · Inspur Yunzhou Industrial Internet Co., Ltd. No.1036 Langchao Road, Lixia District, Jinan, Shandong, 250101, China. · Guoqi Zhimo (Chongqing) Technology Co., Ltd. 5th Floor, Building B15, Xiantao Data Valley, Yubei District, Chongqing, 401122, China. +4

  5. Which Eviction Policy Should an LLM Cache Use? A Systematic Study Across Workloads, Capacities, and Encoders

    Aug 20, 2026Yash Kulkarni, Shubham Harkare, Arvind Yogesh Suresh BabuSemantic CachingLLM Inference Efficiency

    University of Michigan Ann Arbor, Michigan, USA

  6. LipCache: A Local Inference Proxy with Certified Caching for Edge Image Classification Service

    Aug 13, 2026Zhengzhe Xiang, Yinlin Chen, Fuli Ying +3Efficient Neural Network InferenceEdge Inference

  7. LaCache: Robust Semantic Caching for LLM Serving

    Aug 3, 2026Jiacheng Liang, Yuhui Wang, Tanqiu Jiang +1LLM ServingLLM Security

    Stony Brook University

  8. From ambiguous utterances to governed reuse classes: canonicalization, quotient invariance, and conditional decidability

    Jul 11, 2026Cosimo Spera, Ray GarciaSemantic Caching

    Minerva CQ (Bourbaki Intelligent Systems, Inc.), Los Gatos, CA, USA

  9. Risk-Constrained Freshness-Aware Semantic Caching for Open-Web Retrieval-Augmented LLMs

    Jul 5, 2026Muhammad Mansoor, Tahir Ahmad, Yeo-Chan YoonLLM Inference EfficiencyRetrieval-Augmented Generation

    Jeju National University, Jeju, Republic of Korea

  10. When Classic Cache Policies Fail: Learning-Augmented Replacement for Semantic Retrieval Buffers

    Jul 1, 2026Yushi Sun, Bowen Cao, Wai LamLearning-Augmented AlgorithmsSemantic Caching

    LIGHTSPEED, Tencent · Shenzhen, China · The Chinese University of Hong Kong +1

  11. SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference

    Jun 30, 2026Amirhossein Abaskohi, Giuseppe Carenini, Peter West +1KV-Cache OffloadingKV Caching

    University of British Columbia · Microsoft Research

  12. Closing the Operational Gap in Semantic Caching

    Jun 18, 2026Aditeya Baral, Radoslav Ralev, Iliya Sotirov Zhechev +2LLM EvaluationCost-Aware Inference

    New York University · Redis

  13. Hallucination Mitigation with Agentic AI, Nested Learning, and AI Sustainability via Semantic Caching

    May 27, 2026Diego Gosmar, Deborah A. DahlAI Agent ReliabilityMulti-Agent LLM Systems

    Head of AI, Tesisquare · Member, Open Voice Interoperability Initiative · Linux Foundation AI & Data +4

  14. Grounded Cache Routing for Retrieval-Augmented Generation: When Is It Safe to Reuse an Answer?

    May 26, 2026Syed Huma ShahRetrieval-Augmented GenerationRAG Security

    Duke University · Syed Huma Shah$^*$

  15. MVR-cache: Optimizing Semantic Caching via Multi-Vector Retrieval and Learned Prompt Segmentation

    May 24, 2026Ali Noshad, Zishan Zheng, Yinjun WuLLM Inference EfficiencyRL for Combinatorial Optimization

    School of Computer Science, Peking University, Beijing, China · School of Information, Renmin University of China, Beijing, China

  16. CacheRAG: A Semantic Caching System for Retrieval-Augmented Generation in Knowledge Graph Question Answering

    Apr 28, 2026Yushi Sun, Lei ChenRetrieval-Augmented GenerationGraph Retrieval-Augmented Generation

    HKUST Hong Kong, China · HKUST(GZ) / HKUST Guangzhou / Hong Kong, China

  17. Continuous Semantic Caching for Low-Cost LLM Serving

    Apr 21, 2026Baran Atalar, Xutong Liu, Jinhang Zuo +3LLM ServingCost-Aware Inference

    Carnegie Mellon University Pittsburgh, Pennsylvania, USA · University of Washington Tacoma Tacoma, Washington, USA · City University of Hong Kong China +1

  18. Local-Splitter: A Measurement Study of Seven Tactics for Reducing Cloud LLM Token Usage on Coding-Agent Workloads

    Apr 14, 2026Justice Owusu Agyemang, Jerry John Kponyo, Elliot Amponsah +2LLM Agent Workflow OptimizationLLM Inference Efficiency

    Sperix Labs · VIA Cybersecurity Lab, KNUST · Quantum and Assistive Technologies Lab, KNUST

  19. RedParrot: Accelerating NL-to-DSL for Business Analytics via Query Semantic Caching

    Mar 7, 2026Tong Wang, Yongqin Xu, Jianfeng Zhang +6Enterprise AnalyticsLLM Inference Acceleration

    State Key Laboratory of Blockchain and Data Security, Zhejiang University · Xiaohongshu

  20. Semantics-Aware Caching for Concept Learning

    Mar 6, 2026Louis Mozart Kamdem Teyou, Caglar Demir, Axel-Cyrille Ngonga NgomoConcept LearningSemantic Caching

    Heinz Nixdorf Institute, Furstenalle 11, 33102 Paderborn, Germany · Paderborn University, Warburgerstr. 100, 33098 Paderborn, Germany

  21. VoiceAgentRAG: Solving the RAG Latency Bottleneck in Real-Time Voice Agents Using Dual-Agent Architectures

    Mar 2, 2026Jielin Qiu, Jianguo Zhang, Zixiang Chen +12Agentic RAGConversational Agents

    Salesforce AI Research

  22. From Similarity to Vulnerability: Key Collision Attack on LLM Semantic Caching

    Jan 30, 2026Zhixiang Zhang, Zesen Liu, Yuchong Xie +2Adversarial AttacksLLM Agent Security

    Department of Computer Science and Engineering, The Hong Kong University of Science and Technology · Fudan University