cs.DCNov 4, 2026

Chiron: Cache Management Empowered by LLM Intelligence for AI Workloads

Authors: Bohan Zhang, Renshu Ge, Tianze Wang, Chen Chen

Organizations: Shanghai Jiao Tong University

Abstract

AI clusters keep data in remote object stores and rely on local caches to stop GPUs from waiting for data. A cache performs well only when its prefetching and eviction policies match the access pattern, yet these patterns differ across AI workloads and change between phases of one workflow. Existing methods either need code changes, which compromises user transparency, or use low-end heuristics, which cannot follow such changes. Large language models (LLMs) can infer access behavior without user hints, but direct LLM control of cache actions suffers from long reasoning delay, high token cost, and poor decision quality. We present Chiron, a cache system based on advisory switching control. An LLM asynchronously diagnoses the access pattern and recommends a policy. A deterministic harness gates this advice, executes expert-designed policies locally, and returns cache feedback for review. We implement Chiron atop JuiceFS and evaluate it on nine AI workloads. It reduces job completion time by 14.3-41.2% over the best baseline on popular AI workloads.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. WAKMan: Workload Aware KVCache Manager

    Oct 26, 2026Suhas Kumar Morisetty, Vaibhavi Lokegaonkar, Sachin Mathew Varghese +1KV-Cache ManagementKV-Cache Eviction

  2. CACHEFORGE: LLM-Guided End-to-End Generative Cache Replacement Policy for Performance and Hardware Efficiency

    Oct 6, 2026Kaushal Mhapsekar, Bita Aslrousta, Brijesh Kumar Bhayana +5Large Language Model-Guided OptimizationLLM-Based Program Synthesis

  3. Recency/Frequency Adaptive KV Caching for Large Language Model Serving

    Jun 19, 2026Yang Shen, Meghana Madhyastha, Robert Underwood +2LLM ServingKV Caching