ICaRus: Identical Cache Reuse for Efficient Multi Model Inference
Authors: Sunghyeon Woo, Jaeeun Kil, Hoseung Kim, Minsub Kim, Joonghoon Kim, Ahreum Seo, Sungjae Lee, Minjung Jo, Jiwon Ryu, Baeseong Park, Se Jung Kwon, Dongsoo Lee
Multi model inference has recently emerged as a prominent paradigm, particularly in the development of agentic AI systems. However, in such scenarios, each model must maintain its own Key-Value (KV) cache for the identical prompt, leading to substantial memory consumption. This explosive growth of KV caches forces LLM serving systems to evict previously stored caches, which in turn introduces significant recomputation overhead whenever the evicted caches are required again. Moreover, prefix caching is inherently infeasible across different models, forcing each model to recompute KV cache for the identical prompt, which leads to significant overhead. To alleviate these issues, we propose Identical Cache Reuse (ICaRus), a novel architecture that allows multiple models to share identical KV caches across all layers. ICaRus is based on the key observation that a decoder-only Transformer can be conceptually decomposed into a logical encoder, which generates KV caches, and a logical decoder, which predicts output tokens from the KV caches. ICaRus fine-tunes only the logical decoder while freezing the logical encoder, enabling multiple models to share an identical KV cache. This eliminates cache memory explosion and unexpected evictions while also allowing cross-model reuse of KV caches for new input tokens, thereby removing redundant recomputation in multi model inference achieving both efficiency and scalability. Moreover, by incorporating lightweight adapters such as LoRA, ICaRus parallelizes KV cache generation and next-token prediction during decoding. ICaRus achieves comparable accuracy to task-specific fine-tuned model across a diverse set of tasks, while allowing multiple specialized models to fully share KV caches. ICaRus achieves up to 11.1x lower P95 latency and 3.8x higher throughput in multi agent workflow with 8 different models, compared to conventional multi model system.
Figures & tables
Figure 1: Comparison of KV cache management strategies and effectiveness in multi model scenarios between conventional approaches and ICaRus.
Figure 2: Training loss curves of conventional fine-tuning and ICaRus, both applied with LoRA on LLaMA-3.1-8B, trained on the MetaMathQA-40k and Evol-Instruct-Code-80k dataset.
Figure 3: Overview of the ICaRus architecture. The base model, a pretrained decoder-only Transformer, serves as the logical encoder, while the adapter-tuned model (consisting of the base model and a tunable adapter) serves as the logical decoder. The blue and orange lines indicate computations performed by the base model and the adapter-tuned model, respectively. The purple square denotes that the same base model generates the KV cache during both the prefill and decoding phases. In ICaRus architecture, KV caches can be reused regardless of the task, since the KV cache is always generated by the same base model (i.e., the logical encoder).
Scenario
Method
Memory
Latency
Total
Prefill
Decode (per token)
Memory Access
Compute
Single Model
—
O(M+Lt)
O(MLt+Lt2)
O(M+Lt)
O(M+Lt)
Multi Model
BaseLine
O(M+NLt)
O(N(MLt+Lt2))
O(M+Lt)
O(M+Lt)
ICaRus
O(M+Lt)
O(MLt+Lt2)
O(M+Lt)
O(2M+2Lt)
Table 1: Complexity comparisons between single model and multi model scenarios.
Model
Method
KV Sharing
Math
Coding
Knowledge
GSM8K
GSM+
HEval
HEval+
GPQA
LLaMA-3.1-8B
Base Model
.
25.9
18.0
36.6
29.9
16.7
Multi Model
X
69.7
48.5
48.2
41.5
27.3
ICaRus (Ours)
O
67.9
45.8
48.2
43.9
28.8
Qwen3-8B-Base
Base Model
.
11.8
12.5
68.3
61.6
24.2
Multi Model
X
85.4
66.1
81.7
75.6
34.3
Table 2: Comparison of conventional methods and ICaRus on diverse datasets. Base Model denotes the pretrained decoder-only Transformer without fine-tuning. Multi Model consists of three independently fine tuned models: one on MetaMathQA-40K, one on Evol-Instruct-Code, and one on Oasst1. ICaRus uses the same three specializations, but trains only task-specific logical decoders on a shared logical encoder, enabling KV cache sharing across models.
Model
Qwen3-1.7B-Base
Qwen3-8B-Base
Qwen3-14B-Base
Method
Baseline
ICaRus
Baseline
ICaRus
Baseline
ICaRus
GSM8K
73.2
74.0
85.4
87.3
85.6
88.8
GSM+
53.7
54.1
66.1
67.5
66.7
68.8
Table 3: Comparison of conventional fine-tuning and ICaRus across different model sizes (Qwen3-1.7B/8B/14B-Base) trained on the MetaMathQA-40K dataset.
# Model
Method
KV Sharing
Math
Coding
Knowledge
Avg.
GSM8K
GSM-Plus
HEval
HEval+
GPQA
1
Base Model
.
25.9
18.0
36.6
29.9
16.7
25.4
Math Model
.
69.7
48.5
42.7
36.6
20.7
43.6
Coding Model
.
22.8
17.5
48.2
41.5
21.7
30.3
IF Model
.
24.5
16.5
44.5
39.0
27.2
30.3
3
Multi Model
X
69.7
48.5
48.2
41.5
27.2
47.0
Table 4: Comparison of conventional methods and ICaRus in multi-model inference scenarios. Base Model denotes the LLaMA-3.1-8B-Base model without fine-tuning, while Math, Coding, and IF denote models fine-tuned on MetaMathQA-40K, Evol-Instruct-Code, and OASST1, respectively. Multi Model and ICaRus both consist of these three task-specific models; in ICaRus, however, only the logical decoders are fine-tuned while the logical encoder is shared across models.
Figure 4: P95 latency and throughput of ICaRus compared with multiple task-specific agents fine-tuned from the LLaMA-3.1-8B base model under the ReAct pattern. Here, N denotes the number of LoRA modules, which are integrated into multi model system built using either the conventional approach or ICaRus.
Figure 5: Comparison of P95 latency and maximum throughput across QPS for LLaMA3.1-8B and Qwen-3-14B Base under ReAct and Reflexion patterns.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Inference workflow of the logical encoder-decoder.
Figure 7: Training loss curves of conventional fine-tuning and ICaRus, both applied with LoRA on Qwen-3-32B, trained on the ToolAce dataset.
Model
Method
BFCL Non-live (AST)
Simple Python
Simple Java
Simple JavaScript
Qwen3-32B
Baseline
96.5
62.0
74.0
ICaRus (Ours)
94.5
63.0
76.0
Appendix
Table 5: Comparison of conventional fine-tuning and ICaRus when training Qwen3-32B on the ToolAce dataset.
Figure 8: P95 latency and throughput of ICaRus compared with multiple task-specific agents fine-tuned from the LLaMA-3.1-8B base model under the ReAct workflow with swap-based KV cache management. Here, N denotes the number of LoRA modules, which are integrated into multi model system built using either the conventional approach or ICaRus.
Figure 9: P95 latency and throughput of ICaRus compared with multiple task-specific agents fine-tuned from the LLaMA-3.1-8B base model under the ReAct workflow where the agent invocation pattern is random and skewed. Here, N denotes the number of LoRA modules, which are integrated into multi model system built using either the conventional approach or ICaRus.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
Multi-LoRA agent systems enable efficient role specialization by sharing a common backbone model. However, each agent repeatedly processes the growing shared trajectory and constructs its own KV cache, introducing substantial memory and computation redundancy in long-horizon tasks. Existing KV cache sharing methods reduce this repeated prefill, but they either require additional training or architectural constraints or retain substantial model computation. Moreover, direct cache reuse causes the current agent to rely on cache states generated by the previous agent's adapter, weakening the role-specific behavior encoded by its own LoRA. We present PReCache, a training-free KV cache sharing framework with two designs, namely PreLRShared and ReBaseShared, that share the base cache computed using the pretrained weights and precompute a compact agent-specific low-rank (LR) cache. To remove repeated prefill, PreLRShared precomputes each agent's LR cache when the shared context is first processed, allowing the current agent to use its own LR cache without reprocessing context processed by previous agents. To improve sharing accuracy, ReBaseShared reconstructs the shared base cache from adapter-free hidden states, reducing the remaining error caused by the previous agent's adapted representation. To minimize its reconstruction cost, we propose two inference schemes tailored to single-stream inference and concurrent serving, performing the same reconstruction after each agent's turn or alongside its execution, respectively. Across multiple models and agent benchmarks, PreLRShared achieves up to a 3.1x TTFT speedup and a 2.3x improvement in per-request throughput over inference without KV cache sharing. ReBaseShared best preserves accuracy overall among the evaluated cache-sharing methods, with an average drop of only 1.1 points relative to inference without cache sharing.
Hyesung Jeon, Hyeongju Ha, Jae-Joon Kim
Department of Electrical and Computer Engineering Seoul National University
Non-prefix KV cache reuse is a promising approach for improving inference efficiency in multi-agent systems as agents share the same context blocks. However, when agents have conflicting tasks, directly reusing non-prefix KV caches across agents can severely degrade model performance, causing up to a 90% drop. Existing remedies, including selective recomputation and offset-based calibration, recover a limited fraction of this loss. We argue that the problem lies not only in contaminated KV representations but also in the agent's difficulty interpreting reused context under its own instructions. To this end, we introduce LatentRemedy, which uses a few task-conditioned latent tokens to help the target agent leverage reused context without recomputing the shared block. Specifically, we use an MLP to map the last hidden state of the target prefix into a small number of latent tokens, which are placed after the reused KV block to interact with the shared context. To guide subsequent predictions toward the target task, we train the MLP through full-prefill distillation. Experiments show that LatentRemedy consistently outperforms existing reuse baselines on conflicting tasks while maintaining TTFT close to direct reuse as cache size grows. Code and data are available at https://github.com/chenzhwsysu57/kvreuse.
Zhiwei Chen, Zhongtao Rao, Waikit Xiu +4
The Hong Kong University of Science and Technology (Guangzhou) · Microsoft · The University of Hong Kong
The increasing complexity of AI tasks has shifted the paradigm from monolithic models toward multi-agent large language model (LLM) systems. However, these collaborative architectures introduce a critical bottleneck: redundant prefill computation for shared content generated by previous agents, which significantly increases KV cache memory usage and time-to-first-token (TTFT). While various KV cache methods have been proposed to mitigate prefill redundancy, they either fail to maintain accuracy on agent-generated outputs or exhibit low reuse rates due to rigid constraints. We present RelayCaching, a training-free inference method that directly reuses decoding phase KV caches from previous agents in subsequent prefill phases. Our key insight is that KV caches for identical content are highly consistent across phases, while prefix-induced deviations are sparse and localized within a limited range of layers and token positions. By selectively recomputing KV caches at these positions, RelayCaching preserves model accuracy with minimal overhead, yielding a superior accuracy-efficiency trade-off over existing methods. Experiments on diverse collaborative LLM tasks spanning mathematical reasoning, general knowledge, and code generation demonstrate that RelayCaching achieves over 80% KV cache reuse, reduces TTFT by up to 4.7× compared to the standard pipeline, all with negligible accuracy degradation.
Yingsheng Geng, Yuchong Gao, Weihong Wu +2
Beijing University of Posts and Telecommunications, Beijing, China · Tsinghua University, Beijing, China · University of Electronic Science and Technology of China, Chengdu, China +1