LLM-based multi-agent systems solve complex tasks through collaboration, but their growing, simultaneously resident key-value (KV) caches create a memory bottleneck. Applying single-model quantization policies independently ignores differences in agents' precision needs, potentially degrading collective reasoning. We introduce Coordinated Mixed-Precision KV Cache Quantization (CoMixKV), which combines online sensitivity estimation with joint agent-layer bit-width allocation under a shared memory budget. To our knowledge, CoMixKV is the first mixed-precision KV-cache quantization framework to jointly allocate precision across simultaneously resident agent caches under such a budget. The estimator derives sensitivity from a local attention-error bound using recent queries, keys, and values, without calibration data, trial quantization, or additional model passes. A shared controller combines these scores with remaining quantizable capacity and byte-level storage costs to minimize total estimated error. Allocations are refreshed as agents start or finish and new observations become available. Selected widths apply only to newly quantized blocks; recent states retain native precision and previously quantized blocks remain unchanged. Across four benchmarks and three LLM backbones at average 2-bit, 3-bit, and 4-bit settings, CoMixKV achieves strong accuracy against the evaluated quantization baselines. At the average 3-bit setting on Gemma-4-31B-it, it achieves 68.35% average accuracy, exceeding the strongest quantized baseline by 0.59%. On Qwen3-32B, it reduces peak KV-cache memory by 32.08% and achieves a 1.38x inference speedup over FullKV.
Figures & tables
Figure 1: Multi-agent inference as a spatial-temporal graph. Solid blue arrows denote spatial communication ( EqS ), while dashed orange arrows denote temporal information flow ( EqT ). Node subscripts m , r , p , and v denote the manager, Worker 1, Worker 2, and verifier, respectively.
Figure 2: Graph-conditioned cache residency and shared-budget quantization. Left: cache lifetimes across manager, worker, and verifier runs. Right: agent–layer width choices for resident private KV caches under a shared byte envelope.
Figure 3: Overview of CoMixKV . Observe and score: recent Q/K/V observations provide online sensitivity scores by agent and layer. Allocate: the controller assigns paired lower- and higher-bit layers with the target packed-code mean for each invocation under the shared byte envelope. Quantize new blocks: selected widths apply only to newly quantized blocks; sealed blocks remain unchanged, and the most recent R tokens retain native precision. Allocations are updated when the resident set changes or fresh observations become available.
Model
Method
Knowledge Search
Agentic Reasoning
Mathematical Reasoning
Scientific Reasoning
Average
Bamboogle
GAIA
AIME25
GPQA
Qwen3.8-27B
Vanilla
87.20
58.25
96.67
77.78
79.98
KIVI
90.40
58.25
90.00
81.82
80.12
TurboQuant
88.80
58.25
93.33
81.82
80.55
PM-KVQ
88.80
56.31
86.67
77.78
77.39
CoMixKV
91.20
57.28
93.33
81.82
80.91
Table 1: Accuracy (%) across four benchmarks and two model backbones at an average 3-bit KV-cache bit-width budget.
Figure 4: Accuracy and system efficiency on Qwen3-32B. (a) Average accuracy across four benchmarks under average 2-bit and 4-bit KV-cache bit-width budgets; the dashed line denotes the unquantized 16-bit Vanilla reference. (b) System efficiency on the controlled long-context agent workload, including batch wall time, physical KV-cache peak, and total GPU-memory peak. Lower is better for all three efficiency metrics.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Bits
Method
Knowledge Search
Agentic Reasoning
Mathematical Reasoning
Scientific Reasoning
Average
Bamboogle
GAIA
AIME25
GPQA
16
Vanilla
81.60
28.16
30.00
60.61
50.09
2
KIVI
71.20
26.21
36.67
48.48
45.64
TurboQuant
53.60
21.36
23.33
47.98
36.57
PM-KVQ
71.20
24.27
36.67
51.01
45.79
CoMixKV
69.60
25.24
40.00
48.48
45.83
Appendix
Table 2: Accuracy (%) of Qwen3-32B at average KV-cache bit-width budgets of 2 and 4 bits, with the unquantized 16-bit Vanilla reference.
Notation
Definition
q
Original task query in Eq. ( 1 ); not the numerical attention query in Eq. ( 10 ).
i∈{m,r,p,v}
Fixed role labels for manager, researcher, analyst, and verifier. The upright label r denotes the researcher, not allocation round r .
a
One invocation of role i , owning the history ha and cache KVa defined in the main text.
va(t)
Invocation-indexed refinement of vi(t) , distinguishing separate runs of the same role.
ha(t)
The local history ha before step t of invocation a .
ya(t)
Serialized model output at that step, including its action, refining yi(t) in Eq. ( 3 ).
Appendix
Table 3: Agent-side notation and its connection to the main-text formulation.
Figure 5: Invocation-level execution and cache branches. (a) The manager pauses during a managed-worker call while the worker extends its own local action–observation history; the returned textual report becomes the observation of the manager’s delegation action. (b) Reports are serialized and tokenized as part of the receiver’s prompt rather than transferred as KV tensors. Distinct context-dependent suffixes therefore form separate cache branches, although identical physical prefixes may be shared by the serving backend. (c) Recovery starts a fresh local history with reset=True ; any earlier evidence is supplied explicitly as text.
Role
Open-Web tasks
Closed-book tasks
Manager
Calls researcher or analyst
Calls analyst only
Researcher
Search, Visit, Python
Not active
Analyst
Python, Search, Visit
Python only
Verifier
Python and guarded Search
Python only
Appendix
Table 4: Role-specific actions in the multi-agent implementation. Every role also has final_answer . The verifier’s VerificationSearchTool is exposed under the name search , but it returns a fixed message and never accesses the network.
Figure 6: Task-conditioned teams and tool permissions. Open-Web tasks expose both researcher and analyst workers to the manager, whereas closed-book tasks expose only the analyst and disable Web tools. The verifier is a separate runtime stage, not a managed worker callable by the manager. Its guarded search action on open-Web tasks returns a fixed message without network access. Arrows show available task/report channels rather than a mandatory execution sequence, and every role also has final_answer .
Figure 7: Role-conditioned prompt construction. The system prompt combines the shared ReAct protocol, the role’s exposed action schemas, its responsibility, and the task-class profile. The query or delegated subtask is supplied separately as the run input, and each invocation retains its own local history. Roles use the same model backbone but differ in available actions, instructions, and visible histories.
Benchmarks
Agent organization and task instruction
Bamboogle, GAIA
Use the open-Web team. Delegate source discovery to the researcher and calculations or structured extraction to the analyst.
AIME25
Use the closed-book team. Delegate derivation or Python verification to the analyst, disable Web access, and return only the requested exact numeric or symbolic expression.
GPQA
Use the closed-book team. Delegate scientific reasoning to the analyst, disable Web access, and return only the requested option, term, or short answer.
Appendix
Table 5: Task profiles for the benchmark groups reported in the paper.
Message route
Payload visible to the receiver
Manager → worker
A delegated subtask and optional additional arguments.
Worker → manager
The worker’s final textual report, optionally followed by a textual run summary.
Runtime → verifier
Original task, manager proposal, and extracted worker report(s).
Verifier → recovery worker (via runtime)
One focused evidence gap, together with the original task and earlier candidate/evidence.
Recovery worker → verifier (via runtime)
Recovery summary and the relevant Search, Visit, or Python observations.
Appendix
Table 6: Textual payloads passed along the agent graph.
Figure 8: Bounded verification and targeted recovery. (a) Before initial verification, the runtime performs a forced analyst review if no worker was called; otherwise, an explicitly uncertain manager proposal can trigger manager-gap recovery. (b) An evidence request or unusable verification output triggers at most one targeted recovery in this branch, using a researcher for open-Web tasks or an analyst without Web access for closed-book tasks. A new verifier invocation checks the augmented evidence, with a final evidence-only adjudication if necessary and no further search branch. (c) Re-verification receives relevant tool observations together with the recovery summary, rather than treating repeated candidate claims as evidence.
The key-value (KV) cache is the dominant memory bottleneck of long-context large language model (LLM) inference, growing linearly with context length. We show that uniform KV quantization on a fractional-bit grid does not degrade gracefully: under a prespecified multi-seed statistical protocol, Llama-3.1-8B-Instruct with an affine quantizer is statistically indistinguishable from FP16 KV down to 2.322 code bits/value and collapses at 2.0 bits - a quality cliff in (2.0, 2.322] that reappears in generation-time quantization and multi-turn dialogue and transfers to Mistral-7B. The cliff reframes importance-aware mixed precision: above it, eight model-internal importance indicators are statistically interchangeable, so the benefit of mixing is grid interpolation, reaching average precisions uniform quantization cannot realize. SemKV preserves every token, ranks tokens by a model-internal score, and assigns two adjacent above-cliff precisions, achieving a measured 6.0x storage reduction with no statistically detectable quality difference from full KV (n=900, three seeds), and outperforming FP16 token pruning granted a 1.5x larger memory budget. Replacing the affine base with a distortion-optimized quantizer (TurboQuant-MSE) lowers the cliff in every protocol tested, raising the no-detectable-loss operating point to 7.9x. The recipe: measure the cliff for the target deployment setting, then interpolate above it.
Daeha Lee, Do-Hyung Kim, Jae-Hong Kim
Electronics and Telecommunications Research Institute (ETRI), Daejeon, Republic of Korea
We present PolyKV, a system in which multiple concurrent inference agents share a single, asymmetrically compressed KV cache pool. Rather than allocating a separate KV cache per agent -- the standard paradigm -- PolyKV writes a compressed cache once and injects it into N independent agent contexts via HuggingFace DynamicCache objects. Compression is asymmetric: Keys are quantized at int8 (q8_0) to preserve softmax stability, while Values are compressed using TurboQuant MSE -- a Fast Walsh-Hadamard Transform (FWHT) rotation followed by 3-bit Lloyd-Max quantization with centroids tuned to N(0,1). We evaluate across two model scales (SmolLM2-1.7B-Instruct and Llama-3-8B-Instruct), three context lengths (600-7,194 tokens), and up to 15 concurrent agents. PolyKV achieves a stable 2.91x compression ratio across all configurations. On Llama-3-8B with 15 agents sharing a 4K-token context, PolyKV reduces KV cache memory from 19.8 GB to 0.45 GB -- a 97.7% reduction -- while maintaining only +0.57% perplexity degradation and a mean BERTScore F1 of 0.928. PPL delta does not grow with agent count and improves as context length increases, inverting to -0.26% at 1,851 coherent tokens. To our knowledge, no prior work combines a single shared, lossy-compressed KV pool with multi-reader concurrent agent access.
The Key-Value (KV) cache constitutes a dominant source of memory consumption during large language model (LLM) inference, particularly in long-context and complex reasoning scenarios. Existing KV cache quantization methods have explored different compression granularities, including layer-wise precision adjustment and token-level information preservation. In practical deployment, however, the key challenge is not merely compression, but how to allocate limited quantization resources among heterogeneous precision preservation strategies under a fixed KV cache memory budget. We propose MultiAlloc, a profiling-guided precision allocation framework for KV cache quantization that adaptively distributes limited quantization resources between layer-wise precision enhancement and token-wise preservation. MultiAlloc employs lightweight offline profiling to characterize layer sensitivity and token importance, enabling adaptive precision allocation under a fixed memory constraint. Guided by the profiled memory--accuracy trade-off, it automatically derives an effective precision allocation strategy without online allocation search, and applies the resulting policy during inference. Extensive experiments on reasoning and long-context benchmarks demonstrate that MultiAlloc consistently improves over layer-wise, token-wise, and recent multi-granularity KV cache quantization methods under matched memory budgets across diverse models, including Mistral, Llama, DeepSeek-Qwen, and QwQ.
Yangchengyu Zhou, Zhiping Wu, Runchang Jiang +5
Nanjing University · The Hong Kong University of Science and Technology · Geely +1