Coordinated Mixed-Precision KV Cache Quantization for Efficient LLM-based Multi-agent Systems
Organizations: Xiamen University · WeChat AI, Tencent Inc., China
Abstract
LLM-based multi-agent systems solve complex tasks through collaboration, but their growing, simultaneously resident key-value (KV) caches create a memory bottleneck. Applying single-model quantization policies independently ignores differences in agents' precision needs, potentially degrading collective reasoning. We introduce Coordinated Mixed-Precision KV Cache Quantization (CoMixKV), which combines online sensitivity estimation with joint agent-layer bit-width allocation under a shared memory budget. To our knowledge, CoMixKV is the first mixed-precision KV-cache quantization framework to jointly allocate precision across simultaneously resident agent caches under such a budget. The estimator derives sensitivity from a local attention-error bound using recent queries, keys, and values, without calibration data, trial quantization, or additional model passes. A shared controller combines these scores with remaining quantizable capacity and byte-level storage costs to minimize total estimated error. Allocations are refreshed as agents start or finish and new observations become available. Selected widths apply only to newly quantized blocks; recent states retain native precision and previously quantized blocks remain unchanged. Across four benchmarks and three LLM backbones at average 2-bit, 3-bit, and 4-bit settings, CoMixKV achieves strong accuracy against the evaluated quantization baselines. At the average 3-bit setting on Gemma-4-31B-it, it achieves 68.35% average accuracy, exceeding the strongest quantized baseline by 0.59%. On Qwen3-32B, it reduces peak KV-cache memory by 32.08% and achieves a 1.38x inference speedup over FullKV.
Figures & tables
| Model | Method | Knowledge Search | Agentic Reasoning | Mathematical Reasoning | Scientific Reasoning | Average |
|---|---|---|---|---|---|---|
| Bamboogle | GAIA | AIME25 | GPQA | |||
| Qwen3.8-27B | Vanilla | 87.20 | 58.25 | 96.67 | 77.78 | 79.98 |
| KIVI | 90.40 | 58.25 | 90.00 | 81.82 | 80.12 | |
| TurboQuant | 88.80 | 58.25 | 93.33 | 81.82 | 80.55 | |
| PM-KVQ | 88.80 | 56.31 | 86.67 | 77.78 | 77.39 | |
| CoMixKV | 91.20 | 57.28 | 93.33 | 81.82 | 80.91 |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Bits | Method | Knowledge Search | Agentic Reasoning | Mathematical Reasoning | Scientific Reasoning | Average |
|---|---|---|---|---|---|---|
| Bamboogle | GAIA | AIME25 | GPQA | |||
| 16 | Vanilla | 81.60 | 28.16 | 30.00 | 60.61 | 50.09 |
| 2 | KIVI | 71.20 | 26.21 | 36.67 | 48.48 | 45.64 |
| TurboQuant | 53.60 | 21.36 | 23.33 | 47.98 | 36.57 | |
| PM-KVQ | 71.20 | 24.27 | 36.67 | 51.01 | 45.79 | |
| CoMixKV | 69.60 | 25.24 | 40.00 | 48.48 | 45.83 |
| Notation | Definition |
|---|---|
| Original task query in Eq. ( 1 ); not the numerical attention query in Eq. ( 10 ). | |
| Fixed role labels for manager, researcher, analyst, and verifier. The upright label denotes the researcher, not allocation round . | |
| One invocation of role , owning the history and cache defined in the main text. | |
| Invocation-indexed refinement of , distinguishing separate runs of the same role. | |
| The local history before step of invocation . | |
| Serialized model output at that step, including its action, refining in Eq. ( 3 ). |
| Role | Open-Web tasks | Closed-book tasks |
|---|---|---|
| Manager | Calls researcher or analyst | Calls analyst only |
| Researcher | Search, Visit, Python | Not active |
| Analyst | Python, Search, Visit | Python only |
| Verifier | Python and guarded Search | Python only |
| Benchmarks | Agent organization and task instruction |
|---|---|
| Bamboogle, GAIA | Use the open-Web team. Delegate source discovery to the researcher and calculations or structured extraction to the analyst. |
| AIME25 | Use the closed-book team. Delegate derivation or Python verification to the analyst, disable Web access, and return only the requested exact numeric or symbolic expression. |
| GPQA | Use the closed-book team. Delegate scientific reasoning to the analyst, disable Web access, and return only the requested option, term, or short answer. |
| Message route | Payload visible to the receiver |
|---|---|
| Manager worker | A delegated subtask and optional additional arguments. |
| Worker manager | The worker’s final textual report, optionally followed by a textual run summary. |
| Runtime verifier | Original task, manager proposal, and extracted worker report(s). |
| Verifier recovery worker (via runtime) | One focused evidence gap, together with the original task and earlier candidate/evidence. |
| Recovery worker verifier (via runtime) | Recovery summary and the relevant Search, Visit, or Python observations. |
| Accuracy (%) | ||
|---|---|---|
| 16 | 128 | 33.33 |
| 32 | 128 | 40.00 |
| 64 | 128 | 40.00 |
| 32 | 64 | 40.00 |
| Allocation policy | Accuracy (%) |
|---|---|
| Online sensitivity updates | 40.00 |
| Uniform allocation | 36.67 |
| Fixed sensitivity | 36.67 |
KV-Cache Quantization
Key-value cache quantization compresses transformer attention state to reduce memory use and bandwidth during long-context inference.
See all 118 papers