Anatomy of a Quantized Agent: VRAM Stability and Forecasting in Code-Synthesis Agentic Workloads

Anubhab Banerjee

cs.AIAug 15, 2026

Anatomy of a Quantized Agent: VRAM Stability and Forecasting in Code-Synthesis Agentic Workloads

Authors: Anubhab Banerjee

Organizations: Nokia Germany

Abstract

Analytical models of peak VRAM consumption for LLM inference decompose memory into weight-storage, KV-cache, and activation terms parameterized by step count, tool invocations, and context expansion. We evaluate this decomposition empirically within a strictly scoped measurement study: a LangGraph-based CUDA-kernel-synthesis agent (AgentK), a 4-bit quantization family (Q4 K M), a single NVIDIA H100 GPU, and four LLM backbones across 1,920 trajectories. Focusing on peak-memory forecasting behavior, we report two primary observations. First, closed-form analytical models achieve competitive accuracy when provided with two empirical constants: loaded-weight VRAM and a fixed activation-memory overhead. Supplied with live GPU readings and ground-truth trajectory parameters, the closed-form model matches or outperforms the best learned baseline on three of the four backbones (test MAPE 2.2-4.4% vs. 3.4-6.5%, p = 0.76). The exception is the smallest backbone (Phi-4-mini), where minimal VRAM variance (CV 0.3%) causes dynamic modeling to underperform simple regression. Second, compile success strictly bifurcates by backbone capacity (from 5.7% for Phi-4-mini to 62.0% for Qwen2.5-Coder-14B), demonstrating that functional code synthesis remains constrained by intrinsic LLM capabilities rather than available memory. Furthermore, because overall peak-memory variance is remarkably low across all backbones (CV 0.3-9.4%), learned prompt-feature regression offers statistically insignificant improvements over a constant-mean baseline. Consequently, we find no justification for deploying complex predictive VRAM models in highly quantized, weight-dominated regimes. We release the evaluated corpus and anonymized framework to support replication.

Figures & tables

Backbone LLM	mean	std	CV (%)	min	p25	median	p75	max	range
Phi-4-mini	5659.4	18.2	0.32	5540.1	5662.1	5662.1	5662.1	5662.1	122.0
Mistral-7B-Instruct	7371.1	689.9	9.36	6116.1	6922.1	7538.1	7604.1	9106.1	2990.0
Qwen2.5-Coder-7B	6815.0	362.2	5.32	6212.1	6562.1	6910.1	6910.1	7808.1	1596.0
Qwen2.5-Coder-14B	12047.8	844.4	7.01	10542.1	11704.1	12208.1	12278.1	14358.1	3816.0

TABLE I:

M_{\mathrm{peak,true}}

distribution on the v2 test split (MiB).

Agent LLM	Method	MAPE (%)	MAE (MiB)	Undercov. rate	OAW (% ceil.)
mistral-7b-instruct-v0.3	B0	10.84	789.3	0.000	0.96
mistral-7b-instruct-v0.3	B1	7.09	512.7	0.067	1.76
mistral-7b-instruct-v0.3	B2	6.52	458.6	0.067	1.56
mistral-7b-instruct-v0.3	B3	7.07	510.6	0.067	1.77
mistral-7b-instruct-v0.3	B5	4.44	333.0	0.373	0.26
phi-4-mini	B0	14.96	846.7	0.000	1.03

TABLE II: Test-set peak VRAM forecast metrics across the four evaluated LLM backbones

	$r{=}0$	$r{=}1$	$r{=}2$	$r{=}3$
Pilot ( $n{=}1800$ )	1054	50	24	672
Final ( $n{=}1920$ )	579	399	174	768

TABLE III: Global retry-count histogram: pilot vs. final

Backbone LLM	$N{=}0$	mean	median	max
Phi-4-mini	0/300	2301	2009	9,812
Mistral-7B-Instruct	0/540	2185	1050	7,704
Qwen2.5-Coder-7B	1/540	1929	980	29,488
Qwen2.5-Coder-14B	0/540	1315	1172	5,844
Global	1/1920 (0.05 %)	—	—	—

TABLE IV:

N_{\mathrm{true}}

summary per LLM backbone (

n=1920

)

Backbone LLM	KernelBench	clean_synthetic	retry_provoking
Phi-4-mini	4.0	7.0	6.0
Mistral-7B-Instruct	4.8	11.1	14.7
Qwen2.5-Coder-7B	28.9	40.0	40.2
Qwen2.5-Coder-14B	53.6	64.2	67.4

TABLE V: Corpus-wide compile_success rate (%) per LLM backbone and prompt category

Fig. 1: Peak VRAM versus correctness per LLM backbone across the evaluated corpus. Left:

M_{\mathrm{peak,true}}

vs. compile_success (jittered) for all 1,920 traces. Right:

M_{\mathrm{peak,true}}

vs. numerical_correct_kbench (jittered) for the 149 KernelBench rows with a defined label.

Backbone LLM	Grouping	$\eta^{2}$	$F$	$p$	$n$	Residual frac.
Mistral	Prompt family	0.000	0.02	9.81e-01	540	1.000
Mistral	Retry count	0.001	0.23	8.77e-01	540	0.999
Mistral	Config sweep	0.913	493.97	3.83e-122	240	0.087
Phi	Prompt family	0.126	21.34	2.18e-09	300	0.874
Phi	Retry count	0.002	0.21	8.90e-01	300	0.998
Phi	Config sweep	—	—	—	0	—

TABLE VI: One-way ANOVA of

M_{\mathrm{peak,true}}

(MiB) by grouping factor

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 2: AgentK end-to-end control flow. Solid arrows represent the forward pipeline; the dashed arrow denotes the out-of-scope short-circuit exit; the red arrow highlights the compiler-diagnostic feedback edge, taken on every compile failure until the retry_budget is reached. Each traversal of the Generator or Critic constitutes one node execution contributing to

N

and to

N_{\mathrm{true}}

(§ III ).

Fig. 3: Per-backbone histograms of

M_{\mathrm{peak,true}}-\bar{M}_{\mathrm{peak}}

(MiB) across the full evaluated corpus. The narrow residual spreads under Q4_K_M are visible for Phi-4-mini and Qwen2.5-Coder-7B; Mistral-7B and Qwen2.5-Coder-14B exhibit wider tails driven specifically by the

n_{\mathrm{ctx}}

and KV-cache hardware sweep rows.

LLM Inference

Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.

See all 280 papers

Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices
Feb 17, 2026Yakov Pyotr ShkolnikovMemory Management for LLM Agents LLM Agent Memory
Prior Knowledge or Search? A Study of LLM Agents in Hardware-Aware Code Optimization
May 19, 2026Dmitry Redko, Albert Fazlyev, Konstantin Sozykin +3GPU Kernel Optimization Large Language Model-Guided Optimization
Applied AI Institute · AI Talent Hub, ITMO University · YSDA +1
Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs
May 19, 2026Haiquan Lu, Zigeng Chen, Gongfan Fang +2LLM Quantization Long-Context Language Model Inference
National University of Singapore

Feb 17, 2026J/K move · Enter open · S save

Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices

Yakov Pyotr Shkolnikov

Memory Management for LLM Agents LLM Agent Memory LLM Inference Efficiency Persistent Memory for Language Models KV-Cache Offloading Multi-Agent LLM Systems+1

Open paper AskPDF arXiv

Multi-agent LLM systems on edge devices face a memory management problem: device RAM is too small to hold every agent's KV cache simultaneously. On Apple M4 Pro with 10.2 GB of cache budget, only 3 agents fit at 8K context in FP16. A 10-agent workflow must constantly evict and reload caches. Without persistence, every eviction forces a full re-prefill through the model -- 15.7 seconds per agent at 4K context. We address this by persisting each agent's KV cache to disk in 4-bit quantized format and reloading it directly into the attention layer, eliminating redundant O(n) prefill computation via direct cache restoration. The system comprises three components: a block pool providing per-agent isolated Q4 KV caches in safetensors format, a BatchQuantizedKVCache for concurrent inference over multiple agents' quantized caches, and cross-phase context injection that accumulates attention state across conversation phases without re-computation. Evaluated on three architectures (Gemma 3 12B, dense GQA, 48 layers; DeepSeek-Coder-V2-Lite 16B, MoE MLA, 27 layers; Llama 3.1 8B, dense GQA, 32 layers), cache restoration reduces time-to-first-token by up to 136x (Gemma: 22--136x at 4K--32K; DeepSeek: 11--76x at 4K--32K; Llama: 24--111x at 4K--16K; 3--10x at 1K). Q4 quantization fits 4x more agent contexts into fixed device memory than FP16. Perplexity measured with actual Q4 KV caches shows -0.7% for Gemma, +2.8% for Llama, and +3.0% for DeepSeek. Open-source at https://github.com/yshk-mxim/agent-memory

Abstract

LLM Inference

Explore similar work

Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices

Prior Knowledge or Search? A Study of LLM Agents in Hardware-Aware Code Optimization

Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs