Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol for Agentic LLM Serving

Jiaqi Zhao; Haodong Chen; Jitai Hao; Wei Zhao; Jinghao Pang; Qiang Huang; Jun Yu

cs.AIOct 5, 2026

Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol for Agentic LLM Serving

Authors: Jiaqi Zhao, Haodong Chen, Jitai Hao, Wei Zhao, Jinghao Pang, Qiang Huang, Jun Yu

Organizations: Harbin Institute of Technology (Shenzhen)

Abstract

LLM agents increasingly execute complex workflows involving multi-turn reasoning, tool use, and parallel agents. Efficient serving requires decisions that span two layers with complementary information: the agent harness understands workflow dependencies, context lifecycles, and execution objectives, whereas the inference engine observes request queues, KV-cache state, resource pressure, and execution capabilities. Existing interfaces do not systematically connect these views, limiting workflow-aware execution. HEAR, a bidirectional Harness--Engine Pairing protocol for agentic LLM serving. HEAR standardizes how the harness communicates workflow intent and execution requirements and how the engine returns runtime state, capabilities, and outcomes. By separating protocol semantics from optimization policies, HEAR supports diverse coordination strategies without changing workflow or model semantics. We instantiate HEAR for online cache-aware runtime coordination and workload-aware execution-mode selection for agent roles. Across four conversational and research-agent benchmarks under memory-constrained, concurrent serving, HEAR achieves a $1.61\times$ batch speedup and reduces median time-to-first-token by $2.23\times$ on SCBench. Mooncake shows that workflow intent and live engine state provide complementary benefits across load regimes. On BrowseComp-Plus and DeepResearchBench, workload-specific configurations yield $1.23\times$ and $2.45\times$ end-to-end speedups, respectively, without observed task-quality degradation. These results establish HEAR as a reusable coordination substrate for efficient agentic LLM serving.

Figures & tables

Figure 1 : Motivating example. Without workflow intent, the engine may evict KV state shortly before reuse; without KV-residency feedback, the harness may schedule a cold request before an equivalent cache-resident one.

Figure 2 : Overview of HEAR and an illustrative dynamic-agent trace. Left: HEAR organizes bidirectional harness-engine communication into four semantic message categories. Right: As the workflow evolves, the harness sends workflow descriptions and execution controls (blue), while the engine returns runtime state and execution outcomes (orange), forming a feedback loop for scheduling and KV-cache management. KV-cache colors denote decoding, resident, preparing, and empty states. The trace illustrates one policy instantiation; HEAR does not prescribe this sequence of interactions.

Category	Representative Contents
Harness $\rightarrow$ Engine
Execution Description and Intent	Agent roles and stages; workflow dependencies and readiness; request sizes and output limits; context lifecycles; known future uses and explicitly marked predictions.
Execution Requirements and Control	Scheduling preferences and waiting-time constraints; cache preparation, retention, and release requests; allowed or requested execution configurations and serving targets.
Engine $\rightarrow$ Harness
State and Capabilities	Reusable prefix ranges and residency tiers; allocatable capacity, queue state, and resource pressure; supported controls and configurations; calibrated or estimated execution costs.
Execution Outcomes	Status updates for inference and control requests (accepted, completed, rejected, unsupported, or failed); actual configuration, permitted fallback, and cache reuse; latency, resource use, retries, and errors.

Table 1: Bidirectional message categories and representative information.

Distinction	Protocol Interpretation
Intent vs. Control	Descriptions expose current or anticipated execution conditions; only explicit control requests ask the receiver to act. For example, reporting that context $c$ will be reused is distinct from requesting that its KV cache be retained or prepared.
Preference vs. Requirement	Preferences are best-effort, whereas requirements constrain valid execution. An unsatisfied requirement must be rejected or handled by an explicitly permitted fallback.
	For example, a priority hint may affect request ordering, but it cannot override a mandatory protection condition.
Observation vs. Guarantee	State reports describe observed conditions, and capability reports describe supported operations. Neither reserves resources nor guarantees that a future request will be admitted.
	For example, GPU-residency feedback may guide scheduling without reserving the prefix.
Acceptance vs. Completion	Acceptance means that an operation has beenadmitted for processing; completion confirms that its stated effect has been realized. Rejected, unsupported, and failed operations remain distinct outcomes.

Table 2: Key semantic distinctions in bidirectional protocol interactions.

Configuration	TTFT $\downarrow$			Batch Completion $\downarrow$	Reuse $\uparrow$
Configuration	P50	P95	MAX	Batch Completion $\downarrow$	Reuse $\uparrow$
FCFS	63.1 1.00×	74.2 1.00×	79.0 1.00×	481 1.00×	19.2
Cache-Aware	4.6 13.6×	75.8 0.98×	93.9 0.84×	224 2.15×	87.6
Cache-Aware + Guard-40	28.3 2.23×	51.1 1.45×	65.0 1.22×	299 1.61×	66.1
Cache-Aware + Guard-60	4.5 13.9×	64.0 1.16×	80.4 0.98×	230 2.09×	84.7

Table 3: Cache-aware runtime coordination results on SCBench. All times are in seconds. Batch completion is the elapsed time to complete all users’ multi-turn conversations. Gray subscripts report speedups relative to FCFS . Reuse denotes the KV-cache reuse rate for follow-up prompt tokens.

Figure 3 : KV reuse and GPU KV occupancy on Mooncake Bench under different load levels.

Load	Configuration	TTFT $\downarrow$			Session $\downarrow$	Throughput $\uparrow$	Reuse $\uparrow$
Load	Configuration	P50	P95	MAX	Session $\downarrow$	Throughput $\uparrow$	Reuse $\uparrow$
50%	FCFS	0.7 $\times$ 1.00	2.8 $\times$ 1.00	6.2 $\times$ 1.00	13.7 $\times$ 1.00	86.1 $\times$ 1.00	19.5
	Cache-Aware	0.6 $\times$ 1.21	2.4 $\times$ 1.17	5.9 $\mathbf{\times 1.06}$	13.3 $\times$ 1.02	86.0 $\times$ 1.00	20.2
	Cache-Aware + Guard	0.6 $\times$ 1.10	2.8 $\times$ 1.01	7.4 $\times$ 0.84	13.6 $\times$ 1.01	86.2 $\times$ 1.00	19.6
	Session-Aware	0.6 $\times$ 1.16	2.5 $\times$ 1.14	6.3 $\times$ 0.99	13.0 $\times$ 1.05	86.6 $\times$ 1.01	31.9
	Session-Aware + Cache-Aware + Guard	0.5 $\mathbf{\times 1.35}$	2.3 $\mathbf{\times 1.21}$	6.1 $\times$ 1.02	12.7 $\mathbf{\times 1.08}$	86.8 $\mathbf{\times 1.01}$	32.1

Table 4: Cache-aware runtime coordination results on Mooncake Bench. Session is the total response-waiting time per multi-turn conversation; throughput is measured in turns per minute.

Main	Reader	Browscomp-plus			Deep-research-bench
Main	Reader	Wall (h) $\downarrow$	P95 E2E (min) $\downarrow$	Acc. (%) $\uparrow$	Makesp (h) $\downarrow$	P95 E2E (min) $\downarrow$	RACE (%) $\uparrow$
vLLM	vLLM	$5.69_{\color[rgb]{0.5,0.5,0.5}1.00\times}$	$40.0_{\color[rgb]{0.5,0.5,0.5}1.00\times}$	46.15	$5.93_{\color[rgb]{0.5,0.5,0.5}1.00\times}$	$39.8_{\color[rgb]{0.5,0.5,0.5}1.00\times}$	40.6
OmniKV	vLLM	$5.37_{\color[rgb]{0.5,0.5,0.5}1.06\times}$	$40.3_{\color[rgb]{0.5,0.5,0.5}0.99\times}$	47.12	$6.17_{\color[rgb]{0.5,0.5,0.5}0.96\times}$	$39.9_{\color[rgb]{0.5,0.5,0.5}1.00\times}$	41.1
vLLM	$\mathbf{H_{2}O}$	$\mathbf{4.62}_{\color[rgb]{0.3,0.3,0.3}\mathbf{1.23\times}}$	$39.6_{\color[rgb]{0.5,0.5,0.5}1.01\times}$	46.63	$2.76_{\color[rgb]{0.5,0.5,0.5}2.15\times}$	$15.6_{\color[rgb]{0.5,0.5,0.5}2.55\times}$	40.5
OmniKV	$\mathbf{H_{2}O}$	$4.98_{\color[rgb]{0.5,0.5,0.5}1.14\times}$	$37.2_{\color[rgb]{0.5,0.5,0.5}1.08\times}$	43.27	$\mathbf{2.42}_{\color[rgb]{0.3,0.3,0.3}\mathbf{2.45\times}}$	$15.8_{\color[rgb]{0.5,0.5,0.5}2.52\times}$	41.1
vLLM	SnapKV	$4.83_{\color[rgb]{0.5,0.5,0.5}1.18\times}$	$40.4_{\color[rgb]{0.5,0.5,0.5}0.99\times}$	44.23	$2.85_{\color[rgb]{0.5,0.5,0.5}2.08\times}$	$17.7_{\color[rgb]{0.5,0.5,0.5}2.25\times}$	40.0

Table 5: Capability-aware execution configuration results.

Benchmark	Main Output Tok./Req.	Main Latency Alt. $\rightarrow$ Selected	Reader/Sub Latency Dense $\rightarrow$ Selected	Cache Evidence
BrowseComp-Plus	$\sim 0.4$ K	OmniKV $\rightarrow$ vLLM $17.0\rightarrow 14.9$ s 1.14×	vLLM $\rightarrow$ H 2 O $53.4\rightarrow 9.5$ s 5.60×	Prefill $-76.2\%$ Reuse $6.2\%\rightarrow 80.3\%$
DeepResearchBench	$\sim 4.4$ K	vLLM $\rightarrow$ OmniKV $69.4\rightarrow 64.2$ s 1.08×	vLLM $\rightarrow$ H 2 O $421.5\rightarrow 58.4$ s 7.22×	Chain reuse $49.8\%$

Table 6: Role-level execution attribution.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Frequency	Additional Online Work	Included in Formal E2E Time
KV-state update	Per cache event	Residency-directory metadata update	Yes
Scheduling decision	Per dispatch point	Prefix lookup, priority construction, and Guard check	Yes
Role-to-mode lookup	Per model request	Lookup in the frozen role mapping	Yes
Outcome reporting	Per inference/control outcome	Status, actual mode, reuse, retry, and error metadata	Yes
Profile calibration	Offline development stage	Candidate profiling, filtering, ranking, and verification	No

Appendix

Table 7: HEAR protocol cost accounting.

Mode	Cache and Token Settings	Execution Capacity	Mode-specific Parameters
Main Role
vLLM	Prefix caching Max. context 202,752; 32,768 batched tokens	16 sequences	Dense execution
OmniKV	Radix cache 32,768-token prefill chunks	4 batch / 4 decode / 16 GPU sequences	Full-attention layers: automatic; Retention: 8 sink / 256 recent / 4,096 decode tokens; Prefill workspace: 8 GiB
Reader Role
vLLM	Prefix caching Max. context 202,752; 16,384 batched tokens	16 sequences	Dense execution
H 2 O	Chain cache 16,384-token prefill chunks	12 batch / 12 decode / 36 GPU sequences	Budgets: 16,384 prefill / 8,192 decode tokens; Recent ratio: 0.2; Scoring: 128-token window with logits-based prefill scoring

Appendix

Table 8: Execution-mode configurations for BrowseComp-Plus.

Mode	Cache and Token Settings	Execution Capacity	Mode-specific Parameters
Main Role
vLLM	Prefix caching Max. context 200,000; 32,768 batched tokens	10 sequences	Dense execution
OmniKV	Radix cache 32,768-token prefill chunks	10 batch / 10 decode / 16 GPU sequences	Full-attention layers: $\{0,3,9,18,22,27,43\}$
Sub-Agent Role
vLLM	Prefix caching Max. context 200,000; 32,768 batched tokens	10 sequences	Dense execution
H 2 O	Chain cache 32,768-token prefill chunks	10 batch / 10 decode / 44 GPU sequences	Budgets: 8,192 prefill / 4,096 decode tokens; Recent ratio: 0.2; Scoring: 128-token window with logits-based prefill scoring

Appendix

Table 9: Execution-mode configurations for DeepResearchBench.

Figure 4 : Runtime execution timelines on SCBench. Each session is decomposed into queue wait, model service (prefill + decode), and user gap.

Figure 5 : Runtime behavior under the Mooncake replay. GPU KV-pool occupancy, cumulative KV restoration, and the number of waiting requests over time for different scheduling policies.

Main	Reader	Efficiency and Latency						Quality and Completion Outcomes
Main	Reader	Wall (h) $\downarrow$	Speedup $\uparrow$	Throughput $\uparrow$	Median $\downarrow$	P95 $\downarrow$	Max $\downarrow$	Acc. (%) $\uparrow$	Retr. Recall (%) $\uparrow$	C	I	U	F
vLLM	vLLM	5.69	$1.00\times$	36.57	14.63	40.00	41.32	46.15	69.22	96	46	54	12
	H 2 O	4.62	$\mathbf{1.23\times}$	45.00	10.98	39.58	41.81	46.63	70.17	97	59	45	7
	SnapKV	4.83	$1.18\times$	43.03	10.75	40.38	43.26	44.23	70.56	92	63	40	13
OmniKV	vLLM	5.37	$1.06\times$	38.72	13.67	40.28	46.54	47.12	68.75	98	42	53	15
	H 2 O	4.98	$1.14\times$	41.73	12.64	37.24	42.37	43.27	68.33	90	61	49	8

Appendix

Table 10: Complete BrowseComp-Plus six-arm results.

Main	Sub-Agent	Efficiency and Latency							Quality and Completion Outcomes
		Makespan $\downarrow$	Speedup $\uparrow$	Throughput $\uparrow$	E2E Latency (min) $\downarrow$				$N$	RACE (%) $\uparrow$	Comp.	Insight	Instr.	Read.	Errors	Empty
		Makespan $\downarrow$	Speedup $\uparrow$	Throughput $\uparrow$	Mean	Median	P95	Max	$N$	RACE (%) $\uparrow$	Comp.	Insight	Instr.	Read.	Errors	Empty
vLLM	vLLM	5.93	$1.00\times$	16.7	29.46	30.64	39.81	41.18	97	40.65	39.93	38.68	42.98	42.91	1	4
	$\mathbf{H_{2}O}$	2.76	$2.15\times$	36.3	13.17	13.17	15.64	21.27	96	40.46	39.65	38.68	42.81	42.57	0	4
	SnapKV	2.85	$2.08\times$	35.1	15.13	15.19	17.70	20.26	99	40.01	39.33	37.78	42.51	42.45	0	1
OmniKV	vLLM	6.17	$0.96\times$	16.2	30.41	30.64	39.93	40.64	98	41.14	40.30	39.26	43.57	43.20	0	2

Appendix

Table 11: Complete DeepResearchBench six-arm results.

Benchmark	Role	Req./Item	Input/Req. (K)	Output/Req. (K)	Workflow Pattern
BrowseComp-Plus	Main	3.0	52.5	0.36	Dynamic coordination and answer synthesis
BrowseComp-Plus	Reader	13.3	59.1	0.36	Question-dependent document reads
DeepResearchBench	Main	5.4	32.5	4.42	Round-synchronized review and synthesis
DeepResearchBench	Sub-Agent	12.0	135.3	0.51	Four parallel agents over three fixed rounds

Appendix

Table 12: Role-level workload shapes under the selected configurations.

Benchmark	Role	Alternative $\rightarrow$ Selected	Mean Latency (s)	Gain	Execution Evidence
BrowseComp-Plus	Main	OmniKV $\rightarrow$ vLLM	$17.0\rightarrow 14.9$	$1.14\times$	Selected Main output: 0.36K tokens/request
BrowseComp-Plus	Reader	vLLM $\rightarrow$ H 2 O	$53.4\rightarrow 9.5$	$5.60\times$	Prefill $-76.2\%$ ; reuse $6.2\%\rightarrow 80.3\%$
DeepResearchBench	Main	vLLM $\rightarrow$ OmniKV	$69.4\rightarrow 64.2$	$1.08\times$	Selected Main output: 4.42K tokens/request
DeepResearchBench	Sub-Agent	vLLM $\rightarrow$ H 2 O	$421.5\rightarrow 58.4$	$7.22\times$	Chain reuse: $49.8\%$

Appendix

Table 13: Role-level execution behavior under arm-pair comparisons.

Main	Reader	Outcome Counts				Answer Quality
Main	Reader	Correct	Incorrect	Unresolved	Failure	Acc. (%)	95% CI
vLLM	vLLM	96	46	54	12	46.15	$[39.51,\,52.94]$
OmniKV	vLLM	98	42	53	15	47.12	$[40.45,\,53.89]$
vLLM	$\mathbf{H_{2}O}$	97	59	45	7	46.63	$[39.98,\,53.41]$
OmniKV	$\mathbf{H_{2}O}$	90	61	49	8	43.27	$[36.72,\,50.06]$
vLLM	SnapKV	92	63	40	13	44.23	$[37.65,\,51.02]$

Appendix

Table 14: BrowseComp-Plus answer-quality outcomes.

Main	Reader	Answers Produced	Evidence Quality (%)
Main	Reader	Answers Produced	Retrieval Recall $\uparrow$	Citation Precision $\uparrow$	Citation Recall $\uparrow$
vLLM	vLLM	142	69.22	78.01	30.82
OmniKV	vLLM	140	68.75	77.71	33.90
vLLM	$\mathbf{H_{2}O}$	156	70.17	75.44	28.62
OmniKV	$\mathbf{H_{2}O}$	151	68.33	74.78	27.87
vLLM	SnapKV	155	70.56	72.63	28.10

Appendix

Table 15: BrowseComp-Plus retrieval and citation quality.

Main	Sub-Agent	Evaluation Coverage			RACE (%)
Main	Sub-Agent	Judged	Empty	Task Errors	Overall	Comp.	Insight	Instr Follow.	Read.
vLLM	vLLM	97	4	1	40.65	39.93	38.68	42.98	42.91
OmniKV	vLLM	98	2	0	41.14	40.30	39.26	43.57	43.20
vLLM	$\mathbf{H_{2}O}$	96	4	0	40.46	39.65	38.68	42.81	42.57
OmniKV	$\mathbf{H_{2}O}$	99	1	0	41.05	39.91	39.59	43.26	43.10
vLLM	SnapKV	99	1	0	40.01	39.33	37.78	42.51	42.45

Appendix

Table 16: DeepResearchBench RACE quality and evaluation coverage.

Benchmark	Selected	Paired $N$	Quality Diff. (%)	95% Bootstrap CI (%)	Paired Test
BrowseComp-Plus	vLLM–H 2 O	208	$+0.48$	$[-5.29,\,6.25]$	McNemar $p=1.000$
DeepResearchBench	OmniKV–H 2 O	89	$+0.51$	$[-0.36,\,1.46]$	Paired $t$ -test $p=.278$

Appendix

Table 17: Paired quality comparison against all-vLLM.

Explore similar work

Apr 28, 2026cs.MA

Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving

As LLM applications grow more complex, developers are increasingly adopting multi-agent architectures to decompose workflows into specialized, collaborative components, introducing structure that constrains agent behavior and exposes useful semantic predictability. Unlike traditional LLM serving, which operates under highly dynamic and uncertain conditions, this structured topology enables opportunities to reduce runtime uncertainty

\unicode{x2015}

yet existing systems fail to exploit it, treating agentic workloads as generic traffic and incurring significant inefficiencies. Our analysis of production traces from an agent-serving platform and an internal coding assistant reveals key bottlenecks, including low prefix cache hit rates, severe resource contention from long-context requests, and substantial queuing delays due to suboptimal scaling. To address these challenges, we propose Pythia, a multi-agent serving system that captures workflow semantics through a simple interface at the serving layer, unlocking new optimization opportunities and substantially improving throughput and job completion time over state-of-the-art baselines.

Shan Yu, Junyi Shu, Yuanjiang Ni +14

UCLA · Alibaba Cloud Computing · Alibaba Group +6

Workload Cache

Ask PDF

Feb 14, 2026cs.OS

ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System

Large language models(LLMs) are now used to power complex multi-turn agentic workflows. Existing systems run agentic inference by loosely assembling isolated components: an LLM inference engine (e.g., vLLM) and a tool orchestrator (e.g., Kubernetes). Although agentic workflows involve multiple LLM and tool requests, these systems schedule and allocate resources separately on a per-request basis, without end-to-end knowledge of the workflow. This leads to sub-optimal management of KV cache and tool execution environments. To address the challenges, we propose ThunderAgent, a fast, simple, and program-aware agentic inference system. We first abstract agentic workflows as LLM Programs, enabling a unified view of heterogeneous resources, including KV caches, system states, and external tool assets such as disk memory and network ports. Built upon this abstraction, ThunderAgent introduces a program-aware scheduler and a tool resource manager designed to maximize KV cache hit rates, mitigate memory imbalances, and enable asynchronous environment preparation. Evaluations across coding, routing, and scientific discovery agents demonstrate that ThunderAgent achieves 1.5-3.6x throughput improvements in serving, 1.8-3.9x in RL rollout, and up to 4.2x disk memory savings compared to state-of-the-art inference systems. To facilitate reproducibility and support future development, we open-source the system implementations of the whole ThunderAgent at: https://github.com/Agentic-Kinetics/ThunderAgent.

Hao Kang, Ziyang Li, Weili Xu +7

University of Illinois Urbana-Champaign · 5Together AI · 1Georgia Institute of Technology

Agentic Inference LLM Inference Optimization Agentic Workflows Key-Value Cache

Ask PDF

May 15, 2026cs.LG

Harnesses for Inference-Time Alignment over Execution Trajectories

Harness engineering has emerged as an important inference-time technique for large language model (LLM) agents, aiming to improve long-term performance through task decomposition and guided execution. However, more elaborate harnesses are not uniformly better: increasing decomposition or guidance can sometimes improve execution, but can also reduce final task success. We study harness design through the lens of inference-time trajectory alignment. This perspective separates harness into two mechanisms: task decomposition, which structures a task into sub-goals, and guided execution, which reshapes local action distributions during execution. This decomposition allows us to quantify how workflow granularity, retry budgets, and guidance-induced action reweighting shape the performance limits of harness design. It further reveals concrete failure modes, including over-decomposition, over-pruning, and hallucinated execution. We validate these predictions through controlled synthetic experiments and real terminal agent benchmarks. Inspired by the theory, we further show that effective harnesses can be partial: specifying only the initial steps and leaving the remaining execution to agent can achieve higher pass rate than fully structured workflows.

Boyuan Wang, Bochao Li, Minghan Wang +2

Southern University of Science and Technology

Agent Harness Agentic Inference

Ask PDF