How Much Heavy Lifting Can an Agent Harness Do?: Measuring the LLM's Residual Role in a Planning Agent

Sungwoo Jung; Seonil Son

cs.AIApr 8, 2026

How Much Heavy Lifting Can an Agent Harness Do?: Measuring the LLM's Residual Role in a Planning Agent

Authors: Sungwoo Jung, Seonil Son

Organizations: Independent Researcher · RLWRLD.AI Seoul, South Korea

Abstract

Agent harnesses -- the stateful programs that wrap a language model and decide what it sees at each step -- are now known to change end-to-end performance on a fixed model by as much as six times. That raises a question asked less often than it should be: how much of an agent's competence does the harness itself already carry, and how much genuinely still needs the LLM? We externalize a planning harness for noisy Collaborative Battleship into four progressively richer layers -- posterior belief tracking, declarative planning, symbolic reflec tion, and an LLM-backed revision gate -- under a common runtime, taking win rate as the primary metric and F1 as secondary, and pre-specifying heavy lifting as the single largest positive marginal to the primary metric. Across 54 games, declarative pla nning carries the heavy lifting ( $+24.1$ pp win rate over a belief-only harness, zero LLM calls); symbolic reflection is mechanistically real but calibration-sensitive, with signed board-level effects up to $\pm0.140$ F1 that cancel on aggregate; and LLM-backed revision ac tivates on only $4.3\%$ of turns with a bounded, non-monotonic effect. The contribution is methodological: once harness layers are made externally measurable, the LLM's role can be quantified as residual rather than assumed central.

Figures & tables

Layer	What it isolates	Closest prior-work analog
L1. Belief	Posterior over hidden world state	Bayesian experimental design; ( Grand et al., 2025 ) belief backend
L2. Planning	World-model rollout for action and question scoring	LLM+P ( Liu et al., 2023 ) , WorldCoder ( Tang et al., 2024 )
L3. Symbolic reflection	Confidence-gated, LLM-free in-episode revision	Reflexion ( Shinn et al., 2023 ) , MIDCA ( Cox et al., 2016 )
L4. LLM-backed revision	Conditional LM intervention on the same gate	ReAct ( Yao et al., 2023 )

Table 1. The four harness layers and the prior-work line each isolates. Decomposition is what makes each

\Delta_{i}

measurable against a fixed substrate rather than against a different bundle.

Harness layer	Role added on top of the previous layer	LLM?
L1. Belief-only	Posterior belief tracking	No
L2. + Planning	Hypothetical transition evaluation and question timing	No
L3. + Symbolic reflection	Confidence tracking and guarded revision actions	No
L4. + LLM-backed revision	Residual revision when the confidence gate opens	Conditional

Table 2. Progressive externalization of a planning harness. Each row adds one layer that the harness carries without calling the LLM; the LLM enters only at L4, and its invocation rate is measured rather than pre-budgeted.

Panel of Figure 1 — Figure 1. Planning harness main loop, externalized into declarative runtime events. Layers L1–L4 in Table 2 correspond to which phases are active: L1 uses only the belief-driven shoot in phase 2; L2 adds sim.next -based action selection with question timing; L3 adds the Reflect and Revise phases with symbolic presets; L4 replaces the Revise body with an LLM-backed patch when the confidence gate opens.

LM-only Captains (no posterior / EIG / decision rule):
Captain / Harness	LLM Rate	Avg F1	Win Rate	95% CI	Avg Q
gemma3n:e4b ( $\sim$ 4B, no reasoning)	100%	0.263	0.0%	[0.0, 6.6]	14.2
Llama-4-Scout (109B MoE, no reasoning)	100%	0.353	0.0%	[0.0, 6.6]	14.8
gpt-5-nano (small reasoning)	100%	0.459	29.6%	[19.1, 42.8]	12.4
gpt-5-mini (mid reasoning)	100%	0.565	77.8%	[65.1, 86.8]	14.2
Harness decomposition (this paper):

Table 3. Unified within-suite comparison (

54

games =

18

boards

\times

3

seeds;

8\times 8

noisy Battleship,

\varepsilon{=}0.1

; same MCMC belief backend). LM-only Captains call the LLM on every turn with no posterior, EIG, or decision rule; openai/gpt-5-* models use reasoning_effort=medium (default). Harness layers (this paper) progressively add structure on top of a shared belief; the LLM is gated to L4 only ( gemma4:e4b via Ollama). Win-rate CI is Wilson

95\%

on the game-completion proportion; we rely on its width to regulate the strength of claims. Bold : best per block.

Figure 2. Win rate per harness layer with Wilson

95\%

binomial confidence interval (

n{=}54

; harness block of Table 3 ). The L1

\rightarrow

L2 step is the only vertical jump in the harness; within the reflective family (L3, L4) the intervals are flat against each other and the LLM-backed L4 sits in the same band as the LLM-free L3 baselines. The picture is what “the harness does the heavy lifting, not the LLM” looks like at the level of confidence intervals.

Figure 3. Per-board

\Delta

F1 (symbolic reflection on minus off) across the

12

boards recoverable from the registry top/bottom-5 overlap, sorted. Green bars are recovery boards (revision helps); red are over-revision boards (revision hurts). The dotted grey line is the aggregate

\Delta{=}-0.001

. The bimodal, signed-both-ways shape is visual evidence that the right diagnosis of the flat aggregate is calibration, not absence of mechanism.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Scope	$\tau$	LLM Rate	Avg F1	Win Rate
18 games	0.0	0.0%	0.522	50.0%
18 games	0.5	0.0%	0.522	50.0%
18 games	0.72	1.1%	0.515	50.0%
18 games	1.0	4.6%	0.569	61.1%
54 games	0.0	0.0%	0.552	57.4%
54 games	1.0	4.3%	0.557	53.7%

Appendix

Table 4. Threshold sweep across evaluation scales (§4.3). Thresholds

0.0

and

0.5

collapse into a no-LLM basin; threshold

1.0

activates LLM revision on a measured

4

–

5\%

of turns. The 18-game and 54-game rows diverge at

\tau{=}1.0

, underscoring the need for larger-scale evaluation while preserving the qualitative finding of non-monotonicity.

Automated Agent Evaluation

See all 43 papers

Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution
Sep 29, 2026Subba Reddy Oota, Francisco Herrera, Jordi Cabot Sagrera +2LLM Agents Long-Horizon LLM Agents
ADIA Lab, Abu Dhabi, United Arab Emirates · University of Granada, Granada, Spain · Luxembourg Institute of Science and Technology, Luxembourg +2
Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents
Jun 3, 2026Haoyu Sun, Wenxuan Wang, Mingyang Song +5Tool-Augmented Language Model Agents LLM Agent Evaluation
Tongji University · Shanghai AI Laboratory · Harbin Institute of Technology +5
HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
Sep 1, 2026Yuhao Wu, Jingyuan Zhang, Jiajun Shi +16LLM Agent Evaluation Agent Harness Optimization

Sep 29, 2026J/K move · Enter open · S save

Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution

Subba Reddy Oota, Francisco Herrera, Jordi Cabot Sagrera, Marcos López de Prado+1

ADIA Lab, Abu Dhabi, United Arab Emirates · University of Granada, Granada, Spain · Luxembourg Institute of Science and Technology, Luxembourg+2

LLM Agents Long-Horizon LLM Agents LLM Agent Planning LLM Agent Routing

Open paper AskPDF arXiv

Large language models (LLMs) enable agents to solve long-horizon tasks by generating a plan and then executing it in an environment. However, successful planning requires two distinct capabilities: selecting an appropriate plan for the task and executing it faithfully. Existing planner--executor systems can fail at either stage, while final task success alone cannot distinguish selection from execution failures. We therefore study the Plan Declaration--Execution Gap and introduce Planning-as-Routing, where an LLM declares one of four planning modes: Predefined, Sequential, Hierarchical, or Search, and a deterministic router dispatches the task to the corresponding pattern-specific executor. Across four benchmarks and three LLMs, we find three consistent patterns. First, generic Plan+ReAct often fails to preserve declared planning structure, especially for longer plans: across three benchmarks, only (22)--(45%) of trajectories preserve it, whereas pattern-specific executors enforce the intended structure. Second, planning-mode effectiveness varies across environments and models: Search performs best on ALFWorld, Hierarchical on SWE-bench, and the strongest pattern can vary across models within the same benchmark. Third, the largest gains come from execution: pattern-specific executors improve task success from (0.48) to (0.92) on ALFWorld and from (0.36) to (0.44) on SWE-bench Verified over Plan+ReAct. Current LLMs, however, do not reliably select the strongest mode for each task, although few-shot examples improve selection in some benchmark--model combinations. Overall, reliable agent planning requires both effective mode selection and faithful execution: routing substantially closes the execution gap, while task-specific mode selection remains open.

Abstract

Automated Agent Evaluation

Explore similar work

Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution

Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?