Optimize the Harness, Not the Model: Fitting a Minimal Agent to a Self-Hostable LLM for Investigative Analysis

Shai Nahum Gefen; Vered Chassida; Dmitry Basin; Guy Feigenblat

cs.AINov 2, 2026

Optimize the Harness, Not the Model: Fitting a Minimal Agent to a Self-Hostable LLM for Investigative Analysis

Authors: Shai Nahum Gefen, Vered Chassida, Dmitry Basin, Guy Feigenblat

Organizations: Dream Security, Tel Aviv, Israel

Abstract

Government intelligence teams need LLM agents that can analyze large, messy evidence collections, but security and cost constraints often rule out hosted frontier models and broad, hard-to-audit tool surfaces. We report CLEO: Constrained Loop for Environment Optimization, a method for fitting a minimal agent harness to a fixed, self-hostable LLM. Given an expert-validated dataset, CLEO admits only the tools, guardrails, memory policies, and loop components that improve accuracy while remaining general and auditable. On our held-out dataset, the CLEO harness reaches an 83.8% pass rate, within 1.2 points of the strongest out-of-the-box Claude Code SDK baseline, while being two orders of magnitude smaller.

Figures & tables

Harness	LOC	Pass	Soft $\mu\pm\sigma$	Lat.
Claude Code SDK (bare)	$\sim$ 512k 1 1 1 The SDK is closed-source and distributed minified, so no exact source-line count is publicly verifiable. The $\sim$ 512k figure is the one widely reported following the 2026 source-map disclosure; we could not independently verify it.	85.0%	8.48 $\pm$ 2.25	76s
PI #1 (bare)	$\sim$ 75k	80.8%	8.12 $\pm$ 2.70	126s
PI #2 (CLEO’s full prompts)	$\sim$ 75k	83.8%	8.45 $\pm$ 2.39	121s
CLEO	$\sim$ 3k	83.8%	8.41 $\pm$ 2.51	114s

Table 1. Combined held-out benchmark results for four harnesses under the same model

LLM Agent Harnesses

LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.

See all 406 papers

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
Aug 6, 2026Varun Ursekar, Apaar Shanker, Yash Maurya +4Large Language Model-Guided Optimization LLM Agent Evaluation
Scale AI
Harness as a Language: A Minimalist Agent Framework With Maximal Expressivity
Sep 22, 2026Zhening Li, Joshua Liu, Mateja Vukelic +7LLM Agent Memory LLM Agent Self-Improvement
MIT CSAIL · Independent Researcher
EnvOpt: Automated Optimization for Agent Environments
Nov 5, 2026Mingxuan Li, Huanzhi Mao, Qiuyang Mang +3LLM Agents LLM Agent Efficiency
University of California, Berkeley · University of Chicago · Bespoke Labs

Aug 6, 2026J/K move · Enter open · S save

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

Varun Ursekar, Apaar Shanker, Yash Maurya, Shehab Yasser+3

Scale AI

Large Language Model-Guided Optimization LLM Agent Evaluation Agent Harness Optimization LLM Agent Workflow Optimization AI Agent Benchmarks

Open paper AskPDF arXiv

As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them. This makes automated harness optimization -- the iterative and evaluation-guided improvement of a harness by an AI system -- both an important route to improving AI systems and a demanding capability for AI systems themselves. Yet the community lacks a common protocol for measuring how well frontier LLMs perform at this task. We introduce HarnessOpt-Bench, a benchmark for end-to-end harness optimization under expensive and stochastic evaluation. An optimizer, an LLM paired with a coding harness, receives a target agent's seed harness, graded evaluation feedback, and a fixed target-evaluation budget. It edits the harness and nominates a final candidate, which is scored by its normalized gain over the seed on a held-out test partition that remains inaccessible throughout search. A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit. We evaluate 5 frontier LLMs as optimizers both under a shared coding harness and under their native harnesses across 4 downstream tasks, over 111 scored runs. Experiment results show that optimizer models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks and seed regimes. These results establish harness optimization as a measurable and discriminative capability with large space for improvement.

Abstract

LLM Agent Harnesses

Explore similar work

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

Harness as a Language: A Minimalist Agent Framework With Maximal Expressivity

EnvOpt: Automated Optimization for Agent Environments