kw-bench: Breaking Frontier Harnesses on Real-World Tasks

George Juraj Salapa

cs.AIOct 26, 2026

kw-bench: Breaking Frontier Harnesses on Real-World Tasks

Authors: George Juraj Salapa

Organizations: G.S. s.r.o. · PwC Austria

Abstract

Each new generation of frontier model pushes the edge of what is possible. Inside a harness -- a loop, a shell, a filesystem, the ability to write and run its own code -- a model can take on increasingly difficult work. A growing body of work treats the coding-agent harness as infrastructure for knowledge work rather than a coding tool, and a new wave of benchmarks tests it there. Most of them are generative: build a financial model, a memo, a spreadsheet from a brief, judged by a rubric or by another model. Few test the expensive part of knowledge work, which is investigative. In audit, tax, due diligence, regulatory reporting, claims or any dispute over numbers, the answer already exists -- hidden in incomplete and inconsistent evidence spread across systems -- and has to be recovered and defended to the cent. We introduce kw-bench, a benchmark built from real engagements specialists described to us. Its first task family is a five-year VAT reconstruction across three generations of SAP -- legacy ECC, a partial S/4HANA migration and a consultant's redesign -- delivered as raw system extracts: page headers, local number formats, partial link tables, no documentation of how anything connects. Grading is deterministic, with no LLM judge. We ran four harnesses -- Claude Code, Codex, OpenCode and micro-cc -- on the same model, DeepSeek v4.1 Flash, for two hours. The two independent, model-agnostic harnesses delivered (OpenCode 0.542, micro-cc 0.528); the two lab harnesses, away from their home model, delivered nothing (0.00 each). What defeated them was inertia: keeping many weak signals alive across a long session. On a frontier model the task collapses. micro-cc and Claude Code on Claude Opus 5.5 scored 0.9986 and 1.000, in 22 and 25 minutes. A weeks-long engagement now costs minutes and a few dollars; below the frontier, the harness decides how far a model gets.

Figures & tables

Figure 1: Figure 1. One run, every tool call: micro-cc 0.2.124 on DeepSeek v4.1 Flash, two hours, 186 model calls. Each column band is one tool call, drawn from its own command text. The bars at the bottom are per-turn context (up) and output tokens (down); red marks a tool error, green a deliverable written. The end card shows the seven component scores and the total. The cost in the header is a list-price estimate. The full run is in the supplementary video.

Figure 2: Figure 2. The prompt is an email, exactly as the harness receives it. It asks for a chain register, a reconciliation to the cent, a correction file, a two-page memo and a classifier the S/4 team can rerun.

Figure 3: Figure 3a. Regime 1, legacy ECC: one delivery, DE01’s intercompany invoice and AT01’s customer invoice, linked through the document flow. From 2021 to mid-2022 a chain is easy to see.

Figure 4: Figure 3b. Regime 2, S/4HANA from July 2022: the migration log covers only part of the runs, migrated deliveries keep their legacy number only partly, and open items came over without tax. Linking mid-2022 documents means combining a partial log with partial legacy references.

Figure 5: Figure 3c. Regime 3, the 2024 chain cockpit: legs linked only by a chain ID; the manually billed intercompany leg has no chain ID and no Incoterm, only a hand-typed reference (“RKC-2024-6”, “rkc-2024-000008”, “RKC 2024 000054”). Finding chains now means normalising what people typed.

Figure 6: Figure 4a. Evidence trail for the repost error: a June 2022 ECC invoice re-entered in S/4 (“Nacherfassung”) with the same tax, next to the migrated open item that carries none. The migration declared it twice.

Figure 7: Figure 4b. Evidence trail for the procflag error: a Vienna assembly order carries a chain ID, but the goods receipt, assembly consumption and second delivery show two movements. A chain ID is not a chain.

Figure 8: Figure 5. Template for the VAT reconciliation: filed against recomputed per filer, period and return line. More than a thousand cells, graded to the cent; the filed returns are the anchor.

Component	Weight	How it is checked
Chain register	0.20	Per chain billing document: F1 on (document, treatment, jurisdiction) 0.30; mis-filed documents found, with false claims penalised 0.35; moving supply 0.15; filed treatment 0.10; chain grouping 0.10
Reconciliation	0.22	Every filer, period and line; cells that differ must be recomputed within EUR 0.02 (tax lines half by count, half by amount); false differences penalised; filed values copied
Correction file	0.12	Every amended return complete (all seven lines) and correct; extra returns penalised
run.sh on hidden data	0.20	Run as an unprivileged user on a holdout with another seed, other counts and other registration and go-live dates; F1 plus moving supply
Memo	0.08	Total and yearly exposure, root cause per planted mechanism, recommendation (text via pdftotext )
Traps	0.06	No assembly or stock-transfer documents in the register, no false tax differences; credited only alongside real findings

Table 9

Harness	Version	Route	How the 1M window was set	Other adjustments
Claude Code (Anthropic)	2.1.310	OpenRouter, Anthropic-compatible endpoint	CLAUDE_CODE_MAX_CONTEXT_TOKENS=1000000 (it does not know the window of a model it does not recognise)	Effort high, 32k thinking budget; all helper models redirected to DeepSeek
Codex CLI (OpenAI)	0.164.0	OpenRouter, Responses API	A model catalog entry declaring 1M (Codex caps unknown models at 272k)	Custom provider with remote compaction off (Codex compacts through an OpenAI-only endpoint); its freeform patch and search tools switched off, as this route cannot call them; Codex then runs with that catalog entry’s system prompt
OpenCode	1.19.4	OpenRouter, built-in provider	Read from its model registry (1,048,576)	--variant high ; none
micro-cc	0.2.124	OpenRouter	Read from its model catalog	None

Table 10

Step	OpenCode	micro-cc	Codex	Claude Code
1 Rules first	Done (0.3 min)	Partly: one rule (3.9, goods that never leave Germany) left out	Done, re-read three times	Read many times, never applied
2 Readable extracts	Done	Done (5 min)	Done (2 min)	Only after 94 min
3 Tie out the filed returns	Gap diagnosed, tie-out not built	Missed: “start from the filed amounts and add the delta”	Done, late (87–111 min)	Not reached
4 Linking per era	Partly: 156 documents never linked	Partly: the same 156	ECC and early S/4 only; 0 of 1,072 cockpit documents	Not reached
5 What is not a chain	Partly: 162 assembly documents kept	Partly: 162 kept	Never reached the cockpit era	Not reached
6 Classify	90.1% right	90.6% right	1,860 documents, precision 0.94	Not reached

Table 11

Harness	Total	Register (0.20)	Recon (0.22)	Correction (0.12)	Tool (0.20)	Memo (0.08)	Traps (0.06)	Format (0.12)
OpenCode	0.542	0.81	0.30	0.00	0.88	0.24	0.00	1.00
micro-cc 0.2.124	0.528	0.81	0.13	0.00	0.89	0.50	0.00	1.00
Codex	0.00	0	0	0	0	0	0	0
Claude Code	0.00	0	0	0	0	0	0	0

Table 12

Figure 9: Figure 6a. micro-cc on DeepSeek: the chain register, with a mis-filed cockpit chain highlighted (filed as intra-EU/triangular, correct as domestic taxable/intra-EU). The classification layer works: usable as a triage list.

Figure 10: Figure 6b. micro-cc on DeepSeek: the DE01 reconciliation, differences marked against the answer key (green matches to the cent, red is wrong). Invented differences in quiet months; the July 2022 double posting never removed.

Figure 11: Figure 7. Same harness, only the model changed. micro-cc 0.2.124 on DeepSeek v4.1 Flash (left: EUR 2,069,077.69, 33% short, with invented 2021–23 refunds) and on Claude Opus 5.5 (right: EUR 3,093,692.38, exact to the cent). The answer key underneath comes from the task generator, not from micro-cc.

Figure 12: Figure 8. The DE01 reconciliation in both runs against the answer key. DeepSeek adds chain-list deltas to the filed figure, and every difference is wrong; Opus recomputes every line from the posted tax lines, and every difference matches — including the July 2022 double posting ( $-$ 44,209.33) and the FX residue of a quiet month ( $-$ 11.95).

Run	Score	Wall time	Model calls	Tool calls	Input tokens	Output tokens	Tool output fed back	Cost	Harness memory
OpenCode $\cdot$ DeepSeek	0.542	74 min (stopped itself)	83	86	14.5M (97% cached)	52k	291k chars	$0.256 (exact)	705 MB
micro-cc $\cdot$ DeepSeek	0.528	120 min (limit)	186	186	45.7M	502k	277k chars	~ $2 (est.,$ 1.2–2.7)	147 MB
Codex $\cdot$ DeepSeek	0.00	121 min (limit)	167	333	65.7M (87% cached)	477k	3.75M chars	~$3.3 (est.)	486 MB
Claude Code $\cdot$ DeepSeek	0.00	121 min (limit)	358	445	164.0M (96% cached)	180k	6.78M chars	~$3.0 (est.)	343 MB
micro-cc $\cdot$ Opus 5.5	0.9986	22 min (stopped itself)	74	75	10.76M	117k	302k chars	$5.80 (exact)	148 MB
Claude Code $\cdot$ Opus 5.5	1.000	24.6 min (stopped itself)	79	78	11.73M	117k	1.05M chars	$5.89 (exact)	288 MB

Table 17

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,503 papers

What Does a Harness Buy? Tokens, Mostly
Oct 3, 2026Yangze Liu, Zhongyi HanLLM Agent Efficiency LLM Agent Harnesses
Shandong University
Same Model, Different Harness: Different Coding-Agent Results
Aug 26, 2026Sydney LewisAutomated Agent Evaluation LLM Agent Harnesses
Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents
Oct 14, 2026Yuyao Ge, Shenghua Liu, Peizhong Gao +8LLM Agent Harnesses AI Agent Benchmarks
Institute of Computing Technology, Chinese Academy of Sciences · Tsinghua University · Peking University +1

Oct 3, 2026J/K move · Enter open · S save

What Does a Harness Buy? Tokens, Mostly

Yangze Liu, Zhongyi Han

Shandong University

LLM Agent Efficiency LLM Agent Harnesses AI Coding Agents LLM Agent Evaluation LLM Agents Coding Agent Benchmarks

Open paper AskPDF arXiv

A coding agent is a language model wrapped in a harness: the system prompt, the tool set, and the context management that turn a chat model into something that can work inside a repository. Production harnesses ship releases daily, vendors advertise pass-rate gains from harness changes, and leaderboards mix harnesses freely. What is rarely measured is how much the harness itself moves the score when the model is held fixed. We run five models through three production harnesses, Claude Code, mini-SWE-agent, and OpenCode, on SWE-bench Verified, and rerun the same configurations to calibrate how much a score moves when nothing changes but the run. On 447 tasks and the two models we ran there, Claude Code and mini-SWE-agent, the heaviest and the lightest harness, are equivalent within five points. On a 45-task hard subset and five models, swapping the harness flips as many tasks as rerunning the same harness, 13% in both cases, and the tasks a harness wins in one run are not the tasks it wins in the next. The one harness effect that clears the noise is a loss, not a gain: OpenCode trails by up to 9 points on the large pool, and on one model half of that gap sits in runs its output cap cut short. What the harness does decide is the bill. With the same model, the same tasks, and one price list, cost per task differs by up to 3x across harnesses. The gap is set at the first call, by the preamble of system prompt and tool schemas each harness sends with every step, and scaled by the number of steps; per-step growth and per-call tool output differ far less. The provider's price for cached input scales the bill and does not reorder it. The rerun data also give the resolution a harness comparison needs: at the discordance we observe, 45 tasks catch a 13-point gap only half the time and no gap with 80% power, and 447 tasks resolve 5 points, still coarser than the gains many harness changes claim.

Abstract

AI Agent Benchmarks

Explore similar work

What Does a Harness Buy? Tokens, Mostly

Same Model, Different Harness: Different Coding-Agent Results

Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents