Anatomy of an Agent Harness: Decisions, Invariants and Corner Cases
Abstract
An agent harness is the program around a language model that turns its replies into actions: it builds each request, runs the tools the model asks for, decides what the model may do without asking, stores the conversation, recovers from failures, and shows a person what happened. Most of these decisions are made in code and rarely written down, so each harness meets the same corner cases again. This paper writes them down as invariants to test. It gives a system model with trust boundaries; 30 decision cards in six groups, each with options and costs, an invariant, the corner cases that break it, a test and an evidence tag; six cause classes for failures; a test method built on scripted providers, recorded behaviour and planted changes; and a checklist. The cards come from building one desktop harness, Crewroom, the author's own, and from manually testing ten other harnesses and studying open-source harnesses as reference; some patterns in the cards were adopted from those, so the catalogue is not independent of them. Of the 83 invariant clauses, 56 are tagged Tested, 12 Believed and 15 Open. The checklist is applied to the author's harness as one worked example, by its builder with no second rater: of its 30 items, 11 hold, 14 do not and 5 were not tested. It is not a ranking. The paper makes no claims about answer quality or cost or other platforms, and does not evaluate or rank other harnesses.
Figures & tables
| Card | Decision | Core clause, short | Checked by | Tag | Verdict |
|---|---|---|---|---|---|
| T1 | How a turn ends | One recorded end reason per turn | script | Tested | N |
| T2 | Every tool call ends exactly once | Each started call shown once as ended | script | Tested | H |
| T3 | Early start, discarded attempts | Only read-only calls start early | script | Tested | N |
| T4 | Arrival order, reasoning, what is shown | Streamed text stays, once, in order | script | Tested | H |
| T5 | Tool results and long-running tools | Results non-empty, bounded, kept | script | Tested | N |
| F1 | Retry budgets that do not multiply | Each failure episode bounded | script, real | Tested | N |
| Cause class | What it is | Cards | Cheapest detection |
|---|---|---|---|
| Two readers | Two components read one thing differently: checker and shell, prompt and tool list, timer and record | T1, T4, T5, F3, C1, C3, S5, P2, P3, P6, V1 | Feed both readers the same generated inputs and compare |
| Unlisted path | An exit, caller, tool, key or output mode that no list enumerates, so no rule reaches it | T1, T3, T5, F2, C3, S2, P4 | Walk the registry or the syntax tree; one test per entry |
| Untrue text | Something the model or person is told that is false, often a guess shown as fact | T3, F2, F3, F4, F5, C1, S2, S6, P7, V2 | Assert exact texts per cause; scan for guessed values |
| Loose state | A counter, clock, queue or id kept in several places, matched by position, or not reset | T2, F1, C2, S1, S3, S6, V1 | Fake clock; count requests and rows; kill and restart |
| Borrowed authority | A turn, file, link or agent acting with rights it did not get from the person | P1, P3, P5, P6, P8, P9 | Plant the widening input; assert nothing changes |
| Blind test | A test that cannot fail, is not run, or runs a mode no real use reaches | F4, S4, P4 | Plant changes; count tests discovered against tests run |
| Fault injected | What our harness did | Wall time, s | As card? |
|---|---|---|---|
| None | One request, reply | 1.7, 1.3, 1.5 | yes |
| Cut after 2 events | Retry row “connection dropped, trying again in 1 s (1 of 10)”, second request, reply | 2.3, 3.9, 5.1 | yes (F1) |
| First byte delayed 30 s | Waiting line at 20 s naming the reconnect time, cleared when output came; one request | 31.1, 31.3, 31.3 | yes (F2), but its countdown rose |
| Stall of 60 s after 1 event | Waiting line at 21 s, cleared at 61 s; one request (60 s is under the 300 s silence limit) | 61.5, 61.3, 61.5 | yes (F2) |
| Second request stalls 60 s after a tool call | Tool row, waiting line at about 22 s, cleared; reply | 61.7, 62.8, 62.4 | yes (F2) |
| Overload 3 times | Retry row; 4 requests; a wait of 17 to 18 s after the third; reply | 19.2, 19.4, 20.2 | yes (F1) |
| Group | Size | Records | Made comparable by |
|---|---|---|---|
| Fault rows, in the same process | 150 rows; 187 row-and-wire runs, plus one test that every replaced global is restored | Every event, every request (method, path, version headers, body), the saved record, the final message list, any error | A virtual clock for dates, timers and intervals; random numbers fixed at one half; fresh temporary home and folder; one time zone; the network call replaced by the scripted provider |
| Parity | 65 scenarios | Requests sent, saved record, tool results, terminal text | A scripted provider replaying each scenario; a normaliser for paths, ids, times, durations, system strings, commit ids, image bytes and process-based ids |
| Parity over the pipe | 41 of those scenarios (those a pipe can drive) | The same, plus the frames exchanged, in order | The same normaliser; its own baselines |
| Pipe traces | 40 situations | Every frame sent and received, saved records, exit code and signal | Timed actions wait for a named frame, never a sleep; parallel frames sorted into call order |
| Card | Verdict |
|---|---|
| T1 | N : a stream with no finish reason shown and stored as complete, exit 0; step limit named |
| T2 | H : two calls, the first failing: ids and order kept |
| T3 | N : a stream with no finish reason was shown and stored as complete, exit 0 (clause (b) Open); nothing ran from a cut reply and the cut text was not in the output (3 of 3) |
| T4 | H : reasoning not shown |
| T5 | N : a timed-out command ran on and finished; result “timed out” after 45 s. Empty and large output held |
| F1 | N : 18 requests in six bursts of three, the request layer inside each of six rounds; the stop time shown is not the bound that ends it |
| Event | Events | Chats |
|---|---|---|
| Turn completed | 320 | 160 |
| Turn stopped by the person | 40 | 36 |
| Request retried, all labels | 38 | 15 |
| Switch to a backup model | 4 | 4 |
| Model call ended aborted | 10 | 9 |
| Model call ended in error | 13 | 5 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| # | Ending | Cause | What the model is told | What the person sees |
|---|---|---|---|---|
| 1 | Finished | The tool ran | Its output, never empty (say “no output”), bounded; the full text kept retrievable | Done, with the result |
| 2 | Failed | The tool ran and raised or reported an error | The error text | Failed, with the text |
| 3 | Malformed call | Arguments unparseable, unknown tool, or tool not yet loaded | That nothing ran, why, and how to send it again | Failed row |
| 4 | Refused by a limit | A run-level rail (calls, time, spend) | That it did not run, and to wrap up with what it has | Failed row naming the limit |
| 5 | Refused by an extension | A hook or plug-in vetoed it | The veto text; a veto may also halt the whole run | Failed row; a stop notice if halted |
| 6 | Refused by mode | The current mode does not allow it | That permission was denied, other tools may be tried, and the person decides | Failed row |
| # | Failure | How recognised | What the harness does | What the person reads |
|---|---|---|---|---|
| 1 | Bad credentials | Status 401 or 403 | Never retry; refresh a login token once, re-key once; then stop | Which credential is bad and where to fix it |
| 2 | Model not allowed | 403 or 400 naming the plan or model | Never retry | The model is not on the plan; pick another |
| 3 | Bad request | 400 | Never resend the same bytes; one repair per known cause; resend once if the error names a limit | The provider’s words |
| 4 | Unprocessable payload | 413 or 422; image size | Shrink or strip once; then stop | What was too large |
| 5 | Context too long | The error names the context limit | Summarise and resend, at most three times | A final “too long, start fresh” line |
| 6 | Output cap too large | The error names a limit | Resend once with the stated limit | Nothing unless it recurs |
LLM Agent Harnesses
LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.
See all 428 papers