Veritas: A Verifier Generation Harness for Agent Tasks
Organizations: ByteDance Inc., USA · University of Illinois Chicago · New York University
Abstract
The effectiveness of a self-improving system relies heavily on the quality of its verifier, which judges the outcome of each improvement round. However, the verifiers of most tasks are written by hand, which is expensive and does not scale to tasks that come without one. We propose Veritas, a harness that generates verifiers automatically. Unlike general-purpose agent harnesses, Veritas adds three mechanisms for verifier generation: an evidence header stating what the verifier will observe, a library of domain rules that tasks leave unstated, and authoring principles, partly refined by a self-improvement loop. We evaluate on six benchmarks: four scored by execution, whose official verifiers we first certify, and two that compare pairs of responses. With the model fixed, the verifiers Veritas generates agree better with the official verdicts than those of seven other harnesses on all four execution-scored benchmarks. On the two pairwise benchmarks they are at or near the best. They also agree better than those of the vendor's own harness for two of three other model families. Its mechanisms also improve 9 of 12 harness-benchmark pairs scored by execution across three harnesses.
Figures & tables
| Benchmark | Harness | Concretely | |
|---|---|---|---|
| -bench | messages, DB before, DB after | ||
| SWE-bench Verified | patched repository; runs its tests inside it | ||
| Terminal-Bench 4.0 | final-state filesystem | ||
| TB-Science 0.1 | final-state filesystem | ||
| LLMBar, JudgeBench | the two responses |
| plain harnesses | ||||||||
| Benchmark ( tasks) | Codex | OpenCode | pi | Terminus 2 | Hermes | Claude Code | Gemini CLI | Veritas |
| -bench (58) | 0.688 | 0.563 | 0.238 | 0.185 | 0.287 | 0.726 | 0.164 | 0.787 |
| SWE-bench Verified (138) | 0.599 | 0.678 | 0.693 | 0.518 | 0.503 | 0.557 | 0.396 | 0.720 |
| Terminal-Bench (33) | 0.387 | 0.336 | 0.048 | 0.087 | 0.238 | 0.022 | 0.465 | |
| TB-Science (39) | 0.351 | 0.474 | 0.165 | 0.023 | 0.144 | 0.084 | 0.548 | |
| LLMBar (75 pairs; acc.) | 0.920 | 0.760 | 0.893 | 0.933 | 0.893 | 0.827 | 0.840 | 0.920 |
| Codex | Claude Code | |||||
| plain | + ERH | plain | + ERH | |||
| -bench | 0.688 | 0.751 | 0.726 | 0.688 | ||
| SWE | 0.599 | 0.640 | 0.557 | 0.524 | ||
| TB | 0.387 | 0.337 | 0.238 | 0.586 | ||
| TB-Sci | 0.351 | 0.533 | 0.144 | 0.465 | ||
| LLMBar | 0.920 | 0.893 | 0.827 | 0.880 | ||
| Full | E | R | H | Baseline | |
|---|---|---|---|---|---|
| -bench | 0.787 | 0.726 | 0.472 | 0.250 | 0.563 |
| SWE | 0.741 | 0.579 | 0.741 | 0.709 | 0.678 |
| LLMBar | 0.920 | 0.880 | 0.920 | 0.893 | 0.760 |
| JudgeBench | 0.980 | 0.961 | 0.961 | 0.980 | 0.980 |
| Harness | SWE | TB | TB-Sci | LLMB. | JB | |
|---|---|---|---|---|---|---|
| GPT-5.6-sol | ||||||
| Veritas | — | 0.692 | 0.277 | 0.423 | 0.933 | 0.980 |
| Codex CLI | — | 0.720 | 0.311 | 0.540 | 0.933 | 0.980 |
| Qwen3.7-Plus | ||||||
| Veritas | 0.546 | 0.554 | 0.680 | 0.712 | 0.893 | 0.941 |
| Qwen Code | 0.376 | 0.542 | 0.508 | 0.514 | 0.800 | 0.961 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Phase | Rounds | Proposed by | What was edited | Accepted rounds |
|---|---|---|---|---|
| 1 | 1–11 | model | add a principle to H retry or repeat generation change timeouts or concurrency | 3 and 4 (two principles in H) |
| 2 | 12–25 | researchers | change the instructions to the agent add a delivery safeguard change the instructions and delivery together change scoring on DeepSWE | 15 (MCP-Atlas only) 25 (DeepSWE only) |
| 3 | 26–32 | researchers ∗ | change a principle in H test the verifier on synthetic wrong candidates declare the evidence each check reads split the verifier into separate assertions | none |
| oracle | no-op | trunc | linedrop | C1 | C2 | C3 | C4 | C5 | C6 | all derived | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Terminal-Bench | 33/33 | 0/33 | 9/33 | 15/32 | 4/12 | 4/13 | 3/7 | 6/7 | 3/29 | — | 44/133 |
| TB-Science | 39/39 | 0/39 | 20/39 | 13/28 | 5/19 | 0/20 | 0/2 | 5/8 | 0/11 | 3/19 | 46/146 |
| Benchmark | Official verifier | Tasks | Confirmed by execution |
|---|---|---|---|
| Agents’ Last Exam [ 56 ] | deterministic evaluator per task | 82 | 27 tasks accept a wrong candidate or reject a correct one |
| -bench [ 7 , 55 ] | state comparison + expected actions | 146 | 57 tasks lack full power: 33 reject the no-op but accept a near-miss; 24 more pass the no-op on the database comparison |
| DeepSWE [ 24 ] | hand-written checks per task | 106 | 36 tasks give full marks to the gold patch minus one hunk |
| MCP-Atlas [ 6 ] | one shared model judge | 500 | 72 tasks change verdict on identical input; 47 reject a correct paraphrase |
| Terminal-Bench 4.0 [ 36 ] | the task’s tests, run on the final state | 66 | 6 tasks accept a corrupted output that adjudication found defective |
| TB-Science 0.1 [ 60 ] | the task’s tests, run on the final state | 70 | 3 tasks accept a corrupted output that adjudication found defective |
| plain harnesses | + ERH | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Codex | OpenCode | pi | Terminus 2 | Hermes | Claude Code | Gemini CLI | Veritas | Codex | Claude Code | |
| oracle TPR | 0.36 | 0.36 | 0.39 | 0.30 | 0.30 | 0.42 | 0.30 | 0.45 | 0.36 | 0.52 |
| no-op TNR | 0.94 | 0.94 | 0.64 | 0.73 | 0.82 | 0.82 | 0.70 | 0.97 | 0.94 | 1.00 |
| cut or corrupted TNR | 0.91 | 0.89 | 0.63 | 0.70 | 0.73 | 0.79 | 0.65 | 0.89 | 0.92 | 0.94 |
| Candidate | Official | Plain OpenCode | Veritas |
|---|---|---|---|
| agent run | PASS | PASS | PASS |
| agent run without the verification call | FAIL | PASS | FAIL |
| empty | FAIL | FAIL | FAIL |
| Candidate | Official | Plain OpenCode | Veritas |
|---|---|---|---|
| oracle | PASS | PASS | PASS |
| reference script cut to three quarters | PASS | PASS | PASS |
| reference script without its last line | PASS | PASS | PASS |
| one comparison flipped in the solution | PASS | PASS | PASS |
| no-op (placeholder left in place) | FAIL | PASS | FAIL |
| solution file emptied | FAIL | FAIL | FAIL |
| Candidate | Official | Plain Claude Code | Claude Code + ERH |
|---|---|---|---|
| agent run (one free bag) | PASS | FAIL | PASS |
| agent run without the booking call | FAIL | FAIL | FAIL |
| empty | FAIL | FAIL | FAIL |
| Qwen3.7-Plus | Gemini 3.5 Flash | ||||
|---|---|---|---|---|---|
| Candidate | Official | Qwen Code | Veritas | Gemini CLI | Veritas |
| oracle | PASS | FAIL | PASS | FAIL | PASS |
| no-op | FAIL | FAIL | FAIL | FAIL | FAIL |
| reference script cut to three quarters | FAIL | FAIL | FAIL | FAIL | FAIL |
| reference script without its last line | FAIL | FAIL | FAIL | FAIL | FAIL |
| one number scaled by 1.1 | FAIL | FAIL | FAIL | FAIL | FAIL |
LLM Agent Verification
LLM agent verification checks whether agents’ actions, evidence, policies, and completion claims satisfy explicit requirements.
See all 170 papers