Contain, Conform, Compare (Extended Version): A Red-Teamed Evaluation Kit for Small Coding Agents and the Full Record of Its Experiments
Organizations: VNU University of Science, 334 Nguyen Trai, Thanh Xuan, Hanoi, Vietnam
Abstract
A hosted score in the Gemma 4 Developer Agent competition is one noisy number, so development needs a local evaluator that is fast, safe for agent-written code, and matching the hosted scorer's prompts. drydock-eval evaluates a small coding agent (Gemma 4 E4B) at about 250 runs per hour on one RTX 5090. Contain runs each job in a fresh container without network and passed 200 scripted escape tests; conform provides eight checks that compare a local stack's requests with the scorer's configuration, down to rendered tokens; compare provides paired statistics that respect repository clusters. AI agents built and attacked the kit; a human owner approved machine use and release. This extended version repeats the short paper and adds the full record of the experiments made with and around the kit: 9,660 rows of agent runs, 2,494 grading controls and 72 serving measurements, all released. On four L4 GPUs the 31B model decodes 38.1 tokens per second in one stream and E4B 105.8. Six configurations of a 31B agent solved between 37 and 44 of 70 development tasks, and no pairwise difference was detected. A pre-registered five-arm experiment (231 tasks, 4,620 jobs) measured four prompt divergences: no E4B solve-rate change was detected (largest estimate -2.1 points, 95% interval -4.4 to +0.4). Three changes of the E4B agent led their screens by +1.0 to +6.2 points; each was then tested once on new tasks under a frozen pass rule, and none was confirmed (gains -0.35, -0.39 and +4.10 points). The unchanged baseline, run again on the same 128 tasks, moved by 1.17 points, and 65% of these tasks were solved in none of eight runs. Prompts and agent bundles are withheld; every run-level result is public.
Figures & tables
| Model | Context | Streams | Prefill tok/s | Decode tok/s per stream | Decode tok/s, all streams |
|---|---|---|---|---|---|
| 31B | 2,048 | 1 | 1,539 | 38.1 | 38 |
| 31B | 2,048 | 2 | 1,545 | 34.4 | 66 |
| 31B | 2,048 | 4 | 1,546 | 31.2 | 122 |
| 31B | 2,048 | 8 | 1,550 | 17.9 | 78 |
| 31B | 2,048 | 16 | 1,548 | 18.2 | 279 |
| 31B | 8,192 | 1 | 1,479 | 36.3 | 36 |
| Server | KV cache, tokens | 2k context, 1 stream | 8k, 1 stream | 8k, 4 streams |
|---|---|---|---|---|
| no adapter (control) | 46,352 | 38.3 | 36.4 | 29.4 |
| rank 16, attention projections | 37,328 | 33.9 ( 12%) | 32.4 ( 11%) | 26.9 ( 9%) |
| rank 16, all linear layers | 37,328 | 33.8 ( 12%) | 32.5 ( 11%) | 26.9 ( 9%) |
| rank 8, two module types | 37,568 | 33.3 ( 13%) | 32.1 ( 12%) | 26.7 ( 9%) |
| Runner version | Canaries passed | Security findings of independent verifiers |
|---|---|---|
| 1. Container per command | 52/52 | 4 high: host-side install ran the agent’s setup.py (builder-found); symlink leaked a host file; 10 GiB of host disk reserved in 0.005 s; host copied a sparse file in full |
| 2. Hardened, per command | 60/60 (57/60 before canary fixes) | 4 high, 1 untested: host overwrote a file through a planted link; cleanup followed links; 6 GiB reserved in 0.012 s; 200,000 files in 2.4 s |
| 3. Container per job | 98/98 | 1 high: the model server could fetch URLs named in requests; 2 medium |
| 4–6. Request filter added, tightened | 134/134, 159/159, 191/191 | medium at most |
| 7. Final | 200/200 | no high or medium |
| Check | Compares | Mutation runs caught |
|---|---|---|
| role | role of tool results | 53/53 |
| sampling | temperature, top-p, token and thinking budgets | 52/52 |
| tools | declared tool schemas | 20/20 |
| shape | message skeleton, system text, field encoding | 72/72 |
| template | rendered prompt against the Kaggle template | 13/21 by token count, 24/24 by token id |
| thoughts | history thoughts, under the key the server reads | 52/52 |
| Simulated scenario | Small panel | Large panel |
|---|---|---|
| No effect, exchangeable tasks | 0.23–1.90% | 0.00–0.50% |
| No mean effect, candidate breaks 20% of repositories | 7.50% | 8.43% |
| No effect, worst tasks dropped after seeing outcomes | 31.75% | 40.75% |
| Even gain of 3/58 (power) | 60% | 90% |
| Gain confined to four repositories (power) | 15.9% | 24.9% |
| Block | Arm | Tasks | Solved | % | 95% interval | Wall | Out | Calls | T/O | Empty | Sum. |
|---|---|---|---|---|---|---|---|---|---|---|---|
| pub31-1 | BF5 | 30 | 17 | 57 | 39–73 | 303 | 6,893 | 24.2 | 5 | 1 | 18 |
| pub31-1 | BF5-p | 30 | 18 | 60 | 42–75 | 296 | 6,921 | 23.1 | 4 | 4 | 19 |
| pub31-1 | W1 | 30 | 20 | 67 | 49–81 | 254 | 5,942 | 22.4 | 3 | 4 | 0 |
| pub31-2 | W1-b | 30 | 17 | 57 | 39–73 | 290 | 6,394 | 23.4 | 7 | 7 | 0 |
| pub31-3 | W1-p | 30 | 15 | 50 | 33–67 | 304 | 6,412 | 25.2 | 11 | 12 | 0 |
| pub31-4 | W3 | 30 | 16 | 53 | 36–70 | 272 | 7,448 | 30.6 | 4 | 6 | 0 |
| Configuration | First 30 | Next 40 | All 70 | % | 95% interval | Only it | Only BF5 | |
|---|---|---|---|---|---|---|---|---|
| BF5 | 17 | 20 | 37 | 53 | 41–64 | — | — | — |
| W1/W2 | 20 | 19 | 39 | 56 | 44–67 | 6 | 4 | 0.75 |
| W3 | 16 | 22 | 38 | 54 | 43–65 | 6 | 5 | 1.00 |
| SH1/SH3 | 20 | 21 | 41 | 59 | 47–69 | 8 | 4 | 0.39 |
| SH5 | 17 | 24 | 41 | 59 | 47–69 | 7 | 3 | 0.34 |
| SH6 | 21 | 23 | 44 | 63 | 51–73 | 10 | 3 | 0.09 |
| Arm | Standard | Masked | Only masked | Only standard |
|---|---|---|---|---|
| BF5 | 17 | 19 | 2 | 0 |
| SH5 | 17 | 17 | 2 | 2 |
| Arm | Tasks | Solved, L4 | Solved, emul. | Both | Only L4 | Only emul. | Wall, L4 | Wall, emul. | Diff. |
|---|---|---|---|---|---|---|---|---|---|
| SH5 | 10 | 6 | 3 | 3 | 3 | 0 | 280 | 292 | 4% |
| SH6 | 24 | 14 | 17 | 14 | 0 | 3 | 289 | 314 | 8% |
| Arm | Solved | Agent | Setup | Verify | Tokens out | Tool calls |
|---|---|---|---|---|---|---|
| BF5 | 4/10 | 191 | 2.9 | 10.5 | 6,454 | 18.5 |
| BF5-s | 4/10 | 233 | 4.8 | 10.1 | 7,710 | 19.8 |
| ID24 | 4/10 | 193 | 3.8 | 10.7 | 6,551 | 16.7 |
| ID24-r | 4/10 | 179 | 4.0 | 7.8 | 5,792 | 17.1 |
| Date (UTC) | Arm | Model | Public score | Local result of the same arm |
|---|---|---|---|---|
| 2026-11-05 | W3 | 31B | 0.12 | 38 of 70 development tasks (emulated latency) |
| 2026-11-08 | ID24 | 31B | 0.17 | 4 of 10 Berth tasks (4 L4) |
| 2026-11-09 | base | E4B | 0.06 | 20.2% of 1,616 Berth runs (workstation) |
| 2026-11-10 | BF5 | 31B | 0.18 | 37 of 70 development tasks (emulated latency) |
| 2026-11-11 | v-a3 | E4B | 0.05 | 118 of 512 Berth runs (panel H) |
| Block | Arm | Tasks | Runs | Solved | % | 95% interval | Time | Tool calls |
|---|---|---|---|---|---|---|---|---|
| pubE4-1 | e-single | 24 | 24 | 5 | 21 | 9–40 | 124 | 8.7 |
| pubE4-1 | e-wrap | 24 | 24 | 5 | 21 | 9–40 | 108 | 10.3 |
| pubE4-1 | e-wrap-ad | 24 | 24 | 5 | 21 | 9–40 | 118 | 10.5 |
| berthE4-k1 | e-BF | 24 | 48 | 4 | 8 | 3–20 | 70 | 9.3 |
| berthE4-k1 | base | 24 | 48 | 8 | 17 | 9–30 | 79 | 11.9 |
| berthE4-k2 | base | 8 | 16 | 7 | 44 | 23–67 | 82 | 12.4 |
| Tasks and machine | Reference patch passes | Empty patch passes |
|---|---|---|
| whole pool, workstation, no container | 580/580 | 1/580 |
| panels C and H, final runner | 200/200 | 0/200 |
| screen set S, final runner | 48/48 | 0/48 |
| case-study panel and two tasks of S, final runner | 233/233 | 0/233 |
| hosted notebook, first batch | 38/38 | 0/38 |
| hosted notebook, second batch | 87/87 | 0/209 |
| Run | Solved | 95% interval, % | Agent |
|---|---|---|---|
| hosted 4 L4, repeat 1 | 4/24 | 7–36 | 79 |
| hosted 4 L4, repeat 2 | 4/24 | 7–36 | 80 |
| workstation, no sandbox, 8 workers | 3/24 | 4–31 | 65 |
| workstation, no sandbox, 1 worker | 5/24 | 9–40 | 63 |
| workstation, container runner, version 1 | 3/24 | 4–31 | 69 |
| workstation, container runner, version 3 | 2/24 | 2–26 | 58 |
| Task set | Sessions | Solved runs | % | Range | Same | Never | Always | |
|---|---|---|---|---|---|---|---|---|
| S (48 tasks, 2 repeats) | 3 | 19, 21, 20 | 19.8, 21.9, 20.8 | 2.1 | 39 | 34 | 5 | 0.91 |
| H (128 tasks, 4 repeats) | 2 | 103, 97 | 20.1, 18.9 | 1.2 | 99 | 83 | 10 | 0.92 |
| Repository | Tasks | Runs | Solved | % | 95% interval |
|---|---|---|---|---|---|
| executablebooks/markdown-it-py | 6 | 36 | 19 | 52.8 | 37–68 |
| Kludex/starlette | 1 | 6 | 3 | 50.0 | 19–81 |
| jd/tenacity | 15 | 98 | 33 | 33.7 | 25–43 |
| mahmoud/boltons | 17 | 118 | 38 | 32.2 | 24–41 |
| pypa/packaging | 24 | 128 | 36 | 28.1 | 21–36 |
| marshmallow-code/marshmallow | 8 | 56 | 15 | 26.8 | 17–40 |
| Setting | Runs | Solved % | Mean | Median | Solved | Unsolved | Out | In (k) | Tools | T/O | Budget | Empty |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 31B, development tasks, emulated latency | 570 | 57.2 | 278 | 255 | 231 | 341 | 7,089 | 240 | 20.6 | 12.6 | 0.4 | 13.5 |
| 31B, development tasks, 4 L4 | 36 | 58.3 | 286 | 256 | 234 | 358 | 8,515 | 264 | 23.8 | 11.1 | 2.8 | 11.1 |
| 31B, Berth, 4 L4 | 104 | 38.5 | 226 | 212 | 165 | 265 | 7,495 | 196 | 19.2 | 8.7 | 0.0 | 4.8 |
| E4B, development tasks, 4 L4 | 72 | 20.8 | 117 | 106 | 85 | 125 | 4,665 | 102 | 9.8 | 1.4 | 0.0 | 4.2 |
| E4B, Berth, 4 L4 | 160 | 20.0 | 73 | 61 | 43 | 80 | 6,261 | 110 | 10.6 | 0.0 | 1.9 | 11.9 |
| E4B, Berth, workstation | 4,082 | 21.3 | 75 | 60 | 45 | 84 | 8,146 | 156 | 14.2 | 0.3 | 1.2 | 5.5 |
| Arm | Difference from the scorer path | Requests changed by the divergence | Change against F [95% interval]; Holm p |
|---|---|---|---|
| F | none: role tool , Kaggle template, compaction at 14,336 tokens, thoughts under both keys | — | solve rate 23.8% |
| R | model name contains “gemma4”: tool results get role tool_responses | 89% | 0.3 [ 2.8, +2.2]; p = 1 |
| C | compaction unset: prompts grow until rejected above 24,576 tokens (the run ends; edits are still graded) | 21% | +0.2 [ 2.0, +2.5]; p = 1 |
| H | history thoughts removed from every request, as when a client omits the key the server reads | 90% | 2.1 [ 4.4, +0.4]; p = 0.44 |
| T | the Hugging Face E4B template: a thought opener is forced after each tool result | 85% | +0.8 [ 1.4, +2.9]; p = 1 |
| Batch | Jobs started | Analysed runs | Interrupted, row written | Interrupted, no row | How the batch ended |
|---|---|---|---|---|---|
| 1 | 967 | 959 | 0 | 8 | runner reached its open-file limit |
| 2 | 378 | 370 | 8 | 0 | external termination signal |
| 3 | 3,074 | 3,066 | 8 | 0 | model server ended (out of memory) |
| 4 | 201 | 201 | 0 | 0 | job list completed |
| 5 | 24 | 24 | 0 | 0 | re-runs of the interrupted jobs completed |
| Arm | Solved | Unsolved, run completed | Five-minute limit | Over-long prompt rejected | Tool-call budget (28) | No submission | Runs |
|---|---|---|---|---|---|---|---|
| F | 220 | 676 | 17 | 0 | 6 | 5 | 924 |
| R | 217 | 677 | 20 | 0 | 9 | 1 | 924 |
| C | 222 | 568 | 17 | 114 | 1 | 2 | 924 |
| H | 201 | 597 | 106 | 1 | 18 | 1 | 924 |
| T | 227 | 668 | 19 | 0 | 4 | 6 | 924 |
| Arm | Solve rate (%) | 95% interval | Solved runs | Repeat 1 | 2 | 3 | 4 | Runs |
|---|---|---|---|---|---|---|---|---|
| F | 23.81 | [16.7, 30.9] | 220 | 50 | 59 | 57 | 54 | 924 |
| R | 23.48 | [17.7, 29.3] | 217 | 53 | 53 | 54 | 57 | 924 |
| C | 24.03 | [16.6, 31.5] | 222 | 58 | 56 | 55 | 53 | 924 |
| H | 21.75 | [15.9, 27.6] | 201 | 50 | 49 | 54 | 48 | 924 |
| T | 24.57 | [18.6, 30.5] | 227 | 59 | 56 | 51 | 61 | 924 |
| Arm | 0 solved | 1 solved | 2 solved | 3 solved | 4 solved | Tasks |
|---|---|---|---|---|---|---|
| F | 149 | 16 | 21 | 18 | 27 | 231 |
| R | 152 | 14 | 21 | 15 | 29 | 231 |
| C | 151 | 15 | 18 | 17 | 30 | 231 |
| H | 155 | 21 | 13 | 14 | 28 | 231 |
| T | 151 | 14 | 13 | 25 | 28 | 231 |
| Arm | Exact | Tasks lower; higher | Exact | Holm | Monte Carlo | 95% interval | 98.75% simultaneous | |
|---|---|---|---|---|---|---|---|---|
| R | 26; 33 | 0.8652 | 1.000 | 0.8650 | [ , ] | [ , ] | ||
| C | 26; 22 | 0.9253 | 1.000 | 0.9243 | [ , ] | [ , ] | ||
| H | 34; 20 | 0.1102 | 0.441 | 0.1106 | [ , ] | [ , ] | ||
| T | 25; 30 | 0.5604 | 1.000 | 0.5599 | [ , ] | [ , ] |
| Arm | Repositories that differ | Exact | Webb | 95% interval | CR1 interval | Repository-equal | Leave-one-out range |
|---|---|---|---|---|---|---|---|
| R | 12 | 0.8188 | 0.7780 | [ , ] | [ , ] | [ , ] | |
| C | 8 | 0.8438 | 0.7788 | [ , ] | [ , ] | [ , ] | |
| H | 12 | 0.1831 | 0.1736 | [ , ] | [ , ] | [ , ] | |
| T | 9 | 0.8047 | 0.7374 | [ , ] | [ , ] | [ , ] |
| Repository | Tasks | F solved runs | R | C | H | T |
|---|---|---|---|---|---|---|
| pypa/packaging | 65 | 65 of 260 | ||||
| pydantic/pydantic-settings | 48 | 27 of 192 | 0 | |||
| python-poetry/tomlkit | 35 | 31 of 140 | 0 | |||
| more-itertools/more-itertools | 23 | 39 of 92 | 0 | |||
| pallets/click | 22 | 13 of 88 | ||||
| pallets/werkzeug | 9 | 3 of 36 | 0 |
| Arm | Analysis | Tasks | Runs affected | Exact | |
|---|---|---|---|---|---|
| R | five-minute-limit runs of both arms treated as missing | 229 | 37 | 0.8652 | |
| slots of the arm with an infrastructure row scored unsolved | 231 | 5 | 0.8011 | ||
| slots of F with an infrastructure row scored unsolved | 231 | 3 | 0.9321 | ||
| C | five-minute-limit runs of both arms treated as missing | 229 | 34 | 0.9253 | |
| slots of the arm with an infrastructure row scored unsolved | 231 | 5 | 1.0000 | ||
| slots of F with an infrastructure row scored unsolved | 231 | 3 | 0.8506 |
| Arm | Runs | Requests checked | Changed by the divergence | Share (%) | Failed calls recorded |
|---|---|---|---|---|---|
| F | 924 | 15,043 | 0 | 0.0 | 17 |
| R | 924 | 15,290 | 13,616 | 89.1 | 20 |
| C | 924 | 13,036 | 2,709 | 20.8 | 131 |
| H | 924 | 17,450 | 15,684 | 89.9 | 112 |
| T | 924 | 14,787 | 12,629 | 85.4 | 19 |
| Measure | F mean | R F | C F | H F | T F |
|---|---|---|---|---|---|
| history summaries per run | 0.69 | ||||
| [ , ] | [ , ] | [ , ] | [ , ] | ||
| runs with a summary (share) | 0.482 | ||||
| [ , ] | [ , ] | [ , ] | [ , ] | ||
| largest prompt (tokens) | 12,721 | ||||
| [ , ] | [ , ] | [ , ] | [ , ] |
| Block | Arm | Tasks | Runs | Solved | % | Lead | Agent |
|---|---|---|---|---|---|---|---|
| pilot-2a | v-t1 | 15 | 15 | 4 | 26.7 | — | 98 |
| pilot-2a | v-t2 | 15 | 15 | 5 | 33.3 | — | 71 |
| pilot-2b | v-t3 | 15 | 15 | 3 | 20.0 | — | 96 |
| pilot-2b | v-t4 | 15 | 15 | 7 | 46.7 | — | 89 |
| pilot-2b | v-t5 | 15 | 15 | 6 | 40.0 | — | 77 |
| pilot-2b | v-t6 | 15 | 15 | 6 | 40.0 | — | 66 |
| Application 1 | Application 2 | Application 3 | |
| Task set | small panel | large panel minus small | as application 2 |
| Tasks; repositories; repeats | 72; 18; 4 | 128; 19; 4 | 128; 19; 4 |
| Runs per arm (all with a result row) | 288 | 512 | 512 |
| Solved runs: baseline, candidate | 63, 62 | 103, 101 | 97, 118 |
| Mean gain (points; exact fraction) | ( ) | ( ) | ( ) |
| (1) gain at least | not met | not met | met |
| Comparison of two arms | Runs per arm | SD, points | Time as run |
|---|---|---|---|
| Screen on S (48 tasks, 2 repeats) | 96 | 2.9 | 1–2 hours (3–6 arms) |
| Confirmation on H (128 tasks, 4 repeats) | 512 | 1.44 | about 3 hours (2 arms) |
| Hosted, one draw per file (58 public tasks) | 58 | 3.8 | 2 days |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Arm | Model | Runs | What it is |
|---|---|---|---|
| BF5 | 31B | 142 | Single agent: the public Budget-Fit V5 configuration, run unchanged ( reference ) |
| BF5-p | 31B | 30 | BF5 with one changed section of the task prompt ( prompt ) |
| BF5-c | 31B | 32 | BF5 with a prompt-only checking step before the patch is submitted ( prompt ) |
| BF5-s | 31B | 10 | BF5 with changed sampling settings ( sampling ) |
| W1 | 31B | 30 | Wrapped: a root agent calls a coder sub-agent as a tool; the coder has its own context ( architecture ) |
| W2 | 31B | 40 | W1 with a bounded restart of the coder after a malformed tool call ( architecture ) |
| Block | Model | Machine | Source | Tasks | Arms | Rep. | Runs | Note |
|---|---|---|---|---|---|---|---|---|
| pub31-1 | 31B | emulated | dev. | 30 | 3 | 1 | 90 | first 30 development tasks |
| pub31-2 | 31B | emulated | dev. | 30 | 1 | 1 | 30 | same 30 tasks |
| pub31-3 | 31B | emulated | dev. | 30 | 1 | 1 | 30 | same 30 tasks |
| pub31-4 | 31B | emulated | dev. | 30 | 2 | 1 | 60 | same 30 tasks, both arms in one queue |
| pub31-5 | 31B | emulated | dev. | 40 | 2 | 1 | 80 | 40 further development tasks |
| pub31-6 | 31B | emulated | dev. | 40 | 2 | 1 | 80 | same 40 tasks, both arms in one queue |
| 95% interval | Monte Carlo | |||||
|---|---|---|---|---|---|---|
| Arm | Registered | Kit, panel order | Kit command | Registered | Kit, panel order | Kit command |
| R | [ , ] | [ , ] | [ , ] | 0.8650 | 0.8650 | 0.8642 |
| C | [ , ] | [ , ] | [ , ] | 0.9243 | 0.9243 | 0.9268 |
| H | [ , ] | [ , ] | [ , ] | 0.1106 | 0.1106 | 0.1095 |
| T | [ , ] | [ , ] | [ , ] | 0.5599 | 0.5599 | 0.5624 |
| No. | Candidate | Tasks | Repos | Runs/arm | Base | Cand. | Gain | LOO | Time | Verdict | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | v-p2 | 72 | 18 | 288 | 63 | 62 | 0.35 | 0.6034 | 3/6 | 1.47 | 1.03 | not confirmed |
| 2 | v-t6 | 128 | 19 | 512 | 103 | 101 | 0.39 | 0.6034 | 7/6 | 0.85 | 1.18 | not confirmed |
| 3 | v-a3 | 128 | 19 | 512 | 97 | 118 | +4.10 | 0.0405 | 9/6 | +2.94 | 1.10 | not confirmed |
| File | Records | Content |
|---|---|---|
| runs.csv | 5,024 | one row per agent run of Sections 7 , 8 and 10 |
| controls.csv | 2,494 | one row per control grading (reference patch or empty patch) |
| serving_bench.csv | 72 | one row per cell of the serving benchmark of Section 3 |
| serving_servers.csv | 7 | start-up time and cache size of each model server of the benchmark |
| experiments.csv | 31 | one row per block ( Table 33 ) |
| arms.csv | 37 | one row per arm ( Table 32 ) |
| Column | Content |
|---|---|
| batch | 1 to 5: the batch of the block in which the job ran; batch 5 holds the re-runs |
| stage | 1, 2 or 3: the nested stage of the job list that holds the task (118, 37 and 76 tasks) |
| arm | F, R, C, H or T |
| task | task identifier in the public task collection the panel was drawn from |
| repository | repository of the task (owner/name); the unit of clustering |
| repeat | 1 to 4 |
AI Agent Evaluation
AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.
See all 555 papers