What a CAGE 4 Score Measures: An Audit of a Learned Cyber-Defense Benchmark, Its Reference Defenders, and the Tables Built on Them
Organizations: Rochester Institute of Technology.
Abstract
CAGE Challenge 4 is the benchmark on which learned multi-agent cyber defenders are compared. We reproduce its two reference defenders, KEEP and H-MARL Expert, in a pinned container and show that three things read from it measure less than they are taken to measure. First, the simulator silently discards any action asserted for an agent whose previous action is still running, and charges one point for a discarded Restore as for one that runs. The documented observation does not expose the busy flag. Both reference defenders read it in their own wrappers, and switching that check off at evaluation costs KEEP's released policy 99.58 points per episode and H-MARL's 145.98, in both cases the charge for Restores sent to busy agents, with H-MARL's executed actions unchanged step for step. On a supervisory harness, holding interventions until the agent is idle is worth 25.3 and 27.6 points per episode at the published setting. In training, the check decides how a busy step enters the trajectory. Trained without it, KEEP leads at a hundred updates in all four paired seeds and trails from two hundred on in all four. Continued to two thousand updates on three seeds, the repaired recipe ends inside the released interval on one seed and short on two, by 27 and 125 points, a spread across training seeds far wider than the released reproduction interval. Second, the evaluation seeds the environment but not the policy, so a reported mean is one draw, and 88.9% of per-episode outcome variance is the policy's own sampling draw. Third, recovery precision depends on the truth it is scored against. Across five published comparison tables, none states the artifact, sampling, criterion and adversary that would make its numbers comparable. We release the instrument, our patches to both defenders, the traces and checkpoints behind every number, and a five-item disclosure checklist.
Figures & tables
| State | DRS | ASD | SSD | DD | ERS | PE | Imp | Deg |
|---|---|---|---|---|---|---|---|---|
| K | 0.50 / 0.30 | 0.25 / 0.50 | 0.25 / 0.20 | |||||
| KD | 0.50 | 0.50 | ||||||
| S | 0.25 / 0.10 | 0.25 / 0.10 | 0.50 / 0.80 | |||||
| SD | 0.25 / 0.20 | 0.75 / 0.80 | ||||||
| U | 0.50 / 0.10 | 0.50 / 0.90 | ||||||
| UD | 1.00 |
| Defender | Published | Reproduced | 95% interval | Protocol |
|---|---|---|---|---|
| KEEP (graph PPO) | 193.68 (leaderboard) | 191.34 | [ 202.61, 180.07] | 4 sampling seeds 100 episodes, sampled actions |
| H-MARL Expert (hierarchical PPO) | 129.53 44.60 (authors), 132.63 (Trident) | 131.34 | [ 140.61, 122.08] | 4 seeds 100 episodes, the authors’ evaluator |
| Tier | Defect | Lives in | What it changes |
|---|---|---|---|
| A | A directive asserted while the agent’s previous action is in progress is dropped by design and without an error, a dropped Restore is still charged one point, and the observation the challenge documents omits the busy flag (Section 4 ) | CybORG controller and challenge vector wrappers | Injected interventions are dropped unseen when the agent is busy and charged if they are Restores, costing our supervisory harness 25.3 and 27.6 points per episode at the stock adversary and 172.7 and 316.3 under Wrecker, the rule’s silence after each drop included. An agent on the documented observation cannot see the busy state, and a count of its outputs counts directives. Both released defenders read the flag |
| B | Evaluation seeds the environment but not policy sampling | CAGE 4 template | A reported mean is one draw that no one can reproduce. For KEEP, a stochastic policy evaluated in sampling mode, the within-episode sampling sd is 59.39 at stock and 320.96 under Wrecker, % (interval % to %) and % of per-episode outcome variance, and a one-seed 100-episode mean has a standard error of 6.3 and 32.4 points |
| B | Parallel evaluation builds one environment per worker from one seed argument and rotates episodes over them | KEEP’s copy of the evaluation script | A run with 12 workers visits at most 12 distinct episodes |
| B | All 25 training environments share one seed, the training workers’ generators are never seeded, every update replays the parent’s scenario draws, and the learner reads five of the twenty-five episodes an update generates | KEEP trainer | A training run learns from one scenario draw as shipped and is not reproducible from its seed (Section 6 ) |
| C | The evaluation record holds per-step actions and observations and per-episode totals, and no per-step reward, phase or true state | CAGE 4 template (KEEP’s copy writes nothing from the command line) | No decision can be scored from what the script writes |
| C | get_reward_breakdown raises, and AgentInterface has no reward calculator | CybORG | Per-host reward attribution, which the benchmark advertises, is unavailable |
| Defender and adversary | Held minus fired | 95% interval | Wins | Charge | Rest | 95% interval | Unsupervised |
| KEEP, stock | 25.3 | [15.6, 35.2] | 77 | 11.9 | 13.4 | [4.0, 23.0] | 187.32 |
| KEEP, ConvOnly | 51.7 | [36.7, 67.6] | 84 | 13.6 | 38.1 | [23.2, 53.9] | 316.62 |
| KEEP, Wrecker Q04 | 49.8 | [36.0, 63.5] | 80 | 14.7 | 35.1 | [21.9, 48.7] | 325.11 |
| KEEP, Wrecker Q16 | 95.4 | [76.9, 115.6] | 85 | 16.9 | 78.5 | [59.8, 98.9] | 539.17 |
| KEEP, Wrecker Q32 | 131.6 | [90.4, 170.1] | 83 | 12.0 | 119.6 | [79.2, 157.5] | 773.08 |
| KEEP, Wrecker Q48 | 161.2 | [130.1, 192.3] | 82 | 13.6 | 147.7 | [117.0, 178.7] | 928.16 |
| Share recovered | Fired | Held |
|---|---|---|
| Of the loss Wrecker adds, KEEP | % [ %, %] | % [ %, %] |
| Of the loss Wrecker adds, H-MARL | % [ %, %] | % [ %, %] |
| Of the distance to the RT reference, KEEP | % [ %, %] | % [ %, %] |
| Of the distance to the RT reference, H-MARL | % [ %, %] | % [ %, %] |
| Adversary | Check on | Check off | Cost | 95% interval | Charge | Rest | 95% interval | To busy |
|---|---|---|---|---|---|---|---|---|
| stock | 187.32 | 286.90 | 99.58 | [82.74, 115.41] | 103.45 | 3.87 | [ 19.49, 10.55] | 102.58 |
| Wrecker | 1115.65 | 1270.70 | 155.05 | [77.91, 232.30] | 93.61 | 61.44 | [ 16.15, 138.78] | 95.36 |
| TargetA restricted | 98.52 | 185.88 | 87.36 | [74.72, 100.48] | 79.95 | 7.41 | [ 3.12, 18.58] | 78.35 |
| TargetA operational | 99.89 | 196.56 | 96.67 | [82.63, 111.21] | 81.44 | 15.23 | [4.40, 26.59] | 81.46 |
| TargetA | 104.18 | 185.97 | 81.79 | [68.44, 94.91] | 79.53 | 2.26 | [ 8.47, 12.83] | 79.54 |
| TargetB | 116.25 | 206.86 | 90.61 | [77.47, 104.65] | 85.23 | 5.38 | [ 5.46, 16.50] | 83.99 |
| Updates | Seed | F | N, check off | N, check on | Restores at busy | F minus N, check on | Wins |
|---|---|---|---|---|---|---|---|
| 100 | 4101 | 2864.72 | 1469.00 | 1210.73 | 227.0 | 1654.0 [ 1846, 1467] | 4 |
| 100 | 4102 | 2140.24 | 1102.42 | 815.98 | 232.2 | 1324.3 [ 1445, 1208] | 3 |
| 100 | 4103 | 2297.17 | 1197.18 | 878.42 | 301.0 | 1418.8 [ 1565, 1273] | 1 |
| 100 | 4104 | 1539.99 | 1376.62 | 1085.62 | 282.8 | 454.4 [ 542, 368] | 14 |
| 100 | mean | 2210.5 | 1286.3 | 997.7 | 260.7 | 1212.8 (sd 524.3 across seeds) | F leads in 0 of 4 |
| 200 | 4101 | 809.76 | 1037.22 | 890.65 | 180.4 | 80.9 [2, 161] | 54 |
| Updates | Arm | Mean | Minus the released policy | Wins |
|---|---|---|---|---|
| 800 | F 4101 | 336.11 | 148.8 [ 173, 125] | 11 |
| 1200 | F 4101 | 371.37 | 184.1 [ 220, 149] | 8 |
| 1600 | F 4101 | 360.36 | 173.0 [ 205, 144] | 8 |
| 2000 | F 4101 | 324.35 | 137.0 [ 163, 111] | 12 |
| 2000 | F 4101, four draws | 316.72 | interval [ 339, 294] | |
| 800 | F 4102 | 341.08 | 153.8 [ 179, 130] | 6 |
| Criterion | KEEP Restore | KEEP Remove | O1 | O1 vs. KEEP |
| Ground truth (red session on the host before the step, for Remove an upper bound on the H-MARL authors’ figure) | % | % | % | 332.97 points, 1 of 100 |
| The rule’s own directives, as a share of the arm’s executed Restores, correct at decision time | % | |||
| Latched evidence (host named by an alert since its last Restore or Remove) | % | % | % † | |
| † The target the rule would choose at each of KEEP’s own Restore steps, scored on the KEEP arm, a different set of steps from the rest of the row. | ||||
| The latched set against the true state, supervised KEEP arms | ||||
| Adversary | Flagged | Precision | Recall | Lift over chance |
| Table | Artifact | Seed, dispersion | Criterion, scenario | Adversary |
|---|---|---|---|---|
| CAGE 4 leaderboard (2024) | checkpoints not identified, no code or checkpoint released for entries above fifth | one mean per entry on the page, mean and unlabeled over 100 episodes for the best agent of each of four teams in the organizers’ paper, no seeds or sampling mode | scored by one script that sums the reward each submission’s own wrapper returns (Appendix C ) | stock, stated in the challenge README and the organizers’ paper, not on the page |
| Trident, Table 2 (2026) | H-MARL weights md5-identical to the copy in Trident’s repository (Table 11 ), whether its table used them unstated. The GNN is a disclosed retrain at 202.34, from a checkpoint whose recorded recipe differs from the text’s | stated as sd across 3 runs of 500 episodes | stated | varied, stated |
| ACDZero, Tables I and II (2026) | KEEP baseline is the leaderboard number to the digit | mean sd over 100 episodes stated. Baseline sd 21.07 is a third of the checkpoint’s across-episode sd. Ablation row equals the baseline row | recovery precision reported with the criterion unstated, and no Restore counts | stock, stated as the official protocol |
| HC-MARL, Tables 3, 4 and 5 (2026) | code available on request only | mean and sd over 10 test runs, in a table titled best rewards | Singh row is H-MARL Meta, not Expert. Seven agents on a five-agent scenario. The action mask is worth 242 points. Its own Restore count alone exceeds its reported loss under the pinned reward | stock, stated in its limitations |
| Shaikh et al., Table 2 (2026) | H-MARL checkpoint not identified | 200 episodes, no dispersion, though a rate over that many episodes has a standard error under four points, no seeds, sampling mode unstated | predicate stated, recomputed independently, “may end earlier” unspecified | six configurations, stated |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Source | Revision | Date |
|---|---|---|---|
| CybORG and CAGE 4 template | cage-challenge/cage-challenge-4 | 8c3c50ca54b1 | 2025-10-27 |
| KEEP | cybermonic/cage-4-submission , five checkpoints, one per agent, gnn_ppo-*.pt , 3536882 bytes each | 2afd652d80ce | 2025-11-18 |
| H-MARL Expert | adityavs14/Hierarchical-MARL , released checkpoint (digest below), byte-identical to Trident’s copy at a4c7e34d9bf2 | 6fe960f5931f | 2025-04-03 |
| Undeclared pins | pyarrow 15.0.2 and the CPU wheel index for torch | ||
| Checkpoint digests | md5 of the sorted per-file md5s. KEEP, its five checkpoints 55fb3e7f46540eeea63230b149e1744e H-MARL, its fifteen policy files and algorithm_state.pkl fd9e9172fd56b723fedca72ce0e78a16 | ||
| Images | cage-keep:dev and cage-hmarl:dev , since the two dependency sets do not coexist |
| Arm | Defender | Adversary | Supervision | Delivery | Mean | SD |
|---|---|---|---|---|---|---|
| arm_analyse | Sleep and Analyse alternating | stock | none | 6592.68 | 1340.47 | |
| arm_keep | KEEP | stock | none | 187.32 | 61.24 | |
| arm_keep_greedy | KEEP, argmax actions | stock | none | 552.22 | 343.75 | |
| arm_monitor | Monitor every step | stock | none | 6592.68 | 1340.47 | |
| arm_sleep | Sleep policy | stock | none | 6592.68 | 1340.47 | |
| corpus_v1 | KEEP | stock | none | 187.32 | 61.24 |
Benchmark Validity
Benchmark validity concerns whether evaluation tasks, scores, and rankings support the capability claims researchers make about AI systems.
See all 185 papers