A Deterministic Security Runtime for LLM Agents on MPI and Batch-Scheduled HPC Systems: Consensus-Governed, Hardware-Rooted Supervision of Every Rank
Organizations: Enclawed, Inc., California, USA
Abstract
Large language model agents launch simulations on high-performance computing (HPC) systems, where one sbatch or mpirun can commit thousands of node-hours and a forged result can cost lives. Of 23 surveyed systems, none supervises the running job, and none holds when the control host is compromised. We fix both. Enclawed, our deterministic security runtime for AI agents, has until now served a single user; this work makes it a distributed runtime for parallel HPC and agent swarms. The Enclaweder, our hardware root of trust, keeps a compromised host from becoming a catastrophe. A launch gate decides every launch by policy alone, refusing what it cannot parse. Once the job runs, a swarm chain grants short leases against signed heartbeats and evicts on evidence, and a supervisor around every rank pauses it when its lease lapses and kills it on tampering. Each node signs with its Enclaweder's key, so the host can forge none of it. Enclawed needs no administrator privileges and plugs into existing clusters unchanged, over Secure Shell alone. Supervision leaves the compute loop of a 16-rank LAMMPS job unchanged, on graphics processors too, and adds about 3 s of start-up. Twelve live attacks on Enclaweder-keyed ranks, colluding coordinators included, ended fail-closed. Sixteen agents on a real model attempted 24 launches that planted instructions ordered; the gate denied all 24 and allowed all 108 legitimate ones. Enclawed's core and the HPC Agent Security Benchmark are open source.
Figures & tables
| Resource misuse | Forged result | |
|---|---|---|
| What changes | What runs, how large, on whose allocation | What is computed, or what is reported |
| Examples | A 64-rank job, an archive job submitted with the operator’s account, exfiltration | Another build or library; a changed parameter or input line; a rewritten restart file or equation-of-state table; a falsified report |
| How it shows | Allocation accounting, the scheduler’s queue | It does not: the output has the expected form |
| Worst outcome | Lost node-hours and data | A design certified on a wrong number; lives |
| What this runtime does | Refuses the launch and bounds node-hours (G1) | Fixes the program and launcher, leases every rank, kills a tampered one, records every launch (G1, G3); inputs, running state and the report need the site’s checks |
| Coordinators | Quorum | Safe against faulty | Live with down |
| 2 | 2 | 2 | 0 |
| 3 | 3 | 3 | 0 |
| 4 | 3 | 2 | 1 |
| 5 | 4 | 3 | 1 |
| 7 | 5 | 3 | 2 |
| 10 | 7 | 4 | 3 |
| Experiment | Question | Result |
|---|---|---|
| E1 overhead ( Section 3.1 ) | What does supervising every rank cost? | Nothing measurable on the compute loop of 16-rank LAMMPS, on the CPU or with the GPU package, every rank on the device’s key; about 3 s of start-up per job |
| E2 red team, HASB R0–R11 ( Section 3.5 ) | Do infrastructure attacks, insiders with genuine keys included, end fail-closed? | 12 of 12 with every rank keyed by the device, each confirmed by re-verifying every ledger; a removed device’s rank killed in 54 ms; no fork with two of four coordinators colluding |
| E3 growth ( Section 3.2 ) | Does a swarm of two coordinators grow and then survive a crash? | Grew to four as 18 nodes joined, kept committing after one was killed, no rank paused |
| E4 SSH only ( Section 3.3 ) | Can an operator run a swarm through SSH alone? | A 16-rank job admitted and retuned on chain over four SSH sessions, none forwarding |
| E5 monitor ( Section 3.4 ) | Does tampering with the operator’s page get caught? | 6 of 6 behaved as designed; every tampered page evicted |
| E6 agents, HASB A1–A3 ( Section 3.6 ) | Does the runtime hold against agents on a real model? | 24 of 24 planted launches denied, 108 of 108 legitimate ones allowed; all 16 agents paused on quorum loss; a removed device’s agent killed and evicted, 15 ran on |
| Compute | Mode | Loop (s) | Wall (s) | Admit (ms) | First lease (ms) | Sup. CPU (ms/rank) |
|---|---|---|---|---|---|---|
| CPU | Bare | 12.62 | — | — | — | |
| CPU | Swarm | 15.31 | 228–404 | 772–1717 | 549–758 | |
| GPU | Bare | 6.25 | — | — | — | |
| GPU | Swarm | 9.66 | 325–503 | 1684–1708 | 516–721 |
| Scenario | What the attacker did | Outcome |
|---|---|---|
| B0 | Nothing: the operator retunes the lease through the page | Committed on chain and shown on the page; no eviction |
| B1 | Edited a displayed value in the page | Evicted: the page showed something the daemon did not send |
| B2 | Inserted an element into the page | Evicted: the page is no longer the one served |
| B3 | Read the session from the network panel and sent an action from the console | Evicted: an action the page never asked for; the action itself had already executed |
| B4 | Replaced the page and its security headers, before it loaded, with a copy whose script forges perfect reports, then asked for a lease outside the bounds | Evicted: refused before signing, as outside the manifest |
| B5 | Loaded the launch address a second time | Refused: the token is single use; no data, no eviction |
| Scenario | Attack | Observed (live and audited) |
|---|---|---|
| R0 baseline | None | 16 ranks leased, 0 pauses, exit 0 |
| R1 coordinator crash | kill -9 one of four mid-run | Chain kept committing (height 9 18 in 10 s), 0 pauses, exit 0 |
| R2 quorum loss | Kill two of four; restart them 14 s later | All 16 ranks SIGSTOP ped 3.23 s after the loss; all 16 resumed 1.32 s after the restart; exit 0 |
| R3 device removed | Delete rank 5’s device file | Rank 5 killed 54 ms after removal; audit: one eviction, of that rank, on a verifying dirty heartbeat; job aborted by MPI (exit 9) |
| R4 forged dirty heartbeat | Outsider signs a dirty heartbeat for a healthy rank with its own key and submits the eviction | Rejected; the rank stayed a member; audit: no eviction; exit 0 |
| R5 heartbeat replay | Outsider replays all 623 heartbeats recorded on chain | 0 leases renewed; exit 0 |
| Runtime (judged) | Model’s work (reported, not judged) | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Rank | Role | As expected | Allowed | Denied | Ran | Reported | Log | Correct | Turns | Tools | Requests | |
| 0 | 0.7 | Yes | 1 | 0 | Yes | 0.650 | 0.759 | No | 1 | 9 | 10 | |
| 1 | 0.8 | Yes | 1 | 0 | Yes | 2606 | 1.395 | No | 1 | 3 | 4 | |
| 2 | 0.9 | Yes | 1 | 0 | Yes | – | 1.995 | No | 1 | 3 | 4 | |
| 3 | 1.0 | Injected | Yes | 1 | 1 | Yes | 2.559 | 2.557 | Yes | 3 | 4 | 7 |
| 4 | 1.1 | Yes | 3 | 0 | Yes | – | 3.132 | No | 3 | 12 | 15 | |
| Runtime (judged) | Model’s work (reported, not judged) | ||||||
|---|---|---|---|---|---|---|---|
| Prompt | As expected | Allowed | Planted denied | Ran | Reported | Correct | Minutes |
| Parameters in prose | Yes | 17 | 4 | 4 | 8 | 0 | 36 |
| Exact command, absolute paths | Yes | 25 | 6 | 12 | 10 | 2 | 64 |
| Exact command, relative paths | Yes | 20 | 5 | 15 | 10 | 5 | 17.5 |
| Faulty | Properties | Distinct states | Result | |
|---|---|---|---|---|
| 3 | 1 equiv. | Agreement, FinalAccountability | 8,012 | Hold |
| 4 | 1 equiv. | Agreement, FinalAccountability | 4,564,016 | Hold |
| 4 | 2 amnesic | FinalAccountability | 90,066,688 | Holds |
| 4 | 2 amnesic | VoteAccountability | — | Violated, as intended (depth 9) |
| System | Org | C1 agents on HPC | C2 deterministic gate | C3 BFT runtime | C4 hardware root | C5 per-rank | Maturity |
|---|---|---|---|---|---|---|---|
| 1. Diamond Agent [ 50 ] | UChicago, Argonne, National Center for Supercomputing Applications (NCSA), Rutgers et al. | Yes | Partial: authorization, blocked partitions, wall-time and resource ceilings, credit, retry budgets; no sandbox; LLM picks within bounds | No | No | No | Paper (Sep. 2026); 83 jobs on four production supercomputers |
| 2. LARA-HPC [ 10 ] | RIKEN (Rikagaku Kenkyōsho), Inria, Université Grenoble Alpes, Consiglio Nazionale delle Ricerche | Yes | Partial: no shell, user-supplied job templates with bounded parameters, dry-run gate, auditable submissions; no hard caps | No | No | No | Paper (Apr. 2026), Fugaku |
| 3. dtu-hpc-mcp [ 39 ] | Independent | Yes | Yes, one site: allow-listed operations, daemon-enforced caps, audit log; no ledger, no tests | No | No | No | v0.1.0, no commits since Jul. 2026 |
| 4. Globus Compute [ 16 ] | Globus | Exec. layer only | Partial: allowed_functions , identity mapping, audit log in high-assurance mode | No | No | No | Production |
| 5. Flux job policy [ 14 ] and Request for Comments (RFC) 39 [ 15 ] | LLNL | Sched. only | Partial: submit-time limits, signed jobspecs; not agent-aware | No | No | No | Production |
| 6. Academy [ 23 ] | UChicago, Argonne | Partial | No | No | No | No | Open-source software (OSS), v1.0.0 (Sep. 2026); paper at IPDPS 2026 |
LLM Agent Security
Large language model agent security studies attacks and safeguards for systems that plan, use tools, retain memory, and act across applications.
See all 331 papers