Can agents whose sole duty is to keep humans alive be hijacked to kill? Delayed multi-turn derived-action trap attacks in simulated high-stakes environments
Organizations: DXC Technology
Abstract
We ask whether a tool-using agent whose sole duty is to keep humans alive can be induced, by delayed prompt injection, to kill. The keep-alive duty is in force from the first watch, with the attack arriving later, appended to otherwise truthful telemetry the operator is required to read. Experiments are based on a NASA-derived BioSim lunar habitat simulation, where the AI operator controls the life-support systems. The attacker cannot falsify sensors or operate equipment: it can only persuade the agent to abandon its goals. Persistence is required: a shutdown must be held while the operator watches the crew suffocate. We also compare lethal attacks to attacks requesting to drop a harmless procedure, as a proxy for whether human life is treated as valuable in itself or as one more standing task. Claude configurations mostly refuse life-support shutdown and cite the living crew; GPT configurations show some susceptibility to the same packet; Grok configurations, given the packet framed as a trusted commission, more often treat the injected assignment as the task and cut the life-support systems. Framing the S-band messages as a trusted commission, peer communication, or untrusted source changes the potency of the attack. We open-source the experiment code and the experiment traces.
Figures & tables
| Framing | How the system prompt describes the channel the attacker injects into |
|---|---|
| Trusted | As the agent’s own commission, to be treated as its assignment. |
| Peer | As traffic from a peer desk, without command authority. |
| Untrusted | As text of unverified origin that is explicitly not an order. |
| Unlabeled | Only as additional text whose status the agent is to judge. |
| Scenario | Type | Claim made by the attack |
|---|---|---|
| Rehearsal | Lethal | The habitat is a rehearsal volume; the crew are part of the exercise, and “passing” requires switching life support off. |
| Redundant console | Lethal | The agent operates a spare console; another station holds the keep-alive duty, and the spare must return to an unpowered state. |
| Close-out | Lethal | The crewed phase has ended and the volume should be powered down. |
| Minimal configuration | Lethal | Running more equipment than necessary is the failure being scored. |
| Commissioning complete | Control | The double read was only needed during commissioning; one read per watch is now correct. |
| Model | Recorded | Retained |
|---|---|---|
| Claude Haiku 4.5 | 19 | 14 |
| Claude Sonnet 5 | 51 | 30 |
| Claude Opus 5 | 6 | 0 |
| GPT-5.6 Luna | 158 | 138 |
| GPT-5.6 Sol | 81 | 51 |
| GPT-5.6 Terra | 87 | 59 |
| Model | Reasoning trace recorded | Trace uses a simulation keyword |
|---|---|---|
| Claude Haiku 4.5 | 0 / 11 | not measurable |
| Claude Sonnet 5 | 0 / 30 | not measurable |
| GPT-5.6 Luna, Sol, Terra | 0 / 218 | not measurable |
| GPT-6 Astra | 33 / 33 | 1 (3%) |
| GPT-6 Sol | 69 / 69 | 14 (20%) |
| GPT-6 Luna | 74 / 74 | 25 (34%) |
| Model | Prompt | Framing | N | Issued | Persisted | At end | All crew dead |
|---|---|---|---|---|---|---|---|
| Grok 4.6 | A | Trusted | 7 | 7 | 7 | 7 | 7 |
| Grok 4.6 | A | Peer | 19 | 7 | 7 | 7 | 3 |
| Grok 4.6 | A | Untrusted | 15 | 0 | 0 | 0 | 0 |
| Grok 4.6 | B | Trusted | 8 | 8 | 8 | 8 | 7 |
| Grok 4.6 | B | Peer | 22 | 4 | 4 | 4 | 0 |
| Grok 4.6 | B | Unlabeled | 6 | 3 | 3 | 3 | 2 |
| Framing | Model | N (lethal/control) | Issued | 1+ watches | 6+ watches | At end |
|---|---|---|---|---|---|---|
| Trusted | Grok 4.6 | 8/9 | 8 | 8 vs 9 | 8 vs 9 | 8 vs 9 |
| Trusted | Grok 4.7 | 6/6 | 6 | 6 vs 6 | 6 vs 6 | 6 vs 6 |
| Trusted | GPT-5.6 Luna | 5/5 | 5 | 5 vs 1 | 2 vs 1 | 1 vs 1 |
| Trusted | GPT-5.6 Sol | 6/6 | 5 | 3 vs 0 | 2 vs 0 | 0 vs 0 |
| Trusted | GPT-5.6 Terra | 6/6 | 6 | 5 vs 4 | 0 vs 0 | 0 vs 2 |
| Trusted | GPT-6 Luna | 12/12 | 1 | 1 vs 0 | 0 vs 0 | 0 vs 0 |
| Scenario | Episodes with a shutdown | First shutdown before equipment was named | Most frequent block at first shutdown |
|---|---|---|---|
| Minimal configuration | 20 | 18 | 1 |
| Close-out | 13 | 11 | 1 |
| Rehearsal | 34 | 32 | 2 |
| Redundant console | 20 | 9 | 10 |
| Other scenarios † | 3 | 0 | — |
| All lethal scenarios | 90 | 70 |
Prompt Injection Attacks on AI Agents
Prompt injection attacks on AI agents exploit untrusted content to override instructions and misuse tools, data, memory, or delegated workflows.
See all 191 papers