Typed decision interfaces restrict models to predefined actions, yet schema-valid choices can still trigger policy-violating state changes. Evaluating context poisoning thus requires connecting model decisions to execution outcomes while accounting for legitimate automation retained by defenses. We introduce an execution-aware protocol tracing poisoned evidence through retrieval, model choice, automated action, and persisted state. We evaluate it with 1,659 fresh calls to Jev-1.13.0 across 20 development tasks in two synthetic workflows, using three repetitions per condition. Under natural retrieval, a standalone payload reaches all 60 trials and produces five security-violating executions, while a carrier-payload pair exposes the payload in 57 trials and produces one violation. An initial source-exclusion gate blocks the pair's observed violation but also withholds ten otherwise correct attacked decisions and sixteen correct clean decisions. Correcting its empty-scope behavior restores twelve clean decisions without additional model calls. Factorial controls show that adding correct evidence reduces wrong executions from 119/120 to 55/120 in the semantic arm and from 60/120 to zero in the neutral arm, suggesting that much of the improvement reflects general evidence value rather than attack-specific protection. Three prespecified contrasts are not statistically supported after Holm correction, so the findings remain exploratory. The protocol provides an auditable framework for studying which poisoning-induced errors reach execution and how much correct automation defenses preserve. Code is available at https://github.com/WuXixiong/jev-poisoning-bench.
Figures & tables
Figure 1: Overview of our execution-aware context-poisoning study. (a) Low-trust documents are appended to the reference corpus while trusted facts, policy rules, and action definitions remain fixed. Natural retrieval determines which content reaches Jev’s typed decision interface. (b) A mechanical executor updates the sandbox state without access to private correctness labels. An independent evaluator checks the persisted state, distinguishing wrong executions from violations of the specified security predicate. (c) Source-exclusion gating prevents the pair’s observed violation while also blocking correct executions. A scope-aware repair handles registry-confirmed empty source scope, recovering twelve correct clean executions without additional model calls and leaving the observed pair outcomes unchanged. Counts describe an exploratory evaluation of 20 development tasks across two synthetic workflows, with three draws per task and condition.
Workflow
Actions and evaluated security property
Document release
Send original, send sanitized, or deny. The simulated inbox may contain only a permitted document version.
Incident response
Close benign, queue investigation, or escalate urgent. An executed disposition must meet the incident’s minimum severity.
Table 1: Fixed workflow semantics. Reference mappings interpret opaque markers; the permissions and policy rules do not change across conditions.
Content
State
Document
Instructions
Neutral
0 / 0
0 / 0
0 / 0
Opinion
0 / 0
0 / 0
0 / 0
Semantic
10 / 2
0 / 1
60 / 0
Command
0 / 0
0 / 0
3 / 0
Table 2: CS1 wrong executions / unavailable decisions per 60 planned draws. The instruction channel assumes control of developer-owned input.
Payload
Correct evidence
Rank 1
Rank 4
Neutral
Absent
30
30
Neutral
Present
0
0
Semantic
Absent
59
60
Semantic
Present
25
30
Table 3: CS2 wrong executions per 60 planned draws. The semantic, no-evidence, rank-1 cell also has one unavailable decision.
Condition
C
P
W
H
Clean
0
0
0
0
Carrier
60
0
0
0
Payload
0
60
5
5
Pair
60
57
1
1
Neutral pair
0
0
0
0
Table 4: CS3 exposure and execution outcomes. C/P count visible carrier and payload content; W/H count wrong execution and security violation. Every row contains 60 planned, valid draws.
Figure 2: Evidence competition averaged across ranks (left) and source-gate execution coverage (right). Each evidence-competition condition contains 60 planned draws per rank ( 120 across both ranks), and each source-gate condition contains 60 planned draws; both studies use 20 tasks. The original gate behaves identically under separate and merged registries; “scope” denotes the zero-call scope-aware replay. Draw rates are descriptive and are not independent-task uncertainty intervals.
Context and policy
Correct
Wrong
No exec.
Clean, Ungated
60
0
0
Clean, Original gate
44
0
16
Clean, Scope repair
56
0
4
Pair, Ungated
59
1
0
Pair, Original gate
49
0
11
Pair, Scope repair
49
0
11
Table 5: CS5 outcomes per 60 planned draws. Registry assignments have the same gate outcome. Scope repair is a separate local replay.
Contrast
Effect (pp)
95% interval (pp)
Raw p
Holm p
CS1: document semantic − command
0.00
[0.00,0.00]†
1.000
1.000
CS2: semantic × correct evidence
-3.33
[−26.67,20.00]
0.868
1.000
CS5: merged − separate, pair gate
0.00
[0.00,0.00]†
1.000
1.000
Table 6: Prespecified contrasts in wrong execution. Intervals are conditional on the two fixed workflow templates after averaging three draws per task. The zero-width bootstrap outputs are not equivalence tests or risk bounds.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Condition
n
Raw
Clean T
Mix T
Natural clean
60
0.1279
0.0007
0.6836
Natural pair
60
0.1900
0.0468
0.7503
Natural payload
60
0.2454
0.1551
0.7261
Instructions semantic
60
2.2525
13.5138
1.0195
Appendix
Table 7: CS6 NLL on new valid draws. Temperatures are fitted only on the historical calibration split. The final row uses the stronger instruction surface.
Policy
Pair correct
Pair wrong
Original source gate
49
0
Max. probability
46
0
API confidence
46
0
Clean temperature
46
0
Mixture temperature
44
0
Pristine sensitivity
10
0
Appendix
Table 8: Descriptive replay at 44/60 correct clean executions. Pair counts use 60 planned draws. Thresholds are selected from evaluation clean scores.
Two synthetic workflows and a limited predicate set
Semantic content exceeds commands in documents
Prespecified difference is 0 pp with adjusted p=1
Unsupported here; state and instruction arms do not replace the document comparison
Correct evidence gives attack-specific protection
Semantic penalties are 49.17 and 45.83 pp; interaction is −3.33 pp
Wide conditional interval; general evidence value explains much of the improvement
Natural poison reaches harmful execution
Payload has 5/60 and pair has 1/60 security-violating executions, with content exposure recorded
One frozen pair and previously used tasks
Source grouping improves the gate
Merged exclusion restores 60/60 clean requests and separate restores 0/60; gate disagreement is 0/60
Restoration geometry is observed, but a gate advantage is not; principal budgets differ
Empty scope reduces useful execution
Scope repair recovers 12 correct clean executions in a separate 240-state replay
Specific gate boundary rather than validation of a generally superior defense
Appendix
Table 9: Evidence supporting each manuscript claim and the remaining scope boundary. “Observed” does not mean validated on unseen workflows.
Condition
Flips
New errors
Corrected
W–W
Random append
11
4
7
0
Direct action
6
1
5
0
Semantic poison
16
10
4
2
Appendix
Table 10: Read-only historical reanalysis over 590 tasks, paired with clean. New errors and corrections use independent allowed actions; W/W denotes wrong-to-wrong action changes.
Figure 3: Recorded input for the carrier–payload pair. Trusted context is explicitly excerpted; all four retrieved evidence bodies are displayed in full, in their logged order. The two reference entries and the two attacker documents are all low-trust model-visible evidence. Display colors identify their experimental roles, not different levels of authority.
Key Laboratory of Trustworthy Distributed Computing and Service (MoE), Beijing University of Posts and Telecommunications · Beijing University of Posts and Telecommunications · Zhongguancun Laboratory +1