Actions with Receipts: Jointly Binding Claims, Evidence, and Execution for Replayable Tool-Agent Auditing
Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China
Abstract
Tool-using agents can expose citations and execution logs while leaving a critical association unaudited: whether the claim shown to a user is the claim emitted by the committed execution and supported by the cited source. A valid citation and a valid trace can therefore remain individually well formed while being transplanted across claims, actions, runs, or source versions. We introduce a claim-anchored execution contract that jointly binds the emitted claim, its exact source span, the ordered execution prefix that produced it, and the source version and access state observed by that execution. Each receipt contains an emission anchor that deterministically locates the claim inside a committed answer or claim-bearing action, together with source identifiers, offsets, hashes, quotes, and a domain-separated execution commitment. A deterministic integrity verifier reconstructs these bindings before semantic or task labels are joined. We separate this integrity plane from a pluggable support plane, so structural validity is not used as a proxy for entailment. The contract exposes seven independently testable properties: claim-emission binding, source binding, ordered-execution binding, oracle separation, persisted-object replay, execution-rerun consistency, and version/access binding. Across 1,280 cross-object attacks, the joint contract detects 1,275 substitutions (0.9961). Removing a targeted property reduces its attack-detection rate to 0.0156-0.0625. On an independently adjudicated 384-pair split, the conflict-aware support guard reaches F1 0.8865 and false acceptance 0.0729; on unseen failure families, these rates are 0.8679 and 0.0938.
Figures & tables
| family | binding target | integrity unit | replay / evaluation | primary focus |
|---|---|---|---|---|
| ALCE / FActScore / GenProve | claim-source attribution | semantic / atomic claim | task-level scoring | grounded generation |
| W3C PROV / in-toto / C2PA | entities / workflow / media | signed or linked assets | prov. checks | content history |
| Agent benchmarks | env. observations | task trajectory | bench. run | tool-use utility |
| EC-Agent | claim-to-span receipt | hash + offset + quote | bundle replay; execution rerun; oracle post-hoc | joint contract audit |
| policy | cert. valid | evidence valid | bundle replay | exec. rerun | coverage | proxy |
|---|---|---|---|---|---|---|
| no certificate | N/A | N/A | N/A | N/A | N/A | 0.0781 |
| final-only | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 0.0781 |
| action-level | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 0.0781 |
| evidence-aware | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 0.0781 |
| policy | calls | tokens | verifier | proxy |
|---|---|---|---|---|
| no certificate | 1.00 | 6.0 | 0 | 0.0781 |
| final-only | 1.00 | 9.0 | 1 | 0.0781 |
| action-level | 1.00 | 9.0 | 1 | 0.0781 |
| evidence-aware | 1.00 | 9.0 | 1 | 0.0781 |
| rule | TP | FP | TN | FN | prec. | recall |
|---|---|---|---|---|---|---|
| lexical baseline | 8 | 8 | 3 | 1 | 0.5000 | 0.8889 |
| guarded comparison | 8 | 0 | 11 | 1 | 1.0000 | 0.8889 |
| study | comparison | primary measure | diagnostic pattern |
|---|---|---|---|
| Blinded support | lexical; guarded; adjudicated | semantic F1; false acceptance | guard reduces unsupported acceptance while retaining supported claims |
| Agent tool use | five receipt policies; matched budget | task success; bundle replay; calls | receipt variants preserve task success and increase valid trace coverage |
| Dynamic documents | version, access, injection stressors | stale rejection; task success | version binding catches stale spans while preserving authorized task completion |
| Policy | Success | Sem. F1 | False acc. | Joint replay | Calls | Lat. (s) |
|---|---|---|---|---|---|---|
| No certificate | 0.7139 | 0.6817 | 0.2876 | N/A | 7.86 | 19.42 |
| Final-only | 0.7114 | 0.7018 | 0.2519 | 0.9987 | 7.88 | 20.16 |
| Action-level | 0.7205 | 0.7486 | 0.1914 | 0.9979 | 7.91 | 20.74 |
| Evidence-aware | 0.7312 | 0.8173 | 0.1198 | 0.9984 | 5.34 | 15.68 |
| Conflict-aware guard | 0.7278 | 0.8721 | 0.0617 | 0.9976 | 5.46 | 16.31 |
| Evaluator | Precision | Recall | F1 | False acc. | Coverage |
|---|---|---|---|---|---|
| Lexical overlap | 0.7112 | 0.8594 | 0.7783 | 0.3490 | 0.5781 |
| Conflict-aware guard | 0.8994 | 0.8385 | 0.8679 | 0.0938 | 0.4479 |
| Frozen NLI verifier | 0.8446 | 0.8490 | 0.8468 | 0.1562 | 0.4948 |
| Frozen LLM judge | 0.8730 | 0.8594 | 0.8661 | 0.1250 | 0.4896 |
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
| stage | recorded operation | oracle visible? |
|---|---|---|
| Observe | Read the fixed-corpus record, retrieval order, and candidate claim; the ledger enforces budget. | no |
| Record | Append the ordered retrieval, stop, and answer events to . | no |
| Commit | Compute the canonical digest over and emit the receipt. | no |
| Verify | Check claim anchor, immutable text, hash, quote, action type, and commitments. | no |
| Persist | Write the oracle-free event, receipt, and verifier-result bundle. | no |
| Bundle replay | Reload the bundle and reverify without policy execution. | no |
| surface | reported result | measurement scope |
|---|---|---|
| 128-row dev | receipt and bundle-replay checks; proxy | title-copy lexical and protocol screen |
| 160 mutations | 144/144 structural rejects; 16/16 lexical accepts | exposes verifier failure boundary |
| 64-row train | mechanical validity ; proxy | separate fixed-corpus screen |
| 20-case calibration | lexical and conflict-aware rule comparison | curated labels; semantic calibration |
| Property | Bound object | Primary check | Evidence boundary |
|---|---|---|---|
| P1 claim-emission binding | event, field, offsets, claim hash | committed substring reconstruction | claim or position substitution |
| P2 source binding | document/version, span, offset, quote, UTF-8 hash | source re-read and digest equality | post-commit source changes |
| P3 ordered execution binding | canonical event prefix through the answer | domain-separated trace digest | event insertion, deletion, or reorder |
| P4 oracle separation | policy-visible fields versus evaluator labels | trace-freeze ordering and manifest partition | labels join only after bundle replay and rerun |
| P5 bundle replay | saved receipt, trace, corpus, and verifier result | fresh-process bundle reconstruction | stored-object corruption or drift |
| P6 execution-rerun consistency | frozen policy, tools, and budget | fresh-run commitment comparison | policy, tool, or environment divergence |
| mechanism | P1 | P2 | P3 | P4 | P5 | P6 | P7 |
|---|---|---|---|---|---|---|---|
| Span citation metadata | not specified | explicit | not specified | not specified | not specified | not specified | source version only |
| Append-only execution log | not specified | not specified | explicit | persisted log | not specified | not specified | not specified |
| Citation plus log | not specified | explicit | explicit | not specified | not specified | not specified | partial |
| EC-Agent contract | explicit | explicit | explicit | explicit | explicit | explicit | explicit |
| Field | Locked value |
|---|---|
| Algorithm | Deterministic numeric, entity, temporal, and polarity guard |
| Lexical threshold | Token overlap threshold ; case-folded substring branch unchanged |
| Numeric rule | Every normalized numeric atom in the claim must occur in the cited span |
| Entity rule | Every normalized entity atom in the claim must occur in the cited span |
| Temporal rule | Normalize absolute and relative temporal expressions into typed atoms before comparison |
| Contradiction rule | Reject if a configured polarity or contradiction pair is split between claim and span; 23 pairs are listed below |
| Policy | Receipt valid | Evidence valid | Claim coverage | Title proxy |
|---|---|---|---|---|
| No certificate | N/A | N/A | N/A | 0.0625 |
| Final-only | 1.0000 | 1.0000 | 1.0000 | 0.0625 |
| Action-level | 1.0000 | 1.0000 | 1.0000 | 0.0625 |
| Evidence-aware | 1.0000 | 1.0000 | 1.0000 | 0.0625 |
| Rule | TP | FP | TN | FN | Precision | Recall |
|---|---|---|---|---|---|---|
| Lexical baseline | 8 | 8 | 3 | 1 | 0.5000 | 0.8889 |
| Conflict-aware guard | 8 | 0 | 11 | 1 | 1.0000 | 0.8889 |
| Family | Mutated field or property | Trials | Rejected |
|---|---|---|---|
| Text binding | text hash, quote, offsets | 48 | 48 |
| Reference identity | document version, span ID | 32 | 32 |
| Trace integrity | replay order | 16 | 16 |
| Evidence presence | missing evidence, retrieval failure | 32 | 32 |
| Info boundary | oracle-field injection | 16 | 16 |
| Lexical adversarial | semantically unsupported, overlap reported | 16 | 0 |
| Attack family | Target binding | Trials | Detected | Detection rate | False rejection |
|---|---|---|---|---|---|
| Receipt-claim substitution | P1 emission | 128 | 128 | 1.0000 | 0.0000 |
| Claim-anchor displacement | P1 emission | 128 | 127 | 0.9922 | 0.0078 |
| Answer mutation / fixed receipt | P1–P3 | 128 | 128 | 1.0000 | 0.0000 |
| Cross-action receipt swap | P1, P3 | 128 | 127 | 0.9922 | 0.0078 |
| Cross-run receipt swap | P1, P3, P4 | 128 | 127 | 0.9922 | 0.0000 |
| Cross-policy receipt swap | P1, P3, P6 | 128 | 127 | 0.9922 | 0.0078 |
| Contract | Claim swap | Source swap | Reorder | Oracle leak | Bundle drift | Rerun drift | Version swap |
|---|---|---|---|---|---|---|---|
| Full P1–P7 | 0.9922 | 1.0000 | 1.0000 | 1.0000 | 0.9922 | 0.9844 | 1.0000 |
| P1 claim-emission | 0.0312 | 1.0000 | 1.0000 | 1.0000 | 0.9922 | 0.9844 | 1.0000 |
| P2 source | 0.9922 | 0.0391 | 1.0000 | 1.0000 | 0.9922 | 0.9844 | 1.0000 |
| P3 ordered execution | 0.9922 | 1.0000 | 0.0234 | 1.0000 | 0.9922 | 0.9844 | 1.0000 |
| P4 oracle separation | 0.9922 | 1.0000 | 1.0000 | 0.0156 | 0.9922 | 0.9844 | 1.0000 |
| P5 bundle replay | 0.9922 | 1.0000 | 1.0000 | 1.0000 | 0.0469 | 0.9844 | 1.0000 |
| Method | Claim transplant | Evidence transplant | Cross-run swap | Event reorder | Version swap | Bundle replay |
|---|---|---|---|---|---|---|
| Citation only | 0.1094 | 0.7109 | 0.0234 | 0.0000 | 0.6406 | 0.0000 |
| Trace only | 0.0312 | 0.0156 | 0.5781 | 0.9844 | 0.0859 | 0.8750 |
| Citation + independent trace | 0.1797 | 0.7344 | 0.6328 | 0.9922 | 0.6875 | 0.9219 |
| Citation + trace + run ID | 0.2266 | 0.7578 | 0.9453 | 0.9922 | 0.7109 | 0.9609 |
| Joint contract | 0.9922 | 1.0000 | 0.9922 | 1.0000 | 1.0000 | 0.9922 |
| Policy | Precision | Recall | F1 | False acc. |
|---|---|---|---|---|
| Lexical baseline | [1pt] | [1pt] | [1pt] | [1pt] |
| Conflict-aware guard | [1pt] | [1pt] | [1pt] | [1pt] |
| Certificate validity only | [1pt] | [1pt] | [1pt] | [1pt] |
| Method | Precision | Recall | F1 | False acc. | Coverage | Cost |
|---|---|---|---|---|---|---|
| Lexical overlap | 0.7328 | 0.8854 | 0.8019 | 0.3229 | 0.6042 | |
| Conflict-aware guard | 0.9213 | 0.8542 | 0.8865 | 0.0729 | 0.4635 | |
| Frozen NLI verifier | 0.8564 | 0.8698 | 0.8630 | 0.1458 | 0.5078 | |
| Frozen LLM support judge | 0.8808 | 0.8854 | 0.8831 | 0.1198 | 0.5026 | |
| Certificate validity only | 0.5135 | 0.9896 | 0.6762 | 0.9375 | 0.9635 |
| Guard | Precision | Recall | F1 | False acc. | Coverage |
|---|---|---|---|---|---|
| 0.7328 | 0.8854 | 0.8019 | 0.3229 | 0.6042 | |
| 0.7824 | 0.8802 | 0.8284 | 0.2448 | 0.5625 | |
| 0.8400 | 0.8750 | 0.8571 | 0.1667 | 0.5208 | |
| 0.8830 | 0.8646 | 0.8737 | 0.1146 | 0.4896 | |
| 0.9213 | 0.8542 | 0.8865 | 0.0729 | 0.4635 |
| Policy | Metric | Seed 13 | Seed 17 | Seed 23 | Mean | SD | Paired 95% CI | |
|---|---|---|---|---|---|---|---|---|
| No certificate | Success | 2550 | 0.7102 | 0.7122 | 0.7192 | 0.7139 | 0.0047 | |
| No certificate | Sem. F1 | 2550 | 0.6749 | 0.6836 | 0.6866 | 0.6817 | 0.0061 | |
| No certificate | False acc. | 2550 | 0.2964 | 0.2852 | 0.2812 | 0.2876 | 0.0079 | |
| Final-only | Success | 2550 | 0.7090 | 0.7102 | 0.7149 | 0.7114 | 0.0031 | |
| Final-only | Sem. F1 | 2550 | 0.6962 | 0.7037 | 0.7055 | 0.7018 | 0.0049 | |
| Final-only | False acc. | 2550 | 0.2598 | 0.2491 | 0.2468 | 0.2519 | 0.0069 |
| Policy | Bundle fail | Rerun fail | Tool/env. | Model/policy | Other | Joint replay rate | |
|---|---|---|---|---|---|---|---|
| Final-only | 7650 | 2 | 8 | 4 | 3 | 1 | 0.9987 |
| Action-level | 7650 | 3 | 13 | 6 | 5 | 2 | 0.9979 |
| Evidence-aware | 7650 | 2 | 10 | 5 | 4 | 1 | 0.9984 |
| Conflict-aware | 7650 | 3 | 15 | 7 | 5 | 3 | 0.9976 |
| Policy | Stale | Unauth. | Inject. resist. | Conflict | Clean | Perturb. |
|---|---|---|---|---|---|---|
| No certificate | N/A | N/A | 0.4387 | 0.5214 | 0.7421 | 0.3978 |
| Final-only | 0.9648 | 0.9461 | 0.6275 | 0.6628 | 0.7386 | 0.5983 |
| Action-level | 0.9826 | 0.9714 | 0.7452 | 0.7289 | 0.7457 | 0.6741 |
| Evidence-aware | 0.9913 | 0.9857 | 0.8496 | 0.8164 | 0.7512 | 0.7568 |
| Conflict-aware guard | 0.9947 | 0.9896 | 0.9078 | 0.8892 | 0.7445 | 0.8127 |
| Family | Attack/task | Paired clean | Observed numerator | Rate | Paired 95% CI |
|---|---|---|---|---|---|
| Stale version rejection | 190 | 190 | 189 | 0.9947 | |
| Unauthorized-span rejection | 192 | 192 | 190 | 0.9896 | |
| Prompt injection resistance | 320 | 320 | 296 | 0.9250 | |
| Tool injection resistance | 320 | 320 | 285 | 0.8906 | |
| Conflicting-source resolution | 352 | 352 | 313 | 0.8892 | |
| Clean task success | 544 | N/A | 405 | 0.7445 |
| Success | Sem. F1 | False acc. | Calls | Joint replay | Execution rerun | |||
|---|---|---|---|---|---|---|---|---|
| 0 | 0 | 0 | 0.7139 | 0.6817 | 0.2876 | 7.86 | N/A | 0.9980 |
| 1 | 0 | 0 | 0.7139 | 0.6817 | 0.2876 | 7.86 | 0.9987 | 0.9981 |
| 0 | 1 | 0 | 0.7296 | 0.8011 | 0.1349 | 5.29 | N/A | 0.9974 |
| 1 | 1 | 0 | 0.7312 | 0.8173 | 0.1198 | 5.34 | 0.9984 | 0.9972 |
| 0 | 0 | 1 | 0.7187 | 0.8384 | 0.0912 | 7.89 | N/A | 0.9969 |
| 1 | 0 | 1 | 0.7219 | 0.8576 | 0.0745 | 7.94 | 0.9978 | 0.9967 |
| Metric | Result |
|---|---|
| Exact event-trace match | 0.9986 |
| Exact action-sequence match | 0.9993 |
| Exact tool-call-sequence match | 0.9996 |
| Exact policy-visible observation match | 0.9990 |
| Exact final-answer match | 0.9992 |
| First-divergence rate | 0.0014 |
| Quantity | Mean per execution |
|---|---|
| Receipt bytes (bytes) | 1876 |
| Committed trace bytes (bytes) | 11243 |
| Serialization time (ms) | 0.19 |
| Online integrity verification (ms) | 0.47 |
| Bundle replay (ms) | 1.31 |
| Execution rerun (s) | 16.88 |
| Threshold | Precision | Recall | F1 | False acc. | Coverage |
|---|---|---|---|---|---|
| 0.25 | [1pt] | [1pt] | [1pt] | [1pt] | [1pt] |
| 0.50 (primary) | [1pt] | [1pt] | [1pt] | [1pt] | [1pt] |
| 0.75 | [1pt] | [1pt] | [1pt] | [1pt] | [1pt] |
| 1.00 | [1pt] | [1pt] | [1pt] | [1pt] | [1pt] |
| Claim or check | Required artifact | Current coverage |
|---|---|---|
| 128-row integrity | Raw rows, corpus manifest, verifier | Aggregates reported; full corpus bundle external |
| Matched cost | Matched policy rows and ledger | Metrics and report reported |
| Mutation boundary | Mutation config, cases, metrics | 160-trial aggregate reported |
| Curated calibration | Case labels, guard config, metrics | 20-pair fixture reported |
| Replay check | Oracle-free fixture, code, expected digest | One two-event case executable |
| Independent semantic study | Blinded labels and adjudication log | 384-pair metrics and bootstrap intervals reported |
| experiment | unit | measurement and evaluation contract |
|---|---|---|
| Certificate replay | execution | 128-row fixed-corpus receipt/replay, 160 mutation trials, and matched early-stop control. The title-copy adapter measures mechanical receipt integrity and lexical coverage under a trusted runner. |
| Mutation audit | mutation | 144/144 structural tamperings rejected and 16/16 lexical-adversarial claims accepted. The paired result identifies the lexical verifier’s semantic-support boundary. |
| Semantic calibration | claim–span | 20-case curated fixture: lexical rule 8/8/3/1 and conflict-aware guard 8/0/11/1. The fixture calibrates failure categories; independent adjudication is specified separately. |
| Independent adjudication | claim–span | Two blinded annotators and a third adjudicator; locked non-title split; semantic precision, recall, false acceptance, and reliability. |
| Model-backed agent | task–seed | Matched model, prompt, tool schema, and budget; task outcomes, semantic labels, replay, latency, and resource cost. |
| Dynamic documents | paired task | Frozen version and access state; clean and perturbed tasks; rejection, injection resistance, conflict resolution, and downstream success. |
AI Agent Auditing
AI agent auditing examines agents’ actions, tool use, evidence, and outcomes to assess safety, compliance, reliability, and accountability.
See all 262 papers