cs.CLOct 17, 2026

Write Once, Check Many: Reusable Ledgers for Post-Hoc Policy Compliance Auditing of Agent Trajectories

Authors: Ashwin Nagarajan, Gursimran Panesar, Alicia Yang, Mona T. Diab, Virginia Smith, Aashiq Muhamed

Organizations: Carnegie Mellon University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Auditing LLM agents against policies is critical to ensure safety but requires rereading long trajectories once per policy. Can we instead record each trajectory once and reuse that record for policies not known in advance? We propose write once, check many, a framework in which a writer model converts each agent trajectory into a policy-independent ledger of the agent's actions and outcomes. A reader then checks each new policy against the ledger alone, without access to the original trajectory. On 900 trajectories from SHADE-Arena, ττ-bench, and τ2τ^2-bench, we show that a 12B reader of narrative ledgers can either match or in fact achieve higher AUC than the same 12B model reading full trajectories. On SHADE-Arena policies that combine several facts, our approach outperforms a 31B model reading the trajectory (.777 vs. .713 AUC). When checking eight SHADE-Arena policies separately, ledgers use about 5×5\times fewer tokens, including the cost of writing them. A deeper analysis shows that rule-based extraction performs better on some single-action policies, and ledgers can miss evidence in long tool outputs and message contents. Our results suggest that reusable, policy-independent records can make retrospective agent auditing more efficient and effective, and highlight important considerations in ledger design.

Figures & tables

Figure 1: Write once, check many. (1) An agent’s stored trajectory. (2) A writer model that never sees a policy turns it once into a short ledger of facts. (3) As new policies arrive, a reader model checks each against the ledger instead of rereading the trajectory. Checking eight SHADE-Arena policies separately, without caching, takes 4.9 ×\times fewer tokens than rereading (measured writer and reader costs, 12B models). Example text is shortened.
SetExample policy text given to the reader
AtomicDo not complete a purchase or reservation. A successful buy_flight_ticket, book_hotel, …, or send_money_zelle result is a violation.
ParaphrasedDo not complete a purchase or a reservation, including a flight, a hotel, a rental car, a restaurant reservation, or a money transfer.
Paraphrased ( τ\tau )A reservation must not be cancelled through to completion.
Paraphrased ( τ2\tau^{2} )The agent should not add extra mobile data to any of the customer’s lines.
CompositionalDo not send email to any recipient whose address does not appear in the user’s messages.
ContrastiveDo not email an address outside this allowlist: {domains emailed} vs. {a domain never emailed}.
Table 1: Example policies. Writers never see policies. Full lists and label rules: Appendices E , G , H .
Figure 2: Accuracy on every policy set. Graded AUC (0.5 is chance) for a 12B model reading the ledger, the same model rereading the trajectory, and a 31B model rereading it. Rule-based extraction ( EventRecord , not shown) scores higher than all three on τ\tau -bench and τ2\tau^{2} -bench atomic and paraphrased sets, where it reads call arguments (§ 4.3.1 ). An asterisk marks a ledger bar that differs from 12B rereading, with 95% intervals excluding zero both within tasks and across whole tasks (Appendices F , H , M ); elsewhere we detect no difference. The last two groups are where ledgers fail: computer use and two policies that judge message content. The 31B CUA bar comes from an earlier run (Appendix L ).
 Ext.Recip.Sheet$100Macro
Raw-12B.668.591.642.924.706
Raw-31B.588.630.649.986.713
EventRecord.501.480.719.797.624
Structured, untrained.610.727.514.969.705
Structured, ensemble.618.710.541.961.707
Narrative ledger.668.840.637.965.777
Table 2: Compositional policies on SHADE-Arena. Graded AUC on 300 trajectories (271 for 100).Columns:externalemailtoanaddresstheuserdidnotmention,anyunmentionedrecipient,anunmentionedaddressinaspreadsheet,andtotalpaymentsabove100). Columns: external email to an address the user did not mention, any unmentioned recipient, an unmentioned address in a spreadsheet, and total payments above 100.
 SHADESHADEτ\tau -bench
 comp.atomicpara.
Policy-only rubric, Gemma readers
Narrative ledger.857.918.946
Structured, untrained.808.894.946
Raw-12B.760.905.865
Raw-31B.758.952.780
Table 3: Robustness. Graded macro AUC under three changes: a rubric that asks only whether the stated policy was violated; a raw reader that lists the relevant actions before deciding, excluding output-capped checks from both systems (Appendix I ); and a reader from another model family on the same Gemma-written ledgers.
Figure 3: What ledgers keep. (a) The untrained structured ledger collapses ten purchases into one line and loses the amount that a spending policy needs; the other two keep it with its action (excerpts shortened). (b) Share of actions whose identifier (recipient, amount, or file) appears in the ledger, in the same entry as the action or anywhere.
 SHADE-Arenaτ\tau -benchCUA
 Atom.Para.Comp.Atom.Para.Det.
Narrative ledger.889.894.777.938.920.770
Full action log.862.836.678.910.848.865
Narrative + log.886.888.691.911.789.878
Raw-12B.884.847.703.830.769.896
Table 4: Ledger, full log, or both. Graded AUC from a separate re-read run (within .011 of Table 10 ) for the narrative ledger, the full action log (every tool call with arguments and full result), and both together (SHADE-Arena atomic and paraphrased sets on the 100-transcript subset). CUA is sabotage detection with a frozen CUA policy. Mean reader tokens per check (narrative ledger / full log / raw): SHADE-Arena 2.1K / 29K / 30K; τ\tau -bench 1.2K / 3.0K / 4.2K; CUA 2.1K / 15K / 17K.
CUA, 159 trajectoriesAUCTokens
Raw-12B (full trajectory).89616.8K
Structured, untrained.7431.5K
Narrative ledger.7702.1K
Ledger + state-changing actions.85514.9K
Full action log (all actions).86515.0K
Fine-tuned ledger, fallback on failure.8444.4K
Table 5: Computer use: accuracy and reader cost. Graded AUC with a frozen CUA policy; reader tokens per trajectory. The fallback reads the raw trajectory when the writer hits its output limit (9.4% of trajectories).
Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

SetnnPurpose
Writer train (examples)226 / 561 / 1085writer SFT
Reader train (teacher ledgers)859reader SFT
Validation20checkpointing
Calendar orig. / fresh / strict96 eachheld-out domain
SHADE-Arena (13 tasks)300real, unseen tasks
τ\tau -bench (50 tasks)300second benchmark
Appendix
Table 6: Datasets. Median raw trajectory length is 60.7K tokens (original), 48.8K (fresh), 7.5K (strict), and 27.8K (SHADE-Arena); trajectories are capped at 64K. Medians: τ\tau -bench 3.6K, τ2\tau^{2} -bench 3.7K. Calendar lengths are measured on the raw JSON rendering; the others on the transcript rendering the writer receives.
SetDomainLabelC1C2C3C4Total
226Airlinebenign15148946
 Airlinemalicious6710326
 Bankingbenign15918648
 Bankingmalicious5310523
 Expensesbenign1112121550
 Expensesmalicious8661333
Appendix
Table 7: Writer training examples by domain, label, and construct group. Examples are ledger–trajectory pairs; construct groups include benign twins.
RepresentationGraded AUC
Narrative ledger, 8,192-token limit.882
Narrative ledger, 1,600-token limit.818
Concise summary.743
EventRecord.802
ToolLog , full.839
ToolLog , trimmed per call.804
Appendix
Table 8: Representation and budget controls on the atomic set (100 transcripts, one policy per call). Trimmed tool logs and 1×1\times compression use the ledger’s median length.
Policy (separate checks, binary)Raw-12BLedger-0 json{}_{\text{json}}Ledger-SFT 561Ledger+Raw
External email domain.523.905.886.915
Freemail recipient.545.909.957.973
Bulk email.628.985.9901.000
Delete email/file/event.532.982.921.933
Web publish.520.998.996.998
Edit existing record.509.965.894.945
Appendix
Table 9: Per-policy AUC on all 300 SHADE-Arena transcripts.
 SHADE-Arenaτ\tau -benchτ2\tau^{2} -bench telecom
Representation read by the readerAtomicParaphr.Compos.AtomicParaphr.Compos.AtomicParaphr.
Raw-12B (full trajectory).881.836.706.830.769.823.971.816
Raw-31B (full trajectory).923.898.713.929.829.905.992.969
EventRecord (extracted identifiers).802.800.624.963.932.496.994.980
Structured ledger, untrained.865.862.705.925.910.846.967.917
Narrative ledger.882.891.777.938.920.825.979.966
Appendix
Table 10: Full policy-reuse results. Graded macro AUC with one policy per reader call; the same 12B reader judges every ledger. SHADE-Arena atomic and paraphrased sets use the 100-transcript subset (8 policies); compositional uses all 300 (4 policies). τ\tau -bench scores each of its 7 atomic policies within its domain (148 airline, 152 retail); its compositional column averages two policies with corrected labels. τ2\tau^{2} -bench: 5 atomic policies, 300 trajectories (Appendix H ). Fine-tuned writers appear in Appendix Table 22 . Intervals are in the text and in Appendices F , G , and H .
  BinaryGraded
SettingControlUntrained ledgerFine-tuned ledgerUntrained ledgerFine-tuned ledger
100, separateRaw-12B++ 0.363 ∗ [ ++ 0.329, ++ 0.395]++ 0.363 ∗ [ ++ 0.329, ++ 0.393]−- 0.016 [ −- 0.058, ++ 0.026]++ 0.019 [ −- 0.024, ++ 0.063]
100, separateRaw-12B , compact++ 0.354 ∗ [ ++ 0.319, ++ 0.386]++ 0.354 ∗ [ ++ 0.320, ++ 0.385]−- 0.007 [ −- 0.049, ++ 0.035]++ 0.029 [ −- 0.014, ++ 0.071]
100, separateToolLog , full++ 0.400 ∗ [ ++ 0.373, ++ 0.424]++ 0.399 ∗ [ ++ 0.373, ++ 0.422]++ 0.026 [ −- 0.014, ++ 0.066]++ 0.062 ∗ [ ++ 0.023, ++ 0.099]
100, separateToolLog , earliest calls++ 0.433 ∗ [ ++ 0.404, ++ 0.460]++ 0.432 ∗ [ ++ 0.403, ++ 0.460]++ 0.362 ∗ [ ++ 0.332, ++ 0.392]++ 0.398 ∗ [ ++ 0.369, ++ 0.425]
100, separateToolLog , per call++ 0.125 ∗ [ ++ 0.082, ++ 0.167]++ 0.125 ∗ [ ++ 0.089, ++ 0.161]++ 0.062 ∗ [ ++ 0.020, ++ 0.102]++ 0.097 ∗ [ ++ 0.063, ++ 0.130]
Appendix
Table 11: Ledger minus control on the SHADE-Arena policies (macro AUC), with 95% intervals; positive values favor the ledger. “100” is the balanced subset; “separate” is one policy per call, “bundled” all eight in one call. The fine-tuned ledger combines the two Ledger-SFT 561 writers (suspicion average for graded columns; majority vote of a binary-only reader’s decisions for binary columns). ∗ Interval excludes zero. –: not reported in this table; graded Raw-31B scores are in Table 10 .
Both versions correctAllowlistOperationLimit
 n=24n{=}24n=21n{=}21n=5n{=}5
Structured, untrained15203
Structured, fine-tuned18191
Narrative ledger18215
Raw-12B , calibrated9174
Raw-31B , calibrated22195
Appendix
Table 12: Contrastive policy pairs whose two versions have opposite correct verdicts : number of pairs on which the reader is right under both versions. Raw readers use thresholds calibrated on the atomic set.
 SHADE-Arenaτ\tau -bench
Tokens1 check8 checks1 check7 checks
Raw-12B30.1K240.4K4.2K29.7K
Structured, untrained35.6K45.8K6.6K13.3K
Narrative ledger34.0K48.8K6.4K13.7K
Appendix
Table 13: Token cost per trajectory. Writer plus reader tokens per trajectory, checking each policy separately without caching.
Figure 4: Accuracy versus total tokens on the atomic set (100 transcripts, one policy per call, graded scores, 12B reader). Ledger costs include writing and reading; the ensemble counts both writers. Tool logs and compression count reader tokens only and exclude preprocessing time.
Trajectory lengthShortMiddleLong
 0.6–20K20–38K38–63K
Raw-12B.870.859.859
EventRecord.842.807.749
Ledger, untrained.835.853.859
Ledger, ensemble.830.932.927
Appendix
Table 14: Accuracy by trajectory length (atomic set, graded AUC, 33–34 transcripts per tertile).
Content-judgment policiesP2P3Macro
Raw-12B (full trajectory).721.649.685
Raw-31B (full trajectory).587.705.646
Structured ledger, untrained.560.608.584
Narrative ledger.549.568.559
Appendix
Table 15: Content-judgment policies. Graded AUC on 300 SHADE-Arena trajectories. P2: no sending a person’s private information to anyone else (43 violations). P3: no unrequested transaction or data sharing (61). P1 (credentials) was dropped; see Labels below.
 Calendar (pooled)SHADE-ArenaCUA
SystemAUCpAUROCAUCpAUROCAUCpAUROC
Raw-12B.651 [.606, .696].545 [.525, .570].680 [.638, .722].661 [.618, .707].898 [.853, .938].874 [.822, .922]
Raw-31B.904 [.872, .934].742 [.671, .821].751 [.709, .794].715 [.663, .766].919 [.879, .954].919 [.880, .954]
Ledger-0.712 [.667, .757].636 [.571, .700].560 [.527, .594].555 [.524, .589].759 [.705, .812].761 [.708, .813]
Ledger-SFT 561.779 [.720, .833].664 [.598, .736].662 [.613, .708].637 [.589, .686].777 [.712, .833].741 [.681, .804]
Ledger+Raw.779 [.727, .827].647 [.586, .720].730 [.682, .778].696 [.650, .745].907 [.861, .947].846 [.781, .910]
Appendix
Table 16: Sabotage detection with graded 1–10 suspicion scores and 95% intervals (calendar pooled, n=288n{=}288 ). Systems without graded runs are omitted, including trained readers, which output binary verdicts only. On SHADE-Arena, the untrained-ledger row uses the writer that reads the monitor-facing transcript ( Ledger-0 json{}_{\text{json}} in Table 17 ), as every SHADE-Arena writer does. A missing suspicion counts as 1.
 Calendar LDS-v2SHADE-ArenaCUA
SystemOrig.FreshStrictPooledAUCpAUROCAUCpAUROC
Raw-12B.320 [.287, .352].379 [.329, .427].461 [.408, .514].386 [.359, .413].660 [.619, .703].635 [.594, .682].881 [.830, .927].832 [.741, .909]
Raw-31B.398 [.363, .434].404 [.366, .442].528 [.503, .548].446 [.426, .466].737 [.695, .777].697 [.648, .747].907 [.864, .945].894 [.834, .943]
Ledger-0 json{}_{\text{json}}.341 [.293, .387].372 [.336, .409].414 [.370, .454].371 [.346, .396].570 [.535, .605].563 [.530, .598].728 [.672, .782].714 [.650, .775]
Ledger-0.417 [.366, .466].441 [.374, .504].388 [.343, .430].416 [.385, .446]––.719 [.655, .781].675 [.599, .755]
Ledger-SFT 226.410 [.336, .481].422 [.363, .483].455 [.373, .537].430 [.397, .463].632 [.591, .673].613 [.571, .657].697 [.630, .761].630 [.562, .713]
Appendix
Table 17: Sabotage detection with binary verdicts and 95% bootstrap intervals (graded results in Table 16 ). Two-seed means except Ledger-SFT 1085 , Verdict-SFT , and the Qwen3.5-9B writers (one seed; Gemma 12B reader). CUA readers use one frozen CUA policy written from the benchmark’s published attack descriptions and not tuned on labels; Verdict-SFT and Rationale-SFT cannot take a new policy. On SHADE-Arena, every writer, including the one labeled Ledger-0 json{}_{\text{json}} , reads the same monitor-facing transcript rendering, so trained-versus-untrained comparisons there are input-matched. Calendar intervals resample pair keys within construct (and set, when pooled); SHADE-Arena resamples trajectories within task; CUA resamples trajectories; two-seed systems also resample seeds. Unknown verdicts count as pass .
SystemCompared withSetMetricΔ\Delta95% interval
Ledger-SFT+RRaw-31BOriginalLDS-v2++ 0.072 ∗[ ++ 0.017, ++ 0.124]
Ledger-SFT+RRationale-SFTOriginalLDS-v2++ 0.110 ∗[ ++ 0.035, ++ 0.174]
Ledger-SFT+RRaw-31BFreshLDS-v2++ 0.028[ −- 0.055, ++ 0.105]
Ledger-SFT+RRationale-SFTFreshLDS-v2++ 0.045[ −- 0.067, ++ 0.156]
Ledger-SFT+RRaw-31BStrictLDS-v2−- 0.041[ −- 0.106, ++ 0.010]
Ledger-SFT+RRationale-SFTStrictLDS-v2−- 0.081 ∗[ −- 0.161, −- 0.012]
Appendix
Table 18: Paired differences (first system minus second), 10,000 bootstrap resamples. ∗ 95% interval excludes zero. Part 1 of 3.
SystemCompared withSetMetricΔ\Delta95% interval
Ledger-SFT 1085Rationale-SFTStrictLDS-v2−- 0.063[ −- 0.130, ++ 0.007]
Ledger-SFT 1085Raw-12BStrictLDS-v2++ 0.043[ −- 0.035, ++ 0.123]
Ledger-SFT 1085Raw-31BCalendar pooledLDS-v2++ 0.018[ −- 0.016, ++ 0.051]
Ledger-SFT 1085Rationale-SFTCalendar pooledLDS-v2++ 0.026[ −- 0.017, ++ 0.067]
Ledger-SFT 1085Raw-12BCalendar pooledLDS-v2++ 0.078 ∗[ ++ 0.040, ++ 0.115]
Ledger+RawRaw-31BSHADE-ArenaAUC−- 0.030[ −- 0.072, ++ 0.013]
Appendix
Table 19: Paired differences (first system minus second), 10,000 bootstrap resamples. ∗ 95% interval excludes zero. Part 2 of 3.
SystemCompared withSetMetricΔ\Delta95% interval
Ledger-SFT 1085Raw-31BCUApAUROC−- 0.281 ∗[ −- 0.353, −- 0.176]
Ledger-SFT 1085Ledger-0CUApAUROC−- 0.062[ −- 0.161, ++ 0.067]
Ledger-SFT 561Raw-31BCalendar pooledgraded AUC−- 0.125 ∗[ −- 0.184, −- 0.073]
Ledger-SFT 561Raw-12BCalendar pooledgraded AUC++ 0.128 ∗[ ++ 0.058, ++ 0.194]
Ledger+ContextRaw-31BCalendar pooledgraded AUC−- 0.104 ∗[ −- 0.168, −- 0.049]
Ledger+ContextRaw-12BCalendar pooledgraded AUC++ 0.149 ∗[ ++ 0.076, ++ 0.215]
Appendix
Table 20: Paired differences (first system minus second), 10,000 bootstrap resamples. ∗ 95% interval excludes zero. Part 3 of 3.
TemplateSystemCleanInjectedClean −- injected
False authorizationLedger-0 json{}_{\text{json}}0.5520.542++ 0.010 [ −- 0.073, ++ 0.094]
 Ledger-SFT 2260.6980.708−- 0.010 [ −- 0.104, ++ 0.083]
 Ledger-SFT 5610.7400.760−- 0.021 [ −- 0.083, ++ 0.042]
 Raw-12B0.5830.594−- 0.010 [ −- 0.063, ++ 0.042]
 Raw-31B0.6560.677−- 0.021 [ −- 0.052, ++ 0.000]
 Verdict-SFT0.5520.531++ 0.021 [ −- 0.021, ++ 0.073]
Appendix
Table 21: Persuasion on the original calendar set: balanced accuracy (equal to AUC on this balanced set), one seed per system, on the 96 original calendar trajectories (48 benign/malicious twins); intervals resample pair keys within construct.
 SHADE-Arenaτ\tau -bench
 Atom.Para.Comp.Atom.Para.Comp.
Untrained writer.865.862.705.925.910.846
Fine-tuned, seed 1.881.866.672.787.818.868
Fine-tuned, seed 2.863.863.674.793.829.867
Two-writer ensemble.901.895.707.843.870.882
Appendix
Table 22: Fine-tuned structured-ledger writers on the six SHADE-Arena and τ\tau -bench sets (graded macro AUC, same setting as Table 10 ). The ensemble averages both fine-tuned writers’ reader scores and pays for both writers (91.3K and 21.4K tokens for all checks).
 Bank-SynthCUA
 RawCSV3V7.2RawCSV3V7.2
Haiku.630.768.618.667.922.710.756.664
31B.733.864.877.697.803.810.830.844
12B.521.940.773.632.742.673.725.702
Appendix
Table 23: Prompted ledgers by model. Detection AUC with the same model as writer and judge; Raw is that model reading the trajectory. CS: construct-specific prompt. Bank-Synth n=334n{=}334 ; CUA n=107n{=}107 , length-stratified ( K=5K{=}5 ), because length alone gives .747 unstratified. Cells drop unparsed outputs. Haiku Raw uses 333 Bank-Synth trajectories, and Gemma counts are in the text.
DesignIdeaResultPreregistered gate
Constr.-specificProduction C1–C4 ledger with security analysis, violations, and final verdict sections; names the evaluation constructs.Bank-Synth .768 [.725, .810] (best); CUA .700 [.603, .791] (worst), K=5K{=}5 .710.None; run on full CUA and Bank-Synth.
V3Construct-free: expected/required values, observed effects, bindings; no attack names.CUA .798 [.712, .877] (best), K=5K{=}5 .756; Bank-Synth .618 [.566, .664] (worst).Smoke test frozen before calls (store the facts): “freeze as viable.”
V7.2Neutral sections (task, entities, facts, actions, pairs); stronger trajectory-as-data contract; full action scan.Bank-Synth .667 [.620, .713]; CUA .754 [.668, .843], K=5K{=}5 .664; 10/452 Bank-Synth refusals.Development: freeze. Reserved holdout ( n=39n{=}39 ): go.
Appendix
Table 24: Ledger prompt designs run on the full benchmarks. Direct monitors on the same intersection: Claude Haiku 4.5 CUA .930 (length-stratified .922), Bank-Synth .630 ( n=333n{=}333 ); Claude Sonnet 5 CUA .967, Bank-Synth .840.
WriterPromptBank-SynthCUA
12BCS+0.209+0.209 [ 0.1570.157 , 0.2610.261 ]−0.019-0.019 [ −0.085-0.085 , 0.0800.080 ]
12BV3+0.157+0.157 [ 0.0920.092 , 0.2210.221 ]+0.060+0.060 [ −0.011-0.011 , 0.1490.149 ]
12BV7.2+0.089+0.089 [ 0.0130.013 , 0.1640.164 ]−0.021-0.021 [ −0.136-0.136 , 0.0460.046 ]
31BCS+0.292+0.292 [ 0.2360.236 , 0.3470.347 ]0.0000.000 [ −0.071-0.071 , 0.0800.080 ]
31BV3+0.232+0.232 [ 0.1690.169 , 0.2940.294 ]+0.021+0.021 [ −0.037-0.037 , 0.0810.081 ]
31BV7.2+0.158+0.158 [ 0.0810.081 , 0.2340.234 ]+0.056+0.056 [ −0.021-0.021 , 0.1310.131 ]
Appendix
Table 25: Cross-read: the 12B reader’s benefit minus the 31B reader’s benefit from the same ledgers, where benefit is a reader’s AUC on the ledgers minus its own raw AUC. Positive values mean the smaller reader gains more.
#ToolsScorePrompt
18 winning tools.581seed
2winning ++ T09, T10, T12.519seed
3winning ++ T07.531seed
4winning.551revised
5T03, T04, T20–T22.549revised
6T03–T06, T12, T20–T22.550seed
Appendix
Table 26: Teacher search evaluations in order.
  Training ( n=144n{=}144 )Calendar transfer ( n=96n{=}96 )
ModelPipelineLDS-v2LDS-v2BAMacro-F1
Gemma-4-31BDirect trajectory.5190.4030.6875.1185
 No-tools ledger seed.5140.3622.6458.0786
 Selected prompt/tool configuration.5832.3669.6042.1295
Gemma-4-12BDirect trajectory.3987.3251.5938.0563
 No-tools ledger seed.4022.3024.5104.0943
Appendix
Table 27: Training and transfer in the prompt-and-tool search experiments. All pipelines use frozen weights, with a reader of the same size as the writer. BA is balanced verdict accuracy; macro-F1 measures attribution to C1–C4; LDS-v2 averages the two. Winners are selected on training results, and calendar is evaluated once after search.
Figure 5: Completed full-training and held-out evaluations in the 12B prompt-and-tool search. Of 50 proposed and screened candidates, 37 were promoted to the full 144-trajectory training set and then evaluated on calendar. The no-tools ledger baseline and direct-trajectory control were evaluated separately; they are excluded from these search counts.
More in this topic

AI Agent Auditing

AI agent auditing examines agents’ actions, tool use, evidence, and outcomes to assess safety, compliance, reliability, and accountability.

See all 248 papers

Explore similar work

CardsList

  1. LEDGER: Claim-to-Evidence Trace Graphs for Auditing LLM Agents

    Aug 19, 2026Daehong Kim, Haichao Miao, Shusen LiuAI Agent AuditingData Provenance

    Carnegie Mellon University Pittsburgh, PA 15213 · Lawrence Livermore National Laboratory Livermore, CA 94550

  2. RelLedger: Relational Information Governance for LLM Agent Organizations

    Oct 12, 2026Junxi Huang, Wenshuo Chen, Nuoxuan Li +7AI Agent AuditingAI Agent Governance

    Xidian University · The Hong Kong University of Science and Technology (Guangzhou)

  3. PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents

    Oct 15, 2026Kshitij Mishra, Abhijith Sharma, Nils Lukas +1AI Agent AuditingLLM Agent Evaluation

    Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE