CascadeLedger: Counterfactual Auditing of Information and Resource Integrity in LLM Cascades

Miaobo Hu; Shuhao Hu; Xiaobo Guo; Xin Wang; Bokun Wang; Daren Zha

cs.CROct 14, 2026

CascadeLedger: Counterfactual Auditing of Information and Resource Integrity in LLM Cascades

Authors: Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang, Daren Zha

Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China

Abstract

We introduce CascadeLedger, a counterfactual execution-audit framework for budgeted language-model cascades. The protocol enforces call-before-read semantics, binds successful reads to service and cache identities, records availability and component-wise budget transitions, and freezes the event trace before any target or task scorer is joined. The central object is aggregate-equivalent trace validity: two executions can share a final answer, declared resource projection, and service histogram while differing in whether their routing decisions used only legitimately observable information. We complement single-trace invariants with counterfactual prefix non-interference (CPNI), which couples a visible prefix and policy randomness while changing only a declared hidden family. The counterfactual influence frontier (CIF) localizes the earliest violating prefix and a minimal hidden witness; projection-preserving adversarial mutation (PPAM) searches for invalid traces that remain acceptable to a non-counterfactual checker. Across frozen-cache, independently generated, and serving evaluations, the two-layer audit reports a clean false-positive rate of 0.0000, detects 0.9983 of serialized faults and 0.9982 of unlogged hidden-read faults, and changes 0.3403 of corrupted quality-resource conclusions after deterministic recomputation. The online recorder and offline CPNI audit have separate overhead measurements. Learned routing and semantic verification provide downstream compatibility studies under the same execution contract.

Figures & tables

Figure 1: CascadeLedger overview. Online execution (panels 1–5) reveals a cached candidate only after its paid call; the router reads the observed state, stops when small-model confidence reaches $\tau_{a}$ or verifier confidence reaches $\tau_{v}$ , and records unavailable or unaffordable calls as skip or fallback events. The trace is frozen and hashed (panel 6) before the post-freeze audit (panel 7) joins the target, scores the returned candidate, and runs the invariant and counterfactual checks.

measurement	value or unit	evaluation object
$m_{\mathrm{proxy}}$	match fraction	cache–target proxy match
$c_{\mathrm{decl}}$	ledger units	declared resource accounting
$\ell_{\mathrm{decl}}$	accounting ms	declared latency component
$d_{\mathrm{ledger}}$	detection fraction	detected/injected mutations
$m_{\mathrm{task}},\ell_{\mathrm{live}}$	normalized task accuracy; ms/request	held-out task score and p50/p95 latency for the learned router

Table 1: Measurement objects and evaluation units. Task scores and latency share a request denominator but use separate scoring and timing interfaces.

policy	action rule	fallback / budget rule
one-shot	Pay small and return it.	Return no candidate if small is blocked.
fixed	Pay small; if $\kappa_{s}<\tau_{a}$ , pay large directly.	Return small if large is unavailable or blocked.
sequential	Pay small; if rejected, pay verifier. A passing verifier stops with small; otherwise pay large.	Return the last candidate after a blocked escalation.
budget-conditioned	Pay verifier only when it and a possible large call fit; otherwise prefer a direct large call.	Same fallback; budget is checked before every call.

Table 2: Observed-only policy definitions. A verifier call never reveals a target and, in the sequential policy, supplies only a stop signal.

invariant	required relation
oracle isolation	Online rows contain no target/oracle fields or future service outputs.
path–action alignment	$p_{T}$ equals the ordered service IDs of successful call actions.
action schema	Every event has a supported action/service form; verifier events follow a verifier call.
action order	Accepted ledger entry $k$ has action ID service:j:k for its path position.
cache binding	A called observation and final candidate match the cache digest for $(i,j)$ .
ledger reconciliation	Spent cost equals the component-wise sum of accepted service-call costs and the row cost.

Table 3: Machine-checkable ledger invariants. The checker is applied before the offline oracle join; the mutation audit tests the first five failure modes below while the clean replay checks all eight.

Policy	Score	Calls	Tokens	p50	p95
One-shot	0.5823	1.000	236	41.8	72.9
Fixed cascade	0.6936	1.608	462	76.4	151.3
Sequential	0.7043	2.096	518	89.7	172.4
Budget-conditioned	0.7031	1.884	486	82.6	159.1
Learned router	0.7195	1.642	427	68.7	134.2

Table 4: Independent held-out comparison on 1,319 records per seed. Score is normalized task accuracy (

\uparrow

); calls, tokens, and p50/p95 latency in milliseconds are per request (

\downarrow

). Bold marks the best and underline the second-best value in each column.

Figure 2: Held-out task score against generated tokens and p95 latency. Points summarize router seeds 13/17/23; latency is the per-policy request p95. Paired score intervals and coverage are in Table 38 .

Shift	Policy	Proxy	Units	p95 ledger ms	Unavail.
baseline	one-shot	0.2109	4.00	2.00	0.0000
	fixed	0.3203	12.34	10.00	0.0000
	sequential	0.3203	13.73	11.00	0.0000
large unavailable	one-shot	0.2109	4.00	2.00	0.0000
	fixed	0.2109	4.00	2.00	0.6953
	sequential	0.2109	5.39	3.00	0.6953

Table 5: Cascade match rate and ledger stress response on the primary seed-13 cache (128 records). Proxy is exact cache–target match; units and milliseconds are declared ledger quantities. The large-cost

+50\%

row applies the joint shift in Table 10 . Bold marks the maximal proxy and minimal resource/event rates within each shift.

Figure 3: Audit-component recall on matched injected faults ( $\uparrow$ ). Each row removes one check from the full audit; colors share a 0–1 scale and bold marks column maxima. Values are point recall estimates from Table 23 . Removing CPNI eliminates unlogged-read and joint hidden-influence detection while leaving structural checks intact; removing joint interventions affects only the interaction-only family.

Appendix figures & tables51 assets

Supplementary material from the paper’s appendix.

Appendix

symbol	meaning	phase
$q_{i}$	query text for record $i$	online
$C_{i}(j)$	cached observation returned by service $j$	paid call
$A_{t}(j)$	availability bit for service $j$ at step $t$	online
$b_{t}$	component-wise remaining ledger budget	online
$p_{t}$	ordered successful-call path	online
$T_{i}$	complete event trace for record $i$	trace freeze

Appendix

Table 6: Notation used by the replay contract. The online state contains the first five entries; oracle and live fields are joined after routing.

index	action / debit ID	observations	units	ms	path
initial	request small	$\emptyset$	8	6	$[]$
0	call small / service:small:0	$O_{s}$	6	5	$[s]$
1	call verify / service:verify:1	$O_{s},O_{v}$	5	4.5	$[s,v]$
2	verify: reject	$O_{s},O_{v}$	5	4.5	$[s,v]$
3	large unavailable	$O_{s},O_{v}$	5	4.5	$[s,v]$
return	fallback to small	$O_{s},O_{v}$	5	4.5	$[s,v]$

Appendix

Table 7: Public trace example. Index is zero-based within actions ; debit suffix

k

is the paid-path position.

O_{s},O_{v}

are paid observations; units and ms are remaining budget.

field	content	written by	validation
query_id	source-relative record handle	producer	manifest membership
action position / action_id	list index / debit identifier	router / ledger	action order and accepted path index
service_id	small, verify, or large	router	service catalog
action	call, verify, unavailable, budget_skip, or budget_exhausted	router	serialized action schema
terminal fields	answer, abstained, unavailable, budget_exhausted	router	stop/fallback decision
cache_digest / read_set	cache binding / paid services	recorder	observation and read-set consistency

Appendix

Table 8: Serialized replay schema and evaluation extensions. Action positions and debit IDs have distinct roles; live timing attaches to the same request identity.

artifact	selection	held-out	seed	role	status
primary route cache	n/a	128	13	policy/shift replay and proxy table	complete
split-aware gate	64	128	23	feature/cost diagnostic	complete
early threshold gate	64	128	17	threshold selection audit	no selection
supplemental gate	256	1,319	29	paired threshold recommendation	pair selected
live routing study	1,280/seed	1,319/seed	13/17/23	independent task and serving evaluation	complete

Appendix

Table 9: Primary, diagnostic, supplemental, and task-study partitions. Each completed replay row has its own source manifest and digest.

service	baseline units	baseline ms	large-shift units	large-shift ms
small	4	2	4	2
verify	2	1	2	1
large	12	8	18	12

Appendix

Table 10: Declared service catalog for the primary frozen replay. The joint shift multiplies large-service units and accounting milliseconds by 1.5. Appendix E defines generated-token, monetary, and measured-latency comparisons.

mutation	ledger rejects	aggregate unchanged	violated invariant(s)
target leak	yes	yes	oracle isolation
future-output leak	yes	yes	oracle isolation
action reorder	yes	yes	path–action alignment; cache binding; budget conservation
remaining-budget corruption	yes	yes	budget conservation
cache-ID swap	yes	yes	cache binding

Appendix

Table 11: Aggregate-preserving fault injection on a representative frozen trace. “Aggregate unchanged” means that the path/cost/event-count projection is identical before and after mutation.

family	mutation operation	invariant	preserved aggregate	recall
target leak	copy the joined answer into an online row	oracle isolation	path, cost, event count	1.0000
future output	insert an uncalled service output	oracle isolation	path, cost, event count	1.0000
action reorder	swap two successful calls	action order and path alignment	service histogram and cost	1.0000
budget corruption	alter remaining budget after a call	budget conservation	spent cost and path	1.0000
cache swap	replace an observation digest	cache binding	service histogram and cost	1.0000
retry omission	remove a retry event while keeping final output	retry accounting	final candidate and service count	1.0000

Appendix

Table 12: Declared mutation families and invariant targets. The first five rows use the private frozen-cache campaign; additional rows test serving events. Recall is detected/injected faults within each family.

Figure 4: Structural mutation audit. The event-level ledger detects the declared private and public mutation families, while the aggregate-only projection detects none.

category	event pattern	count
small-only	call small; stop	39
verifier stop	call small; call verifier; stop	0
large escalation	call small; call verifier; call large	89
availability fallback	call small; skip large/unavailable	89 under shift
budget fallback	call small; skip large/budget	89 under shift
semantic error	task scorer marks returned answer incorrect	370/1,319 per seed

Appendix

Table 13: Route taxonomy for the primary seed-13 cache and independent task analysis. Counts use the named policy and shift; task errors and timeouts use the task-study request set.

Case	Boundary event	Returned candidate	Task accuracy	Units	Latency	Repair action
small-only	threshold accept	small	0.846	4.00	42.1 ms	retain or adjust threshold
verifier stop	verifier accept	small	0.807	6.00	58.4 ms	calibrate verifier
large escalation	verifier reject	large	0.726	18.00	129.7 ms	compare matched baseline
availability fallback	service unavailable	last candidate	0.544	6.00	64.8 ms	outage policy
budget fallback	budget skip	last candidate	0.557	6.00	61.5 ms	budget schedule
timeout/retry	timeout event	retry/ fallback	0.601	7.42	236.2 ms	retry policy

Appendix

Table 14: Per-record error analysis. Quantitative fields report the independent task scorer and serving telemetry summaries.

method	contract	trace recall	hidden recall	bytes/ request
Aggregate summary	final answer, calls, and cost only	0.0000	0.0000	184
Structured logger	serialized event rows without checks	0.0000	0.0000	1,168
Schema validator	action and field schema	0.3618	0.0000	1,206
Hash/cache binding	schema plus service/cache digests	0.5746	0.0000	1,542
Single-trace ledger	binding, paid reads, order, and budget	0.9983	0.0000	2,074
Ledger + CPNI	ledger plus paired hidden-state replay	0.9983	0.9982	2,248

Appendix

Table 15: Audit baseline ladder. Each row exposes the contract available to the checker; recall and clean false-positive rate are measured on the same clean and faulty traces.

method	target	future verifier	future large	future queue
Aggregate summary	0.0000	0.0000	0.0000	0.0000
Schema validator	0.0000	0.0000	0.0000	0.0000
Single-trace ledger	0.0000	0.0000	0.0000	0.0000
Ledger + CPNI	1.0000	0.9974	0.9987	0.9967

Appendix

Table 16: Detection of unlogged hidden influences. Columns report recall for the indicated fault family; clean divergence is measured on paired executions without an inadmissible dependency.

method	direct	target bit	compressed	joint
Aggregate summary	0.0000	0.0000	0.0000	0.0000
Schema validator	0.0000	0.0000	0.0000	0.0000
Single-trace ledger	0.0000	0.0000	0.0000	0.0000
Ledger + marginal CPNI	0.9984	0.9961	0.9948	0.0000
Ledger + CPNI + joint interventions	0.9986	0.9964	0.9952	0.9929

Appendix

Table 17: Detection of direct, transformed, and interaction-only hidden influence. Values are recall on the corresponding injected family.

method	delayed	stochastic
Aggregate summary	0.0000	0.0000
Schema validator	0.0000	0.0000
Single-trace ledger	0.0000	0.0000
Ledger + marginal CPNI	0.9947	0.9713
Ledger + CPNI + joint interventions	0.9951	0.9746

Appendix

Table 18: Detection of delayed and stochastic hidden influence. Clean divergence is measured on fault-free paired executions.

fault population	family accuracy	exact prefix	prefix MAE	replays/trace
Direct hidden reads	0.9984	0.9940	0.006	6.31
Transformed hidden reads	0.9946	0.9869	0.018	7.08
Joint hidden reads	0.9897	0.9752	0.039	9.62
Stochastic hidden reads	0.9698	0.9437	0.072	8.84
All declared families	0.9881	0.9750	0.034	7.96

Appendix

Table 19: Counterfactual influence frontier localization. Prefix error is measured in routing decisions and replay count includes all paired interventions.

projection	generated	baseline pass	full recall	conclusion changes	search success
$\Phi_{1}$	1,012	1.0000	0.9980	0.3656	0.9414
$\Phi_{2}$	947	1.0000	0.9979	0.3464	0.8877
$\Phi_{3}$	823	1.0000	0.9976	0.3220	0.8037

Appendix

Table 20: Projection-preserving adversarial mutation. Baseline pass is the fraction of generated invalid traces accepted by the strongest non-counterfactual checker.

pairing scheme	clean divergence	hidden-fault recall	replay variance
Independent randomness	0.0786	0.9980	0.0734
Common randomness	0.0000	0.9982	0.0013

Appendix

Table 21: Effect of coupling policy randomness in CPNI. Replay variance is the variance of the paired action-difference estimator.

setting	stages	held-out requests	serialized recall	hidden recall	exact prefix
Three-stage reference	3	1,319	0.9983	0.9982	0.9746
Longer-horizon cascade	6	1,319	0.9974	0.9968	0.9613

Appendix

Table 22: Longer-horizon cascade transfer. Recall uses the corresponding injected-fault denominator; exact-prefix accuracy uses traces with a known violating prefix.

configuration	leakage	order	binding	budget	unlogged read	joint
Full audit	1.0000	1.0000	0.9991	0.9973	0.9982	0.9929
$-$ oracle isolation	0.4876	1.0000	0.9991	0.9973	0.9982	0.9926
$-$ path/action order	1.0000	0.4167	0.9984	0.9958	0.9982	0.9918
$-$ cache binding	1.0000	0.9992	0.3281	0.9973	0.9982	0.9923
$-$ paid-read consistency	0.7124	0.9986	0.9979	0.9951	0.9982	0.9907
$-$ budget conservation	1.0000	0.9978	0.9987	0.3047	0.9982	0.9921

Appendix

Table 23: Leave-one-component-out audit recall. The full row uses all event invariants and CPNI; each subsequent row removes one check.

fault family	$N$	full recall	ACR	changed conclusion
Serialized structural faults	1,024	0.9980	0.3018	309/1,024
Unlogged hidden reads	768	0.9987	0.3490	268/768
Compound read–budget faults	512	0.9980	0.4043	207/512
All declared families	2,304	0.9983	0.3403	784/2,304

Appendix

Table 24: Audit consequence results. Recall measures rejected faulty traces; ACR measures the fraction for which accepting the invalid trace changes the predeclared conclusion.

source	fault population	$N$	log-only recall	full recall	ACR
Independent generator	property-based and faulty-router implementations	1,856	0.7430	0.9978	0.3254
Natural implementation set	issue- or history-derived faults	128	0.7812	0.9922	0.3516

Appendix

Table 25: Independent and natural fault generalization. Each recall uses its own clean and faulty denominator; ACR is computed with the same conclusion operator.

layer	events/request	bytes/request	median ms	p95 ms	throughput
Online recorder	4.31	1,168	0.24	0.57	24,800
Hash/cache binding	4.31	1,542	0.41	0.89	15,600
Single-trace replay	4.31	2,074	0.96	2.14	6,950
CPNI replay (full)	8.62	2,248	4.68	10.72	1,480
CPNI replay (sampled)	5.38	2,116	1.63	3.86	4,360

Appendix

Table 26: Audit overhead by layer. Latency columns are median/p95 per request or trace, and throughput is measured on the declared audit machine.

configuration	prefixes	interventions /trace	recall	p95 ms/trace	replays /trace
Single-trace only	0	0	0.6653	2.14	1
CPNI sampled	1.36	4.38	0.9836	3.86	5.38
CPNI full	2.41	7.62	0.9983	10.72	8.62
CPNI + CIF	2.41	7.96	0.9983	13.48	8.96

Appendix

Table 27: Audit cost as intervention coverage increases. Runtime is p95 milliseconds per trace; replay count includes the original execution. The zero-intervention row checks serialized events.

shift	policy	proxy	units	unavail./failure	budget/failure
baseline	sequential	0.3203	13.73	0.0000	0.0000
baseline	learned gate	0.3203	15.91	0.0000	0.0000
large unavailable	learned gate	0.2109	4.00	0.9922	0.0000
tight budget	learned gate	0.2109	4.00	0.0000	0.9922
large cost $+50\%$	learned gate	0.3203	21.86	0.0000	0.0000

Appendix

Table 28: Learned-gate cache audit. The fit and tune records are disjoint from the 128 held-out records; the first event column reproduces the separate runner’s unavailable or route-failure field, while canonical event semantics are defined above.

Split	Records	Producer seeds	Router seeds
Train	1,024	31/37/43	13/17/23
Calibration	256	47/53/59	13/17/23
Held-out	1,319	61/67/71	13/17/23
Live shadow	512	73/79/83	13/17/23

Appendix

Table 29: Data partitions and independent generation seeds. The task scorer and the Qwen3.5-0.8B/2B model pair are version-bound across all policies. Record counts are per router seed.

Condition	p50 (ms)	p95 (ms)	Timeout/ retry (%)	Paired CI
Baseline	41.8–89.7	72.9–172.4	0.4–1.8	$[+0.0061,+0.0244]$
Large unavailable	40.9–63.6	71.5–121.8	0.5–2.1	$[+0.0108,+0.0327]$
Tight budget	40.7–61.9	70.8–118.6	0.4–1.6	$[+0.0094,+0.0305]$
Price $+50\%$	42.2–93.4	74.1–181.7	0.5–1.9	$[+0.0058,+0.0240]$
Cross-cache	43.6–96.8	76.4–187.2	0.7–2.4	$[+0.0032,+0.0218]$

Appendix

Table 30: Serving latency and event rates over five policies. p50 and p95 columns give policy ranges in milliseconds; timeout/retry entries are percentages. Paired task-score intervals compare learned with sequential routing.

Figure 5: Learned-minus-sequential paired task-score intervals (95%, hierarchical bootstrap). Left: train, calibration, and held-out splits, with filled markers at the reported score difference. Right: held-out serving conditions, with hollow markers at the interval midpoint. The dotted line marks the

-0.05

non-inferiority margin.

hypothesis	acceptance criterion	rejection condition
H1 verifier stop	Independent task score non-inferior within $-0.05$ and measured cost lower	quality lower bound $<-0.05$
H2 transfer	Paired task-score delta non-inferior on router seeds 13/17/23	sign changes or harmful CI
H3 audit generalization	clean false positive 0 and mutation recall $\geq 0.995$	hidden/future-read mutation escapes
H4 cost calibration	declared cost predicts measured latency with predeclared calibration	correlation reverses under a shift

Appendix

Table 31: Evaluation hypotheses and acceptance criteria.

Partition	Data and scorer	Budget and pairing	Shift boundary
Train	Independent seeds; version-bound scorer; fit router	Frozen quality/token budget; shared queries	Baseline and declared offline shifts
Calibration	Independent seeds; select thresholds	Matched budget; disjoint calibration records	Calibration cache and fixed shift grid
Held-out	Frozen generation seeds and scorer	Matched budgets; query/seed pairs	Held-out cache and service shifts
Live	Version-bound model and scorer on served outputs	Matched budget; repeated serving windows	Outage, price, cache, and traffic conditions

Appendix

Table 32: Split and provenance protocol. Threshold selection is confined to calibration; held-out and serving comparisons preserve query and seed pairing.

Policy	Verifier use	Route decision	Serving controls
One-shot	No verifier call	Return the small candidate	Fixed arrival, timeout, and retry rules
Fixed cascade	Direct small/large decision	Escalate at the small threshold	Shared request mix and service versions
Sequential	A verifier can accept or reject	Stop, escalate, or fall back	Shared queueing and failure rules
Budget-conditioned	Same verifier contract	Condition route on remaining budget	Same budget and service grid
Learned router	Evaluation labels stay outside online state	Fit on train; tune on calibration	Measured queueing, retries, and timeouts

Appendix

Table 33: Baseline and verifier contracts. All policies share the declared budget, model versions, request mix, and scorer for each comparison.

Train
Policy	Tokens	USD	p50 (ms)	p95 (ms)	Req/s
One-shot	231	0.00018	40.9	70.8	24.3
Fixed	452	0.00069	74.8	147.1	13.4
Sequential	507	0.00079	87.5	168.3	11.7
Budget-conditioned	474	0.00073	80.1	155.0	12.6
Learned router	414	0.00062	65.4	129.1	15.3

Appendix

Table 34: Complete resource result surface by split and policy. Tokens, USD cost, and p50/p95 latency are per request; throughput is requests/s. Bold marks the best value within each split. Task quality and coverage use the same request denominator as Table 38 .

Condition	Task score	Coverage / abstention	Calls	Failure
Baseline	0.5823–0.7195	0.9947–0.9992 / 0.0008–0.0053	1.000–2.096	0.003–0.012
Large unavailable	0.5823–0.6648	0.9910–0.9991 / 0.0009–0.0090	1.000–1.476	0.002–0.608
Tight budget	0.5823–0.6701	0.9924–0.9990 / 0.0010–0.0076	1.000–1.508	0.003–0.591
Price $+50\%$	0.5823–0.7191	0.9942–0.9991 / 0.0009–0.0058	1.000–2.088	0.004–0.015
Cross-cache	0.5634–0.7016	0.9918–0.9986 / 0.0014–0.0082	1.000–2.114	0.006–0.031

Appendix

Table 35: Task and route behavior across serving conditions. Each cell is the range over the five policies. Coverage and abstention use all requests as denominator. Failure includes recorded service or budget events, whereas timeout/retry rates are separated in Table 30 .

shift	policy	proxy	units	depth	p95 ms	unavail.
baseline	one-shot	0.2109	4.00	1.00	2.00	0.0000
baseline	fixed	0.3203	12.34	1.70	10.00	0.0000
baseline	sequential	0.3203	13.73	2.39	11.00	0.0000
large unavailable	one-shot	0.2109	4.00	1.00	2.00	0.0000
large unavailable	fixed	0.2109	4.00	1.00	2.00	0.6953
large unavailable	sequential	0.2109	5.39	1.70	3.00	0.6953

Appendix

Table 36: Complete frozen-cache shift ledger for the primary 128-record cache. Proxy is an exact cache–target match; units and milliseconds are declared ledger fields.

artifact	train	held-out	pair	Proxy/ task	selection rule
early gate	64	128	none	0.3203	quality and cost gate
supplemental gate	256	1,319	$(0.95,0.40)$	0.3374	4,000 paired bootstrap replicates
seed 13 live	1,024	1,319	$(0.70,0.45)$	0.7210	maximize calibrated utility subject to 95% paired lower bound $\geq-0.05$
seed 17 live	1,024	1,319	$(0.75,0.45)$	0.7172	maximize calibrated utility subject to 95% paired lower bound $\geq-0.05$
seed 23 live	1,024	1,319	$(0.70,0.50)$	0.7204	maximize calibrated utility subject to 95% paired lower bound $\geq-0.05$

Appendix

Table 37: Threshold transfer by artifact. The supplemental pair is selected on its frozen 256-record partition; each live seed has its own calibration protocol.

Figure 6: Proxy match versus declared cost. The primary seed-13 frontier and split-aware seed-23 gate use separate caches. The learned gate has 41/128 matches and higher cost than sequential routing, which has 42/128.

Policy	Score	Cover.	Abst.	Calls	Failure	Paired CI
Train
One-shot	0.5964	0.9990	0.0010	1.000	0.0020	$[-0.1291,-0.1116]$
Fixed	0.7048	0.9976	0.0024	1.594	0.0046	$[-0.0196,-0.0044]$
Sequential	0.7168	0.9968	0.0032	2.101	0.0078	$[0.0000,0.0000]$
Budget-conditioned	0.7156	0.9971	0.0029	1.889	0.0059	$[-0.0059,+0.0035]$
Learned router	0.7421	0.9983	0.0017	1.626	0.0037	$[+0.0162,+0.0344]$

Appendix

Table 38: Task score and route behavior by split and policy. Coverage, abstention, and failure are fractions of requests; calls are per request. Paired confidence intervals are task-score differences relative to sequential routing. Bold marks the best score, coverage, abstention, calls, and failure within each split. Resource metrics use the same split in Table 34 .

(a) Resource use
Condition	Units	Tokens	USD	GPU-hours
Baseline	4.00–13.82	236–518	0.00018–0.00081	0.36–0.82
Large unavailable	4.00–6.18	236–318	0.00018–0.00029	0.27–0.49
Tight budget	4.00–6.42	236–327	0.00018–0.00031	0.26–0.47
Price $+50\%$	4.00–18.91	236–518	0.00018–0.00103	0.37–0.89
Cross-cache	4.00–14.26	244–536	0.00019–0.00085	0.39–0.93

Appendix

Table 39: Live serving measurements across five policies. Each range spans policies within a condition. Panel (a) gives mean request resources and total GPU-hours; panel (b) gives per-policy queue/service p50/p95 milliseconds and throughput in requests/s.

axis	paired comparison	result
model version	same queries, new small/large hashes	paired task delta $+0.0126$ ; no policy-order reversal
cache producer	independent generation seeds 13/17/23	learned-minus-sequential delta $+0.0121$ – $+0.0180$ across seeds
verifier schema	strict format versus semantic scorer	semantic verifier gains $+0.0187$ task score with 17.4% fewer large calls
service availability	baseline versus outage windows	paired quality remains within the $-0.05$ margin; outage events remain explicit
budget	canonical versus tight budget vectors	budget-aware routes retain a $+0.010$ – $+0.024$ paired advantage over sequential fallback
traffic	fixed request mix versus repeated live windows	Spearman $\rho=0.86$ – $0.91$ with no calibration sign reversal

Appendix

Table 40: Robustness and transfer axes for the task and serving study.

Fault / preserved projection	Evaluation	Detection rate	Task delta	Cost change	Policy ranking
Hidden paid read	aggregate-only	0.0000	+0.027	$-19.1\%$	$F\succ L\succ S$
path, cost, event count	CascadeLedger	1.0000	+0.015	$-17.6\%$	$L\succ S\succ F$
Stale cache binding	aggregate-only	0.0000	+0.031	$-16.4\%$	$S\succ L\succ F$
service histogram, cost	CascadeLedger	0.9991	+0.014	$-17.2\%$	$L\succ S\succ F$
Missing failure event	aggregate-only	0.0000	+0.022	$-24.8\%$	$L\succ F\succ S$
final candidate, path length	CascadeLedger	0.9984	+0.011	$-12.6\%$	$L\succ S\succ F$

Appendix

Table 41: Decision-value impact of aggregate-preserving faults. For each fault, the first row evaluates aggregate-only summaries and the second row evaluates CascadeLedger. Both evaluations have zero clean false positives.

F

S

B

, and

L

denote fixed, sequential, budget-conditioned, and learned routing. The preserved projection is listed in italics below each fault.

Family	Generator contract	$N$	Recall	Decision changes
held-out field mutation	independent map	320	319/320 (0.9969)	91/320 (0.2844)
property-based sequence	bounded grammar	384	384/384 (1.0000)	104/384 (0.2708)
semantic-equivalent trace	same output; invalid read	256	255/256 (0.9961)	83/256 (0.3242)
compound fault	read/order/budget mix	512	511/512 (0.9980)	207/512 (0.4043)
independently seeded router fault	independent faulty router	384	383/384 (0.9974)	119/384 (0.3099)

Appendix

Table 42: Held-out fault families for audit generalization. Each family uses

N

clean traces and

N

faulty traces; clean false positives are zero throughout. Recall and decision changes give count/

N

, followed by the fraction.

domain	model family	verifier	TP	FP	TN	FN
GSM8K	Qwen family	format control	62	31	118	45
GSM8K	Qwen family	semantic answer verifier	94	14	135	13
GSM8K	second model family	format control	57	35	113	51
GSM8K	second model family	semantic answer verifier	88	18	130	20
non-math task	Qwen family	format control	54	39	110	53
non-math task	Qwen family	semantic/process verifier	87	20	129	20

Appendix

Table 43: Nontrivial semantic-verifier and transfer experiment. Confusion fields use the verifier-labeled denominator; task and resource fields use the request denominator.

domain	family	verifier	ECE	task score	units	p95 ms	coverage
GSM8K	Qwen	format	0.187	0.7043	13.74	172.4	0.9947
GSM8K	Qwen	semantic	0.061	0.7230	11.35	148.6	0.9971
GSM8K	second family	format	0.204	0.6721	14.02	180.7	0.9939
GSM8K	second family	semantic	0.073	0.6954	11.84	154.9	0.9965
non-math	Qwen	format	0.221	0.6413	13.66	169.5	0.9928
non-math	Qwen	semantic	0.082	0.6661	11.58	146.2	0.9958

Appendix

Table 44: Semantic-verifier calibration and routing. ECE is expected calibration error on the labeled verifier sample; routing metrics use all requests. Bold marks the better value within each domain/family pair.

field	value
GPU model and count	$2\times$ NVIDIA A100 80GB PCIe
CPU and host memory	AMD EPYC 7543 32-core, 256 GB DDR4
inference engine and version	vLLM 0.10.1 with CUDA 12.4
small/large placement	Qwen3.5-0.8B + verifier on GPU0; Qwen3.5-2B on GPU1
batching policy and maximum batch	continuous dynamic batching; maximum batch 32
tensor/data parallel configuration	TP=1 per model, DP=1, no cross-GPU tensor sharding

Appendix

Table 45: Serving execution environment contract.

condition	method	$N$	Spearman	Pearson	slope	$R^{2}$	calib. error	shift resid.
baseline	five policies	19,785	0.914	0.923	6.18	0.852	5.7 ms	$+0.8$ ms
large unavailable	five policies	19,785	0.887	0.901	6.04	0.812	6.3 ms	$-1.4$ ms
tight budget	five policies	19,785	0.896	0.908	5.97	0.824	6.1 ms	$-0.9$ ms
price shift	five policies	19,785	0.921	0.929	6.11	0.863	6.0 ms	$+1.7$ ms
cross-cache	five policies	19,785	0.861	0.876	6.26	0.768	7.4 ms	$+2.6$ ms

Appendix

Table 46: Request-level declared-cost calibration under serving shifts.

field	contract
model class and architecture	two-layer MLP router, 64/32 hidden units with GELU, masked three-action output head
online input features	paid small/verifier confidence, extraction flags, normalized response length and token fraction, availability bits, remaining unit/ms budget, route depth, and missing-observation masks
feature normalization	training-split mean/standard deviation for continuous fields; binary state bits unchanged; missing paid observations represented by explicit masks
training objective and labels	cost-sensitive cross-entropy over the offline oracle-best admissible action with $\lambda=0.15$ normalized resource penalty; labels joined only after trace freeze
optimizer and learning rate	AdamW, learning rate $3\times 10^{-4}$
regularization and batch size	weight decay $10^{-3}$ , dropout 0.10, batch size 128

Appendix

Table 47: Independent learned-router fitting contract.

field	protocol
confidence level	95%
resampling method and replicates	paired hierarchical bootstrap with 5,000 replicates
primary resampling unit	query identifier, with all repeated policy outcomes kept in the same cluster
router-seed treatment	router seeds 13/17/23 retained as an outer resampling level
candidate-generation-seed variance	held-out producer seeds 61/67/71 treated as an independent random-effect level
hierarchical aggregation	outer router/producer seed resampling followed by inner paired-query resampling

Appendix

Table 48: Paired uncertainty protocol for the learned-minus-sequential comparison.

artifact	path	role	validation
manuscript	source entry in delivery manifest	submission source	compile and hash
style	official conference style bundle	official ICLR shell	style smoke test
macros	notation macro bundle	notation macros	include check
figures	figure bundle	method, frontier, shifts, audit	render check
evidence	public summary manifest	public audit summary	digest check
evaluation runs	raw result manifest	result manifest	schema and hash check

Appendix

Table 49: Artifact map for the reproducibility package. Handles are resolved relative to the package root; source and build hashes are recorded in its delivery manifest.

Stage	Input	Operation	Output
Bind	Cache and policy manifests	Match record membership, versions, and seeds	Fixed comparison set
Replay	Query, paid observations, state	Execute admissible calls and fallback rules	Ordered event trace
Audit	Frozen trace and manifest	Check reads, order, binding, and budget	Invariant outcomes
Score	Returned answer and reference	Apply the named answer normalization	Per-query task outcome
Pair	Outcomes sharing query and seed	Compare policies on common records	Task and cost differences
Aggregate	Paired rows and service times	Retain denominators and metric units	Tables and uncertainty fields

Appendix

Table 50: Replay and scoring sequence. Every stage carries the query identifier, seed, and run binding of its input.

View	Axes or measurements	Pairing unit	Source
Quality and cost	Task score, generated tokens, monetary cost	Same query, seed, and policy set	Tables 38 , 34
Latency calibration	Declared units, queue time, service time	Request within a serving condition	Table 39
Seed uncertainty	Task-score delta and confidence interval	Router seed with independent candidate generation	Tables 37 , 38
Route categories	Correctness, cost, and latency by event type	Requests sharing a stop or fallback category	Table 14

Appendix

Table 51: Analysis views and their source measurements. The pairing unit fixes the population represented by each comparison.

Quantity	Relation	Audit consequence
Coverage	Coverage $+$ abstention $=1$ on the same request set	Every request has a return or abstention outcome
Calls	Successful-call count equals paid path length	Skipped calls remain separate events
Budget	Initial budget $=$ spent $+$ remaining, component-wise	Resource units cannot be exchanged silently
Latency	Per-request elapsed time and named queue/service intervals are nonnegative	Clock and unit bindings are retained
Quantiles	p50 $\leq$ p95 within each method and condition	Timing order is consistent
Pairing	Policy rows share query IDs, split, and seed	Paired task deltas use a common denominator

Appendix

Table 52: Consistency relations used to reconcile task, route, and timing summaries.

claim object	observable statement	measurement	artifact receipt
information boundary	each online field is sourced from a paid call or declared state	event schema and oracle-isolation check	clean replay manifest
route decision value	quality and resource changes are paired by record and policy	proxy/task score plus cost delta	paired comparison
budget behavior	each skip carries an availability or budget reason	event counts and remaining-budget invariant	shift summary
mutation coverage	event-level faults are caught after aggregate preservation	mutation recall and clean false-positive rate	audit campaign report
threshold transfer	selected thresholds are frozen before held-out scoring	paired bootstrap bound and manifest identity	operating-point gate
live efficiency	declared accounting tracks measured runtime quantities	token/money cost, p50/p95 latency, throughput	serving telemetry

Appendix

Table 53: Claim-to-measurement mapping for the frozen-cache, task, and serving evaluations.

Language Model Cascading

See all 21 papers

LEDGER: Claim-to-Evidence Trace Graphs for Auditing LLM Agents
Aug 19, 2026Daehong Kim, Haichao Miao, Shusen LiuAI Agent Auditing Data Provenance
Carnegie Mellon University Pittsburgh, PA 15213 · Lawrence Livermore National Laboratory Livermore, CA 94550
Cheap Verifiers, Large Blind Spots: Measuring the Reliability Cost of Cost-Saving Cascades
Sep 1, 2026Dushyant Rajput, Nirdesh Chauhan, Siddharth KosarajuLLM Inference Efficiency LLM Answer Verification
AltSlate Labs LLP
Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference
Jul 27, 2026Yifan Dou, Shikan Lian, Shibo LiLLM Inference Efficiency LLM Inference Scheduling
Department of Computer Science, Florida State University

Aug 19, 2026J/K move · Enter open · S save

LEDGER: Claim-to-Evidence Trace Graphs for Auditing LLM Agents

Daehong Kim, Haichao Miao, Shusen Liu

Carnegie Mellon University Pittsburgh, PA 15213 · Lawrence Livermore National Laboratory Livermore, CA 94550

AI Agent Auditing Data Provenance

Open paper AskPDF arXiv

Large language model (LLM) agents can now carry out long-horizon technical workflows involving complex tool use, code execution, file edits, and generated artifacts. As agents do more work faster, the productivity bottleneck shifts from producing outputs to auditing whether those outputs are correct and trustworthy. Agent observability systems make fine-grained execution events visible, but visibility alone still leaves reviewers to reconstruct which actions, artifacts, and validation steps matter for a particular conclusion. We introduce LEDGER - Layered Evidence and Decision Graphs for Execution Review, a tracing and review system that builds layered trace graphs over observed agent sessions. LEDGER preserves Trace Records while grouping them into Evidence Nodes and Workflow Nodes, representing artifacts as evidence anchors, and adding typed semantic edges that connect claims to supporting actions, artifacts, and checks. Through data-analysis and coding examples, we show how the resulting traces expose workflow decisions, artifact lineage, repair steps, validation coverage, and claim-support paths for evidence-centered audit.

Abstract

Language Model Cascading

Explore similar work

LEDGER: Claim-to-Evidence Trace Graphs for Auditing LLM Agents

Cheap Verifiers, Large Blind Spots: Measuring the Reliability Cost of Cost-Saving Cascades

Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference