Agent Policy-Value Audit: Separating Transition Composition from Event Selection in Financial LLM Agents

Mingyang; Chen; Yida; Xu; Huiwen; Chen; Yiming Lu; Wei Jin

cs.AIOct 2, 2026

Agent Policy-Value Audit: Separating Transition Composition from Event Selection in Financial LLM Agents

Authors: Mingyang, Chen, Yida, Xu, Huiwen, Chen, Yiming Lu, Wei Jin

Organizations: Alex · Emory University Atlanta, USA · Andrew · Aurora

Abstract

Financial LLM agents are often evaluated by comparing their end-to-end returns with those of a baseline and testing the paired difference against zero. This measures whether deploying the agent changes realized performance, but it does not isolate event-selection skill. An agent that frequently changes positions from flat to long can earn a positive paired return from an upward-drifting event pool even when it selects events at random. We propose the Agent Policy-Value Audit, which holds fixed the observed count of each ordered action-change type and randomly reassigns them across eligible events. The average payoff from these reassignments is the composition benchmark; the difference between observed deployment value and this benchmark is selection value. In semi-synthetic benchmarks based on real earnings-event returns, a zero-centered paired test falsely attributes passive exposure to selection skill in $11.6\%$ of no-skill replications, while the transition-matched audit reduces this rate to $5.3\%$ . Applied retrospectively to 723 earnings events at 44 U.S. consumer-facing firms, the audit decomposes the agent's gross deployment value of $+15.2$ bps/event into a $+25.8$ composition benchmark and a $-10.6$ selection value. The agent does not detectably outperform matched random assignments. Financial-agent evaluations should report deployment value separately from event-selection value.

Figures & tables

Panel of Figure 1 — Figure 1. Agent Policy-Value Audit. Both policies act on the same frozen inputs, and realized outcomes enter evaluation only. The observed deployment value $\hat{V}$ is compared with a benchmark $C_{Z}$ that preserves the agent’s transition composition while reassigning eligible events; the difference $S_{Z}$ is the selection value. Safeguards: firm-clustered uncertainty on $\hat{V}$ ; future-only scoring under decision-time overlap; leakage diagnostics on inputs. Flow diagram: both policies read the same frozen inputs; paired payoffs give the deployment value, which is compared with a composition benchmark built by reassigning the observed action changes within pools; the difference is the selection value.

Stage	$N$	Note
Raw earnings events	$1{,}291$	$1{,}219$ main $+$ $72$ megacap
Main-cohort candidate events	$917$	2020–2025 calendar window
Scoreable out-of-sample events	$723$	CRSP availability; eff. $\leq$ 2024
Full fallback (ITT default; counted)	$53$	retries exhausted
Genuine agent decisions (per-protocol)	$670$	$=723-53$
Rule-overrides (agent vs. rule)	$310$	$302$ flat $\to$ trade $+\,8$ other

Table 1. Sample flow, evaluation windows, and decision timing.

Arm	Policy class	Information set	Role
$R_{0}$	Deterministic	Structured only	Factorial control
$R_{1}$	Deterministic	$+$ five aggregates	Factorial treated
$L_{0}$	LLM	Structured only	Factorial control
$L_{1}^{\mathrm{agg}}$	LLM	$+$ five aggregates	Matched-information cell

Table 2. Matched-information factorial cells. Prompts are hash-frozen.

Figure 2. Known-skill calibration. Semi-synthetic agents with directional accuracy

q

on their flat

\to

long picks, real returns, and the real transition composition (

R{=}2{,}000

replications per point,

600

reassignment draws each). (a) Recovered vs. injected selection value (dashed: identity). (b) Power of the one-sided matched test (dotted: Table 3 ’s

.783

q{=}0.60

; vertical guide: pool base accuracy

0.523

). Two side-by-side panels: recovered selection value against injected value along the identity line, and the power curve of the matched test rising with directional accuracy, matching 0.783 at q equal to 0.60.

Procedure	P	K	F	Centered	Exposure	Timing	$q{=}0.60$
				False attribution (no skill)			Power
Standalone book, i.i.d.	–	–	–	.050	.139	.578	.890
Standalone book, clustered	–	–	–	.053	.200	.577	.917
Unpaired ablation	–	–	–	.010	.014	.164	.625
Zero-centered paired test	$\checkmark$	–	–	.057	.116	.592	.846
Intervention-count permutation	$\checkmark$	–	–	.048	.021	.490	.665

Table 3. Evaluator-component benchmark. One-sided false-attribution rates under three no-skill designs, and power under known selection (

q{=}0.60

	Rejection / sensitivity rate				Power
Assignment / test	U	SX	RX	NL	PW
Matched, baseline pools (primary)	$.050$	$.061$	$.013$	$.020$	$.643$
Matched, $\times$ year	$.045$	$.060$	$.025$	$.046$	$.702$
Matched, $\times$ firm	$.057$	$.069$	$.040$	$.022$	$.770$
Zero-centered paired, clustered	$.171$	$.173$	$.066$	$.107$	$.861$

Table 4. Size and exchangeability stress tests. Rejection rates under clean no-skill assignments (U, SX), covariate-dependent assignments (RX, NL), and injected power (PW) (

R{=}1{,}000

; one-sided

5\%

; real returns and transition counts).

Figure 3. Structural stress grid. One-sided false-attribution rates for no-skill flat

\to

long agents across premium scale

\kappa

, sample fraction

f

, and intervention rate

\rho

(slices through the

64

-cell grid;

R{=}800

per cell; dashed line:

5\%

nominal). Three line panels: false-attribution rates rise toward 0.88 for the zero-centered test and stay near 0.05 for the matched audit as premium scale, sample fraction, and intervention rate increase.

Contrast	$N$	Chg. rate	Gross	Net	$95\%$ CI
Full agent vs. rule (ITT)	$723$	$0.43$	$\mathbf{+15.2}$	$\mathbf{-1.2}$	$[-45.2,+69.2]$
Scaffold-free LLM vs. rule (PP)	$710$	$0.71$	$-7.0$	$-28.4$	$[-138.2,+101.7]$
External Llama: social $-$ orig	$719$	$0.45$	$-18.5$	—	$[-100.1,+56.1]$

Table 5. Deployment value (bps/event; gross, and net of the pre-stated

40

bps cost). Firm-clustered

95\%

CIs on the gross estimate (

B_{\mathrm{boot}}=10^{4}

); quarter-clustered confidence intervals are similar. External row: TradingAgents with the Llama backbone, future-only

[0,+1]

window; “social

-

orig” compares its social-augmented and original configurations (§ 6.4 ).

Transition	$n$	Observed	Null mean	Residual	To $\hat{V}$ (bps/event)
flat $\to$ long	$294$	$+47.7$	$+67.4$	$-19.7$	$+19.4$
flat $\to$ short	$8$	$+145.1$	$-67.4$	$+212.5$	$+1.6$
other	$8$	$-522.8$	$-79.4$	$-443.4$	$-5.8$
Total	$\mathbf{310}$	$\mathbf{+35.5}$	$\mathbf{+60.1}$	$\mathbf{-24.6}$	$\mathbf{+15.2}$

Table 6. Transition-level decomposition (bps/intervention except the last column). Per-transition null means are exact pool means; their intervention-weighted average is the exact

+60.1

(Monte Carlo mean

+59.5

). “Other”

=

4

long

\to

flat,

2

short

\to

flat,

2

long

\to

short.

Figure 4. Audit of the observed interventions. (a) Event-level decomposition of

\hat{V}

into

C_{Z}

and

S_{Z}

. (b) The agent’s payoff per intervention within the matched reassignment distribution (Table 6 ). Two panels: a bar decomposition of deployment value into composition benchmark and selection value, and a histogram of the matched reassignment distribution with the observed value at its 32nd percentile.

Contrast	Change rate	Gross	$95\%$ CI
Social info., deterministic ( $R_{1}{-}R_{0}$ ) ∗	$0.01$	$+0.5$	$[-7.5,+7.2]$
Social info., LLM ( $L_{1}^{\mathrm{agg}}{-}L_{0}$ )	$0.41$	$+1.3$	$[-58.0,+64.6]$
Policy, matched info. ( $L_{1}^{\mathrm{agg}}{-}R_{1}$ )	$0.59$	$-31.9$	$[-149.0,+86.8]$
Policy, structured only ( $L_{0}{-}R_{0}$ )	$0.76$	$-32.7$	$[-171.5,+96.1]$
Information $\times$ policy interaction	—	$+0.8$	$[-57.6,+62.0]$

Table 7. Matched-information factorial. Gross bps/event with firm-clustered

95\%

CIs; both policy classes receive the same five aggregate social features.

Null specification	Conditioning	Payoff pctl.	Win pctl.
Primary	baseline action	$32$ nd	$13$ th
Year-stratified	$\times$ year	$34$ th	$15$ th
Firm-preserving ‡	within firm	$13$ th	$12$ th
Demeaned §	firm $+$ quarter	$22$ nd	—
Propensity-stratified †	$\times$ propensity quintile	$28$ th	$12$ th

Table 8. Null-specification robustness. Percentiles of the observed payoff/intervention and win rate under each transition-matched null (

5{,}000

draws).

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,179 papers

Measurement Boundaries in LLM Financial Agent Evaluation: Fixed-Tape Execution and Multi-Defect Auditing
Sep 26, 2026Weicheng XueLLM Agent Evaluation AI Agent Auditing
Virginia Tech
From Knowing to Doing: A Memory-Controlled Benchmark for LLM Trading Agents on Stock Markets
May 27, 2026Taojie Zhu, Wentao Zhao, Rui Sun +7Benchmark Contamination LLM Agent Evaluation
1Tsinghua University, 2Stepfun · 2Stepfun · 4Shanghai Jiao Tong University +3
ValueBlindBench: Agreement-Gated Stress Testing of LLM-Judged Investment Rationales Before Returns Are Observable
Apr 28, 2026Sidi Chang, Peiying Zhu, Yuxiao ChenLLM-as-a-Judge AI Agent Evaluation
Blossom AI Labs, Tokyo, Japan · Blossom AI, San Francisco, USA

Sep 26, 2026J/K move · Enter open · S save

Measurement Boundaries in LLM Financial Agent Evaluation: Fixed-Tape Execution and Multi-Defect Auditing

Weicheng Xue

Virginia Tech

LLM Agent Evaluation AI Agent Auditing Counterfactual Evaluation

Open paper AskPDF arXiv

What controls are needed to interpret execution performance and audit scores in LLM agent evaluations? We study two limits on these interpretations in a financial agent harness. In StudyA, comparing independent runs under idealized and stressed execution on three synthetic settings that share one 24-day upward phase mixes the execution rule with fresh model responses and portfolio feedback: the parsed decision paths agree in only $19.8\%$ of $450$ pairs. Replaying each stored response tape through both execution destinations gives a narrower result. Conditional on those responses, stressed execution changes total return by $-0.0170$ (95% interval $[-0.0230,-0.0117]$ ), or $10.4\%$ of the idealized baseline, and ten seed clusters do not resolve the model ranking. StudyB corrects an incomplete answer key and replaces legacy tasks with matched zero-, one-, and two-defect tasks under an explicit multi-label prompt. The drop in target violation recall from one to two defects is positive in five of six combinations of auditor and source (median

0.267

), with three surviving Holm correction. Yet the auditor that includes both target labels most often has micro-precision

0.149

, emits findings on

98/100

zero-defect tasks, and returns the exact dual-defect set in only

21/100

cases. Target recall by itself therefore gives a poor account of audit quality on this construction. The studies address different limits: what an execution comparison estimates, and what target recall captures. Together, they show how fixed conditions and diagnostic controls bound the claims a score can support.

Abstract

LLM Agent Evaluation

Explore similar work

Measurement Boundaries in LLM Financial Agent Evaluation: Fixed-Tape Execution and Multi-Defect Auditing

From Knowing to Doing: A Memory-Controlled Benchmark for LLM Trading Agents on Stock Markets

ValueBlindBench: Agreement-Gated Stress Testing of LLM-Judged Investment Rationales Before Returns Are Observable