Measurement Boundaries in LLM Financial Agent Evaluation: Fixed-Tape Execution and Multi-Defect Auditing

Weicheng Xue

cs.LGSep 26, 2026

Measurement Boundaries in LLM Financial Agent Evaluation: Fixed-Tape Execution and Multi-Defect Auditing

Authors: Weicheng Xue

Organizations: Virginia Tech

Abstract

What controls are needed to interpret execution performance and audit scores in LLM agent evaluations? We study two limits on these interpretations in a financial agent harness. In StudyA, comparing independent runs under idealized and stressed execution on three synthetic settings that share one 24-day upward phase mixes the execution rule with fresh model responses and portfolio feedback: the parsed decision paths agree in only $19.8\%$ of $450$ pairs. Replaying each stored response tape through both execution destinations gives a narrower result. Conditional on those responses, stressed execution changes total return by $-0.0170$ (95% interval $[-0.0230,-0.0117]$ ), or $10.4\%$ of the idealized baseline, and ten seed clusters do not resolve the model ranking. StudyB corrects an incomplete answer key and replaces legacy tasks with matched zero-, one-, and two-defect tasks under an explicit multi-label prompt. The drop in target violation recall from one to two defects is positive in five of six combinations of auditor and source (median $0.267$ ), with three surviving Holm correction. Yet the auditor that includes both target labels most often has micro-precision $0.149$ , emits findings on $98/100$ zero-defect tasks, and returns the exact dual-defect set in only $21/100$ cases. Target recall by itself therefore gives a poor account of audit quality on this construction. The studies address different limits: what an execution comparison estimates, and what target recall captures. Together, they show how fixed conditions and diagnostic controls bound the claims a score can support.

Figures & tables

Figure 1: The fixed-tape execution contrast. (a) The

2\times 2

decomposition on total return. The downstream mechanical contrast excludes zero under either tape origin; the tape-origin term is descriptive rather than causal. (b) Kendall

\tau_{b}

between E0 and E1 Sharpe rankings under a fixed tape, with the frequency of exact-order agreement across cluster-bootstrap resamples. Confidence intervals come from a bootstrap over the

10

shared seed clusters; rank-correlation values are discrete because only five models are ranked.

Setting	Tape	$\tau_{b}$	95% CI	Exact-order freq.	Winner E0 $\to$ E1
calm trend	E0	1.000	[0.80, 1.00]	0.7635	glm-5.2 $\to$ glm-5.2
calm trend	E1	1.000	[0.60, 1.00]	0.2608	glm-5.2 $\to$ glm-5.2
high volatility	E0	1.000	[0.60, 1.00]	0.5368	glm-5.2 $\to$ glm-5.2
high volatility	E1	1.000	[0.60, 1.00]	0.5678	glm-5.2 $\to$ glm-5.2
jump–tail	E0	1.000	[0.80, 1.00]	0.7369	glm-5.2 $\to$ glm-5.2
jump–tail	E1	0.800	[0.40, 1.00]	0.3265	glm-5.2 $\to$ glm-5

Table 1: Sharpe-rank comparison under a fixed response tape. Kendall

\tau_{b}

is near one, but exact-order agreement occurs in only

0.26

–

0.76

of seed-cluster bootstrap resamples, and the winner changes in the jump–tail setting with tapes from E1. “Exact order” is a bootstrap agreement frequency, not a posterior probability that the population order is correct.

Figure 2: Behaviour on the legacy corpus, whose prompts contain domain-specific quantity cues. (a) On tasks where two defects are objectively true, the share of responses reporting both target labels, one, or neither; this is not the total number of findings in a response. (b) Violation recall with one true defect versus two, with within-pair McNemar

p

-values, scored against the corrected key so that reporting the edit is credited rather than penalized. The low frequency of “both” does not identify a quantity-cue effect. The new matched experiment uses an explicit multi-label instruction.

Auditor	Source	Both targets	Violation target	Edit target	Neither target
deepseek-v4-pro	tool-use	35	3	0	2
deepseek-v4-pro	trading A	22	0	7	1
deepseek-v4-pro	trading B	21	0	9	0
glm-5	tool-use	19	0	21	0
glm-5	trading A	4	9	16	1
glm-5	trading B	0	5	25	0

Table 2: Presence of the two target labels on the two-defect arm. The four columns partition responses only by whether the target violation and target edit appear. Additional or duplicate findings do not change a response’s column, so “both targets” is not exact-set accuracy and “violation target” does not mean that no unsupported finding was emitted.

Auditor	Zero any	Zero exact	Micro-P	Exact/300	Dual both	Dual exact
deepseek-v4-pro	$98/100$	$1/100$	0.149	$29/300$	$78/100$	$21/100$
glm-5	$45/100$	$55/100$	0.199	$108/300$	$23/100$	$7/100$

Table 3: Target-label counts beside specificity outcomes. “Zero any” is the number of zero-defect tasks with at least one emitted finding. “Zero exact” is a parse-valid, duplicate-free empty finding set; parse failures are not credited. “Dual both” requires both target labels but permits extras, whereas “dual exact” requires a parse-valid, duplicate-free match. Micro-precision pools all findings over the

300

tasks per auditor, and overall exact set applies the same strict match to all

300

Auditor	Source	$n$	Single	Dual	Drop	Holm $p$
deepseek-v4-pro	tool-use	40	0.900	0.950	$-0.050$	0.625
deepseek-v4-pro	trading A	30	0.900	0.733	$+0.167$	0.125
deepseek-v4-pro	trading B	30	0.967	0.700	$+0.267$	0.031
glm-5	tool-use	40	1.000	0.475	$+0.525$	$\mathbf{6\times 10^{-6}}$
glm-5	trading A	30	0.833	0.433	$+0.400$	0.009
glm-5	trading B	30	0.433	0.167	$+0.267$	0.116

Table 4: Internally hash-bound multi-label experiment. Target violation recall on the one-defect and two-defect conditions within matched triplets, with exact two-sided McNemar tests and Holm correction across six cells. Positive drop means that the target violation label appears less often when the specified edit defect also occurs.

Figure 3: Two aggregate summaries and their cell structure. (a) Parsed-path agreement in the independently sampled E0/E1 source runs. The pooled

19.8\%

consists of

89/90

agreements from one nearly inactive model and

0/90

from each active model. This shows that the diagonal is not a fixed-output comparison; it does not show that execution caused the divergence, since most divergent pairs already differ in the first response. (b) The six cells recorded in the internally hash-bound multi-label plan. Five target-recall drops are positive, three survive Holm correction, and one is negative.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Outcome	Estimand	Estimate	95% CI
total return	$D_{\mathrm{E0}}$ : execution $\mid$ tape from E0	-0.0193	$[-0.0273,-0.0121]$
	$D_{\mathrm{E1}}$ : execution $\mid$ tape from E1	-0.0147	$[-0.0197,-0.0100]$
	$\bar{D}$ : balanced destination	-0.0170	$[-0.0230,-0.0117]$
	$O_{\mathrm{E0}}$ : response origin $\mid$ replay at E0	+0.0012	$[-0.0075,+0.0101]$
	$O_{\mathrm{E1}}$ : response origin $\mid$ replay at E1	+0.0059	$[-0.0053,+0.0176]$
	$\bar{O}$ : balanced tape origin	+0.0035	$[-0.0061,+0.0136]$

Appendix

Table 5: Study A, every estimand of the fixed-tape

2\times 2

replay, on the four outcomes the body draws on. Intervals are seed-cluster bootstrap (

10{,}000

draws,

10

clusters,

450

matched pairs). The replay records five further outcomes; they are in factorial_estimands.csv .

Auditor	Source	$n$	Single	Dual	Drop	95% CI	$b/c$	Holm $p$
deepseek-v4-pro	tool-use	40	0.900	0.950	-0.050	$[-0.150,+0.050]$	1/3	0.625
deepseek-v4-pro	trading A	30	0.900	0.733	+0.167	$[+0.033,+0.300]$	5/0	0.125
deepseek-v4-pro	trading B	30	0.967	0.700	+0.267	$[+0.133,+0.433]$	8/0	0.0312
glm-5	tool-use	40	1.000	0.475	+0.525	$[+0.375,+0.675]$	21/0	5.7e-06
glm-5	trading A	30	0.833	0.433	+0.400	$[+0.200,+0.600]$	13/1	0.00916
glm-5	trading B	30	0.433	0.167	+0.267	$[+0.067,+0.467]$	10/2	0.116

Appendix

Table 6: Study B, the six auditor

\times

source cells recorded in the internally hash-bound multi-label plan. Recall is on the target violation, within matched triplets; the discordant counts are the McNemar pairs. Intervals are unadjusted 95% paired percentile-bootstrap confidence intervals based on

10{,}000

resamples.

p

-values are from two-sided exact McNemar tests with Holm correction across the six primary comparisons.

Explore similar work

Execution Realism and Reproducibility in LLM-Based Trading Systems: A Systematic Scoping Review and Evidence Audit
Jun 6, 2026Junyi Yao, Zihao Zheng, Baichuan Li +1ML Reproducibility Finance
Washington University in St. Louis, St. Louis, MO 63130 USA. · Department of Operations Research and Engineering Management, Southern Methodist University, Dallas, TX 75275 USA.
ReguSim: Evaluating LLM Agent Rule Grounding in Financial Compliance
Aug 20, 2026Yiyang Luo, Yihang Jiang, Qijun Xie +4LLM Agent Evaluation AI Agent Auditing
HKUST · HKBU · NTU
Representation Signatures and Risk-Feedback Alignment in LLM Trading Agents
May 16, 2026Weicheng XueLLM Alignment LLM Auditing
Virginia Tech

Jun 6, 2026J/K move · Enter open · S save

Execution Realism and Reproducibility in LLM-Based Trading Systems: A Systematic Scoping Review and Evidence Audit

Junyi Yao, Zihao Zheng, Baichuan Li, Jiayu Long

Washington University in St. Louis, St. Louis, MO 63130 USA. · Department of Operations Research and Engineering Management, Southern Methodist University, Dallas, TX 75275 USA.

ML Reproducibility Finance Algorithmic Trading

Open paper AskPDF arXiv

Execution realism remains weakly standardized in research on large-language-model-based trading systems, limiting comparison and reproduction across backtests, simulations, and portfolio benchmarks. This article presents a PRISMA-ScR systematic scoping review with a nested execution-reproducibility evidence audit. Eight reproducible query families returned 796 query-level records; DOI/title deduplication left 687 unique records, 101 reports were sought for retrieval, and 59 full texts were recovered through direct and fallback open-access routes. Fifty-three reports are provisionally eligible for expanded evidence charting, subject to reconciliation of two blind author-validation packets. The charting framework separately evaluates point-in-time controls, temporal splits, held-out evaluation, trading costs and turnover, execution semantics, universe construction, architecture reporting, and artifact availability. Direct-trading, portfolio or alpha-construction, and benchmark studies are retained as distinct descriptive subgroups rather than pooled as a common performance estimand. Search responses, retrieval attempts, file hashes, evidence excerpts, screening decisions, and coding worksheets are archived with the review materials. Final field-level counts are intentionally withheld until author validation is complete.

Abstract

Explore similar work

Execution Realism and Reproducibility in LLM-Based Trading Systems: A Systematic Scoping Review and Evidence Audit

ReguSim: Evaluating LLM Agent Rule Grounding in Financial Compliance

Representation Signatures and Risk-Feedback Alignment in LLM Trading Agents