FIDES-Live: Evaluating Behavioral Concordance in Sequential Trading Agents

Arther Tian; Alex Ding; Simon Wu; Aaron Chan

cs.CROct 27, 2026

FIDES-Live: Evaluating Behavioral Concordance in Sequential Trading Agents

Authors: Arther Tian, Alex Ding, Simon Wu, Aaron Chan

Organizations: DGrid AI

Abstract

Sequential trading agents produce decisions, orders, execution receipts, account states, and reports that need not agree, even when the final portfolio value is correctly computed. We present FIDES-Live, an operational extension of FIDES from generated strategy concordance to the audit of sequential agent traces. The protocol separates structured decision validity, optional numerical commitment adherence, execution and ledger reconciliation, and receipt- grounded reporting, with explicit coverage and unscorable outcomes. We retrospectively audit ten requested inference routes over 7,299 cryptocurrency account-rounds and a 310-round equity pilot, using real market observations and local paper execution. Cryptocurrency decision validity is 97.35% and structured reporting accuracy is 90.81% among scorable responses, but route-level decision and reporting performance can diverge sharply. Commitment adherence is 94.36% overall; it falls to 62.63% on constraints emitted with nonempty order proposals, compared with 97.56% for valid empty proposals. All retained cash and position transitions reconcile with recorded fills, while execution controls and unavailable quotes explain separate rejection populations. Short-window marked returns provide descriptive context rather than a new test of whether concordance predicts alpha. These observations motivate reporting concordance, evidence coverage, activity, and financial outcomes separately. They do not establish long-run trading skill, semantic faithfulness of explanations, or causal benefits from audit feedback.

Figures & tables

Panel of Figure 1 — Figure 1: The FIDES-Live measurement object. Intended orders, execution outcomes, and reported facts are distinct artifacts. The deterministic audit checks supported properties and preserves unavailable evidence; it does not infer hidden reasoning from a correct account balance.

Property	Earlier FIDES	FIDES-Live
Observation unit	One generated strategy; 40 outputs	One retained account-round; 7,609 rounds
Evidence	Strategy description, generated code, historical backtest	Decision, commitments, order receipts, account state, report
Measurements	Language–code, implementation, and claim–outcome concordance	Action-contract validity, optional commitments, ledger reconciliation, structured report agreement
Financial setting	Historical out-of-sample strategy evaluation	Forward-running local paper execution on real market observations

Table 1: The studies share an audit perspective but use different observations and measurement definitions. Their scores are not directly comparable.

Environment	$N$	Valid decisions	Accurate reports	Kept commitments	Unknown
Crypto	7,299	7,052/7,244	6,597/7,265	2,892/3,065	2
Equities	310	297/309	274/309	196/202	53

Table 2: Recomputed primary counts. Decision and report denominators exclude explicitly unavailable calls. Commitment denominators include only nontrivial constraints with eligible marked-state evidence. Unknown refers to declared constraints excluded for unavailable or stale valuation evidence.

Model	$N$	$V$ (%)	$A$ (%)	Cover. (%)	Adhere. (%)	Fills
Claude Haiku 4.5	739	76.58	82.65	99.28	98.09	38
DeepSeek V3.2	631	98.73	14.33	0.48	100.00	9
Gemini 2.5 Flash	740	99.86	99.86	10.20	100.00	5
MiniMax M2.5	741	100.00	100.00	1.08	100.00	49
Kimi K2.5	741	100.00	100.00	4.19	85.25	39
GPT-4.1 mini	742	100.00	99.73	31.30	90.04	6

Table 3: Cryptocurrency audit by requested model route, combining the settled and partial sessions.

V

and

A

use their own scorable denominators. Coverage is the percentage of valid decisions declaring a nontrivial commitment; adherence is per scorable constraint, not per round. Fills are local simulated executions. Exact numerators and denominators appear in Appendix Table 11 and the accompanying data.

Figure 2: Decision validity and structured report accuracy are distinct operational properties. Rates are conditional on scorable responses; unequal sample sizes and infrastructure failures remain visible in the tables.

Figure 3: Left: declared nontrivial commitment coverage among valid cryptocurrency decisions. Right: compliance among scorable constraints, conditioned on a nonempty or valid empty proposal. Counts above bars are passed/evaluated constraints. Empty proposals dominate the corpus and can conceal materially weaker adherence on active rounds.

Env.	Round stratum	$N$	Reports	%	Commitments	%
Crypto	Nonempty proposal	219	205/219	93.61	176/281	62.63
Crypto	Valid empty proposal	6833	6258/6812	91.87	2716/2784	97.56
Crypto	Any rejection	87	82/87	94.25	37/131	28.24
Equities	Nonempty proposal	53	48/52	92.31	25/31	80.65
Equities	Valid empty proposal	244	221/244	90.57	171/171	100.00
Equities	Any rejection	25	24/25	96.00	0/0	–

Table 4: Activity-conditioned audit. Reports and commitments have different eligibility denominators. Any-rejection strata can overlap other strata; these rows are not a partition to sum. Undefined adherence is shown as a dash. An empty valid proposal is distinct from a malformed or unavailable decision.

Decision category	Account-rounds	Gate pass	Gate fail
Valid nonempty	219	132	87
Valid empty	6833	6073	760
Invalid or unavailable	247	0	247

Table 5: Cryptocurrency gate populations. Contract-invalid or unavailable responses form a third input category; zero admitted orders in that category does not establish a valid choice to do nothing. Counts partition retained account-rounds.

Model	$N$	Return (%)	$A$ (%)	Cash (%)
Claude Haiku 4.5	664	-0.970	80.67	27.4
DeepSeek V3.2	615	-0.691	14.38	71.4
Gemini 2.5 Flash	665	-0.313	99.85	75.2
MiniMax M2.5	666	-1.448	100.00	20.8
Kimi K2.5	666	-0.833	100.00	20.3
GPT-4.1 mini	667	-0.157	100.00	70.4

Table 6: Settled seven-day cryptocurrency session only.

N

is retained account-rounds;

A

is report accuracy conditional on a scorable response. Return uses the same saved endpoint for all accounts. Cash is the final fraction of marked equity. These are descriptive marked outcomes, not estimates of alpha.

Figure 4: Available explicitly nonstale saved marks, with returns measured from initial cash. The shaded initial 20-hour-49-minute interval has no eligible marks; the later October 5 gap is approximately 97 minutes. Curves are subsampled and broken across source gaps longer than five minutes, without imputing missing observations. The full-session endpoint is retained. The cash reference assumes no interest or trading and is not an exposure-matched market benchmark.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Abbreviation	Exact requested route
GPT-4.1 mini	openai/gpt-4.1-mini
GPT-5.4	openai/gpt-5.4
DeepSeek V3.2	deepseek/deepseek-v3.2
Gemini 2.5 Flash	google/gemini-2.5-flash
Claude Haiku 4.5	anthropic/claude-haiku-4.5
Qwen3 235B	qwen/qwen3-235b-a22b-instruct-2507

Appendix

Table 7: Model names denote request routes, not independently attested immutable model weights.

Setting	Cryptocurrency	Equity pilot
Assets	Six spot assets	Eight ETFs initially; fifteen assets in main run
Decision interval	15 minutes	5 minutes initially; 15 minutes in main run
Initial allocation	$10,000 per model/session	$10,000 per model/run
Historical context	Up to 90 daily bars	Up to 30 daily bars
Fee / adverse slippage	10 / 2 basis points	1 / 2 basis points
Cash / position / exposure	20% / 30% / 80% fixed gate	No matching balanced gate

Appendix

Table 8: Material recorded differences. These conditions are not pooled into a causal cross-market comparison.

Model	Calls	Tokens (M)	Usage pairs	Median seconds
Claude Haiku 4.5	1484	27.54	1463	9.43
DeepSeek V3.2	1484	16.52	1476	5.48
Gemini 2.5 Flash	1484	28.83	1476	2.47
MiniMax M2.5	1483	17.31	1479	10.77
Kimi K2.5	1483	22.50	1480	11.68
GPT-4.1 mini	1484	16.02	1479	2.40

Appendix

Table 9: Cryptocurrency call-table accounting, including attempts outside committed rounds. Tokens are recorded prompt plus completion counts; missing usage contributes no measured amount. Usage pairs counts calls with both components available. Median elapsed seconds uses recorded values and is descriptive of the gateway/system, not a hardware-normalized speed benchmark.

UTC date	Account-rounds	Accurate/scorable reports	Accuracy (%)
2026-09-30	820	686/809	84.80
2026-10-01	959	805/952	84.56
2026-10-02	960	860/960	89.58
2026-10-03	959	872/957	91.12
2026-10-04	957	867/949	91.36
2026-10-05	906	814/900	90.44

Appendix

Table 10: Cryptocurrency observation counts by UTC date. Partial first/last days and operational gaps remain visible. Accurate/scorable reports use the same primary definition as the main text.

Route	Valid/scorable	Accurate/scorable	Declaring/valid	Passed/evaluable
Claude Haiku 4.5	556/726	605/732	552/556	1081/1102
DeepSeek V3.2	620/628	90/628	3/620	6/6
Gemini 2.5 Flash	735/736	738/739	75/735	150/150
MiniMax M2.5	740/740	738/738	8/740	8/8
Kimi K2.5	739/739	740/740	31/739	52/61
GPT-4.1 mini	738/738	739/741	231/738	416/462

Appendix

Table 11: Cryptocurrency exact numerators and denominators corresponding to Table 3 . Declaring/valid is commitment coverage; passed/evaluable counts individual nontrivial constraints. A perfect rate based on a few constraints is not evidence of broad coverage.

AI Agent Auditing

AI agent auditing examines agents’ actions, tool use, evidence, and outcomes to assess safety, compliance, reliability, and accountability.

See all 262 papers

FIDES: A Concordance Protocol for LLM-Generated Trading Strategies
Aug 24, 2026Arther Tian, Alex Ding, Simon Wu +1Look-Ahead Bias in Language Models Language Model Generation Evaluation
DGrid AI
Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agents
Jan 17, 2026Raffi KhatchadourianAI Agent Reliability LLM Agent Evaluation
IBM Financial Services Market, New York, NY, USA
The Control Layer Behind Rule-Following Agents Trading Across Market Conditions
Oct 19, 2026Saad Naja, Nasser Odtallah, Jake Van-Praag +1AI Agent Reliability Process-Based Agent Evaluation
Hygentra AI

Aug 24, 2026J/K move · Enter open · S save

FIDES: A Concordance Protocol for LLM-Generated Trading Strategies

Arther Tian, Alex Ding, Simon Wu, Aaron Chan

DGrid AI

Look-Ahead Bias in Language Models Language Model Generation Evaluation Language Model Self-Assessment Algorithmic Trading

Open paper AskPDF arXiv

An LLM asked for a trading strategy returns three artifacts at once: a natural-language rationale, an executable implementation, and once run, a track record. Whether these are the same object is rarely checked. We present FIDES, a measurement protocol that treats them as three views to be reconciled rather than one deliverable to be graded. Through dual delivery, a single model call returns both a natural-language strategy with an explicit claimed edge and a self-contained strategy(df) function. FIDES executes the code in a sandbox against a lag-one out-of-sample backtest and scores three concordance gaps: say to do, do to real, and say to result. On 8 liquid US ETFs across four models plus a two-stage elicitation arm, 40 strategies, 2023 to 2024 out-of-sample, three findings stand out. First, concordance does not predict profit: only 2 of 40 strategies beat buy-and-hold, and a plain sma(50,200) rule outperforms every model's mean Sharpe. Second, self-assessment is badly calibrated: 32 of 40 strategies claim to beat buy-and-hold and exactly one does. Third, swapping the language-code judge for a second model flips say to do on more than half of items. Injecting Close.shift(-1) drops do to real by 0.33 on average, while our runtime future-information probe fired on neither clean nor injected code. We frame FIDES as a protocol for measurement fidelity, not a claim about market performance.

Abstract

AI Agent Auditing

Explore similar work

FIDES: A Concordance Protocol for LLM-Generated Trading Strategies

Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agents

The Control Layer Behind Rule-Following Agents Trading Across Market Conditions