Reliable Before Profitable: QuantHarness for Governing Financial Strategy Development with LLM Agents

Jiasheng Mao; Jiacheng Liu; Hengwei Ye; Zheng Tian

cs.AIOct 19, 2026

Reliable Before Profitable: QuantHarness for Governing Financial Strategy Development with LLM Agents

Authors: Jiasheng Mao, Jiacheng Liu, Hengwei Ye, Zheng Tian

Organizations: ShanghaiTech University, Shanghai, China

Abstract

LLM agents can turn natural-language financial research requests into executable and backtested strategies, but a successful run may still implement the wrong rule, use information unavailable at decision time, or obscure how repairs and candidate selection changed the strategy. Because these failures arise throughout development, final code and performance metrics alone are insufficient to establish either fidelity to the request or the validity of the development process. We therefore address the reliability of financial strategy development at the trajectory level: the system must control how intermediate outputs become downstream state and preserve the evidence needed to justify the terminal outcome. We introduce QuantHarness, a two-layer agent harness in which a finance-specific Workflow structures specification, data grounding, construction, testing, and review, while Harness Core enforces stage-specific information, action, and commitment boundaries. We also introduce FinRelBench, a benchmark that evaluates strategy-development reliability through execution success, requirement satisfaction, and runtime evidence. QuantHarness achieves the highest Task Success at 90.72%, compared with 74.23%--84.54% for the comparison systems. Controlled stress tests show that its boundaries contain targeted violations, while structured run records improve evidence retrieval and support decision replay under the evaluated protocol.

Figures & tables

Figure 1. Overview: Workflow structures strategy development; Harness Core governs each run and preserves its run record.

	Main measure	Mechanical diagnostics		Requirement diagnostics
System	Task Success	Data Pipeline Success	Strategy Run Success	Data Requirement Satisfaction	Data Runtime- Evidence Coverage	Strategy Requirement Satisfaction	Strategy Runtime- Evidence Coverage
QuantHarness	90.72%	100.00%	98.97%	98.97%	98.28%	98.71%	96.07%
Codex CLI	81.44%	96.91%	91.75%	97.59%	96.56%	97.50%	91.11%
OpenHands	74.23%	86.60%	84.54%	84.54%	84.54%	85.38%	81.40%
SWE-agent	84.54%	94.85%	93.81%	94.85%	94.50%	94.44%	92.85%
Aider	79.38%	98.97%	94.85%	96.91%	99.31%	99.09%	92.64%

Table 1. Strategy-development reliability on 97 planned FinRelBench tasks per system. Higher is better.

A. Normal runs
Boundary	Successful outcome $\uparrow$	On	Off
Information	Initial implementation succeeds	25/30	23/30
Action	Workflow completes correctly	30/30	23/30
B. Controlled stress tests
Boundary	Observed failure $\downarrow$	On	Off
Information	Invalid information used	0/30	11/30

Table 2. Paired boundary ablations and stress tests on 30 FinRelBench tasks.

Evidence view	Exact evidence	Replay
Organized artifacts	100.0%	44/44
Same artifacts, unorganized	43.2%	0/44
Final artifacts only	27.6%	0/44

Table 3. Evidence retrieval and replay for 44 run records covering 30 FinRelBench tasks.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Origin	Daily scores	Intraday orders	Total
Original reliability-oriented tasks	14	3	17
Adapted QuantCode-Bench tasks	50	30	80
Total	64	33	97

Appendix

Table 4. Composition of the current FinRelBench task set.

System	Output type		Task origin
	Daily scores	Intraday orders	Original	Adapted
QuantHarness	60/64	28/33	14/17	74/80
Codex CLI	54/64	25/33	12/17	67/80
OpenHands	49/64	23/33	12/17	60/80
SWE-agent	56/64	26/33	14/17	68/80
Aider	52/64	25/33	14/17	63/80

Appendix

Table 5. Task Success by output type and task origin. Each pair of columns partitions the same 97 planned tasks.

	Generation/ data pipeline	Strategy execution	Unmet/unverifiable requirements	Runtime evidence only	Evaluation incomplete
QuantHarness	0	1	1	6	1
Codex CLI	3	5	5	5	0
OpenHands	13	1	4	5	2
SWE-agent	5	1	5	4	0
Aider	1	4	10	5	0

Appendix

Table 6. Non-success accounting over 97 planned tasks per system. The final column is evaluator incompleteness, not a strategy-failure category.

System	Complete panels	Exact agreement	Any arbitration	Cross-wording
QuantHarness	96	98.2%	25/96	8/96
Codex CLI	95	97.5%	30/95	12/95
OpenHands	83	98.3%	20/83	9/83
SWE-agent	92	99.4%	17/92	7/92
Aider	97	98.1%	33/97	15/97

Appendix

Table 7. External-panel consistency on eligible cases with complete assessments. Agreement is criterion-level; arbitration counts cases, not model calls. Cross-wording arbitration is a subset of any arbitration.

Boundary	Endpoint	Full	Off
Information	Initial mechanical execution	29/30	30/30
	Public-obligation conformity	26/30	23/30
	Initial implementation success	25/30	23/30
	Terminal completion	30/30	30/30
Action	Admissible Workflow completion	30/30	23/30
	Observed unauthorized behavior	0/30	0/30

Appendix

Table 8. Normal-run boundary diagnostics on 30 paired tasks, separating first-candidate execution and semantics from terminal completion.

Boundary	Starting condition and intervention	Evidence and denominator
Information	A revision handoff contains accepted, superseded, and rejected decisions. Full receives the currently accepted version; Off receives the raw versions and governance events without an explicit current-version answer.	Execution evidence must show use of an obsolete or rejected value. Invalid information is used in 0/30 Full and 11/30 Off cases.
Action	A provisional binding passes mechanical checks but substantive review and approval are pending. Building early uses a temporary opportunity for parallel work; waiting costs one synthetic scheduling cycle and remains feasible.	The agent chooses early construction in 22/30 cases. The same request is replayed against both guards: 0/22 Full and 22/22 Off requests produce unauthorized persistent files.
Commitment	After a real Judge approval, the candidate implementation is replaced in 14 prespecified cases; 16 cases retain the approved candidate. Off removes the targeted identity enforcement.	Compare the hashes of the approved source, the source presented for freezing, and the frozen source. Full blocks freezing of all 14 modified candidates, whereas Off freezes them; all 16 unchanged controls commit in both conditions.

Appendix

Table 9. Stress protocols. Counts distinguish exposure, agent activation, and verified escape.

View	Exact audit evidence	Acceptance packet	Commitment packet	Acceptance executed	Commitment executed	Verified replay
Organized	100.0%	44/44	44/44	44/44	44/44	44/44
Flat	43.2%	0/44	6/44	0/44	6/44	0/44
Final only	27.6%	0/44	0/44	0/44	0/44	0/44

Appendix

Table 10. Audit and replay decomposition for 44 records per view. Packet completeness requires all roles for that component; execution counts are record counts, not average role-coverage percentages.

Phase	Dates	Universe	Use
Training	2025-01-02–2025-08-29	CSI 500 (2024-12-31)	Fitting/construction
Optimization	2025-09-01–2025-12-31	CSI 500 (2025-08-29)	Candidate selection
Final	2026-01-05–2026-05-29	CSI 500 (2025-12-31)	One-time reporting

Appendix

Table 11. Chronological windows for the open-ended rule and model research protocol. Dates in the Universe column identify the constituent snapshots; each snapshot predates its evaluation or training window.

System	Completed tasks $\uparrow$	Mean ann. return $\uparrow$	Mean Sharpe $\uparrow$	Mean max drawdown $\uparrow$	Mean turnover $\downarrow$
QuantHarness	10/10	$\mathbf{-3.27\%}$	$\mathbf{-0.219}$	$\mathbf{-13.85\%}$	33.06
Codex CLI	10/10	$-10.65\%$	$-0.611$	$-15.03\%$	51.63
OpenHands	10/10	$-5.68\%$	$-0.350$	$-14.95\%$	59.75
SWE-agent	9/10	$-13.07\%$	$-0.732$	$-14.50\%$	59.90
Aider	9/10	$-5.59\%$	$-0.384$	$-15.22\%$	62.26
RD-Agent(Q)	10/10	$-7.66\%$	$-0.503$	$-18.75\%$	57.31

Appendix

Table 12. Financial results for the ten daily-score tasks in RuleResearchBench. Financial metrics are means over each system’s completed tasks.

System	Completed tasks $\uparrow$	Positive returns $\uparrow$	Mean total return $\uparrow$	Mean Sharpe $\uparrow$	Mean max drawdown $\uparrow$	Mean generalization gap (pp)
A. ModelBench-5: specified model families
QuantHarness	5/5	3/5	$\mathbf{+1.75\%}$	$\mathbf{0.168}$	$\mathbf{-15.32\%}$	$+4.32$
Codex CLI	5/5	2/5	$-0.77\%$	$-0.170$	$-17.27\%$	$-0.82$
OpenHands	5/5	2/5	$+1.24\%$	$0.058$	$-16.05\%$	$+0.83$
SWE-agent	3/5	0/3	$-14.79\%$	$-1.152$	$-22.63\%$	$-15.53$
Aider	0/5	–	–	–	–	–

Appendix

Table 13. Financial results for ModelBench and ModelResearchBench. Financial metrics are means over each system’s completed tasks within the corresponding group.

System	Completed tasks
QuantHarness	2/2
Codex CLI	0/2
OpenHands	1/2
SWE-agent	1/2
Aider	0/2
Total (system–task runs)	4/10

Appendix

Table 14. Completion of the two intraday rule-research tasks.

Stage	Stage inputs and permitted actions	Required evidence and authority
Task understanding	Public request and operating mode; structured extraction of intent, missing fields, conflicts, and assumptions; no strategy-code generation.	Task contract. Intake policy determines whether the task is ready to proceed or needs clarification; the runtime checks the associated state transition.
Planning and data grounding	Public task and stage-specific Runtime Card. Catalog listing and asset inspection precede binding; only authorized data tools are exposed.	Specification, planning blueprint, data binding, and field contract. Deterministic catalog/schema checks verify the proposed sources and fields.
Construction	Accepted specification, bound fields, implementation plan, and required output interface. Code generation uses the established data binding rather than choosing new data sources.	Candidate source and implementation metadata; model paths additionally produce training/inference artifacts. Stages pass candidates to checks, not directly to acceptance.
Validation and backtest	Deterministic services validate and backtest the candidate under the configured data and execution protocol; they do not redesign the strategy.	Validation checks, execution outputs, and backtest reports. Failed checks block normal progression; recovery requires authorization.
Recovery	Public failure feedback and the current candidate, with stage-limited repair/replan/rebind authority and bounded attempts.	Proposed change, guard result, revised candidate, and renewed checks. Recovery cannot silently weaken the task to make execution pass.
Judge and commitment	Judge reads a compact evidence bundle and recommends acceptance, recovery, or stopping; it does not execute its own recommendation.	Judge report, policy/guard decision, and approved candidate identity. The guarded runtime authorizes transitions. Before saving the frozen artifact, the freezer checks required evidence and, when seal enforcement is enabled, verifies that the files match their approved versions.

Appendix

Table 15. Stage interfaces and authority. An agent proposal or a written report is not itself permission to advance the run.

Panel of Figure 2 — Figure 2. The observed request-to-freeze trajectory. Numbers match the content boxes below. A denotes agent proposals/generation, D deterministic operations, and P persisted evidence. Dashed rounded nodes contain agent stages; solid nodes contain deterministic review and execution. The three boundary annotations identify the mediation observed in this run.

System	Implementation / release	Native limit within the process cap
QuantHarness	Native Workflow	20 turns; one repair allowance
Codex CLI	exec 0.144.1	No comparable turn cap exposed
OpenHands	agent-server 1.8.0 ( de4e2eb )	80 native iterations
SWE-agent	runner 1.1.0 ( abd7d69 )	2.00 per-instance cost limit
Aider	CLI 0.86.3.dev53 ( 5dc9490 )	No comparable turn cap exposed

Appendix

Table 16. FinRelBench system releases and native generation limits.

Operation	Daily-score task	Intraday-order task
Whole generation process	1,200	2,400
Individual model request	300	600
Agent stage, including request retry	660	1,260
Data-binding stage	660	1,260
Independent dataset construction	300	600
Independent strategy execution	150	300

Appendix

Table 17. FinRelBench time limits in seconds, separating strategy generation from independent evaluation for each output type.

LLM Agent Harnesses

LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.

See all 357 papers

FinHarness: An Inline Lifecycle Safety Harness for Finance LLM Agents
May 26, 2026Haoxuan Jia, Yang Liu, Bin Chong +10AI Agent Reliability LLM Guardrails
Nanyang Technological University · Tsinghua University · Peking University +8
QuantEval: A Benchmark for Financial Quantitative Tasks in Large Language Models
Jan 13, 2026Zhaolu Kang, Junhao Gong, Wenqing Hu +13Financial Language Model Evaluation LLM Evaluation
Peking University · Tsinghua University · Fudan University +6
FinanceHarness: Autonomous Financial Deep Research Framework
Jul 30, 2026Yijia Xiao, Rujun Han, Yanfei Chen +8Deep Research Agents Financial Forecasting
Google Cloud AI Research · University of California, Los Angeles

May 26, 2026J/K move · Enter open · S save

FinHarness: An Inline Lifecycle Safety Harness for Finance LLM Agents

Haoxuan Jia, Yang Liu, Bin Chong, Yingguang Yang+9

Nanyang Technological University · Tsinghua University · Peking University+8

AI Agent Reliability LLM Guardrails LLM Agent Verification AI Agent Monitoring Safety Filtering Runtime Enforcement for AI Agents+4

Open paper AskPDF arXiv

Finance LLM agents must simultaneously block prompt-induced unauthorized actions and approve legitimate multi-step business workflows. However, boundary filters often miss irreversible mid-trajectory tool calls, while post-hoc LLM judges perform auditing only after termination -- too late for intervention and at a computational cost that scales linearly with trace length. We present FinHarness, an inline safety harness that wraps a finance agent end-to-end with three components: a Query Monitor that fuses single-turn intent with cross-turn drift, a Tool Monitor that evaluates each prospective tool call, and a Cascade module that integrates per-step risk and adaptively routes verification between a lightweight and an advanced-tier LLM judge. Fired risk factors are re-injected into the agent input as ex-ante evidence, enabling the agent to refuse, re-plan, or approve on its own. On FinVault, routed FinHarness cuts ASR from 38.3% to 15.0% while largely preserving benign approval (

41.1\% \to 39.3\%

), and uses

4.7\times

fewer advanced-judge calls than an always-advanced ablation.

Abstract

LLM Agent Harnesses

Explore similar work

FinHarness: An Inline Lifecycle Safety Harness for Finance LLM Agents

QuantEval: A Benchmark for Financial Quantitative Tasks in Large Language Models

FinanceHarness: Autonomous Financial Deep Research Framework