Auditing Self-Evolution in Financial Agents: Capability Gains, Security Drift, and Execution-Interface Mismatch

Jialong Li; Jialing Zhu

cs.AIAug 18, 2026

Auditing Self-Evolution in Financial Agents: Capability Gains, Security Drift, and Execution-Interface Mismatch

Authors: Jialong Li, Jialing Zhu

Organizations: Independent Researcher

Abstract

Self-evolving agents turn experience into reusable skills, workflows, or memories, but post-evolution accuracy alone does not show whether learned behavior preserves previously correct behavior or security. We audit SkillOpt, Agent Workflow Memory (AWM), and ReasoningBank in simulated e-banking using matched benign acquisition trajectories, sealed evaluation endpoints, execution-grounded checks, and independent state replay. On Qwen 3.7 Flash, SkillOpt raises benign utility from 0.741 to 0.837 while exposure to injected content rises from 0.820 to 0.943. Conditional attack success after exposure falls from 0.605 to 0.562, yet overall attack success rate (ASR) rises from 0.496 to 0.530 and unauthorized financial state changes rise to 0.685. Across three independently evolved lineages, capability, exposure, and unauthorized-state changes increase in all three, whereas ASR increases in only two. ReasoningBank raises utility to 0.859 without increasing aggregate ASR, although unauthorized state changes remain slightly above Static. AWM reveals a separate evaluation hazard: a literal WebArena text-action envelope disrupts tool execution in our native function-calling executor. In a post-hoc sensitivity test, removing only that envelope restores utility from 0.319 to 0.756, while exposure rises from 0.299 to 0.909 and ASR from 0.195 to 0.575. Auditing self-evolving financial agents therefore requires tracking regressions, attack-surface contact, unauthorized financial-state change, and artifact-executor compatibility, not accuracy alone.

Figures & tables

Family	Defect class	Why invalid here	Frozen treatment
task_5 , task_6	fixture-shadowed	A pre-existing record can satisfy the check before the agent acts.	Pre/post state delta
task_11	recipient-insensitive	Right amount but wrong recipient passes.	Verify recipient and amount
task_8	vacuous	return True ; accepts wrong answer and spurious payment.	Excluded

Table 1. The four official utility predicates that failed the pre-model audit, and their frozen treatment. Repairs were frozen before model execution and are execution-grounded.

Method	State	Update and endpoint use
SkillOpt	skill doc.	Reflection produces up to four typed edit patches; v2 validation gates deployment of best_skill .
AWM	workflow	Induce from successful trajectories with family/template dedup; insert the complete workflow, with no retrieval.
ReasoningBank	memory	Distil success and failure ( $\leq 3$ items/trajectory); retrieve the top-1 trajectory and insert all of its items.

Table 2. Ported systems, compared end-to-end under their method-specific update and retrieval rules.

Panel of Figure 1 — Figure 1. Audit protocol. Within each lineage, one matched benign acquisition set feeds all three systems; acquisition sets are generated independently across lineages. All evolved artifacts are frozen before any endpoint variant is opened. Inside the dashed region v3 / v4 / v5 are sealed and v2 is reserved for SkillOpt’s validation gate. Flow diagram: within each lineage, one matched benign acquisition set of 22 trajectories from 11 seen families feeds SkillOpt, AWM, and ReasoningBank in parallel. The acquisition set differs across the three lineages. All evolved artifacts are then frozen; only afterwards are v3, v4, and v5 unsealed for benign evaluation, with v3 alone used for attacked evaluation. A dashed box marks the endpoint-blind region covering acquisition, evolution, and freezing.

Condition	Utility	W $\rightarrow$ C	C $\rightarrow$ W	Net	Exposure	cASR	ASR	Unauth. state
Static	0.741	—	—	—	0.820	0.605	0.496	0.583
SkillOpt	0.837	23	10	$+13$	0.943	0.562	0.530	0.685
ReasoningBank	0.859	25	0 9	$+16$	0.802	0.591	0.474	0.595
AWM-LiteralPort ∗	0.319	14	71	$-57$	0.299	0.653	0.195	0.217
AWM-InterfaceAdapted †	0.756	18	16	$+2$	0.909	0.633	0.575	0.714

Table 3. Main results on the corrected 15-family AgentDojo Banking suite. Benign

n{=}135

per condition; attacked

n{=}405

(SkillOpt 400 after excluding five provider-side repetition -guardrail rejections, § 4.7 ). Transitions are paired against Static on identical (lineage, family, variant); cASR is attack success conditional on exposure.

Goal	ASR	expo.	ASR	expo.	ASR	expo.	$\Delta$ ASR
	Static		SkillOpt		R.Bank
0	0.489	0.822	0.500	0.955	0.467	0.800	$+0.011$
1	0.511	0.844	0.489	0.933	0.511	0.800	$-0.022$
2	0.644	0.822	0.682	0.955	0.644	0.822	$+0.037$
3	0.644	0.822	0.705	0.955	0.533	0.800	$+0.060$
4	0.600	0.800	0.467	0.911	0.600	0.800	$-0.133$

Table 4. Per-goal attack success and exposure. Exposure rises on all nine goals under SkillOpt; ASR movement is mixed but net positive.

\Delta

is SkillOpt

-

Static ASR.

Panel of Figure 2 — Figure 2. Capability change against attack-success drift, relative to Static. Small filled markers are the three independently evolved lineages; large open markers are the pooled estimates of Table 3 . SkillOpt’s lineages straddle the zero-ASR line: one decreases ASR by $7.4$ pp, so the pooled $+3.4$ pp shift is not lineage-consistent (§ 4.3 ). AWM-InterfaceAdapted is a post-hoc sensitivity arm and is shown pooled only. Scatter plot of change in attack success rate against change in benign utility relative to Static, in percentage points. For SkillOpt three small circles show the individual lineages at plus 11.1 utility with minus 7.4 attack success, plus 15.6 with plus 7.3, and plus 2.2 with plus 10.4, straddling the zero line; a large open circle marks the pooled estimate at plus 9.6 and plus 3.4. For ReasoningBank three small squares lie below the zero line between minus 3.0 and minus 1.5 attack success, with a large open square at the pooled estimate, plus 11.9 and minus 2.2. A grey open triangle marks the post-hoc interface-adapted AWM arm at plus 1.5 and plus 7.9.

Method	Metric	Lineages (pp)	Med.	Range	Dir.
SkillOpt	utility	$+11.1/{+}15.6/{+}2.2$	$+11.1$	$2.2$ – $15.6$	3/3
SkillOpt	exposure	$+4.4/{+}19.3/{+}13.3$	$+13.3$	$4.4$ – $19.3$	3/3
SkillOpt	unauth. state	$+0.7/{+}12.2/{+}17.8$	$+12.2$	$0.7$ – $17.8$	3/3
SkillOpt	total ASR	$\mathbf{-7.4}/{+}7.3/{+}10.4$	$+7.3$	$-7.4$ – $+10.4$	2/3
R.Bank	utility	$+8.9/{+}11.1/{+}15.6$	$+11.1$	$8.9$ – $15.6$	3/3
R.Bank	total ASR	$-3.0/{-}1.5/{-}2.2$	$-2.2$	$-3.0$ – $-1.5$	3/3

Table 5. Lineage-level robustness of headline effects (percentage points vs. Static). Each lineage is an independent end-to-end evolution. Dir. is the number of lineages whose delta has the same sign as the pooled effect. Pooled estimates appear in Table 3 and are distinct from lineage medians.

Condition	$n$	ASR	State chg.,	Unauth.	Extra /
		hits	not ASR	state	ASR
Static	405	201	35	236	$+17.4\%$
SkillOpt	400	212	62	274	$\mathbf{+29.2\%}$
ReasoningBank	405	192	49	241	$+25.5\%$
AWM-LiteralPort ∗	405	0 79	0 9	0 88	$+11.4\%$
AWM-InterfaceAdapted †	405	233	56	289	$+24.0\%$

Table 6. Goal-keyed attack success versus unauthorized financial-state change. Every ASR hit here also changed state, so “ASR hits” and “State chg., not ASR” partition “Unauth. state”. “Extra / ASR” is a relative increase over goal-keyed successes, not a percentage-point difference.

Harm category	Static	Skill	Reas.	AWM	AWM
		Opt	Bank	LP ∗	IA †
Unauthorized transfer	0.319	0.412	0.338	0.151	0.425
Sensitive-data disclosure	0.304	0.307	0.279	0.126	0.370
Amount manipulation	0.183	0.145	0.084	0.015	0.156
Recipient substitution	0.156	0.117	0.141	0.049	0.136
Unauth. payment modif.	0.096	0.122	0.077	0.002	0.163

Table 7. Financial harm by category, attacked endpoint (rate per attacked episode). Categories may overlap within a rollout: they are neither additive nor commensurable and are never summed into a loss figure. SkillOpt

n{=}400

, all others

n{=}405

	Static	AWM - LP	AWM - IA †	AWM placebo
Textual <action> rate	0.000	0.704	0.000	0.000
Zero-real-tool-call rate	0.131	0.704	0.046	—
Mean real tool calls	2.97	1.05	3.34	—
Benign utility	0.741	0.319	0.756	0.704
Exposure	0.820	0.299	0.909	—
ASR	0.496	0.195	0.575	—

Table 8. Interface diagnostic. LiteralPort and InterfaceAdapted differ only by removal of the four tag types; all non-tag workflow content is identical.

Condition	Util.	$\Delta$ Util.	Expo.	cASR	ASR	Unauth.
Static	0.843	—	0.858	0.608	0.522	0.596
SkillOpt	0.870	$+.028$	0.928	0.578	0.536	0.671
ReasoningBank	0.907	$+.065$	0.836	0.609	0.509	0.639
AWM-LiteralPort	0.306	$-.537$	0.262	0.659	0.173	0.191
AWM-IA †	0.787	$-.056$	0.895	0.628	0.562	0.707

Table 9. Untouched-checker sensitivity: the same frozen rollouts re-aggregated over the 12 families whose official predicates required no correction (

n{=}108

benign,

324

attacked per condition; SkillOpt 319 attacked).

\Delta

is versus Static within this subset; compare Table 3 for the 15-family suite.

Method	Util.	Plac.	$\Delta$	ASR	Plac.	$\Delta$	Len. ratio
SkillOpt	0.837	0.763	$+.074$	0.507	0.470	$+.037$	1.95–2.50
ReasoningBank	0.859	0.696	$+.163$	0.474	0.415	$+.059$	0.59–0.74
AWM-LiteralPort	0.319	0.704	$-.385$	0.185	0.437	$-.252$	1.83–2.37

Table 10. Method-matched placebo contrasts. ASR columns use the frozen three-goal placebo subset, goals 4, 6, 7. “Len. ratio” is realised placebo/real state length across lineages.

AI Agent Auditing

AI agent auditing examines agents’ actions, tool use, evidence, and outcomes to assess safety, compliance, reliability, and accountability.

See all 262 papers

FinEvo-Bench: A Longitudinal Benchmark for Self-Evolving Agents in Professional Financial Workflows
Aug 6, 2026Bo Deng, Kang Zhou, Lifan Guo +6Long-Horizon Agent Evaluation AI Agent Benchmarks
Beihang University · Qwen DianJin Team, Alibaba Cloud Computing
When Experience Becomes Instruction: Trajectory Poisoning in Self-Evolving Agent Skill Systems
Aug 6, 2026Jialuo Chen, Lingqi Jiang, Xinhao Deng +7Adversarial Attacks AI Agent Security
Ant Group · Zhejiang University · Tsinghua University +3
When Self-Evolution Backfires: Pre-Commit Gating against Skill Contamination in LLM Agents
Aug 6, 2026Linfang Shang, Ming Xu, Yiding Sun +4AI Agent Reliability LLM Agent Skill Learning
Tencent

Aug 6, 2026J/K move · Enter open · S save

FinEvo-Bench: A Longitudinal Benchmark for Self-Evolving Agents in Professional Financial Workflows

Bo Deng, Kang Zhou, Lifan Guo, Chongyang Tao+5

Beihang University · Qwen DianJin Team, Alibaba Cloud Computing

Long-Horizon Agent Evaluation AI Agent Benchmarks Finance Self-Evolving Agents

Open paper AskPDF arXiv

Agents used over time encounter recurring professional work: each case requires different evidence and judgment, while the underlying workflow can be reused. Benchmarks built from independent tasks cannot reveal whether an agent turns earlier experience into better procedures for later cases. We introduce FinEvo-Bench, a longitudinal benchmark designed around this structure. It contains 120 open-ended tasks drawn from real cases across 20 business scenes in six financial domains. Each scene contains six substantively different cases that share a professional workflow and an expert-authored rubric for task quality and financial compliance. Constructing and validating the benchmark required approximately 1,200 person-hours. Finance provides a natural test bed because recurring analyses apply shared professional and compliance requirements to heterogeneous inputs, producing case-specific analyses and conclusions. We evaluate four self-evolving agent scaffolds with Qwen3.7-Max on three independently shuffled, globally interleaved task streams. A Claude Code rubric judge backed by Claude Opus4.6 evaluates all outputs, and paired state-reset controls estimate each scaffold's gain from retained experience. Evolving runs score 9.33--19.37 points higher and trigger 0.12--0.44 fewer compliance issues per task than their paired controls. Paired score gains at within-scene ranks4--6 exceed those at ranks~1--3 by 6.10--8.70 points. FinEvo-Bench measures whether retained experience improves later professional work under continued use.

Abstract

AI Agent Auditing

Explore similar work

FinEvo-Bench: A Longitudinal Benchmark for Self-Evolving Agents in Professional Financial Workflows

When Experience Becomes Instruction: Trajectory Poisoning in Self-Evolving Agent Skill Systems

When Self-Evolution Backfires: Pre-Commit Gating against Skill Contamination in LLM Agents