MemTrial: Learning When to Trust Memory in LLM Portfolio Agents

Guanghao Wu; Zhuo Cai; Shoujin Wang

cs.AIOct 8, 2026

MemTrial: Learning When to Trust Memory in LLM Portfolio Agents

Authors: Guanghao Wu, Zhuo Cai, Shoujin Wang

Organizations: University of Technology Sydney

Abstract

Large language model (LLM) agents for portfolio management learn from experience: they credit each experience in their memory with the outcome of the decisions that used it. In financial markets, however, this outcome mostly reflects the market move shared by all decisions on that date, so the credit tracks the market rather than the experience, and these agents often do worse than simply holding the equal-weight (1/ $N$ ) portfolio. We ask how an agent can credit an experience with what it changes, and answer it by putting memory on trial: drafts of the same decision with and without an experience face the same market, so the outcome they share cancels in their difference. Our agent, MemTrial, drafts each decision with eight combinations of its retrieved experiences, chosen by a fractional factorial design, and credits each experience with its Banzhaf value, the average of these differences. As each date occurs once and each draft is a noisy LLM sample, these credits are noisy and may not hold on new dates. MemTrial therefore pools them across dates and similar experiences with a hierarchical Bayesian model, acts on them only after they have predicted unseen dates, and otherwise stays anchored at a conservative reference such as 1/ $N$ . On four benchmarks, MemTrial not only benefits from experiences that matter (the best of 15 methods on a semi-synthetic benchmark with known experience quality) but also limits its losses when its values do not hold (at most 2.2% below 1/ $N$ on PortBench and InvestorBench, against 15--38% for the best experience-learning agent). Averaged over five settings, it improves the utility of the best experience-learning agent by 21.2%, and with eight LLMs it beats every LLM-based baseline on InvestorBench.

Figures & tables

Figure 1. (a) Existing agents credit experiences with market-driven outcomes and always use these credits. (b) MemTrial measures what each experience changes on the same date and acts on its values only after they have predicted unseen dates; otherwise it stays near 1/

N

. Two rows of icons. Row (a), existing agents: a robot retrieves four experiences; the outcome is mostly the market; all experiences are credited, and the resulting values track the market; the agent always acts on them without a check, marked with a warning sign. Row (b), MemTrial: drafts with and without an experience h on the same date; the shared move cancels, and their difference is what h changes; differences are pooled across dates; a trust test asks whether the values predict unseen dates; if yes, the agent acts on the values; if no, it stays near 1/N, shown with an anchor.

	Spearman correlation ( $\rho$ )
Benchmark	Outcome credit vs. 1/ $N$	Contribution vs. 1/ $N$	Outcome credit vs. contribution	Dates with a detectable effect
PortBench-Full	$+0.91$ $\pm$ 0.00	$+0.04$ $\pm$ 0.03	$+0.11$ $\pm$ 0.06	3–4 / 56
PortBench-Raw	$+0.92$ $\pm$ 0.00	$+0.03$ $\pm$ 0.04	$+0.04$ $\pm$ 0.03	2–4 / 56
InvestorBench	$+0.95$ $\pm$ 0.00	$-0.07$ $\pm$ 0.02	$-0.03$ $\pm$ 0.02	126–127 / 149
ClassAlloc	$+0.97$ $\pm$ 0.00	$+0.02$ $\pm$ 0.07	$+0.02$ $\pm$ 0.07	53–57 / 71

Table 1. Outcome credit tracks the market, not the experience.

Figure 2. One decision date of MemTrial. Before the outcome, it acts on its learned values only if they have predicted unseen dates; after the outcome, it updates the values and scores their predictions. Contributions compare drafts of one date, so their shared market move cancels. Flow chart with two bands. Top band, act before the outcome: state and memory with four experiences; eight subsets give eight drafts; a trust test leads to the best draft if trusted or to an anchored blend of 1/N and the drafts if not; the deployed portfolio is held until the outcome is realized. Bottom band, learn after the outcome: Banzhaf contributions compare drafts with and without each experience on the same date, so the shared move cancels; Bayesian values per experience with error bars, pooled across dates and content; forward scores compare earlier values with the new contributions; values and past scores return to the trust test of the next date.

	PortBench	InvestorBench	ClassAlloc	PlantedMem
Assets	99 (6 classes)	4 stocks, cash	5 classes	4 classes
Decisions	monthly	daily	monthly	monthly
Development	2019–22	none	none	15 worlds
Test dates	20 (2023–24)	149 (10/20–05/21)	71 (05/20–03/26)	120 / world
Experiences	338 decisions	63 lessons	58 lessons	12/48 planted
Seeds	3	3	3	100 worlds

Table 2. Three real benchmarks and one semi-synthetic benchmark with planted experiences of known quality.

	Method	PortBench-Full	PortBench-Raw	InvestorBench	ClassAlloc	PlantedMem
		%/month	%/month	bp/day	%/month	%/month
Rule-based	1/ $N$ ( DeMiguel et al., 2009 )	2.02 $\pm$ 0.00	2.02 $\pm$ 0.00	10.45 $\pm$ 0.00	0.623 $\pm$ 0.000	0.57 $\pm$ 0.00
Rule-based	Minimum variance ( Markowitz, 1955 )	0.25 $\pm$ 0.00	0.25 $\pm$ 0.00	5.53 $\pm$ 0.00	0.230 $\pm$ 0.000	$-$ 0.04 $\pm$ 0.00
LLM w/o experience learning	Memory-free ( Brown et al., 2020 )	1.60 $\pm$ 0.05	1.07 $\pm$ 0.02	8.43 $\pm$ 0.29	0.657 $\pm$ 0.001	0.76 $\pm$ 0.08
	Self-consistency ( Wang et al., 2022 )	1.56 $\pm$ 0.11	1.17 $\pm$ 0.03	7.89 $\pm$ 0.21	0.686 $\pm$ 0.007	0.79 $\pm$ 0.03
	Similarity retrieval (top-2) ( Lewis et al., 2020 )	1.43 $\pm$ 0.19	1.24 $\pm$ 0.29	8.67 $\pm$ 0.19	0.649 $\pm$ 0.033	0.72 $\pm$ 0.07

Table 3. Main results: utility per decision period (higher is better). MemTrial is the best LLM-based method on PortBench and InvestorBench, where it stays within 2.2% of 1/

N

, and the best method in PlantedMem.

Figure 3. Wealth of the balanced investor (top, start

=1

) and relative to 1/

N

for the conservative, balanced and aggressive investors (rows 2–4). For every investor, MemTrial stays close to 1/

N

, while the other LLM-based methods drift away, mostly below. Twelve line charts in four rows for PortBench-Full, InvestorBench and ClassAlloc. Top row: wealth of the balanced investor over the test period for 1/N, the memory-free LLM, FinMem, MemRL, Reflexion, ExpeL and MemTrial; rows two to four: wealth relative to 1/N in percent for the conservative, balanced and aggressive investors. Falling markets are shaded red, and the ClassAlloc panels mark the LLM's knowledge cutoff with a dotted line. MemTrial stays near 1/N for every investor, while the other agents drift away from it, mostly below.

		PortBench-Full			InvestorBench			ClassAlloc
	Method	CR % $\uparrow$	SR $\uparrow$	MDD % $\downarrow$	CR % $\uparrow$	SR $\uparrow$	MDD % $\downarrow$	CR % $\uparrow$	SR $\uparrow$	MDD % $\downarrow$
Rule-based	1/ $N$ ( DeMiguel et al., 2009 )	61.2 $\pm$ 0.0	1.77 $\pm$ 0.00	5.6 $\pm$ 0.0	19.1 $\pm$ 0.0	2.87 $\pm$ 0.00	5.7 $\pm$ 0.0	82.0 $\pm$ 0.0	1.22 $\pm$ 0.00	12.0 $\pm$ 0.0
Rule-based	Minimum variance ( Markowitz, 1955 )	5.4 $\pm$ 0.0	1.19 $\pm$ 0.00	1.5 $\pm$ 0.0	9.9 $\pm$ 0.0	1.47 $\pm$ 0.00	6.4 $\pm$ 0.0	19.1 $\pm$ 0.0	4.19 $\pm$ 0.00	0.2 $\pm$ 0.0
LLM w/o exp. learning	Memory-free ( Brown et al., 2020 )	48.5 $\pm$ 0.8	1.47 $\pm$ 0.06	7.3 $\pm$ 1.0	14.6 $\pm$ 0.5	2.24 $\pm$ 0.08	6.3 $\pm$ 0.1	88.3 $\pm$ 1.5	1.11 $\pm$ 0.00	13.9 $\pm$ 0.3
Experience- learning agents	FinMem ( Yu et al., 2025 )	49.6 $\pm$ 9.2	1.53 $\pm$ 0.21	6.2 $\pm$ 1.4	14.7 $\pm$ 0.4	2.27 $\pm$ 0.03	6.1 $\pm$ 0.1	75.4 $\pm$ 2.6	1.07 $\pm$ 0.03	15.4 $\pm$ 0.5
	MemRL ( Zhang et al., 2026 )	47.7 $\pm$ 11.1	1.45 $\pm$ 0.19	6.9 $\pm$ 1.1	15.4 $\pm$ 0.7	2.33 $\pm$ 0.09	6.1 $\pm$ 0.3	84.0 $\pm$ 1.2	1.06 $\pm$ 0.03	15.4 $\pm$ 1.1

Table 4. Return and risk of the balanced investor: cumulative net return (CR), annualized Sharpe ratio (SR) and maximum drawdown (MDD). On PortBench and InvestorBench, MemTrial is the only LLM-based method close to 1/

N

in both return and Sharpe ratio.

Figure 4. Ablation: change in utility when one step of MemTrial is changed (%, mean over seeds; $=$ : same actions; bold: $p<0.05$ ). On real markets, the anchor and the trust test matter most; with signal, acting on the learned values does. Heat map with eight ablation variants in rows, grouped into value estimation, trust test and anchored action, and seven settings in columns: PortBench-Full, PortBench-Raw, InvestorBench and ClassAlloc, and PlantedMem without signal, with signal and with informative content. Each cell gives the change in utility relative to MemTrial in percent; red cells are losses and blue cells gains. Falling back to the draft average loses 16 to 41 percent on PortBench and InvestorBench, and never acting on the learned values loses 15 percent in PlantedMem with signal.

Figure 5. Outcome credit does not learn which experiences are useful; counterfactual contributions do, and pooling them by content helps when content is informative. Two line charts over ten years of PlantedMem decisions (balanced investor, mean over 100 worlds); the y-axis is the rank correlation between the learned values and the planted quality of the experiences. Panel (a), beta 0.25: outcome credit stays below 0.1; counterfactual selection rises from about 0.36 to 0.81; MemTrial rises from about 0.17 to 0.81. Panel (b), 48 experiences with informative content: outcome credit stays below 0.1; counterfactual selection rises from 0.26 to 0.55; MemTrial rises from 0.02 to 0.66 and overtakes counterfactual selection after about two years.

Figure 6. PlantedMem by regime. MemTrial abstains when experiences carry no signal and acts on its values when they matter. Two bar charts over seven PlantedMem regimes. Top: gain over the reference in pp per month for outcome credit (MemRL), counterfactual selection and MemTrial. Outcome credit loses in every regime; counterfactual selection loses up to 0.13 when experiences carry no signal and gains up to 0.31 when they matter; MemTrial stays near zero without signal and gains up to 0.30. Bottom: MemTrial's trust rate, 2 to 3 percent without signal and 64 to 85 percent at beta 0.25 and 0.5.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$t$ , $T$	decision date, number of dates
$s_{t}$	market state at date $t$
$N$	number of assets (plus cash)
$\theta=(M,m,\gamma)$	investor profile: maximum share of risky assets, minimum share of defensive assets, risk aversion
$P_{\theta}$	projection of a portfolio onto the investor’s mandate
$u_{t}(x)$ ; $R_{t}$ , $V_{t}$	investor utility (certainty-equivalent return) of portfolio $x$ , Eq. ( 1 ); net return and realized variance over the holding period

Appendix

Table B.1. Notation.

	PortBench (Full / Raw)	InvestorBench	ClassAlloc	PlantedMem
Decision dates	59 monthly, Aug 2019 – Dec 2024 (a month whose 20-day window would overlap the previous one is skipped); 56 usable per configuration	63 warm-up and 149 test trading days, Jul 2020 – May 2021	129 monthly, Jul 2015 – Mar 2026: 58 warm-up and 71 test	132 monthly, Apr 2015 – Mar 2026, per episode; the first 12 are a warm-up
Development / test	36 / 20 months	none (fully held out) / 149 days	none (fully held out) / 71 months	seeds 0–14 / 15–114
Assets	99 in 6 classes, including cash	HON, JNJ, MSFT, UVV and cash	equities, bonds, commodities, real estate and cash: equal-weighted indices of 89 assets	equities, bonds, cash, commodities
Experiences	338 records of earlier decisions (Apr 2015 – Jul 2019); 29 retrieved	63 lessons written in the warm-up	58 lessons written in the warm-up	12 or 48 planted per episode
Drafts per date and seed	19: the 16 subsets of the 4 retrieved experiences, FinMem, MemRL and the benchmark’s native memory; Reflexion runs separately	20: the 16 subsets, ExpeL, FinMem, MemRL and Reflexion	20: the 16 subsets, ExpeL, FinMem, MemRL and Reflexion	resampled from 2,828 logged LLM allocations
LLM executions	6,612 (6,518 valid) and 354 Reflexion decisions	9,003	4,318 (58 in the warm-up), and 210 redrawn (Appendix D.3 )	none

Appendix

Table D.1. Benchmark statistics and cost.

Benchmark	Method	Utility	Gross return	Trading cost	Risk penalty	L1
PortBench-Full	1/ $N$	2.02 $\pm$ 0.00	2.25 $\pm$ 0.00	0.03 $\pm$ 0.00	0.21 $\pm$ 0.00	0.17 $\pm$ 0.00
	Memory-free	1.60 $\pm$ 0.05	2.12 $\pm$ 0.05	0.17 $\pm$ 0.02	0.34 $\pm$ 0.03	1.13 $\pm$ 0.11
	Sim. retrieval (top-4)	1.70 $\pm$ 0.06	2.19 $\pm$ 0.04	0.18 $\pm$ 0.02	0.31 $\pm$ 0.02	1.17 $\pm$ 0.14
	Draft averaging	1.62 $\pm$ 0.05	2.09 $\pm$ 0.06	0.16 $\pm$ 0.00	0.32 $\pm$ 0.02	1.02 $\pm$ 0.01
PortBench-Raw	1/ $N$	2.02 $\pm$ 0.00	2.25 $\pm$ 0.00	0.03 $\pm$ 0.00	0.21 $\pm$ 0.00	0.17 $\pm$ 0.00
	Memory-free	1.07 $\pm$ 0.02	1.43 $\pm$ 0.03	0.18 $\pm$ 0.01	0.17 $\pm$ 0.00	1.19 $\pm$ 0.08

Appendix

Table D.2. Decomposition of the test-period utility. The LLM drafts trail 1/

N

in all three parts on PortBench-Full, mostly in gross return on PortBench-Raw and mostly in trading costs on InvestorBench.

Variant	PortBench	InvestorBench	ClassAlloc	PlantedMem (core)	PlantedMem (informative)
Ablation (Figure 4 )
MemTrial (ours)	0.3 $\pm$ 0.5	0.3 $\pm$ 0.5	14.6 $\pm$ 21.7	33.7 $\pm$ 9.2	44.9 $\pm$ 22.5
per-experience only (no content)	0.0 $\pm$ 0.0	0.3 $\pm$ 0.5	14.6 $\pm$ 21.7	31.8 $\pm$ 8.9	22.0 $\pm$ 20.7
content-aware only (no learner selection)	1.1 $\pm$ 1.3	0.3 $\pm$ 0.5	14.6 $\pm$ 21.7	32.6 $\pm$ 9.0	44.5 $\pm$ 23.1
in-sample F-test	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	2.3 $\pm$ 4.1	29.3 $\pm$ 8.4	15.1 $\pm$ 17.8
no trust test	24.7 $\pm$ 21.4	34.5 $\pm$ 18.8	53.8 $\pm$ 24.7	54.3 $\pm$ 12.6	64.9 $\pm$ 18.6

Appendix

Table D.3. Trust rate (%) of every variant. With the trust test, MemTrial acts on its values on at most 0.3% of PortBench and InvestorBench decisions; without it, on 25–35%.

						PlantedMem
Step changed	Variant	PortBench-Full	PortBench-Raw	InvestorBench	ClassAlloc	no signal	signal	informative content	core
–	MemTrial (ours)	1.98 $\pm$ 0.00	2.01 $\pm$ 0.00	10.40 $\pm$ 0.04	0.613 $\pm$ 0.017	0.784 $\pm$ 0.031	0.932 $\pm$ 0.070	0.862 $\pm$ 0.068	0.873 $\pm$ 0.045
Value estimation	per-experience learner only	1.98 $\pm$ 0.00	2.01 $\pm$ 0.00	10.40 $\pm$ 0.04	0.613 $\pm$ 0.017	0.786 $\pm$ 0.030	0.929 $\pm$ 0.069	0.817 $\pm$ 0.048	0.872 $\pm$ 0.045
	content-aware learner only	1.98 $\pm$ 0.00	1.94 $\pm$ 0.12	10.40 $\pm$ 0.04	0.613 $\pm$ 0.017	0.785 $\pm$ 0.031	0.929 $\pm$ 0.072	0.866 $\pm$ 0.072	0.871 $\pm$ 0.046
Trust test	in-sample F-test	1.98 $\pm$ 0.00	2.01 $\pm$ 0.00	10.37 $\pm$ 0.00	0.628 $\pm$ 0.001	0.786 $\pm$ 0.029	0.924 $\pm$ 0.064	0.814 $\pm$ 0.045	0.869 $\pm$ 0.043
	none (always act on values)	1.89 $\pm$ 0.11	1.77 $\pm$ 0.41	9.91 $\pm$ 0.45	0.612 $\pm$ 0.005	0.767 $\pm$ 0.038	0.940 $\pm$ 0.080	0.882 $\pm$ 0.068	0.871 $\pm$ 0.052

Appendix

Table D.4. Utilities behind Figure 4 , with the variant that always keeps the reference (

q=1

; Appendix D.8 ) (% per month, bp per day on InvestorBench; mean

\pm

sd over seeds). Without signal, the variants differ little in PlantedMem; with signal, never acting on the learned values stays at the reference.

					PlantedMem
Method	PortBench-Full	PortBench-Raw	InvestorBench	ClassAlloc	no signal	signal	informative content
1/ $N$	2.021 $\pm$ 0.000	2.021 $\pm$ 0.000	10.45 $\pm$ 0.00	0.623 $\pm$ 0.000	0.575 $\pm$ 0.000	0.575 $\pm$ 0.000	0.575 $\pm$ 0.000
MemTrial (ours)	1.976 $\pm$ 0.005	2.007 $\pm$ 0.002	10.40 $\pm$ 0.04	0.613 $\pm$ 0.017	0.784 $\pm$ 0.031	0.932 $\pm$ 0.070	0.862 $\pm$ 0.068
Hedge, prior 0.9 on the reference	1.992 $\pm$ 0.009	2.003 $\pm$ 0.003	10.41 $\pm$ 0.01	0.627 $\pm$ 0.001	0.786 $\pm$ 0.029	0.787 $\pm$ 0.029	0.787 $\pm$ 0.030
With the anchored action of MemTrial
Memory-free	1.985 $\pm$ 0.008	1.998 $\pm$ 0.004	10.34 $\pm$ 0.02	0.630 $\pm$ 0.001	0.789 $\pm$ 0.030	0.789 $\pm$ 0.032	0.788 $\pm$ 0.030

Appendix

Table D.5. Every LLM-based baseline with the anchored action of MemTrial (Eq. ( 11 ); % per month, bp per day on InvestorBench; mean

\pm

sd over seeds). With the same anchor, the baselines do as well as MemTrial on the real benchmarks; with signal in PlantedMem, MemTrial earns 11% more than the best of them.

Figure D.1. Wealth relative to 1/

N

in every setting. MemTrial stays within 2% of 1/

N

throughout, whereas the other agents fall up to 21% behind. Twelve line charts in four rows (PortBench-Full, PortBench-Raw, InvestorBench, ClassAlloc) and three columns (conservative, balanced and aggressive investor) showing wealth relative to 1/N in percent for 1/N, the memory-free LLM, FinMem, MemRL, Reflexion, ExpeL and MemTrial. MemTrial stays near zero; the other agents fall up to 21 percent behind.

Figure D.2. Wealth in every setting (start

=1

). MemTrial tracks 1/

N

; on PortBench and InvestorBench, the others end below it. Twelve line charts in the same grid showing wealth, starting at 1, for the same methods. MemTrial tracks 1/N; the other agents end below it on PortBench and InvestorBench.

Figure D.3. InvestorBench with other LLMs and temperatures: wealth relative to 1/

N

. MemTrial stays within 1.2% of 1/

N

in every configuration; the other agents end 1.5–6.7% behind. Eleven line charts, one per InvestorBench configuration (eight LLMs and three further temperatures), showing wealth relative to 1/N for the balanced investor. MemTrial stays near zero; the other agents end 1.5 to 6.7 percent behind.

Method	PortBench			InvestorBench			ClassAlloc
	End	Falling	Rising	End	Falling	Rising	End	Falling	Rising
Memory-free	$-11.10$ $\pm$ 0.46	$-0.65$ $\pm$ 0.58	$-10.50$ $\pm$ 0.61	$-3.15$ $\pm$ 0.36	$-0.65$ $\pm$ 0.06	$-2.52$ $\pm$ 0.39	$+0.31$ $\pm$ 0.97	$-1.59$ $\pm$ 0.25	$+2.08$ $\pm$ 0.89
FinMem	$-9.05$ $\pm$ 1.93	$+0.75$ $\pm$ 1.48	$-9.80$ $\pm$ 0.96	$-2.54$ $\pm$ 0.20	$-0.41$ $\pm$ 0.03	$-2.14$ $\pm$ 0.22	$-4.62$ $\pm$ 0.25	$-3.17$ $\pm$ 0.10	$-1.33$ $\pm$ 0.35
MemRL	$-11.44$ $\pm$ 2.82	$-0.93$ $\pm$ 0.38	$-10.59$ $\pm$ 2.64	$-2.46$ $\pm$ 0.10	$-0.50$ $\pm$ 0.11	$-1.97$ $\pm$ 0.11	$-2.50$ $\pm$ 0.22	$-3.53$ $\pm$ 0.20	$+1.35$ $\pm$ 0.07
Reflexion	$-11.42$ $\pm$ 1.16	$+0.11$ $\pm$ 1.31	$-11.50$ $\pm$ 1.78	$-4.60$ $\pm$ 0.10	$-0.52$ $\pm$ 0.06	$-4.09$ $\pm$ 0.11	$-2.53$ $\pm$ 0.87	$-1.45$ $\pm$ 0.67	$-0.98$ $\pm$ 0.71
ExpeL	$-11.29$ $\pm$ 0.88	$-0.05$ $\pm$ 1.03	$-11.27$ $\pm$ 0.83	$-2.75$ $\pm$ 0.16	$-0.38$ $\pm$ 0.03	$-2.37$ $\pm$ 0.18	$-3.88$ $\pm$ 0.39	$-3.44$ $\pm$ 0.26	$+0.17$ $\pm$ 0.33

Appendix

Table D.6. Wealth relative to 1/

N

(%) at the end and by market phase. MemTrial stays within 1% of 1/

N

in every phase; the other agents lose most in the rising phases of PortBench and InvestorBench.

Method	Shortfall $<2\%$	Shortfall $<5\%$	Shortfall $<10\%$	Median largest shortfall (%)
Memory-free	27.8 $\pm$ 3.7	77.8 $\pm$ 3.7	94.4 $\pm$ 0.0	3.15 $\pm$ 0.19
FinMem	19.1 $\pm$ 3.9	67.9 $\pm$ 3.9	90.7 $\pm$ 3.2	3.59 $\pm$ 0.12
MemRL	31.5 $\pm$ 4.9	74.7 $\pm$ 1.1	93.2 $\pm$ 1.1	3.04 $\pm$ 0.26
Reflexion	14.2 $\pm$ 4.3	55.6 $\pm$ 1.9	88.9 $\pm$ 3.7	4.60 $\pm$ 0.24
ExpeL	26.5 $\pm$ 2.1	69.8 $\pm$ 3.9	83.3 $\pm$ 3.2	3.01 $\pm$ 0.36
MemTrial (ours)	93.2 $\pm$ 1.1	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	0.20 $\pm$ 0.01

Appendix

Table D.7. Survival against 1/

N

(% of runs). MemTrial never falls 5% behind 1/

N

; the others do in 22–44% of runs.

Method	PortBench			InvestorBench			ClassAlloc
	CR	SR	MDD	CR	SR	MDD	CR	SR	MDD
1/ $N$	53.1 $\pm$ 0.0	1.72 $\pm$ 0.00	5.7 $\pm$ 0.0	18.1 $\pm$ 0.0	2.87 $\pm$ 0.00	5.4 $\pm$ 0.0	82.0 $\pm$ 0.0	1.22 $\pm$ 0.00	12.0 $\pm$ 0.0
Memory-free	36.0 $\pm$ 0.8	1.43 $\pm$ 0.04	6.8 $\pm$ 0.8	14.4 $\pm$ 0.4	2.26 $\pm$ 0.06	6.0 $\pm$ 0.1	82.5 $\pm$ 1.8	1.14 $\pm$ 0.02	12.7 $\pm$ 0.3
FinMem	39.1 $\pm$ 3.0	1.53 $\pm$ 0.08	6.2 $\pm$ 0.6	15.1 $\pm$ 0.2	2.42 $\pm$ 0.03	5.8 $\pm$ 0.0	73.5 $\pm$ 0.5	1.12 $\pm$ 0.00	13.4 $\pm$ 0.0
MemRL	35.4 $\pm$ 4.4	1.39 $\pm$ 0.05	6.5 $\pm$ 0.4	15.2 $\pm$ 0.1	2.39 $\pm$ 0.03	5.9 $\pm$ 0.1	77.4 $\pm$ 0.4	1.11 $\pm$ 0.00	13.5 $\pm$ 0.3
Reflexion	35.4 $\pm$ 1.8	1.43 $\pm$ 0.09	6.8 $\pm$ 0.8	12.6 $\pm$ 0.1	2.02 $\pm$ 0.01	5.9 $\pm$ 0.0	77.4 $\pm$ 1.6	1.14 $\pm$ 0.01	13.0 $\pm$ 0.4

Appendix

Table D.8. Cumulative net return (CR, %), annualized Sharpe ratio (SR) and maximum drawdown (MDD, %). On every benchmark, MemTrial has the highest Sharpe ratio and the smallest drawdown of the LLM-based agents.

Figure D.4. Effect of the trust-test level

\alpha

(top: utility; bottom: trust rate). On real markets, a looser test acts on chance and loses; in PlantedMem, it acts earlier on values that hold. Six small line charts in two rows for PortBench (Full and Raw), InvestorBench and PlantedMem (informative regime), with the trust-test level alpha at 0.01, 0.02, 0.05 (default), 0.1 and 0.2. Top row, utility: PortBench is flat up to 0.05 and falls by about 2 to 4 percent at 0.2; InvestorBench is flat; PlantedMem rises from 0.84 to 0.88 percent per month. Bottom row, trust rate: PortBench rises from 0 to 17 percent (Full) and 7 percent (Raw) at 0.2; InvestorBench stays below 2 percent; PlantedMem rises from 33 to 56 percent.

Hyperparameter	Value	PortBench-Full	PortBench-Raw	InvestorBench	PlantedMem (core)	PlantedMem (informative)
Test level $\alpha$	0.01	1.98 $\pm$ 0.00	2.01 $\pm$ 0.00	10.37 $\pm$ 0.00	0.87 $\pm$ 0.04	0.84 $\pm$ 0.07
	0.02	1.98 $\pm$ 0.00	2.01 $\pm$ 0.00	10.37 $\pm$ 0.00	0.87 $\pm$ 0.04	0.85 $\pm$ 0.07
	0.05 (default)	1.98 $\pm$ 0.00	2.01 $\pm$ 0.00	10.40 $\pm$ 0.04	0.87 $\pm$ 0.04	0.86 $\pm$ 0.07
	0.1	2.01 $\pm$ 0.05	1.98 $\pm$ 0.04	10.42 $\pm$ 0.09	0.87 $\pm$ 0.05	0.87 $\pm$ 0.07
	0.2	1.94 $\pm$ 0.05	1.93 $\pm$ 0.13	10.39 $\pm$ 0.07	0.87 $\pm$ 0.05	0.88 $\pm$ 0.07
Minimum forward scores	5	1.98 $\pm$ 0.00	2.01 $\pm$ 0.00	10.40 $\pm$ 0.04	0.87 $\pm$ 0.05	0.86 $\pm$ 0.07

Appendix

Table D.9. Sensitivity to the trust test. Test levels up to 0.05 give identical PortBench results; looser levels help only in the informative PlantedMem regime.

$k$	Design	Drafts	PortBench-Full	PortBench-Raw	InvestorBench	PlantedMem (core)	PlantedMem (informative)
2	full factorial	4	1.98 $\pm$ 0.01	2.01 $\pm$ 0.00	10.27 $\pm$ 0.11	0.812 $\pm$ 0.043	0.798 $\pm$ 0.042
3	half fraction	4	1.97 $\pm$ 0.02	2.01 $\pm$ 0.00	10.14 $\pm$ 0.21	0.829 $\pm$ 0.041	0.799 $\pm$ 0.047
3	full factorial	8	1.98 $\pm$ 0.01	2.01 $\pm$ 0.00	9.96 $\pm$ 0.36	0.865 $\pm$ 0.051	0.850 $\pm$ 0.070
4	half fraction (default)	8	1.98 $\pm$ 0.00	2.01 $\pm$ 0.00	10.40 $\pm$ 0.04	0.868 $\pm$ 0.045	0.862 $\pm$ 0.072
4	full factorial	16	1.97 $\pm$ 0.00	1.95 $\pm$ 0.11	9.99 $\pm$ 0.34	0.889 $\pm$ 0.047	0.903 $\pm$ 0.085
5	half fraction	16	–	–	–	0.896 $\pm$ 0.044	0.914 $\pm$ 0.078

Appendix

Table D.10. Number of retrieved experiences

k

and the design. More experiences help when experiences matter, with diminishing returns per draft; at equal cost, the half fraction is the better design.

$\alpha_{0}$	$\lambda_{0}$	PortBench- Full	PortBench- Raw	InvestorBench	PlantedMem (core)
0.5	1	1.80 $\pm$ 0.02	2.02 $\pm$ 0.00	9.99 $\pm$ 0.03	0.87 $\pm$ 0.04
0.5	4	1.83 $\pm$ 0.02	1.92 $\pm$ 0.01	9.71 $\pm$ 0.02	0.87 $\pm$ 0.04
0.5	16	1.85 $\pm$ 0.02	1.72 $\pm$ 0.01	9.65 $\pm$ 0.04	0.86 $\pm$ 0.04
0.75	1	1.86 $\pm$ 0.03	2.02 $\pm$ 0.00	10.29 $\pm$ 0.05	0.87 $\pm$ 0.04
0.75	4	1.92 $\pm$ 0.01	1.98 $\pm$ 0.00	10.16 $\pm$ 0.02	0.87 $\pm$ 0.04
0.75	16	1.94 $\pm$ 0.01	1.89 $\pm$ 0.00	10.12 $\pm$ 0.01	0.87 $\pm$ 0.04

Appendix

Table D.11. Strength of the anchor (

\alpha_{0}=1

: always the reference). A uniform prior (

\alpha_{0}=0.5

) costs the most; with

\alpha_{0}\geq 0.9

, every

\lambda_{0}

stays close to the default.

Design	Drafts	No influence	Noise only	$\beta=0.1$	$\beta=0.25$	$\beta=0.5$	Many exp.	Informative	Core avg.
Reference	8	0.782 $\pm$ 0.031	0.782 $\pm$ 0.031	0.782 $\pm$ 0.031	0.782 $\pm$ 0.031	0.782 $\pm$ 0.031	0.782 $\pm$ 0.031	0.782 $\pm$ 0.031	0.782 $\pm$ 0.031
$k=2$ , full factorial	4	0.784 $\pm$ 0.029	0.782 $\pm$ 0.038	0.785 $\pm$ 0.029	0.804 $\pm$ 0.048	0.904 $\pm$ 0.130	0.782 $\pm$ 0.029	0.798 $\pm$ 0.042	0.812 $\pm$ 0.043
$k=3$ , half fraction	4	0.784 $\pm$ 0.032	0.784 $\pm$ 0.042	0.787 $\pm$ 0.038	0.821 $\pm$ 0.061	0.970 $\pm$ 0.123	0.789 $\pm$ 0.036	0.799 $\pm$ 0.047	0.829 $\pm$ 0.041
$k=3$ , full factorial	8	0.782 $\pm$ 0.028	0.778 $\pm$ 0.032	0.784 $\pm$ 0.031	0.904 $\pm$ 0.089	1.078 $\pm$ 0.154	0.812 $\pm$ 0.048	0.850 $\pm$ 0.070	0.865 $\pm$ 0.051
$k=4$ , half fraction (default)	8	0.783 $\pm$ 0.028	0.777 $\pm$ 0.037	0.794 $\pm$ 0.045	0.901 $\pm$ 0.084	1.085 $\pm$ 0.114	0.836 $\pm$ 0.079	0.862 $\pm$ 0.072	0.868 $\pm$ 0.045
$k=4$ , full factorial	16	0.780 $\pm$ 0.030	0.778 $\pm$ 0.029	0.792 $\pm$ 0.049	0.950 $\pm$ 0.080	1.143 $\pm$ 0.134	0.869 $\pm$ 0.094	0.903 $\pm$ 0.085	0.889 $\pm$ 0.047

Appendix

Table D.12. PlantedMem utility by regime for each

k

and design (% per month). A larger

k

gains only where experiences matter.

Trust rule	No influence	Noise only	$\beta=0.1$	$\beta=0.25$	$\beta=0.5$	Many exp.	Informative
MemTrial (trust test)	2.3 $\pm$ 6.6	3.0 $\pm$ 10.1	13.7 $\pm$ 20.7	64.5 $\pm$ 23.3	84.8 $\pm$ 8.9	26.1 $\pm$ 22.6	44.9 $\pm$ 22.5
Per-experience learner only	1.5 $\pm$ 5.1	2.0 $\pm$ 8.3	10.5 $\pm$ 18.0	61.4 $\pm$ 24.0	83.5 $\pm$ 10.1	22.0 $\pm$ 20.7	22.0 $\pm$ 20.7
In-sample F-test	2.4 $\pm$ 6.7	2.1 $\pm$ 7.1	8.1 $\pm$ 13.7	52.4 $\pm$ 22.9	81.7 $\pm$ 10.5	14.8 $\pm$ 17.7	15.1 $\pm$ 17.8

Appendix

Table D.13. Trust rate (%) by PlantedMem regime. MemTrial rarely trusts its values when experiences carry no signal and mostly trusts them when experiences matter.

Method	No influence	Noise only	$\beta=0.1$	$\beta=0.25$	$\beta=0.5$	Many exp.	Informative
Reference (memory-free average)	0.788 $\pm$ 0.032	0.788 $\pm$ 0.032	0.788 $\pm$ 0.032	0.788 $\pm$ 0.032	0.788 $\pm$ 0.032	0.788 $\pm$ 0.032	0.788 $\pm$ 0.032
Outcome credit (MemRL†)	0.759 $\pm$ 0.097	0.658 $\pm$ 0.089	0.758 $\pm$ 0.090	0.755 $\pm$ 0.098	0.749 $\pm$ 0.144	0.760 $\pm$ 0.085	0.760 $\pm$ 0.085
Counterfactual selection	0.768 $\pm$ 0.097	0.659 $\pm$ 0.087	0.805 $\pm$ 0.093	0.926 $\pm$ 0.100	1.101 $\pm$ 0.138	0.852 $\pm$ 0.078	0.852 $\pm$ 0.078
MemTrial w/o trust test	0.776 $\pm$ 0.047	0.758 $\pm$ 0.056	0.803 $\pm$ 0.059	0.924 $\pm$ 0.092	1.094 $\pm$ 0.131	0.843 $\pm$ 0.061	0.882 $\pm$ 0.068
MemTrial, never acting on its values	0.788 $\pm$ 0.028	0.787 $\pm$ 0.029	0.788 $\pm$ 0.028	0.788 $\pm$ 0.029	0.788 $\pm$ 0.029	0.789 $\pm$ 0.029	0.789 $\pm$ 0.029
MemTrial (ours)	0.787 $\pm$ 0.030	0.782 $\pm$ 0.035	0.793 $\pm$ 0.043	0.912 $\pm$ 0.088	1.090 $\pm$ 0.124	0.821 $\pm$ 0.050	0.862 $\pm$ 0.068

Appendix

Table D.14. Utility by PlantedMem regime (% per month). MemTrial stays at the reference when experiences carry no signal and gains when they matter.

Method	No influence	Noise only	$\beta=0.1$	$\beta=0.25$	$\beta=0.5$	Many exp.	Informative	Core avg.
1/ $N$	0.575 $\pm$ 0.000	0.575 $\pm$ 0.000	0.575 $\pm$ 0.000	0.575 $\pm$ 0.000	0.575 $\pm$ 0.000	0.575 $\pm$ 0.000	0.575 $\pm$ 0.000	0.57 $\pm$ 0.00
Draft averaging	0.782 $\pm$ 0.032	0.715 $\pm$ 0.052	0.777 $\pm$ 0.031	0.764 $\pm$ 0.044	0.719 $\pm$ 0.080	0.775 $\pm$ 0.041	0.775 $\pm$ 0.041	0.75 $\pm$ 0.03
MemTrial, always 1/ $N$ when not trusted ( $q=1$ )	0.578 $\pm$ 0.016	0.580 $\pm$ 0.021	0.596 $\pm$ 0.055	0.783 $\pm$ 0.124	1.030 $\pm$ 0.136	0.626 $\pm$ 0.073	0.700 $\pm$ 0.096	0.71 $\pm$ 0.05
MemTrial, never acting on its values	0.633 $\pm$ 0.009	0.608 $\pm$ 0.017	0.630 $\pm$ 0.009	0.624 $\pm$ 0.014	0.611 $\pm$ 0.026	0.626 $\pm$ 0.013	0.626 $\pm$ 0.013	0.62 $\pm$ 0.01
MemTrial, anchored ( $\alpha_{0}=0.9$ )	0.635 $\pm$ 0.015	0.612 $\pm$ 0.024	0.646 $\pm$ 0.047	0.805 $\pm$ 0.114	1.033 $\pm$ 0.133	0.669 $\pm$ 0.063	0.733 $\pm$ 0.085	0.75 $\pm$ 0.05
MemTrial, uniform prior ( $\alpha_{0}=0.5$ )	0.735 $\pm$ 0.022	0.675 $\pm$ 0.041	0.736 $\pm$ 0.039	0.852 $\pm$ 0.100	1.043 $\pm$ 0.130	0.751 $\pm$ 0.055	0.797 $\pm$ 0.073	0.81 $\pm$ 0.05

Appendix

Table D.15. PlantedMem with 1/

N

as the reference (% per month). The anchored blend learns to leave a weak reference.

Method	All 71 months	50 months before cutoff	21 months after cutoff
1/ $N$	0.623 $\pm$ 0.000	0.554 $\pm$ 0.000	0.787 $\pm$ 0.000
Minimum variance	0.230 $\pm$ 0.000	0.178 $\pm$ 0.000	0.353 $\pm$ 0.000
Memory-free	0.657 $\pm$ 0.001	0.636 $\pm$ 0.008	0.705 $\pm$ 0.017
Self-consistency	0.686 $\pm$ 0.007	0.672 $\pm$ 0.010	0.721 $\pm$ 0.013
Similarity retr. (top-2)	0.649 $\pm$ 0.033	0.616 $\pm$ 0.043	0.728 $\pm$ 0.008
Similarity retr. (top-4)	0.643 $\pm$ 0.039	0.621 $\pm$ 0.047	0.696 $\pm$ 0.027

Appendix

Table D.16. ClassAlloc before and after the knowledge cutoff of gpt-4.1-mini (% per month). Most LLM-based methods beat 1/

N

before the cutoff; after it, all trail it, MemTrial the least.

LLM, temperature	Memory-free	Best experience- learning agent	Best other LLM-based method	MemTrial (ours)	MemTrial w/o trust test	Improv. (%, $p$ )	Trust rate (%)
(a) InvestorBench (bp per day)
gpt-4.1-mini, 0	7.74 $\pm$ 0.35	ExpeL 9.46 $\pm$ 0.09	ExpeL 9.46 $\pm$ 0.09	9.84 $\pm$ 0.51	9.69 $\pm$ 0.22	$+4.0$ $\pm$ 6.0 (0.32)	23.2 $\pm$ 7.9
gpt-4.1-mini, 0.3	7.63 $\pm$ 0.25	ExpeL 9.40 $\pm$ 0.15	ExpeL 9.40 $\pm$ 0.15	10.54 $\pm$ 0.32	9.33 $\pm$ 0.28	$+12.1$ $\pm$ 4.0 (0.09)	7.0 $\pm$ 12.1
gpt-4.1-mini, 0.7 (main)	8.43 $\pm$ 0.29	ExpeL 8.93 $\pm$ 0.32	Hedge 8.97 $\pm$ 0.23	10.40 $\pm$ 0.04	9.91 $\pm$ 0.45	$+16.4$ $\pm$ 3.2 (0.12)	0.3 $\pm$ 0.5
gpt-4.1-mini, 1.0	7.83 $\pm$ 0.47	ExpeL 9.51 $\pm$ 0.45	ExpeL 9.51 $\pm$ 0.45	10.36 $\pm$ 0.01	10.10 $\pm$ 0.54	$+9.0$ $\pm$ 4.8 (0.15)	0.0 $\pm$ 0.0
gpt-4.1-nano, 0.7	8.45 $\pm$ 0.74	MemRL 7.75 $\pm$ 0.55	Self-consistency 9.07 $\pm$ 0.39	10.30 $\pm$ 0.06 *	9.22 $\pm$ 1.25	$+33.0$ $\pm$ 7.4 (0.02)	2.1 $\pm$ 3.6

Appendix

Table D.17. Robustness to the LLM and its temperature, all configurations. MemTrial beats every LLM-based baseline in all eleven InvestorBench configurations and stays within 0.021 pp of 1/

N

with all eight LLMs on ClassAlloc.

	gpt-4.1-mini, temperature
Method	0	0.3	0.7 (main)	1.0
1/ $N$	10.45 $\pm$ 0.00	10.45 $\pm$ 0.00	10.45 $\pm$ 0.00	10.45 $\pm$ 0.00
Minimum variance	5.53 $\pm$ 0.00	5.53 $\pm$ 0.00	5.53 $\pm$ 0.00	5.53 $\pm$ 0.00
Memory-free	7.74 $\pm$ 0.35	7.63 $\pm$ 0.25	8.43 $\pm$ 0.29	7.83 $\pm$ 0.47
Self-consistency	7.86 $\pm$ 0.17	7.81 $\pm$ 0.08	7.89 $\pm$ 0.21	7.83 $\pm$ 0.24
Similarity retrieval (top-2)	8.36 $\pm$ 0.14	8.73 $\pm$ 0.28	8.67 $\pm$ 0.19	8.60 $\pm$ 0.45

Appendix

Table D.18. InvestorBench with gpt-4.1-mini at four temperatures, all methods (bp per day). At every temperature, no LLM-based baseline beats 1/

N

and MemTrial is the best LLM-based method.

Method	gpt-4.1-mini	gpt-4.1-nano	gpt-5-mini	Llama-3.3-70B	Qwen3-235B	DeepSeek-V3.1	Gemini 2.5 Flash	Claude Haiku 4.5
1/ $N$	10.45 $\pm$ 0.00	10.45 $\pm$ 0.00	10.45 $\pm$ 0.00	10.45 $\pm$ 0.00	10.45 $\pm$ 0.00	10.45 $\pm$ 0.00	10.45 $\pm$ 0.00	10.45 $\pm$ 0.00
Minimum variance	5.53 $\pm$ 0.00	5.53 $\pm$ 0.00	5.53 $\pm$ 0.00	5.53 $\pm$ 0.00	5.53 $\pm$ 0.00	5.53 $\pm$ 0.00	5.53 $\pm$ 0.00	5.53 $\pm$ 0.00
Memory-free	8.43 $\pm$ 0.29	8.45 $\pm$ 0.74	8.36 $\pm$ 0.54	8.34 $\pm$ 0.39	8.31 $\pm$ 0.21	8.37 $\pm$ 0.55	7.66 $\pm$ 0.23	8.42 $\pm$ 0.10
Self-consistency	7.89 $\pm$ 0.21	9.07 $\pm$ 0.39	8.31 $\pm$ 0.31	8.26 $\pm$ 0.22	8.37 $\pm$ 0.17	8.63 $\pm$ 0.51	7.92 $\pm$ 0.23	8.48 $\pm$ 0.07
Similarity retrieval (top-2)	8.67 $\pm$ 0.19	6.89 $\pm$ 0.92	8.41 $\pm$ 0.17	8.90 $\pm$ 0.28	8.15 $\pm$ 0.44	8.66 $\pm$ 0.47	8.50 $\pm$ 0.41	8.98 $\pm$ 0.12
Similarity retrieval (top-4)	8.12 $\pm$ 0.61	7.71 $\pm$ 0.66	8.46 $\pm$ 0.24	8.51 $\pm$ 0.14	8.00 $\pm$ 0.32	8.55 $\pm$ 0.45	8.22 $\pm$ 0.08	8.83 $\pm$ 0.15

Appendix

Table D.19. InvestorBench with eight LLMs, all methods (bp per day). With every LLM, no LLM-based baseline beats 1/

N

and MemTrial beats every LLM-based baseline.

Method	gpt-4.1-mini	gpt-4.1-nano	gpt-5-mini	Llama-3.3-70B	Qwen3-235B	DeepSeek-V3.1	Gemini 2.5 Flash	Claude Haiku 4.5
1/ $N$	0.623 $\pm$ 0.000	0.623 $\pm$ 0.000	0.623 $\pm$ 0.000	0.623 $\pm$ 0.000	0.623 $\pm$ 0.000	0.623 $\pm$ 0.000	0.623 $\pm$ 0.000	0.623 $\pm$ 0.000
Minimum variance	0.230 $\pm$ 0.000	0.230 $\pm$ 0.000	0.230 $\pm$ 0.000	0.230 $\pm$ 0.000	0.230 $\pm$ 0.000	0.230 $\pm$ 0.000	0.230 $\pm$ 0.000	0.230 $\pm$ 0.000
Memory-free	0.657 $\pm$ 0.001	0.587 $\pm$ 0.007	0.713 $\pm$ 0.013	0.619 $\pm$ 0.059	0.688 $\pm$ 0.022	0.597 $\pm$ 0.018	0.616 $\pm$ 0.012	0.668 $\pm$ 0.020
Self-consistency	0.686 $\pm$ 0.007	0.588 $\pm$ 0.014	0.719 $\pm$ 0.012	0.614 $\pm$ 0.017	0.698 $\pm$ 0.021	0.615 $\pm$ 0.024	0.618 $\pm$ 0.003	0.656 $\pm$ 0.004
Similarity retrieval (top-2)	0.649 $\pm$ 0.033	0.557 $\pm$ 0.000	0.701 $\pm$ 0.019	0.602 $\pm$ 0.009	0.599 $\pm$ 0.006	0.603 $\pm$ 0.009	0.656 $\pm$ 0.026	0.640 $\pm$ 0.006
Similarity retrieval (top-4)	0.643 $\pm$ 0.039	0.554 $\pm$ 0.019	0.655 $\pm$ 0.006	0.598 $\pm$ 0.005	0.631 $\pm$ 0.019	0.625 $\pm$ 0.016	0.586 $\pm$ 0.016	0.641 $\pm$ 0.009

Appendix

Table D.20. ClassAlloc with eight LLMs, all methods (% per month). MemTrial stays within 0.021 pp of 1/

N

with every LLM; the memory-free LLM beats 1/

N

with four of them.

			Memory-free		MemTrial (ours)
LLM	Cutoff	Months	before	after	before	after
gpt-4.1-mini	Jun 2024	50/21	$+0.082$	$-0.082$	$+0.003$	$-0.041$
gpt-4.1-nano	Jun 2024	50/21	$-0.018$	$-0.078$	$-0.018$	$-0.007$
gpt-5-mini	May 2024	49/22	$+0.157$	$-0.059$	$+0.040$	$-0.024$
Llama-3.3-70B	Dec 2023	44/27	$+0.047$	$-0.087$	$+0.005$	$-0.009$
Qwen3-235B	–

Appendix

Table D.21. ClassAlloc before and after each LLM’s knowledge cutoff (utility minus that of 1/

N

, pp per month). With five of the six LLMs whose cutoff is published, the memory-free draft trails 1/

N

after the cutoff.

Figure D.5. ClassAlloc with other LLMs: wealth relative to 1/

N

. MemTrial stays within 3% of 1/

N

with every LLM, whereas the memory-free LLM and the other agents move up to 14% away. Eight line charts, one per LLM on ClassAlloc, showing wealth relative to 1/N for the balanced investor, with a dotted line at the LLM's knowledge cutoff. MemTrial stays within 3 percent of 1/N; the other agents move up to 14 percent away.

		Absolute recall		Relative recall
LLM	Cutoff	before	after	before	after
gpt-4.1-mini	Jun 2024	$+0.21$ $\pm$ 0.02	$-0.05$ $\pm$ 0.12	$+0.19$ $\pm$ 0.01	$-0.02$ $\pm$ 0.03
gpt-4.1-nano	Jun 2024	$+0.10$ $\pm$ 0.04	$0.00$ $\pm$ 0.04	$+0.06$ $\pm$ 0.06	$-0.10$ $\pm$ 0.09
gpt-5-mini	May 2024	$+0.22$ $\pm$ 0.06	$-0.04$ $\pm$ 0.09	$+0.20$ $\pm$ 0.04	$-0.10$ $\pm$ 0.13
Llama-3.3-70B	Dec 2023	$+0.40$ $\pm$ 0.09	$+0.09$ $\pm$ 0.15	$+0.36$ $\pm$ 0.08	$+0.01$ $\pm$ 0.12
Qwen3-235B	–	$+0.27$ $\pm$ 0.03	–	$+0.21$ $\pm$ 0.05	–

Appendix

Table D.22. Memorization probe: correlation of the asset-class returns that each LLM recalls from a date alone with the realized ClassAlloc returns. Every LLM recalls the outcomes before its knowledge cutoff, and hardly any after it.

	gpt-4.1-mini				gpt-5-mini
	dates shown		dates hidden		dates shown		dates hidden
Method	before	after	before	after	before	after	before	after
Memory-free	$+0.082$ $\pm$ 0.008	$-0.082$ $\pm$ 0.017	$+0.127$ $\pm$ 0.017	$-0.099$ $\pm$ 0.027	$+0.157$ $\pm$ 0.020	$-0.059$ $\pm$ 0.025	$+0.136$ $\pm$ 0.038	$-0.066$ $\pm$ 0.006
Self-consistency	$+0.118$ $\pm$ 0.010	$-0.066$ $\pm$ 0.013	$+0.125$ $\pm$ 0.030	$-0.104$ $\pm$ 0.012	$+0.160$ $\pm$ 0.013	$-0.048$ $\pm$ 0.016	$+0.118$ $\pm$ 0.028	$-0.068$ $\pm$ 0.015
FinMem †	$-0.016$ $\pm$ 0.021	$-0.062$ $\pm$ 0.026	$-0.047$ $\pm$ 0.029	$-0.076$ $\pm$ 0.003	$+0.019$ $\pm$ 0.025	$-0.013$ $\pm$ 0.045	$-0.024$ $\pm$ 0.016	$-0.088$ $\pm$ 0.035
MemRL †	$+0.058$ $\pm$ 0.030	$-0.104$ $\pm$ 0.033	$+0.088$ $\pm$ 0.032	$-0.093$ $\pm$ 0.050	$+0.077$ $\pm$ 0.046	$-0.048$ $\pm$ 0.034	$+0.077$ $\pm$ 0.049	$-0.012$ $\pm$ 0.041

Appendix

Table D.23. ClassAlloc with every calendar date hidden from the LLM (utility minus that of 1/

N

, pp per month). Hiding the dates does not remove the memory-free draft’s advantage before the knowledge cutoff.

Figure D.6. ClassAlloc at each LLM’s knowledge cutoff. The LLMs recall dated outcomes from before their cutoffs, but hiding the dates does not remove the memory-free draft’s advantage before the cutoff. Two bar charts for six LLMs. Top: correlation between the asset-class returns an LLM recalls from a date alone and the realized returns, before and after its knowledge cutoff: 0.10 to 0.63 before and minus 0.16 to 0.09 after. Bottom: utility of the memory-free draft minus that of 1/N in pp per month before and after each LLM's cutoff, positive before and negative after for five of the six LLMs; hatched bars show gpt-4.1-mini and gpt-5-mini with all dates hidden, which keep the same pattern.

	New months (2025–2026)		All months after the cutoff
Method	Full	Raw	Full	Raw
Rule-based portfolios
1/ $N$	1.025 $\pm$ 0.000	1.025 $\pm$ 0.000	2.075 $\pm$ 0.000	2.075 $\pm$ 0.000
Minimum variance	0.029 $\pm$ 0.000	0.029 $\pm$ 0.000	0.078 $\pm$ 0.000	0.078 $\pm$ 0.000
LLM agents without experience learning
Memory-free	1.825 $\pm$ 0.228	1.055 $\pm$ 0.103	1.290 $\pm$ 0.277	1.363 $\pm$ 0.053

Appendix

Table D.24. PortBench after the knowledge cutoff (% per month). Over the eleven months after the cutoff, MemTrial is the best LLM-based method and stays close to 1/

N

; in the six new months, the memory-free draft beats 1/

N

on PortBench-Full.

1 April 2022							1 September 2022
Lessons in the draft	Eq.	Bd.	Cm.	RE	Cash	Utility	Lessons in the draft	Eq.	Bd.	Cm.	RE	Cash	Utility
none	35	10	25	20	10	$-$ 4.63	none	15	35	15	15	20	$-$ 5.43
L18/04, L16/08	35	10	20	25	10	$-$ 5.14	L18/04, L16/08	15	30	10	15	30	$-$ 5.05
L18/04, L17/05	35	10	15	30	10	$-$ 5.67	L18/04, L19/10	10	30	10	10	40	$-$ 3.89
L16/08, L17/05	35	15	10	30	10	$-$ 6.02	L16/08, L19/10	15	25	10	15	35	$-$ 4.81
L18/04, L15/12	35	15	15	25	10	$-$ 5.48	L18/04, L17/05	15	35	10	15	25	$-$ 5.29

Appendix

Table E.1. Case study on ClassAlloc (balanced investor). The contributions separate the lessons although every draft lost money; in seed 1, acting on the values cost 1.66 pp in April, and keeping the anchored action gave up 2.02 pp in September.

Method	Memory block
Memory-free, Self-consistency	none
MemTrial, Similarity retrieval, Counterfactual selection, Uplift credit, Draft averaging	experiences block with a subset of the four retrieved lessons
FinMem	experiences block with four lessons ranked by similarity, recency and importance
MemRL	experiences block with four of the ten most similar lessons, ranked by learned value ( $\varepsilon$ -greedy)
Reflexion	reflections block with its three most recent reflections
ExpeL	insights block with at most eight insights distilled from all warm-up lessons

Appendix

Table G.1. Memory block of each LLM-based method in the decision prompt on InvestorBench and ClassAlloc.

Explore similar work

From Knowing to Doing: A Memory-Controlled Benchmark for LLM Trading Agents on Stock Markets
May 27, 2026Taojie Zhu, Wentao Zhao, Rui Sun +7Benchmark Contamination LLM Agent Evaluation
1Tsinghua University, 2Stepfun · 2Stepfun · 4Shanghai Jiao Tong University +3
FinPerMA: A Theory-Informed, Event-Grounded Personalized-Memory Benchmark for LLM Agents
Aug 4, 2026Ben Wang, Kang Zhou, Lifan Guo +2Financial Services LLM Agent Memory
Qwen DianJin Team, Alibaba Cloud Computing
OpenPM: Auditable Point-in-Time Evaluation for LLM Portfolio-Management Agents
Aug 6, 2026Xinying Cai, Minghao Guo, Jiahe Liu +7LLM Auditing LLM Agent Evaluation
Rutgers University · Technical University of Denmark · New Jersey Institute of Technology +4

May 27, 2026, cs.AIJ/K move · Enter open · S save

From Knowing to Doing: A Memory-Controlled Benchmark for LLM Trading Agents on Stock Markets

Taojie Zhu, Wentao Zhao, Rui Sun, Beidi Luan+6

1Tsinghua University, 2Stepfun · 2Stepfun · 4Shanghai Jiao Tong University+3

Benchmark Contamination LLM Agent Evaluation AI Agent Benchmarks Finance Algorithmic Trading

Open paper AskPDF arXiv

Evaluating whether large language model (LLM) agents can profit in capital markets is increasingly framed as end-to-end trading: place an agent in a historical market, let it trade, and measure portfolio returns. This setup is vulnerable to two evaluation failures. First, long backtests often overlap with the knowledge cutoffs of frontier LLMs, allowing memorized tickers, dates, prices, and market narratives to substitute for investment reasoning. Second, raw returns are a noisy proxy for stock-selection ability, since positive performance may come from market beta, style exposure, or favorable regimes rather than genuine alpha. We introduce KTD-Fin (Knowing-To-Doing Financial Benchmark), an end-to-end stock-market trading benchmark that addresses both issues. KTD-Fin uses a data-side masking protocol to anonymize key identifiers and calendar information consistently across prompts and tools, separating historical market memory from investment decision-making. It also incorporates a Barra-style performance attribution framework that decomposes portfolio returns into market, style, and stock-selection alpha components. Across ten frontier LLM agents evaluated on the Chinese CSI300 over a 2024--2026 window, masking substantially changes agent rationales, pushing them towards anonymized factor-based reasoning. Attribution analysis further shows that LLM agents' cumulative returns under leakage-controlled evaluation are largely explained by passive market and style exposure, with limited evidence of persistent stock-selection alpha. These findings suggest that financial LLM benchmarks should evaluate not only whether an agent makes money, but also whether the source of returns reflects transferable investment skill. We release KTD-Fin as a reproducible template for leakage-controlled and attribution-aware evaluation of LLM trading agents.

Abstract

Explore similar work

From Knowing to Doing: A Memory-Controlled Benchmark for LLM Trading Agents on Stock Markets

FinPerMA: A Theory-Informed, Event-Grounded Personalized-Memory Benchmark for LLM Agents

OpenPM: Auditable Point-in-Time Evaluation for LLM Portfolio-Management Agents