AlphaPareto: Formulaic Alpha Discovery with LLM-Guided Multi-Objective Reinforcement Learning

Yingbo Zhao; Zeyu Yang; Zhoufan Zhu

stat.MLSep 28, 2026

AlphaPareto: Formulaic Alpha Discovery with LLM-Guided Multi-Objective Reinforcement Learning

Authors: Yingbo Zhao, Zeyu Yang, Zhoufan Zhu

Organizations: School of Economics Xiamen University Fujian, China · Paula and Gregory Chow Institute for Studies in Economics Xiamen University Fujian, China · School of Economics & Wang Yanan Institute for Studies in Economics Xiamen University Fujian, China

Abstract

Formulaic alpha discovery is a core challenge in quantitative trading, as identifying alphas that work well together remains difficult. Recent reinforcement learning (RL) methods formulate this task as a Markov decision process (MDP), but two important issues remain unresolved. First, as the alpha pool evolves, the reward function changes accordingly, making the MDP inherently non-stationary. Second, most existing methods optimize a single objective, typically predictive power, while ignoring other important properties of a high-quality alpha pool. Motivated by these challenges, we propose AlphaPareto, an RL method for formulaic alpha discovery. To address non-stationarity, AlphaPareto augments the state to include both the alpha under construction and the current alpha pool, and applies a large language model (LLM) to encode the pool. This design allows the agent to adapt to the evolving search environment. To overcome the limitation of single-objective reward design, AlphaPareto replaces the scalar reward with a multi-objective vector-valued reward that simultaneously captures predictive power, temporal stability, perturbation robustness, and diversity, and optimizes these objectives through a Pareto-regularized learning procedure. Empirical applications to real-world datasets show that our AlphaPareto method outperforms its competitors.

Figures & tables

Figure 1 : Overview of the AlphaPareto framework.

Method	Mean	Std	Mean	Std	Mean	Std
	CSI300		CSI800		Market
Alpha158	2.99%	-	4.77%	-	4.04%	-
MLP	1.51%	(2.33%)	4.63%	(0.90%)	7.00%	(0.18%)
GP	1.09%	(0.82%)	2.74%	(1.40%)	7.52%	(1.53%)
AlphaAgent	0.51%	(0.34%)	0.48%	(0.46%)	1.10%	(0.59%)
R&D-Agent-Quant	2.39%	(0.96%)	2.45%	(0.73%)	1.48%	(0.91%)

Table 1 : Out-of-sample ICs of different methods across different datasets.

Figure 2 : Radar chart comparison in terms of temporal stability, perturbation robustness, and diversity.

Model	AV ( $\uparrow$ )	MDD ( $\uparrow$ )	IR ( $\uparrow$ )	SR ( $\uparrow$ )
Alpha158	33.17%	-18.71%	1.08	1.34
MLP	47.09%	-25.21%	1.57	1.58
GP	24.49%	-13.29%	0.77	1.18
AlphaGen	40.03%	-15.91%	1.65	1.61
AlphaAgent	35.07%	-29.94%	0.71	0.89
R&D-Agent-Quant	37.30%	-30.18%	0.76	0.96

Table 2 : Out-of-sample monthly portfolio performance. AV denotes the annualized return, MDD denotes the maximum drawdown, IR denotes the information ratio, and SR denotes the Sharpe ratio.

Model	Mean	Std	Mean	Std	Mean	Std
	CSI300		CSI800		Market
DeepSeek-1.5B	2.65%	(0.99%)	5.39%	(0.44%)	9.33%	(0.92%)
Qwen-Embedding-4B	3.92%	(0.46%)	5.70%	(0.87%)	10.10%	(1.01%)
DeepSeek-7B	3.38%	(0.56%)	5.50%	(1.23%)	9.50%	(0.56%)
DeepSeek-14B	3.71%	(1.73%)	5.59%	(0.46%)	9.74%	(0.64%)
DeepSeek-32B	3.26%	(1.43%)	4.76%	(0.52%)	9.88%	(1.13%)

Table 3 : Out-of-sample ICs of AlphaPareto across different LLM sizes.

Included Components		CSI300		CSI800		Market
LLM	MORL	Mean	Std	Mean	Std	Mean	Std
		3.69%	(0.66%)	5.07%	(0.63%)	8.44%	(1.09%)
$\checkmark$		2.37%	(0.68%)	5.04%	(0.42%)	9.68%	(0.83%)
	$\checkmark$	3.99%	(1.27%)	5.54%	(1.03%)	9.63%	(0.80%)
$\checkmark$	$\checkmark$	3.92%	(0.46%)	5.70%	(0.87%)	10.10%	(1.01%)

Table 4 : Ablation study of AlphaPareto, where “LLM” denotes the LLM-based alpha pool semantic reasoning, and “MORL” denotes the MAP-based multi-objective reinforcement learning.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3 : (a) A formulaic alpha discovered by our AlphaPareto method. (b) Its expression tree. (c) Its RPN representation.

Features
Tokens	Description
Open/High/Low/Close/Vwap/Volume	Opening/high/low/closing/vwap price or volume of stock $i$ at time $s$ .
Constant	Number from $\{-30,-10,-5,-2,-1,-0.5,-0.01,0.01,0.5,1,2,5,10,30\}$ .
Time delta	Integer from $\{10,20,30,40,50\}$ , which is used in the time-series operators.
Time-series Operators
$\textit{Ref}\,(u_{i,s},d)$	Return the value of $u_{i,s-d}$ , where $d$ is the time delta and $u_{i,s}$ is a feature of

Appendix

Table 5 : Description of available features and operators.

Chosen model	Embedding dimension $d_{e}$
DeepSeek-1.5B	1536
Qwen-Embedding-4B	2560
DeepSeek-7B	3584
DeepSeek-14B	5120
DeepSeek-32B	5120

Appendix

Table 6 : Embedding dimension of LLM candidates.

Hyperparameter	Values
Instruments	CSI300 / CSI800 / Market
$P$ (Alpha Pool Size)	10, 20
Leading days	20
Batch size	128
Optimizer	Adam
Learning rate	$5\times 10^{-4}$

Appendix

Table 7 : Additional hyperparameters.

Method	$t$	$p$ -value	$t$	$p$ -value	$t$	$p$ -value
	CSI300		CSI800		Market
Alpha158	4.55	(0.52%)	2.40	(3.73%)	13.39	(0.01%)
MLP	2.73	(2.62%)	7.05	(0.11%)	8.00	(0.07%)
GP	16.11	(0.00%)	9.58	(0.03%)	10.87	(0.02%)
AlphaAgent	55.88	(0.00%)	25.41	(0.00%)	44.84	(0.00%)
R&D-Agent-Quant	6.59	(0.14%)	34.67	(0.00%)	116.93	(0.00%)

Appendix

Table 8 : Paired

t

-tests of AlphaPareto against each baseline across five matched seeds.

p

-values are shown in parentheses as percentages, rounded to two decimal places.

Figure 4 : Cumulative returns of monthly portfolios constructed by different methods.

	CSI800
Pool representation	Mean	Std
None (MORL only)	5.54%	(1.03%)
Matched noise	4.22%	(1.62%)
LLM embedding (AlphaPareto)	5.70%	(0.87%)

Appendix

Table 9 : Out-of-sample ICs on CSI800 with different pool representations.

	S&P 500
Method	Mean	Std
Alpha158	3.92%	-
MLP	5.43%	(0.31%)
AlphaAgent	-0.01%	(0.56%)
R&D-Agent-Quant	1.29%	(0.49%)
AlphaGen	7.03%	(0.30%)

Appendix

Table 10 : Out-of-sample ICs of different methods on the S&P 500 universe.

Experiment	Runs	Hardware per run	Time per run	Total compute
Main comparison	195	GPU	8h	1560h
LLM scaling study	150	GPU	8h	1200h
Ablation study	360	GPU	7h	2520h

Appendix

Table 11 : Computational resources used for the reported experiments.

Explore similar work

From Feedback Loops to Policy Updates: Reinforcement Fine-Tuning for LLM-Based Alpha Factor Discovery
May 14, 2026Lingzhe Zhang, Tong Jia, Yunpeng Zhai +5Exploratory Factor Analysis Reinforcement Fine-Tuning
Peking University, Beijing, China · Alibaba Group, China · Nanjing University, Nanjing, China +1
AlphaRJM: Reward-Jump Memory for Stochastic Return-Guided Alpha Discovery
Sep 8, 2026Sayan Dhan, Selvaraju NatarajanSoft Actor-Critic Stochastic
Department of Mathematics Indian Institute of Technology Guwahati
Towards Autonomous Formulaic Alpha Discovery: An Evolutionary Computation Perspective
Aug 3, 2026Xinwei Yu, Yiyang Fu, Mingcheng Fan +3Evolutionary Optimization Model Discovery
Xi’an Jiaotong-Liverpool University, Suzhou 215123, China · Shanghai University, Shanghai 200444, China

May 14, 2026, cs.CEJ/K move · Enter open · S save

From Feedback Loops to Policy Updates: Reinforcement Fine-Tuning for LLM-Based Alpha Factor Discovery

Lingzhe Zhang, Tong Jia, Yunpeng Zhai, Zixuan Xie+4

Peking University, Beijing, China · Alibaba Group, China · Nanjing University, Nanjing, China+1

Exploratory Factor Analysis Reinforcement Fine-Tuning Feedback Loop Complementarity

Open paper AskPDF arXiv

Modern quantitative trading increasingly relies on systematic models to extract predictive signals from large-scale financial data, where alpha factor discovery plays a central role in transforming market observations into tradable signals. Recent LLM-based methods have shown promise in automating factor generation, but most of them still rely on prompt-level generation--evaluation--feedback loops for iterative optimization. As the loop becomes longer, repeatedly appended historical candidates and feedback can cause context explosion, increase inference cost, dilute useful information, and introduce feedback drift. Moreover, these methods often depend on very large LLMs whose stable generation preferences may lead to structurally similar expressions, redundant candidates, and search stagnation. To address these limitations, we propose \textsc{QuantEvolver}, a self-evolving alpha factor discovery framework based on reinforcement fine-tuning. Instead of accumulating feedback in the prompt, \textsc{QuantEvolver} converts executable quantitative evaluation into policy updates, enabling a Miner LLM to internalize historical optimization experience through parameter learning. Specifically, \textsc{QuantEvolver} constructs high-quality seed factors, builds diverse seed--time-window training tasks, generates executable Factor DSL expressions, evaluates them through Regime Backtest, and optimizes the Miner LLM with Diversity-Complementarity Reward. During training, high-quality factors are continuously accumulated in a Mined Factor Database, which serves as the final discovered factor library. Extensive experiments on three realistic market benchmarks demonstrate the effectiveness of \textsc{QuantEvolver}, which consistently improves the primary evaluation metric of each task over existing LLM-based alpha factor discovery baselines, produces higher-quality and more complementary factor pools.

Abstract

Explore similar work

From Feedback Loops to Policy Updates: Reinforcement Fine-Tuning for LLM-Based Alpha Factor Discovery

AlphaRJM: Reward-Jump Memory for Stochastic Return-Guided Alpha Discovery

Towards Autonomous Formulaic Alpha Discovery: An Evolutionary Computation Perspective