When Valid Signals Fail: Regime Boundaries Between LLM Features and RL Trading Policies

Zhengzhe Yang

cs.CLApr 13, 2026

When Valid Signals Fail: Regime Boundaries Between LLM Features and RL Trading Policies

Authors: Zhengzhe Yang

Organizations: Independent Researcher

Abstract

Can large language models (LLMs) generate continuous numerical features that improve reinforcement learning (RL) trading agents? We build a modular pipeline where a frozen LLM serves as a stateless feature extractor, transforming unstructured daily news and filings into a fixed-dimensional vector consumed by a downstream PPO agent. We introduce an automated prompt-optimization loop that treats the extraction prompt as a discrete hyperparameter and tunes it directly against the Information Coefficient - the Spearman rank correlation between predicted and realized returns - rather than NLP losses. The optimized prompt discovers genuinely predictive features (IC above 0.15 on held-out data). However, these valid intermediate representations do not automatically translate into downstream task performance: during a distribution shift caused by a macroeconomic shock, LLM-derived features add noise, and the augmented agent under-performs a price-only baseline. In a calmer test regime the agent recovers, yet macroeconomic state variables remain the most robust driver of policy improvement. Our findings highlight a gap between feature-level validity and policy-level robustness that parallels known challenges in transfer learning under distribution shift.

Figures & tables

Figure 1: Cumulative portfolio value across 2025. Solid lines: 5-seed mean; shaded:

\pm 1

std. H1 (red) captures a tariff-driven shock; H2 (yellow) is calmer. The regime split isolates when LLM features succeed and fail.

Panel of Figure 2 — Figure 2: System overview. A Go-based ingestion pipeline collects news, filings, and macro data into a relational store. The LLM produces per-ticker feature vectors, which the PPO agent consumes alongside OHLCV and technical indicators.

Feature	Group	Mean	Std	Min	Max	% Non-zero	Scale
sentiment	LLM News	-0.007	0.290	-1.000	0.900	71.8%	[-1, 1]
impact	LLM News	0.292	0.233	0.000	1.000	71.9%	[0, 1]
conflicting_signals	LLM News	0.076	0.182	0.000	1.000	18.5%	[0, 1]
news_novelty	LLM News	0.737	0.424	0.000	1.000	77.9%	[0, 1]
market_sentiment	Market Regime	0.175	0.405	-1.000	0.800	98.7%	[-1, 1]
macro_event_flag	Market Regime	0.481	0.500	0.000	1.000	48.1%	{0, 1}

Table 1: Feature Value Distributions (2023-2024 training period)

Figure 3: Training convergence (seed 42). Validation Sharpe at each 100k-step checkpoint on H1 2025. Both agents plateau by 400–500k steps; the vertical line marks the multi-seed cutoff.

Panel of Figure 4 — Figure 4: Prompt optimization workflow. Claude iteratively refines the extraction prompt based on downstream financial metric feedback.

Table 2: Prompt mutation results. Top: January 2025 optimization set (769 bundles, 38 tickers). Bottom: February 2025 out-of-sample validation.

Metric	Gate Threshold	Value	Status
signal_coverage	$\geq 0.25$	0.408	PASS
ic_ir_5d	$\geq 0.05$	+0.104	PASS
quintile_spread	$>0$	+0.002	PASS
hit_rate	$\geq 0.52$	0.714	PASS

Table 3: Adequacy gate assessment ( mut2-few-shot , Jan 2025).

Panel of Figure 5 — Figure 5: Abbreviated visualization of v1-stable-core . Few-shot calibration examples anchoring numerical output ranges proved necessary to maximize IC.

Feature	Group	IC Mean	IC IR	$t$ -stat	% Pos	$N$
sentiment	LLM News	0.016	0.093	1.00	0.5	117
impact	LLM News	0.029	0.177	1.91	0.6	117
conflicting_signals	LLM News	0.040	0.233	2.52	0.6	117
news_novelty	LLM News	0.011	0.065	0.70	0.5	117

Table 4: Feature IC analysis (5-day forward return, 2023–2024 training period). Cross-sectional IC for macro features is zero by construction: they are constant across tickers on any given day and carry regime information detectable only by the RL policy, not by cross-sectional ranking.

Config	N Feats	Sharpe	Return%	Max DD%	$\Delta$ Sharpe
Baseline (OHLCV + Tech)	0	0.010 $\pm$ 0.618	-5.03 $\pm$ 18.00	34.91 $\pm$ 6.01	—
LLM Signals Only †	6	-0.411 $\pm$ 0.690	-16.42 $\pm$ 18.81	42.12 $\pm$ 8.11	-0.421
Macro Only ‡	5	-0.007 $\pm$ 0.355	-6.16 $\pm$ 9.74	34.78 $\pm$ 9.28	-0.017
LLM + Macro (Full)	10	-0.267 $\pm$ 0.284	-13.91 $\pm$ 7.41	38.66 $\pm$ 6.34	-0.276

Table 5: H1 2025 validation ablation (

N\!=\!5

seeds, 120 trading days). Mean

\pm

std across seeds.

\Delta

Sharpe vs. Baseline.

Paired $t$ -test (LLM+Macro vs. Baseline): $t(4)=0.76$ , $p=0.4873$
Config	N Feats	Sharpe	Return%	Max DD%	$\Delta$ Sharpe
Baseline (OHLCV + Tech)	0	0.809 $\pm$ 0.333	11.29 $\pm$ 6.21	14.96 $\pm$ 4.04	—
LLM Signals Only †	5	1.001 $\pm$ 0.853	12.98 $\pm$ 8.93	16.27 $\pm$ 6.59	+0.192
Macro Only ‡	5	1.099 $\pm$ 0.695	16.04 $\pm$ 9.53	15.30 $\pm$ 3.15	+0.290
LLM + Macro (Full)	10	1.038 $\pm$ 0.424	15.14 $\pm$ 6.19	17.27 $\pm$ 2.18	+0.229
SPY buy-and-hold §	—	1.756 §	11.87 §	5.07 §	+0.947

Table 6: H2 2025 held-out test (

N\!=\!5

seeds). Mean

\pm

std.

\Delta

Sharpe vs. Baseline.

Figure 8: Regime-conditional performance. During elevated VIX (shaded), the LLM-augmented agent under-performs the baseline. In low-volatility windows the gap narrows or reverses.

Cost (bp)	Cost (%)	Baseline Sharpe	LLM+Macro Sharpe	$\Delta$ Sharpe	LLM Win?
0	0.00%	1.231	0.753	-0.478	×
5	0.05%	1.202	0.730	-0.472	×
10	0.10%	1.156	0.674	-0.482	×
20	0.20%	1.108	0.549	-0.559	×
50	0.50%	0.929	0.580	-0.349	×

Table 7: Transaction cost sensitivity (2025 H1, seed 42). The baseline advantage over LLM+Macro is robust across all cost levels, confirming the H1 null result is not a friction artifact.

LLM-Guided RL

Large language model-guided reinforcement learning uses language models to shape rewards, exploration, planning, and policy learning.

See all 140 papers

RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training
Jun 2, 2026Rachit Bansal, Clara Mohri, Tian Qin +2Language Model Pretraining Supervised Fine-Tuning
Harvard University
When are LLMs Sufficient Policy Optimizers for Sequential RL Tasks?
May 29, 2026Stephane Hatgis-Kessell, Emma BrunskillLanguage Model-Based Control Policy Optimization
Department of Computer Science, Stanford University
Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models
Apr 27, 2026Dan Shi, Zhuowen Han, Simon Ostermann +3Reinforcement Learning RL for Language Model Reasoning
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China · German Research Center for Artificial Intelligence (DFKI), Saarbrücken, Germany · Saarland University, Saarbrücken, Germany

Jun 2, 2026J/K move · Enter open · S save

RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training

Rachit Bansal, Clara Mohri, Tian Qin, David Alvarez-Melis+1

Harvard University

Language Model Pretraining Supervised Fine-Tuning RL for Language Models Reinforcement Learning RL Post-Training

Open paper AskPDF arXiv

The standard LLM training pipeline applies reinforcement learning (RL) only after pre-training and supervised fine-tuning (SFT). We question this status quo by training a LLM from scratch and applying RL, SFT, and SFT followed by RL directly to intermediate pre-training checkpoints. We find that RL is effective very early, and often matches the full SFT

\to

RL pipeline early as well. Through experiments on harder problems, we find that targeted pre-training data composition is a strong lever for RL effectiveness, even more so than model scale. Beyond reasoning accuracy, applying RL directly to base checkpoints expands the model's distribution; the sharpening effect reported in recent work arises only when RL follows SFT. The general capabilities of the model remain essentially unchanged by RL, while they degrade following SFT. Finally, we merge RL and SFT objectives by parallel averaging, which outperforms across all other training methods discussed, across metrics, while preserving general capabilities. Together, these results suggest that LLM training might benefit from an expanded use of RL.

Abstract

LLM-Guided RL

Explore similar work

RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training

When are LLMs Sufficient Policy Optimizers for Sequential RL Tasks?

Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models