Financial market states lack a unique ground truth and must therefore be constructed from price data. Conventional labeling methods typically define states independently of the downstream classifier, leaving unresolved whether the resulting labels can be recovered from information available at prediction time. We propose DDSL, an iterative self-labeling framework that couples a price- and grammar-guided Viterbi labeler with a Transformer classifier through model feedback. DDSL constructs a nine-state representation that jointly encodes current and future market trends while retaining structured price emissions and temporal constraints throughout refinement. Experiments on SSE50, CSI300, CSI500, and NASDAQ100 show that DDSL increases held-out joint-state macro F1 by 78%-96% over single-pass dynamic-programming labeling. An independent GRU classifier that never participates in the labeling loop reproduces the same trend, while label entropy and class coverage remain stable, indicating that the gain transfers across architectures and is not caused by label collapse. We further show that learnability is controlled by both iteration and the model-feedback weight βmodel. Iterative gains diminish as successive label systems converge, whereas stronger feedback increasingly widens the performance gap between the feedback Transformer and the independent GRU. Increased learnability is accompanied by declining segment-level price-feature separation after an initial improvement. Finally, a non-anticipative event study provides partial external validation: refinement strengthens a conditional BEAR→BULL reversal signal on the three A-share datasets, while the corresponding effect has the opposite sign on NASDAQ100. These results position DDSL as a structured framework for constructing market-state labels whose learnability can be explicitly controlled and evaluated.
Figures & tables
Figure 1: DDSL label evolution at selected iterations for stock 600760 (Shanghai Stock Exchange, 2000/03/28–2000/11/08). Segment boundaries shift under model feedback, with progressively smaller changes at later iterations.
Parameter
Value
Parameter
Value
Smoothing and windows
EWMA span
10
Lc
10
Lf
20
τ0
5×10−4
Price-emission scoring
λc
1.0
λf
1.2
wα
1.0
wρ
0.8
Table 1: Main-configuration labeling hyperparameters, shared across all datasets, seeds, and iterations unless varied in the βmodel ablation. Lc is the half-width of the symmetric current window.
Dataset
F1(Tr.) 0→5
F1(GRU) 0→5
ΔH
FRseg0→1→5
SSE50
.246→.482
.245→.460
+0.83%
.668→.772→.647
CSI300
.247→.458
.253→.429
+0.01%
.791→.933→.730
CSI500
.246→.460
.245→.441
+1.13%
.744→.868→.648
NASDAQ100
.231→.411
.240→.385
−0.78%
.727→.840→.662
Table 2: Summary of label learnability and its cost. F1 and FRseg entries are means over three seeds. ΔH is the relative change in normalized label entropy from iter 0 to iter 5 .
Figure 2: Full iteration trajectories underlying Table 2 . Transformer joint-state macro F1 increases across all four datasets (a), whereas FRseg peaks at iter 1 and then declines (b). Lines show means over three seeds; shaded bands indicate ±1 standard deviation.
Transition
SSE50
CSI300
CSI500
NASDAQ100
0→1
.209/.741
.214/.735
.220/.729
.198/.749
1→2
.096/.881
.099/.877
.100/.877
.088/.887
2→3
.058/.929
.077/.904
.066/.919
.060/.924
3→4
.047/.942
.058/.929
.057/.930
.039/.951
4→5
.040/.951
.050/.938
.050/.939
.032/.960
Table 3: Convergence between adjacent DDSL iterations on the test split, averaged over three seeds. Each entry reports joint-state change rate (CR) followed by Cohen’s κ .
SSE50
NASDAQ100
βmodel
F1(Tr.)
F1(GRU)
FRseg
F1(Tr.)
F1(GRU)
FRseg
0
.246
.245
.668
.231
.240
.727
0.1
.358
.348
.774
.317
.313
.795
0.2
.418
.404
.693
.367
.352
.699
0.3
.482
.460
.647
.411
.385
.662
0.4
.580
.528
.641
.463
.431
.655
Table 4: Effect of βmodel on Transformer macro F1, GRU macro F1, and segment-level Fisher Ratio at iter 5 , means over three seeds. Seed standard deviations do not exceed .021 for F1 and .040 for FRseg .
Figure 3: Learnability–discriminability map on SSE50 (Transformer mean over three seeds). The iteration sweep ( βmodel=0.3 , solid) and weight sweep (iter 5 , dashed) intersect at the shared main configuration. Dotted lines mark the pure-DP reference, and the shaded region contains configurations that improve both metrics over this baseline. Subsequent refinement increases learnability while reducing segment-level separation. Pseudo-labeling ( × ) lies outside the structured DDSL trajectories.
Transition
Dataset
iter 0
iter 1
iter 5
BEAR → BULL
SSE50
+1.13±0.69
+1.63±0.42
+2.11±0.37
CSI300
+1.17±0.40
+1.08±0.22
+1.50±0.38
CSI500
+1.47±0.73
+0.68±0.23
+2.00±0.79
NASDAQ100
−1.51±0.50
−1.65±1.15
−2.32±0.75
NEUT → BULL
SSE50
+0.70±0.13
+0.20±0.13
−0.25±0.08
CSI300
+0.22±0.33
+0.42±0.34
+0.49±0.15
Table 5: Event-minus-control differences (%) under non-anticipative online Viterbi decoding, mean ± std over three seeds. The two blocks isolate the same predicted BULL call under different current states. Cells contain 3×102 to 8×103 events.
Building a world model requires a state representation useful for planning and decision-making---potentially over tasks unknown at training time. In the context of financial markets, planning and decision-making may require a model to reason about market-wide conditions, asset-specific expected returns, liquidity, volatility, and cross-asset relationships. Yet financial representation learning has largely been evaluated on individual predictive tasks, oftentimes on a single time period using comparatively narrow datasets. We address this through three primary contributions. First, we introduce Market-1T, a dataset containing nearly one trillion observations across U.S. equities from 2008 to 2025 at 1 Hz resolution. Second, we develop and implement a rigorous evaluation protocol. Third, we conduct a systematic large-scale study of financial representation learning, comparing 18 encoder-training strategies across nearly two decades of market regimes. We evaluate learned representations both by their predictive utility on common finance tasks and through probes of latent structure. We find that encoders with similar predictive performance can organize market state very differently. Collectively, we establish a foundation for training and evaluating financial market representations in support of world models such as DINO-WM, V-JEPA 2, and LeWM.
Humzah Merchant, Alec Guthrie, Simon Mahns +2
University of Chicago · Johns Hopkins University · Brown University
Early warning of extreme market volatility is central to financial risk management, but actionable events are rare, nonstationary, and often triggered by exogenous information shocks. In our CSI300 setting, only ∼80 positive samples are observed across 791 training days, making heavily supervised multi-source models unstable. We first analyze a 100K-parameter hierarchical text-signal fusion model (HTSF) and find that added parameterization hurts in this low-label regime. Motivated by this failure, we propose AAMSF (Anomaly-Augmented Multi-Signal Fusion), a semisupervised framework that combines Isolation Forest anomaly scores over market indicators, GDELT events, Chinese financial news, and English media with lightweight Ridge score fusion. We further introduce T-AAMSF, a temporal extension for multi-day anomaly accumulation. On CSI300 (2018--2023), AAMSF achieves test AUC-ROC 0.680, outperforming the strongest unsupervised baseline (0.630) and neural baseline (0.588), while T-AAMSF improves PR-AUC to 0.291. Ablations reveal strong source asymmetry: GDELT and domestic financial news provide complementary risk signals, whereas English media consistently reduces performance, and learned weighting is unreliable under validation noise. These results suggest an empirical design principle for label-scarce financial risk warning: robust anomaly geometry and source reliability can matter more than supervised representation capacity.
In this work, we study portfolio optimization under the stochastic discount factor (SDF) framework by learning market state representations that capture the underlying risk structures of financial data. This is challenging due to several factors: financial markets exhibit non-stationary dynamics with shifting regimes, multimodal inputs such as price and news data often contain stochastic noise, and existing diffusion-based approaches, while effective for modeling stochastic dynamics, rely on assumptions such as isotropic Gaussian noise that fail to capture the state-dependent nature of financial uncertainty. To address these challenges, we introduce RADAR, a retrieval-augmented diffusion framework that learns market representations by conditioning on similar historical regimes. RADAR leverages retrieval to construct context-dependent noise distributions, applies conditional diffusion to denoise multimodal representations, and initializes the diffusion process using empirical statistics to reflect state-dependent uncertainty. Experiments show that RADAR achieves state-of-the-art performance on key risk-adjusted metrics while producing economically meaningful signals on asset returns and correlations.
Kelvin J. L. Koa, Xinyang Li, Ke-Wei Huang
National University of Singapore Asian Institute of Digital Finance