Integrating Inductive Biases in Transformers via Distillation for Financial Time Series Forecasting

Yu-Chen Den; Kuan-Yu Chen; Kendro Vincent; Darby Tien-Hao Chang

cs.LGMar 17, 2026

Integrating Inductive Biases in Transformers via Distillation for Financial Time Series Forecasting

Authors: Yu-Chen Den, Kuan-Yu Chen, Kendro Vincent, Darby Tien-Hao Chang

Organizations: SinoPac Holdings, Taipei, Taiwan · National Chengchi University, Taipei, Taiwan

Abstract

Transformer-based models have been widely adopted for time-series forecasting due to their high representational capacity and architectural flexibility. However, many Transformer variants implicitly assume stationarity and stable temporal dynamics -- assumptions routinely violated in financial markets characterized by regime shifts and non-stationarity. Empirically, state-of-the-art time-series Transformers often underperform even vanilla Transformers on financial tasks, while simpler architectures with distinct inductive biases, such as CNNs and RNNs, can achieve stronger performance with substantially lower complexity. At the same time, no single inductive bias dominates across markets or regimes, suggesting that robust financial forecasting requires integrating complementary temporal priors. We propose TIPS (Transformer with Inductive Prior Synthesis), a knowledge distillation framework that synthesizes diverse inductive biases -- causality, locality, and periodicity -- within a unified Transformer. TIPS trains bias-specialized Transformer teachers via attention masking, then distills their knowledge into a single student model with regime-dependent alignment across inductive biases. Across four major equity markets, TIPS achieves state-of-the-art performance, outperforming strong ensemble baselines by 55%, 9%, and 16% in annual return, Sharpe ratio, and Calmar ratio, while requiring only 38% of the inference-time computation. Further analyses show that TIPS generates statistically significant excess returns beyond both vanilla Transformers and its teacher ensembles, and exhibits regime-dependent behavioral alignment with classical architectures during their profitable periods. These results highlight the importance of regime-dependent inductive bias utilization for robust generalization in non-stationary financial time series.

Figures & tables

Figure 1. Performance–efficiency trade-off across generic time-series models, financial forecasting models, and classical architectures evaluated across multiple equity markets. The figure highlights substantial variation in performance and computational cost across model families, with TIPS achieving the strongest performance among the evaluated methods while maintaining low inference-time overhead.

Inductive Bias	Classical Models	Transformer Design
Global Context	Standard Transformer	Vanilla Attention
Causality	RNNs, Mamba	Causal Attention Masks
Locality	TCN	Input Patching, Recency Decay Bias
Periodicity	Dilated CNN, LSTM Memory	Positional Bias (fixed/learnable)

Table 1. Inductive biases in financial time series and their instantiation in classical models and Transformers. Global context denotes unconstrained attention.

Single Inductive Bias
Method	CSI300	CSI500	NI225	SP500	Avg.
Vanilla Transformer	0.915	1.662	0.774	1.297	1.162
+ Causality	1.032 †	1.510	0.841 †	1.261	1.161
+ Locality	0.972 †	1.477	0.695	1.283	1.107
+ Periodicity	1.056 †	1.496	0.693	1.234	1.120
Multiple Biases (Merged)

Table 2. Performance (Annual Sharpe Ratio) of different bias integration strategies. Bold : best per market;

\dagger

: outperforms Vanilla Transformer (row 1). Higher is better.

Figure 2. Overview of the TIPS training framework. (a) Bias-specialized Transformer (TFM) teachers are constructed via different attention masks or positional biases (Colors indicate where the masks and biases are applied). (b) Teachers are trained independently for ranking prediction. (c) Teacher predictions are averaged and distilled into a single student model.

Market	Region	# Stocks	Tr/Va/Te Split	SR	Auto Corr.
CSI300	China	295	Tr: 2008-2019 Va: 2020 Te: 2021-2024	0.294	0.003
CSI500	China	514		0.265	0.006
NI225	Japan	209		0.304	-0.012
SP500	US	525		0.524	-0.013

Table 3. Dataset statistics, including market, region, number of stocks, train/validation/test splits, annual Sharpe ratio (SR), and weekly autocorrelation (Auto Corr.).

Markets	CSI300			CSI500			NI225			SP500			Avg. across markets			Efficiency
Metrics $\rightarrow$	AR	SR	CR	AR	SR	CR	AR	SR	CR	AR	SR	CR	AR	SR	CR	FLOPs (G)
Generic Time Series SOTA
DLinear	0.071	0.259	0.173	0.192	0.604	0.481	0.134	0.553	0.472	0.378	0.914	1.168	0.194	0.583	0.574	< 0.001
TimeMixer	0.155	0.642	0.533	0.170	0.650	0.412	0.120	0.538	0.392	0.203	0.795	0.732	0.162	0.656	0.517	1.123
AutoFormer	0.155	0.667	0.592	0.235	0.921	0.778	0.124	0.562	0.449	0.258	0.873	0.871	0.193	0.756	0.673	2.175
PatchTST	0.149	0.614	0.506	0.177	0.668	0.492	0.159	0.719	0.554	0.248	0.936	0.796	0.183	0.734	0.587	0.484

Table 4. Main results across four equity markets. Metrics: Annual Return (AR), Sharpe Ratio (SR), Calmar Ratio (CR). Bold: best; underline: second best. Bias Teacher Ensemble consists of seven mask-based bias-specialized teachers, while TIPS denotes the distilled student model. Results are averaged over five random seeds. Detailed efficiency analysis is provided in Appendix C .

Supervision	CSI300	CSI500	NI225	SP500	Avg. SR
$0.5\times\text{distill}+0.5\times\text{target}$	1.403	2.080	0.883	1.462	1.457
$0.9\times\text{distill}+0.1\times\text{target}$	1.307	2.036	0.985	1.505	1.458
$1.0\times\text{distill}$ (TIPS)	1.343	2.010	0.958	1.506	1.454

Table 5. Ablation of supervision ratio. Results report Sharpe Ratio (SR) under different weightings of the distillation and ground-truth ranking targets. Bold: best; underline: second best.

Method	CSI300	CSI500	NI225	SP500	Avg. SR	SR Gap ( $\downarrow$ )
Bias Teacher Ensemble	1.152	1.883	0.902	1.665	1.401	3.172
TIPS
Vanilla Distill	1.203	1.826	0.882	1.573	1.371	0.341
+ Low Temp.	1.440	1.810	0.973	1.473	1.424	0.546
+ Aggressive LS	1.324	2.024	0.905	1.554	1.452	0.560
+ SWA	1.343	2.010	0.958	1.506	1.454	0.436

Table 6. Ablation of distillation regularization components. Results report Sharpe Ratio (SR) obtained by incrementally adding regularization terms to vanilla distillation. SR Gap denotes the absolute train-test SR difference averaged over markets. Bold: best; underline: second best.

Method	CSI300	CSI500	NI225	SP500
Causality	0.257	0.295	0.240	0.447
Locality	0.236	0.241	0.238	0.369
Periodicity	0.235	0.244	0.243	0.373
Vanilla	0.239	0.250	0.253	0.413
TIPS w/o LS & SWA	0.563	0.655	0.470	0.641
TIPS w/o SWA	0.359	0.414	0.315	0.458

Table 7. Average top-1 portfolio weight as a proxy for extreme-score concentration across distillation variants and individual teacher groups.

Distill Steps	CSI300	CSI500	NI225	SP500	Average
Vanilla Distill	$0.607\pm 0.059$	$0.658\pm 0.064$	$0.342\pm 0.282$	$0.517\pm 0.311$	0.531 $\pm$ 0.179
+ Low Temp.	$0.543\pm 0.054$	$0.584\pm 0.047$	$0.252\pm 0.190$	$0.433\pm 0.232$	0.453 $\pm$ 0.131
+ Aggressive LS	$0.472\pm 0.040$	$0.488\pm 0.022$	$0.251\pm 0.190$	$0.283\pm 0.104$	0.374 $\pm$ 0.089
+ SWA	0.454 $\pm$ 0.035	0.460 $\pm$ 0.037	0.238 $\pm$ 0.190	0.297 $\pm$ 0.125	0.362 $\pm$ 0.097
Total Reduction	-25.2%	-30.1%	-30.6%	-42.6%	-31.8%

Table 8. Teacher–student rank similarity across distillation regularization stages. Values report mean Spearman correlation

\pm

standard deviation computed over the seven teachers.

Bias Configuration	Method	CSI300	CSI500	NI225	SP500	Avg. SR
Causality only	Ensemble	1.250	1.868	0.877	1.636	1.408
Causality only	Distill	1.235	2.082	1.010	1.476	1.451
Locality only	Ensemble	1.118	1.813	0.754	1.453	1.285
Locality only	Distill	1.215	1.815	0.904	1.486	1.355
Periodicity only	Ensemble	1.155	1.774	0.788	1.487	1.301
Periodicity only	Distill	1.232	1.828	0.793	1.449	1.326

Table 9. Ensemble vs. distillation across different bias configurations. Bold : better method per bias setting and market.

	vs. Vanilla Transformer				vs. Bias Teacher Ensemble
Market	Alpha ( $\alpha$ )	$t$ -stat	Beta ( $\beta$ )	$R^{2}$	Alpha ( $\alpha$ )	$t$ -stat	Beta ( $\beta$ )	$R^{2}$
CSI300	$0.75\times 10^{-3}$	$3.724^{***}$	0.975	0.585	$0.50\times 10^{-3}$	$2.706^{**}$	0.973	0.651
CSI500	$0.93\times 10^{-3}$	$3.711^{***}$	1.207	0.682	$0.63\times 10^{-3}$	$2.780^{**}$	1.156	0.742
NI225	$0.26\times 10^{-3}$	$1.965^{*}$	1.019	0.703	$0.12\times 10^{-3}$	1.056	0.988	0.768
SP500	$1.03\times 10^{-3}$	$1.655^{*}$	1.319	0.698	$0.21\times 10^{-3}$	0.341	1.231	0.705

Table 10. Performance attribution analysis. We regress TIPS returns on vanilla Transformer and Bias Teacher Ensemble returns, reporting daily Alpha (

\alpha

t

-statistics of Alpha, Beta (

\beta

), and

R^{2}

{}^{*}p<0.05

{}^{**}p<0.01

{}^{***}p<0.001

(one-tailed).

Teacher Group	CSI300	CSI500	NI225	SP500
Causality	0.090	0.258	0.000	0.001
Locality	0.319	0.280	0.323	0.313
Periodicity	0.409	0.337	0.373	0.434
Vanilla	0.166	0.104	0.238	0.147
Average Teacher	0.017	0.022	0.066	0.105

Table 11. Fraction of test days on which the student attention map is closest to each teacher group or the average-teacher baseline, measured via cosine similarity over portfolio-weighted two-layer attentions of the top-5 selected stocks.

Market	Baseline	SR	$\bar{\rho}$	$\Delta\rho$	$t$ -stat	$p$ -value
CSI300	Vanilla Transformer	0.915	0.660	0.038	2.57	0.005**
	LSTM	0.842	0.649	0.025	1.18	0.120
	GRU	0.875	0.650	0.049	2.19	0.014*
	Mamba	0.648	0.608	0.055	2.12	0.017*
	TCN	1.169	0.599	0.023	1.47	0.071
CSI500	Vanilla Transformer	1.662	0.738	-0.012	-0.80	0.787

Table 12. Conditional alignment analysis using MA5 features.

\bar{\rho}

denotes overall pattern similarity, and

\Delta\rho

measures the increase in similarity during profitable periods (top 30% returns) relative to unprofitable periods. Bold / underline indicate the best/second-best Sharpe ratio (SR) per market.

{}^{*}p<0.05

{}^{**}p<0.01

{}^{***}p<0.001

(one-tailed).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Hidden	Layers	FFN	Heads	Dropout
GRU / LSTM	64	2	—	—	0.0
TCN †	64	2	—	—	0.0
Transformer	64	2	256	4	0.0
PatchTST ‡	64	2	256	—	0.0
iTransformer	64	2	256	—	0.0
AutoFormer	64	2	—	—	0.1

Appendix

Table 13. Baseline hyperparameter configurations.

Metrics $\rightarrow$	FLOPs(G)	Inf. Time(ms)	Memory(MB)	Params
Baselines
DLinear	< 0.001	0.28	8.88	72
TimeMixer	1.123	4.79	77.39	7183
AutoFormer	2.175	20.00	46.08	244417
PatchTST	0.485	1.43	15.73	25878
iTransformer	0.611	1.07	28.41	101505

Appendix

Table 14. Computational efficiency comparison across models. We report FLOPs (G), inference time per sample (ms), peak memory usage (MB), and total number of trainable parameters.

Market	$(3,128)$	$(3,256)$	$(4,128)$	$(4,256)$	TIPS
	Larger Transformer $(L,d)$
CSI300	0.65	0.52	0.45	0.61	1.34
CSI500	1.02	0.57	0.58	0.50	2.01
NI225	0.76	0.68	0.70	0.70	0.96
SP500	1.07	1.13	1.11	1.20	1.61

Appendix

Table 15. SR of larger vanilla Transformers compared to TIPS.

Figure 3. Regime analysis of four different markets.

Time Series Forecasting

Time series forecasting predicts future values from ordered observations using statistical, neural, and probabilistic models to support decisions under uncertainty.

See all 858 papers

Jun 28, 2026

Adaptive Financial Transformer with Regime-Gated Attention for Stock Return Prediction

Adaptive Financial Transformer (AFT) is proposed for stock return prediction under non-stationary financial markets. The model incorporates a Market Regime Encoder, an Adaptive Gate Network, and an Adaptive Financial Context module to dynamically bias self-attention based on semantic relationships between financial indicators. Unlike conventional Transformer architectures that treat all input features uniformly, the proposed approach groups 95 engineered financial features into 11 semantic categories and adapts attention according to latent market regimes. The study also identifies and corrects sequence alignment and backtesting issues that can inflate reported trading performance, and introduces a financially-aware composite objective that jointly optimizes prediction error, directional accuracy, and non-overlapping Sharpe ratio. Extensive experiments compare the proposed architecture against classical machine learning models, recurrent neural networks, and Transformer baselines using chronological evaluation, five random seeds, ablation studies, hyperparameter optimization, explainability analysis, and multi-stock validation. Results demonstrate competitive predictive performance while reducing model complexity by 15.2% and improving parameter efficiency through feature selection, providing an interpretable Transformer architecture for financial time-series forecasting.

Dishan Sarkar

Department of Artificial Intelligence and Machine Learning Birla Institute of Technology, Mesra Ranchi, India

Multivariate Time Series Forecasting Time Series Forecasting Non-Stationary Time Series Forecasting Attention Mechanisms Time Series Transformers Finance

Ask PDF

Mar 31, 2026

Forecast collapse of transformer-based models under squared loss in financial time series

We study trajectory forecasting under squared loss for time series with weak conditional structure, using highly expressive prediction models. Building on the classical characterization of squared-loss risk minimization, we emphasize regimes in which the conditional expectation of future trajectories is effectively degenerate, leading to trivial Bayes-optimal predictors (flat for prices and zero for returns in standard financial settings). In this regime, increased model expressivity does not improve predictive accuracy but instead introduces spurious trajectory fluctuations around the optimal predictor. These fluctuations arise from the reuse of noise and result in increased prediction variance without any reduction in bias. This provides a process-level explanation for the degradation of Transformerbased forecasts on financial time series. We complement these theoretical results with numerical experiments on high-frequency EUR/USD exchange rate data, analyzing the distribution of trajectory-level forecasting errors. The results show that Transformer-based models yield larger errors than a simple linear benchmark on a large majority of forecasting windows, consistent with the variance-driven mechanism identified by the theory.

Pierre Andreoletti

IDP · Institut Denis Poisson, UMR C.N.R.S. 7013, France.

Time Series Forecasting Time Series Transformers Financial Time Series

Ask PDF

Jun 2, 2026

FinStressTS: A Parametric Synthetic Benchmark for Time-Series Forecasting in Finance

Financial forecasting is difficult due to low signal-to-noise ratios, latent factors, heavy tails, regime shifts, and jumps. Real-world benchmarks offer limited failure attribution: researchers can observe underperformance, but often cannot isolate why because mechanisms are unobservable and entangled. Real financial data reveal only one realized path, making it difficult to assess tail-risk calibration or data efficiency. We introduce FinStressTS, a mechanism-aware synthetic benchmark that links model behavior to controlled structural causes. FinStressTS comprises 30 diagnostic environments around six mechanism families: volatility clustering, multi-scale persistence, heavy-tailed shocks, regime switching, self-exciting jumps, and zero-inflated processes. We evaluate two tasks: point forecasting, using NMAE across five settings, and probabilistic forecasting, using CRPS under known data-generating mechanisms. We benchmark 15 models, from classical methods (HAR, VAR) to Transformer forecasters (PatchTST, iTransformer) and deep probabilistic architectures (DeepAR, TSFlow), and use learning curves to measure sample efficiency. Our evaluation reveals three insights. First, performance is mechanism-dependent: autoregressive and linear models are highly competitive, and often outperform Transformer-based models, in several volatility-, tail-, and jump-driven environments. Second, distributional alignment matters: parametric probabilistic models such as DeepAR calibrate well in stationary settings, while flexible models can help when distributions become multimodal or sparse. Third, neural models often require more data to match simple baselines, with larger gains mainly when learning latent regimes or complex distributions. FinStressTS provides an open framework for diagnosing failure modes and advancing risk-aware forecasting.

Jiaze Sun, Kelvin J. L. Koa, Ruiyang Ni +3

National University of Singapore Singapore · National University of Singapore Asian Institute of Digital Finance Singapore · Nanyang Technological University Singapore

Financial Forecasting Time Series Forecasting Forecasting Benchmarks Synthetic Benchmark Generation Financial Time Series Finance+1

Ask PDF

Abstract

Time Series Forecasting

Explore similar work

Adaptive Financial Transformer with Regime-Gated Attention for Stock Return Prediction

Forecast collapse of transformer-based models under squared loss in financial time series

FinStressTS: A Parametric Synthetic Benchmark for Time-Series Forecasting in Finance