Fine-tuning Timeseries Predictors Using Reinforcement Learning

Hugo Cazaux; Ralph Rudd; Hlynur Stefánsson; Sverrir Ólafsson; Eyjólfur Ingi Ásgeirsson

cs.LGMar 20, 2026

Fine-tuning Timeseries Predictors Using Reinforcement Learning

Authors: Hugo Cazaux, Ralph Rudd, Hlynur Stefánsson, Sverrir Ólafsson, Eyjólfur Ingi Ásgeirsson

Organizations: Reykjavik University, Department of Engineering, Menntavegur 1, Reykjavik, 102, Iceland

Abstract

This chapter presents three major reinforcement learning algorithms used for fine-tuning financial forecasters. We propose a clear implementation plan for backpropagating the loss of a reinforcement learning task to a model trained using supervised learning, and compare the performance before and after the fine-tuning. We find an increase in performance after fine-tuning, and transfer learning properties to the models, indicating the benefits of fine-tuning. We also highlight the tuning process and empirical results for future implementation by practitioners.

Figures & tables

Date	Open	Low	High	Close	Volume
2005-12-05	2.17	2.15	2.19	2.16	5.84e8
2005-12-06	2.23	2.21	2.25	2.23	8.57e8
2005-12-07	2.24	2.20	2.24	2.23	6.79e8
2005-12-08	2.21	2.19	2.23	2.23	7.90e8
2005-12-09	2.24	2.21	2.25	2.24	5.55e8
2005-12-12	2.26	2.25	2.27	2.26	5.25e8

Table 1: Sample daily financial data for AAPL

Environment	Reward
HalfCheetah-v4	$R=w_{f}F-w_{\mathrm{ctrl}}\,C$
Hopper-v4	$R=w_{f}F+w_{h}H-w_{\mathrm{ctrl}}\,C$
Humanoid-v4	$R=w_{f}F+w_{h}H-w_{\mathrm{ctrl}}\,C-w_{\mathrm{ctct}}\,C_{\mathrm{tct}}$

Table 2: MuJoCo environment reward functions (forward reward

F

, healthy reward

H

, control cost

C

, contact cost

C_{\mathrm{tct}}

)

Dataset	Latent		Actor		Reference
Metric	MSE	MAE	MSE	MAE	MSE	MAE
Financial	0.202	0.206	0.200	0.271	0.203	0.118
Industrials	0.274	0.251	0.119	0.116	0.128	0.121
Technology	0.341	0.264	0.126	0.119	0.131	0.119

Table 3: Latent vs Actor paradigms comparison. The backbone is fine-tuned using PPO on Financial, Industrials and Technology. Reference is the base model without fine-tuning. Lower is better, in bold the best metric.

Model	PPO	CMAPPO	GRPO
Environment	Reward	Reward	Reward
HalfCheetah-v4	-150.54	-111.10	-137.18
Hopper-v4	1185.06	1960.75	624.86
Humanoid-v4	2897.81	3201.09	2659.32

Table 4: Results of MuJoCo environment training. Higher is better, best value in bold.

Frozen %	Model	Financial		Industrials		Technology
Metric		MSE	MAE	MSE	MAE	MSE	MAE
0%	PPO	0.200	0.271	0.119	0.116	0.126	0.119
	CMAPPO	0.324	0.208	0.146	0.160	0.203	0.189
	GRPO	0.198	0.109	0.118	0.113	0.124	0.115
25%	PPO	0.199	0.114	0.120	0.116	0.125	0.118
	CMAPPO	0.300	0.204	0.202	0.211	0.525	0.341

Table 5: Results of fine-tuning models on Financial, Industrials and Technology dataset compared to the original model. In rows, the model’s layers are progressively frozen. In columns, each sector represents the testing set of the model. Lower is better, best value in bold.

Trained on	Financial		Industrials		Technology
Tested on	MSE	MAE	MSE	MAE	MSE	MAE
Financial	0.203	0.118	0.207	0.113	0.203	0.110
Industrials	0.224	0.227	0.128	0.121	0.122	0.114
Technology	0.256	0.229	0.132	0.118	0.131	0.117

Table 6: Reference values before fine-tuning.

Trained on $\rightarrow$		Financial		Industrials		Technology
Fine-tuned on $\downarrow$	Method	MSE	MAE	MSE	MAE	MSE	MAE
Financial	PPO	0.279	0.196	0.213	0.219	0.247	0.219
	CMAPPO	0.224	0.143	0.145	0.152	0.151	0.151
	GRPO	0.230	0.156	0.155	0.167	0.170	0.165
	Baseline	0.203	0.118	0.207	0.113	0.203	0.110
Industrials	PPO	0.201	0.115	0.123	0.119	0.131	0.121

Table 7: Results of fine-tuning models on Financial, Industrials and Technology dataset compared to the original model. The model is fine-tuned and tested on the specified sector for each row. In columns, each sector represents the original training set of the model. Lower is better, best value in bold.

Figure 1: Training time vs MSE. Dotted line is the original model performance before fine-tuning. Training time is scaled down from 1e6 for readability.

Number of Subagents	Financial
Metric	MSE	MAE
2	0.925	0.401
4	0.461	0.275
6	0.394	0.256
8	0.337	0.211
10	0.271	0.183

Table 8: The influence of the number of subagents when fine-tuning the backbone compared to the non fine-tuned backbone.

Group Size	Financial
Metric	MSE	MAE
2	0.200	0.204
4	0.198	0.199
6	0.197	0.199
8	0.195	0.196
10	0.199	0.201

Table 9: The influence of the group size when fine-tuning the model on the Financial dataset.

RL Fine-Tuning

Reinforcement learning fine-tuning adapts pretrained models and policies through task rewards, preference feedback, or environment interaction.

See all 231 papers

TimeRFT: Stimulating Generalizable Time Series Forecasting for TSFMs via Reinforcement Finetuning
Apr 18, 2026Siyang Li, Yize Chen, Zijie Zhu +4Time Series Forecasting OOD Generalization
HKUST(GZ) Guangzhou, China · University of Alberta Edmonton, Canada · Alibaba Cloud Guangzhou, China +3
DPAL: A Test-Time Adaptation Loss for Financial Time Series Forecasting
Oct 9, 2026Wuzhou Lian, Haochun Wang, Yuefan DengTime Series Forecasting Financial Forecasting
University of Michigan · Stony Brook University
Integrating Inductive Biases in Transformers via Distillation for Financial Time Series Forecasting
Mar 17, 2026Yu-Chen Den, Kuan-Yu Chen, Kendro Vincent +1Time Series Forecasting Financial Time Series
SinoPac Holdings, Taipei, Taiwan · National Chengchi University, Taipei, Taiwan

Apr 18, 2026J/K move · Enter open · S save

TimeRFT: Stimulating Generalizable Time Series Forecasting for TSFMs via Reinforcement Finetuning

Siyang Li, Yize Chen, Zijie Zhu, Yuxin Pan+3

HKUST(GZ) Guangzhou, China · University of Alberta Edmonton, Canada · Alibaba Cloud Guangzhou, China+3

Time Series Forecasting OOD Generalization Non-Stationary Time Series Forecasting RL Fine-Tuning Time-Series Foundation Models

Open paper AskPDF arXiv

Time Series Foundation Models (TSFMs) advance generalization and data efficiency in time series forecasting by unified large-scale pretraining. But TSFMs remain lacking when adapting to specific downstream forecasting tasks for two reasons. First, the non-stationary and uncertain nature of time series data lead to inevitable temporal distribution shifts between historical training and future testing data, while current Supervised FineTuning (SFT)-based methods are prone to overfitting and may degrade generalization. Second, training data availability varies across forecasting tasks, requiring TSFMs to generalize well under diverse data regimes. To address these challenges, we introduce the Time series Reinforcement Finetuning (TimeRFT) paradigm for TSFM downstream adaptation, which consists of two task-specific training recipes: i) A forecasting quality-based temporal reward mechanism that conducts a multi-faceted evaluation of the contribution of each prediction step to overall forecasting accuracy. ii) A forecasting difficulty-based data selection strategy to identify time series samples with generalizable predictive patterns and informative training signals. Extensive experiments demonstrate TimeRFT can consistently outperform SFT-based adaptation methods across various real-world forecasting tasks and training data regimes, enhancing prediction accuracy and generalization against unforeseen distribution shifts.

Abstract

RL Fine-Tuning

Explore similar work

TimeRFT: Stimulating Generalizable Time Series Forecasting for TSFMs via Reinforcement Finetuning

DPAL: A Test-Time Adaptation Loss for Financial Time Series Forecasting

Integrating Inductive Biases in Transformers via Distillation for Financial Time Series Forecasting