The impact of temporal resampling on the safety of clinical reinforcement learning: A simulated trial

Thomas Frost; Hrisheekesh Vaidya; Steve Harris

cs.LGFeb 6, 2026

The impact of temporal resampling on the safety of clinical reinforcement learning: A simulated trial

Authors: Thomas Frost, Hrisheekesh Vaidya, Steve Harris

Organizations: University College London, London, United Kingdom · University College London Hospitals NHS Foundation Trust, London, United Kingdom

Abstract

Reinforcement learning (RL) is a type of artificial intelligence for making optimal choices. In healthcare, researchers generally use offline RL (ORL), where models are trained and evaluated from retrospective observational data. To accommodate naturally irregular clinical records, researchers often resample the data into uniform time intervals before training (known as binning). However, binning may create counterfactual trajectories that do not represent clinical practice. We first quantified these distortions in patients receiving continuous insulin infusions in the MIMIC-IV intensive care database. Binning reversed the apparent causal order for over 70% of paired glucose measurements and insulin adjustments within the same interval, and frequently masked significant dysglycaemic events and merged distinct treatment decisions together. We then conducted an in silico clinical trial using 30 virtual patients with type 1 diabetes from the FDA-approved UVA/Padova simulator. The simulator was modified to include stochastic intervals between decisions and used to generate a training dataset for offline RL. We trained three ORL algorithms on both the unprocessed dataset and equivalent datasets resampled at 10-minute, 2-hour, and 4-hour intervals. When deployed back into the simulated environment, temporal resampling was found to reduce model performance by up to 60% relative to unprocessed data, with 4-hour binning causing all agents to perform worse than the dataset's baseline. Retrospective evaluation on resampled data actively obscured this effect, predicting 1.5-3x better returns than agents achieved in practice. We recommend that future research in this area prioritises datasets with natural clinical timings between decisions, which may be a necessary step before these models can be safely deployed into patient care.

Figures & tables

Figure 1: Example patient trajectories from MIMIC-IV and the UVA/Padova simulator. A) The top and bottom panels display a patient trajectory from MIMIC-IV receiving an insulin infusion, before and after temporal binning (respectively). A causal inversion artefact can be seen, where insulin was escalated in response to a spike in the blood glucose level. In the binned trajectory, this action appears to occur in response to glucose levels from the previous time window, with the glucose spike occurring after the action. B) A patient trajectory from the UVA/Padova simulator, also before and after binning. A causal inversion artefact is observed, this time in response to a carbohydrate intake.

Figure 2: Quantification of artefacts in MIMIC-IV following temporal binning. A) The proportion of causally inverted artefacts (from reactive glucose-insulin pairs that occupy the same time window) is shown to be high even with 1-hour binning, and increases monotonically with greater bin widths. B) The proportion of hypo- and hyperglycaemic events that are masked as a result of averaging blood glucose levels within each bin window. C) The proportion of bins containing 2 or more insulin adjustment decisions (which are then aggregated into a single action).

Figure 3: Impact of temporal resampling on offline RL performance. Models trained via behavioural cloning (BC), implicit Q-learning (IQL), or conservative Q-learning (CQL) were evaluated on the UVA/Padova insulin control task. Models were trained using unprocessed, interpolated, or temporally binned datasets and deployed in both regularly and irregularly timed versions of the environment. Models trained on the unprocessed dataset consistently achieved the highest returns, which were heavily degraded using binned or interpolated datasets. The pink band indicates the expert proximal policy optimisation (PPO) baseline used to generate the training data. Average returns are normalised (0.0 for a random policy; 1.0 for the highest observed score). Shaded regions and error bars represent 95% confidence intervals (CIs).

Figure 4: Calibration plot showing reliability of off-policy evaluation across different types of dataset preprocessing. The plot compares the true online performance of trained agents in the UVA/Padova environment against the performance predicted by fitted Q-evaluation (FQE). Performance is normalised such that 0.0 represents a random policy and 1.0 represents the dataset’s behaviour policy. While agents trained on unprocessed (blue) and interpolated (orange) data show high calibration (clustering near the diagonal), those trained on temporally binned data (green, red) exhibit severe overestimation bias. Error bars represent 95% confidence intervals.

Hyperparameters	LavaGap	UVA/Padova	UVA/Padova (FQE)
Learning rate	$1\cdot 10^{-3}$	$3\cdot 10^{-4}$	$3\cdot 10^{-4}$
Batch size	64	1024	256
Hidden dimensions	64	128	64
$\gamma$ — discount factor	0.99	0.99	1.0
$\tau$ — expectile (IQL)	0.8	0.9	—
$\beta$ — temperature (IQL)	10	10	—

Table 1: Hyperparameters for LavaGap and UVA/Padova environments.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1 : LavaGap Environment. To reach the green goal square, the agent (red triangle) must navigate the grid without contacting the lava (orange squares). The model uses partial observations (light grey) to calculate the optimal path. The episode ends with a reward of 1 if the agent reaches the goal, or 0 if it hits lava or exceeds the maximum number of steps.

Figure S2 : Quantification of artefacts in UVA/Padova following temporal binning. A) The proportion of causally inverted artefacts from insulin adjustments paired with the preceding onset of hypo-/hyperglycaemia, as a result of both events sharing the same time window. B) The proportion of hypo- and hyperglycaemic events that are masked as a result of averaging blood glucose levels within each bin window. C) The proportion of bins containing 2 or more insulin adjustment decisions (which are then aggregated into a single action).

Figure S3 : Results from LavaGap. Offline reinforcement learning agents (BC, IQL, CQL) demonstrate universal reductions in performance when trained on temporally resampled datasets (interpolated, binned) compared to unprocessed datasets with natural timings. In combination with Fig 3 , this effect appears to generalise across both discrete and continuous action domains. Error bars and shaded regions represent 95% confidence intervals.

Figure S4 : Time in target range performance in the UVA/Padova simulator. Proportion of time spent within the target glucose range for each offline reinforcement learning agent (BC, IQL, CQL). Under both irregular (left) and regular (right) sampling, IQL and CQL achieve peak performance using unprocessed datasets, matching or exceeding the online PPO baseline. However, training on temporally resampled datasets (interpolated and binned) leads to a consistent degradation in performance for both algorithms. Error bars and the shaded horizontal region represent 95% confidence intervals.

Figure S5 : Early termination rates in the UVA/Padova simulator : Proportion of episodes ending in early termination for offline reinforcement learning agents (BC, IQL, CQL) under irregular (left) and regular (right) sampling conditions. BC shows a high propensity for early termination, approaching 1.0 when trained on binned datasets for both environments. IQL and CQL both lead to a lower proportion of episodes with early termination, with a trend towards the lowest values from interpolated data and the highest values from binned data. Error bars and shaded regions represent 95% confidence intervals, with the pink band indicating the PPO reference baseline.

Figure S6 : Example patient trajectory in the UVA/Padova simulator (glucose and rewards only). The figure shows how the per-step and cumulative rewards change for different values of glucose in the UVA/Padova T1DM simulator over the course of an example patient trajectory.

Paper	Date	Authors	Title	Data	Model	Evaluation	Intervals
1	2016 Aug	Nemati et al. [ 71 ]	Optimal medication dosing from suboptimal clinical examples	MIMIC-II	Fitted Q-Iteration (model-free)	Retrospective evaluation only	1 hour
2	2017 Aug	Prasad et al. [ 12 ]	A reinforcement learning approach to weaning of mechanical ventilation in intensive care units	MIMIC-III	Fitted Q-Iteration (model-free)	Retrospective evaluation only	10 minutes
3	2017 Nov	Raghu et al. [ 47 ]	Continuous state-space models for optimal sepsis treatment: a deep reinforcement learning approach	MIMIC-III	Dueling Double Deep Q-Network (model-free)	Retrospective evaluation only	4 hours
4	2018 Jul	Lin et al. [ 72 ]	A deep deterministic policy gradient approach to medication dosing and surveillance in the ICU	MIMIC-III, Emory University	Deep Deterministic Policy Gradient (model-free)	Retrospective evaluation only	Unspecified (3 hours?)
5	2018 Jul	Wang et al. [ 73 ]	Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation	MIMIC-III	Supervised RL with Recurrent Neural Network (model-free)	Retrospective evaluation only	24 hours
6	2018 Nov	Komorowski et al. [ 7 ]	The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care	MIMIC-III, eICU	Policy Iteration (model-based)	Retrospective evaluation only	4 hours

Appendix

Table S1 : Literature Review : Summary of relevant literature from 2016 to 2026 for offline reinforcement learning using electronic health record data, including choice of resampling interval where applicable.

Item	Description	Page	Notes
Module 1: Study goals
1a	Population or distribution about which the scientific claim is made.	Introduction	Offline reinforcement learning on any temporally resampled healthcare data.
1b	Motivation for choosing this population or distribution (1a.).	Introduction	Explore risks to patients from decision models trained on resampled (rather than unprocessed) data.
1c	Motivation for the use of ML methods in the study.	Introduction	Clinical reinforcement learning could improve healthcare decision models, but often mandates an offline training approach.
Module 2: Computational reproducibility
2a	Dataset used for training and evaluating the model along with link or DOI to uniquely identify the dataset.	Methods for dataset generation	Datasets were generated uniquely by interacting with the LavaGap and UVA/Padova environments. Available at the GitHub repo below.

Appendix

Table S2: REFORMS checklist : As per Kapoor et al. [ 70 ] .

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Insulin4RL: Real-Time Insulin Management in the Intensive Care Unit for Offline Reinforcement Learning
Jun 17, 2026Thomas Frost, Steve HarrisRL Benchmarks Offline RL
Institute of Health Informatics University College London London, United Kingdom
Unmasking Toxic Mimicry in Medical Offline Reinforcement Learning for ICU Sepsis Management via Counterfactual Clinical Audits
Aug 11, 2026Hangqi Ren, Junyi LiaoCounterfactual Evaluation Offline RL
School of Engineering, Vanderbilt University, Nashville, TN, USA · Pratt School of Engineering, Duke University, Durham, NC, USA
ResidencyRL: Reinforcement Learning in Simulated Clinical Environments
Aug 7, 2026Valentin Liévin, Samuel Schmidgall, Tim Strother +32Clinical Decision-Making Clinical Reasoning
Google DeepMind · Google Research · Department of Oncology, Houston Methodist Hospital, Houston, TX +3

Jun 17, 2026J/K move · Enter open · S save

Insulin4RL: Real-Time Insulin Management in the Intensive Care Unit for Offline Reinforcement Learning

Thomas Frost, Steve Harris

Institute of Health Informatics University College London London, United Kingdom

RL Benchmarks Offline RL

Open paper AskPDF arXiv

Offline reinforcement learning (ORL) offers the potential to improve the quality of clinical decision-making using historical electronic health record (EHR) data. Current training and evaluative practices in this field rely heavily on EHR datasets that have been temporally discretised into fixed, regular time intervals. Discretisation creates fictional representations of complex clinical scenarios and compromises the generalisability of retrospective model evaluations. In this paper, we introduce Insulin4RL, a healthcare ORL dataset featuring naturally irregular inputs and actions from real clinical trajectories. Derived from MIMIC-IV, Insulin4RL comprises over 375,000 labelled decisions across 12,209 patients requiring insulin infusion titration in the Intensive Care Unit. The dataset can thus be used for research into ORL model performance under realistic clinical sampling assumptions. We provide a description of the dataset's structure and characteristics, baseline performance metrics using model-free offline reinforcement learning, and a standardised evaluation protocol using fitted Q-evaluation. We conclude with suggested areas for future research that could be addressed using this resource.

Abstract

Reinforcement Learning

Explore similar work

Insulin4RL: Real-Time Insulin Management in the Intensive Care Unit for Offline Reinforcement Learning

Unmasking Toxic Mimicry in Medical Offline Reinforcement Learning for ICU Sepsis Management via Counterfactual Clinical Audits

ResidencyRL: Reinforcement Learning in Simulated Clinical Environments