What Next-Event Accuracy Cannot See: Closed-Loop Evaluation of Emergency Department Trajectory Simulators

Zhen Xuen Brandon Low

cs.LGSep 10, 2026

What Next-Event Accuracy Cannot See: Closed-Loop Evaluation of Emergency Department Trajectory Simulators

Authors: Zhen Xuen Brandon Low

Organizations: Monash University

Abstract

Clinical trajectory models are usually evaluated by next-event accuracy on observed histories. Simulation is different: models must condition on their own generated events, allowing errors to compound. Although this problem is well known in sequence modelling, it has not been systematically quantified for clinical trajectory simulators. We developed EDSim-Bench to evaluate this failure mode using 425,028 MIMIC-IV-ED stays, with external replication on MC-MED, and release the evaluation protocol and scoring code. Starting from held-out visit prefixes, models generate the remainder of each visit and are evaluated on termination, event composition, timing, conditional fidelity, and occupancy forecasting, with a train-only order-3 n-gram as a reference baseline. Despite next-event accuracies within 0.001, three neural architectures behaved very differently under rollout. Across seeds, one Transformer recipe ranged from 0.43 to 0.96 in termination score and from 4.2- to 137-fold the divergence of the n-gram; no prefix-trained neural model approached the n-gram on termination or event composition. Inference-time interventions improved termination but did not jointly recover composition and timing. Supervising every eligible sequence position rather than only the final prefix position was associated with one to two orders of magnitude lower divergence across Transformer, GRU, and LSTM models, with the pattern persisting under model scaling, temporal shift, and external-site evaluation. Nevertheless, even the best model generated visits approximately half as long as observed, and model rankings reversed on occupancy forecasting, a downstream quantity relevant to bed management. These results show that next-event accuracy is insufficient to evaluate clinical trajectory simulators and motivate closed-loop evaluation across seeds, rollout criteria, and downstream tasks.

Figures & tables

A. Evaluation gap: representative full-test runs
Model	Open-loop acc.	Term.	Raw JSD ( $\times$ F)	Dur.
$n$ -gram-3	–	0.9998	0.0023 (1)	1.14
Transformer	0.442	0.434	0.310 (136)	0.97
GRU	0.442	0.599	0.073 (32)	0.67
LSTM	0.443	0.568	0.084 (37)	0.45
Model (supervision)	Seeds	Term. range	Raw JSD range	$\times$ F range

Table 1: Core internal results with the 32-token vocabulary and vanilla decoding. Panel A gives representative full-test runs; Dur. is generated over observed remaining duration on terminated rollouts. Panel B gives seed ranges on the matched 100k-prefix audit. Raw JSD is followed by its multiple of the train-only order-3 reference. Architecture-level claims use panel B and Appendix E .

Figure 1: Training supervision and closed-loop behaviour. Each point represents one training seed using the 32-token vocabulary. Event-distribution divergence is reported relative to the train-only order-3

n

-gram reference (

\times

F, log scale). Models trained with supervision at every eligible sequence position show substantially lower divergence and less variation across seeds, particularly for the causal Transformer.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Vocab	Model	Acc.	Top-3	LOS MAE
7-tok	Transformer	0.594	0.932	216.8
7-tok	GRU	0.604	0.936	213.3
7-tok	LSTM	0.605	0.937	213.2
32-tok	Transformer	0.442	0.768	216.6
32-tok	GRU	0.442	0.763	213.5
32-tok	LSTM	0.443	0.763	213.5

Appendix

Table 2: Open-loop next-event and remaining-LOS metrics. Standard-model rows use the full test split; the causal all-position and decoder-only rows use a 30k stratified sample. Decoder event accuracy is evaluated after teacher-forcing the preceding observed time-bin token and is therefore not strictly comparable with the continuous-time models.

Vocab	Model	Own term.	Matched
32-tok	$n$ -gram-3	1.14	1.16
32-tok	Transformer	0.97	1.18
32-tok	GRU	0.67	0.58
32-tok	LSTM	0.45	0.38
7-tok	$n$ -gram-3	1.15	1.18
7-tok	Transformer	0.08	0.07

Appendix

Table 3: Duration ratio computed over each model’s own terminated rollouts and over launch points where all four models terminate. The matched sets contain 288,146 launch points for the 32-token vocabulary and 730,046 for the 7-token vocabulary. A ratio of 1.00 indicates calibrated mean remaining duration.

Vocab	Model/Dec.	Term.	JSD $\times$ F	Edit	Dur.	LOS	RD
7-tok	Transf. Van	0.905	0.091 (56)	0.679	0.08	300	0.623
7-tok	Transf. PM	0.889	0.088 (55)	0.666	0.09	295	0.776
7-tok	GRU Van	0.680	0.126 (78)	0.718	0.07	279	0.463
7-tok	GRU PM	0.664	0.125 (77)	0.710	0.08	275	0.551
7-tok	LSTM Van	0.683	0.134 (83)	0.718	0.07	279	0.466
7-tok	LSTM PM	0.668	0.132 (82)	0.710	0.09	276	0.554

Appendix

Table 4: Full internal closed-loop performance. Van

=

vanilla decoding; PM

=

process-masked decoding; Dur.

=

generated over observed remaining duration on terminated rollouts, with 1.00 indicating calibration; LOS

=

prefix-stripped remaining-LOS MAE in minutes (Appendix J ); RD

=

reached-discharge rate. LOS MAE can favor substantially miscalibrated duration, particularly in the 7-token setting. Process masking can improve termination while simultaneously worsening temporal calibration.

	Transformer			GRU
Config	Term.	JSD	Dur.	Term.	JSD	Dur.
Vanilla	0.905	0.091	0.08	0.680	0.126	0.07
Greedy	0.370	0.209	0.09	0.274	0.203	0.05
Temp. 0.7	0.720	0.148	0.13	0.461	0.185	0.16
Temp. 1.3	0.974	0.053	0.05	0.879	0.081	0.04
Top- $p$ 0.9	0.461	0.191	0.16	0.371	0.199	0.27

Appendix

Table 5: Decoding ablation with the 7-token vocabulary on 100k prefixes. Dur. is generated over observed remaining duration on terminated rollouts. No tested decoding rule brings the clock close to 1.00. For the Transformer, repetition penalty 1.75 gives the best termination and JSD of the tested settings but leaves generated duration at only 4% of the observed remaining stay.

	Transformer			GRU
Config	Term.	JSD	Dur.	Term.	JSD	Dur.
Vanilla	0.434	0.310	0.97	0.599	0.073	0.67
Greedy	0.052	0.343	0.01	–	–	–
Temp. 0.7	0.271	0.311	0.83	0.380	0.125	0.68
Temp. 1.3	0.554	0.300	0.92	0.709	0.066	0.48
Top- $p$ 0.9	0.173	0.310	0.18	–	–	–

Appendix

Table 6: Decoding ablation with the 32-token vocabulary on 100k prefixes. Dur. is defined as in Table 5 . For the GRU, each tested intervention that increases termination above the vanilla value also shortens the simulated visit. Count-model blending substantially improves Transformer termination and JSD but does not match the pure count reference.

Vocab	Model	Term.	JSD $\times$ F	Dur.	LOS
32-tok	Transf. small	0.434	0.310 (136)	0.97	356
32-tok	Transf. large	0.751	0.059 (26)	0.41	271
7-tok	Transf. small	0.905	0.091 (56)	0.08	300
7-tok	Transf. large	0.964	0.079 (49)	0.15	301

Appendix

Table 7: Scale ablation under vanilla decoding. Small (=d=128), 2 layers, 4 heads, feed-forward dimension 512 (0.42M parameters); Large (=d=256), 4 layers, 8 heads, feed-forward dimension 1024 (3.25M parameters, approximately

8\times

). The

d=512

, 6-layer all-position model summarized in Section 6.2 has 19.2M parameters (approximately

45\times

the baseline). Dur. and LOS are defined as in Table 4 .

Model	Seed	Term.	JSD $\times$ F	Cap
Transformer	1	0.432	0.314 (137)	0.568
Transformer	2	0.901	0.010 (4.2)	0.099
Transformer	3	0.684	0.138 (61)	0.316
Transformer	4	0.962	0.033 (14)	0.038
Transformer	5	0.930	0.033 (14)	0.070
Transformer	6	0.697	0.122 (54)	0.303

Appendix

Table 8: Per-seed closed-loop metrics with the 32-token vocabulary and vanilla decoding. Model labels follow Table 1 : (prefix)

=

loss at the final prefix position only; (all-pos.)

=

supervision at every eligible position; Decoder

=

decoder-only tokenized-time model; Cap

=

fraction of rollouts reaching the event cap.

Order	JSD	Contexts	Size	Cov.
3	0.00234	7,945	23 MB	0.9999
4	0.00152	48,198	97 MB	0.9966
5	0.00110	154,683	286 MB	0.9829
6	0.00071	340,010	660 MB	0.9506
7	0.00041	585,196	1.27 GB	0.8950

Appendix

Table 9: Order-

k

count models with the 32-token vocabulary on 100k matched prefixes. Contexts

=

distinct training contexts stored; Size

=

serialized table size; Cov.

=

fraction of test contexts observed exactly in training at that order.

Model	JSD	LOS-W1
$n$ -gram-3	0.070	0.63
$n$ -gram-7	0.048	0.65
Semi-Markov	–	0.37
Causal all-position	0.032–0.036	0.09–0.12

Appendix

Table 10: Conditional fidelity on matched late-visit prefixes (completion fraction

\geq 0.75

). JSD

=

suffix event-distribution JSD within this subset; LOS-W1

=

Wasserstein-1 distance in log-minutes between generated and observed remaining-stay distributions . LOS-W1 is a distributional rather than paired per-visit metric and is therefore compatible with the global duration biases reported in Table 1 . Causal-model values span three training seeds.

Model	Marg.	Cond.	Head.	Logit
$n$ -gram-3	0.927	0.962	0.49	0.71
$n$ -gram-7	0.927	0.963	0.49	0.71
Semi-Markov	0.950	0.976	0.52	0.77
Causal all-position	0.967–0.977	0.980–0.987	0.40–0.44	0.53–0.59
Standard Transformer	0.998	0.999	0.78	1.50

Appendix

Table 11: Conditional classifier two-sample test on 100k matched prefixes using a gradient-boosted classifier with grouped 3-fold cross-validation. Marg. and Cond. are classifier AUC using continuation-only and prefix-plus-continuation features, respectively. Head. is the conditional increment normalized by available AUC headroom; Logit is the corresponding increment on the logit scale. Lower corrected increments indicate less additional separability after conditioning on the prefix. Causal-model values span three training seeds.

Arm	$n=1$	$n=2$	$n=3$
$n$ -gram-3 reference	1.0	1.0	1.0
Transformer	136	119	79
GRU	32	46	37
LSTM	37	52	40
$n$ -gram-3 reference	1.0	1.0	1.0
Causal Tf. (all-position)	0.7	1.1	1.4

Appendix

Table 12: Suffix-distribution divergence under increasingly order-sensitive representations. Values are multiples of the order-3 count reference scored at the same sequence order. The standard models use the full test set; all-position and decoder rows use the matched 100k-prefix evaluation.

Arm	Pooled	Per-rollout	First 10
Transformer (prefix)	136	361	4445
GRU (prefix)	32	99	449
LSTM (prefix)	36	83	291
Causal Tf. (all-position)	0.6	2.3	15
LSTM (all-position)	1.0	2.2	15
GRU (all-position)	1.4	3.3	37

Appendix

Table 13: Sensitivity of event-distribution divergence to rollout weighting. Values are multiples of the order-3 count reference scored using the same weighting and the same rollout sample. “Per-rollout” gives each continuation equal weight regardless of length. The first-10 metric is included only as a sensitivity analysis because its count-reference denominator is extremely small.

Vocab	Order	Coverage	# contexts
7-tok	1	1.000000	7
7-tok	2	1.000000	26
7-tok	3	0.999996	86
32-tok	1	1.000000	31
32-tok	2	0.999998	576
32-tok	3	0.999853	7,945

Appendix

Table 14: Exact train-context coverage of test transitions. Contexts are the distinct histories stored in the corresponding training count table.

Metric	Earlier	Corrected
Termination	0.564	0.905
Non-termination	0.436	0.095
Inter-event timing	implementation artifact	artifact removed

Appendix

Table 15: Illustrative earlier versus corrected rollout results for the 7-token Transformer. ‘Earlier” uses the pre-audit generation path; ‘Corrected” uses the common rollout protocol reported in this manuscript.

Arm	$\times$ F	+1h	+2h	+4h	+8h	+12h	MAE
$n$ -gram-3	1	(+1.4)	(+2.9)	(+6.6)	(+8.9)	(+4.5)	4.9
Transformer	136	(+2.2)	(+10.1)	(+26.7)	(+48.5)	(+53.2)	28.1
GRU	32	(-7.9)	(-2.1)	(+12.3)	(+30.0)	(+33.2)	17.1
LSTM	37	(-7.4)	(-5.9)	(+6.2)	(+27.1)	(+34.1)	16.1
$n$ -gram-3	1	(+1.4)	(+2.9)	(+6.9)	(+9.2)	(+4.7)	5.0
Causal all-position	0.7	(-5.8)	(-12.4)	(-18.9)	(-13.3)	(-7.9)	11.7

Appendix

Table 16: Occupancy forecast error in percentage points (predicted minus observed) at five horizons and mean absolute error across horizons. Top: full test set with 1.13M matched launch points. Bottom: matched 100k-prefix comparison of the count reference and two all-position models.

Model	A1	A2	A3	A4	A5	Non-triage
Transformer	0.994	0.996	0.996	0.996	1.000	0.899
GRU	0.536	0.528	0.523	0.523	0.515	0.689
LSTM	0.457	0.444	0.441	0.447	0.412	0.697

Appendix

Table 17: Termination rate with the 7-token vocabulary by launch context under vanilla decoding. A1–A5 correspond to prefixes ending at the triage event with the indicated recorded acuity. “Non-triage” contains all other launch points (

n=1{,}070{,}586

; 94.5% of test prefixes).

Transformer
Recorded group	$n$	Term.	RD	Dur.
Asian	34,611	0.474	0.666	1.130
Black	240,372	0.431	0.599	0.927
Hispanic/Latino	87,113	0.437	0.609	1.024
White	693,761	0.432	0.607	0.944
Not recorded	16,097	0.437	0.709	2.039

Appendix

Table 18: Closed-loop performance with the 32-token vocabulary and vanilla decoding by recorded race.

n

denotes rollout prefixes; Term.

=

termination rate, RD

=

reached-discharge rate, and Dur.

=

within-group duration ratio. “Not recorded” is a documentation category and is shown separately rather than interpreted as a demographic group.

Quantity	7-token	32-token
Real vs. real	$4.5\times 10^{-6}$	$1.0\times 10^{-5}$
Population vs. subsample	$1.5\times 10^{-6}$	$4.2\times 10^{-6}$
$n$ -gram-3 reference	0.0016	0.0023
Transformer (vanilla)	0.091	0.310

Appendix

Table 19: Reference values for suffix event-distribution JSD. Real vs. real compares two independent 100k samples of observed test suffixes and reports the mean across five trials. “Neural” is the standard vanilla Transformer.

Vocab	Model	Cap 128	Cap 256
7-token	Transformer	0.905	0.979
7-token	GRU	0.680	0.815
7-token	LSTM	0.683	0.801
32-token	Transformer	0.434	0.447

Appendix

Table 20: Termination under 128- and 256-event caps with vanilla decoding.

Figure 2: The simulation gap. Open-loop next-event accuracy (

x

-axis) versus closed-loop suffix event-distribution JSD (log

y

-axis) for the three standard architectures. With the 32-token vocabulary, nearly identical open-loop accuracy corresponds to substantially different closed-loop divergence. Open markers denote the 7-token vocabulary and filled markers the 32-token vocabulary; the order-3 count reference is dashed.

Figure 3: Per-step generated-versus-observed token JSD. Left, 7-token vocabulary: the standard models begin near the observed distribution and diverge with rollout depth. Right, 32-token vocabulary: the Transformer shows elevated first-step miscalibration , whereas the recurrent models begin closer to the observed distribution and exhibit compounding drift . The order-3

n

-gram reference (dashed) also accumulates some divergence with rollout depth but remains lowest.

Figure 4: The grammar–statistics split (Section 5.2 ). Decoding, masking, and tuning interventions are shown in the termination–suffix-JSD plane for both event vocabularies. Colors identify intervention families and marker shapes identify model architectures. Several interventions improve termination and, particularly with the 7-token vocabulary, reduce event-distribution divergence, but none simultaneously reaches near-complete termination and the order-3 reference’s composition. Temporal calibration is evaluated separately in Table 4 .

Vocab	Model	Run	$\geq 10$	Uniq	Tail
7-token	Transformer	9.0	0.40	0.43	0.011
7-token	GRU	11.6	0.57	0.34	0.038
7-token	LSTM	12.5	0.57	0.34	0.060
32-token	Transformer	4.4	0.05	0.28	0.007
32-token	GRU	6.9	0.24	0.30	0.025
32-token	LSTM	5.3	0.09	0.32	0.003

Appendix

Table 21: Repeated- and low-diversity-generation diagnostics under vanilla decoding on the full test split. Run

=

mean longest identical-token run per rollout;

\geq 10

=

fraction containing an identical-token run of at least ten events; Uniq

=

number of distinct generated tokens divided by suffix length; Tail

=

fraction whose final ten generated events are identical.

Model	Decoding	Term.	JSD $\times$ F	RD
$n$ -gram-3 reference	–	0.991	0.00416 (1)	–
Transformer	Vanilla	0.000	0.0236 (5.7)	0.000
Transformer	Process mask	0.011	0.0206 (4.9)	0.000
GRU	Vanilla	0.000	0.0713 (17)	0.000
GRU	Process mask	0.011	0.0649 (16)	0.000
Causal all-pos. s1	Vanilla	0.945	0.0103 (2.5)	0.934

Appendix

Table 22: MC-MED replication on 100k prefixes with a 512-event cap.

\times

F is normalized to the train-only MC-MED order-3

n

-gram reference (JSD 0.0041649), not to the MIMIC denominator. Standard-model JSD is computed over capped, non-terminating rollouts and is therefore not length-matched to the terminating rows. RD

=

reached-discharge rate. Only the Transformer and GRU standard architectures were retrained on MC-MED.

Trajectory Prediction

Trajectory prediction forecasts how people, vehicles, robots, and other dynamic systems will move from observed histories and context.

See all 186 papers

Oct 2, 2026

Forecasting from Counterfactual Simulator Rollouts: A Sim2Real Evaluation

Deploying a new decision policy creates a cold-start problem for prediction models whose targets depend on the policy's actions: historical observations reflect earlier policies, while real observations under the new policy are not yet available. Simulation offers a way to address this gap by rolling out the target policy across counterfactual scenarios and using the resulting trajectories to learn how the system responds to those controls. The simulation-to-reality (Sim2Real) transfer of this simulator-trained model can then be backtested by evaluating it against real observations from past deployments. Using two real-world inventory-control deployments, we evaluate this process from three angles: simulator fidelity, zero-shot transfer to real behavior, and adaptation as real target-policy observations accumulate. The simulator-trained forecaster achieves lower point-estimate mean absolute percentage error (MAPE) than the same architecture trained on historical real data, reducing MAPE by 1.2-3.1 percentage points in Study 1 and 12.5-18.7 points in Study 2. After deployment, lightweight calibration using early real observations further reduces error by up to 2.5 percentage points. These results provide empirical evidence that simulator-generated counterfactual data can support cold-start forecasting under a new policy, and the resulting model can be further refined as real deployment data become available.

Angel Wang, Dominique Perrault-Joncas, Alvaro Maggiar +2

Amazon

Sim-to-Real Transfer Time Series Forecasting Counterfactual Prediction

Ask PDF

May 14, 2026

DT-Transformer: A Foundation Model for Disease Trajectory Prediction on a Real-world Health System

Accurate disease trajectory prediction is critical for early intervention, resource allocation, and improving long-term outcomes. While electronic health records (EHRs) provide a rich longitudinal view of patient health in clinical environments, models trained on curated research cohorts may not reflect routine deployment settings, and those trained on single-hospital datasets capture only fragments of each patient's trajectory. This highlights the importance of leveraging large, multi-hospital health systems for training and validation to better reflect real-world clinical complexity. In this work, we develop DT-Transformer, a foundation model trained on 57.1M structured EHR entries over 1.7M patients from Mass General Brigham (MGB), spanning 11 hospitals and a broad network of outpatient clinics. DT-Transformer achieves strong discrimination in both held-out and prospective validation settings. Next-event prediction achieves a median age- and sex-stratified AUC of 0.871 across 896 disease categories, with all categories exceeding AUC 0.5. These results support health system-scale training as a path toward foundation models suited to real-world clinical forecasting.

Yunying Zhu, Andrew R Weckstein, Kueiyu Joshua Lin +1

Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, USA · Harvard T.H. Chan School of Public Health, Harvard University, Boston, MA, USA · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University, MA, USA +2

Clinical Prediction Time Series Transformers Time-Series Foundation Models Healthcare Longitudinal EHR Modeling EHR Foundation Models+1

Ask PDF

Oct 12, 2026

Sequence Composition in EHR Foundation Model Pretraining

Autoregressive electronic health record (EHR) foundation models predict the next event token in patient trajectories: sequences of structured medical events. A typical pretraining strategy is to concatenate all trajectories into a global stream and sample token windows from it. This gives patients with longer sequences more training exposure and allows tokens to attend across patient boundaries. We argue that these choices create a misalignment between pretraining and downstream clinical tasks. To explore this, we investigate how the composition of pretraining sequences affects clinical prediction performance. Using the global-stream approach as a baseline, we propose alternative sequence composition methods that control how patient trajectories contribute to pretraining sequences. Across four MIMIC-IV v2.2 tasks, our best method improves macro AUROC, AUPRC, F1, and balanced accuracy, with relative gains of 9.3% in F1 and 2.7% in AUPRC under a fixed training recipe. We find that these gains generalize across four additional evaluation settings, including the INSPIRE and eICU-CRD datasets, with improvements in macro AUROC, AUPRC, and F1 score. Our results show that simply changing how EHR data are presented to the model during pretraining can improve downstream performance, without changing the model or collecting additional data.

Joshua Placidi, Inigo Stibbe, Alfred J. Balston +2

Department of Computing, Imperial College London

EHR Foundation Models Electronic Health Records Longitudinal EHR Modeling Self-Supervised Pre-Training Clinical Outcome Prediction

Ask PDF

Abstract

Trajectory Prediction

Explore similar work

Forecasting from Counterfactual Simulator Rollouts: A Sim2Real Evaluation

DT-Transformer: A Foundation Model for Disease Trajectory Prediction on a Real-world Health System

Sequence Composition in EHR Foundation Model Pretraining