Autoregressive (AR) models have proved effective for zero-shot prediction from longitudinal electronic health records, but impose strictly sequential generation on trajectories containing temporally clustered and interdependent clinical events. We investigate block-based masked diffusion (BD) as an alternative on MIMIC-IV under matched tokenisation, context length, model capacity, and evaluation conditions. Across block sizes, BD exhibits a clear quality--efficiency trade-off, with small-block configurations retaining much of the predictive performance of AR while enabling substantially faster inference. A representative B=8 configuration achieves approximately 1.43× faster evaluation. Although generative rollout remains below AR in most settings, frozen linear probes on BD and AR representations achieve near-identical performance across clinical prediction tasks, indicating that BD learns comparably informative clinical representations. Increasing the denoising budget yields diminishing returns, while alternative decoding strategies provide limited gains. Overall, our results show that block diffusion is a viable and computationally attractive approach to longitudinal EHR modelling, with further improvements in iterative decoding offering a promising route toward closing the remaining generative performance gap.
Figures & tables
Figure 1: Quality, efficiency, and representation trade-offs of block diffusion. Top-left: BD–AR AUROC difference under rollout and linear probing. Top-right: AUROC across denoising budgets. Bottom: ICU mortality AUROC and evaluation time across block sizes at S/B=0.5 . Dashed lines denote AR.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Illustration of AR and block-diffusion decoding for longitudinal EHR trajectories. Clinical events frequently occur in temporally clustered groups. AR generates each event sequentially, whereas BD preserves causal generation across blocks while jointly denoising multiple positions within the active block.
Figure 3: Clinical event multiplicity in MIMIC-IV. Survival distribution of the number of non-time-gap event tokens sharing a recorded timestamp in the ETHOS representation.
Statistic
Value
Number of timestamps
23 274 367
Mean events per timestamp
16.16
Median
6
25th / 75th percentile
3 / 20
P(N>1)
96.8%
P(N≥4)
67.6%
Appendix
Table 1: Multiplicity of clinical event tokens sharing a timestamp in the MIMIC-IV / ETHOS training set. N is the number of non-time-gap event tokens at a timestamp.
AR
BD ( B=8 , S=4 )
AUROC
0.844[0.831,0.858]
0.883[0.873,0.892]
AUPRC
0.306[0.276,0.335]
0.204[0.183,0.227]
Precision@100
0.74
0.47
Mean score (deaths / survivors)
0.242 / 0.015
0.353 / 0.090
Score =0 (deaths / survivors)
23.7% / 86.7%
2.5% / 39.1%
Appendix
Table 2: Hospital-mortality rollout scores on the test set. AUROC/AUPRC brackets are 95% instance-bootstrap CIs. Score statistics are Monte Carlo P(MEDS_DEATH) over 20 samples.
Figure 4: Predictive performance versus evaluation speed on ICU mortality. Each point denotes a block-diffusion configuration with a particular block size and denoising budget. The x-axis shows evaluation speedup relative to the autoregressive (AR) baseline, while the y-axis shows AUROC. The highlighted B=8 , S=4 configuration provides a representative operating point, achieving approximately 1.43× faster evaluation while retaining much of the predictive performance of AR.
Figure 5: Left: Difference in AUPRC between block diffusion and AR under generative rollout and frozen linear probing across four clinical tasks. Middle: ICU mortality AUPRC as block size varies under the fixed normalised denoising budget S/B=0.5 . Dashed lines denote the AR baseline. Shaded regions and error bars indicate 95% bootstrap confidence intervals. Right: ICU mortality AUPRC as the normalized denoising budget S/B increases for fixed block sizes.
Strategy
AUROC
AUPRC
Random
0.8476 [0.8331, 0.8614]
0.3440 [0.3123, 0.3792]
ReMDM η=0.05
0.8421 [0.8268, 0.8562]
0.3338 [0.3028, 0.3679]
ReMDM η=0.10
0.8446 [0.8307, 0.8590]
0.3515 [0.3176, 0.3871]
ReMDM η=0.20
0.8433 [0.8287, 0.8580]
0.3474 [0.3158, 0.3800]
Confidence
0.8089 [0.7921, 0.8257]
0.3487 [0.3154, 0.3844]
Appendix
Table 3: ICU mortality decoding strategies for block size B=8 and S=4 diffusion steps. Metrics are medians over completed full-validation runs where available; Confidence uses the available full split run. Brackets are 95% CIs over the three random seeds.
Longitudinal electronic health record modeling learns from patient histories over time to forecast outcomes, characterize disease trajectories, and guide care.
Is the uniform-state diffusion framework a more powerful paradigm for discrete diffusion? Recent studies indicate that this may be the case. In combination with predictor-corrector samplers, uniform-state diffusion models (USDMs) produce samples of higher-quality than masked diffusion models (MDMs), and USDMs equal or outperform MDMs in downstream tasks, even though they exhibit greater perplexity. Two issues remain unresolved. First, existing work compares uniform and masked diffusion with un-informed correctors that re-inject noise at random positions, rather than targeting tokens most likely to be wrong. Second, prior work compares full-sequence diffusion models, so we do not know whether the same conclusion holds when tokens are generated block by block. To address these issues, we introduce BlockGen, a blockwise sequence model that we instantiate with both masked and uniform diffusion. BlockGen trains on a mixture of block sizes and its likelihood interpolates between AR and pure diffusion more finely than models with a fixed block size. BlockGen enables AR-informed predictor-corrector sampling (ARPC), which combines AR and diffusion predictions to re-generate unlikely tokens without an auxiliary verifier. Under ancestral sampling, uniform outperforms masked in the block-by-block setting, especially in the few-step regime. Under ARPC, the gap closes and reverses at high NFE. With block size 16 on GSM8K, MDMs reach slightly higher accuracy than USDMs, and we observe a similar trend in Generative Perplexity on OpenWebText. Find our code at https://github.com/jdeschena/blockgen.
Justin Deschenaux, Caglar Gulcehre
EPFL Lausanne, Switzerland · EPFL, Lausanne, Switzerland · Microsoft AI
Diffusion language models enable flexible arbitrary-order generation, but existing sampling methods are mostly designed for early masked diffusion models (MDMs). In this work, we study sampling for recent block diffusion language models (BDLMs). We show empirically and analytically that these models are naturally more aligned with left-to-right decoding than MDMs. Based on this observation, we propose Parallel Autoregressive Decoding (PARD), a simple training-free sampling method that preserves left-to-right unmasking structure while allowing parallel token commitment. Extensive experiments show that PARD consistently outperforms existing parallel samplers in generation quality, while achieving substantial speedups over pure AR decoding with only a small quality gap.
Kai Syun Hou, James Kwok
Department of Computer Science and Engineering The Hong Kong University of Science and Technology
Masked language modelling (MLM) is the dominant pretraining objective for transformer encoders applied to longitudinal electronic health records (EHRs). However, reconstructing individual events may overemphasise noisy or weakly informative observations and does not explicitly encourage learning higher-level patient states. Joint-Embedding Predictive Architectures (JEPAs) instead learn by predicting representations in latent space and have shown strong performance in a variety of domains. In this paper, we investigate to what extent JEPA can be adapted to EHR sequence representation. Our empirical investigation using MIMIC-IV shows that although all three of our JEPA formulations train stably without representational collapse, they consistently underperform standard MLM. We find that target size, not target heterogeneity, increases prediction error, yet reducing the target size does not recover downstream performance. This suggests that stable latent prediction alone is insufficient for competitive EHR representation learning and highlights the challenge of defining clinically meaningful latent targets for EHR.