ViSTA: A Simple Bridge Extends Visual Alignment to Clinical Time-Series Understanding in Multimodal LLMs

Junyi Gao; Yu Shi; Pingzhao Hu; Ewen M Harrison

cs.CLSep 25, 2026

ViSTA: A Simple Bridge Extends Visual Alignment to Clinical Time-Series Understanding in Multimodal LLMs

Authors: Junyi Gao, Yu Shi, Pingzhao Hu, Ewen M Harrison

Organizations: Centre for Medical Informatics, University of Edinburgh · Health Data Research UK · Biostatistics Division, Dalla Lana School of Public Health, University of Toronto · Department of Biochemistry, Western University

Abstract

Clinical prediction models estimate risk from patient measurements, while large language models support medical text understanding and question answering. Yet their language capabilities do not ensure accurate prediction from structured, high-dimensional clinical time series. Improving this ability would connect risk estimation with flexible questions about a patient's evolving condition. We introduce ViSTA, a compact adapter that incorporates irregular numerical measurements into a pretrained vision-language model's chart representations. It learns corrections to visual tokens while leaving all pretrained parameters unchanged. On MIMIC-IV, ViSTA has the highest mean scores among the compared adaptations on all four metrics for acute kidney injury and mortality prediction across models with 2-9 billion parameters. With 0.516 million trainable parameters, the 2-billion-parameter model reaches an area under the ROC curve of 0.7376 for acute kidney injury, compared with GPT-5.6 Sol's 0.7380 with text input and high reasoning effort. Training for temporal question answering yields 69.27% accuracy at 4 billion parameters with over 90% fewer trainable parameters than low-rank adaptation using charts or numerical text, at a 2.82-4.88 percentage-point accuracy gap. ViSTA extends pretrained language models to numerical prediction and temporal questions.

Figures & tables

Figure 1: Numerical correction of an established visual interface. Direct numerical interfaces learn how to present a new continuous representation to a language model. Chart inputs use its pretrained visual pathway. ViSTA adds numerical corrections to original visual tokens at their existing positions, preserving token order and positional indices. The language model and its output head remain frozen.

Figure 2: ViSTA architecture. An event encoder pools irregular numerical observations into latent summaries. Queries derived from visual grid positions and variable identity read these summaries and produce residuals with the same shape as the original visual tokens. The residual output projection starts at zero. Answer supervision propagates through the frozen language model to the event encoder and bridge.

Method	Update	Trainable params (M)
Unadapted text	None	0
Unadapted plots	None	0
Chart LoRA	LoRA	6.193
Text LoRA	LoRA	7.032
ChatTS	LoRA	6.692
MLLM4TS	Layer norm	0.884

Table 1: Comparison methods and adaptation size. Update describes adaptation within the language decoder. Parameters are trainable counts at 4B. ViSTA uses 0.516M/0.582M/0.780M at 2B/4B/9B.

Proprietary references
	AKI		Mortality		QA
Method / input	AUROC	AP	AUROC	AP	Acc. (%)
GPT 5.6 Sol-high / text	0.7380 $\pm 0.0293$	0.4212 $\pm 0.0246$	0.7515 $\pm 0.0231$	0.2935 $\pm 0.0254$	-
GPT 5.6 Sol-high / plots	0.7283 $\pm 0.0292$	0.4126 $\pm 0.0330$	0.7221 $\pm 0.0291$	0.2733 $\pm 0.0285$	-
GPT 5.6 Luna-high / text	0.7422 $\pm 0.0297$	0.4382 $\pm 0.0312$	0.6870 $\pm 0.0306$	0.2780 $\pm 0.0324$	-
GPT 5.6 Luna-high / plots	0.6689 $\pm 0.0284$	0.3358 $\pm 0.0342$	0.6684 $\pm 0.0306$	0.2605 $\pm 0.0368$	-

Table 2: Prediction and QA performance after outcome training. Values are mean

\pm

patient-bootstrap SD; bold and underline mark the best and second-best Qwen means per scale and metric. Adapted Qwen models use outcome supervision only. GPT models receive no task-specific training, and their QA scores appear in Table 3 .

GPT reference / input	QA without task-specific training (%)
GPT 5.6 Sol-high / text	90.51 $\pm 0.70$
GPT 5.6 Sol-high / plots	84.24 $\pm 0.79$
GPT 5.6 Luna-high / text	80.30 $\pm 0.93$
GPT 5.6 Luna-high / plots	74.05 $\pm 1.13$
Method	2B		4B		9B
	Acc. (%)	$\Delta$ (pp)	Acc. (%)	$\Delta$ (pp)	Acc. (%)	$\Delta$ (pp)

Table 3: Large QA gains with a compact frozen interface. Accuracy (%) is mean

\pm

patient-bootstrap SD;

\Delta

is the gain over the unadapted chart model in percentage points. GPT models receive no task-specific training. Qwen adaptations are trained on QA; bold and underline mark the best and second-best Qwen means per scale.

Figure 3: Prediction gains persist as the backbone grows. Adapted models use outcome supervision only; the right panel evaluates QA without QA training. Error bars show patient-bootstrap SD. Both encodings have identical parameter counts at each scale.

Figure 4: Supervised QA gains across most categories. At 4B, ViSTA -Separate is compared with the category-wise best of ChatTS, MLLM4TS, ITFormer, and OpenTSLM. Error bars show paired patient-bootstrap SD with baseline selection repeated per replicate. Absolute scores appear in Appendix G .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Answer construction
TG: Event localization	Locate the start of the first consecutive pair of observed threshold breaches within one of four quarters of a time window.
ASR: Cross-variable retrieval	Find an extremum of one variable and retrieve the nearest observation of another; resolve time ties using the earlier reading.
TPR: Segmented trends	Split ordered observations into five nearly equal blocks, compare medians using a stated tolerance, and merge adjacent repeated directions.
MA: Missingness counting	Count a variable’s observed and absent entries on the union of recorded timestamps in a specified window.
TSS: Numerical summaries	Select the correct first, last, minimum, and maximum values for two variables from summaries with numerical distractors.
CVR: Ordering of changes	Compare the first specified directional changes of two variables, including simultaneous and absent-change cases.

Appendix

Table 4: Temporal QA definitions. Each question specifies its observation window, thresholds, and tolerances.

Setting	Value
Clinical variables $C$ ; slots per variable	11; 12
Total internal slots $K$	132
Visual tokens per chart; total $N$	192; 2,112
Numerical optimizer	AdamW, learning rate $10^{-4}$
Weight decay; gradient norm ceiling	0.01; 1.0
Numerical precision	BF16 backbone

Appendix

Table 5: Selected ViSTA interface and training configuration. Settings are selected on the validation set.

	Prediction $\Delta$ AUROC		QA after prediction (pp)		Supervised QA $\Delta$ (pp)
Scale	AKI	Mortality	vs. plots	vs. text	ViSTA -Joint	ViSTA -Separate
2B	+0.2079	+0.1754	+3.00	+0.42	+32.78	+33.28
4B	+0.1190	+0.1478	+6.23	+2.33	+32.77	+33.77
9B	+0.2298	+0.1916	+5.91	+4.00	+25.46	+24.23

Appendix

Table 6: Prediction and QA gains over the unadapted models. Prediction AUROC changes use the unadapted chart baseline as the reference. QA after prediction training is compared with both unadapted input baselines. These columns use ViSTA -Joint; supervised QA gains use their respective encodings and are relative to the unadapted chart baseline.

Scale	AKI AP	Mortality AP	QA after prediction (pp)	Supervised QA (pp)
2B	+0.0099	+0.0344	+0.09	-0.50
4B	+0.0218	+0.0259	+0.09	-1.00
9B	+0.0213	+0.0375	-0.73	+1.23

Appendix

Table 7: ViSTA -Joint minus ViSTA -Separate encoding. Positive entries favor ViSTA -Joint. Prediction and supervised QA use separately trained adapters. The first QA column evaluates the prediction-trained adapter without further training.

	AKI		Mortality		Transfer QA
Method / input	AUROC	AP	AUROC	AP	Acc. (%)
ViSTA : full $V+R$	0.7440	0.4650	0.7350	0.2651	41.73
Inference-time input interventions
Visual only: $V$	0.6250	0.2795	0.5872	0.1493	35.50
Residual only: $R$	0.5418	0.2216	0.5690	0.1332	29.95
Training ablations

Appendix

Table 8: Input interventions and design ablations at 4B. The full model is ViSTA -Joint from Table 2 . Here

R=s\odot\Delta

denotes the scaled numerical residual. Visual-only and residual-only inputs are evaluated at inference without retraining. The last two rows remove the indicated component during training. QA is evaluated without QA supervision; higher is better for every metric.

Category	Best baseline (%)	ViSTA -Joint (%)	ViSTA -Separate (%)	$\Delta$ (pp)
ASR: Cross-variable retrieval	$41.50\pm 3.29$	$52.00\pm 3.49$	$53.00\pm 3.47$	$11.50\pm 4.47$
CVR: Ordering of changes	$68.50\pm 3.25$	$63.00\pm 3.42$	$62.50\pm 3.42$	$-6.00\pm 2.67$
IID: Next intervention	$83.50\pm 2.44$	$78.00\pm 2.96$	$80.50\pm 2.87$	$-3.00\pm 2.27$
IR: Intervention response	$59.50\pm 3.40$	$64.50\pm 3.39$	$62.00\pm 3.40$	$2.50\pm 4.30$
MA: Missingness counting	$59.00\pm 3.19$	$54.00\pm 3.52$	$63.00\pm 3.41$	$4.00\pm 4.30$
MED: Monitoring rules	$75.50\pm 3.07$	$76.00\pm 3.04$	$75.50\pm 3.07$	$0.00\pm 0.00$

Appendix

Table 9: Supervised QA gains over the best baseline in each category at 4B. All methods are trained on QA. The reference is the maximum of ChatTS, MLLM4TS, ITFormer, and OpenTSLM in each category.

\Delta

is ViSTA -Separate minus this reference, in percentage points. Values include patient-bootstrap SD.

Category	Test $n$	Majority letter	Majority (%)	Random (%)
Event localization	200	A	27.00	25.00
Cross-variable retrieval	200	A	28.50	25.00
Segmented trends	200	A	21.00	25.00
Missingness counting	200	A	29.50	25.00
Numerical summaries	200	D	25.50	25.00
Ordering of changes	200	C	30.50	25.00

Appendix

Table 10: QA baselines without patient information. Majority selects each category’s most frequent training answer letter and reports test accuracy. Random is the exact expectation of uniform four-option guessing. All accuracies are percentages.

	AKI		Mortality
Method	AUROC	AP	AUROC	AP
LSTM	$0.7793\pm 0.0178$	$0.5244\pm 0.0354$	$0.7780\pm 0.0212$	$0.3357\pm 0.0473$
GRU	$0.7761\pm 0.0183$	$0.5170\pm 0.0341$	$0.7728\pm 0.0233$	$0.3248\pm 0.0474$
Transformer	$0.7378\pm 0.0183$	$0.4400\pm 0.0340$	$0.7288\pm 0.0256$	$0.2997\pm 0.0465$
ConCare	$0.7946\pm 0.0175$	$0.5367\pm 0.0331$	$0.7881\pm 0.0218$	$0.3505\pm 0.0484$
VITA	$0.8006\pm 0.0170$	$0.5429\pm 0.0354$	$0.7969\pm 0.0211$	$0.3569\pm 0.0485$

Appendix

Table 11: Specialized clinical prediction references. Reported AUROC/AP

\pm

SD. These systems have no QA evaluation in this study.

ECE $\downarrow$
	AKI			Mortality
Scale	Unadapted plots	ViSTA -Joint	ViSTA -Separate	Unadapted plots	ViSTA -Joint	ViSTA -Separate
2B	0.1965	0.0468	0.0444	0.4962	0.0328	0.0338
4B	0.0378	0.0437	0.0614	0.3860	0.0383	0.0373
9B	0.2418	0.0820	0.0511	0.7261	0.0666	0.0284
Brier score $\downarrow$

Appendix

Table 12: Numerical adaptation improves prediction probability quality. Probabilities are evaluated without recalibration; lower is better. Bold marks the minimum for each model size, outcome, and metric. Table 13 reports both unadapted input baselines, including patient-bootstrap SD.

		AKI		Mortality
Scale	Method	ECE $\downarrow$	Brier $\downarrow$	ECE $\downarrow$	Brier $\downarrow$
2B	Unadapted text	0.3462 $\pm 0.0133$	0.2895 $\pm 0.0021$	0.5665 $\pm 0.0096$	0.4214 $\pm 0.0044$
	Unadapted plots	0.1965 $\pm 0.0130$	0.2026 $\pm 0.0030$	0.4962 $\pm 0.0097$	0.3482 $\pm 0.0035$
4B	Unadapted text	0.0861 $\pm 0.0124$	0.1780 $\pm 0.0059$	0.1300 $\pm 0.0095$	0.1127 $\pm 0.0048$
	Unadapted plots	0.0378 $\pm 0.0111$	0.1590 $\pm 0.0080$	0.3860 $\pm 0.0100$	0.2537 $\pm 0.0040$
9B	Unadapted text	0.4843 $\pm 0.0145$	0.2990 $\pm 0.0033$	0.4502 $\pm 0.0092$	0.4586 $\pm 0.0088$

Appendix

Table 13: Calibration of unadapted models with text and chart inputs. Values are mean

\pm

patient-bootstrap SD. ECE uses ten equal-width probability bins. Bold marks the smallest mean within each scale and metric.

Explore similar work

Training Large Language Models to Predict Clinical Events
May 12, 2026Benjamin Turtel, Paul Wilczewski, Kris SkotheimHealthcare Language Model Calibration
Lightning Rod Labs
LiteMedCoT-VL: Parameter-Efficient Adaptation for Medical Visual Question Answering
May 10, 2026Runze Ma, Shunbo Jia, Haonan Lyu +2CoT Distillation Healthcare
School of Information Technology Monash University Malaysia · Faculty of Innovation Engineering Macau University of Science and Technology · Department of Bioelectronics Faculty of Biomedical Engineering Shenzhen University of Advanced Technology +1
Revitalizing Medical Time Series with Vision-Informed Retrieval: A Vision-Language Perspective
Sep 28, 2026Guoqi Yu, Juncheng Wang, Shujun WangMultivariate Time Series Classification Time Series Classification
Department of Biomedical Engineering and Sports Technology The Hong Kong Polytechnic University

May 12, 2026, cs.LGJ/K move · Enter open · S save

Training Large Language Models to Predict Clinical Events

Benjamin Turtel, Paul Wilczewski, Kris Skotheim

Lightning Rod Labs

Healthcare Language Model Calibration Clinical Prediction Longitudinal EHR Modeling Electronic Health Records Medical LLMs+1

Open paper AskPDF arXiv

Longitudinal clinical notes contain rich evidence of how patients evolve over time, but converting this signal into training supervision for clinical prediction remains challenging. We extend Foresight Learning to clinical prediction by converting time-ordered MIMIC-III notes into examples consisting of past patient context, a natural-language question about a possible future event, and a label resolved from later documentation. This process yields 6,900 prediction examples from 702 admissions across medications, procedures, organ support, microbiology, and mortality. A small LoRA adapter trained on these examples improves over the prompted base model, reducing expected calibration error from 0.1269 to 0.0398 and Brier score from 0.199 to 0.145, while slightly outperforming GPT-5 point estimates on held-out questions. The approach enables reusable clinical prediction supervision from longitudinal notes without hand-engineered structured features or endpoint-specific classifiers.

Abstract

Explore similar work

Training Large Language Models to Predict Clinical Events

LiteMedCoT-VL: Parameter-Efficient Adaptation for Medical Visual Question Answering

Revitalizing Medical Time Series with Vision-Informed Retrieval: A Vision-Language Perspective