Revitalizing Medical Time Series with Vision-Informed Retrieval: A Vision-Language Perspective

Guoqi Yu; Juncheng Wang; Shujun Wang

cs.CVSep 28, 2026

Revitalizing Medical Time Series with Vision-Informed Retrieval: A Vision-Language Perspective

Authors: Guoqi Yu, Juncheng Wang, Shujun Wang

Organizations: Department of Biomedical Engineering and Sports Technology The Hong Kong Polytechnic University

Abstract

Medical time series (MedTS) underpin many clinical classification tasks, yet existing methods usually represent them only as numerical sequences and underuse the morphology that is explicit in waveform inspection. To bridge this gap, we introduce Vision-Informed Retrieval (ViRe), which uses a frozen VLM-derived waveform representation as a morphology-aware Query to guide retrieval from raw numerical MedTS features. Specifically, a Vision Query is extracted using pre-trained vision-language models (VLMs) to obtain morphology-aware priors from waveform plots. A tailored attention-based cross-modal retrieval mechanism then uses the Vision Query to select morphology-relevant temporal and channel evidence from the numerical representation. ViRe demonstrates strong effectiveness against ten established baselines, yielding an overall 6.42% relative improvement over the previous state of the art across six public benchmarks. Code, training scripts, and reproducibility materials are publicly available in the GitHub Repository: https://github.com/Levi-Ackman/ViRe.

Figures & tables

Figure 2: Overview of ViRe. Raw MedTS is embedded into Temporal and Channel tokens, while its waveform visualization is processed by a frozen CLIP Vision Encoder to produce a global Vision Query . Cross-attention Encoders retrieve vision-guided temporal and channel summaries, which are fused and projected to produce the final diagnostic prediction.

	ADFTD		APAVA		PTB		MIMIC
Query Type	Accuracy	F1-Score	Accuracy	F1-Score	Accuracy	F1-Score	Accuracy	F1-Score	Avg. Gain
w/o	54.79 $\pm$ 1.81	51.73 $\pm$ 2.22	83.86 $\pm$ 2.87	82.45 $\pm$ 3.69	81.45 $\pm$ 3.44	74.58 $\pm$ 6.89	84.92 $\pm$ 1.11	84.81 $\pm$ 1.12	–
Zero	54.72 $\pm$ 2.82	51.23 $\pm$ 2.72	84.34 $\pm$ 1.72	83.73 $\pm$ 1.96	84.62 $\pm$ 2.55	80.21 $\pm$ 3.96	86.13 $\pm$ 0.14	86.06 $\pm$ 0.13	1.92%
Gaussian	54.46 $\pm$ 2.21	51.54 $\pm$ 1.93	82.89 $\pm$ 2.66	81.65 $\pm$ 3.12	82.83 $\pm$ 2.15	77.34 $\pm$ 3.64	86.50 $\pm$ 0.11	86.44 $\pm$ 0.11	0.76%
Vision	57.83 $\pm$ 1.82	54.02 $\pm$ 2.22	91.43 $\pm$ 1.03	91.19 $\pm$ 1.03	88.26 $\pm$ 1.12	85.81 $\pm$ 1.52	88.61 $\pm$ 0.12	88.54 $\pm$ 0.12	7.72%

Table 3: Ablation of the retrieval Query. Zero replaces the Vision Query with an all-zero vector, and Gaussian replaces it with a vector sampled from a Gaussian distribution.

	ADFTD		APAVA		PTB		MIMIC
Fusion Strategy	Accuracy	F1-Score	Accuracy	F1-Score	Accuracy	F1-Score	Accuracy	F1-Score	Avg. Gain
w/o	54.79 $\pm$ 1.81	51.73 $\pm$ 2.22	83.86 $\pm$ 2.87	82.45 $\pm$ 3.69	81.45 $\pm$ 3.44	74.58 $\pm$ 6.89	84.92 $\pm$ 1.11	84.81 $\pm$ 1.12	–
Add	56.67 $\pm$ 1.46	53.53 $\pm$ 2.12	85.95 $\pm$ 1.77	85.45 $\pm$ 1.71	84.81 $\pm$ 2.17	81.26 $\pm$ 3.47	87.57 $\pm$ 0.06	87.49 $\pm$ 0.07	4.05%
Concat	56.86 $\pm$ 1.54	53.55 $\pm$ 1.21	84.23 $\pm$ 2.55	83.02 $\pm$ 3.11	83.60 $\pm$ 2.25	79.13 $\pm$ 3.47	87.88 $\pm$ 0.18	87.81 $\pm$ 0.18	3.02%
Retrieval	57.83 $\pm$ 1.82	54.02 $\pm$ 2.22	91.43 $\pm$ 1.03	91.19 $\pm$ 1.03	88.26 $\pm$ 1.12	85.81 $\pm$ 1.52	88.61 $\pm$ 0.12	88.54 $\pm$ 0.12	7.72%

Table 4: Ablation of modality fusion. Cross-attention retrieval is compared with simple addition ( Add ) and concatenation ( Concat ) of the vision prior and the numerical features.

	ADFTD		APAVA		PTB		MIMIC
Vision Backbone	Accuracy	F1-Score	Accuracy	F1-Score	Accuracy	F1-Score	Accuracy	F1-Score	Avg. Gain
w/o	54.79 $\pm$ 1.81	51.73 $\pm$ 2.22	83.86 $\pm$ 2.87	82.45 $\pm$ 3.69	81.45 $\pm$ 3.44	74.58 $\pm$ 6.89	84.92 $\pm$ 1.11	84.81 $\pm$ 1.12	–
ViT (Random Init)	37.39 $\pm$ 3.11	27.34 $\pm$ 1.92	80.62 $\pm$ 0.60	77.18 $\pm$ 0.54	80.65 $\pm$ 1.68	73.50 $\pm$ 2.79	83.67 $\pm$ 0.14	83.58 $\pm$ 0.13	-11.81%
ViT (ImageNet)	52.21 $\pm$ 0.73	50.55 $\pm$ 1.13	84.48 $\pm$ 2.80	84.86 $\pm$ 3.67	82.06 $\pm$ 3.54	75.72 $\pm$ 5.83	86.52 $\pm$ 0.21	86.42 $\pm$ 0.21	0.34%
CLIP-Vision	57.83 $\pm$ 1.82	54.02 $\pm$ 2.22	91.43 $\pm$ 1.03	91.19 $\pm$ 1.03	88.26 $\pm$ 1.12	85.81 $\pm$ 1.52	88.61 $\pm$ 0.12	88.54 $\pm$ 0.12	7.72%

Table 5: Ablation of the vision backbone: Random ViT, ImageNet ViT, and CLIP-Vision.

Figure 3: F1-Score with and without vision priors across training-set proportions.

	ADFTD				PTB				MIMIC
Metrics/Features	Vision	Zero	Gaussian	Raw	Vision	Zero	Gaussian	Raw	Vision	Zero	Gaussian	Raw
DBI $\downarrow$	4.034	7.962	18.098	9.607	1.156	1.466	1.722	1.385	0.893	1.216	32.475	1.562
NMI $\uparrow$	0.101	0.054	0.004	0.042	0.425	0.345	0.117	0.194	0.398	0.327	0.002	0.276
Homogeneity $\uparrow$	0.112	0.055	0.004	0.040	0.441	0.371	0.122	0.202	0.403	0.332	0.002	0.274
Completeness $\uparrow$	0.102	0.053	0.003	0.039	0.411	0.332	0.113	0.186	0.401	0.329	0.003	0.281

Table 6: Clustering quality of retrieved features. Raw denotes no retrieval. Lower DBI and higher NMI, Homogeneity, and Completeness indicate better clustering.

Figure 4: (a) t-SNE visualization of retrieved features under different queries. (b) t-SNE geometry of the Vision Query and the temporal embeddings across four datasets.

	MAR $\uparrow$		CAR $\uparrow$
Dataset	ViRe	Gaussian	ViRe	Gaussian
PTB	1.81	1.46	1.65	1.33
PTB-XL	1.67	1.38	1.51	1.27
MIMIC	1.72	1.41	1.63	1.35

Table 7: Quantitative Attention Alignment on ECG Datasets. MAR and CAR measure attention-density enrichment on high-curvature and QRS regions, respectively; higher is better.

A. Diagnosis F1
Diagnosis	ViRe	Medformer	Gain	Patient-bootstrap 95% CI
MI	72.21	71.35	+0.85	[-1.42,+3.08]
STTC	74.91	73.44	+1.47	[-0.99,+3.97]
HYP	59.49	44.17	+15.32	[+10.39,+20.25]
AMI	72.93	71.01	+1.92	[-1.23,+5.14]
ISCA	37.07	30.15	+6.92	[-2.03,+15.94]

Table 8: PTB-XL diagnosis results and patient-matched visual intervention controls.

Attribute	Metric	CLIP	Random	Gain
Amplitude range	$\rho$	0.903	0.501	+0.402
PR interval	$\rho$	0.633	0.248	+0.384
Cross-lead synchrony	$\rho$	0.684	0.320	+0.364
Prolonged QT	AUROC	0.904	0.656	+0.248

Table 9: Representative MEETI concept-decoding results [ 56 ] .

Zero-shot text alignment	12/12 directions
Key results	AF AUROC +0.423 ; QTc $\rho$ +0.284 ; amplitude +0.239 ; synchrony +0.232 .
Same-patient natural changes	5/5 measurements
Key results	Direction accuracy: HR +0.128 ; PR +0.154 ; QRS +0.127 ; QT +0.238 ; QTc +0.144 .
Raw-space retrieval	21/21 positive CIs
Key results	Bradycardia +0.337 ; HR +0.331 ; amplitude +0.262 ; QT/QRS +0.256/+0.245 .

Table 10: Morphology-aware representation analyses on MEETI.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Operation	Transformation	Default
Temporal flipping	Reverse the sequence along the time axis.	$\textit{prob}=0.5$
Channel shuffling	Randomly permute the channel order.	$\textit{prob}=0.5$
Temporal masking	Mask timestamps shared across all channels.	$\textit{ratio}=0.1$
Frequency masking	Suppress randomly selected frequency bands and transform the signal back to the time domain.	$\textit{ratio}=0.1$
Jittering	Add random noise sampled from $[0,1]$ and scale its magnitude.	$\textit{scale}=0.1$
Dropout	Randomly set a fraction of signal values to zero.	$\textit{ratio}=0.1$

Appendix

Table 11: Training-time augmentations and default settings.

Item	Setting
Batch size	$B=128$
Learning rate	$1\mathrm{e}{-4}$
Model dimension	$D=128$
Temporal Encoder depth	$M=6$ for all datasets
Channel Encoder depth	$N=6$ by default; $N=0$ for TDBrain
Temporal granularity $L$	Dataset order: APAVA, TDBrain, ADFTD, PTB, PTB-XL, MIMIC; $L=\{1,3,8,1,6,6\}$

Appendix

Table 12: Implementation configuration of ViRe.

Figure 5: Visualization example. Stacked twelve-lead PTB waveform rendered by $\mathcal{V}(\cdot)$ .

Metric	ViRe	Medformer	FEDformer
F1-Score $\uparrow$	91.19 $\pm$ 1.01	76.31 $\pm$ 0.71	73.51 $\pm$ 3.30
Training Time (ms) $\downarrow$	211.9	313.1	477.7
Training Memory (MB) $\downarrow$	698.2	838.4	1103.9
Inference Time (ms) $\downarrow$	267.7	152.6	282.6
Inference Memory (MB) $\downarrow$	740.8	408.5	491.6

Appendix

Table 14: Computational Overhead on APAVA. F1-Score, per-batch latency (

B=128

), and peak memory are measured. Higher is better for F1-Score; lower is better for time and memory.

Figure 6: Retrieval Attention Visualization on PTB. Temporal attention from the CLIP-derived Vision Query peaks around a morphologically salient interval with synchronous cross-lead variations.

Figure 7: Rendering sensitivity of ViRe. F1-Score on APAVA and PTB under controlled variations of rendering resolution (left), waveform scaling/line width (middle), and channel layout coloring (right). The results show that ViRe is sensitive to excessively low resolution and inappropriate line width, while being largely insensitive to whether channels are colorized.

Evaluation	Gain	Wins	Statistical evidence
PTB-XL, 13 diagnoses (64.93 vs. 61.57)	+3.36	n/a	Cluster 95% CI [+1.46,+5.10] ; paired $p=0.0103$ .
PTB-XL, five superclasses (72.26 vs. 68.52)	+3.74	n/a	Cluster 95% CI [+2.36,+5.14] ; paired $p=0.0043$ .
PTB-XL, 13 diagnoses; repeated splits	+2.46	7/7	Wilcoxon $p=0.0156$ ; corrected paired $t$ -test $p=0.0038$ ; 95% CI [+1.14,+3.77] .
PTB-XL, five superclasses; repeated splits	+2.42	7/7	Wilcoxon $p=0.0156$ ; corrected paired $t$ -test $p=0.0060$ ; 95% CI [+0.99,+3.84] .
APAVA; repeated splits	+6.48	8/10	Exact Wilcoxon/Holm $p=0.0371$ .
PTB; repeated splits	+3.98	9/10	Exact Wilcoxon/Holm $p=0.0078$ .

Appendix

Table 15: Subject-partition stability and paired statistical evidence.

Dataset	Metric	ViRe	Hi-Patch	t-PatchGNN	GRU-D
Human Activity	MSE $\times 10^{-3}$	2.45 $\pm$ 0.04	2.57 $\pm$ 0.02	2.66 $\pm$ 0.03	3.94 $\pm$ 0.29
Human Activity	MAE $\times 10^{-2}$	3.04 $\pm$ 0.02	3.11 $\pm$ 0.03	3.15 $\pm$ 0.02	4.37 $\pm$ 0.21
PhysioNet	MSE $\times 10^{-3}$	4.56 $\pm$ 0.04	4.86 $\pm$ 0.03	4.98 $\pm$ 0.08	5.76 $\pm$ 0.34
PhysioNet	MAE $\times 10^{-2}$	3.44 $\pm$ 0.03	3.62 $\pm$ 0.07	3.72 $\pm$ 0.03	4.53 $\pm$ 0.15
MIMIC-III	MSE $\times 10^{-2}$	1.56 $\pm$ 0.09	1.75 $\pm$ 0.26	1.69 $\pm$ 0.03	2.35 $\pm$ 0.06
MIMIC-III	MAE $\times 10^{-2}$	6.77 $\pm$ 0.11	7.24 $\pm$ 0.18	7.22 $\pm$ 0.09	8.34 $\pm$ 0.22

Appendix

Table 16: Irregular forecasting under the Hi-Patch protocol [ 32 ] . Lower is better.

Dataset	Metric	ViRe	MTM	STraTS	t-PatchGNN	GRU-D
P12	AUROC	88.5 $\pm$ 1.2	88.0 $\pm$ 1.0	86.4 $\pm$ 1.1	84.5 $\pm$ 0.9	81.9 $\pm$ 2.1
P12	AUPRC	60.3 $\pm$ 2.4	58.6 $\pm$ 4.1	53.9 $\pm$ 3.1	50.8 $\pm$ 2.6	46.1 $\pm$ 4.7
P19	AUROC	91.3 $\pm$ 2.0	90.3 $\pm$ 2.0	89.7 $\pm$ 1.8	87.0 $\pm$ 1.4	83.9 $\pm$ 1.7
P19	AUPRC	62.3 $\pm$ 4.3	58.3 $\pm$ 5.3	57.9 $\pm$ 3.3	51.5 $\pm$ 5.2	46.9 $\pm$ 2.1
PAM	Accuracy	98.3 $\pm$ 0.6	97.5 $\pm$ 0.2	96.4 $\pm$ 0.8	93.9 $\pm$ 1.2	83.3 $\pm$ 1.6
PAM	F1	98.4 $\pm$ 0.6	97.6 $\pm$ 0.2	95.3 $\pm$ 0.7	94.8 $\pm$ 1.2	84.8 $\pm$ 1.2

Appendix

Table 17: Irregular classification under the MTM protocol [ 59 ] . Higher is better.

Explore similar work

Clinical Trajectory Alignment for Medical Vision-Language Pre-training
Sep 28, 2026Huimin Yan, Xian Yang, Zhi Wang +1Medical Vision-Language Models Longitudinal Imaging
Institute of Intelligent Information Processing, Shanxi University, Taiyuan, China · Alliance Manchester Business School, The University of Manchester, Manchester, UK
LoMeVQA: A Comprehensive Benchmark for Longitudinal Medical VQA
Jul 30, 2026Zhilin Wu, Zhangkai Ni, Chengmei Yang +4Medical Visual Question Answering Multimodal Clinical Data
Tongji University Shanghai, China · East China Normal University Shanghai, China
MedMamba: Multi-View State Space Models with Adaptive Graph Learning for Medical Time Series Classification
May 24, 2026Da Zhang, Bingyu Li, Zhiyuan Zhao +3Time-Series Classification State Space Models
Northwest Polytechnical University · TeleAI, China Telecom · TeleAI +1

Sep 28, 2026, cs.CVJ/K move · Enter open · S save

Clinical Trajectory Alignment for Medical Vision-Language Pre-training

Huimin Yan, Xian Yang, Zhi Wang, Liang Bai

Institute of Intelligent Information Processing, Shanxi University, Taiyuan, China · Alliance Manchester Business School, The University of Manchester, Manchester, UK

Medical Vision-Language Models Longitudinal Imaging Multimodal Clinical Data Longitudinal Patient Records Vision-Language Alignment Patient Trajectories+1

Open paper AskPDF arXiv

Medical vision-language pre-training largely follows a visit-level image-report matching paradigm, aligning paired images and reports at individual visits. While effective for static cross-modal correspondence, this paradigm provides limited supervision for longitudinal clinical change, such as whether abnormalities improve, remain stable, or worsen over time. Learning such change is challenging because temporal semantics are implicit in free-text reports, and different abnormalities within the same patient may evolve asynchronously or even in opposite directions. We propose MedCTA, which reframes medical vision-language pre-training from visit-level cross-modal matching to learning clinical change. Rather than compressing a patient history into a single temporal representation, MedCTA models clinical change at two complementary scopes. At the abnormality scope, clinically grounded queries construct abnormality-conditioned visual and textual trajectories to capture heterogeneous abnormality evolution. At the patient-course scope, global image and report sequences are modeled to capture overall clinical progression beyond any individual abnormality. Structured trend supervision is extracted from longitudinal reports by an offline LLM parser, removing the need for manual temporal annotations. Combined with static image-report alignment, MedCTA learns representations that preserve visit-level cross-modal correspondence while encoding longitudinal change semantics. Experiments on temporal image classification, image-text retrieval, and zero-shot classification show consistent gains over strong medical vision-language baselines.

Abstract

Explore similar work

Clinical Trajectory Alignment for Medical Vision-Language Pre-training

LoMeVQA: A Comprehensive Benchmark for Longitudinal Medical VQA

MedMamba: Multi-View State Space Models with Adaptive Graph Learning for Medical Time Series Classification