Benchmarking LLM Summaries of Multimodal Clinical Time Series for Remote Monitoring

Aditya Shukla; Yining Yuan; Ben Tamo; Yifei Wang; Micky Nnamdi; Shaun Tan; Jieru Li; Benoit Marteau

cs.AIMar 2, 2026

Benchmarking LLM Summaries of Multimodal Clinical Time Series for Remote Monitoring

Authors: Aditya Shukla, Yining Yuan, Ben Tamo, Yifei Wang, Micky Nnamdi, Shaun Tan, Jieru Li, Benoit Marteau,

Organizations: Georgia Institute of Technology · Shepherd Center

Abstract

Large language models (LLMs) can generate fluent clinical summaries of remote therapeutic monitoring time series. However, it remains unclear whether these narratives faithfully capture clinically significant events, such as sustained abnormalities. Existing evaluation metrics primarily focus on semantic similarity and linguistic quality, leaving event-level correctness largely unmeasured. To address this gap, we introduce an event-based evaluation framework for multimodal time-series summarization using the Technology-Integrated Health Management (TIHM)-1.5 dementia monitoring dataset. Clinically grounded daily events are derived through rule-based abnormal thresholds and temporal persistence criteria. Model-generated summaries are then aligned with these structured facts. Our evaluation protocol measures abnormality recall, duration recall, measurement coverage, and hallucinated event mentions. We benchmark three approaches: zero-shot prompting, statistical prompting, and a vision-based pipeline that uses rendered time-series visualizations. The results reveal a striking decoupling between conventional metrics and clinical event fidelity. Models that achieve high semantic similarity scores often exhibit near-zero abnormality recall. In contrast, the vision-based approach demonstrates the strongest event alignment, achieving 45.7% abnormality recall and 100% duration recall. These findings underscore the importance of event-aware evaluation to ensure reliable clinical time-series summarization.

Figures & tables

Figure 1: Overview of the proposed event-based evaluation framework for clinical time-series summarization. Left: Summary generation pipelines, including (1) image-based vision–language summarization via signal segmentation and plot rendering, and (2) structured statistical prompting with aggregated numerical features. Both pipelines generate daily LLM summaries from raw physiological data. Right: Event-based factual alignment evaluation. Ground-truth clinical events are derived through rule-based thresholding to detect abnormalities and sustained durations. Generated summaries are evaluated using strict event matching (clinical factual recall) and measurement coverage analysis, producing recall and coverage scores that quantify event-level correctness beyond reference-based semantic similarity metrics.

Method	Backbone	Clinical Event Metrics - Ours (%)			Standard NLP Metrics ( $\uparrow$ )
Method	Backbone	Abnormality	Duration	Coverage	SummaC	Align	Clarity
Zero-shot	Llama-3.2-3B	0.00	0.00	46.44	0.59	0.32	3.27
	Llama-3-8B	0.00	0.00	41.40	0.57	0.31	4.00
	Gemma-3-12B	0.00	0.00	31.50	0.28	0.39	1.70
	Gemma-3-27B	2.10	0.00	15.60	0.55	0.44	1.44
Stat Based	Llama-3.2-3B	18.80	0.00	99.32	0.44	0.23	3.38

Table 1: Event-level clinical correctness diverges from conventional NLP metrics. Statistical conditioning improves grounding, and the visualization-grounded pipeline achieves the strongest event alignment despite lower NLP scores.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Systolic blood pressure (SBP) over time for patient d7a46 on 2019-06-11. The SBP peaks at 188 mmHg (14:36) and remains markedly elevated across multiple readings through 17:55, contradicting the summary’s reported peak (150 mmHg) and time window (13:00–13:30).

Summarization Evaluation

Summarization evaluation measures whether summaries preserve important information, remain faithful to their sources, and serve readers’ needs.

See all 44 papers

Sep 4, 2026

MMTClinic: Multimodal, Multilingual Time Series Question Answering and Reasoning Benchmark for Clinical Domain

Time-series data in clinical settings is crucial for capturing dynamic changes in a patient's health over time, enabling timely diagnosis, personalized treatment, and early detection of critical events. However, the development of clinically reliable and linguistically inclusive medical AI systems remains a significant challenge, primarily due to the lack of multimodal, multilingual, and time-series-grounded benchmarks that reflect the complexity of real-world clinical scenarios. To fill this gap, we present MMTClinic, a benchmark designed to evaluate large language models (LLMs) on complex reasoning and question-answering tasks involving clinical time-series. MMTClinic combines text, medical images, and multivariate physiological signals and includes 30,000 QA pairs (15,000 multiple choice questions (MCQs) and 15,000 open-ended questions) across five languages: English, Hindi, Bengali, Marathi, and Tamil. These questions cover three important clinical tasks---mortality prediction, heart rate forecasting, and SOFA score estimation. We evaluate 13 state-of-the-art LLMs in zero-shot, few-shot, and chain-of-thought settings. Our evaluation reveals notable differences in model performance across tasks, languages, and modalities, highlighting current limitations in clinical reasoning capabilities. MMTClinic provides a valuable resource for advancing multilingual, multimodal, and time-series-aware medical AI research. The dataset will be made publicly available on successful acceptance of the work.

Sourav Malakar, Harshit Nigam, Akash Ghosh +4

Department of CSE Institute of Engineering and Management Kolkata, West Bengal, India · Department of CSE IIT Patna Bihar, India · Department of A. K. Choudhury School of IT University of Calcutta Kolkata, West Bengal, India +1

Time Series QA Clinical Language Model Evaluation Multimodal Model Evaluation Multilingual Language Model Evaluation Clinical Outcome Prediction Multivariate Time Series Forecasting

Ask PDF

Sep 7, 2026

Large language models in medical time series analysis

Medical time series (MedTS), including electrocardiograms (ECG), electroencephalograms (EEG), photoplethysmography (PPG), and vital-sign recordings, are central to clinical diagnosis and health monitoring. As large language models (LLMs) have advanced, a growing body of work has examined how their reasoning, generation, and knowledge-integration capabilities can support MedTS analysis. Yet existing studies remain scattered, and the field still lacks a clear view of how these models should be designed, integrated into clinical workflows, and evaluated. This review synthesizes recent work on large language models for medical time series analysis (MedTSLLMs), covering both methodological progress and issues related to real-world deployment. We review model architectures, data resources, and processing pipelines, and prompt design strategies adapted for diverse clinical scenarios. We further organize existing MedTS applications, ranging from diagnostic interpretation and report generation to longitudinal health monitoring and physiological signal synthesis, highlighting task-specific design choices, common evaluation protocols, and empirical findings reported across studies. By bringing together current practices and open challenges, this review aims to provide a clearer foundation for developing, evaluating, and deploying MedTSLLMs responsibly in healthcare. We also maintain a regularly updated list of MedTSLLM studies and resources at: https://github.com/hy727/MedTSLLM-Review.

Yu Han, Cigdem Beyan, Xiang Zhang +6

AI for Good (AIGO), Istituto Italiano di Tecnologia, Genoa, Liguria, Italy · DITEN, University of Genoa, Genoa, Liguria, Italy · Department of Computer Science, University of Verona, Verona, Veneto, Italy +9

Medical LLMs

Ask PDF

Sep 17, 2026

CliniCIRCA: A Modular LLM Framework for Constructing Longitudinal Mental Health Patient Journeys from Raw EHR Narratives

In mental health care, reasoning over patient journeys is a key task for clinicians. Yet these journeys, encompassing a longitudinal progression of biological, psychological, and social events, are often spread across disparate unstructured text narratives, making temporal recovery challenging. We present CliniCIRCA, a multi-stage LLM framework for Calendar-anchored, Imprecision-aware Reconstruction of Clinical Annals. To our knowledge, CliniCIRCA is the first to temporally classify clinical events across unstructured discharge summaries without event-level timestamps. From 14,882 MIMIC-III mental health admissions, we first construct a benchmark of 52 discharge summaries on which CliniCIRCA produces 15,891 temporally tagged events. After correcting 629 errors based on a clinician-in-the-loop evaluation, we produce verified gold-standard labels. Finally, the corrected timelines drive a temporally grounded summarization stage that compresses each source 1.52 times into a date-grouped chronological record. We then scale the framework to generate 1,000 silver-standard timelines and evaluate them as training data. Compared with zero- and few-shot prompting, instruction tuning generally improves five open-weight models on event extraction, temporal tagging, and summarization across silver and clinician-verified evaluations.

Aiwei Ivy Zhang, Nimra Ishfaq, Mohit Chandra +5

Georgia Institute of Technology · University of Texas at Austin · Northwell Health

Temporal Reasoning in Language Models Mental Health LLM-Assisted Annotation Document Information Extraction Longitudinal EHR Modeling

Ask PDF

Abstract

Summarization Evaluation

Explore similar work

MMTClinic: Multimodal, Multilingual Time Series Question Answering and Reasoning Benchmark for Clinical Domain

Large language models in medical time series analysis

CliniCIRCA: A Modular LLM Framework for Constructing Longitudinal Mental Health Patient Journeys from Raw EHR Narratives