Organizations: Department of CSE Institute of Engineering and Management Kolkata, West Bengal, India · Department of CSE IIT Patna Bihar, India · Department of A. K. Choudhury School of IT University of Calcutta Kolkata, West Bengal, India · Department of Computer Science Bangabasi Morning College Kolkata, West Bengal, India
Time-series data in clinical settings is crucial for capturing dynamic changes in a patient's health over time, enabling timely diagnosis, personalized treatment, and early detection of critical events. However, the development of clinically reliable and linguistically inclusive medical AI systems remains a significant challenge, primarily due to the lack of multimodal, multilingual, and time-series-grounded benchmarks that reflect the complexity of real-world clinical scenarios. To fill this gap, we present MMTClinic, a benchmark designed to evaluate large language models (LLMs) on complex reasoning and question-answering tasks involving clinical time-series. MMTClinic combines text, medical images, and multivariate physiological signals and includes 30,000 QA pairs (15,000 multiple choice questions (MCQs) and 15,000 open-ended questions) across five languages: English, Hindi, Bengali, Marathi, and Tamil. These questions cover three important clinical tasks---mortality prediction, heart rate forecasting, and SOFA score estimation. We evaluate 13 state-of-the-art LLMs in zero-shot, few-shot, and chain-of-thought settings. Our evaluation reveals notable differences in model performance across tasks, languages, and modalities, highlighting current limitations in clinical reasoning capabilities. MMTClinic provides a valuable resource for advancing multilingual, multimodal, and time-series-aware medical AI research. The dataset will be made publicly available on successful acceptance of the work.
Figures & tables
Figure 1: Overview of MMTClinic
Table 2Table 3
Figure 2: Construction of MMTClinic. Step 1 indicates data collection. Step 2 implies the detailed pre-processing steps for extracting complete data of a patient. Step 3 focused on question answer generation in two experimental setup (a) Text + Time Series and (b) Text + Time Series + Image. Step 4 indicates the question answer validation by medical and linguistic expert.
Figure 5: Radar plot indicating language-specific accuracy of LLMs for Text and Time Series modality: (a) MCQ and (b) Reasoning.
Figure 8: Radar plot indicating language-specific accuracy of LLMs for Text, Time Series, and Image modality: (a) MCQ and (b) Reasoning.
Figure 11: Box plot indicating few-shot and CoT specific accuracy of LLMs for Text and Time Series modality (a) MCQ (b) Reasoning
Figure 12: The LHS displays the correctly answered questions, while the RHS highlights the incorrectly answered ones by the language models on the MMTClinic dataset.
LLMs
Parameters
DeepSeek-R1 Guo et al. (2025a)
Commercial
GPT-4.1-nano Popov et al. (2025)
Commercial
gemini-2-flash Rakshith et al. (2025)
Commercial
DeepSeek-R1-Llama-8B Guo et al. (2025b)
8B
Gemma-3-27B-IT Team et al. (2025)
27B
Llama-3.2-11B-Vision-Instruct Lee et al. (2025)
11B
Table 5: List of LLMs with corresponding parameter sizes
Dataset
#Samples
Modalities
Question Types
Multi.
Reason.
MCQ
#Tasks
#Langs.
ECG-QA
∼ 2K
ECG Signal
MCQ
No
No
Yes
1
1
ECG-Expert-QA
47K
ECG Signal
MCQ + Open-ended
No
Yes
Yes
Multi-cond.
1
Q-Heart
8.7K
ECG + Text
Open-ended
No
Yes
No
Multi-cond.
1
TIMER-Bench
∼ 10K
Text + TS
Instructional QA
No
Yes
No
Temporal QA
1
EHRNoteQA
∼ 2K
Text + TS
Open-ended
No
Yes
No
Patient QA
1
EHRXQA
N/S
Text + Image + TS
Open-ended
No
Yes
No
EHR + X-ray
1
Table 6: Comparison of clinical QA and reasoning benchmarks across key dimensions.
Figure 15: Radial plot indicating language-specific accuracy of LLMs under few-shot and CoT setup for Text, Time Series, and Image modality (a) MCQ (b) Reasoning
Clinical time series are central to patient monitoring, risk assessment, and clinical decision support. However, they are often sparse, irregularly sampled, and asynchronous, making it difficult for models to identify the temporal evidence required for clinical Question Answering (QA). Existing benchmarks primarily focus on regularly sampled time-series QA or medical QA over static data, and therefore rarely assess whether models can faithfully ground their answers in irregular temporal observations. To fill this gap, we introduce CLIR-Bench, a benchmark for irregular clinical time series QA constructed from de-identified ICU records through a principled four-stage pipeline. CLIR-Bench contains 6,600 QA instances spanning 11 clinical variables, organized into four capability dimensions and 11 tasks. Each question is linked to explicit temporal evidence and task-specific answer derivation rules, enabling evaluation of both answer accuracy and evidence use. Experiments show that existing generalist models struggle to retrieve and reason over sparse clinical evidence, highlighting the need for stronger irregular time-series reasoning methods. Our code and data are available at https://huggingface.co/datasets/winall/CLIR-Bench.
Frank Nie, Ethan B. Liu, Yuan Zhu +3
Shandong University China · University of Auckland New Zealand
Large language models (LLMs) have been widely adopted in healthcare, yet they still encounter significant challenges in complex clinical decision-making scenarios. Existing benchmarks primarily assess LLM performance in single-course settings and lack systematic evaluation in multi-course scenarios, where a patient's condition evolves over time. To address this gap, we propose ClinicalMC, a benchmark for multi-course clinical decision-making. It includes 1,275 Chinese and 5,804 English samples across four stages from admission to discharge. These stages cover triage, first-course examination/diagnosis/treatment, subsequent multi-course examination/assessment/treatment, and final diagnosis. In ClinicalMC, patients in the English dataset undergo an average of 5.11 clinical courses, whereas those in the Chinese dataset undergo 3.42. To assess LLM performance, we construct a multi-agent evaluation framework that includes patient, examiner, and doctor agents. Based on the benchmark and framework, we design two experimental settings -- a single-turn static setting and a multi-turn dynamic setting -- and assess three categories of LLMs: 1) closed-source LLMs like GPT5-mini; 2) open-source LLMs like DeepSeek-V3.2; and 3) medical LLMs like HuatuoGPT-o1. Through extensive evaluation, we aim to better understand LLM performance in the medical domain and support its effective deployment in healthcare.
Ruihui Hou, Siyi Zhu, Ziyue Huai +4
East China University of Science and Technology, Shanghai, China · Renji Hospital Affiliated to Shanghai Jiaotong University School of Medicine, Shanghai, China.
The rise of large language models (LLMs) has shifted time series analysis from narrow analytics to general-purpose reasoning. Yet, existing benchmarks cover only a small set of health time series modalities and tasks, failing to reflect the diverse domains and extensive temporal dependencies inherent in real-world physiological modeling. To bridge these gaps, we introduce HEARTS (Health Reasoning over Time Series), a unified benchmark for evaluating hierarchical reasoning capabilities of LLMs over general health time series. HEARTS integrates 16 real-world datasets across 12 health domains and 20 signal modalities, and defines a comprehensive taxonomy of 110 tasks grouped into four core capabilities: Perception, Inference, Generation, and Deduction. Evaluating 16 state-of-the-art LLMs on more than 20K test samples reveals intriguing findings. First, LLMs substantially underperform specialized models, and their performance is only weakly related to general reasoning scores. Moreover, LLMs often rely on simple heuristics and struggle with multi-step temporal reasoning. Finally, performance declines with increasing temporal complexity, with similar failure modes within model families, indicating that scaling alone is insufficient. By making these gaps measurable, HEARTS provides a standardized testbed and living benchmark for developing next-generation LLM agents capable of reasoning over diverse health signals.
Sirui Li, Shuhan Xiao, Mihir Joshi +4
University of California, Los Angeles, 2Google Research · 2Google Research