MMTClinic: Multimodal, Multilingual Time Series Question Answering and Reasoning Benchmark for Clinical Domain

Sourav Malakar; Harshit Nigam; Akash Ghosh; Sriparna Saha; Amlan Chakrabarti; Saptarsi Goswami; Priti Singh

cs.CLSep 4, 2026

MMTClinic: Multimodal, Multilingual Time Series Question Answering and Reasoning Benchmark for Clinical Domain

Authors: Sourav Malakar, Harshit Nigam, Akash Ghosh, Sriparna Saha, Amlan Chakrabarti, Saptarsi Goswami, Priti Singh

Organizations: Department of CSE Institute of Engineering and Management Kolkata, West Bengal, India · Department of CSE IIT Patna Bihar, India · Department of A. K. Choudhury School of IT University of Calcutta Kolkata, West Bengal, India · Department of Computer Science Bangabasi Morning College Kolkata, West Bengal, India

Abstract

Time-series data in clinical settings is crucial for capturing dynamic changes in a patient's health over time, enabling timely diagnosis, personalized treatment, and early detection of critical events. However, the development of clinically reliable and linguistically inclusive medical AI systems remains a significant challenge, primarily due to the lack of multimodal, multilingual, and time-series-grounded benchmarks that reflect the complexity of real-world clinical scenarios. To fill this gap, we present MMTClinic, a benchmark designed to evaluate large language models (LLMs) on complex reasoning and question-answering tasks involving clinical time-series. MMTClinic combines text, medical images, and multivariate physiological signals and includes 30,000 QA pairs (15,000 multiple choice questions (MCQs) and 15,000 open-ended questions) across five languages: English, Hindi, Bengali, Marathi, and Tamil. These questions cover three important clinical tasks---mortality prediction, heart rate forecasting, and SOFA score estimation. We evaluate 13 state-of-the-art LLMs in zero-shot, few-shot, and chain-of-thought settings. Our evaluation reveals notable differences in model performance across tasks, languages, and modalities, highlighting current limitations in clinical reasoning capabilities. MMTClinic provides a valuable resource for advancing multilingual, multimodal, and time-series-aware medical AI research. The dataset will be made publicly available on successful acceptance of the work.

Figures & tables

Table 2

Table 3

Figure 2: Construction of MMTClinic. Step 1 indicates data collection. Step 2 implies the detailed pre-processing steps for extracting complete data of a patient. Step 3 focused on question answer generation in two experimental setup (a) Text + Time Series and (b) Text + Time Series + Image. Step 4 indicates the question answer validation by medical and linguistic expert.

Panel of Figure 5 — Figure 5: Radar plot indicating language-specific accuracy of LLMs for Text and Time Series modality: (a) MCQ and (b) Reasoning.

Panel of Figure 8 — Figure 8: Radar plot indicating language-specific accuracy of LLMs for Text, Time Series, and Image modality: (a) MCQ and (b) Reasoning.

Panel of Figure 11 — Figure 11: Box plot indicating few-shot and CoT specific accuracy of LLMs for Text and Time Series modality (a) MCQ (b) Reasoning

Figure 12: The LHS displays the correctly answered questions, while the RHS highlights the incorrectly answered ones by the language models on the MMTClinic dataset.

LLMs	Parameters
DeepSeek-R1 Guo et al. (2025a)	Commercial
GPT-4.1-nano Popov et al. (2025)	Commercial
gemini-2-flash Rakshith et al. (2025)	Commercial
DeepSeek-R1-Llama-8B Guo et al. (2025b)	8B
Gemma-3-27B-IT Team et al. (2025)	27B
Llama-3.2-11B-Vision-Instruct Lee et al. (2025)	11B

Table 5: List of LLMs with corresponding parameter sizes

Dataset	#Samples	Modalities	Question Types	Multi.	Reason.	MCQ	#Tasks	#Langs.
ECG-QA	$\sim$ 2K	ECG Signal	MCQ	No	No	Yes	1	1
ECG-Expert-QA	47K	ECG Signal	MCQ + Open-ended	No	Yes	Yes	Multi-cond.	1
Q-Heart	8.7K	ECG + Text	Open-ended	No	Yes	No	Multi-cond.	1
TIMER-Bench	$\sim$ 10K	Text + TS	Instructional QA	No	Yes	No	Temporal QA	1
EHRNoteQA	$\sim$ 2K	Text + TS	Open-ended	No	Yes	No	Patient QA	1
EHRXQA	N/S	Text + Image + TS	Open-ended	No	Yes	No	EHR + X-ray	1

Table 6: Comparison of clinical QA and reasoning benchmarks across key dimensions.

Panel of Figure 15 — Figure 15: Radial plot indicating language-specific accuracy of LLMs under few-shot and CoT setup for Text, Time Series, and Image modality (a) MCQ (b) Reasoning

Per-Task Expert Ratings (Medical, Linguistic, Mean)
MCQ (Text + TimeSeries)
•Medical Expert Rating: 4.3
•Linguistic Expert Rating: 4.1
•Mean Rating: 4.2
MCQ (Text + TimeSeries + Images)
•Medical Expert Rating: 4.6

Table 7: Vertically formatted expert review scores for each task.

Modality	Type	Task	Language	Model	Accuracy
Small LLMs
Text + Timeseries	MCQ	Task3	Bengali	DeepSeek-R1-Llama-8B	53.3
Text + Timeseries	MCQ	Task1	Bengali	DeepSeek-R1-Llama-8B	51.5
Text + Timeseries	MCQ	Task2	Bengali	DeepSeek-R1-Llama-8B	52.21
Text + Timeseries	MCQ	Task1	Bengali	Mistral-7B-Instruct	35.05
Text + Timeseries	MCQ	Task2	Bengali	Mistral-7B-Instruct	34.38

Table 8: Model performance across tasks using different Bengali multimodal inputs

Modality	Type	Task	Language	Model	Accuracy
Small LLMs
Text + Timeseries	MCQ	Task3	English	DeepSeek-R1-Llama-8B	59.6
Text + Timeseries	MCQ	Task1	English	DeepSeek-R1-Llama-8B	61.24
Text + Timeseries	MCQ	Task2	English	DeepSeek-R1-Llama-8B	58.6
Text + Timeseries	MCQ	Task1	English	Mistral-7B-Instruct	45.99
Text + Timeseries	MCQ	Task2	English	Mistral-7B-Instruct	44.55

Table 9: Model performance across tasks using different English multimodal inputs

Modality	Type	Task	Language	Model	Accuracy
Small LLMs (≤7B)
Text + Timeseries	MCQ	Task3	Hindi	DeepSeek-R1-Llama-8B	55.2
Text + Timeseries	MCQ	Task1	Hindi	DeepSeek-R1-Llama-8B	58.6
Text + Timeseries	MCQ	Task2	Hindi	DeepSeek-R1-Llama-8B	54.1
Text + Timeseries	MCQ	Task1	Hindi	Mistral-7B-Instruct	35.63
Text + Timeseries	MCQ	Task2	Hindi	Mistral-7B-Instruct	40.27

Table 10: Model performance across tasks using different Hindi multimodal inputs

Modality	Type	Task	Language	Model	Accuracy
Small LLMs (≤7B)
Text + Timeseries	MCQ	Task3	Marathi	DeepSeek-R1-Llama-8B	58.7
Text + Timeseries	MCQ	Task1	Marathi	DeepSeek-R1-Llama-8B	59.2
Text + Timeseries	MCQ	Task2	Marathi	DeepSeek-R1-Llama-8B	52.6
Text + Timeseries	MCQ	Task1	Marathi	Mistral-7B-Instruct	33.94
Text + Timeseries	MCQ	Task2	Marathi	Mistral-7B-Instruct	32.86

Table 11: Model performance across tasks using different Marathi multimodal inputs

Small LLMs (≤7B)
Text + Timeseries	MCQ	Task3	Tamil	DeepSeek-R1-Llama-8B	61.2
Text + Timeseries	MCQ	Task1	Tamil	DeepSeek-R1-Llama-8B	60.2
Text + Timeseries	MCQ	Task2	Tamil	DeepSeek-R1-Llama-8B	56.6
Text + Timeseries	MCQ	Task1	Tamil	Mistral-7B-Instruct	21.48
Text + Timeseries	MCQ	Task2	Tamil	Mistral-7B-Instruct	31.56
Text + Timeseries	MCQ	Task3	Tamil	Mistral-7B-Instruct	25.09

Table 12: Model performance across tasks using different Tamil multimodal inputs

Modality	Type	Task	Language	Model	Accuracy
Small to Mid LLMs (≤32B)
Text + Timeseries	Reasoning	Task3	English	DeepSeek-R1-Llama-8B	60
Text + Timeseries	Reasoning	Task1	English	DeepSeek-R1-Llama-8B	64.8
Text + Timeseries	Reasoning	Task2	English	DeepSeek-R1-Llama-8B	53.2
Text + Timeseries	Reasoning	Task1	English	Mistral-7B-Instruct	32.05
Text + Timeseries	Reasoning	Task2	English	Mistral-7B-Instruct	31.23

Table 13: Performance of models on Reasoning tasks using English multimodal inputs

Modality	Type	Task	Language	Model	Accuracy
Small to Mid LLMs (≤32B)
Text + Timeseries	Reasoning	Task1	Bengali	DeepSeek-R1-Llama-8B	76.6
Text + Timeseries	Reasoning	Task2	Bengali	DeepSeek-R1-Llama-8B	54.8
Text + Timeseries	Reasoning	Task3	Bengali	DeepSeek-R1-Llama-8B	64.6
Text + Timeseries	Reasoning	Task1	Bengali	Mistral-7B-Instruct	30.0
Text + Timeseries	Reasoning	Task2	Bengali	Mistral-7B-Instruct	25.3

Table 14: Performance of models on Reasoning tasks using Bengali multimodal inputs

Modality	Type	Task	Language	Model	Accuracy
Small to Mid LLMs (≤32B)
Text + Timeseries	Reasoning	Task2	Hindi	DeepSeek-R1-Llama-8B	57.6
Text + Timeseries	Reasoning	Task3	Hindi	DeepSeek-R1-Llama-8B	66.6
Text + Timeseries	Reasoning	Task1	Hindi	DeepSeek-R1-Llama-8B	75.6
Text + Timeseries	Reasoning	Task1	Hindi	Mistral-7B-Instruct	28.6
Text + Timeseries	Reasoning	Task2	Hindi	Mistral-7B-Instruct	24.7

Table 15: Performance of models on Reasoning tasks using Hindi multimodal inputs

Modality	Type	Task	Language	Model	Accuracy
Small to Mid LLMs (≤32B)
Text + Timeseries	Reasoning	Task1	Marathi	DeepSeek-R1-Llama-8B	70.6
Text + Timeseries	Reasoning	Task2	Marathi	DeepSeek-R1-Llama-8B	51.4
Text + Timeseries	Reasoning	Task3	Marathi	DeepSeek-R1-Llama-8B	60
Text + Timeseries	Reasoning	Task1	Marathi	Mistral-7B-Instruct	26.5
Text + Timeseries	Reasoning	Task2	Marathi	Mistral-7B-Instruct	21.3

Table 16: Performance of models on Reasoning tasks using Marathi multimodal inputs

Modality	Type	Task	Language	Model	Accuracy
Small to Mid LLMs (≤32B)
Text + Timeseries	Reasoning	Task1	Tamil	DeepSeek-R1-Llama-8B	80.8
Text + Timeseries	Reasoning	Task3	Tamil	DeepSeek-R1-Llama-8B	65.2
Text + Timeseries	Reasoning	Task2	Tamil	DeepSeek-R1-Llama-8B	56.2
Text + Timeseries	Reasoning	Task1	Tamil	Mistral-7B-Instruct	23.87
Text + Timeseries	Reasoning	Task2	Tamil	Mistral-7B-Instruct	23.43

Table 17: Performance of models on Reasoning tasks using Tamil multimodal inputs

Modality	Type	Task	Language	Model	Accuracy
Text + Timeseries + Image	MCQ	Task1	Bengali	Llama-3.2-11B-Vision-Instruct	30.05
Text + Timeseries + Image	MCQ	Task2	Bengali	Llama-3.2-11B-Vision-Instruct	39.77
Text + Timeseries + Image	MCQ	Task3	Bengali	Llama-3.2-11B-Vision-Instruct	27.81
Text + Timeseries + Image	MCQ	Task1	English	Llama-3.2-11B-Vision-Instruct	34.48
Text + Timeseries + Image	MCQ	Task2	English	Llama-3.2-11B-Vision-Instruct	39.26
Text + Timeseries + Image	MCQ	Task3	English	Llama-3.2-11B-Vision-Instruct	30.96

Table 18: Performance of models on MCQ tasks with Text + Timeseries + Image inputs across multiple languages

Modality	Type	Task	Language	Model	Accuracy
Text + Timeseries	MCQ	Task1	Bengali	Llama-3.2-11B-Vision-Instruct	33.07
Text + Timeseries	MCQ	Task2	Bengali	Llama-3.2-11B-Vision-Instruct	44.61
Text + Timeseries	MCQ	Task3	Bengali	Llama-3.2-11B-Vision-Instruct	31.74
Text + Timeseries	MCQ	Task1	English	Llama-3.2-11B-Vision-Instruct	39.07
Text + Timeseries	MCQ	Task2	English	Llama-3.2-11B-Vision-Instruct	43.98
Text + Timeseries	MCQ	Task3	English	Llama-3.2-11B-Vision-Instruct	34.92

Table 19: Performance of Few-Shot prompting for MCQ tasks across different languages and models

Modality	Type	Task	Language	Model	Accuracy
Text + Timeseries	MCQ	Task1	Bengali	Llama-3.2-11B-Vision-Instruct	38.63
Text + Timeseries	MCQ	Task2	Bengali	Llama-3.2-11B-Vision-Instruct	50.72
Text + Timeseries	MCQ	Task3	Bengali	Llama-3.2-11B-Vision-Instruct	38.39
Text + Timeseries	MCQ	Task1	English	Llama-3.2-11B-Vision-Instruct	45.51
Text + Timeseries	MCQ	Task2	English	Llama-3.2-11B-Vision-Instruct	50.28
Text + Timeseries	MCQ	Task3	English	Llama-3.2-11B-Vision-Instruct	41.01

Table 20: Performance of CoT prompting for MCQ tasks across different languages and models

Modality	Type	Task	Language	Model	Accuracy
Text + Timeseries + Image	MCQ	Task1	Bengali	Llama-3.2-11B-Vision-Instruct	38.26
Text + Timeseries + Image	MCQ	Task2	Bengali	Llama-3.2-11B-Vision-Instruct	52.51
Text + Timeseries + Image	MCQ	Task3	Bengali	Llama-3.2-11B-Vision-Instruct	36.60
Text + Timeseries + Image	MCQ	Task1	English	Llama-3.2-11B-Vision-Instruct	47.00
Text + Timeseries + Image	MCQ	Task2	English	Llama-3.2-11B-Vision-Instruct	50.72
Text + Timeseries + Image	MCQ	Task3	English	Llama-3.2-11B-Vision-Instruct	41.71

Table 21: Few-Shot Accuracy for reasoning tasks across different languages and models using Text + Timeseries + Image modality

Modality	Type	Task	Language	Model	Accuracy
Text + Timeseries + Image	MCQ	Task1	Bengali	Llama-3.2-11B-Vision-Instruct	45.63
Text + Timeseries + Image	MCQ	Task2	Bengali	Llama-3.2-11B-Vision-Instruct	58.76
Text + Timeseries + Image	MCQ	Task3	Bengali	Llama-3.2-11B-Vision-Instruct	39.82
Text + Timeseries + Image	MCQ	Task1	English	Llama-3.2-11B-Vision-Instruct	48.82
Text + Timeseries + Image	MCQ	Task2	English	Llama-3.2-11B-Vision-Instruct	55.39
Text + Timeseries + Image	MCQ	Task3	English	Llama-3.2-11B-Vision-Instruct	47.87

Table 22: CoT Accuracy for reasoning tasks across different languages and models using Text + Timeseries + Image modality

Modality	Type	Task	Language	Model	Accuracy
Text + Timeseries + Image	Reasoning	Task1	Bengali	Llama-3.2-11B-Vision-Instruct	29.25
Text + Timeseries + Image	Reasoning	Task2	Bengali	Llama-3.2-11B-Vision-Instruct	33.34
Text + Timeseries + Image	Reasoning	Task3	Bengali	Llama-3.2-11B-Vision-Instruct	21.31
Text + Timeseries + Image	Reasoning	Task1	English	Llama-3.2-11B-Vision-Instruct	30.12
Text + Timeseries + Image	Reasoning	Task2	English	Llama-3.2-11B-Vision-Instruct	37.19
Text + Timeseries + Image	Reasoning	Task3	English	Llama-3.2-11B-Vision-Instruct	23.8

Table 23: Performance of models on Reasoning tasks with Text + Timeseries + Image inputs across multiple languages.

Figure 16: Example illustration of both MCQ and Reasoning predictions by LLM

Time Series QA

Time-series question answering connects natural-language questions to evidence and reasoning over sequential data.

See all 32 papers

CLIR-Bench: Benchmarking Multimodal Question Answering over Irregular Clinical Time Series
Jul 10, 2026Frank Nie, Ethan B. Liu, Yuan Zhu +3Irregular Time-Series Modeling Time Series QA
Shandong University China · University of Auckland New Zealand
ClinicalMC: A Benchmark for Multi-Course Clinical Decision-Making with Large Language Models
Jun 2, 2026Ruihui Hou, Siyi Zhu, Ziyue Huai +4Clinical Decision-Making Clinical Language Model Evaluation
East China University of Science and Technology, Shanghai, China · Renji Hospital Affiliated to Shanghai Jiaotong University School of Medicine, Shanghai, China.
HEARTS: Benchmarking LLM Reasoning on Health Time Series
Feb 25, 2026Sirui Li, Shuhan Xiao, Mihir Joshi +4LLM Evaluation Time Series Reasoning
University of California, Los Angeles, 2Google Research · 2Google Research

Jul 10, 2026J/K move · Enter open · S save

CLIR-Bench: Benchmarking Multimodal Question Answering over Irregular Clinical Time Series

Frank Nie, Ethan B. Liu, Yuan Zhu, Loe Yan+2

Shandong University China · University of Auckland New Zealand

Irregular Time-Series Modeling Time Series QA Clinical QA

Open paper AskPDF arXiv

Clinical time series are central to patient monitoring, risk assessment, and clinical decision support. However, they are often sparse, irregularly sampled, and asynchronous, making it difficult for models to identify the temporal evidence required for clinical Question Answering (QA). Existing benchmarks primarily focus on regularly sampled time-series QA or medical QA over static data, and therefore rarely assess whether models can faithfully ground their answers in irregular temporal observations. To fill this gap, we introduce CLIR-Bench, a benchmark for irregular clinical time series QA constructed from de-identified ICU records through a principled four-stage pipeline. CLIR-Bench contains 6,600 QA instances spanning 11 clinical variables, organized into four capability dimensions and 11 tasks. Each question is linked to explicit temporal evidence and task-specific answer derivation rules, enabling evaluation of both answer accuracy and evidence use. Experiments show that existing generalist models struggle to retrieve and reason over sparse clinical evidence, highlighting the need for stronger irregular time-series reasoning methods. Our code and data are available at https://huggingface.co/datasets/winall/CLIR-Bench.

Abstract

Time Series QA

Explore similar work

CLIR-Bench: Benchmarking Multimodal Question Answering over Irregular Clinical Time Series

ClinicalMC: A Benchmark for Multi-Course Clinical Decision-Making with Large Language Models

HEARTS: Benchmarking LLM Reasoning on Health Time Series