HealthLoopQA: A Context-Aware Question Answering Benchmark for Interpreting Wearable Monitoring Data in Diabetes Care

Yuchen Niu; Yanan Ma; Srinivasan Nandakumar; Maolin Chen; Viktor Schlegel; Kexin Wei; Ling Cheng; Anna Bird

cs.LGSep 7, 2026

HealthLoopQA: A Context-Aware Question Answering Benchmark for Interpreting Wearable Monitoring Data in Diabetes Care

Authors: Yuchen Niu, Yanan Ma, Srinivasan Nandakumar, Maolin Chen, Viktor Schlegel, Kexin Wei, Ling Cheng, Anna Bird,

Organizations: Nanyang Technological University · Mayo Clinic · The University of Manchester · Imperial College London, Imperial Global Singapore ·

Abstract

As medical wearables become integrated into daily chronic disease care, effectively interpreting longitudinal monitoring data is essential for patients and clinicians to understand health trends, detect safety-critical events, and make informed decisions. While large language models (LLMs) show promise for transforming this streaming physiological data into personalized health insights, evaluating their reasoning capability and analytical rigor in diverse monitoring tasks remains a fundamental challenge. Existing medical wearable question answering (QA) benchmarks primarily assess short-horizon classification or statistical summaries, largely ignoring the long-term patterns, therapeutic and behavioural contexts, and potential system failures inherent in real-world deployments. To address this, we introduce HealthLoopQA, a comprehensive diagnostic benchmark for evaluating LLM reasoning over continuous diabetes monitoring data. Grounded in a novel taxonomy of eleven atomic reasoning abilities, HealthLoopQA comprises 127 tasks and over 1,500 QA instances spanning process mining, anomaly detection, and prediction over 30-day horizons. To systematically evaluate safety awareness, we complement real-world datasets with a fault-injected simulation testbed modeling diverse device malfunctions and cyber-physical attacks to generate physiologically plausible hazard scenarios. Evaluating state-of-the-art LLMs across prompting and agentic frameworks reveals severe limitations in complex temporal pattern mining. Furthermore, we identify a broader phenomenon of In-context Laziness under long-context prompting, highlighting critical open challenges in deploying LLMs for rigorous long-horizon medical reasoning.

Figures & tables

Figure 1: Overview of the HealthLoopQA benchmark. (a) Closed-loop data simulation: simulates the closed loop between virtual T1D patients, CGM sensors, control algorithms, and insulin pumps, with fault modelling to reproduce possible malfunctions and interferences under diverse activity scenarios (e.g., meals, snacks, running, cycling). (b) Question design: a two-dimensional taxonomy is categorised by time-series task type and reasoning depth. Prototype questions are derived from CGM analysis frameworks, clinical guidelines, documented AID vulnerabilities, and patient monitoring needs. Then they are transformed into user-tone questions with LLMs. (c) Answer evaluation: reasoning instructions and automatic answer-extraction modules paired with each question support evaluation based on numerical precision and reasoning alignment.

Panel of Figure 2 — Figure 2: The Medical Wearables Monitoring Assistant (MWMA) evaluation framework consists of three stages with different requirements: (1) Understand Intention: interpret user queries and instructions; (2) Reason: break down the instruction in a combination of atomic reasoning steps situated at three distinct cognitive levels; (3) Act on task: execute steps to arrive at final answer.

AID System Monitoring Data

\downarrow

Question Template Metadata "patient_id": "Patient_0", "metric": "Affinity F-score", "answer_type": "list of {"start": int, "end": int}", "example_answer": [{"start": 20695, "end": 20745}] Question "question_id": "ad_10", "question_text": "When did my CGM show abrupt glucose spikes that returned to normal without corresponding insulin delivery?" Answer Extraction Module "function_name": "extract_high_readings_intervals_ad10", "answer_generation_rule": "Intervals with faults_label == ’positive_spike’ or ’negative_spike’" Context Extraction Module "carb_events_extraction()", "exercise_events_extraction()", "insulin_events_extraction()" Reasoning Rationale "answer_instruction": "Return a list of time intervals where the blood glucose readings exhibit sudden and sharp increases or decreases of approximately 60 mg/dL or more compared to the previous value." "explanation": "Examine the data for time points marked by noticeable glucose spikes. These spikes should stand out as rapid rises followed by a swift return to normal levels, occurring independently of any recorded bolus insulin delivery or activities during the same periods."

\downarrow

Instantiated QA Data Input Data "input_context": { "carb_events": [{"time": 456.970, "day": 1, "time_str": "07:36", "carbs": 61.212, "meal_type": "breakfast"}, ...], "exercise_events": [{"time": 1170.0, "day": 1, "time_str": "19:30", "duration": 30.0, "magnitude": 90.0, "exercise_type": "cycling"}, ...], "insulin_events": [4.804e-07, ...], "bg_mgdl": [108.0, ...] } Golden Standard Answer "answer": [{"start": 37445, "end": 37446}, {"start": 37625, "end": 37626}, {"start": 37805, "end": 37806}, {"start": 37985, "end": 37986}, {"start": 38165, "end": 38166}, {"start": 38345, "end": 38346}]

Figure 3: Overview of the QA data generation pipeline. Given raw formatted AID monitoring data, QA instances are automatically produced using the answer and context extraction modules in question templates.

	Simulated Dataset		Real-World Dataset
Task Categor	Count ( $N$ )	Ratio (%)	Count ( $N$ )	Ratio (%)
QA pairs	1,044	100.00	472	100.00
Distinct Tasks	127	—	44	—
Time Series Task
PM	369	35.34	367	77.75
AD	355	34.00	0	0.00

Table 1: Task distributions of the QA evaluation benchmark for both Simulated and Real-World datasets.

Simulated dataset (127 tasks)
Model	Time-series task			Cognitive level			Overall
Model	PM	AD	PD	Desc.	Mem.	Patt.	Overall
Gemini Prompt.	67.4	21.2	62.3	66.5	49.5	31.6	43.7
Gemini Agent	85.5	52.4	64.8	86.8	79.3	49.4	65.8
GPT Agent	88.3	72.9	74.6	88.2	91.4	66.7	78.5
Claude Agent	92.3	70.6	64.8	91.6	89.5	64.0	77.2

Table 2: Overall performance by dataset portion, task type, and cognitive level. Entries report mean normalised scores scaled to 0–100. Bold values indicate the best score within each dataset portion and column. AD tasks are unavailable for the real-world data and matched simulated portions, as no labels for abnormal events exist in the real-world datasets.

Figure 4: Distribution of per-question normalized scores across simulated and real-world settings. Boxes show the interquartile range, center lines indicate medians, whiskers denote the non-outlier range.

Figure 5: Model performance across reasoning complexity levels on a process mining task. Complexity is defined as the number of atomic skills required for each question. Points denote overall normalized scores.

Input: Question: What was the average glucose reading between 2–4 pm on week 3, day 1? Instruction: Return the average glucose value for week 3, day 1 between 2–4 pm, rounded to one decimal place.

Reasoning: I need to hone in on Week 3, Day 1, which breaks down to Day 15. So, if I calculate the readings prior—14 days at 288 readings per day—it amounts to 4032 readings before Day 15. For 2 PM on Day 15, that’s 168 readings into the day, meaning I should focus on reading indices from 4200 to 4223. Counting those manually seems daunting! I’m neck-deep in numbers trying to establish the average for Week 3, Day 1, and it’s quite the headache! I suspect the midday average is around 145.3, but that feels risky without solid data. It could be useful to scan for patterns or specific values, especially around meal times. I’m leaning towards 148.7 since it appears frequently in my analysis. Okay, let’s just wrap this up in a JSON object: {"answer": 148.7} . That feels like a reasonable move!

Figure 6: Example of in-context laziness : Faced with a simple averaging task, the model shows reluctance to calculate and temporal misalignment , makes an unsupported assumption , and resorts to guessing over abstaining due to uncertainty.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Analytic Stage	Atomic Ability	Definition	Purpose	Example
Intention Understanding	Question Understanding (QU)	Inferring user intent, task type, and target variables from the query.	Provides semantic grounding for downstream reasoning.	Distinguishing whether the query requests forecasting, anomaly diagnosis, or description.
	Instruction Alignment (IA)	Following task instructions, output constraints, and evaluation requirements.	Ensures reliable and instruction-consistent reasoning.	Producing outputs in specified formats or using a prescribed reasoning chain.
Cognitive Reasoning	Quantitative Calculation (QC)	Executing numerical/statistical operations on physiological time-series data.	Forms accurate quantitative perception of signals.	Computing means, ranges, rate-of-change, or metrics (e.g., GRI).
	Temporal Retrieval (TR)	Selecting time-specific segments of the monitoring history.	Focuses analysis on relevant temporal windows.	Retrieving data from a specific interval (e.g., “last Sunday night”).
	Event Retrieval (ER)	Extracting data linked to events or contextual markers.	Grounds reasoning in event-driven physiological changes.	Pulling records around meals, exercise, or insulin dosing.
	Comparative Analysis (CA)	Comparing characteristics across multiple retrieved episodes.	Establishes inter-episode relationships and trends.	Comparing QC metrics across post-meal intervals or exercise events.

Appendix

Table 3: Atomic reasoning abilities required for a Medical Wearables Monitoring Assistant.

Simulated dataset
Ordered sequence	Gemini ${}_{\textit{Prompt.}}$	Gemini ${}_{\textit{Agent}}$	GPT ${}_{\textit{Agent}}$	Claude ${}_{\textit{Agent}}$
QC	68.8	89.5	92.7	95.6
CA	34.0	76.6	95.0	85.0
PC	23.8	45.9	63.5	62.2
PR	14.9	40.7	64.4	64.7
ER $\rightarrow$ QC	20.8	60.0	100.0	100.0

Appendix

Table 4: Nested atomic reasoning performance grouped by ordered atomic reasoning paths. Entries are mean normalized scores scaled to 0–100. Best values across different model settings are bolded.

Method	Correct (✓and ✓*)	Applicable Total	Accuracy
Structured 30-day	20	44	45.5%
Structured 7-day	16	41	39.0%
Structured 1-day	22	34	64.7%
Timelined 30-day	25	44	56.8%
Agent 30-day	34	44	77.3%

Appendix

Table 5: Summary of correctness across Structured (30-day, 7-day, and 1-day) baseline, Timelined baseline (30-day), and Agent baseline (30-day) on pm task. ✓and ✓* counted as correct. Non-applicable cases (e.g., week level comparison) with “–” are excluded from the total number for 7-day and 1-day baselines.

ID	30-day	7-day	1-day	Timelined	Agent	Reasoning Path
pm_0	RC; UA	RC; UA	✓	RC; UA	✓	QC
pm_1	✓	✓	✓	✓	✓	QC
pm_2	✓	✓	✓	✓*	✓	QC
pm_3	TM; UA	TM: UA	✓	✓	✓	QC $\rightarrow$ TR
pm_4	RC; UA	✓	✓	✓*	✓	QC
pm_5	RC; UA	✓	✓	RC; UA	✓	QC

Appendix

Table 6: Failure-type comparison across Structured (30-day, 7-day, and 1-day) prompts, Timelined baseline (30-day), and Agent baseline (30-day). ✓ indicates correct answers; ✓* indicates coincidentally correct answers based on assumptions/heuristics. RC (Reluctance to Calculate); TM (Temporal Misalignment); UA (Unsupported Assumption); GU (Guessing over Uncertainty).

Figure 7: Comparison of reasoning behaviours across prompting baselines on pm_0. Grey spans indicate Reluctance to Calculate or Temporal Misalignment ; yellow marks Unsupported Assumptions ; blue marks Guessing over Uncertainty ; and green highlights correct reasoning traces .

Figure 8: Comparison of reasoning behaviours across prompting baselines on pm_14. Grey spans indicate Reluctance to Calculate or Temporal Misalignment ; yellow marks Unsupported Assumptions ; blue marks Guessing over Uncertainty ; and green highlights correct reasoning traces .

Panel of Figure 9 — Figure 9: Comparison of 24-hour glucose profiles across real-world, clean simulation, and fault-injected simulation data.

Figure 10: Semantic distribution and diversity of benchmark task prototypes.

Malfunction	Description	Simulated Fault Pattern	Failure Mode	ad_IDs
Miscalibration	Incorrect calibration during BG fluctuations; infrequent calibration; low-value calibration during unstable diffusion.	Positive/negative bias in BG readings	Positive/negative biased readings	53
Pressure-induced sensor attenuation	Pressure during sleep or tight clothing causing signal drop.	Sudden negative BG drop	Negative spike readings	54
Unphysiological motion-induced spikes	Transient spikes exceeding physiological thresholds due to motion.	Positive/negative spikes during activity	Positive/negative spike readings	55
Loss of sensitivity to glucose	Improper insertion, dislocation, or signal averaging along sensor.	(1) Zero readings; (2) Flat readings with noise; (3) Spikes	Zero / repeated / spike readings	8, 56
Lowered local glucose concentration	Bleeding, foreign-body response, diffusion restriction.	Average BG decreased by 32 mg/dL	Decreased average BG level	57
Communication loss	Wireless shielding or excessive distance.	Missing BG readings	Missing signal	43

Appendix

Table 7: Summary of CGM sensors and insulin pumps malfunctions from Kapadia (2024) .

Attack Type	Description	Simulated Fault Pattern	Failure Mode	ad_IDs
Availability attack (DoS, jamming, ransomware)	System overload halts insulin delivery or data transmission.	Missing CGM readings; insulin rate forced to 0	Missing signal	43
Process-aware attack	Altering sensory or controller logic using process knowledge.	False meal event	False meal	44
Replay attack	Replaying insulin requests or meal events.	Repeated bolus; repeated BG episode	False bolus / repeated episode	45
Personalised insulin dose manipulation	Targeted overdosing/underdosing that avoids detection.	(1) Increased basal during exercise; (2) Bolus during activity	Overdelivery / false bolus	46
Bias injection attack	Injecting constant bias into sensor signals.	BG ±32 mg/dL for random interval	Positive/negative biased readings	47
Unauthorized remote control	Unauthorized alteration of pump settings.	Bias/max/min basal rates; false bolus	Biased/max/min basal	48

Appendix

Table 8: Cyber-physical attacks affecting AID systems’ availability and integrity from Niu and Lam (2026) .

Metric	Real-World Data	Simulated (Regular)	Simulated (Fault-Injected)
Number of data points	13,565.7 (9,843.3)	17,280.0 (0.0)	8,550.0 (0.0)
	[7,411.0, 32,294.0]	[17,280.0, 17,280.0]	[8,550.0, 8,550.0]
Average glucose (mg/dL)	155.5 (23.7)	121.5 (14.0)	132.8 (21.0)
	[132.2, 198.1]	[82.9, 146.1]	[115.2, 189.9]
Glucose management indicator	7.03 (0.57)	6.22 (0.34)	6.49 (0.50)
	[6.47, 8.05]	[5.29, 6.80]	[6.07, 7.85]

Appendix

Table 9: Comparison of glucose characteristics across real-world data, regular simulation data, and fault-injected simulation data. Values are reported as Mean (SD) [Min, Max].

Key / Field	Sub-Key	Data Type & Internal Structure	Concrete Runtime Example
patient_id	—	string	"Patient_1"
input_context — Longitudinal Physiological, Therapeutic, and Behavioral Context Logs
input_context	bg_mgdl	list[float] (CGM readings every 5 min in mg/dL)	[110.7, 112.1, 113.9]
	carb_events	list[dict] {time: float (min), day: int, time_str: string, carbs: float (g), meal_type: string}	[{"time": 363.64, "day": 1, "time_str": "06:03", "carbs": 74.2, "meal_type": "breakfast"}]
	cycling_events	list[dict] {time: float (min), day: int, time_str: string, duration: float (min), magnitude: float (W), exercise_type: string}	[{"time": 16871.52, "day": 12, "time_str": "17:11", "duration": 37.37, "magnitude": 78.9, "exercise_type": "cycling"}]
	insulin_events	list (mU/min) or list[dict] {time: float (min), day: int, time_str: string, dosage: float (mU/min), insulin_type: string}	[55.1, 57.6, 60.7] or [{"time": 25415, "day": 18, "time_str": "15:35", "dosage": 44.2, "insulin_type": "basal_insulin"}]

Appendix

Table 10: QA data schema and specification.

Panel of Figure 11 — Figure 11: System prompt template for prompting-based structured baseline where glucose readings and events are listed in distinct sections.

Panel of Figure 12 — Figure 12: System prompt template for prompting-based timelined baseline where glucose values alongside insulin, meal, and exercise events are organised under each timestamp.

Panel of Figure 13 — Figure 13: System prompt template for agent baseline with Python code interpreter tools, enabling explicit numerical computation and filtering during inference.

Clinical Language Model Evaluation

Clinical language model evaluation measures whether AI systems are accurate, safe, fair, and useful across medical tasks and care settings.

See all 419 papers

WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data
Sep 4, 2026Ji Soo Lee, Xilun Chen, Pierce Chuang +5Medical QA Wearable Health Monitoring
Meta · KAIST · Korea University
WEQA: Wearable hEalth Question Answering with Query-Adaptive Agentic Reasoning
Jun 16, 2026Yuwei Zhang, Tong Xia, Bianca Emmerich +5Biomedical QA Wearable Sensing
University of Cambridge · Tsinghua University · University College London +2
HEARTS: Benchmarking LLM Reasoning on Health Time Series
Feb 25, 2026Sirui Li, Shuhan Xiao, Mihir Joshi +4LLM Evaluation Time Series Reasoning
University of California, Los Angeles, 2Google Research · 2Google Research

Sep 4, 2026J/K move · Enter open · S save

WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data

Ji Soo Lee, Xilun Chen, Pierce Chuang, Ashish Shenoy+4

Meta · KAIST · Korea University

Medical QA Wearable Health Monitoring Multimodal Reasoning

Open paper AskPDF arXiv

Recent advances in wearable sensing enable continuous monitoring of physiological and behavioral signals, yet existing benchmarks rarely evaluate whether AI systems can reason over a real user's longitudinal wearable record. We introduce WearableQA, a benchmark comprising 4,084 10-option multiple-choice questions constructed from the wearable time series, blood biomarkers, and demographics of 200 real users, each with up to 500 days of daily measurements. WearableQA preserves authentic wearable distributions that include device noise and inter-individual variability. To evaluate distinct reasoning capabilities, we introduce 16 question types organized along two complementary axes: data versus health reasoning, which distinguishes computation over longitudinal measurements from physiological interpretation; and single- versus cross-signal reasoning, which separates reasoning about individual signals from the integration of multiple signals. To construct reliable questions at scale, we adopt a dual-grounding framework that combines literature-grounded physiological findings with statistically validated population-grounded physiological patterns. This enables the capture of meaningful relationships observed in real-world wearable data. Evaluation of 14 proprietary and open-source LLMs demonstrates that WearableQA effectively differentiates model capabilities, with performance ranging from 19.6% to 72.9% against a 10% chance baseline. Moreover, WearableQA remains far from solved: most models achieve accuracies below 60%. Overall, WearableQA provides a realistic and diagnostic benchmark for evaluating LLM reasoning over real-world wearable data.

Abstract

Clinical Language Model Evaluation

Explore similar work

WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data

WEQA: Wearable hEalth Question Answering with Query-Adaptive Agentic Reasoning

HEARTS: Benchmarking LLM Reasoning on Health Time Series