HealthLoopQA: A Context-Aware Question Answering Benchmark for Interpreting Wearable Monitoring Data in Diabetes Care
Authors: Yuchen Niu, Yanan Ma, Srinivasan Nandakumar, Maolin Chen, Viktor Schlegel, Kexin Wei, Ling Cheng, Anna Bird, Anil Anthony Bharath, Siew-Kei Lam
Organizations: Nanyang Technological University · Mayo Clinic · The University of Manchester · Imperial College London, Imperial Global Singapore · National University of Singapore
As medical wearables become integrated into daily chronic disease care, effectively interpreting longitudinal monitoring data is essential for patients and clinicians to understand health trends, detect safety-critical events, and make informed decisions. While large language models (LLMs) show promise for transforming this streaming physiological data into personalized health insights, evaluating their reasoning capability and analytical rigor in diverse monitoring tasks remains a fundamental challenge. Existing medical wearable question answering (QA) benchmarks primarily assess short-horizon classification or statistical summaries, largely ignoring the long-term patterns, therapeutic and behavioural contexts, and potential system failures inherent in real-world deployments. To address this, we introduce HealthLoopQA, a comprehensive diagnostic benchmark for evaluating LLM reasoning over continuous diabetes monitoring data. Grounded in a novel taxonomy of eleven atomic reasoning abilities, HealthLoopQA comprises 127 tasks and over 1,500 QA instances spanning process mining, anomaly detection, and prediction over 30-day horizons. To systematically evaluate safety awareness, we complement real-world datasets with a fault-injected simulation testbed modeling diverse device malfunctions and cyber-physical attacks to generate physiologically plausible hazard scenarios. Evaluating state-of-the-art LLMs across prompting and agentic frameworks reveals severe limitations in complex temporal pattern mining. Furthermore, we identify a broader phenomenon of In-context Laziness under long-context prompting, highlighting critical open challenges in deploying LLMs for rigorous long-horizon medical reasoning.
Figures & tables
Figure 1: Overview of the HealthLoopQA benchmark. (a) Closed-loop data simulation: simulates the closed loop between virtual T1D patients, CGM sensors, control algorithms, and insulin pumps, with fault modelling to reproduce possible malfunctions and interferences under diverse activity scenarios (e.g., meals, snacks, running, cycling). (b) Question design: a two-dimensional taxonomy is categorised by time-series task type and reasoning depth. Prototype questions are derived from CGM analysis frameworks, clinical guidelines, documented AID vulnerabilities, and patient monitoring needs. Then they are transformed into user-tone questions with LLMs. (c) Answer evaluation: reasoning instructions and automatic answer-extraction modules paired with each question support evaluation based on numerical precision and reasoning alignment.
Figure 2: The Medical Wearables Monitoring Assistant (MWMA) evaluation framework consists of three stages with different requirements: (1) Understand Intention: interpret user queries and instructions; (2) Reason: break down the instruction in a combination of atomic reasoning steps situated at three distinct cognitive levels; (3) Act on task: execute steps to arrive at final answer.
AID System Monitoring Data
↓
Question Template Metadata "patient_id": "Patient_0", "metric": "Affinity F-score", "answer_type": "list of {"start": int, "end": int}", "example_answer": [{"start": 20695, "end": 20745}] Question "question_id": "ad_10", "question_text": "When did my CGM show abrupt glucose spikes that returned to normal without corresponding insulin delivery?" Answer Extraction Module "function_name": "extract_high_readings_intervals_ad10", "answer_generation_rule": "Intervals with faults_label == ’positive_spike’ or ’negative_spike’" Context Extraction Module "carb_events_extraction()", "exercise_events_extraction()", "insulin_events_extraction()" Reasoning Rationale "answer_instruction": "Return a list of time intervals where the blood glucose readings exhibit sudden and sharp increases or decreases of approximately 60 mg/dL or more compared to the previous value." "explanation": "Examine the data for time points marked by noticeable glucose spikes. These spikes should stand out as rapid rises followed by a swift return to normal levels, occurring independently of any recorded bolus insulin delivery or activities during the same periods."
Figure 3: Overview of the QA data generation pipeline. Given raw formatted AID monitoring data, QA instances are automatically produced using the answer and context extraction modules in question templates.
Simulated Dataset
Real-World Dataset
Task Categor
Count ( N )
Ratio (%)
Count ( N )
Ratio (%)
QA pairs
1,044
100.00
472
100.00
Distinct Tasks
127
—
44
—
Time Series Task
PM
369
35.34
367
77.75
AD
355
34.00
0
0.00
Table 1: Task distributions of the QA evaluation benchmark for both Simulated and Real-World datasets.
Model
Time-series task
Cognitive level
Overall
PM
AD
PD
Desc.
Mem.
Patt.
Simulated dataset (127 tasks)
Gemini Prompt.
67.4
21.2
62.3
66.5
49.5
31.6
43.7
Gemini Agent
85.5
52.4
64.8
86.8
79.3
49.4
65.8
GPT Agent
88.3
72.9
74.6
88.2
91.4
66.7
78.5
Claude Agent
92.3
70.6
64.8
91.6
89.5
64.0
77.2
Table 2: Overall performance by dataset portion, task type, and cognitive level. Entries report mean normalised scores scaled to 0–100. Bold values indicate the best score within each dataset portion and column. AD tasks are unavailable for the real-world data and matched simulated portions, as no labels for abnormal events exist in the real-world datasets.
Figure 4: Distribution of per-question normalized scores across simulated and real-world settings. Boxes show the interquartile range, center lines indicate medians, whiskers denote the non-outlier range.
Figure 5: Model performance across reasoning complexity levels on a process mining task. Complexity is defined as the number of atomic skills required for each question. Points denote overall normalized scores.
Input: Question: What was the average glucose reading between 2–4 pm on week 3, day 1? Instruction: Return the average glucose value for week 3, day 1 between 2–4 pm, rounded to one decimal place.
Reasoning: I need to hone in on Week 3, Day 1, which breaks down to Day 15. So, if I calculate the readings prior—14 days at 288 readings per day—it amounts to 4032 readings before Day 15. For 2 PM on Day 15, that’s 168 readings into the day, meaning I should focus on reading indices from 4200 to 4223. Counting those manually seems daunting! I’m neck-deep in numbers trying to establish the average for Week 3, Day 1, and it’s quite the headache! I suspect the midday average is around 145.3, but that feels risky without solid data. It could be useful to scan for patterns or specific values, especially around meal times. I’m leaning towards 148.7 since it appears frequently in my analysis. Okay, let’s just wrap this up in a JSON object: {"answer": 148.7} . That feels like a reasonable move!
Figure 6: Example of in-context laziness : Faced with a simple averaging task, the model shows reluctance to calculate and temporal misalignment , makes an unsupported assumption , and resorts to guessing over abstaining due to uncertainty.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Analytic Stage
Atomic Ability
Definition
Purpose
Example
Intention Understanding
Question Understanding (QU)
Inferring user intent, task type, and target variables from the query.
Provides semantic grounding for downstream reasoning.
Distinguishing whether the query requests forecasting, anomaly diagnosis, or description.
Instruction Alignment (IA)
Following task instructions, output constraints, and evaluation requirements.
Ensures reliable and instruction-consistent reasoning.
Producing outputs in specified formats or using a prescribed reasoning chain.
Cognitive Reasoning
Quantitative Calculation (QC)
Executing numerical/statistical operations on physiological time-series data.
Forms accurate quantitative perception of signals.
Computing means, ranges, rate-of-change, or metrics (e.g., GRI).
Temporal Retrieval (TR)
Selecting time-specific segments of the monitoring history.
Focuses analysis on relevant temporal windows.
Retrieving data from a specific interval (e.g., “last Sunday night”).
Event Retrieval (ER)
Extracting data linked to events or contextual markers.
Grounds reasoning in event-driven physiological changes.
Pulling records around meals, exercise, or insulin dosing.
Comparative Analysis (CA)
Comparing characteristics across multiple retrieved episodes.
Establishes inter-episode relationships and trends.
Comparing QC metrics across post-meal intervals or exercise events.
Appendix
Table 3: Atomic reasoning abilities required for a Medical Wearables Monitoring Assistant.
Ordered sequence
Gemini Prompt.
Gemini Agent
GPT Agent
Claude Agent
Simulated dataset
QC
68.8
89.5
92.7
95.6
CA
34.0
76.6
95.0
85.0
PC
23.8
45.9
63.5
62.2
PR
14.9
40.7
64.4
64.7
ER → QC
20.8
60.0
100.0
100.0
Appendix
Table 4: Nested atomic reasoning performance grouped by ordered atomic reasoning paths. Entries are mean normalized scores scaled to 0–100. Best values across different model settings are bolded.
Method
Correct (✓and ✓*)
Applicable Total
Accuracy
Structured 30-day
20
44
45.5%
Structured 7-day
16
41
39.0%
Structured 1-day
22
34
64.7%
Timelined 30-day
25
44
56.8%
Agent 30-day
34
44
77.3%
Appendix
Table 5: Summary of correctness across Structured (30-day, 7-day, and 1-day) baseline, Timelined baseline (30-day), and Agent baseline (30-day) on pm task. ✓and ✓* counted as correct. Non-applicable cases (e.g., week level comparison) with “–” are excluded from the total number for 7-day and 1-day baselines.
ID
30-day
7-day
1-day
Timelined
Agent
Reasoning Path
pm_0
RC; UA
RC; UA
✓
RC; UA
✓
QC
pm_1
✓
✓
✓
✓
✓
QC
pm_2
✓
✓
✓
✓*
✓
QC
pm_3
TM; UA
TM: UA
✓
✓
✓
QC → TR
pm_4
RC; UA
✓
✓
✓*
✓
QC
pm_5
RC; UA
✓
✓
RC; UA
✓
QC
Appendix
Table 6: Failure-type comparison across Structured (30-day, 7-day, and 1-day) prompts, Timelined baseline (30-day), and Agent baseline (30-day). ✓ indicates correct answers; ✓* indicates coincidentally correct answers based on assumptions/heuristics. RC (Reluctance to Calculate); TM (Temporal Misalignment); UA (Unsupported Assumption); GU (Guessing over Uncertainty).
Figure 7: Comparison of reasoning behaviours across prompting baselines on pm_0. Grey spans indicate Reluctance to Calculate or Temporal Misalignment ; yellow marks Unsupported Assumptions ; blue marks Guessing over Uncertainty ; and green highlights correct reasoning traces .
Figure 8: Comparison of reasoning behaviours across prompting baselines on pm_14. Grey spans indicate Reluctance to Calculate or Temporal Misalignment ; yellow marks Unsupported Assumptions ; blue marks Guessing over Uncertainty ; and green highlights correct reasoning traces .
Figure 9: Comparison of 24-hour glucose profiles across real-world, clean simulation, and fault-injected simulation data.
Figure 10: Semantic distribution and diversity of benchmark task prototypes.
Malfunction
Description
Simulated Fault Pattern
Failure Mode
ad_IDs
Miscalibration
Incorrect calibration during BG fluctuations; infrequent calibration; low-value calibration during unstable diffusion.
Positive/negative bias in BG readings
Positive/negative biased readings
53
Pressure-induced sensor attenuation
Pressure during sleep or tight clothing causing signal drop.
Sudden negative BG drop
Negative spike readings
54
Unphysiological motion-induced spikes
Transient spikes exceeding physiological thresholds due to motion.
Positive/negative spikes during activity
Positive/negative spike readings
55
Loss of sensitivity to glucose
Improper insertion, dislocation, or signal averaging along sensor.
(1) Zero readings; (2) Flat readings with noise; (3) Spikes
Table 7: Summary of CGM sensors and insulin pumps malfunctions from Kapadia (2024) .
Attack Type
Description
Simulated Fault Pattern
Failure Mode
ad_IDs
Availability attack (DoS, jamming, ransomware)
System overload halts insulin delivery or data transmission.
Missing CGM readings; insulin rate forced to 0
Missing signal
43
Process-aware attack
Altering sensory or controller logic using process knowledge.
False meal event
False meal
44
Replay attack
Replaying insulin requests or meal events.
Repeated bolus; repeated BG episode
False bolus / repeated episode
45
Personalised insulin dose manipulation
Targeted overdosing/underdosing that avoids detection.
(1) Increased basal during exercise; (2) Bolus during activity
Overdelivery / false bolus
46
Bias injection attack
Injecting constant bias into sensor signals.
BG ±32 mg/dL for random interval
Positive/negative biased readings
47
Unauthorized remote control
Unauthorized alteration of pump settings.
Bias/max/min basal rates; false bolus
Biased/max/min basal
48
Appendix
Table 8: Cyber-physical attacks affecting AID systems’ availability and integrity from Niu and Lam (2026) .
Metric
Real-World Data
Simulated (Regular)
Simulated (Fault-Injected)
Number of data points
13,565.7 (9,843.3)
17,280.0 (0.0)
8,550.0 (0.0)
[7,411.0, 32,294.0]
[17,280.0, 17,280.0]
[8,550.0, 8,550.0]
Average glucose (mg/dL)
155.5 (23.7)
121.5 (14.0)
132.8 (21.0)
[132.2, 198.1]
[82.9, 146.1]
[115.2, 189.9]
Glucose management indicator
7.03 (0.57)
6.22 (0.34)
6.49 (0.50)
[6.47, 8.05]
[5.29, 6.80]
[6.07, 7.85]
Appendix
Table 9: Comparison of glucose characteristics across real-world data, regular simulation data, and fault-injected simulation data. Values are reported as Mean (SD) [Min, Max].
Key / Field
Sub-Key
Data Type & Internal Structure
Concrete Runtime Example
patient_id
—
string
"Patient_1"
input_context — Longitudinal Physiological, Therapeutic, and Behavioral Context Logs
Figure 11: System prompt template for prompting-based structured baseline where glucose readings and events are listed in distinct sections.
Figure 12: System prompt template for prompting-based timelined baseline where glucose values alongside insulin, meal, and exercise events are organised under each timestamp.
Figure 13: System prompt template for agent baseline with Python code interpreter tools, enabling explicit numerical computation and filtering during inference.
Recent advances in wearable sensing enable continuous monitoring of physiological and behavioral signals, yet existing benchmarks rarely evaluate whether AI systems can reason over a real user's longitudinal wearable record. We introduce WearableQA, a benchmark comprising 4,084 10-option multiple-choice questions constructed from the wearable time series, blood biomarkers, and demographics of 200 real users, each with up to 500 days of daily measurements. WearableQA preserves authentic wearable distributions that include device noise and inter-individual variability. To evaluate distinct reasoning capabilities, we introduce 16 question types organized along two complementary axes: data versus health reasoning, which distinguishes computation over longitudinal measurements from physiological interpretation; and single- versus cross-signal reasoning, which separates reasoning about individual signals from the integration of multiple signals. To construct reliable questions at scale, we adopt a dual-grounding framework that combines literature-grounded physiological findings with statistically validated population-grounded physiological patterns. This enables the capture of meaningful relationships observed in real-world wearable data. Evaluation of 14 proprietary and open-source LLMs demonstrates that WearableQA effectively differentiates model capabilities, with performance ranging from 19.6% to 72.9% against a 10% chance baseline. Moreover, WearableQA remains far from solved: most models achieve accuracies below 60%. Overall, WearableQA provides a realistic and diagnostic benchmark for evaluating LLM reasoning over real-world wearable data.
Language models are remarkably capable at medical question answering, in some cases surpassing the accuracy of general physicians. However, answering questions about wearable health data remains challenging and understudied, as these ubiquitous sensors produce continuous, high-dimensional, and longitudinal data, which is non-trivial to align with text-centric distributions in LLM pretraining. The diversity of sensor modalities and user intents cannot be effectively handled by a fixed reasoning workflow or a single pretrained foundation model. To address these challenges, we propose WEQA, a query-adaptive agent framework that unifies LLM reasoning with specialized wearable analytical and modeling tools. An LLM controller is employed to synthesize execution plans and dynamically route each query to the appropriate combination of sensor analysis and pretrained models, and perform grounded response auditing with external knowledge. We also curate a benchmark spanning four open wearable datasets comprising analytic and predictive tasks in three different health domains. Experiments show that our framework is 24% more accurate than LLM and agentic baselines, and a blinded study with 12 medical experts and 8 users shows substantial gains in usefulness and clinical soundness.
Yuwei Zhang, Tong Xia, Bianca Emmerich +5
University of Cambridge · Tsinghua University · University College London +2
The rise of large language models (LLMs) has shifted time series analysis from narrow analytics to general-purpose reasoning. Yet, existing benchmarks cover only a small set of health time series modalities and tasks, failing to reflect the diverse domains and extensive temporal dependencies inherent in real-world physiological modeling. To bridge these gaps, we introduce HEARTS (Health Reasoning over Time Series), a unified benchmark for evaluating hierarchical reasoning capabilities of LLMs over general health time series. HEARTS integrates 16 real-world datasets across 12 health domains and 20 signal modalities, and defines a comprehensive taxonomy of 110 tasks grouped into four core capabilities: Perception, Inference, Generation, and Deduction. Evaluating 16 state-of-the-art LLMs on more than 20K test samples reveals intriguing findings. First, LLMs substantially underperform specialized models, and their performance is only weakly related to general reasoning scores. Moreover, LLMs often rely on simple heuristics and struggle with multi-step temporal reasoning. Finally, performance declines with increasing temporal complexity, with similar failure modes within model families, indicating that scaling alone is insufficient. By making these gaps measurable, HEARTS provides a standardized testbed and living benchmark for developing next-generation LLM agents capable of reasoning over diverse health signals.
Sirui Li, Shuhan Xiao, Mihir Joshi +4
University of California, Los Angeles, 2Google Research · 2Google Research