HealthCLBench:Evaluating Context Learning in Health Conversations through Controlled Case Clusters
Abstract
Complex health conversations require answers grounded in individual information and currently valid evidence. We evaluate health context learning: acquiring, integrating, and applying relevant context during inference with fixed model parameters to answer health questions and respond appropriately to changes in facts, evidence, or task goals. To our knowledge, Chinese-language medical benchmarks have yet to systematically combine explicit contextual difficulties, controlled case variations, and member-specific medical criteria for open-ended answers. We introduce HealthCLBench, comprising 328 controlled clusters, 1,103 cases, and 4,931 rubric criteria across medical consultation and everyday health management. Members share backgrounds while evidence, patient states, or task goals change; each has its own medical scoring requirements. A 70-person medical team annotates and reviews the simulated cases and their rubrics. Human review of 700 full-context responses from the original evaluation yields 90.5% pooled agreement with the original automated judgments at the rubric-item level. We report full-context and query-only results for seven general-purpose models and Ant Afu. Across the eight systems, full- query differences range from 11.83 to 31.76 case points and from 10.87 to 38.97 geometric cluster points. GPT-6-Astra leads both full-context metrics at 75.03 and 60.42, followed by Claude-Opus-5-5 and Ant Afu. Cross-condition differences describe score changes under the reported evaluation settings. Response comparisons reveal failures in checking historical information, expressing uncertainty when warranted, and ensuring recommendations respect required conditions. The resource supports model comparison and targeted follow-up evaluation.
Figures & tables
| Resource | Tasks and relations | Responses and scoring |
|---|---|---|
| HealthBench [ 5 ] | Health conversations; includes context awareness | Open-ended final responses; conversation-specific rubrics |
| CMB-Clin [ 41 ] | Chinese multi-turn clinical tasks; 74 cases, 208 questions | Open answers; fluency, relevance, completeness, and medical proficiency |
| ClinConsensus v6 [ 43 ] | 2,500 Chinese open-ended clinical cases | 30 binary criteria per case; physician-calibrated coverage thresholds |
| MedBench v4 [ 16 ] | Chinese medical tasks; includes long conversations and documents | Text, multimodal, and agent tracks; task-specific scoring |
| MedEinst [ 10 ] | Diagnostic pairs with changed discriminating evidence | Diagnostic labels; case and paired accuracy, bias trap rate |
| MedPIC-Bench [ 23 ] | Medication-safety pairs with changed patient information | Multiple choice; option exact match and pair accuracy |
| Member and current information | Response goal | Checkable rubric requirements (selected) |
|---|---|---|
| B1 : Adds parotid aspirate containing pus and a positive Staphylococcus aureus culture. | How to manage the condition. | P0: Identify suppurative bacterial parotid infection and distinguish it from viral mumps; prioritize clinician-directed antibiotics informed by culture or susceptibility. P1: Recommend follow-up for recurrent pus and possible further drainage. |
| B2 : Adds positive mumps-virus IgM. | Explain cause, management, and possible consequences. | P1: Treat IgM as supporting evidence, not standalone confirmation, and recommend clinical assessment; cover transmission precautions, supportive care, and complications; retain possible independent gum disease. |
| B3 : Adds neither test result; asks about care if symptoms persist until Friday. | Which department to visit first. | P0: Choose dentistry first if both symptoms persist. P1: Choose otolaryngology (ENT) if mainly throat or palate pain remains; seek earlier care for fever, marked facial swelling, or suddenly worsening pain before Friday. |
| Category | Definition / primary task | Example summary: input and task |
| Medical scenarios (S1–S5) | ||
| Medical Explanation | Explain diseases, medications, or medical concepts in relation to the request. | S1 : Given C-reactive protein of 71.32 mg/L, explain the value and whether it alone identifies a specific disease. |
| Health Assessment | Integrate symptoms, history, and reports to assess individual health status. | S2 : Earlier turns report about three days of nasal congestion, concurrent headache, both clear and yellow discharge, and no allergic rhinitis history. The current turn adds cough and sore throat for continued assessment. |
| Clinical Decision-Making | Address medication, treatment, discontinuation, switching, or choices among plans. | S3 : A breastfeeding user finds codeine phosphate on a cough syrup label and asks whether it can be taken. |
| Care Triage | Assess risk and determine whether, where, and in which department to seek care. | S4 : Outer lower-front gum pain and swelling and left-sided swallowing pain persist. The user asks which department to visit first if still unwell on Friday. |
| Health Management | Support diet, exercise, sleep, chronic care, and follow-up. | S5 : After five sets of 20 squats on Sunday, leg pain and tenderness on days two and three restrict sitting, squatting, and walking. The user asks whether to continue lower-body training. |
| Statistic | Value |
|---|---|
| Clusters | 328 |
| Cases | 1,103 |
| Members per cluster: median [Q1,Q3] | 3.5 [3,4] |
| Members per cluster: minimum–maximum | 1–5 |
| Rubric criteria | 4,931 |
| P0 / P1 / P2 criteria | 1,028 / 3,648 / 255 |
| Judgment axis | Conditions to control or distinguish | Implications for response requirements |
|---|---|---|
| Diagnostic thresholds and corrected values | The 75 g OGTT protocol at 24–28 weeks of gestation, measurement times, units, and corrected values | Reaching any corresponding threshold can change the GDM determination. If corrected values do not cross the relevant thresholds and the other applicable conditions remain unchanged, the core determination is preserved. Meeting diagnostic criteria and initiating insulin are separate questions [ 28 , 2 ] . |
| Infection etiology and management | Parotid secretion cultures, identified pathogens, and the evidential status of serum IgM | Bacterial culture, identification, and antimicrobial susceptibility testing inform antibiotic treatment. Viral IgM provides supporting evidence rather than a definitive diagnosis on its own; management also considers transmission prevention and supportive care [ 11 , 7 , 8 ] . |
| Medication ingredients and lactation contraindications | The complete formulation, specific product, and applicable local labeling | Avoid codeine-containing products during breastfeeding. Chinese labeling contraindicates oral single-ingredient dextromethorphan hydrobromide preparations during lactation; low milk exposure or absence of alcohol does not override this restriction [ 39 , 31 ] . |
| Input / metric | GPT-6- Astra | Claude- Opus-5-5 | Ant Afu | Qwen3.8- 27B | GLM-5.3 | Kimi-K3 | DeepSeek- V4.1-Flash | Gemini- 3.8-Flash |
|---|---|---|---|---|---|---|---|---|
| Full / Case | 75.03 | 68.21 | 61.99 | 57.31 | 52.29 | 48.46 | 45.71 | 42.77 |
| Full / | 60.42 | 49.50 | 43.30 | 36.44 | 29.20 | 24.77 | 23.67 | 21.79 |
| Query / Case | 44.16 | 36.46 | 39.69 | 40.52 | 33.68 | 31.66 | 33.01 | 30.95 |
| Query / | 21.45 | 14.78 | 17.75 | 17.03 | 12.33 | 9.74 | 12.43 | 10.92 |
| / Case | +30.87 | +31.76 | +22.30 | +16.79 | +18.61 | +16.80 | +12.71 | +11.83 |
| / | +38.97 | +34.72 | +25.56 | +19.41 | +16.87 | +15.03 | +11.24 | +10.87 |
| System | Reference | Variant 1 | Variant 2 | Task variant | |
|---|---|---|---|---|---|
| Breathing discomfort: symptom interpretation and test planning | |||||
| GPT-6-Astra | 73.33 | 100.00 | 60.00 | 100.00 | 81.44 |
| Claude-Opus-5-5 | 73.33 | 100.00 | 100.00 | 100.00 | 92.54 |
| Qwen3.8-27B | 33.33 | 0.00 | 60.00 | 83.33 | 0.00 |
| Diet and sleep: glucose evidence and fat-loss assessment | |||||
| GPT-6-Astra | 100.00 | 100.00 | 55.56 | 100.00 | 86.34 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Member | Translated current question | Sample scoring requirements, summarized |
|---|---|---|
| A1 Original values | “Please analyze this again; these are results from 25 weeks of gestation.” | Interpret 4.39/8.62/8.31 mmol/L using pregnancy-specific 75 g OGTT criteria and identify that all three are below their thresholds. Distinguish the report’s reference range; do not invent a borderline GDM diagnosis solely because a value is close to a threshold. |
| A2 Upward correction | “The hospital corrected the report. My 2-hour glucose after the glucose load in the OGTT at 25 weeks of gestation was actually 8.81 mmol/L, not the 8.31 reported before. Please analyze it again.” | Use 8.81 mmol/L, compare it with the 8.5 mmol/L threshold, identify GDM, and recommend obstetric monitoring and management. Do not assume the report belongs to a spouse solely because the profile is male. |
| A3 Downward correction | “Please analyze this again; these are results from 25 weeks of gestation, and the hospital corrected the report. The glucose value 2 hours after the glucose load was actually 8.21 mmol/L.” | Use 8.21 rather than 8.31 mmol/L, retaining the determination that the test does not meet GDM criteria. Verify report attribution and continue prenatal follow-up. |
| A4 Treatment question | “Based on these glucose tolerance results at 25 weeks of gestation, do I need to start insulin immediately?” | Do not self-initiate or adjust insulin based only on the original values. Compare all three values with their thresholds and have obstetric care determine treatment using subsequent monitoring. |
| Member | Translated current question | Core response required by the rubrics |
|---|---|---|
| B1 Bacterial evidence | “I went to the hospital today. They aspirated pus from my parotid gland, and the bacterial culture was positive for Staphylococcus aureus. How should this be managed?” | Identify suppurative bacterial infection and prioritize antibiotics selected by a clinician using culture or susceptibility results. Reassess recovery and reaccumulation of pus, with further drainage determined by the clinician if needed. |
| B2 Viral evidence | “I had a blood serology test at the hospital today, and the result was positive for mumps-virus IgM antibodies. What usually causes parotitis? How should it be managed? What effects can it have?” | Treat positive IgM as supporting evidence and evaluate it alongside symptoms. Explain transmission and complication risks, appropriate isolation, and supportive care; do not recommend antibiotics or drainage routinely. |
| B3 Care navigation | “Given the swelling and pain in my outer lower gum and the left-sided pain on swallowing, which department should I visit first if they have not improved by Friday?” | Visit dentistry first if both symptoms persist; consider otolaryngology if throat or palatal pain is the main remaining problem. Seek earlier care for fever, marked facial swelling, or sudden worsening pain. |
| Member | Translated current question | Core response required by the rubrics |
|---|---|---|
| C1 Unknown formula | “Can I take cough syrup while breastfeeding?” | Verify the product name, complete ingredients, and labeling before judging suitability. If it is a Chinese oral single-ingredient dextromethorphan hydrobromide product, identify the lactation contraindication; low milk exposure alone does not justify recommending use. |
| C2 Codeine | “I’m breastfeeding, and the ingredient list on the bottle includes codeine phosphate. Can I take this cough syrup?” | Advise against self-administration and recommend consulting a physician or pharmacist about a suitable option. If the product has already been taken and the infant develops unusual drowsiness, feeding difficulty, or abnormal breathing, advise immediate medical care. |
| C3 Dextromethorphan | “I’m breastfeeding and have a cough. I checked the bottle’s ingredient list: dextromethorphan is the only cough-suppressant ingredient, and there is no alcohol. Can I take it?” | State the Chinese lactation contraindication for oral single-ingredient dextromethorphan hydrobromide preparations and advise against self-administration. Verify the complete formula and labeling to determine whether this restriction applies to the user’s product. Alcohol absence, infant characteristics, or overseas milk-transfer data do not remove the contraindication; do not directly provide dosing or duration, or advise self-directed breastfeeding cessation. |
| Member / cluster | GPT-6- Astra | Claude- Opus-5-5 | Ant Afu | Qwen3.8- 27B | GLM-5.3 | Kimi-K3 | DeepSeek- V4.1-Flash | Gemini- 3.8-Flash |
|---|---|---|---|---|---|---|---|---|
| Case cluster A | ||||||||
| A1 | 100.00 | 100.00 | 0.00 | 66.67 | 100.00 | 66.67 | 100.00 | 100.00 |
| A2 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 |
| A3 | 55.56 | 100.00 | 0.00 | 55.56 | 100.00 | 55.56 | 55.56 | 55.56 |
| A4 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 55.56 | 100.00 |
| 86.34 | 100.00 | 0.00 | 78.01 | 100.00 | 78.01 | 74.54 | 86.34 | |
| Difficulty subcategory | Definition |
|---|---|
| Complex Context | |
| Multiple health subjects | The context concerns multiple individuals with distinct health information, requiring symptoms, histories, tests, or medications to be assigned to the correct person. |
| Interleaved topics | Multiple health topics run in parallel, alternate, pause, or recur, requiring the relevant topic thread to be tracked across the conversation. |
| Reference and ellipsis dependence | Pronouns, demonstratives, ellipsis, or continuation expressions depend on earlier context to recover the intended person, measurement, medication, test, event, or action. |
| Multiple questions in one query | A single query contains at least two questions, judgment goals, or task requirements that must be addressed separately. |
| Dynamic context updates | Later information explicitly corrects, changes, or supersedes an earlier fact, health state, plan, or conclusion, requiring identification of the currently valid information. |
| Departments (785 related cases) | Diseases (924 related cases) | ||||
|---|---|---|---|---|---|
| Category | Cases | Share (%) | Category | Cases | Share (%) |
| Endocrinology | 229 | 29.17 | Diabetes | 184 | 19.91 |
| Dermatology and Venereology | 209 | 26.62 | Hypertension | 155 | 16.77 |
| Gastroenterology | 205 | 26.11 | Bacterial Infection | 128 | 13.85 |
| Cardiovascular Medicine | 181 | 23.06 | Common Cold | 116 | 12.55 |
| Infectious Diseases | 168 | 21.40 | Anemia | 98 | 10.61 |
| Category | GPT-6- Astra | Claude- Opus-5-5 | Ant Afu | Qwen3.8- 27B | GLM-5.3 | Kimi-K3 | DeepSeek- V4.1-Flash | Gemini- 3.8-Flash | |
|---|---|---|---|---|---|---|---|---|---|
| Medical scenario | |||||||||
| Medical Explanation | 41 | 73.22 | 58.84 | 59.84 | 51.41 | 55.48 | 43.40 | 44.44 | 42.09 |
| Health Assessment | 498 | 73.52 | 68.83 | 61.81 | 57.45 | 50.91 | 45.94 | 44.25 | 40.47 |
| Clinical Decision-Making | 273 | 76.42 | 63.88 | 60.62 | 56.06 | 51.01 | 45.16 | 46.74 | 45.29 |
| Care Triage | 127 | 81.84 | 80.38 | 70.88 | 66.96 | 63.56 | 64.16 | 55.46 | 52.12 |
| Health Management | 164 | 72.50 | 66.48 | 58.49 | 52.93 | 49.08 | 50.67 | 41.22 | 38.53 |
| Category | GPT-6- Astra | Claude- Opus-5-5 | Ant Afu | Qwen3.8- 27B | GLM-5.3 | Kimi-K3 | DeepSeek- V4.1-Flash | Gemini- 3.8-Flash | |
|---|---|---|---|---|---|---|---|---|---|
| Department | |||||||||
| Endocrinology | 229 | 73.61 | 74.54 | 67.24 | 62.32 | 58.48 | 52.63 | 47.27 | 44.07 |
| Dermatology and Venereology | 209 | 77.56 | 70.09 | 66.40 | 57.22 | 53.43 | 50.86 | 45.57 | 45.86 |
| Gastroenterology | 205 | 71.82 | 68.52 | 61.42 | 56.40 | 52.83 | 49.20 | 44.69 | 42.56 |
| Cardiovascular Medicine | 181 | 71.24 | 66.84 | 61.87 | 57.92 | 53.18 | 48.63 | 45.51 | 41.96 |
| Infectious Diseases | 168 | 74.38 | 70.64 | 66.20 | 60.03 | 55.01 | 51.84 | 45.51 | 47.26 |
| Category | GPT-6- Astra | Claude- Opus-5-5 | Ant Afu | Qwen3.8- 27B | GLM-5.3 | Kimi-K3 | DeepSeek- V4.1-Flash | Gemini- 3.8-Flash | |
|---|---|---|---|---|---|---|---|---|---|
| Medical scenario | |||||||||
| Medical Explanation | 41 | +28.43 | +24.48 | +17.77 | +17.11 | +24.44 | +14.63 | +9.96 | +18.31 |
| Health Assessment | 498 | +30.29 | +33.95 | +23.10 | +17.69 | +18.72 | +16.48 | +12.08 | +11.45 |
| Clinical Decision-Making | 273 | +32.68 | +25.17 | +20.94 | +14.30 | +16.02 | +13.31 | +15.50 | +11.50 |
| Care Triage | 127 | +27.05 | +35.63 | +23.98 | +17.10 | +19.81 | +22.01 | +12.09 | +12.55 |
| Health Management | 164 | +33.17 | +34.86 | +22.02 | +17.87 | +20.22 | +20.09 | +11.10 | +11.34 |
Clinical Language Model Evaluation
Clinical language model evaluation measures whether AI systems are accurate, safe, fair, and useful across medical tasks and care settings.
See all 461 papers