InclusionMed: Benchmarking AI for Inclusive Healthcare
Abstract
Advancing medical AI for inclusive healthcare requires evaluation across diverse healthcare roles, tasks, and care settings, but benchmark coverage remains uneven. We introduce InclusionMed, an open medical AI evaluation initiative for inclusive healthcare. The InclusionMed Atlas organizes benchmarks across three healthcare roles and three AI capabilities. For the first release, we selected four public benchmarks from 388 candidates through structured audits and independent reproduction. With InclusionMed-DxFiles, our private diagnostic reasoning benchmark, the Atlas contains five benchmarks covering three of its nine role--capability cells. We evaluated frontier large language models on the Atlas and aggregated their results into the InclusionMed Index, with overall scores ranging from 17.5 to 49.1 out of 100. The Index also reports scores by healthcare role and AI capability, showing that a model's rank varied with the role or capability considered. A within-benchmark analysis using task-level tags showed that the ten highest-ranked models in the Index all scored higher on patient communication than on clinical decision-making. Models with similar Index scores also differed substantially in estimated inference cost. These findings highlight the need for evaluation covering more diverse healthcare needs and for context-specific model selection. Through open collaboration and community contributions, InclusionMed seeks to ensure that diverse healthcare needs worldwide shape how medical AI progress is defined and measured.
Figures & tables
| Benchmark | Tasks | Runs/task | Metric | Weight |
|---|---|---|---|---|
| HealthBench Professional | 525 | 8 | Length-adjusted rubric score | 20% |
| PhysicianBench | 100 | 1 | Pass@1 | 20% |
| ALE-CLI v1.0: Health and Medicine | 22 | 1 | Task score (partial credit) | 20% |
| CHI-Bench | 75 | 3 | Pass@1 | 20% |
| InclusionMed-DxFiles | 312 | 1 | Perfect-grade rate | 20% |
| Healthcare role | AI capability | |||||
|---|---|---|---|---|---|---|
| Model | Patients and the public | Healthcare professionals | Healthcare administration professionals | Knowledge | Reasoning | Agency |
| Gemini 4 Argon | — | 55.0 | 25.3 | — | 54.2 | 45.7 |
| Claude Opus 5.5 | — | 55.6 | 22.2 | — | 55.4 | 44.7 |
| Claude Fable 5.1 | — | 54.8 | 25.3 | — | 53.3 | 46.1 |
| GPT-6 Astra | — | 52.4 | 16.0 | — | 58.7 | 36.1 |
| GPT-6.1 Sol | — | 52.1 | 14.2 | — | 58.3 | 35.4 |
| Clinical specialty | Medical expertise | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Internal medicine specialties | Surgical specialties | Obstetrics and gynecology | Pediatrics | Psychiatry | General practice | Other specialties | Medical understanding | Clinical decision- making | Safety and ethics | Evidence- based medicine | Patient communication | Medical calculation |
| Gemini 4 Argon | 56.6 | 52.3 | 60.3 | 51.1 | 53.5 | 64.2 | 63.2 | 60.4 | 52.8 | 51.0 | 57.0 | 64.4 | 54.0 |
| Claude Opus 5.5 | 63.6 | 56.4 | 67.0 | 49.3 | 64.6 | 58.3 | 64.1 | 62.7 | 59.5 | 58.9 | 61.0 | 69.8 | 48.4 |
| Claude Fable 5.1 | 60.8 | 54.8 | 64.3 | 52.5 | 45.4 | 54.0 | 61.6 | 60.6 | 57.5 | 48.4 | 62.8 | 68.4 | 51.1 |
| GPT-6 Astra | 65.0 | 59.0 | 74.0 | 56.5 | 70.2 | 68.5 | 69.1 | 63.7 | 63.6 | 62.5 | 62.4 | 82.0 | 56.3 |
| GPT-6.1 Sol | 64.8 | 57.5 | 73.0 | 54.7 | 70.0 | 67.1 | 69.3 | 64.6 | 62.9 | 59.9 | 63.1 | 78.5 | 51.8 |
| Healthcare workflow | |||||
|---|---|---|---|---|---|
| Model | Treatment planning and implementation | Research and teaching | Clinical assessment and diagnosis | Healthcare services and management | Follow-up and monitoring |
| Gemini 4 Argon | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘553.8 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘8 | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘555.4 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘4 | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘559.4 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘4 | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘774.6 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘6 | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘661.8 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘8 |
| Claude Opus 5.5 | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘558.4 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘4 | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘660.0 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘0 | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘662.9 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘9 | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘884.5 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘5 | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘885.2 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘2 |
| Claude Fable 5.1 | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘555.5 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘5 | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘557.4 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘4 | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘660.8 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘8 | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘778.0 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘0 | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘770.1 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘1 |
| GPT-6 Astra | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘663.3 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘3 | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘661.4 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘4 | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘667.7 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘7 | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘991.2 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘2 | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘772.3 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘3 |
| GPT-6.1 Sol | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘662.1 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘1 | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘662.4 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘4 | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘664.4 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘4 | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘888.3 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘3 | \XeTeXglyphbounds 1 \XeTeXcharglyph ‘663.5 \XeTeXglyphbounds 3 \XeTeXcharglyph ‘5 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Family | Releases |
|---|---|
| Claude | Opus 5.5, Fable 5.1 |
| DeepSeek | V4.1 Flash 1017, V4 Pro 0813 |
| Gemini | 4 Argon, 3.8 Flash |
| GLM | 5.3, 5.3 Flash |
| GPT | 6 Astra, 6.1 Sol, 6 Luna, 5.6 Luna |
| Grok | 4.3, 4.7 |
| Benchmark | Model | Harness | Version / Tag of Harness |
| CHI-Bench | GPT series | Codex | 0.128.0 |
| Gemini series | Gemini CLI | 0.67.0 | |
| Other models | Claude Code | 2.1.123 | |
| ALE-CLI v1.0: Health and Medicine | GPT series | Codex | 0.0.0-agenthle-20260614 |
| Gemini series | Gemini CLI | 0.38.8-agenthle | |
| Claude Opus 5.5 | Claude Code | 2.1.280 |
| Model | Input | Cache hit | Output | Cache write |
|---|---|---|---|---|
| Claude Opus 5.5 | 4.00 | 0.20 | 20.00 | 5.00 |
| Claude Fable 5.1 | 10.00 | 0.25 | 50.00 | 12.50 |
| DeepSeek V4.1 Flash | 0.30 | 0.006 | 1.20 | — |
| DeepSeek V4 Pro | 1.32 | 0.044 | 3.96 | — |
| Gemini 4 Argon | 2.00 | 0.10 | 10.00 | — |
| Gemini 3.8 Flash | 0.75 | 0.075 | 3.75 | — |
| Category | Gemini 4 Argon | Claude Opus 5.5 | Claude Fable 5.1 | GPT-6 Astra | GPT-6.1 Sol | DeepSeek V4.1 Flash | Grok 4.7 | Kimi K3.1 Pro | Qwen3.8 Max | Kimi K3 | Median |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Physiological system | |||||||||||
| Nervous system | 58.1 | 64.4 | 57.0 | 67.8 | 67.9 | 50.9 | 62.3 | 59.3 | 42.1 | 48.9 | 58.7 |
| Sensory system | 61.3 | 66.1 | 64.5 | 60.3 | 58.7 | 56.6 | 55.2 | 68.7 | 55.7 | 58.3 | 59.5 |
| Cardiovascular system | 51.6 | 57.5 | 59.0 | 60.5 | 57.7 | 47.7 | 53.1 | 51.6 | 37.6 | 49.8 | 52.4 |
| Respiratory system | 59.8 | 59.8 | 61.8 | 62.7 | 62.2 | 52.2 | 57.5 | 49.8 | 41.8 | 46.4 | 58.6 |
| Digestive system | 41.2 | 51.0 | 45.8 | 56.6 | 58.6 | 40.1 | 43.3 | 47.5 | 23.9 | 37.3 | 44.5 |
| Model | Treatment planning and implementation | Clinical assessment and diagnosis | Overall score |
|---|---|---|---|
| Gemini 4 Argon | 56.5 | 85.7 | 59.0 |
| Claude Opus 5.5 | 60.9 | 57.1 | 61.0 |
| Claude Fable 5.1 | 60.9 | 71.4 | 62.0 |
| GPT-6 Astra | 44.6 | 57.1 | 46.0 |
| GPT-6.1 Sol | 39.1 | 42.9 | 40.0 |
| DeepSeek V4.1 Flash | 53.3 | 71.4 | 55.0 |
| Model | Data retrieval | Clinical reasoning | Action execution | Documentation |
|---|---|---|---|---|
| n = 103 | n = 356 | n = 112 | n = 99 | |
| Gemini 4 Argon | 1.9% | 10.1% | 13.4% | 15.2% |
| Claude Opus 5.5 | 4.9% | 12.6% | 13.4% | 10.1% |
| Claude Fable 5.1 | 2.9% | 9.6% | 10.7% | 13.1% |
| GPT-6 Astra | 1.9% | 14.9% | 19.6% | 11.1% |
| GPT-6.1 Sol | 3.9% | 14.9% | 19.6% | 13.1% |
Clinical Language Model Evaluation
Clinical language model evaluation measures whether AI systems are accurate, safe, fair, and useful across medical tasks and care settings.
See all 461 papers