cs.CLOct 29, 2026
SaveDecision and System One models in rheumatology: accuracy and calibration on specialty-training examination questions and diagnostic case vignettes
Organizations: Independent researcher, Madrid, Spain · Rheumatology Department, Hospital Universitari de Bellvitge, Barcelona, Spain · Institute for Digital Medicine, Marburg University, Marburg, Germany · Division of Rheumatology and Systemic Inflammatory Disorders, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
Abstract
Objectives: To evaluate the accuracy and calibration of Decision and System One models on rheumatology examination questions and diagnostic vignettes. Methods: Two benchmarks were analysed: 172 rheumatology questions from Spanish specialty-training examinations (expanded RheumaMIR dataset) in Spanish and English, and 70 English diagnostic vignettes from RheumBench with 5-18 candidate diagnoses, 52 of which included the reference diagnosis. Four System One models (Jev 1.13, Clef, Microsoft-Decision-1, Laya) and GPT-6 Luna (OpenAI Decisions API) were compared with GPT-6 Sol and two local generative models. The primary outcome was accuracy; secondary outcomes included calibration, discrimination, accuracy of answers with probability >=0.90 and language differences. Results: In RheumaMIR, Jev's accuracy was 93.0% in Spanish and 91.9% in English, below GPT-6 Sol (99.4%; Holm-adjusted P<=0.005); GPT-6 Luna reached 97.7%, Clef 91.9-93.0% and Microsoft-Decision-1 86.0-87.8%, without detectable language differences. Jev's answers with probability >=0.90 (76-77% of answers) were 98.5% correct. In RheumBench, accuracy was lower: GPT-6 Sol 76.9%, GPT-6 Luna 63.5%, Clef 59.6%, Jev 50.0% and Microsoft-Decision-1 40.4% (each System One model below GPT-6 Sol, Holm-adjusted P<=0.045; chance 9.6%). Jev became overconfident (mean probability 0.659; over/underconfidence index +0.159), and its answers with probability >=0.90 (25% of cases) were 84.6% correct, against 95.5% for GPT-6 Luna (42%). Laya performed near chance in both benchmarks. Conclusion: The best-performing System One models were highly accurate on examination questions but less accurate on diagnostic vignettes, remaining below GPT-6 Sol. Jev's high-confidence predictions were less reliable on vignettes, highlighting the need for task-specific evaluation and calibration.
Figures & tables
Figure 1: How a large language model (LLM) and a System One model process the same clinical note. Diagram drafted with large language model assistance; content verified by the authors.
| Model | Spanish, n/N; % (95% CI) | English, n/N; % (95% CI) | English Spanish, pp (95% CI) | P (Holm) a |
| System One models | ||||
| Jev 1.13 | 160/172; 93.0 (88.2–96.0) | 158/172; 91.9 (86.8–95.1) | 1.2 ( 5.2 to 2.3) | 1.000 |
| GPT-6 Luna (Decisions API) | 168/172; 97.7 (94.2–99.1) | 168/172; 97.7 (94.2–99.1) | 0.0 ( 2.3 to 2.3) | 1.000 |
| Clef | 158/172; 91.9 (86.8–95.1) | 160/172; 93.0 (88.2–96.0) | +1.2 ( 2.9 to 5.2) | 1.000 |
| Microsoft-Decision-1 | 151/172; 87.8 (82.1–91.9) | 148/172; 86.0 (80.1–90.4) | 1.7 ( 6.4 to 2.3) | 1.000 |
| Laya 0.3.20 | 38/172; 22.1 (16.5–28.9) | 43/172; 25.0 (19.1–32.0) | +2.9 ( 5.2 to 11.0) | 1.000 |
Table 1: Accuracy by model (primary outcome).
| Model, language | Errors, n | Mean probability a | O/U index | ECE | Retained, n (%) | Accuracy of retained, % (95% CI) |
| System One models | ||||||
| Jev 1.13, Spanish | 12 | 0.915 | 0.015 | 0.030 | 130 (75.6) | 98.5 (94.6–99.6) |
| Jev 1.13, English | 14 | 0.919 | 0.000 | 0.010 | 132 (76.7) | 98.5 (94.6–99.6) |
| GPT-6 Luna (Decisions API), Spanish | 4 | 0.928 | 0.049 | 0.049 | 135 (78.5) | 99.3 (95.9–99.9) |
| GPT-6 Luna (Decisions API), English | 4 | 0.914 | 0.062 | 0.062 | 127 (73.8) | 100.0 (97.1–100.0) |
| Clef, Spanish | 14 | 0.898 | 0.021 | 0.035 | 129 (75.0) | 97.7 (93.4–99.2) |
Table 2: Probability quality and answers retained at a selected-answer probability of at least 0.90.
Figure 2: Reliability diagrams of the System One models and GPT-6 Sol (run 1). (A) RheumaMIR, Spanish (top row) and English (bottom row), 172 questions per language. (B) RheumBench, listed option set, 52 cases whose reference diagnosis is among the options. Each point is one of ten equal-width bins (closed on the right) of the selected-answer probability, plotted against the accuracy of those answers; point area is proportional to the number of answers; points below the diagonal indicate overconfidence; panel titles give the ECE. GPT-6 Sol is shown with medium and with no reasoning.
| Characteristic | Historical (2009–2010 to 2022–2023), n=143 | New (2023–2024 to 2025–2026), n=30 | Total, n=173 |
| Number of options, n (%) | |||
| Five (2009–2010 to 2014–2015) | 63 (44.1) | 0 (0.0) | 63 (36.4) |
| Four (2015–2016 onwards) | 80 (55.9) | 30 (100.0) | 110 (63.6) |
| Type of question, n (%) | |||
| Clinical case | 95 (66.4) | 21 (70.0) | 116 (67.1) |
| Factual | 48 (33.6) | 9 (30.0) | 57 (32.9) |
Table S1: Characteristics of the evaluated RheumaMIR questions.
| Model and version (developer) | Access and identifier | Settings | Considerations |
| System One models | |||
| Jev 1.13 (TypeSafe AI) | Remote, OpenRouter (typesafe/jev-1.13); returned as typesafe/jev-1.13-20260917 | Single choice question; native probabilities | Architecture undisclosed; English described by the developer as its primary training language |
| GPT-6 Luna (OpenAI) | Remote, OpenAI Decisions API (gpt-6-luna; public beta); no dated snapshot | Single choice question without fallback choice; native probabilities | Not described by OpenAI as a System One model; architecture undisclosed; returns the selected option and a probability for each option without generating text |
| Clef (Cloudflare) [ 19 ] | Remote, OpenRouter Decisions endpoint (cloudflare/clef; alpha); provider pinned to Cloudflare Workers AI, fallbacks disabled; no dated snapshot | Request of Jev plus provider field; native probabilities | Open weights; 27 billion parameters; fine-tuned from Qwen3.8-27B |
| Microsoft-Decision-1 (Microsoft) | Remote, OpenRouter Decisions endpoint (microsoft/microsoft-decision-1; alpha), returned as microsoft/microsoft-decision-1-20261009; provider pinned to Azure, fallbacks disabled | Request of Jev plus provider field; native probabilities | Text only; post-trained from Qwen3.5-9B; weights not public and updated continually, so the returned identifier names no fixed snapshot |
| Laya 0.3.20 (Convai Innovations) [ 18 ] | Local (Apple GPU); English checkpoint (ModernBERT-large, 421 million parameters) for English RheumaMIR items and RheumBench, multilingual checkpoint (mmBERT-base) for Spanish items | Language passed explicitly; default input limits; no recalibration | Open source; shortened option text in two RheumaMIR questions (three question–language pairs) and, with a 512-token limit, cut the end of 10 RheumBench vignettes (9 of the 52 analysed) |
Table S2: Evaluated models and configurations: version, access, settings and considerations for interpretation.
| System One model | Difference, pp (95% CI) | Correct only GPT-6 Sol, n | Correct only System One model, n | P (Holm) |
| RheumaMIR, Spanish | ||||
| Jev 1.13 | 6.4 ( 10.5 to 2.9) | 11 | 0 | 0.005 |
| GPT-6 Luna (Decisions API) | 1.7 ( 4.1 to 0.0) | 3 | 0 | 0.500 |
| Clef | 7.6 ( 11.6 to 3.5) | 14 | 1 | 0.005 |
| Microsoft-Decision-1 | 11.6 ( 16.3 to 7.0) | 20 | 0 | <0.001 |
| Laya 0.3.20 | 77.3 ( 83.1 to 70.9) | 133 | 0 | <0.001 |
Table S3: Paired comparisons of the System One models with GPT-6 Sol (medium reasoning).
| Model | RheumaMIR, Spanish | RheumaMIR, English | RheumBench |
| System One models | |||
| Jev 1.13 | 0.904 (0.843–0.956) | 0.916 (0.859–0.964) | 0.737 (0.598–0.859) |
| GPT-6 Luna (Decisions API) | NE | NE | 0.876 (0.773–0.959) |
| Clef | 0.851 (0.754–0.932) | 0.789 (0.641–0.915) | 0.800 (0.668–0.899) |
| Microsoft-Decision-1 | 0.831 (0.752–0.901) | 0.933 (0.891–0.966) | 0.811 (0.679–0.922) |
| Laya 0.3.20 | 0.613 (0.503–0.710) | 0.522 (0.425–0.624) | 0.323 (0.031–0.729) |
Table S4: Correctness AUROC of the selected-answer probability (95% CI).
Figure S1: Reliability diagrams for local generative models with thinking disabled (run 1). (A) RheumaMIR in Spanish (top) and English (bottom). (B) RheumBench, listed option set (52 cases). Each point represents one of ten equal-width bins (closed on the right; Section S4 ), plotting mean selected-option probability against accuracy. Point size reflects bin count. Points below the diagonal indicate overconfidence; small bins warrant caution. System One models and GPT-6 Sol appear in Figure 2 of the main text.
| Model | Spanish: same answer in 3 runs, % | Spanish: single-run accuracy, % | English: same answer in 3 runs, % | English: single-run accuracy, % |
| System One models | ||||
| Jev 1.13 | 98.3 | 93.0–93.6 | 98.8 | 91.9–93.0 |
| GPT-6 Luna (Decisions API) | 100.0 | 97.7–97.7 | 100.0 | 97.7–97.7 |
| Clef | 100.0 | 91.9–91.9 | 100.0 | 93.0–93.0 |
| Microsoft-Decision-1 | 97.7 | 86.6–87.8 | 98.8 | 86.0–86.6 |
| Laya 0.3.20 | 100.0 | 22.1–22.1 | 100.0 | 25.0–25.0 |
Table S5: Repeatability over three runs.
| Model | Spanish: latency, s, median (IQR) | Spanish: cost per 1,000 questions, US$ | English: latency, s, median (IQR) | English: cost per 1,000 questions, US$ |
| System One models | ||||
| Jev 1.13 | 0.33 (0.31–0.36) | 0.021 | 0.33 (0.31–0.37) | 0.020 |
| GPT-6 Luna (Decisions API) | 0.21 (0.20–0.24) | 0.027 | 0.21 (0.20–0.23) | 0.025 |
| Clef | 0.47 (0.38–0.54) | 0.075 | 0.47 (0.39–0.56) | 0.069 |
| Microsoft-Decision-1 | 0.40 (0.38–0.43) | 0.008 | 0.39 (0.37–0.42) | 0.007 |
| Laya 0.3.20 | 0.04 (0.03–0.04) | No charge | 0.07 (0.05–0.07) | No charge |
Table S6: Response time and cost.
| Listed option set | None of the above option set | |||||
| Model | Forced answers accepted at 0.90, n/18 | Accepted at 0.90 and correct, 70 cases, n/N (%) | None of the above selected, n/18 | None of the above selected, n/52 | Correct with None of the above, n/52 (listed options) | Accuracy with None of the above, 70 cases, % (95% CI) |
| System One models | ||||||
| Jev 1.13 | 1 | 11/14 (78.6) | 0 | 0 | 25 (26) | 35.7 (25.5–47.4) |
| GPT-6 Luna (Decisions API) | 1 | 21/23 (91.3) | 4 | 0 | 32 (33) | 51.4 (40.0–62.8) |
| Clef | 0 | 5/5 (100.0) | 0 | 0 | 30 (31) | 42.9 (31.9–54.5) |
| Microsoft-Decision-1 | 1 | 11/16 (68.8) | 3 | 2 | 21 (21) | 34.3 (24.2–46.0) |
Table S7: RheumBench: answers when the reference diagnosis is not among the options.
| ID | Question (abridged) and options | Incorrect option selected | Model, language: probability |
|---|---|---|---|
| 138 2010–2011 Clinical case SLE | Woman with SLE on glucocorticoids (0.5 mg/kg/day for several months); sudden, continuous knee pain for 3 weeks; normal radiograph. Next investigation? 1. Knee CT scan. 2. Full blood count with a differential white cell count, and C-reactive protein. 3. Cell count and biochemistry of joint fluid. 4. Culture of joint fluid. 5. Magnetic resonance imaging. | 3. Cell count and biochemistry of joint fluid. | Microsoft-Decision-1, Spanish: 0.999 Microsoft-Decision-1, English: 0.966 Clef, Spanish: 0.827 |
| 151 2023–2024 Clinical case Bone metabolism | 93-year-old man after hemiarthroplasty for an osteoporotic subcapital femoral fracture. Correct statement? 1. He is not a candidate for bone-protective treatment because of his age and dementia. 2. Bisphosphonates should not be started if he has hypocalcaemia. 3. He should receive calcium supplements even if his levels are normal. 4. He should remain on bed rest for six weeks after surgery to improve fracture healing. | 3. He should receive calcium supplements even if his levels are normal. | Microsoft-Decision-1, Spanish: 0.997 Microsoft-Decision-1, English: 0.772 Clef, Spanish: 0.463 |
| 63 2015–2016 Clinical case SLE | 37-year-old man with MCP arthritis, pleuritis and malar erythema; ANA 1/640, anti-DNA positive, anti-Sm negative. Initial treatment of choice? 1. High-dose glucocorticoids. 2. Glucocorticoids and mycophenolate. 3. Non-steroidal anti-inflammatory drugs and antimalarials. 4. The condition is likely to be self-limiting and does not require treatment. | 1. High-dose glucocorticoids. | Microsoft-Decision-1, Spanish: 0.993 Microsoft-Decision-1, English: 0.829 |
| 142 2009–2010 Clinical case Others | 72-year-old man with macroglossia, periorbital ecchymosis, sensorimotor polyneuropathy, monoclonal peak, proteinuria 6 g/24 h and prothrombin activity 35%. Diagnostic approach? 1. Abdominal fat biopsy and bone marrow aspirate. 2. Determination of microalbuminuria and glycated haemoglobin. 3. Determination of plasma vitamin B12 and erythrocyte folate. 4. Serology for HIV (human immunodeficiency virus). 5. Renal biopsy. | 5. Renal biopsy. | Microsoft-Decision-1, Spanish: 0.990 Microsoft-Decision-1, English: 0.860 |
| 161 2024–2025 Clinical case SLE | Woman with SLE diagnosed 1 month earlier (autoimmune haemolytic anaemia, prolonged aPTT), on glucocorticoids, hydroxychloroquine and belimumab; dyspnoea for 48 h, haemoglobin 10.2 g/dL. First laboratory test? 1. D-dimer. 2. Urine protein. 3. NT-proBNP. 4. Coombs’ test. | 4. Coombs’ test. | Microsoft-Decision-1, Spanish: 0.986 Jev, English: 0.620 Jev, Spanish: 0.550 |
| 33 2019–2020 Factual Vasculitis | With regard to temporal arteritis, mark the WRONG answer. 1. It occurs mainly in white people over the age of 50 years. 2. Approximately a quarter of patients develop aneurysm or dilatation of the thoracic aorta. 3. Unilateral or bilateral vision loss is one of the most serious complications. 4. The peak incidence is between the sixth and seventh decades of life. | 2. Approximately a quarter of patients develop aneurysm or dilatation of the thoracic aorta. | Microsoft-Decision-1, Spanish: 0.984 Clef, English: 0.980 Clef, Spanish: 0.960 Microsoft-Decision-1, English: 0.948 Jev, English: 0.940 Jev, Spanish: 0.900 |
Table S8: RheumaMIR: the ten questions answered incorrectly with the highest selected-answer probabilities.
More in this topic
Clinical Language Model Evaluation
Clinical language model evaluation measures whether AI systems are accurate, safe, fair, and useful across medical tasks and care settings.
See all 426 papers