We present Doctorina MedBench, an evaluation framework for agent-based medical AI based on the simulation of physician-patient interactions. Unlike traditional medical benchmarks that rely on solving standardized test questions, the proposed approach models a multi-step clinical dialogue in which an AI system must collect medical history, analyze available synthetic attachments when present, formulate differential diagnoses, and provide diagnostic and management recommendations. System performance is evaluated across separate task-level domains, including diagnosis, differential diagnosis, treatment, safety-critical condition handling, and dialogue-step behavior; the broader D.O.T.S. framework is used as a supplementary summary for diagnosis, observations/investigations, treatment, and step count. The framework also supports testing and quality-monitoring workflows intended to identify changes in model behavior during development. It supports safety-oriented cases, category-based sampling of synthetic clinical scenarios, and regression-style comparisons across system versions. In the reported study, the analyzed paired complete-case cohort consisted of 254 physician-authored synthetic clinical cases retained from 261 attempted case identifiers. The evaluation metrics are intended for comparative research on interactive medical AI systems and for studying clinical reasoning workflows in synthetic dialogue settings. Our results suggest that simulated clinical dialogue can provide a complementary assessment setting to traditional examination-style benchmarks, while the reported findings do not establish independent clinical validity, clinical effectiveness, or readiness for real-world deployment.
Figures & tables
Objective
Benchmark question
Principal outcome domain
Interactive information gathering
Does the system identify and elicit clinically relevant information that is not disclosed in the initial presentation?
Total and Average steps
Diagnostic identification
Does the system identify the principal condition?
Diagnosis Accuracy
Differential diagnosis
Does the system produce a clinically appropriate ranked or unranked differential diagnosis?
Differential Accuracy; ranked retrieval metrics not evaluated in the archived comparison
Diagnostic workup
Does the system recommend required investigations while avoiding unnecessary or irrelevant testing?
Omission and unexpected-item errors
Treatment and management
Does the system provide appropriate treatment, monitoring, follow-up, and escalation recommendations?
Treatment Accuracy based on matching, extra, missing, and different items
Table 1: Pre-specified benchmark objectives and outcome domains.
Manifest field
Purpose
Reporting status
Benchmark name and version
Identifies the exact benchmark release used in the study
Doctorina MedBench; archived research freeze
Freeze date
Defines the date after which case content and reference labels were not modified for the primary analysis
Not recorded in the archived export
Attempted synthetic case identifiers
Defines the number of case identifiers attempted once under each system configuration
261
AI Doctor completed runs
Defines the number of attempted case identifiers completed by the AI Doctor workflow
255/261
GPT-5 basic completed runs
Defines the number of attempted case identifiers completed by the prompt-only reference
260/261
Paired cases in the comparative analysis
Defines the number of attempted case identifiers with completed outputs from both configurations
254 paired complete cases
Table 2: Required metadata for the archived benchmark release and paired analysis.
Conceptual element
Purpose in evaluation
Required
case_id
Unique identifier of the clinical case.
Yes
category
Broad clinical category used for case grouping.
Yes
intro
Initial patient presentation shown at the start of the dialogue.
Yes
additional_answers
Case-grounded information disclosed by the simulated patient when asked.
Yes
control_questions
Clinically relevant questions expected during history-taking.
Yes
diagnosis
Reference diagnosis.
Yes
Table 3: Conceptual clinical and dialogue structure of a benchmark case.
Dimension
GPT-5 basic prompt-only reference
Doctorina workflow
Instruction
Single-sentence diagnostic instruction
Structured multi-component clinical workflow
Dialogue objective
No explicit requirement to conduct iterative history-taking
Explicit iterative information gathering before finalization
Output structure
Unstructured final response
Structured diagnostic and management output
Orchestration
No domain-specific routing or workflow control
Agentic routing, history-taking, planning, and stopping logic
Interpretation
Minimal zero-scaffold reference
Full-system configuration; not an isolated component ablation
Table 4: Configuration differences relevant to interpretation of the descriptive comparison.
Metric
GPT-5 basic reference
AI Doctor
Difference
Diagnosis Accuracy
86.61%
91.34%
+4.73 pp
Differential Accuracy
23.62%
46.69%
+23.07 pp
Treatment Accuracy
37.60%
54.46%
+16.86 pp
Critical Conditions Passed
98.82%
99.61%
+0.79 pp
Total steps (doctor/patient pairs)
167
3009
+2842
Average steps (doctor/patient pairs)
0.66
11.85
+11.19
Table 5: Updated descriptive comparison across 254 paired complete synthetic clinical cases. Metric names and order follow the aggregate export.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Note. The table describes the public JSONL release schema. It does not describe a complete internal benchmark database or run-level audit export. Long machine-readable field identifiers are allowed to wrap at underscores to preserve the printed layout without changing field names.
Field
Type
Cardinality
Definition and validation rule
id
integer
1
Original synthetic case identifier. Must be unique within the public release.
diagnosis
string or null
1
Principal reference diagnosis label for the synthetic case. A null value indicates that the archived public export did not preserve a non-null value for this field.
differential
list[string]
0..*
Reference alternative diagnostic hypotheses used for the released differential-diagnosis field.
num_steps
integer or null
1
Reference dialogue-step target used as a soft behavioral audit. Null indicates that the archived public export did not preserve a numeric value for this field.
intro
string or null
1
Initial synthetic patient presentation shown to the evaluated system at the start of the dialogue.
Appendix
Table 1: Field-level data dictionary for a released synthetic case record.
Field
Status
Comment
GPT-5 basic reference instruction
Preserved verbatim
Reproduced above.
Patient-simulator prompt template
Preserved as template
Runtime placeholders were populated with case-specific values.
Evaluator prompt templates
Preserved as templates
Multiple task-specific evaluator templates are reproduced above.
AI Doctor workflow instruction
Reproduced above
The high-level workflow instruction is provided for interpretability.
Provider/model family
Partially preserved
Described as GPT-5 for the reference; exact provider-side snapshot identifiers were not retained.