cs.LGNov 15, 2026

Scaling large language model-assisted line-of-therapy adjudication with human-in-the-loop evidence-calibrated selection

Authors: Olatunji Oni, Sanhita Sengupta, Marc De Benedetti, Nathanial Eddy

Organizations: Bristol Myers Squibb, NJ, USA

Abstract

Large language models (LLMs) have enabled the ability to scale clinical adjudication tasks traditionally dependent on exhaustive manual review. Line-of-therapy (LoT) adjudication, a major bottleneck in oncology cohort construction, is a prime example. Line boundaries must be inferred from longitudinal treatment histories according to nuanced, study-specific definitions, at every regimen transition for every patient. LLMs can apply this adjudication at scale, but their reliability varies across patients. Treatment histories differ in complexity, and ambiguous or atypical patterns depend on expert clinical judgment. Reliable automation thus requires verified support for model assignments and the ability to identify cases that warrant manual review. Here, we present evidence-calibrated selection (ECS), a human-in-the-loop framework for scaling LLM-assisted LoT adjudication. ECS derives confidence from observed clinician adjudications, prioritizes patients with the least-supported assignments for follow-up review, and returns completed adjudications to the model as precedent. As the evidence base grows, ECS continuously calibrates confidence to estimate residual error, thus enabling more efficient use of clinician time. In simulations on 136 clinician adjudicated patients with multiple myeloma, ECS achieved a residual error of 5% after review of 44.6% of patients, compared to 62.2% when review was directed by LLM self-reported confidence and 81.9% under random sampling. These results support a scalable approach to study-specific LoT adjudication, using clinician evidence to inform automated predictions, minimize error, and reduce review burden.

Figures & tables

Explore similar work

CardsList
  1. A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework

    Apr 3, 2026Chenyu Li, Zohaib Akhtar, Mingu Kwak +13LLM-as-a-JudgeClinical Language Model Evaluation

  2. Can LLMs Accurately Score Medical Diagnoses and Clinical Reasoning?

    Apr 16, 2026Amy Rouillard, Sitwala Mundia, Linda Camara +8LLM-as-a-JudgeHealthcare

  3. Patient-specific rubric-based evaluation of large language models across longitudinal cancer care

    Oct 26, 2026Wei-Wei Zhang, Pengkai Wang, Yan Li +13Clinical Language Model EvaluationMedical LLMs