cs.AINov 13, 2026

When evaluation becomes intervention in clinical AI

Authors: Anthony Abou Mrad, Aviona Conti, Samuel Schmidgall, Jayanth Pratap, Ruhika Iyer, Joseph Kim, King Shi, Yann Sakref, 

Organizations: The Johns Hopkins University, Baltimore, Maryland, USA · Rutgers New Jersey Medical School, Newark, New Jersey, USA · Stanford University, Stanford, California, USA

Abstract

AI judges, models that grade another system's output against a rubric, extend clinical review to the scale at which clinical AI writes. When teams use judge assessments to select a prompt or train a model, the judge becomes part of the intervention. Selection can then favor errors the judge overlooks. We describe what a judge's report must contain and the independent clinical test each judge-driven change needs.

Explore similar work

CardsList
  1. Decision language models as affordable and rapid alternatives to LLM judges of clinical generative AI

    Oct 20, 2026Rotem Gershon, Seffi Cohen, Dean Ariel +11LLM-as-a-JudgeClinical Language Model Evaluation

  2. Objectify, Don't Delete: Complaint-Derived Evaluation of Ambient Clinical Notes

    Nov 14, 2026Nikita Gupta, Narayanan Unny Edakunni, Vikram Okaly +4Clinical Language Model EvaluationRubric-Based Evaluation