cs.AINov 13, 2026
SaveWhen evaluation becomes intervention in clinical AI
Organizations: The Johns Hopkins University, Baltimore, Maryland, USA · Rutgers New Jersey Medical School, Newark, New Jersey, USA · Stanford University, Stanford, California, USA
Abstract
AI judges, models that grade another system's output against a rubric, extend clinical review to the scale at which clinical AI writes. When teams use judge assessments to select a prompt or train a model, the judge becomes part of the intervention. Selection can then favor errors the judge overlooks. We describe what a judge's report must contain and the independent clinical test each judge-driven change needs.
More in this topic
LLM-as-a-Judge
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
See all 603 papers