Decision language models as affordable and rapid alternatives to LLM judges of clinical generative AI

Rotem Gershon; Seffi Cohen; Dean Ariel; Havatzelet Bilavsky-Yarden; Amiel Sberro; Shai Yitzhaki; Osnat Tausky; Naama Golan

cs.AIOct 20, 2026

Decision language models as affordable and rapid alternatives to LLM judges of clinical generative AI

Authors: Rotem Gershon, Seffi Cohen, Dean Ariel, Havatzelet Bilavsky-Yarden, Amiel Sberro, Shai Yitzhaki, Osnat Tausky, Naama Golan,

Organizations: Innovation Division, Clalit Research Institute, Clalit Health Services, Ramat-Gan, Israel. · Stein Faculty of Computer and Information Science, Ben Gurion University of the Negev, Be’er Sheva, Israel. · Department of Family Medicine, Clalit Health Services, Tel Aviv, Israel. · Schneider Children’s Medical Center, Petah-Tikva, Israel. ·

Abstract

Automated judges of clinical artificial intelligence (AI) outputs are necessary, but large language model (LLM) judges are costly and slow at scale. We compared Jev, a decision language model that returns structured judgments, with frontier and open-weight LLM judges in four use cases. On 118 pediatric answers, Jev's agreement with physicians' ordinal quality ratings (Gwet's AC2 0.50; expected score 0.53) matched or exceeded that of seven of 12 LLM judges, including frontier models, Claude Opus 5.5 (0.49) and GPT-6 (0.38). On physician-graded binary HealthBench criteria, Jev matched or exceeded four of six LLM judges. On MedHELM's open-ended tasks, Jev's agreement with a three-LLM jury approached or exceeded the jurors' internal agreement. On HealthBench Professional, Jev matched the benchmark's reference judge, GPT-5.4, on 91% of criteria (other judges, 91-93%). Frontier judges cost 73-294 times more than Jev and took 7-57 times longer. Jev may enable efficient clinical AI evaluation at scale.

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 555 papers

Decision Models vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places
Sep 24, 2026Delip Rao, Chris Callison-BurchLLM Inference Efficiency LLM Evaluation
University of Pennsylvania
JEV-as-a-Judge: Accept When Confident, Escalate When Unsure
Sep 22, 2026Yubo Li, Yidi Miao, Ramayya Krishnan +1LLM-as-a-Judge Language Model Generation Evaluation
Carnegie Mellon University
Can LLMs Accurately Score Medical Diagnoses and Clinical Reasoning?
Apr 16, 2026Amy Rouillard, Sitwala Mundia, Linda Camara +8LLM-as-a-Judge Healthcare
aWits MIND Institute, University of the Witwatersrand, Johannesburg, South Africa · fSchool of Computer Science and Applied Mathematics, University of the Witwatersrand, Johannesburg, South Africa · bGrai Labs, Cape Town, South Africa +3

Sep 24, 2026J/K move · Enter open · S save

Decision Models vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places

Delip Rao, Chris Callison-Burch

University of Pennsylvania

LLM Inference Efficiency LLM Evaluation LLM-as-a-Judge Selective Prediction Classification Rubric-Based Evaluation

Open paper AskPDF arXiv

LLM judges now score outputs against rubrics, in benchmarks and as training rewards. Decision models return probabilities over permitted answers with a confidence score but no free text. Two such models are Jev and an open reproduction, Perplexity's Decider. We compare each with three flash-tier LLM judges on nine panels from seven benchmarks with human judgments, giving every judge identical criterion texts. The LLM judges run in two setups, reading a whole rubric at once or one criterion at a time. Our findings suggest both decision models can often stand in for them. The LLM judges cost 16 to 357 times as much and take 5 to 350 times as long, yet in both setups each decision model trails them significantly in at most 6 of 27 comparisons. The two kinds of judge err alike, agreeing more with one another than with the human-annotated judgment labels on ordinal criteria. Over 85% of flash-tier LLM verdicts on either decision model's most confident errors repeat them, where independent errors would give about half. Each decision model's confidence ranks its errors on most ordinal panels, so each is a natural first stage for a cascade deferring uncertain verdicts to an LLM judge. Yet with flash-tier fallbacks, such cascades gained a mere 2.7 points at best over the strongest single judge, even with oracle thresholds, suggesting they mainly save cost. A cascade needs a fallback that is right where its first stage errs, but the LLM judges are mostly wrong in the same places as Jev, and the Decider is already the strongest single judge on most panels. Even each vendor's higher-tier model repeats those errors, and as a fallback it lifted no cascade more than 1.7 points above both tiers' strongest single judge. Holding across setups, effort levels, and tiers, these findings suggest designing future decision models afresh to get right what LLM judges get wrong, and vice versa.

Abstract

LLM-as-a-Judge

Explore similar work

Decision Models vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places

JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

Can LLMs Accurately Score Medical Diagnoses and Clinical Reasoning?