LLMs as Judges and Examinees for Medical Item Difficulty Prediction

Xiaomeng Xiong; Marcus Walker; Yanlin Jiang

cs.CLOct 28, 2026

LLMs as Judges and Examinees for Medical Item Difficulty Prediction

Authors: Xiaomeng Xiong, Marcus Walker, Yanlin Jiang

Organizations: University of Florida · National Commission on Certification of Physician Assistants

Abstract

Predicting medical item difficulty from item text alone remains challenging. Prior work has largely relied on linguistic features or text representations derived from general-purpose and domain-specific language models, but predictive performance has remained modest. This study takes a different approach by using difficulty-related signals that emerge when open-weight large language models (LLMs) interact with test items. Using USMLE and PANCE medical licensure examination items, we examined LLMs in two roles. As judges, the models provided absolute difficulty ratings and pairwise difficulty comparisons. As simulated examinees, they generated response-based signals including correctness, keyed-option probability, response entropy, and the probability margin between the top two options. We then combined the strongest judgment and response signals with baseline item features in downstream prediction models. Pairwise difficulty judgment was the most consistent individual LLM signal across both datasets, while simulated-examinee signals and absolute judgments provided complementary information in some feature combinations. The best configurations reached a Spearman correlation of $ρ$ = .447 on USMLE, exceeding the best system in the 2024 BEA shared task ( $ρ$ = .25), and $ρ$ = .494 on PANCE. These results were comparable to direct absolute judgments from the proprietary frontier model Claude Opus 5 ( $ρ$ = .443 and .488, respectively). Overall, judgment and response signals from open-weight LLMs captured difficulty-related information beyond text representations. Because these models can be deployed locally, the approach offers a practical option for predicting the difficulty of unreleased or high-security test items.

Figures & tables

Model	Parameters	Architecture	Developer	Role
nemotron-3-super-120b-a12b	120.6B / 12.7B active	Hybrid MoE	NVIDIA	Judge, examinee
nemotron-3-nano-30b-a3b	31.6B / 3.6B active	Hybrid MoE	NVIDIA	Judge, examinee
gpt-oss-120b	117B / 5.1B active	MoE	OpenAI	Judge, examinee
gemma-4-31b-it	30.7B	Dense	Google	Judge, examinee
medgemma-27b-it	27B	Dense	Google	Judge, examinee
gemma-3-27b-it	27B	Dense	Google	Judge, examinee

Table 1: Open-Weight LLMs Used as Difficulty Judges and Simulated Examinees

Configuration	Included Features
$B$	Baseline item features
$B+P$	Baseline features + selected LLM pairwise judgment
$B+A$	Baseline features + selected LLM absolute judgment
$B+S$	Baseline features + four selected LLM examinee readouts
$B+P+A$	Baseline features + LLM pairwise judgment + LLM absolute judgment
$B+P+S$	Baseline features + LLM pairwise judgment + LLM examinee readouts

Table 2: Feature Configurations Evaluated in the Downstream Prediction Models

	USMLE			PANCE
Model	Acc ${}_{\text{cons}}$	Acc ${}_{\text{all}}$	Consistency	Acc ${}_{\text{cons}}$	Acc ${}_{\text{all}}$	Consistency
gemma-4-31b-it	.668	.639	82.6	.668	.645	86.1
nemotron-3-super-120b-a12b	.656	.617	74.7	.653	.614	74.5
gpt-oss-120b	.624	.599	79.8	.633	.608	81.2
nemotron-3-nano-30b-a3b	.603	.573	71.3	.618	.581	69.1
medgemma-27b-it	.599	.577	77.5	.582	.562	76.0

Table 3: Pairwise Judgment Accuracy of the Six Judge Models

Figure 1: Pairwise Judgment Accuracy (Top) and Two-Way Agreement (Bottom) as a Function of the Empirical Difficulty Gap

	Comparisons per item ( $k$ )
Judge	5	10	15	20	25	30	35	40
USMLE
gemma-4-31B	.384	.412	.419	.420	.422	.423	.422	.424
nemotron-120B	.349	.377	.390	.389	.392	.394	.395	.396
gpt-oss-120B	.297	.328	.330	.338	.337	.341	.341	.342
nemotron-30B	.245	.282	.280	.288	.291	.293	.292	.294

Table 4: Spearman Correlations Between Bradley–Terry Scores and Empirical Difficulty Across Comparison Densities

	USMLE		PANCE
Model	Absolute	Pairwise	Absolute	Pairwise
gemma-4-31b-it	.349	.424	.411	.439
nemotron-3-super-120b-a12b	.263	.396	.249	.348
gpt-oss-120b	.288	.342	.226	.318
nemotron-3-nano-30b-a3b	.239	.294	.176	.246
medgemma-27b-it	.134	.232	.131	.158

Table 5: Training-Set Spearman Correlations for Absolute and Pairwise Judgments

Dataset	Configuration	Learner	Spearman $\rho$	Pearson $r$	RMSE
USMLE	B	Ridge	.228	.274	.299
	B+P	Ridge	.425	.417	.283
	B+A	Ridge	.357	.381	.288
	B+S	Ridge	.379	.382	.287
	B+P+A	Ridge	.419	.423	.281
	B+P+S	Ridge	.447	.441	.280

Table 6: Test-Set Performance Across the Eight Feature Configurations

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$S_{1}$	$S_{2}$	$S_{3}$	$S_{4}$
USMLE ( $n=466$ )
Nemotron 3 Super 120B	.276	.251	.205	.274
GPT-OSS 120B	.021	.124	.001	.020
Llama 3.3 70B	.278	.093	.265	.277
Llama 3.1 70B	.291	.132	.277	.289
Gemma 4 31B	.256	.078	.246	.252

Appendix

Table A1: Spearman Correlations Between LLM Examinee Readouts and Empirical Item Difficulty

Dataset	Signal / Readout	Selected Model	$\rho$
USMLE	LLM pairwise judgment ( $P$ )	Gemma 4 31B	.424
	LLM absolute judgment ( $A$ )	Gemma 4 31B	.349
	Keyed-option probability ( $S_{1}$ )	Mistral Small 3.1 24B	.327
	Binary correctness ( $S_{2}$ )	Nemotron 3 Nano 30B	.270
	Response entropy ( $S_{3}$ )	Mistral Small 3.1 24B	.287
	Top-two margin ( $S_{4}$ )	Mistral Small 3.1 24B	.327

Appendix

Table A2: Selected LLM-Derived Signals and Training-Set Spearman Correlations

Item Difficulty Estimation

See all 19 papers

What Makes Medical Examination Items Difficult? Using Large Language Models to Explore Patterns Associated With Empirical Item Difficulty
Oct 29, 2026Xiaomeng Xiong, Marcus Walker, Yanlin JiangItem Difficulty Estimation Medical Education
University of Florida, Gainesville, Florida, USA. · National Commission on Certification of Physician Assistants, USA.
Estimating Item Difficulty with Large Language Models as Experts
May 18, 2026Diana Kolesnikova, Kirill Fedyanin, Abe D. Hofman +2LLM Evaluation LLM-as-a-Judge
Department of Methodology and Statistics, Tilburg University, Tilburg, Netherlands · Smart Business Technologies, Belgrade, Serbia · Department of Psychological Methods, University of Amsterdam, Amsterdam, Netherlands +2
LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment
Jun 17, 2026Han Chen, Ming Li, Chenguang Wang +4LLM Evaluation Educational Assessment
MBZUAI · University of Maryland · Virginia Tech

Oct 29, 2026J/K move · Enter open · S save

What Makes Medical Examination Items Difficult? Using Large Language Models to Explore Patterns Associated With Empirical Item Difficulty

Xiaomeng Xiong, Marcus Walker, Yanlin Jiang

University of Florida, Gainesville, Florida, USA. · National Commission on Certification of Physician Assistants, USA.

Item Difficulty Estimation Medical Education Rubric-Based Evaluation LLM Evaluation

Open paper AskPDF arXiv

Research on medical item difficulty has largely focused on prediction rather than on identifying interpretable patterns associated with empirical difficulty. This study used empirical difficulty labels to guide large language models (LLMs) in inducing difficulty rubrics and examined how rubric guidance affected LLM difficulty judgments. Using training items and their empirical difficulty estimates from the United States Medical Licensing Examination (USMLE) and the Physician Assistant National Certifying Examination (PANCE), Claude Opus 5 induced separate rubrics for absolute and pairwise difficulty judgment. None of the induced rubrics treated surface features, such as stem length, option count, or images, as major sources of difficulty. Several patterns emerged across both examinations, including cue convergence, misleading cues, and option competition, although their specific forms differed. Retrieval of knowledge that could not be derived from the stem was particularly prominent in the PANCE rubrics, whereas the USMLE rubrics placed greater emphasis on integrating multiple clinical cues. We compared commercial and open-weight LLMs on held-out items under unguided and rubric-guided conditions. Rubric guidance changed how models used the difficulty scale but did not consistently improve performance. In absolute judgments, guidance encouraged broader use of the scale and generally improved coarse-grained agreement, without consistently improving rank ordering. In pairwise comparisons, its effects varied across models and examinations, and judgments became more sensitive to presentation order. These findings suggest that the value of label-guided rubric induction may lie more in generating interpretable, testable hypotheses from empirical difficulty data than in reliably improving difficulty judgments.

Abstract

Item Difficulty Estimation

Explore similar work

What Makes Medical Examination Items Difficult? Using Large Language Models to Explore Patterns Associated With Empirical Item Difficulty

Estimating Item Difficulty with Large Language Models as Experts

LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment