AI-Enabled Quality Assurance for Multiple-Choice Assessment Items

Steven Moore; Nicholas Diana

cs.CLOct 3, 2026

AI-Enabled Quality Assurance for Multiple-Choice Assessment Items

Authors: Steven Moore, Nicholas Diana

Organizations: George Mason University · Colgate University

Abstract

Generating multiple-choice questions is increasingly scalable, but establishing their assessment quality remains difficult. We present a focused narrative review of automated item-writing flaw detection, revision, psychometric screening, and NLP benchmark auditing. Database searches, citation retrieval, and nominated sources yield fourteen research reports reviewed in full text. We distinguish surface checks from content-sensitive judgments and map a 19-criterion rubric to detection methods and reported evidence. High label-level accuracy often coexists with weak positive case detection, while rubric definitions and reference standards vary. Revision evidence is mixed, and the associations reported in prior work do not establish the effects of repair. We propose evaluating quality assurance as a sequence of independently validated decisions, with criterion-specific reporting, calibrated human review, and outcome-based assessment of revisions.

Explore similar work

Auditing MCQA Benchmarks through Probability Landscapes
Aug 31, 2026Minsoo Song, Chanjun ParkBenchmark Auditing LLM Auditing
Soongsil University
Automated item evaluation: Predicting item acceptance and rejection using LLM-generated critiques
Aug 6, 2026Hotaka Maeda, Yikai LuEducational Assessment Text Classification
Smarter Balanced, University of California-Santa Cruz · Department of Educational Psychology, University of Minnesota-Twin Cities
The Architect, the Adversary, and the Judge: Closed-Loop Generation of Standards-Aligned Assessment Items at Scale
Aug 26, 2026Wenhui Chen, Ziyao Lin, Jianlin Chen +2Educational Assessment LLM-as-a-Judge
University of Macau · South China University of Technology · Independent Researcher

Aug 31, 2026, cs.CLJ/K move · Enter open · S save

Auditing MCQA Benchmarks through Probability Landscapes

Minsoo Song, Chanjun Park

Soongsil University

Benchmark Auditing LLM Auditing

Open paper AskPDF arXiv

As Large Language Models rapidly advance, performance on standard multiple-choice question answering (MCQA) benchmarks is reaching saturation. While the community has responded by developing increasingly difficult datasets, validating question quality and filtering flawed items remains a labor-intensive process. To provide a scalable diagnostic approach, we propose a two-component probabilistic framework for auditing MCQA benchmarks using model output distributions. First, for benchmark-level analysis, we characterize the probability landscape using the top prediction probability (

P_{top1}

) and normalized residual entropy (

H_{norm}

), summarized globally by Mean Pairwise Distance (MPD). Second, for item-level diagnostics, we introduce noise injection to reduce meaningful distractor competition, enabling us to flag candidate items for targeted human review and categorize residual failure patterns. Across four MCQA benchmarks, our landscape analysis reveals benchmark-level differences in model confidence and residual option competition. Concurrently, our noise-injection method flags potentially actionable item-level issues, showing alignment with expert error annotations from MMLU-Redux. These results suggest that our probability-based framework provides a lightweight audit lens for comparing macro-level benchmark structure and prioritizing individual items for targeted human review.

Abstract

Explore similar work

Auditing MCQA Benchmarks through Probability Landscapes

Automated item evaluation: Predicting item acceptance and rejection using LLM-generated critiques

The Architect, the Adversary, and the Judge: Closed-Loop Generation of Standards-Aligned Assessment Items at Scale