cs.CLNov 16, 2026

Validating LLM-Based Essay Scorers: A Comparative Analysis in Critical Online Reasoning in Medical Education

Authors: Daniel Baumartz, Alexander Mehler, Mevlüt Bagci, Maxim Konca

Organizations: Goethe University Frankfurt, Germany

Abstract

Automated essay scorers (AES) based on large language models (LLMs) promise human-level accuracy. However, training, fine-tuning, and evaluating LLMs for specific assessment tasks is costly. In particular, the question arises whether these AES should be trained for specific domains or whether models based on generic LLMs might be sufficient. Human annotations, by contrast, are often more valid but far more expensive. This study evaluates whether ensembles of AES systems based on general-purpose LLMs can approximate human ratings of medical-education writing tasks by leveraging encoded linguistic and conceptual knowledge. The goal is to assess whether general-purpose models are applicable for assisting or potentially replacing human annotation in this domain. We experiment with 11 AES systems and 8 different LLMs and evaluate 668 student responses to 12 tasks, using \numprint{2076} evaluations by 7 human annotators. These ratings are aggregated to create consistent performance levels that can be used as target classes for classifiers trained on AES outputs. Our results show that learner classes based on human ratings can be accurately reconstructed using a linear SVM trained on a small set of AES features. A single classifier generalized well across tasks and domains, with reasoning and content features being most predictive. This suggests that out-of-the-box LLMs, even without domain-specific fine-tuning, can support automatic essay grading in medical education. While individual AES systems are insufficient alone, their combination approximates human evaluators, pointing to a reduced need for manual scoring or its supplementation with automated methods.

Figures & tables

Explore similar work

CardsList
  1. Rubric-Based AES with LLM Ensembles: Scoring of Student Responses in Critical Online Reasoning Using Ensembles of LLMs

    Nov 16, 2026Daniel Baumartz, Alexander Mehler, Patrick Schrottenbacher +2Automated Essay ScoringEducational Assessment

  2. LLMs Do Not Grade Essays Like Humans

    Mar 24, 2026Jerin George Mathew, Sumayya Taher, Anindita Kundu +1Automated Essay ScoringLLM-as-a-Judge

  3. Inside the Scorer: Mechanistic Evidence on Rubric Control in LLM-Based Automated Essay Scoring from 25 Base Models

    Oct 19, 2026Chungche Huang, Po-Ya Angela Wang, Shu-Kai HsiehAutomated Essay ScoringRubric-Based Evaluation