cs.CLNov 16, 2026

Rubric-Based AES with LLM Ensembles: Scoring of Student Responses in Critical Online Reasoning Using Ensembles of LLMs

Authors: Daniel Baumartz, Alexander Mehler, Patrick Schrottenbacher, Leon Hammerla, Sebastian Gombert

Organizations: Goethe University Frankfurt, Germany · DIPF Leibniz Institute for Research and Information in Education, Germany

Abstract

Grading student responses is costly and time-consuming. Moreover, human annotators are prone to bias, fatigue, and rater drift, which further limits grading reliability. Various methods have been employed to develop automated essay scoring (AES) systems to reduce these costs and address bias-related issues. These methods range from early text-feature-based systems to current machine-learning and transformer-based models. Specifically, large language models (LLMs) offer the potential to streamline the development of AES systems. However, they are subject to well-known limitations, such as hallucinations and prompt sensitivity. This creates a need to explore the potential of a mixed-methods approach that leverages the synergies among individual LLMs through ensemble formation. We evaluate the potential of ensembles of LLMs to match human-rater agreement when grading student responses in economics and medical education across 103 criteria. We use 631 German-language responses and corresponding human ratings by 2 to 5 annotators from a previously conducted assessment. We experiment with 523 rubric-based prompt configurations across 15 open-weight LLMs and assess their consistency with human raters using Krippendorff's alpha for inter-annotator agreement (IAA) and the Alternative Annotator Test (alt-test). Our results show that replacing one human rater with the best-performing LLM--prompt configuration maintains or improves IAA for 91 of the 103 criteria and that LLMs pass the alt-test for 90 criteria. Our findings suggest that current state-of-the-art LLMs can support AES of German responses in economics and medical education, reducing human workload and enabling faster processing, while validation against human ratings remains necessary.

Figures & tables

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Validating LLM-Based Essay Scorers: A Comparative Analysis in Critical Online Reasoning in Medical Education

    Nov 16, 2026Daniel Baumartz, Alexander Mehler, Mevlüt Bagci +1Automated Essay ScoringAutomated Grading

  2. Evaluating Austrian A-Level German Essays with Large Language Models for Automated Essay Scoring

    Mar 6, 2026Jonas Kubesch, Lena Huber, Clemens HavasAutomated Essay ScoringEducational Assessment

  3. Inside the Scorer: Mechanistic Evidence on Rubric Control in LLM-Based Automated Essay Scoring from 25 Base Models

    Oct 19, 2026Chungche Huang, Po-Ya Angela Wang, Shu-Kai HsiehAutomated Essay ScoringRubric-Based Evaluation