cs.CLOct 7, 2026
SaveAlice: A Large-Scale German Benchmark for Rubric-Based Multi-Dimensional Automatic Short Answer Scoring
Organizations: DIPF | Leibniz Institute for Research and Information in Education · IPN | Leibniz Institute for Science and Mathematics Education · Umeå University · Computer Science Department & Studiumdigitale, Goethe University Frankfurt
Abstract
Automatic Short Answer Scoring (ASAS) is central to NLP for Education. However, openly available benchmarks remain scarce, and existing datasets largely address how well students answer a question directly rather than how well they master underlying concepts (knowledge elements) such as thermal energy or epistemic activities (skills) such as reasoning or claim. To address this gap, we introduce Alice, a large-scale, rubric-based German ASAS dataset that is pedagogically aligned and comprises three subtasks: (i) learning performance (Alice-LP), (ii) knowledge elements (Alice-KE), and (iii) skills (Alice-SK). We further formulate rubric-based ASAS as a rubric-retrieval task and benchmark the dataset with a range of language models, from encoder-only models to lightweight LLMs. We also benchmark the dataset with zero-shot prompting via LLMs and a standard classification baseline. The experiments show that LLMs, in particular, struggle to score knowledge elements and skills in the zero-shot setting. They also indicate that rubric text is often useful, especially for Alice-KE and Alice-SK, while on Alice-LP gains over sample-solution-focused inputs are more modest and vary by model and input format.
Figures & tables
Figure 1: An instance from Alice . Each instance includes a question prompt, a sample solution, and per-level rubrics for overall learning performance, knowledge elements, and skills. The complete example is shown in Figure 6 in the appendix.
| Benchmark | Answers | Questions | Context information | Language coverage | Public? | |
| ASAP-SAS | 22K | 10 | question prompt, per-level rubrics | English | ||
| SciEntsBank Dzikovska et al. (2013) | 10K | 197 | solution, question prompt | English | ||
| Beetle Dzikovska et al. (2013) | 3K | 56 | solution, question prompt | English | ||
| PT_ASAG_2018 Galhardi et al. (2018) | 13K | 8 | solution, question prompt | Portuguese | ||
| RIKEN-SAS ( Mizumoto et al., 2019 ; Funayama et al., 2025 ) | 31K | 34 | reading passage, question prompt, analytic rubrics, key phrases | Japanese | ||
| Sung et al. (2019) | 76K | 28 | solution, question prompt | English |
Table 1: Comparison of the Alice version used in this work with existing ASAS benchmarks. Note that we frame all information other than the student’s answer as context information .
| Subject | KE | SK |
| Biology | 0.84 | 0.78 |
| Chemistry | 0.86 | 0.65 |
| Physics | 0.72 | 0.69 |
Table 2: Quadratic weighted kappa ( ; QWK) agreement for KE and SK annotations by subject.
Figure 2: NLI label distribution between adjacent higher- and lower-level rubric pairs.
| LP | KE | SK | |||||
| Split | Subject | #Q | #A | #Q | #A | #Q | #A |
| Train | Biology | 8 | 1,148 | 8 | 1,148 | 8 | 1,148 |
| Chemistry | 54 | 5,921 | 52 | 5,645 | 53 | 5,755 | |
| Mathematics | 9 | 609 | NA | NA | NA | NA | |
| Physics | 19 | 3,103 | 18 | 2,939 | 19 | 3,073 | |
| Total | 90 | 10,781 | 78 | 9,732 | 80 | 9,976 | |
Table 3: Split-wise question (#Q) and answer (#A) counts by subject and subtask.
Figure 3: Distribution of levels across subtasks and subjects. Mathematics has no KE or SK annotations.
Figure 4: Correlation heatmaps between levels of LP and the other two subtasks.
Table 4: Main Macro-F1 results on Alice . Upper panels show fine-tuned UA and UQ results; the lower panel shows zero-shot UQ prompting. In the upper panels, bold marks the best result per subtask; in the lower panel, it marks the best result per input format and subtask. The underline in the upper panels marks cells where sequence classification outperforms rubric-retrieval. Red cells mark full-context results ( +qs / +qr / +qs+r ) that underperform the base input format ( ar / sa / -r ) for that model and subtask.
Figure 5: Per-level F1 across Alice subtasks and evaluation settings.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Illustrative benchmark instance, translated from German to English. Selected rubric levels are highlighted in green.
| Subject | KE items |
| Biology | Gendrift (Genetic Drift), Genetik (Genetics), Mutation (Mutation), Natürliche Selektion (Natural Selection), Sexuelle Selektion (Sexual Selection), Variation (Variation), Wahrscheinlichkeit (Probability), Zufall (Randomness) |
| Chemistry | Abklingfunktion (Decay Function), Aggregatzustand (State of Matter), Aggregatzustände (States of Matter), Aktivierungsenergie (Activation Energy), Anzahl an Gasteilchen (Number of Gas Particles), Dissoziation (Dissociation), Druck (Pressure), Druckänderung (Pressure Change), Dynamisches Gleichgewicht (Dynamic Equilibrium), Eduktregenerierung (Reactant Regeneration), Einflussfaktor Druck (Influencing Factor: Pressure), Einflussfaktor Temperatur (Influencing Factor: Temperature), Endotherme Rückreaktion (Endothermic Reverse Reaction), Energieumsatz (Energy Conversion), Exemplarität/Deduktion (Exemplarity/Deduction), Exotherme Hinreaktion (Exothermic Forward Reaction), Gasentwicklung (Gas Evolution), Geschlossenes System (Closed System), Gleichgewichtseinstellung (Equilibrium Establishment), Gleichgewichtskonstante (Equilibrium Constant), Gleichzeitigkeit (Simultaneity), Hinreaktion (Forward Reaction), Inaktivierung (Inactivation), Katalysator (Catalyst), Katalysatorrückbildung (Catalyst Regeneration), Kollision (Collision), Konzentration (Concentration), Kostenintensivität (Cost Intensity), Ladungsträger (Charge Carrier), Leitfähigkeit (Conductivity), Makroskopisches Reaktionsende (Macroscopic Reaction Endpoint), Masse (Mass), Mindestenergie (Minimum Energy), Mindestenergie/Aktivierungsenergie (Minimum Energy/Activation Energy), Mittelwert (Mean), Neueinstellung (Re-Establishment of Equilibrium), Offenes System (Open System), Produktentfernung (Product Removal), Reaktionsenthalpie (Reaction Enthalpy), Reaktionsgeschwindigkeit (Reaction Rate), Reaktionsrate (Reaction Rate), Reaktionsweg (Reaction Pathway), Reaktionszeit (Reaction Time), Rückreaktion (Reverse Reaction), Sekantensteigung (Secant Slope), Stoffmenge (Amount of Substance), Stoffumsatz (Substance Conversion), Stoffumwandlung (Substance Transformation), Stoßwahrscheinlichkeit (Collision Probability), Stoßwirksamkeit (Collision Effectiveness), System (System), Sättigungskurve (Saturation Curve), Sättigungsfunktion (Saturation Function), Tangentensteigung (Tangent Slope), Teilchengeschwindigkeit (Particle Velocity), Teilcheninteraktion (Particle Interaction), Temperatur (Temperature), Ungleichheit von Entitäten (Inequality of Entities), Unvollständigkeit (Incompleteness), Volumen (Volume), Wiedereinstellung (Re-Establishment), Wirksamkeit Katalysator (Catalyst Effectiveness), Zeitintervall (Time Interval), Zeitunabhängigkeit (Time Independence), Zellgift (Cytotoxin), Zellweger-Syndrom (Zellweger Syndrome), Zerteilungsgrad (Degree of Dispersion), Zufallsfehler (Random Error) |
| Mathematics | – |
| Physics | Chemische Energie (Chemical Energy), Elektrische Energie (Electrical Energy), Energieversorgung (Gesellschaft) (Energy Supply: Society), Energieversorgung (Ökologie) (Energy Supply: Ecology), Kinetische Energie (Kinetic Energy), Strahlungsenergie (Radiation Energy), Thermische Energie (Thermal Energy), Umwandlung (Conversion) |
Table 5: Knowledge Element ( KE ) item names per subject. Each entry is a domain concept that students are scored on independently.
| Subject | SK items |
| Biology | Beschreiben den Experimentaufbau (Describe the Experimental Setup), Bewerten der Daten (Evaluate the Data), Claim (Claim), Erläutern die Erklärungskraft eines Modells (Explain the Explanatory Power of a Model), Evidence (Evidence), Formulieren eine Forschungsfrage (Formulate a Research Question), Formulieren eine Hypothese (Formulate a Hypothesis), Forschungsfragen stellen (Pose Research Questions), Hypothesen/Vermutungen aufstellen (Formulate Hypotheses/Assumptions), Konzentration auf bestimmte Aspekte (Focus on Specific Aspects), Reasoning (Reasoning) |
| Chemistry | Analysieren Zusammenhänge, die im Modell expliziert werden (Analyze Relationships Made Explicit in the Model), Auseinandersetzung mit dem Phänomen (Engagement with the Phenomenon), Beobachten und Messen (Observing and Measuring), Beschreiben auf Basis eines Phänomens relevante Variablen, Systeme und/oder Konzepte (Describe Relevant Variables, Systems, and/or Concepts Based on a Phenomenon), Beschreiben den Experimentieraufbau (Describe the Experimental Setup), Beschreiben die vorliegenden Daten (Describe the Available Data), Bestimmen mithilfe der Daten relevante Aspekte / Werte (durch Berechnung) (Determine Relevant Aspects/Values from Data (by Calculation)), Beziehen die relevanten Aspekte / Werte auf die vorhandenen Informationen (Relate the Relevant Aspects/Values to the Available Information), Claim (Claim), Erklären, warum die Daten die Behauptung stützen oder widerlegen (Explain Why the Data Support or Refute the Claim), Erläutern Rückschlüsse zu den Implikationen ihrer Ergebnisse (Explain Inferences about the Implications of Their Results), Erläutern die Erklärungskraft eines Modells (Explain the Explanatory Power of a Model), Evaluieren Stärken und Schwächen der eigenen Modellierung durch Vergleiche mit Konsensmodellen (Evaluate Strengths and Weaknesses of Own Modelling through Comparison with Consensus Models), Fassen Erkenntnisse der Modellierung zusammen (Summarize Insights from Modelling), Fassen die Daten zusammen, um die Frage zu beantworten (Summarize the Data to Answer the Question), Formulieren eine Hypothese (Formulate a Hypothesis), Identifizieren die relevanten Daten oder den Beleg, der die Behauptung belegt (Identify the Relevant Data or Evidence Supporting the Claim), Konzentration auf bestimmte Aspekte (Focus on Specific Aspects), Nutzen die Manipulation zur Erklärung des Phänomens (Use Manipulation to Explain the Phenomenon), Reasoning (Reasoning), Stellen Veränderungen der Modellkomponenten angemessen dar (Represent Changes in Model Components Appropriately), Verknüpfen im Modell auftretende Variablen und formulieren Hypothesen zu deren Zusammenhang (Link Variables in the Model and Formulate Hypotheses about Their Relationship), Wählen einen geeigneten Versuchsansatz aus (Select an Appropriate Experimental Approach) |
| Mathematics | – |
| Physics | Beschreiben den Experimentieraufbau (Describe the Experimental Setup), Beschreiben die vorliegenden Daten (Describe the Available Data), Bestimmen mithilfe der Daten relevante Aspekte / Werte (durch Berechnung) (Determine Relevant Aspects/Values from Data (by Calculation)), Beziehen die relevanten Aspekte / Werte auf die vorhandenen Informationen (Relate the Relevant Aspects/Values to the Available Information), Claim (Claim), Fassen Erkenntnisse der Modellierung zusammen (Summarize Insights from Modelling), Fassen die Daten zusammen, um die Frage zu beantworten (Summarize the Data to Answer the Question), Formulieren erste Zusammenhänge, Muster und Strukturen in den Daten (Formulate Initial Relationships, Patterns, and Structures in the Data), Forschungsfragen stellen (Pose Research Questions), Identifizieren relevante Daten für die Erklärung/ Argumentation (Identify Relevant Data for Explanation/Argumentation), Reasoning (Reasoning), Wechseln die Darstellung der Daten (Change the Representation of Data) |
Table 6: Scientific-inquiry Skill ( SK ) item names per subject. Each entry is an inquiry competency that students are scored on independently.
Figure 7: Per-subject Macro-F1 on Test-UA for each subtask, averaged across all input format variants within each approach. Fine-tuned models only; zero-shot LLM prompting was not evaluated on Test-UA.
Figure 8: Per-subject Macro-F1 on Test-UQ for each subtask, averaged across all input format variants within each approach.
Table 7: Complementary QWK results on Alice . The table follows the same layout as Table 4 : upper panels show fine-tuned UA and UQ results, while the lower panel shows zero-shot UQ prompting. In the upper panels, bold marks the best result per subtask; in the lower panel, it marks the best result per input format and subtask. The underline in the upper panels marks cells where sequence classification outperforms rubric-retrieval. Red cells mark full-context results ( +qs / +qr / +qs+r ) that underperform the base input format ( ar / sa / -r ) for that model and subtask. Dashes indicate unavailable QWK values.
| Base Model | Input Format | UA (F1/Acc) | UQ (F1/Acc) |
| XLM-RoBERTa-Long | ar | 65.7 / 65.5 | 61.0 / 60.7 |
| +qs | 65.0 / 64.8 | 59.4 / 59.1 | |
| mmBERT | ar | 69.7 / 69.5 | 57.7 / 57.5 |
| +qs | 70.0 / 69.8 | 60.0 / 60.0 | |
| Llama-3.2-1B | ar | 72.1 / 71.9 | 59.0 / 58.8 |
| +qs | 73.5 / 73.2 | 59.9 / 59.5 |
Table 8: Contrastive loss (CL) results on Alice - LP . Gray cells indicate CL results that outperform the corresponding rubric-retrieval setting in Table 4 based on F1.
| Base Model | Input Format | UA (F1/Acc) | UQ (F1/Acc) |
| XLM-RoBERTa-Long | ar | 66.4 / 66.0 | 56.6 / 56.3 |
| +qs | 63.6 / 63.5 | 57.4 / 57.3 | |
| mmBERT | ar | 68.6 / 68.1 | 57.1 / 57.0 |
| +qs | 70.4 / 70.0 | 60.2 / 59.7 | |
| Llama-3.2-1B | ar | 70.0 / 69.6 | 57.8 / 57.9 |
| +qs | 73.6 / 73.2 | 61.6 / 61.0 |
Table 9: Separate encoding results on Alice - LP . Gray cells indicate separate-encoding results that outperform the corresponding rubric-retrieval setting in Table 4 based on F1.
| Joint Rubric-Retrieval | ||||||||||||
| Base Model | Input Format | Q1 | Q2 | Q3 | Q4 | Q5 | Q6 | Q7 | Q8 | Q9 | Q10 | Mean |
| BERT-base-cased | ar | 81.5 | 77.4 | 67.6 | 70.9 | 78.6 | 81.9 | 67.6 | 62.9 | 78.1 | 71.4 | 73.8 |
| +q | 77.2 | 62.0 | 66.1 | 69.5 | 63.3 | 58.7 | 66.3 | 63.6 | 82.8 | 73.8 | 68.3 | |
| RoBERTa-base | ar | 83.3 | 72.0 | 68.9 | 76.0 | 77.2 | 83.2 | 70.3 | 69.4 | 79.8 | 75.5 | 75.5 |
| +q | 81.1 | 67.5 | 66.4 | 71.9 | 68.0 | 66.5 | 69.0 | 67.1 | 81.3 | 76.9 | 71.6 | |
| ModernBERT-base | ar | 76.3 | 57.5 | 68.3 | 65.7 | 73.4 | 75.6 | 56.8 | 52.5 | 79.6 | 70.2 | 67.6 |
Table 10: Comparison of our rubric-retrieval framework with previous results on ASAP-SAS (QWK, %).