cs.CLOct 7, 2026
SaveRubric Spans are Label Representations: Joint LLM Encoding for Short Answer Scoring
Organizations: DIPF | Leibniz Institute for Research and Information in Education · Centre for International Student Assessment (ZIB) · Computer Science Department & Studiumdigitale, Goethe University Frankfurt
Abstract
Automatic Short Answer Scoring (ASAS) requires models that can score student responses against question-specific criteria while remaining efficient and transferable across rubric sets. We propose RUSPAN, a rubric-conditioned ASAS framework that treats rubric descriptions as semantic label representations. RUSPAN serialises the question context, student answer, and all candidate rubric levels into a single sequence, then scores the levels listwise from the rubric-span and whole-sequence representations produced in a single LM pass. We further introduce RUSPAN-RIM, in which a Rubric-Independent Mask prevents rubric spans from attending to one another, making rubric representations depend only on the answer and question context and preventing overfitting to rubric patterns during training for zero-shot transfer. On six ASAS benchmarks spanning English, German, and Portuguese, RUSPAN improves mono-benchmark scoring over discriminative and generative baselines, while RIM with position reindexing delivers consistent and substantial gains on PT-ASAG, the held-out benchmark with the strongest combined language and rubric-structure shift.
Figures & tables
Figure 1: The Ruspan architecture. The formatted input combines the question context, student answer, rubric spans, and scoring instruction in one decoder-only LM pass. Span pooling extracts rubric representations from the final tokens of rubric spans and a sequence representation from the final token; an alignment function constructs each rubric-level representation before a shared scoring head projects it to a scalar score for softmax normalization.
Figure 2: Top : Attention masks for standard autoregressive attention (left) vs. attention after applying RIM (right). Green = allowed; pink = blocked. Under RIM, each rubric block attends only to the context ( , ) and itself, never to other rubric blocks. Bottom : Position ID reindexing applied to each rubric span so that every rubric block starts its positional count from the same anchor offset ( ), preserving relative positions within each span.
| Benchmark | Original Context | Augmented Context | Language | Levels | Test Splits | Sizes |
| Alice-LP Sun et al. (2026) | q, s, r | None | de | 3 | UA, UQ | 10,781 – 2,695 / 3,096 |
| ASAP-SAS | q, r | s | en | 3–4 | UA | 17,043 – 5,224 |
| iStudio Li et al. (2023) | q, s | r | en | 3 | UA | 5,226 653 653 |
| PT-ASAG Galhardi et al. (2018) | q, s | r | pt | 4 | UA | 3,290 – 366 |
| SciEntsBank 3-way Dzikovska et al. (2013) | q, s | r | en | 3 | UA, UQ, UD | 4,969 – 540 / 733 / 4,562 |
| Beetle-3-way Dzikovska et al. (2013) | q, s | r | en | 3 | UA, UQ | 3,941 – 439 / 819 |
Table 1: Overview of ASAS benchmarks and their augmentation. q : question; s : sample solution; r : rubrics. Test splits: UA (unseen answers), UQ (unseen questions), and UD (unseen domains). Sizes are reported as train dev test; – indicates no split; multiple test sizes follow the order of the listed test splits.
| Model | Method | ASAP-SAS | Alice-LP | iStudio | PT-ASAG | SciEntsBank | Beetle | |
| UA | UQ | UA | UA | UQ | UD | UQ | ||
| Llama-3.2-1B-Instruct | Seq. Class. | N/A | 60.9 | 85.4 | 67.4 | 66.0 | 61.8 | 66.2 |
| Rub. Retrieval | 74.3 | 65.5 | 86.9 | 65.0 | 68.0 | 65.6 | 67.2 | |
| Ans. Rub. Ali. | 74.1 | 61.1 | 85.8 | 57.1 | 61.2 | 65.1 | 55.2 | |
| Trained Gen. | 56.1 | 46.5 | 83.5 | 54.9 | 44.5 | 44.9 | 59.4 | |
| Ruspan | 77.6-77.7- 77.8 -76.7 | 64.4-62.3- 66.5 -64.1 | 87.4 -85.9-86.6-85.9 | 66.0-63.8- 68.2 -63.1 | 65.6-65.2-64.2- 69.2 | 68.8 -66.5-68.0-66.7 | 66.4-64.9-61.9-65.7 | |
Table 2: Main mono-benchmark results (macro-F1) on ASAS benchmarks. For Ruspan , hyphen-separated scores are ordered as concat - diff - condiff - no-align . Bold indicates the best score within each backbone and benchmark column. A baseline score marked with indicates a significant one-sided paired approximate-randomization improvement of the best Ruspan variant over that baseline ( ). Rubric Retrieval is omitted for 7B and 8B models due to high computational cost.
| Model | Method | iStudio | Alice-LP | PT-ASAG |
| UA | UQ | UA | ||
| Llama-3.2-1B-Instruct | Rub. Retrieval | 54.4 2.4 | 45.5 0.7 | 38.2 2.7 |
| Trained Gen. | 57.6 3.1 | 48.8 2.6 | 30.6 1.7 | |
| Zero-shot Gen. | 27.7 | 44.7 | 32.8 | |
| Ruspan | 55.8 5.5 | 48.4 2.1 | 28.1 0.6 | |
| Ruspan -RIM | 59.9 2.6 | 42.9 7.8 | 29.9 3.3 |
Table 3: Joint-training cross-benchmark macro-F1 (mean std over 3 random seeds). Bold : best method within each model group. Zero-shot Gen. results are from a single run.
| Method | Variable | Forward-pass per instance | Prediction mechanism | Task-trained |
| Seq. Classification | No | 1 | Fixed -way classification head | Yes |
| Rubric Retrieval | Yes | K | Expand each instance into answer-rubric pairs and take the rubric with the highest score | Yes |
| Trained Generation | Yes | 1 | Autoregressive score generation | Yes |
| Zero-shot Generation | Yes | 1 | Autoregressive score generation | No |
| Ruspan | Yes | 1 | parallel rubric-span scores | Yes |
Table 4: Structural comparison of ASAS methods. denotes the number of available levels. Variable indicates whether one trained model can handle different numbers of candidate levels.
Figure 3: Relative evaluation throughput on iStudio , normalized by Ruspan within each model size. Values above 1.0 indicate faster inference than Ruspan ; values below 1.0 indicate slower inference.
Figure 4: Per-level F1 by method. Top: mono-benchmark training. Bottom: cross-benchmark, averaged over random seeds. Bold marks the best result at each score level.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Field | English | German | Portuguese |
| question | <question> | <Frage> | <pergunta> |
| sample solution | <sample solution> | <Musterlösung> | <solução_exemplar> |
| answer | <answer> | <Antwort> | <resposta> |
| rubric | <rubric> | <rubric> | <rubric> |
Table 5: XML field tag translations by language.
| # | English | German | Portuguese |
| 1 | Determine the correctness of the student answer based on the information given. | Bestimmen Sie die Korrektheit der Schülerantwort basierend auf den gegebenen Informationen. | Determine a correção da resposta do aluno com base nas informações fornecidas. |
| 2 | Assess the response in relation to the given criteria. | Bewerten Sie die Antwort im Hinblick auf die angegebenen Kriterien. | Avalie a resposta em relação aos critérios fornecidos. |
| 3 | Determine if the answer aligns with the expected standards and provides a clear justification. | Bestimmen Sie, ob die Antwort den erwarteten Standards entspricht und eine klare Begründung liefert. | Determine se a resposta está alinhada com os padrões esperados e fornece uma justificativa clara. |
| 4 | Evaluate the answer based on the provided rubric and other context information. | Bewerten Sie die Antwort basierend auf der bereitgestellten Rubrik und anderen Kontextinformationen. | Avalie a resposta com base na rubrica fornecida e outras informações contextuais. |
| 5 | Score the response according to the rubric criteria and supporting materials. | Bewerten Sie die Antwort gemäß den Rubrikkriterien und unterstützenden Materialien. | Pontue a resposta de acordo com os critérios da rubrica e materiais de apoio. |
| 6 | Analyze whether the student’s answer meets the requirements outlined in the rubric. | Analysieren Sie, welche Rubrikkriterien die Schülerantwort erfüllt. | Analise se a resposta do aluno atende aos requisitos descritos na rubrica. |
Table 6: Full instruction pool ( ) in English, German, and Portuguese.
| English | German | Portuguese | |
| System | You are an expert grader of short student answers. You will receive a question, a sample solution, the student’s answer, and a numbered list of rubric levels (starting at 0). Read everything carefully, briefly explain your reasoning, and then state which rubric level best matches the student’s answer. | Sie sind eine Expertin bzw. ein Experte für die Bewertung kurzer Schülerantworten. Sie erhalten eine Frage, eine Musterlösung, die Antwort der Schülerin bzw. des Schülers sowie eine nummerierte Liste von Rubrik-Stufen (beginnend bei 0). Lesen Sie alles sorgfältig, erläutern Sie kurz Ihre Begründung und geben Sie anschließend an, welche Rubrik-Stufe am besten zur Schülerantwort passt. | Você é uma especialista ou um especialista em avaliação de respostas curtas de alunos. Você receberá uma pergunta, uma solução exemplar, a resposta do aluno e uma lista numerada de níveis da rubrica (começando em 0). Leia tudo com atenção, explique brevemente o seu raciocínio e, em seguida, indique qual nível da rubrica melhor corresponde à resposta do aluno. |
| Output format | End your response with a final line in this exact format: Final answer: <integer> , where <integer> is the chosen rubric level (between 0 and {max_score} , inclusive). | Beenden Sie Ihre Antwort mit einer letzten Zeile in genau diesem Format: Endgültige Antwort: <Ganzzahl> , wobei <Ganzzahl> die gewählte Rubrik-Stufe (zwischen 0 und {max_score} , einschließlich) ist. | Termine a sua resposta com uma última linha exatamente neste formato: Resposta final: <inteiro> , onde <inteiro> é o nível da rubrica escolhido (entre 0 e {max_score} , inclusive). |
| User template (structure identical across languages; field labels follow Table 5 ): | |||
| {question_label}: {question} | |||
| {solution_label}: {sample_solution} | |||
| {answer_label}: {answer} | |||
Figure 9: Zero-shot prompt templates in English, German, and Portuguese.
| Level | Human-written rubric | LLM-generated rubric |
| ASAP-SAS . Question: After reading the group’s procedure, describe what additional information you would need in order to replicate the experiment. Make sure to include at least three pieces of information. | ||
| 0 | The response describes little or no accurate or relevant information from the acid rain investigation. | Responses give no specific procedural details needed to replicate the acid-rain experiment and often include unrelated statements. They fail to identify any of the key missing items from the reference answers, such as type or amount of vinegar, sample identity or size, exposure time, container dimensions, rinsing procedure, or measurement methods. |
| 1 | The response describes one additional piece of information that would be needed to accurately replicate the experiment. | Responses identify exactly one specific procedural detail that is missing for replication, for example naming the type of vinegar, the exact volume of vinegar, or the duration the samples are submerged. They clearly state that single item but omit other essential parameters like sample size or shape, container specifications, rinsing steps, or measurement procedures. |
| 2 | The response describes two additional pieces of information that would be needed to accurately replicate the experiment. | Responses identify two distinct, specific pieces of missing procedural information, for example specifying both what the samples are and how much vinegar to use, or the exposure time and the instrument for measuring mass. They demonstrate partial sufficiency for replication but still omit at least one other critical element required for full reproducibility. |
| 3 | The response describes three additional pieces of information that would be needed to accurately replicate the experiment. | Responses list three specific procedural details that would enable replication, such as the material and size of the samples, the exact volume or concentration of vinegar to use, and the duration or environmental conditions for the exposure. They differ from lower scores by covering multiple different categories of missing information rather than repeating similar items. |
Table 7: Qualitative comparison of human-written and LLM-generated rubrics for an ASAP-SAS item.
| Level | Human-written rubric | LLM-generated rubric |
| Alice-LP . Question: Welche wissenschaftlichen Fragen fallen dir zur Färbung und Musterung der Guppys ein? Tipp: Wissenschaftliche Fragen sollten überprüfbar sein, beispielsweise durch Experimente. English: What scientific questions occur to you about the coloration and patterning of guppies? Tip: Scientific questions should be testable, for example through experiments. | ||
| 0 | Die Schüler:innen formulieren keine wissenschaftliche Frage zur Färbung und Musterung der Guppys genannt. ( English : Students do not formulate a scientific question about the coloration and patterning of the guppies. ) | Antworten zeigen ein grundlegendes Verständnis für die Färbung und Musterung der Guppys, jedoch fehlen klare, überprüfbare wissenschaftliche Fragen. Die Formulierungen sind ungenau und betreffen oft allgemeine Beobachtungen oder Vermutungen ohne spezifische wissenschaftliche Ansätze. ( English : Responses show a basic understanding of the coloration and patterning of guppies, but clear, testable scientific questions are missing. The formulations are imprecise and often concern general observations or conjectures without specific scientific approaches. ) |
| 1 | Die Schüler:innen formulieren eine wissenschaftliche Frage zur Färbung und Musterung der Guppys genannt mit Bezug auf Variation zwischen Individuen oder Geschlechtern. ( English : Students formulate one scientific question about the coloration and patterning of the guppies with reference to variation between individuals or sexes. ) | Antworten enthalten einige überprüfbare Fragen zur Färbung und Musterung der Guppys, jedoch sind diese oft weniger präzise und weniger tiefgehend. Die Fragen zeigen ein gewisses Verständnis für Unterschiede zwischen Männchen und Weibchen, thematisieren jedoch nicht alle relevanten wissenschaftlichen Aspekte oder Zusammenhänge. ( English : Responses contain some testable questions about the coloration and patterning of guppies, but these are often less precise and less in-depth. The questions show some understanding of differences between males and females, but do not address all relevant scientific aspects or relationships. ) |
| 2 | Die Schüler:innen formulieren mindestens zwei wissenschaftliche Fragen zur Färbung und Musterung der Guppys genannt mit Bezug auf Variation zwischen Individuen und Geschlechtern. ( English : Students formulate at least two scientific questions about the coloration and patterning of the guppies with reference to variation between individuals and sexes. ) | Antworten formulieren mehrere präzise und überprüfbare wissenschaftliche Fragen zur Färbung und Musterung der Guppys. Die Fragen zeigen ein tiefes Verständnis für biologische Konzepte wie Geschlechtsdimorphismus, genetische Variationen und deren Auswirkungen auf das Überleben und die Fortpflanzung der Guppys. ( English : Responses formulate several precise and testable scientific questions about the coloration and patterning of guppies. The questions show a deep understanding of biological concepts such as sexual dimorphism, genetic variation, and their effects on guppy survival and reproduction. ) |
Table 8: Qualitative comparison of human-written and LLM-generated rubrics for an Alice-LP item. The original German text is shown in gray and followed by an English translation.
| Model | Variant | iStudio | SciEntsBank | Beetle | ASAP-SAS | Alice-LP | |
| UA | UQ | UD | UQ | UA | UQ | ||
| Llama-3.2-1B-Instruct | concat | 84.8 | 65.3 | 64.5 | 64.5 | 77.8 | 61.3 |
| diff | 87.2 | 30.3 | 26.9 | 65.6 | 78.5 | 65.6 | |
| condiff | 86.8 | 66.9 | 65.9 | 66.2 | 76.9 | 62.6 | |
| no-align | 87.2 | 64.3 | 63.2 | 65.5 | 76.9 | 62.1 | |
| Seq. Class. w/o Rub. | 85.4 | 66.2 | 64.9 | 67.0 | N/A | 63.1 | |
Table 9: Quality-control ablations for rubric descriptions (macro-F1). For Ruspan variants, the left block reports performance when detailed generated rubric descriptions are replaced with short label names, and the right block reports performance when human-written rubric descriptions are replaced with LLM-generated rubric descriptions. For Sequence Classification, the row reports performance without appended rubric text, with N/A for ASAP-SAS . Light red marks cases where the ablated condition is worse for human-written rubric replacement or better for label-name and no-rubric ablations.
| Model | Variant | F1 |
| Llama-3.2-1B-Instruct | Ruspan - concat | |
| Ruspan - condiff | ||
| Llama-3.2-3B-Instruct | Ruspan - concat | |
| Ruspan - condiff | ||
| Mean | ||
Table 10: Effect of removing the LLM-generated sample solution from ASAP-SAS inputs. Values are changes in macro-F1 relative to the full-input setting (ablated minus full). This ablation applies only to ASAP-SAS .
Figure 10: Label distributions for the six ASAS benchmarks.
| Model | Method | PIQA | xStance | SemEval-2016 Task 6 | FigQA | IMDB | AG News | EIC | |
| Italian | Seen Targets | Donald Trump | |||||||
| Llama-3.2-1B-Instruct | Seq. Class. w/o Rub. | N/A | 78.4 | 73.6 | 61.3 | N/A | 96.4 | 95.2 | 80.2 |
| Seq. Class. w/ Rub. | 78.3 | 78.8 | 73.7 | 45.8 | 92.3 | 95.8 | 95.1 | 81.5 | |
| Ruspan - concat | 81.7 | 80.6 | 74.0 | 43.1 | 92.8 | 95.8 | 94.8 | 80.8 | |
| Ruspan - diff | 80.8 | 80.4 | 73.7 | 53.9 | 92.6 | 96.0 | 95.0 | 80.7 | |
| Ruspan - condiff | 80.5 | 79.7 | 76.1 | 60.5 | 92.8 | 96.3 | 95.1 | 80.5 | |
Table 11: Available non-ASAS benchmark results (macro-F1). Bold indicates the best completed result within each model group.