Transformer-based language models have achieved remarkable success in automated short answer grading (ASAG). However, current methods in ASAG struggle to comprehend, reason about and evaluate complex mathematical answers involving symbols and formulas. These score-only methods fail to generalize across diverse responses and skill states associated with identical scores. Moreover, they ignore the potential upward bias introduced by generative Artificial Intelligence (AI) usage in assessing students' performance and knowledge mastery. To address these limitations, we propose AICG (AI-aware Cognitive-supervised Grading), a two-branch multi-task framework. The first branch combines RoBERTa-based regression, ordinal, and classification heads for individual-answer grading. The second branch uses an expert-designed Q-matrix to associate responses with assessed skills and jointly predicts G-DINA-derived skill mastery and learning gain. Through the shared encoder, these skill-level targets provide cognitive supervision for answer grading beyond score labels. AICG further uses a Gen-AI detector, trained through supervised contrastive pretraining followed by linear classification, to estimate assistance probabilities and convert them into reliability weights, attenuating unreliable cognitive evidence during learning assessment. We conduct experiments on 49,402 instructor-graded responses spanning 19 academic semesters. Relative to RoBERTa regression, AICG reduces mean absolute error (MAE) by 50.3% and improves quadratic weighted kappa (QWK) from 0.798 to 0.927. Ablations identify Q-matrix-guided cognitive supervision as the primary source of improvement, while AI-aware weighting further reduces overall MAE by 4.2% over the unweighted cognitive model and consistently improves MAE and QWK across the four most recent post-Gen-AI semesters.
Figures & tables
Figure 2: Overview of AICG. The Gen-AI Detector supplies AI-assistance probabilities only to the Q-Matrix-Based Cognitive Supervision branch. The shared RoBERTa encoder supports parallel cognitive-supervision and individual-response-grading branches. Solid and dashed arrows denote prediction and supervision flows, respectively.
Evaluation
N
Acc.
Prec.
Recall
F1
Within-semester Testing (S18-S22)
1,004
.959
.955
.964
.960
Cross-semester Testing (F14-S17)
11,458
.951
.959
.942
.950
Table 1: Gen-AI detector performance. F17 is excluded because no data were collected in that semester.
Method
MAE ↓
± .05 ↑
± .10 ↑
± .20 ↑
Exact ↑
Macro-F1 ↑
QWK ↑
RoBERTa ( Liu et al. 2019 )
.0873 (.002)
.709 (.007)
.752 (.006)
.832 (.005)
.751 (.009)
.564 (.012)
.798 (.004)
GPT-4o Scoring ( Ferreira Mello et al. 2025 )
.1140 (.002)
.606 (.010)
.662 (.008)
.776 (.006)
.636 (.013)
.439 (.011)
.674 (.006)
BERT ( Zhu et al. 2022 )
.0981 (.003)
.675 (.009)
.727 (.007)
.806 (.007)
.704 (.008)
.477 (.009)
.753 (.005)
ASQ-DBN ( Zhang et al. 2020 )
.0946 (.003)
.682 (.009)
.734 (.008)
.814 (.006)
.721 (.010)
.516 (.011)
.769 (.005)
GradeOpt ( Chu et al. 2025 )
.0788 (.003)
.748 (.008)
.796 (.007)
.861 (.006)
.786 (.009)
.612 (.010)
.842 (.005)
RoBERTa+O
.0842 (.002)
.724 (.004)
.763 (.006)
.844 (.003)
.783 (.006)
.611 (.007)
.813 (.003)
Table 2: Overall grading performance across F22–F24. Values are mean (standard deviation) over five repetitions. Best completed results are shown in bold.
Sem.
Method
MAE ↓
± .10 ↑
Exact ↑
Macro-F1 ↑
QWK ↑
F22
RoBERTa
.0904
.744
.739
.553
.791
RoBERTa+O
.0882
.759
.772
.602
.806
Full Scoring
.0873
.763
.776
.615
.826
+ Gain
.0759
.817
.835
.674
.866
+ State
.0578
.889
.906
.732
.900
+ Full Cognitive
.0497
.918
.929
.761
.922
Table 3: Ablation results at the beginning and end of the expanding-window evaluation. Values are means over five seeds.
AI
MAE ↓
QWK ↑
Sem.
Usage
FullCog
AICG
FullCog
AICG
F22
13.40%
.0497
.0493
.922
.918
S23
23.15%
.0507
.0489
.914
.922
F23
23.56%
.0516
.0482
.924
.926
S24
20.41%
.0358
.0336
.923
.929
F24
23.85%
.0402
.0379
.934
.940
Table 4: Semester-level effect of AI-aware weighting. FullCog denotes Full Scoring + Full Cognitive.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Semester
Students
Pre Answers
Post Answers
Total Answers
AI Answers
AI Percentage
Pre-Gen-AI
2,530
19,076
16,452
35,528
0
0.00%
F22
277
2,020
1,084
3,104
416
13.40%
S23
170
1,347
735
2,082
482
23.15%
F23
376
2,946
1,531
4,477
1,055
23.56%
S24
218
890
898
1,788
365
20.41%
F24
290
1,206
1,217
2,423
578
23.85%
Appendix
Table 5: Composition of the longitudinal ASAG dataset.
Test Semester
Historical Training Window
F22
F14–S22
S23
F14–F22
F23
F14–S23
S24
F14–F23
F24
F14–S24
Appendix
Table 6: Expanding-window evaluation splits.
Phase
Item
CT
MET
IE
DMT
A2
A3
CIE
CP
TPT
BR
Pre
60
0
0
0
0
0
0
0
0
1
0
Pre
63
0
0
0
0
0
0
0
1
0
0
Pre
64
1
0
0
0
0
0
1
0
0
1
Pre
65
1
0
0
1
1
0
0
0
0
0
Pre
66
0
0
0
0
0
0
0
0
0
1
Pre
67
0
1
1
0
0
1
0
0
0
0
Appendix
Table 7: Expert Q-matrix for the pretest and aligned posttest items.
Component
Setting
Final Value
Gen-AI Detector: contrastive pretraining
Backbone
roberta-base
Maximum sequence length
512 tokens
Projection dimension
768
Balanced mini-batch size
64 (32 per class)
Optimizer
AdamW
Learning rate
2×10−5
Appendix
Table 8: Implementation settings for each component of AICG.
DIPF | Leibniz Institute for Research and Information in Education · Faculty of Computer Science, Goethe University Frankfurt · 3Chemnitz University of Technology