cs.CLOct 13, 2026
SaveMachine--human agreement in marking undergraduate assignments is specific to model, prompt and institution
Organizations: Centre for the Future of Intelligence, University of Cambridge, Cambridge, UK · Department of Computer Science, University of Lancashire, Preston, UK · School of Psychology, Manchester Metropolitan University, Manchester, UK · Department of Psychology, University of Cambridge, Cambridge, UK ·
Abstract
Large language models (LLMs) promise to automate one of higher education's most resource-intensive activities: assessing student writing. Existing evaluations are typically limited to a single model, a single prompt and a single institution. Using 761 undergraduate psychology assignments from three UK universities, we evaluated three frontier LLMs, individually and in ensemble, on accuracy, prompt sensitivity, error independence, generalisability and evidential basis. Agreement between human and machine marks varied widely, with no model or prompt optimal across institutions. All models compressed marks towards a common central tendency, overvaluing the weakest assignments and undervaluing the strongest. Because this error was shared, ensembling failed to remove it. Machine marks also rewarded linguistic features more strongly than human marks. Since human-machine agreement was a joint product of model, prompt and institution, we conclude that current LLMs lack evaluative capacity transferable across contexts, and that single-context validation overstates readiness for deployment.
Figures & tables
Figure 1: AI models track human marking only moderately, and every model compresses the grade scale. Machine marks plotted against human marks for each model and the inverse-RMSE weighted-mean ensemble (columns) at each institution (rows). Each point is one assignment from the held-out test set (Cambridge n = 106; Manchester Met n = 364; Nottingham n = 138), scored in a single zero-shot pass under the model’s best-calibrated prompt configuration. Solid lines are least-squares fits and dashed lines mark the identity. Panel insets report Spearman’s ρ and QWK. Fitted slopes are shallower than the identity line in every model–institution combination, indicating that machine marks systematically overvalue the weakest assignments and undervalue the strongest.
| Method | EA (%) | AA (%) | BA (%) | QWK | nQWK | MAE | RMSE | Bias | |
|---|---|---|---|---|---|---|---|---|---|
| Cambridge (n = 106) | |||||||||
| Claude | .631 | 14.2 | 37.7 | 54.7 | .493 | .672 | 6.3 | 8.2 | 4.2 |
| Gemini | .579 | 31.1 | 49.1 | 62.3 | .484 | .617 | 5.1 | 7.5 | 1.1 |
| GPT | .671 | 32.1 | 49.1 | 58.5 | .609 | .742 | 5.0 | 7.3 | 2.6 |
| Median† | .666 | 27.4 | 48.1 | 60.4 | .568 | .716 | 5.1 | 7.3 | 2.5 |
| Mean† | .679 | 10.4 | 50.0 | 61.3 | .616 | .719 | 5.2 | 7.3 | 2.7 |
Table 1: Machine–human scoring agreement is moderate, uneven across institutions, and no single model dominates. Scoring accuracy by method and institution on the held-out test set. Each method was evaluated on the identical set of assignments and against the identical human mark, so all within-block comparisons are paired. Complete cases only (Cambridge n = 106, Manchester Met n = 364, Nottingham n = 138). Individual models (Claude Opus 4.6, Gemini 3 Flash, GPT-5.4, abbreviated by provider) were evaluated under their best-calibrated prompt configuration (see Method). Boldface font indicates the best value per column within each institution (for bias, smallest absolute value). † denotes aggregation methods combining the three models' marks; ρ, Spearman correlation; W, inverse-RMSE weighting. EA, exact agreement (identical mark); AA, adjacent agreement (within 3 marks); BA, band agreement (same UK degree classification band, i.e. First, 2:1, 2:2, Third, Fail); QWK, quadratic weighted kappa; nQWK, QWK normalised against the maximum achievable given the two marginal distributions; MAE and RMSE, mean absolute and root mean squared error in grade points; Bias, mean signed error (negative = machine marks below human marks).
Figure 2: Effects of prompt design on marking accuracy are model-specific. (a) The fully crossed 3 × 3 × 3 prompt design: three levels each of criterion specificity, calibration intervention, and scoring strategy, composing 27 prompt configurations per model, scored on the calibration set. (b–d) Marginal root mean squared error (RMSE, in grade points, lower is better) on the calibration set (n = 153, 146 complete cases, assignments across all 81 model–configuration combinations), pooled across institutions, for each prompt property: (b) criterion specificity; (c) calibration intervention; (d) scoring strategy. Each point is a model’s RMSE marginalised over the other two factors (nine configurations per point). The three models diverge in which factor they are sensitive to: Gemini’s error rises sharply with the full rubric (b) and falls under analytic scoring (d), Claude benefits from the band-first strategy (d), and GPT is close to indifferent across all levels. Base rates help Claude and Gemini, but the added debiasing instruction reverses the gain (c).
Figure 3: Shared compression towards a common central tendency resists linear calibration. (a) Mean signed error (machine − human, grade points) as a function of the human mark. Errors cross zero near the lower boundary of the 2:1 band and grow monotonically towards both tails for every model (correlations between signed error and human mark: −0.79 to −0.83), such that assignment position on the grade scale alone explains 62–69% of error variance. (b) Ratio of machine to human mark standard deviations by institution. All cells fall below parity (dashed line), spanning 0.46 (Gemini, Cambridge) to 0.83 (Claude, Nottingham). (c) Density of human marks against raw and affine-corrected weighted-mean predictions at each institution. Matching the calibration set’s mean and variance requires stretch factors of 2.11 (Cambridge), 1.91 (Manchester Met) and 1.42 (Nottingham); the correction restores the spread of the marks but amplifies assignment-specific residuals alongside the signal, trading absolute accuracy for chance-corrected ordinal agreement while leaving rank agreement unchanged.
Figure 4: Machine–human agreement fails to generalise across institutions mainly through within-band accuracy, not grade composition. (a) Distribution of human marks by degree band at each institution. (b) Kitagawa decomposition of the weighted-mean ensemble’s mean absolute error gap between Manchester Met (8.3 grade points) and Cambridge (5.2): equalising cohort composition removes only 0.38 points of the difference; the remaining 2.75 points arise from lower within-band agreement between machine and human marks. (c) Contributions of grade composition and within-band accuracy to each institution’s MAE gap relative to Cambridge. Nottingham’s overall MAE is similar to Cambridge’s only because a favourable composition (−0.72 points) offsets within-band accuracy that is 0.78 points worse. Composition explains approximately one-eighth of the Manchester Met–Cambridge gap. The remainder reflects institution-specific structure, in the assignments, the writing, or the human benchmark itself, that is invisible to single-institution evaluation.
Figure 5: Machine marks are more strongly conditioned on surface linguistic features than human marks, and feature sensitivity varies by institution. Fisher z-transformed Spearman correlations between marks and each of the 33 extracted features (rows, grouped by linguistic domain) at each institution (columns), for human marks (red) and the weighted-mean ensemble (blue); connecting lines show the human–ensemble gap for each feature. Sample sizes are the held-out test set (Cambridge n = 106; Manchester Met n = 364; Nottingham n = 138). Ensemble correlations exceed human correlations in magnitude for 29 of 33 features at Manchester Met and 27 of 33 at Cambridge, but for only 14 of 33 at Nottingham, and the profile of feature–mark associations differs substantially across institutions, indicating that the influence of surface features on marking is context-dependent. Feature definitions are given in Supplementary Table S8.
| Human mark | Ensemble mark | |||||||
| Component | Var. (%) | b | SE | P | b | SE | P | |
| PC1 | Lexical richness & diversity | 24.1 | 0.58 | 0.14 | 0.001 | 0.73 | 0.06 | 0.001 |
| PC2 | Sentence complexity & readability | 19.2 | 0.01 | 0.17 | 0.954 | 0.34 | 0.08 | 0.001 |
| PC3 | Assignment length & information density | 16.2 | 0.98 | 0.19 | 0.001 | 0.84 | 0.09 | 0.001 |
| PC4 | Lexical density vs cohesion | 10.7 | 0.11 | 0.22 | 0.601 | 0.01 | 0.10 | 0.887 |
| PC5 | Conversational vs formal style | 6.8 | 0.52 | 0.26 | 0.051 | 0.14 | 0.12 | 0.255 |
Table 2: Machine marks are conditioned on a linguistic dimension undetected in human marks, and on the linguistic surface overall more strongly. Linear regressions of human and ensemble machine marks on the six PCA-derived linguistic components, held-out test set (n = 608). Var. (%) gives the percentage of variance in the feature set explained by each component (cumulative 79.8%), not variance in marks. Unstandardised coefficients (b) with standard errors and two-sided p values; institution was included as a dummy-coded covariate (Cambridge as reference; covariate coefficients in Supplementary Table S9). Boldface indicates coefficients significant at p < .05. Sentence complexity and readability (PC2) predicts the ensemble mark but not the human mark.
Table S1: Calibration-split RMSE for all 27 prompt configurations, by model. Root-mean-square error against the human mark on the calibration split for each cell of the 3 × 3 × 3 prompt factorial (A: criterion specificity, B: calibration intervention, C: scoring strategy). Bold values mark the pooled optimum for each model (Claude A1-B2-C3; Gemini A1-B2-C2; GPT A2-B3-C2), deployed unchanged across all three institutions.
Table S2: Repeated-measures ANOVA on per-assignment absolute error across the prompt factorial. Within-model analyses treat criterion specificity (A), calibration intervention (B), and scoring strategy (C) as within-assignment factors; the combined analysis adds model as an additional within-assignment factor. p (Holm), Holm-adjusted p values for the pre-specified family of nine per-model prompt-property main effects (three models × three properties). Interactions and the combined analysis were not part of this family and are marked –. p (LB) applies the lower-bound sphericity correction. An effect is recorded as surviving only if the corrected p < .05. ηp², partial eta squared.
Table S3: Planned contrasts for rubric and calibration effects on absolute and signed error. Paired t-tests on per-assignment error (df = 145), with the Holm correction applied across the full family of fifteen contrasts. Wilcoxon signed-rank tests are reported uncorrected as a distribution-free check. Δ is the mean difference in marks between the first- and second-named conditions, so negative values on |error| indicate lower absolute error under the first-named condition and signed contrasts indicate shifts in the mark level. dz, Cohen’s dz for paired designs.
Table S4: Counterfactual per-institution prompt selection and its transfer. For each model and institution: the locally optimal configuration on that institution’s calibration subset (n assignments as shown), its local RMSE, and the rank the deployed pooled optimum would attain among the 27 cells locally. ρ columns give Spearman correlations between institutions’ full cell orderings. Per-institution optima were never deployed; all reported test-set results use the pooled configurations of Supplementary Table S1.
Table S5: Pooled test-set accuracy for individual models and aggregation strategies. Held-out test set, all institutions pooled (n = 608). † indicates an aggregation over the three models’ marks. ρ, Spearman correlation with the human mark; EA/AA/BA, exact, adjacent, and band agreement; QWK and nQWK as defined in Section 3.5; MAE and RMSE in marks; Bias, mean signed error (machine − human). Bold marks the best value in each column. EA values for mean-based aggregates are not comparable to integer-valued methods, as averaging yields non-integer marks.
Table S6: Within-model test–retest stability and inter-model concordance. Upper panel: five repeated scoring runs on the reliability subset (n = 165 assignments) under the deployed configurations; ICC, intraclass correlation across runs, ICC(2,1), two-way random effects, single measures, absolute agreement; sd and Range, mean within-assignment standard deviation and range of marks; Band-stable %, share of assignments receiving the same degree classification on every run. Lower panel: inter-model concordance on the held-out test set, reporting ICC(2,k) for the three-model average and Krippendorff’s α.
Table S7: Self-reported confidence is near-constant within model and uninformative about error. Distribution of self-reported confidence (0–100) across all calibration-split scoring calls (153 assignments × 27 configurations), by model, with the Pearson correlation between confidence and per-assignment absolute error and its p value. Mode (share) gives the most frequent value and the proportion of calls emitting it.
Table S8: Definitions and extraction methods for the 33 assignment-level features. The 32 linguistic features span five domains established in the automated writing assessment literature – lexical diversity and sophistication, readability, syntactic and structural complexity, discourse and cohesion, and writing error – and are analysed together with word count. Extraction used spaCy and NLTK preprocessing in combination with domain-specific libraries: lexicalrichness for diversity indices, wordfreq for lexical sophistication, textstat for readability formulae, and symspellpy for grammar, spelling and punctuation detection (Section 3.6).
Figure S1: Machine marks correlate with a far broader set of surface linguistic features than human marks. Spearman rank-order correlations among the five mark variables (Human; GPT-5.4; Claude Opus 4.6; Gemini 3 Flash; weighted-mean ensemble) and the 33 extracted features, held-out test set (n = 608), with rows and columns grouped and colour-coded by linguistic domain. Asterisks denote significance after Holm correction across all 703 unique pairwise comparisons (*p < .05, **p < .01, ***p < .001). Human marks correlate reliably with 5 features; the weighted-mean ensemble with 20 of the 33, each in the same direction as the corresponding human correlation. Across all 33 features, the ensemble coefficient is the larger in magnitude for 28. The substantial within-domain intercorrelation among features motivates the principal component analysis reported in the main text.
Figure S2: Loadings and interpretation of the six retained principal components. (a) Loadings of the 33 standardised features (rows, colour-coded by linguistic domain) on the six retained components; the strongest loadings are annotated. (b) Label and substantive interpretation of each component. The six components jointly explain 79.8% of feature variance (24.1%, 19.2%, 16.2%, 10.7%, 6.8% and 2.8% respectively); retention was determined by the elbow criterion on the scree plot (Section 3.6).
Table S9: Per-model regressions of machine marks on the six linguistic components, and institutional covariates from the human and ensemble models. Upper panel: linear regressions of each individual model’s marks on the six PCA-derived linguistic components, held-out test set (n = 608), paralleling the human and ensemble models of Table 2; institution was included as a dummy-coded covariate (Cambridge as reference). Unstandardised coefficients (b) with standard errors and two-sided p values; boldface indicates p < .05; model fit gives R² and the F statistic. Lower panel: institutional covariate coefficients from the human and ensemble models of Table 2, omitted there for space. PC2 (sentence complexity & readability) significantly predicts every model’s marks, and per-model R² (0.320–0.365) substantially exceeds that of the human-mark model (0.111).
More in this topic
Automated Grading
Automated grading uses machine learning and language models to score student work against answer keys or rubrics and provide feedback.
See all 88 papers