Stress Testing a Multimodal Large Language Model Grading Assistant on Kinematics Graph Reading
Organizations: LIACC - Laboratory of Artificial Intelligence and Computer Science, Faculty of Engineering, University of Porto, Porto, Portugal · Fraunhofer Portugal AICOS, Porto, Portugal · BrightFactory, Portugal · Rectorate and ETH AI Center, ETH Zurich, 8092 Zurich, Switzerland ·
Abstract
Multimodal large language models (MLLMs) can recognize graph features yet award credit without sufficient graphical support. This exploratory study examines selected grading outputs for synthetic responses to four kinematics exercises, generated through visual distortion and progressive transformation toward reference graphs. Different model and prompt configurations were used. Some judgments correctly distinguished instantaneous zero velocity from rest over a finite interval and recognized positive curvature. Yet a severely distorted velocity--time graph received full credit despite disconnected and branching traces and an unclear zero-velocity reference. Other submissions graded under the same configuration lost substantial credit. Numerical cutoffs intended to distinguish meaningful features from drawing irregularities were supplied without a coordinate convention for unnumbered graphs, leaving the basis of exact partial-credit awards unresolved. Stress testing can expose unsupported awards before classroom use. For the configurations examined, we do not recommend accepting scores as final grades without instructor review, including review of full-credit responses.
Figures & tables
| Criterion | Award | Grader’s explanation (abridged) |
|---|---|---|
| Forward motion | 3/3 | The velocity curve begins above zero. |
| Stopping interval | 4/4 | There is a sustained flat segment on the v = 0 line, indicating a stop. |
| Reverse motion | 3/3 | After the stop, the velocity values are below zero. |
| Total | 10/10 | All three criteria were marked satisfied. |
| Panel | Credit | Observed | Reason for credit |
|---|---|---|---|
| (a) | 0% (0/3) | After the initial plateau, the slope becomes increasingly negative, producing concave-down motion. Later concave-up recovery portions flatten the downward motion rather than establish the requested initial forward acceleration. | The initial acceleration phase has negative curvature, the wrong sign for the required positive second derivative. |
| (b) | 17% (0.5/3) | The early descending segment initially steepens downward. Concave-up bending appears locally around the trough, but the subsequent rise flattens briefly before joining the final line. Unnumbered axes prevent a calibrated numerical curvature measurement. | Positive curvature is limited to local portions rather than sustained throughout the first phase. Much of the early curvature has the wrong sign or is approximately zero. |
| (c) | 50% (1.5/3) | There is a modest increase in slope near the beginning, from a shallow rising segment to the steeper main segment. Positive curvature is concentrated around this short bend; much of the initial portion looks nearly straight. Unnumbered axes prevent a reliable numerical curvature measurement. | The early slope change provides evidence of positive acceleration, but the visible concavity is limited and does not clearly establish the required curvature throughout the first phase. |
| (d) | 80% (2.4/3) | The beginning bends upward as its slope increases from shallow to moderately positive. The curvature is modest and becomes difficult to distinguish from a straight line. Unnumbered axes prevent a calibrated numerical second-derivative measurement. | Positive initial curvature is visible, earning substantial credit, but the evidence for exceeding the specified curvature threshold is limited. |
| (e) | 100% (3/3) | The early section bends upward: its slope increases from roughly 0.25 to roughly 0.45 before the later straight portion. The curvature is consistent with exceeding 0.05 when estimated using the displayed grid spacing, although numerical axis scales are absent. | The first phase visibly has increasing positive slope, providing the required positive second derivative. |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Measure | Value |
|---|---|
| Mean squared error | 0.0372 |
| Root mean squared error | 0.193 |
| Pearson correlation | 0.28 |
| Spearman rank correlation | 0.36 |
Automated Grading
Automated grading uses machine learning and language models to score student work against answer keys or rubrics and provide feedback.
See all 98 papers