physics.ed-phNov 17, 2026

Stress Testing a Multimodal Large Language Model Grading Assistant on Kinematics Graph Reading

Authors: Ons Zammel, Carlos Soares, Flávia Carvalhido, Gerd Kortemeyer

Organizations: LIACC - Laboratory of Artificial Intelligence and Computer Science, Faculty of Engineering, University of Porto, Porto, Portugal · Fraunhofer Portugal AICOS, Porto, Portugal · BrightFactory, Portugal · Rectorate and ETH AI Center, ETH Zurich, 8092 Zurich, Switzerland · 

Abstract

Multimodal large language models (MLLMs) can recognize graph features yet award credit without sufficient graphical support. This exploratory study examines selected grading outputs for synthetic responses to four kinematics exercises, generated through visual distortion and progressive transformation toward reference graphs. Different model and prompt configurations were used. Some judgments correctly distinguished instantaneous zero velocity from rest over a finite interval and recognized positive curvature. Yet a severely distorted velocity--time graph received full credit despite disconnected and branching traces and an unclear zero-velocity reference. Other submissions graded under the same configuration lost substantial credit. Numerical cutoffs intended to distinguish meaningful features from drawing irregularities were supplied without a coordinate convention for unnumbered graphs, leaving the basis of exact partial-credit awards unresolved. Stress testing can expose unsupported awards before classroom use. For the configurations examined, we do not recommend accepting scores as final grades without instructor review, including review of full-credit responses.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks

    Nov 2, 2026Ahmad Jarjis Hasan, Jonathan LwowskiMLLM EvaluationVLM Evaluation

  2. LLMs as Teaching Assistants for Mathematics Exam Grading: Reliability, and Practical Usability

    Jun 1, 2026Aastha Sapkota, M. G. Sarwar MurshedLLM EvaluationMathematics Education

  3. Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams

    Sep 24, 2026Ali Habibullah, Yazan Alshoibi, Mohammad Alshiekh +2Prompt SensitivityLLM-as-a-Judge