Dec 11, 2025, physics.ed-phJ/K move · Enter open · S save
Holger Maus, Fabian Kieser, Stefan Petersen, Peter Wulff+1
Department of Physics Education, Leibniz Institute for Science and Mathematics Education Kiel, Germany · Department of Physics Education Research, Free University of Berlin Berlin, Germany · Department of Physics and Physics Education Research, Ludwigsburg University of Education Ludwigsburg, Germany
Generative AI offers new opportunities for individualized and adaptive learning, e.g., through large language model (LLM)-based feedback systems. While LLMs can produce factually correct feedback for relatively straightforward conceptual tasks, delivering high-quality feedback for tasks that require advanced domain expertise, such as physics problem solving, remains a substantial challenge. This study presents the design and implementation of an LLM-based feedback system for physics problem solving grounded in evidence-centered design and reports a first evaluation within the German Physics Olympiad. Participants rated the usefulness and correctness of the generated feedback for each implemented problem. The collected ratings indicate that the feedback was generally perceived as useful and highly correct. However, an in-depth analysis revealed that the feedback contained errors in 20% of cases; errors that often went unnoticed by the students. We discuss the risks associated with uncritical reliance on LLM-based feedback and outline potential directions for generating more adaptive and reliable LLM-based feedback in the future.