Organizations: Tohoku University · Research and Development Center for Large Language Models, National Institute of Informatics · Osaka Kyoiku University · National Institute of Informatics · University of Tokyo
Although large language models (LLMs) show promise for generating educational feedback, it remains unclear how feedback should be designed (e.g., tone and coverage) to support answer revision and learner evaluations across learner profiles. We define six feedback designs for multiple-choice biology questions, including a baseline design and five variants with additional feedback elements, and conduct an empirical study with 321 high school students. We evaluate feedback using immediate revision performance and six subjective evaluation criteria, and analyze differences in subjective evaluations across learner profiles based on personality traits. Our results show that presenting task-relevant information clearly is associated with better immediate revision performance and is favorably evaluated across learner profiles, while we observe descriptive differences in evaluation patterns, particularly for informational novelty and affective framing. These findings support further investigation of personalized LLM feedback design.
Figures & tables
Figure 1: Overview of this study. First, we define six feedback elements related to feedback content and generate feedback using GPT-5. Next, based on data from 321 participants, we analyze immediate revision performance and subjective evaluations across six criteria. Finally, we examine which feedback elements are more favorably received across learners with different Big Five personality traits.
Figure 2: Overview of the experimental procedure. Participants first provide informed consent and complete a pre questionnaire. They then answer the task questions and evaluate feedback using a three-point scale across six criteria, such as trustworthiness and ease of understanding. If they make mistakes in their responses, they repeat the cycle of answering, reviewing the feedback, and evaluating it until they reach the correct answer. We randomly assign a feedback element to each question and keep it fixed throughout the revision process.
Figure 3: Accuracy of the first revised attempt. Normal , Keywords , and Coverage are the most effective methods for supporting answer revision, in this order.
Figure 4: Results of the subjective evaluations for each feedback element. Keywords , Normal , and Actionability tend to receive higher ratings across many criteria.
Elements
Guidance
Trust
Knowledge
Key Points
Understanding
Expression
Intercept (Normal.)
1.40 [0.92, 2.13]
1.76 [1.13, 2.75] ∗
1.42 [0.98, 2.05]
1.65 [1.09, 2.51] ∗
1.73 [1.16, 2.60] ∗∗
1.40 [0.92, 2.12]
Keywords
1.18 [0.76, 1.82]
0.75 [0.52, 1.09]
0.76 [0.50, 1.14]
1.00 [0.66, 1.51]
0.77 [0.52, 1.15]
0.76 [0.51, 1.14]
Actionability
1.07 [0.67, 1.71]
0.72 [0.48, 1.08]
0.86 [0.57, 1.30]
0.97 [0.62, 1.51]
0.80 [0.53, 1.20]
0.83 [0.54, 1.28]
Novelty
0.62 [0.40, 0.95] ∗
0.55 [0.34, 0.87] ∗
0.74 [0.48, 1.13]
0.60 [0.40, 0.90] ∗
0.47 [0.30, 0.73] ∗∗∗
0.46 [0.29, 0.74] ∗∗
Coverage
1.20 [0.80, 1.79]
0.80 [0.53, 1.20]
0.84 [0.54, 1.30]
1.08 [0.74, 1.58]
0.96 [0.63, 1.46]
0.84 [0.53, 1.33]
Positivity
0.86 [0.56, 1.31]
0.58 [0.41, 0.83] ∗∗
0.75 [0.51, 1.10]
0.75 [0.49, 1.13]
0.72 [0.48, 1.09]
0.71 [0.47, 1.06]
Table 1: Logistic GEE results for subjective evaluations across six criteria. Odds ratios (ORs) and 95% confidence intervals are reported relative to the Normal condition. ∗p<.05 , ∗∗p<.01 , ∗∗∗p<.001 .
Figure 5: Profiles of three learner clusters based on Big Five traits. Cluster 1 shows high Agreeableness and Extraversion, along with low Openness and Conscientiousness. Cluster 2 is characterized by higher Neuroticism and lower Extraversion than the other clusters. Cluster 3 exhibits high levels across all traits except Neuroticism.
Figure 6: Feedback evaluation results by cluster. Values are standardized for each learner and each evaluation dimension. In Social , “Trustworthiness” scores are similar across all feedback elements. In Reserved , Keywords receives high ratings, while in Adaptable , Keywords , Normal , and Actionability receive high ratings.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: User interface of the system used in this study. Participants read the question prompt, select an answer option, and submit their response. The system then provides feedback, which participants evaluate using a three-point scale across six criteria. This example shows Positivity .
Elements
Error
Conceptual
Stepwise
Knowledge
Identification
Explanation
Reasoning
Elaboration
Normal
28
0
29
7
Keywords
31
0
31
9
Actionability
32
0
32
7
Novelty
30
3
17
36
Coverage
30
0
30
11
Appendix
Table 2: Results of Cognitive Scaffolding analysis across feedback elements.
Elements
Preserved
Reduced
Hallucination
Normal
22
7
0
Keywords
31
2
0
Actionability
24
9
0
Novelty
30
0
0
Coverage
24
9
0
Positivity
29
2
0
Appendix
Table 3: Results of Pedagogical Reliability analysis across feedback elements.
Elements
Unfiltered n / Acc (%)
Filtered n / Acc (%)
Δ Acc
Unfiltered β (vs. Normal)
Filtered β (vs. Normal)
Normal
198 / 65.7
148 / 64.2
− 1.5pt
ref
ref
Keywords
204 / 53.0
153 / 58.2
+ 5.3pt
− 0.530 ∗
− 0.254 (n.s.)
Actionability
202 / 49.0
143 / 46.9
− 2.1pt
− 0.688 ∗∗
− 0.710 ∗∗
Novelty
213 / 52.1
213 / 52.1
± 0
− 0.564 ∗
− 0.499 ∗
Coverage
171 / 53.2
136 / 52.9
− 0.3pt
− 0.519 ∗
− 0.466 (marginal, p=.061)
Positivity
208 / 46.6
185 / 48.6
+ 2.0pt
− 0.783 ∗∗∗
− 0.638 ∗∗
Appendix
Table 4: Sensitivity analysis of revision performance using unfiltered and filtered data. Normal shows the highest revision accuracy in both datasets, while the overall pattern of results remains similar after excluding answer-leaking feedback. n denotes the number of corresponding responses, and Acc. denotes revision accuracy. p<.05 , p<.01 , p<.001 .
Criterion
Keywords
Actionability
Novelty
Coverage
Positivity
Guidance for Review
.027 / .040
.013 / .025
−.058 / −.064
−.027 / −.034
−.127∗∗ / −.135∗∗
Trustworthiness
.035 / .050
.025 / .058
−.013 / −.013
.007 / .020
−.093∗ / −.082∗
New Knowledge
−.015 / −.013
−.061 / −.012
.053 / .067
−.042 / −.022
−.123∗∗ / −.115∗∗
Key Points Clarity
−.008 / −.007
−.009 / .010
−.108∗∗ / −.125∗
−.047 / −.056
−.163∗∗∗ / −.169∗∗∗
Ease of Understanding
.005 / −.004
−.058 / −.041
−.148∗∗∗ / −.162∗∗∗
−.050 / −.042
−.119∗∗ / −.125∗∗
Expression Quality
.008 / .007
.001 / .029
−.095∗ / −.110∗∗
−.029 / −.019
−.148∗∗∗ / −.158∗∗∗
Appendix
Table 5: Sensitivity analysis of subjective evaluations using unfiltered and filtered data. Normal is the reference condition. Each cell reports the regression coefficient for the unfiltered / filtered analysis, respectively. ∗p<.05 , ∗∗p<.01 , ∗∗∗p<.001 .
95% CI
Predictor
β
SE
z
p
Lower
Upper
Intercept (Normal)
0.584
0.181
3.227
<.001
0.229
0.938
Keywords
− 0.254
0.242
− 1.048
.295
− 0.729
0.221
Actionability
− 0.710
0.243
− 2.921
.003
− 1.186
− 0.233
Novelty
− 0.499
0.221
− 2.259
.024
− 0.932
− 0.066
Coverage
− 0.466
0.248
− 1.875
.061
− 0.953
0.021
Appendix
Table 6: Generalized Estimating Equation (GEE) Results for First Revised Accuracy. N=978 . Reference category: Normal. GEE with robust covariance estimator; independence working correlation structure. Chi-square test: χ2(5)=12.611 , p=.027 .
95% CI
Predictor
β
SE
z
p
Lower
Upper
Criterion 1: Guidance for Review
Intercept (Normal)
2.499
0.037
66.72
0.000
2.425
2.572
Keywords
0.040
0.041
0.961
0.336
− 0.041
0.120
Actionability
0.025
0.045
0.553
0.580
− 0.063
0.112
Novelty
− 0.064
0.041
− 1.559
0.119
− 0.145
0.017
Appendix
Table 7: Detailed GEE results for each criterion. Positivity receives significantly lower ratings compared to Normal across all criteria. Additionally, Novelty shows significant negative effects in three criteria: “Key Points Clarity”, “Ease of Understanding”, and “Expression Quality”.
Criterion
Predictor
Main OR [95% CI]
Sensitivity OR [95% CI]
Review Guidance
Intercept (Normal)
1.40 [0.92, 2.13]
2.10 [1.26, 3.49] ∗∗
Keywords
1.18 [0.76, 1.82]
1.02 [0.65, 1.59]
Actionability
1.07 [0.67, 1.71]
1.00 [0.61, 1.63]
Novelty
0.62 [0.40, 0.95] ∗
0.59 [0.39, 0.88] ∗
Coverage
1.20 [0.80, 1.79]
1.21 [0.82, 1.78]
Positivity
0.86 [0.56, 1.31]
0.83 [0.56, 1.24]
Appendix
Table 8: Sensitivity analysis controlling for question-level variation. Odds ratios (ORs) and 95% confidence intervals are reported for the main logistic GEE model and the sensitivity model including question ID as a fixed effect. Normal is the reference condition. ∗p<.05 , ∗∗p<.01 , ∗∗∗p<.001 .
Figure 8: The result of elbow method. a distinct “elbow” or inflection point is observed around k=3 and k=4 , after which the rate of decrease becomes gradual.
Figure 9: This prompt is overview of the generation process.
Figure 10: This prompt outlines the specifications for each feedback element, consisting of PEDAGOGICAL_ELEMENT and DETAILED_INSTRUCTIONS . It presents the DETAILED_INSTRUCTIONS for Normal , Actionability , and Keywords . The remaining elements are detailed in Figure 11 .
Figure 11: This prompt continues the specifications for each feedback element from Figure 10 , specifically presenting the DETAILED_INSTRUCTIONS for Novelty , Coverage , and Positivity .
Item
Content
Task
Answer the following questions regarding ecosystem functions and human health. Humans live surrounded by various organisms, some of which can be harmful to our health. For example, diseases such as typhoid and dysentery are caused by (Ki: Bacteria), while others like measles and smallpox are caused by (Ku: Viruses). Some organisms enter the human body through food, leading to food poisoning. In addition, there are illnesses such as athlete \CJK@punctchar \CJK@uniPunct 0"80"99s foot caused by (Ke: Fungi). The processes shown in Figure 1 are essential for maintaining a balanced nutrient cycle within an ecosystem. However, human intervention often disrupts these functions, leading to various environmental issues. From the options 1 – 8 below, select the most appropriate combination of Disruption Causes (u–ka) and the resulting Environmental Problems (b–g).
(Disruption Causes)
[u] Discharge of hazardous substances such as heavy metals and DDT.
[e] Excessive use of chemical fertilizers.
[o] Factory emissions and automobile exhaust gases.
Appendix
Table 9: Examples of leaked feedback. These examples illustrate feedback generated when a learner selects 2 ( Coverage ) or 5 ( Baseline ) instead of the correct answer, 7 . The problem requires matching specific causes with their corresponding issues. Although the feedback does not explicitly state that 7 is the correct answer, it indirectly reveals the solution. By considering the causal combinations presented in the feedback, the learner inevitably identifies 7 as the only logical choice. The task and feedback shown here are translated from the original Japanese.
Item
Content
Task
Glucose is an essential energy source for the cells that constitute the human body, and it is continuously supplied to all cells via the bloodstream. … Therefore, to ensure stable cellular activity, the human body is equipped with a regulatory mechanism that maintains blood glucose levels (the concentration of glucose in the blood) within a constant range. Diabetes mellitus is a disease in which this regulation of blood glucose fails, resulting in the excretion of glucose in the urine. To diagnose diabetes and determine appropriate treatment strategies, there is a test in which 75 g of glucose is administered in a fasting state, and blood glucose levels and blood insulin concentrations are measured before and after ingestion. This test is known as the oral glucose tolerance test (OGTT). With regard to the mechanism described in brackets, select the most appropriate regulatory pathway that operates during hypoglycemia from among options 1 to 4 . グルコース（ブドウ糖）は，私たちの体を構成する細胞にとって重要なエネルギー源であり，血液によってすべての細胞に常に供給されている．… そこで，安定した細胞活動を保証するため，私たちの体には【血糖値（血液中のグルコース濃度）を一定に保つ血糖調節の仕組み】が備わっている． 糖尿病は，この血糖調節がうまくいかなくなり，尿中にグルコースが排出される病気である． 糖尿病の診断と治療方針を決めるため，空腹時に75gのグルコースを飲み，その前後で血糖値や血液中のインスリン濃度などを調べる検査がある． これを糖負荷試験という． 【】に関して，低血糖時にはたらく調節の反応経路として最も適当なものを，次の 1 ～ 4 のうちから一つ選べ．
There is an error in the combination of the parasympathetic nervous system and the hormone secreted by the pancreas in the regulatory pathway during hypoglycemia. During hypoglycemia, it is necessary to increase blood glucose levels; therefore, the sympathetic nervous system becomes dominant, and hyperglycemic hormones are secreted from the pancreas and adrenal glands. … Review the secretion sites of each hormone (pancreatic α -cells and β -cells, adrenal medulla and adrenal cortex) and organize whether each hormone acts to increase or decrease blood glucose levels. 低血糖時の調節経路について，副交感神経と膵臓から分泌されるホルモンの組み合わせに誤りがあります． 低血糖時には血糖値を上昇させる必要があるため，交感神経系が優位になり，膵臓や副腎から血糖上昇ホルモンが分泌されます．… 各ホルモンの分泌部位（膵臓の α 細胞・ β 細胞，副腎髄質・副腎皮質）と，それぞれが血糖値を上昇させるか低下させるかを整理し直してください．
Appendix
Table 10: Examples of feedback generated by each element. These tasks in this study are sourced from Japanese university entrance examinations. All tasks and feedback are originally presented in Japanese. The examples shown illustrate the feedback generated when a learner selects 2 instead of the correct answer, 1 .
English
Japanese
Title: Evaluating the Effects of Generative AI-Based Personalized Feedback on Learning Motivation in High-School Biology
タイトル： 生物基礎における生成AIによる個別フィードバックの学習意欲向上効果の検証
Research Objective: This study investigates how personalized feedback generated by generative AI for high-school biology questions influences learners’ motivation and understanding. The findings will contribute to research and development of generative AI technologies.
Generative AI produces feedback using multiple methods, and participants evaluate the quality of the feedback. By analyzing these evaluations, we identify which feedback characteristics are more effective for learning motivation and understanding.
Procedure: Participants complete the following activities as part of their winter vacation assignment: (1) submit answers through the system, (2) review and evaluate AI-generated feedback, (3) answer pre/post questionnaires, (4) complete confirmation tests, and (5) allow collection of learning logs such as study time and progress.
Data Usage and Consent: Collected data such as answers, questionnaires, and learning logs are gathered as part of educational activities. Whether these data are used for research purposes depends on participant consent.
If participants agree, anonymized data may be used for academic purposes such as research publications and conference presentations. If participants do not agree, they can still complete the assignment and receive AI-generated feedback without disadvantage.
Research Title: Evaluating the Effects of Generative AI-Based Personalized Feedback on Learning Motivation in High-School Biology
研究タイトル： 生物基礎における生成AIによる個別フィードバックの学習意欲向上効果の検証
This is a pre-questionnaire for evaluating the effects of personalized feedback generated by generative AI on learning motivation. Please select the option that best matches your thoughts. The questionnaire takes approximately 15 minutes and consists of 70 yes/no questions.
I often act without carefully considering problems.
問題を綿密に検討しないで、実行に移すことが多い
I tend to be somewhat lazy.
どちらかというと、怠惰な方です。
Compared with others, I am talkative.
他の人と比べると話好きです。
Appendix
Table 12: Big Five personality trait questionnaire items used in this study Murakami and Murakami (1997) . The questionnaire consists of 70 items.
English
Japanese
I tend to be cautious of others’ kindness.
人から親切にされると、何か下心がありそうで警戒しがちです。
I am not good at analyzing problems.
問題を分析するのは苦手な方です。
I have given up on things because I thought I lacked ability.
自分にはそれをする力がないと思って、あきらめてしまったことが何回かあります。
I think I could greatly contribute to society if given the opportunity.
機会さえあれば、大いに世の中に役立つことができるのにと思います。
I tend to have a quiet personality.
どちらかというと、おとなしい性格です。
I often quit things halfway.
何かに取り組んでも、中途半端でやめてしまうことが多い。
Appendix
Table 13: Remaining items from Table 12 of the Big Five personality trait questionnaire used in this study Murakami and Murakami (1997) .
English
Japanese
The following questions ask about your prior understanding, learning experience, and learning motivation regarding biology and AI-assisted learning. Please select the option that best matches yourself.
While recent studies have explored human behavior and preference simulation using large language models (LLMs), it remains unclear how well LLMs can simulate subjective evaluations from real learners in educational settings. We investigate this question using real learner evaluation data on feedback for high-school biology questions at both the group and individual levels. We compare performance with and without learner-specific information, such as personality traits and evaluation examples, across six models. Our results show that LLMs still have a limited ability to simulate learner evaluations. Providing learner profiles and examples improves score calibration and individual-level simulation, but more often fails to improve group-level consistency. These findings highlight the need to investigate which learner information and adaptation strategies are effective for learner preference simulation.
Tohoku University · Research and Development Center for Large Language Models, National Institute of Informatics · National Institute of Informatics +2
We investigate whether state-of-the-art large language models (LLMs) generate feedback that reflects the pedagogical practices of expert teachers in terms of feedback focus and adaptivity. Previous evaluation efforts have examined feedback characteristics, its impact on learning, and its target, yet the focus of feedback and its adaptivity remains largely overlooked. To bridge this gap, we adopt and refine Narciss's taxonomy into seven feedback focus types to annotate teacher and LLM-generated feedback across three university writing courses. We release FeedType, a benchmark containing annotated teacher and LLM feedback from six LLMs under three prompting strategies. We assess the coverage and distribution of feedback focus types, and examine whether LLMs adapt their feedback across draft stages and student performance levels as an expert instructor does. Our findings show that while most LLMs cover most feedback focus types, they fail to reflect teacher feedback distributions and show varying levels of adaptivity, with none matching the teachers' adaptive behavior. We believe FeedType will support future research on pedagogical alignment in LLM feedback generation.
Generative AI offers new opportunities for individualized and adaptive learning, e.g., through large language model (LLM)-based feedback systems. While LLMs can produce factually correct feedback for relatively straightforward conceptual tasks, delivering high-quality feedback for tasks that require advanced domain expertise, such as physics problem solving, remains a substantial challenge. This study presents the design and implementation of an LLM-based feedback system for physics problem solving grounded in evidence-centered design and reports a first evaluation within the German Physics Olympiad. Participants rated the usefulness and correctness of the generated feedback for each implemented problem. The collected ratings indicate that the feedback was generally perceived as useful and highly correct. However, an in-depth analysis revealed that the feedback contained errors in 20% of cases; errors that often went unnoticed by the students. We discuss the risks associated with uncritical reliance on LLM-based feedback and outline potential directions for generating more adaptive and reliable LLM-based feedback in the future.
Holger Maus, Fabian Kieser, Stefan Petersen +2
Department of Physics Education, Leibniz Institute for Science and Mathematics Education Kiel, Germany · Department of Physics Education Research, Free University of Berlin Berlin, Germany · Department of Physics and Physics Education Research, Ludwigsburg University of Education Ludwigsburg, Germany