cs.CLFeb 12, 2026
SaveInvestigating Learner-Aware Design of LLM-Generated Educational Feedback
Organizations: Tohoku University · Research and Development Center for Large Language Models, National Institute of Informatics · Osaka Kyoiku University · National Institute of Informatics · University of Tokyo
Abstract
Although large language models (LLMs) show promise for generating educational feedback, it remains unclear how feedback should be designed (e.g., tone and coverage) to support answer revision and learner evaluations across learner profiles. We define six feedback designs for multiple-choice biology questions, including a baseline design and five variants with additional feedback elements, and conduct an empirical study with 321 high school students. We evaluate feedback using immediate revision performance and six subjective evaluation criteria, and analyze differences in subjective evaluations across learner profiles based on personality traits. Our results show that presenting task-relevant information clearly is associated with better immediate revision performance and is favorably evaluated across learner profiles, while we observe descriptive differences in evaluation patterns, particularly for informational novelty and affective framing. These findings support further investigation of personalized LLM feedback design.
Figures & tables
Figure 1: Overview of this study. First, we define six feedback elements related to feedback content and generate feedback using GPT-5. Next, based on data from 321 participants, we analyze immediate revision performance and subjective evaluations across six criteria. Finally, we examine which feedback elements are more favorably received across learners with different Big Five personality traits.
Figure 2: Overview of the experimental procedure. Participants first provide informed consent and complete a pre questionnaire. They then answer the task questions and evaluate feedback using a three-point scale across six criteria, such as trustworthiness and ease of understanding. If they make mistakes in their responses, they repeat the cycle of answering, reviewing the feedback, and evaluating it until they reach the correct answer. We randomly assign a feedback element to each question and keep it fixed throughout the revision process.
Figure 3: Accuracy of the first revised attempt. Normal , Keywords , and Coverage are the most effective methods for supporting answer revision, in this order.
Figure 4: Results of the subjective evaluations for each feedback element. Keywords , Normal , and Actionability tend to receive higher ratings across many criteria.
| Elements | Guidance | Trust | Knowledge | Key Points | Understanding | Expression |
| Intercept (Normal.) | 1.40 [0.92, 2.13] | 1.76 [1.13, 2.75] ∗ | 1.42 [0.98, 2.05] | 1.65 [1.09, 2.51] ∗ | 1.73 [1.16, 2.60] ∗∗ | 1.40 [0.92, 2.12] |
| Keywords | 1.18 [0.76, 1.82] | 0.75 [0.52, 1.09] | 0.76 [0.50, 1.14] | 1.00 [0.66, 1.51] | 0.77 [0.52, 1.15] | 0.76 [0.51, 1.14] |
| Actionability | 1.07 [0.67, 1.71] | 0.72 [0.48, 1.08] | 0.86 [0.57, 1.30] | 0.97 [0.62, 1.51] | 0.80 [0.53, 1.20] | 0.83 [0.54, 1.28] |
| Novelty | 0.62 [0.40, 0.95] ∗ | 0.55 [0.34, 0.87] ∗ | 0.74 [0.48, 1.13] | 0.60 [0.40, 0.90] ∗ | 0.47 [0.30, 0.73] ∗∗∗ | 0.46 [0.29, 0.74] ∗∗ |
| Coverage | 1.20 [0.80, 1.79] | 0.80 [0.53, 1.20] | 0.84 [0.54, 1.30] | 1.08 [0.74, 1.58] | 0.96 [0.63, 1.46] | 0.84 [0.53, 1.33] |
| Positivity | 0.86 [0.56, 1.31] | 0.58 [0.41, 0.83] ∗∗ | 0.75 [0.51, 1.10] | 0.75 [0.49, 1.13] | 0.72 [0.48, 1.09] | 0.71 [0.47, 1.06] |
Table 1: Logistic GEE results for subjective evaluations across six criteria. Odds ratios (ORs) and 95% confidence intervals are reported relative to the Normal condition. , , .
Figure 5: Profiles of three learner clusters based on Big Five traits. Cluster 1 shows high Agreeableness and Extraversion, along with low Openness and Conscientiousness. Cluster 2 is characterized by higher Neuroticism and lower Extraversion than the other clusters. Cluster 3 exhibits high levels across all traits except Neuroticism.
Figure 6: Feedback evaluation results by cluster. Values are standardized for each learner and each evaluation dimension. In Social , “Trustworthiness” scores are similar across all feedback elements. In Reserved , Keywords receives high ratings, while in Adaptable , Keywords , Normal , and Actionability receive high ratings.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: User interface of the system used in this study. Participants read the question prompt, select an answer option, and submit their response. The system then provides feedback, which participants evaluate using a three-point scale across six criteria. This example shows Positivity .
| Elements | Error | Conceptual | Stepwise | Knowledge |
| Identification | Explanation | Reasoning | Elaboration | |
| Normal | 28 | 0 | 29 | 7 |
| Keywords | 31 | 0 | 31 | 9 |
| Actionability | 32 | 0 | 32 | 7 |
| Novelty | 30 | 3 | 17 | 36 |
| Coverage | 30 | 0 | 30 | 11 |
Table 2: Results of Cognitive Scaffolding analysis across feedback elements.
| Elements | Preserved | Reduced | Hallucination |
| Normal | 22 | 7 | 0 |
| Keywords | 31 | 2 | 0 |
| Actionability | 24 | 9 | 0 |
| Novelty | 30 | 0 | 0 |
| Coverage | 24 | 9 | 0 |
| Positivity | 29 | 2 | 0 |
Table 3: Results of Pedagogical Reliability analysis across feedback elements.
| Elements | Unfiltered n / Acc (%) | Filtered n / Acc (%) | Acc | Unfiltered (vs. Normal) | Filtered (vs. Normal) |
| Normal | 198 / 65.7 | 148 / 64.2 | 1.5pt | ref | ref |
| Keywords | 204 / 53.0 | 153 / 58.2 | 5.3pt | 0.530 ∗ | 0.254 (n.s.) |
| Actionability | 202 / 49.0 | 143 / 46.9 | 2.1pt | 0.688 ∗∗ | 0.710 ∗∗ |
| Novelty | 213 / 52.1 | 213 / 52.1 | 0 | 0.564 ∗ | 0.499 ∗ |
| Coverage | 171 / 53.2 | 136 / 52.9 | 0.3pt | 0.519 ∗ | 0.466 (marginal, p=.061) |
| Positivity | 208 / 46.6 | 185 / 48.6 | 2.0pt | 0.783 ∗∗∗ | 0.638 ∗∗ |
Table 4: Sensitivity analysis of revision performance using unfiltered and filtered data. Normal shows the highest revision accuracy in both datasets, while the overall pattern of results remains similar after excluding answer-leaking feedback. denotes the number of corresponding responses, and Acc. denotes revision accuracy. , , .
| Criterion | Keywords | Actionability | Novelty | Coverage | Positivity |
| Guidance for Review | .027 / .040 | .013 / .025 | / | / | / |
| Trustworthiness | .035 / .050 | .025 / .058 | / | .007 / .020 | / |
| New Knowledge | / | / | .053 / .067 | / | / |
| Key Points Clarity | / | / .010 | / | / | / |
| Ease of Understanding | .005 / | / | / | / | / |
| Expression Quality | .008 / .007 | .001 / .029 | / | / | / |
Table 5: Sensitivity analysis of subjective evaluations using unfiltered and filtered data. Normal is the reference condition. Each cell reports the regression coefficient for the unfiltered / filtered analysis, respectively. , , .
| 95% CI | ||||||
| Predictor | SE | Lower | Upper | |||
| Intercept (Normal) | 0.584 | 0.181 | 3.227 | 0.229 | 0.938 | |
| Keywords | 0.254 | 0.242 | 1.048 | 0.729 | 0.221 | |
| Actionability | 0.710 | 0.243 | 2.921 | 1.186 | 0.233 | |
| Novelty | 0.499 | 0.221 | 2.259 | 0.932 | 0.066 | |
| Coverage | 0.466 | 0.248 | 1.875 | 0.953 | 0.021 | |
Table 6: Generalized Estimating Equation (GEE) Results for First Revised Accuracy. . Reference category: Normal. GEE with robust covariance estimator; independence working correlation structure. Chi-square test: , .
| 95% CI | ||||||
| Predictor | SE | Lower | Upper | |||
| Criterion 1: Guidance for Review | ||||||
| Intercept (Normal) | 2.499 | 0.037 | 66.72 | 0.000 | 2.425 | 2.572 |
| Keywords | 0.040 | 0.041 | 0.961 | 0.336 | 0.041 | 0.120 |
| Actionability | 0.025 | 0.045 | 0.553 | 0.580 | 0.063 | 0.112 |
| Novelty | 0.064 | 0.041 | 1.559 | 0.119 | 0.145 | 0.017 |
Table 7: Detailed GEE results for each criterion. Positivity receives significantly lower ratings compared to Normal across all criteria. Additionally, Novelty shows significant negative effects in three criteria: “Key Points Clarity”, “Ease of Understanding”, and “Expression Quality”.
| Criterion | Predictor | Main OR [95% CI] | Sensitivity OR [95% CI] |
| Review Guidance | Intercept (Normal) | 1.40 [0.92, 2.13] | 2.10 [1.26, 3.49] ∗∗ |
| Keywords | 1.18 [0.76, 1.82] | 1.02 [0.65, 1.59] | |
| Actionability | 1.07 [0.67, 1.71] | 1.00 [0.61, 1.63] | |
| Novelty | 0.62 [0.40, 0.95] ∗ | 0.59 [0.39, 0.88] ∗ | |
| Coverage | 1.20 [0.80, 1.79] | 1.21 [0.82, 1.78] | |
| Positivity | 0.86 [0.56, 1.31] | 0.83 [0.56, 1.24] |
Table 8: Sensitivity analysis controlling for question-level variation. Odds ratios (ORs) and 95% confidence intervals are reported for the main logistic GEE model and the sensitivity model including question ID as a fixed effect. Normal is the reference condition. , , .
Figure 8: The result of elbow method. a distinct “elbow” or inflection point is observed around and , after which the rate of decrease becomes gradual.
Figure 9: This prompt is overview of the generation process.
Figure 10: This prompt outlines the specifications for each feedback element, consisting of PEDAGOGICAL_ELEMENT and DETAILED_INSTRUCTIONS . It presents the DETAILED_INSTRUCTIONS for Normal , Actionability , and Keywords . The remaining elements are detailed in Figure 11 .
Figure 11: This prompt continues the specifications for each feedback element from Figure 10 , specifically presenting the DETAILED_INSTRUCTIONS for Novelty , Coverage , and Positivity .
| Item | Content |
| Task | Answer the following questions regarding ecosystem functions and human health. Humans live surrounded by various organisms, some of which can be harmful to our health. For example, diseases such as typhoid and dysentery are caused by (Ki: Bacteria), while others like measles and smallpox are caused by (Ku: Viruses). Some organisms enter the human body through food, leading to food poisoning. In addition, there are illnesses such as athlete \CJK@punctchar \CJK@uniPunct 0"80"99s foot caused by (Ke: Fungi). The processes shown in Figure 1 are essential for maintaining a balanced nutrient cycle within an ecosystem. However, human intervention often disrupts these functions, leading to various environmental issues. From the options 1 – 8 below, select the most appropriate combination of Disruption Causes (u–ka) and the resulting Environmental Problems (b–g). |
| (Disruption Causes) | |
| [u] Discharge of hazardous substances such as heavy metals and DDT. | |
| [e] Excessive use of chemical fertilizers. | |
| [o] Factory emissions and automobile exhaust gases. |
Table 9: Examples of leaked feedback. These examples illustrate feedback generated when a learner selects 2 ( Coverage ) or 5 ( Baseline ) instead of the correct answer, 7 . The problem requires matching specific causes with their corresponding issues. Although the feedback does not explicitly state that 7 is the correct answer, it indirectly reveals the solution. By considering the causal combinations presented in the feedback, the learner inevitably identifies 7 as the only logical choice. The task and feedback shown here are translated from the original Japanese.
| Item | Content |
| Task | Glucose is an essential energy source for the cells that constitute the human body, and it is continuously supplied to all cells via the bloodstream. … Therefore, to ensure stable cellular activity, the human body is equipped with a regulatory mechanism that maintains blood glucose levels (the concentration of glucose in the blood) within a constant range. Diabetes mellitus is a disease in which this regulation of blood glucose fails, resulting in the excretion of glucose in the urine. To diagnose diabetes and determine appropriate treatment strategies, there is a test in which 75 g of glucose is administered in a fasting state, and blood glucose levels and blood insulin concentrations are measured before and after ingestion. This test is known as the oral glucose tolerance test (OGTT). With regard to the mechanism described in brackets, select the most appropriate regulatory pathway that operates during hypoglycemia from among options 1 to 4 . グルコース（ブドウ糖）は，私たちの体を構成する細胞にとって重要なエネルギー源であり，血液によってすべての細胞に常に供給されている．… そこで，安定した細胞活動を保証するため，私たちの体には【血糖値（血液中のグルコース濃度）を一定に保つ血糖調節の仕組み】が備わっている． 糖尿病は，この血糖調節がうまくいかなくなり，尿中にグルコースが排出される病気である． 糖尿病の診断と治療方針を決めるため，空腹時に75gのグルコースを飲み，その前後で血糖値や血液中のインスリン濃度などを調べる検査がある． これを糖負荷試験という． 【】に関して，低血糖時にはたらく調節の反応経路として最も適当なものを，次の 1 ～ 4 のうちから一つ選べ． |
| Choices | 1 Diencephalon → Pituitary → Adrenal gland → Glucocorticoids 1 間脳 → 脳下垂体 → 副腎 → 糖質コルチコイド |
| 2 Diencephalon → Parasympathetic NS → Pancreas → Adrenaline 2 間脳 → 副交感神経 → 膵臓 → アドレナリン | |
| 3 Spinal cord → Sympathetic NS → Adrenal gland → Glucagon 3 脊髄 → 交感神経 → 副腎 → グルカゴン | |
| 4 Spinal cord → Parasympathetic NS → Pancreas → Insulin 4 脊髄 → 副交感神経 → 膵臓 → インスリン | |
| Normal | There is an error in the combination of the parasympathetic nervous system and the hormone secreted by the pancreas in the regulatory pathway during hypoglycemia. During hypoglycemia, it is necessary to increase blood glucose levels; therefore, the sympathetic nervous system becomes dominant, and hyperglycemic hormones are secreted from the pancreas and adrenal glands. … Review the secretion sites of each hormone (pancreatic -cells and -cells, adrenal medulla and adrenal cortex) and organize whether each hormone acts to increase or decrease blood glucose levels. 低血糖時の調節経路について，副交感神経と膵臓から分泌されるホルモンの組み合わせに誤りがあります． 低血糖時には血糖値を上昇させる必要があるため，交感神経系が優位になり，膵臓や副腎から血糖上昇ホルモンが分泌されます．… 各ホルモンの分泌部位（膵臓の 細胞・ 細胞，副腎髄質・副腎皮質）と，それぞれが血糖値を上昇させるか低下させるかを整理し直してください． |
Table 10: Examples of feedback generated by each element. These tasks in this study are sourced from Japanese university entrance examinations. All tasks and feedback are originally presented in Japanese. The examples shown illustrate the feedback generated when a learner selects 2 instead of the correct answer, 1 .
| English | Japanese |
| Title: Evaluating the Effects of Generative AI-Based Personalized Feedback on Learning Motivation in High-School Biology | タイトル： 生物基礎における生成AIによる個別フィードバックの学習意欲向上効果の検証 |
| Research Objective: This study investigates how personalized feedback generated by generative AI for high-school biology questions influences learners’ motivation and understanding. The findings will contribute to research and development of generative AI technologies. | 研究の目的： この研究は、高校生を対象に、生物基礎の問題に対して生成AIが作る個別フィードバック（コメント）が、どのような要素を持つと学習意欲や理解に効果的かを調査し、得られた知見等を広く生成AIの研究開発や普及に役立てることを目的としています。 |
| Generative AI produces feedback using multiple methods, and participants evaluate the quality of the feedback. By analyzing these evaluations, we identify which feedback characteristics are more effective for learning motivation and understanding. | 複数の方法で生成AIがフィードバックを生成し、その効果を高校生の皆さんに評価してもらいます。その評価結果を分析することで、どの要素のフィードバックがより学習意欲や理解に効果的かを明らかにします。 |
| Procedure: Participants complete the following activities as part of their winter vacation assignment: (1) submit answers through the system, (2) review and evaluate AI-generated feedback, (3) answer pre/post questionnaires, (4) complete confirmation tests, and (5) allow collection of learning logs such as study time and progress. | 実施手順： 冬休みの課題の一環として、以下の手順で活動を行っていただきます。 (1) 課題の解答をシステム上で提出 (2) 生成AIが作成したフィードバック（コメント）の確認・評価 (3) 事前アンケート・事後アンケートへの回答 (4) 確認テストの実施 (5) 学習ログ（取り組み時間、進捗、課題達成率など）の収集 |
| Data Usage and Consent: Collected data such as answers, questionnaires, and learning logs are gathered as part of educational activities. Whether these data are used for research purposes depends on participant consent. | データの取り扱いと同意について： この活動で得られるデータ（課題解答、アンケート、学習ログなど）は、教育活動の一環として収集されます。それらの収集されるデータを研究目的で利用するかどうかについては、皆さんの同意に基づいて決定されます。 |
| If participants agree, anonymized data may be used for academic purposes such as research publications and conference presentations. If participants do not agree, they can still complete the assignment and receive AI-generated feedback without disadvantage. | 「同意する」を選択した場合： 収集したデータを研究・学会発表・論文執筆などの学術目的に使用します。 データは匿名化され、個人が特定されることはありません。 「同意しない」を選択した場合： 研究利用は行いませんが、通常どおりAIによるフィードバックを受け取り、課題を行うことができます。 |
Table 11: Consent Form Used in the Empirical Experiment.
| English | Japanese |
| Title: Evaluating the Effects of Generative AI-Based Personalized Feedback on Learning Motivation in High-School Biology: Pre-questionnaire | タイトル： 生物基礎における生成AIによる個別フィードバックの学習意欲向上効果の検証 事前アンケート |
| Research Title: Evaluating the Effects of Generative AI-Based Personalized Feedback on Learning Motivation in High-School Biology | 研究タイトル： 生物基礎における生成AIによる個別フィードバックの学習意欲向上効果の検証 |
| This is a pre-questionnaire for evaluating the effects of personalized feedback generated by generative AI on learning motivation. Please select the option that best matches your thoughts. The questionnaire takes approximately 15 minutes and consists of 70 yes/no questions. | これは、生成AIによる個別フィードバックの学習意欲向上効果の検証のための事前アンケートです。 以下の質問の中から、最も近い選択肢を選択してください。 所要時間はおよそ15分です。 以下の質問に対して、「はい」または「いいえ」で回答してください。 質問は70問あります。 |
| I often act without carefully considering problems. | 問題を綿密に検討しないで、実行に移すことが多い |
| I tend to be somewhat lazy. | どちらかというと、怠惰な方です。 |
| Compared with others, I am talkative. | 他の人と比べると話好きです。 |
Table 12: Big Five personality trait questionnaire items used in this study Murakami and Murakami (1997) . The questionnaire consists of 70 items.
| English | Japanese |
| I tend to be cautious of others’ kindness. | 人から親切にされると、何か下心がありそうで警戒しがちです。 |
| I am not good at analyzing problems. | 問題を分析するのは苦手な方です。 |
| I have given up on things because I thought I lacked ability. | 自分にはそれをする力がないと思って、あきらめてしまったことが何回かあります。 |
| I think I could greatly contribute to society if given the opportunity. | 機会さえあれば、大いに世の中に役立つことができるのにと思います。 |
| I tend to have a quiet personality. | どちらかというと、おとなしい性格です。 |
| I often quit things halfway. | 何かに取り組んでも、中途半端でやめてしまうことが多い。 |
Table 13: Remaining items from Table 12 of the Big Five personality trait questionnaire used in this study Murakami and Murakami (1997) .
| English | Japanese |
| The following questions ask about your prior understanding, learning experience, and learning motivation regarding biology and AI-assisted learning. Please select the option that best matches yourself. | これから取り組む生物基礎の学習やAIを使った学習に関して、皆さんの事前の理解度・学習経験・学習意欲についてお聞きします。質問はすべて、あなた自身に最もあてはまるものを選んでください。 |
| I am good at basic biology. | 生物基礎は得意な教科である |
| 1: Very poor 5: Very good | 1: とても苦手 5: とても得意 |
| How frequently do you use generative AI (e.g., ChatGPT, Gemini, Copilot) in daily life? | あなたは、日常生活の中で生成AI（例：ChatGPT、Gemini、Copilotなど）をどの程度の頻度で利用していますか |
| Never | 全く利用していない |
| Occasionally (less than once a month) | ときどき利用する（1ヶ月に1回未満） |
Table 14: Additional Pre-questionnaire Items Used in the Empirical Experiment.