cs.AIOct 10, 2026
SaveSocraticRepair: Planning-Guided Socratic Tutoring for Program Debugging and Misconception Repair
Organizations: University of Warwick · Warwick Manufacturing Group
Abstract
Large language models are increasingly used in computer science education. However, the teaching abilities of even the most powerful LLMs are limited: they tend to reveal the solution prematurely and may lose focus on the learner's current debugging objective. To address this, we introduce SocraticRepair, a planning-guided Socratic tutoring framework for program debugging and miscon- ception repair that adapts prior planning-guided tutoring work (Li et al., 2026b). The framework is instantiated in the Python programming domain by fine-tuning Qwen2.5-Coder-7B-Instruct on synthetic multi-turn tutoring dialogues constructed from four preprocessed programming datasets. The system is evaluated using a predefined set of independent metrics and benchmarked against a general-purpose LLM. The experiments show that SocraticRepair achieves higher scores on stepwise scaffolding and question quality while also substantially reducing premature solution disclosure. The results also identify reliable pedagogical plan generation as an important remaining challenge.
Figures & tables
Figure 1: SocraticRepair framework. The same fine-tuned model operates in planning and tutoring modes. In planning mode, the model generates a stepwise pedagogical plan from the programming case. In tutoring mode, the model generates the learner state, step assessments (undemonstrated steps that have been demonstrated in the latest student utterance) and a Socratic reply given a programming case, latest submitted code, execution output, a plan and a progress through the plan.
| Learner state | Tutoring action | Active objective |
| START | Ask an initial guiding question | Unchanged |
| INCORRECT | Re-ask or provide a targeted hint | Not demonstrated |
| CONFUSION | Simplify the current objective or provide a hint | Not demonstrated |
| QUESTION | Answer the clarification and steer the student back to the objective | Not demonstrated |
| IRRELEVANT | Refocus the student on the current objective | Not demonstrated |
| CORRECT / COMPREHENSION | Advance when sufficient evidence demonstrates the objective, otherwise continue eliciting evidence | May be demonstrated |
Table 1: Learner state tutoring policy, adapted from ( Li et al., 2026b ) . Unlike ScaffoldLM, SocraticRepair requires sufficient evidence that a pedagogical objective has been demonstrated before marking it as completed; the learner state alone is not sufficient.
| Dataset | Case ID | Problem | Buggy code | Tests | Failure | Bugs description | Correct code | Source |
| PYMETA | D | G | D | G | X | G | G | D |
| MULTI_DEBUG | D | D | D | G | X | D | D | D |
| QUIXBUGS | D | D | D | D | X | G | D | D |
| INTRO_PROG | D | C | C | D | X | G | C | D |
Table 2: Origin of the fields used in the common benchmark representation. C denotes a value copied directly from the source dataset; D denotes a value deterministically extracted; G denotes content generated or reconstructed by an LLM-based preprocessing agent; and X denotes a value obtained by executing the buggy implementation against the tests.
| Dataset | Train | Validation | Test | Total |
| PYMETA | 242 | 29 | 29 | 300 |
| MULTI_DEBUG | 120 | 15 | 15 | 150 |
| QUIXBUGS | 32 | 4 | 4 | 40 |
| INTRO_PROG | 26 | 5 | 5 | 36 |
| Total | 420 | 53 | 53 | 526 |
Table 3: Distribution of programming cases across data sources and train, validation and test partitions.
| Learner state | RECEPTIVE | UNCERTAIN | PERSISTENT |
| CORRECT | 0.40 | 0.30 | 0.25 |
| INCORRECT | 0.20 | 0.15 | 0.35 |
| QUESTION | 0.10 | 0.25 | 0.15 |
| COMPREHENSION | 0.10 | 0.10 | 0.05 |
| CONFUSION | 0.10 | 0.15 | 0.15 |
| IRRELEVANT | 0.10 | 0.05 | 0.05 |
Table 4: Learner state sampling probabilities for the three simulated student variants.
| Metric | Evaluated property | Literature basis |
| Technical correctness | Technical accuracy and consistency with the program, execution evidence and trusted repair. | ScaffoldEval: Answer Accuracy |
| Debugging relevance | Alignment with the student’s current objective. | ScaffoldEval: Topic Adherence; TreeInstruct: Relevance |
| Stepwise scaffolding | Step by step guidance that leaves meaningful reasoning work to the student. | ScaffoldEval: Stepwise Scaffolding; TreeInstruct: Indirectness |
| Guidance progression | Productive movement towards unresolved objectives without unnecessary repetition. | ScaffoldEval: Guidance Quality; TreeInstruct: Logical Flow |
| Adaptive feedback | Responsiveness to the student’s latest utterance. | ScaffoldEval: Adaptive Feedback |
| Debugging question quality | Use of questions that elicit useful explanation | ScaffoldEval: Question Quality |
Table 5: Turn level tutor quality dimensions adapted from prior tutoring evaluations ( Li et al., 2026b ; Kargupta et al., 2024 ) .
| SocraticRepair | GPT-4o | |
| Evaluation dialogues | 159 | 159 |
| Successful dialogues | 148 (93.1%) | 152 (95.6%) |
| Mean rounds | 8.04 | 4.99 |
| Premature-disclosure turns | 24 / 1,276 (1.88%) | 59 / 794 (7.43%) |
| Dialogues with disclosure | 20 / 159 (12.58%) | 53 / 159 (33.33%) |
Table 6: Dialogue level outcomes and premature solution disclosure for SocraticRepair and GPT-4o.
| Plan source | Attempts | Technically valid |
| SocraticRepair | 53 | 11 |
| GPT-4o + verifier | 53 | 53 |
Table 7: Technical validity of SocraticRepair-generated pedagogical plans for the held-out evaluation cases.
| Metric | SocraticRepair | GPT-4o | Holm-adjusted | |
| Technical correctness | 1.789 | 1.860 | -0.071 | .002 |
| Debugging relevance | 1.929 | 1.949 | -0.021 | .118 |
| Stepwise scaffolding | 1.876 | 1.789 | +0.087 | .006 |
| Guidance progression | 1.655 | 1.661 | -0.006 | .385 |
| Adaptive feedback | 1.792 | 1.843 | -0.051 | .036 |
| Debugging question quality | 1.626 | 1.446 | +0.180 |
Table 8: Dialogue averaged tutor quality scores over 158 matched interactions. Scores range from 0 to 2 and denotes SocraticRepair minus GPT-4o. Statistical significance is assessed using paired two-sided Wilcoxon signed-rank tests with Holm correction across the six metrics.
| Learner state | Turns (SR/GPT) | SR Adapt. | GPT Adapt. | SR state acc. |
| CORRECT | 430 / 213 | 1.840 | 1.939 | 95.3% |
| INCORRECT | 236 / 150 | 1.640 | 1.660 | 91.9% |
| QUESTION | 156 / 103 | 1.833 | 1.864 | 100.0% |
| COMPREHENSION | 117 / 56 | 1.897 | 1.982 | 58.1% |
| CONFUSION | 109 / 67 | 1.560 | 1.522 | 98.2% |
| IRRELEVANT | 69 / 47 | 1.812 | 1.809 | 100.0% |
Table 9: Adaptive feedback scores by controlled learner state and SocraticRepair learner state prediction accuracy. The fixed START state is omitted.
| Bugs | Matched | Tech. | Relevance | Scaff. | Progression | Adapt. | Q. qual. |
| 1 | 74 | -0.094 | -0.004 | +0.137 | -0.015 | -0.037 | +0.157 |
| 2 | 42 | -0.057 | -0.045 | +0.014 | -0.017 | -0.098 | +0.113 |
| 3+ | 42 | -0.044 | -0.025 | +0.072 | +0.020 | -0.027 | +0.289 |
Table 10: Difference between SocraticRepair and GPT-4o by annotated bug count ( SocraticRepair minus GPT-4o).
| Evaluation dimension | Cohen’s |
| Technical correctness | 0.81 |
| Debugging relevance | 0.77 |
| Stepwise scaffolding | 0.70 |
| Guidance progression | 0.68 |
| Adaptive feedback | 0.66 |
| Debugging question quality | 0.72 |
Table 11: Agreement between the human annotator and an LLM judge. Linearly weighted Cohen’s is reported for the six tutor quality dimensions, while standard Cohen’s is used for the binary premature disclosure metric.