SocraticRepair: Planning-Guided Socratic Tutoring for Program Debugging and Misconception Repair
Organizations: University of Warwick · Warwick Manufacturing Group
Abstract
Large language models are increasingly used in computer science education. However, the teaching abilities of even the most powerful LLMs are limited: they tend to reveal the solution prematurely and may lose focus on the learner's current debugging objective. To address this, we introduce SocraticRepair, a planning-guided Socratic tutoring framework for program debugging and miscon- ception repair that adapts prior planning-guided tutoring work (Li et al., 2026b). The framework is instantiated in the Python programming domain by fine-tuning Qwen2.5-Coder-7B-Instruct on synthetic multi-turn tutoring dialogues constructed from four preprocessed programming datasets. The system is evaluated using a predefined set of independent metrics and benchmarked against a general-purpose LLM. The experiments show that SocraticRepair achieves higher scores on stepwise scaffolding and question quality while also substantially reducing premature solution disclosure. The results also identify reliable pedagogical plan generation as an important remaining challenge.
Figures & tables
| Learner state | Tutoring action | Active objective |
| START | Ask an initial guiding question | Unchanged |
| INCORRECT | Re-ask or provide a targeted hint | Not demonstrated |
| CONFUSION | Simplify the current objective or provide a hint | Not demonstrated |
| QUESTION | Answer the clarification and steer the student back to the objective | Not demonstrated |
| IRRELEVANT | Refocus the student on the current objective | Not demonstrated |
| CORRECT / COMPREHENSION | Advance when sufficient evidence demonstrates the objective, otherwise continue eliciting evidence | May be demonstrated |
| Dataset | Case ID | Problem | Buggy code | Tests | Failure | Bugs description | Correct code | Source |
| PYMETA | D | G | D | G | X | G | G | D |
| MULTI_DEBUG | D | D | D | G | X | D | D | D |
| QUIXBUGS | D | D | D | D | X | G | D | D |
| INTRO_PROG | D | C | C | D | X | G | C | D |
| Dataset | Train | Validation | Test | Total |
| PYMETA | 242 | 29 | 29 | 300 |
| MULTI_DEBUG | 120 | 15 | 15 | 150 |
| QUIXBUGS | 32 | 4 | 4 | 40 |
| INTRO_PROG | 26 | 5 | 5 | 36 |
| Total | 420 | 53 | 53 | 526 |
| Learner state | RECEPTIVE | UNCERTAIN | PERSISTENT |
| CORRECT | 0.40 | 0.30 | 0.25 |
| INCORRECT | 0.20 | 0.15 | 0.35 |
| QUESTION | 0.10 | 0.25 | 0.15 |
| COMPREHENSION | 0.10 | 0.10 | 0.05 |
| CONFUSION | 0.10 | 0.15 | 0.15 |
| IRRELEVANT | 0.10 | 0.05 | 0.05 |
| Metric | Evaluated property | Literature basis |
| Technical correctness | Technical accuracy and consistency with the program, execution evidence and trusted repair. | ScaffoldEval: Answer Accuracy |
| Debugging relevance | Alignment with the student’s current objective. | ScaffoldEval: Topic Adherence; TreeInstruct: Relevance |
| Stepwise scaffolding | Step by step guidance that leaves meaningful reasoning work to the student. | ScaffoldEval: Stepwise Scaffolding; TreeInstruct: Indirectness |
| Guidance progression | Productive movement towards unresolved objectives without unnecessary repetition. | ScaffoldEval: Guidance Quality; TreeInstruct: Logical Flow |
| Adaptive feedback | Responsiveness to the student’s latest utterance. | ScaffoldEval: Adaptive Feedback |
| Debugging question quality | Use of questions that elicit useful explanation | ScaffoldEval: Question Quality |
| SocraticRepair | GPT-4o | |
| Evaluation dialogues | 159 | 159 |
| Successful dialogues | 148 (93.1%) | 152 (95.6%) |
| Mean rounds | 8.04 | 4.99 |
| Premature-disclosure turns | 24 / 1,276 (1.88%) | 59 / 794 (7.43%) |
| Dialogues with disclosure | 20 / 159 (12.58%) | 53 / 159 (33.33%) |
| Plan source | Attempts | Technically valid |
| SocraticRepair | 53 | 11 |
| GPT-4o + verifier | 53 | 53 |
| Metric | SocraticRepair | GPT-4o | Holm-adjusted | |
| Technical correctness | 1.789 | 1.860 | -0.071 | .002 |
| Debugging relevance | 1.929 | 1.949 | -0.021 | .118 |
| Stepwise scaffolding | 1.876 | 1.789 | +0.087 | .006 |
| Guidance progression | 1.655 | 1.661 | -0.006 | .385 |
| Adaptive feedback | 1.792 | 1.843 | -0.051 | .036 |
| Debugging question quality | 1.626 | 1.446 | +0.180 |
| Learner state | Turns (SR/GPT) | SR Adapt. | GPT Adapt. | SR state acc. |
| CORRECT | 430 / 213 | 1.840 | 1.939 | 95.3% |
| INCORRECT | 236 / 150 | 1.640 | 1.660 | 91.9% |
| QUESTION | 156 / 103 | 1.833 | 1.864 | 100.0% |
| COMPREHENSION | 117 / 56 | 1.897 | 1.982 | 58.1% |
| CONFUSION | 109 / 67 | 1.560 | 1.522 | 98.2% |
| IRRELEVANT | 69 / 47 | 1.812 | 1.809 | 100.0% |
| Bugs | Matched | Tech. | Relevance | Scaff. | Progression | Adapt. | Q. qual. |
| 1 | 74 | -0.094 | -0.004 | +0.137 | -0.015 | -0.037 | +0.157 |
| 2 | 42 | -0.057 | -0.045 | +0.014 | -0.017 | -0.098 | +0.113 |
| 3+ | 42 | -0.044 | -0.025 | +0.072 | +0.020 | -0.027 | +0.289 |
| Evaluation dimension | Cohen’s |
| Technical correctness | 0.81 |
| Debugging relevance | 0.77 |
| Stepwise scaffolding | 0.70 |
| Guidance progression | 0.68 |
| Adaptive feedback | 0.66 |
| Debugging question quality | 0.72 |