cs.HCOct 19, 2026
SaveValiAssist: Scaling LLM-generated Programming Feedback Validation by Demonstration
Organizations: Virginia Tech, USA
Abstract
Generative AI can create programming feedback at scale, but validating its alignment with instructors' pedagogical goals and strategies remains challenging. Instructors may struggle to fully specify validation criteria upfront, as these often emerge through inspecting concrete examples and revising misaligned feedback. We present ValiAssist, a validation-by-demonstration system that infers reusable validation logic from instructors' revisions of AI-generated feedback and allows instructors to inspect and refine that logic as it is applied across other student submissions. In a comparative study with 16 instructors, ValiAssist helped instructors create and refine validation criteria, producing feedback that better aligned with their pedagogical intent (91% vs. 57%) and more than doubling the accuracy of correcting problematic criteria (77% vs. 32%). Our findings suggest that validation by demonstration can help instructors turn judgments they know how to make on individual cases into explicit, reusable logic, while progressively clarifying where those judgments do and do not generalize.
Figures & tables
Figure 1. The validation-by-demonstration workflow of ValiAssist. The instructor starts by revising LLM-generated feedback in a sample code-feedback pair, then ValiAssist suggests validation criteria by inferring the instructor’s intention according to their interactions (e.g., comments, AI chat, and direct edit) and context information (e.g., original and revised feedback, code submission, and error message). After verifying and tuning the criteria, the instructor can apply the criteria to all other code-feedback pairs to scale the validation. ValiAssist first performs condition matching on the code-feedback pairs and takes actions based on the matched criteria.
| Act | None | Opaque | Readable | Structured |
|---|---|---|---|---|
| State | n/a | n/a | CS50 Duck ( Liu et al., 2024a ) | CodeHelp ( Liffiton et al., 2023 ) Co-Pilot ( Hedderich et al., 2024 ) TeachTune ( Jin et al., 2025 ) |
| Rate | SPHERE ( Tang et al., 2025 ) Edison ( Miroyan et al., 2025 ) | CS50 human feedback ( Liu et al., 2025 ) DPO with teachers ( Woodrow et al., 2025 ) | GradeHITL ( Chu et al., 2025 ) | |
| Demonstrate | Teacher revisions ( Borchers et al., 2026 ) | CoGrader ( Chen et al., 2025 ) | Avalon ( Armfield et al., 2025 ) Formative baseline (§ 3 ) | ValiAssist |
Table 1. Design space for generative AI pedagogical guidance validation. We treat validation as any instructor act that governs whether generated guidance reaches learners. Rows name that act: state a criterion before seeing output, rate outputs presented for review, or demonstrate by revising an output in place. Columns name what the act leaves behind for outputs the instructor never sees: none (applies only to the item touched), opaque (reused by the system but not inspectable), readable (natural language the instructor can rewrite), and structured (an executable specification the instructor can inspect and repair). n/a marks cells that cannot occur, since a stated criterion is itself the persisted, readable artifact. marks cells with no published occupant known to us.
| Workflow | Definition | Example sequence | % (n) | |
|---|---|---|---|---|
| Feedback-only | The instructor settles the generated message without opening the submission. | read message send | 30.34 (54) | |
| Code-first | The instructor reads the submission first and forms their own judgment of what the student needs, then turns to the message and decides against that judgment without going back. | read submission read message send | 11.80 (21) | |
| Interleaved | The instructor enters from either the submission or the message, then keeps moving between the two, revising the feedback and checking it against the code again as they go. | read message check submission revise message check submission send | 57.87 (103) |
Table 2. Validation workflows from the preliminary observations. The last column gives each workflow’s share of the 178 coded episodes, with the episode count in parentheses. Episodes are coded one at a time, so an instructor can appear in more than one workflow, and none of the ten stayed with a single workflow.
Figure 2. Probe system is composed of three main panels: code-feedback, validation, and demonstration. In the code-feedback panel, the instructor can review the generated feedback (A), the system’s validation results (B), and validate and edit feedback. In the validation panel, the instructor can create, manage, and apply validation criteria (C). In the demonstration panel, the instructor can review their revision record (D), and adjust the criteria generated from their demonstration (E).
| Index | Code | Definition | Example criteria | % (n) |
|---|---|---|---|---|
| 1 | conciseness | The criterion cuts material that does not teach: greetings, filler, over-long explanation, or encouragement the student does not need. | “Feedback must exclude any final redundant encouraging message.” “Ensure that the feedback is concise and actionable.” | 33.73 (28) |
| 2 | no_direct_solutions | The criterion keeps complete code solutions and direct answers out of the feedback, so the student has to work the problem out. | “Feedback must not include any source code snippets.” “Confirm that the feedback does not provide direct answers.” | 24.10 (20) |
| 3 | content_requirement | The criterion asks the feedback to cover something specific, such as the range function, list comprehensions, or testing with different inputs. | “Inclusivity of Range: explain how range works.” “Feedback includes an explanation of the range function if it is missing.” | 9.64 (8) |
| 4 | code_style | The criterion asks the feedback to comment on how the code is written, most often its naming. | “The feedback must emphasize the use of descriptive variable names.” “The code lacks clarity in parameter naming. If the parameter naming is unclear, it suggests replacing the function…” | 6.02 (5) |
| 5 | specify_program | The criterion states what the student’s program itself has to do or look like, rather than what the feedback should say about it. | “Ensure the input to the make_list function is a list.” “Check whether the function is returning the correct list or not.” | 6.02 (5) |
| 6 | conceptual_guidance | The criterion steers the feedback toward the idea behind the mistake instead of the particular fix. | “Feedback should encourage problem-solving or exploration without providing exact code solutions.” “Overly specific variable name suggestions to maintain generality in feedback.” | 4.82 (4) |
Table 3. Pedagogical intent codebook. Ten codes derived from the 83 validation criteria participants created in the formative study, ordered by frequency. Each row gives the code, its definition, and two example criteria from the corpus.
Figure 3. ValiAssist UI. ValiAssist’s user interface contains three main components: (A) a panel for initial LLM-powered feedback generation, (B) a panel for reviewing code-feedback pairs, and (C) a panel for validation criteria review and application. The instructor can revise the feedback in three ways: (D) comment on code/feedback, (F) AI chat, and (E) direct edit.
Figure 5. ValiAssist’s validation criterion (B). If the instructor attempts to accept a pending criterion (A) with potential issues, a dialog (C) would pop up. The instructor can switch to the detailed view (D) to relate the criterion to the original demonstration, where the semantic labels are linked to the corresponding matched text spans (E). In the detailed view, the instructor can also directly select a text span to add as an example (F). By clicking the view icon, the instructor can overview the matched text spans of semantic labels by clusters and add examples (G). They can also refine the criterion in edit view (H).
Figure 6. ValiAssist’s criteria execution UI. The instructor can inspect the criteria execution results (A) for each code-feedback pair. The matched text spans are highlighted (B) and instructor can hover on a span to view the corresponding semantic label before confirming the revision.
Figure 7. Participants’ Agreement Level (higher is better). 1 for Strongly Disagree; 7 for Strongly Agree, * denotes , and ** denotes .
Figure 8. NASA-TLX Response. Lower is better.
Figure 9. P12’s interaction timelines during Criteria Creation (sub-task 1) and LLM-assisted Validation (sub-task 2). The top band of each panel shows the inferred stage of validation; the bottom row shows every logged action, with dashed lines marking validation runs and the hatched divider marking the untimed tutorial break (not to scale). Under ValiAssist, P12 created criteria from demonstrations and short description edits, then scaled validation through five run–review cycles, accepting criteria-proposed revisions in bursts and editing criteria between runs. Under Baseline, P12 authored criteria in a single batch, spent sub-task 1 refining their wording, and applied them in fewer, later runs. Runs without adjacent markers (e.g., at the opening of sub-task 2) are re-executions performed while P12 was still reviewing the propagated results: only completed actions were logged, so reading and scrolling leave no marks.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11. Revision and test case annotation UI
Figure 12. Baseline in the user study. User creates validation criteria in pure natural language with an AI assistant. They can write their criteria in prose, and the system will transform it in formal criteria. The user can refine the criteria through a multi-turn clarification.
More in this topic
Generative AI Evaluation
Generative AI evaluation measures the quality, reliability, diversity, and real-world usefulness of generated outputs and AI-assisted workflows.
See all 311 papers