Outcome Over Learning? Evidence from Student Use of AI Code Assistants for Unit Test Writing
Organizations: Ontario Tech University Oshawa, ON, Canada
Abstract
Artificial intelligence (AI) code assistants are rapidly being adopted in programming education, yet little is known about how students naturally use these tools when given unrestricted access. In a user study investigating AI-assisted unit test writing, we observed that a majority of students adopted a one-shot interaction strategy, asking the AI to generate a complete unit test directly from the initial requirements. While this approach is capable of efficiently producing a solution, it shifts students' focus from learning how to write effective unit tests to simply completing the task. Moreover, because students rarely engaged in critically evaluating, refining, or iterating on AI-generated output, they may be missing opportunities to develop the critical thinking skills that are essential for effective and responsible use of AI. In an effort to shift student focus from outcome to learning we also discuss how training, accountability and improved learning task design can better support a more learning-centric responsible use of AI.
Figures & tables
| Answer | Count | Percentage |
|---|---|---|
| 1 - Not confident at all | 3 | 14.3% |
| 2 | 9 | 42.8% |
| 3 - Moderately confident | 8 | 38.1% |
| 4 | 1 | 4.8% |
| 5 - Extremely confident | 0 | 0.0% |
| Answer | Count | Percentage |
|---|---|---|
| Yes | 17 | 81.0% |
| No | 4 | 19.0% |
| Model | Count | Percentage |
|---|---|---|
| GPT | 20 | 95.2% |
| Gemini | 14 | 66.7% |
| Claude | 10 | 47.6% |
| Copilot | 9 | 42.9% |
| Llama | 2 | 9.5% |
| Cursor | 1 | 4.8% |
| Answer | Count | Percentage |
|---|---|---|
| 1 - Not confident at all | 0 | 0.0% |
| 2 | 2 | 9.5% |
| 3 - Moderately confident | 8 | 38.1% |
| 4 | 7 | 33.3% |
| 5 - Extremely confident | 4 | 19.1% |
| Use Case | # Students | % of Students |
|---|---|---|
| Debugging & Problem Solving | 10 | 47.6% |
| Learning | 6 | 28.6% |
| Starter Code Generation | 5 | 23.8% |
| Improvement & Optimization | 4 | 19.1% |
| Code Review | 3 | 14.3% |
| Code Block Generation | 3 | 14.3% |
Generative AI in Education
Generative AI in education uses language and multimodal models to support teaching, learning, feedback, assessment, and educational design.
See all 301 papers