Instructor-designed AI improves learning in a randomized trial across 19 universities
Abstract
Universities are alarmed by recent evidence that artificial intelligence (AI) can undermine learning, with students prescribed generic chatbots or centrally designed AI tools performing worse. In a pre-registered randomized controlled trial across 7,200 students at 19 universities over three semesters, we tested whether AI designed by frontline instructors can improve learning. We encouraged instructors to design AI solutions they believed would best help their students. Drawing on their unique knowledge of their classes and students, the instructors created a wide array of innovations. We show that these instructor-designed AI innovations improved student learning, measured by AI-proof assessments (proctored, in-person examinations). We found no change in student engagement, measured by course activity records, or satisfaction, measured by course evaluations. Our findings demonstrate that AI can improve student learning when its adoption is frontline-driven, with instructors empowered to design and refine AI solutions for their own classrooms.
Figures & tables
| Theme | N (%) | Representative keywords | Sample quotes |
|---|---|---|---|
| Practice more | 696 (49.2%) | Practice questions; exam preparation; study; review; repetition | [1] …give me practice problems and conceptual questions close to the ones I could expect on the exam. [2] It allowed me to have endless problems and ensure that I wasn’t just memorizing the practice problems but actually knew the steps in a new problem |
| Better recall, comprehension, and problem solving | 466 (33.0%) | Recall; understand; clarify; explain; guided steps; problem solving | [3] Help me to figure out some formula that i don’t remember . [4] I liked how the AI tools helped break down complex topics into simpler explanations [5] I liked that the AI tools gave clear, step-by-step explanations for Excel functions and quantitative concepts, which made the material much easier to understand . They also helped me check my work quickly and reinforced what we learned in class … [6] I liked that the AI learning aids made it easier to understand complex coding and modeling concepts, especially when working with large datasets and machine learning models. |
| Tailored content and increased trust | 328 (23.2%) | Course content; instructor methods; personalized; adaptive; trust | [7] They were tailored to the class and built around the class content—it increased my trust . [8] .. referred to the correct vocabulary that was in the course… [9] It helped me understand in the same learning approach that the professor used. [10]… trained on class content , often more accurate than ChatGPT |
| Save time | 205 (14.5%) | Quick; instant; convenient; anytime; accessible; less waiting | [11]… the AI helped me understand the conceptual things well and made it where I didn’t attend office hours for every single minuscule question [12] I didn’t have to go around and spend too much time finding information. [13] I also didn’t have to wait a long time like for the professor to respond to an email to get a basic explanation. |
| Interactive and engaging | 90 (6.4%) | Conversation; follow-up; engaging; fun; interesting; novel experience | [14] I enjoyed the conversational testing nature. It felt like it was engaging a different part of my brain while learning. [15] I have never taken a class that utilized a specialized AI in the way Professor Nils did. It was a pretty cool experience |
| AI literacy | 40 (2.8%) | Prompting; AI models; building agents; responsible use; future skills | [16] Learning how to properly program an AI model and how different prompts impact outcomes. [17] … taught me how to use AI in smarter ways and adapt with it. [18] It made it relatable to what we are likely to see in the future especially with programming our own agent |
| Theme | N (%) | Representative keywords | Sample quotes |
|---|---|---|---|
| Use more | 272 (23.5%) | Earlier; frequent use; lectures; assignments; reminders | [1] … implementing it into the course a little more frequently instead of just midterm/final [2] Incorporate checkpoints after every unit where we can engage with the AI. |
| Increase practice difficulty and variety | 263 (22.7%) | Harder questions; variety; more practice; repetition; conceptual practice | [3] Wider variety of practice problems, with harder questions available. [4] Maybe try to make it closer to the actual exam or harder then it . [5] Help AI create more challenging ideas . |
| More tailored to the course | 232 (20.1%) | Course content; exams; lecture materials; instructor methods; individual needs | [6] …more aligned with class content and adding more examples that match our assignments and lectures. [7] …standardized explanations that follow the same approach taught in lectures… tailored to our specific assignments. |
| Clearer and deeper explanations | 230 (19.9%) | Clarity; depth; concise answers; examples; guided steps | [8] … more in-depth responses that explain what I may be doing wrong and provide step-by-step instructions. [9] Using it to aid in the excel assignments, not giving the answers but guiding us to understand why certain functions in excel were used |
| AI literacy | 129 (11.1%) | Instructions; demonstrations; prompting; appropriate use | [10] One possible improvement would be to provide more structured guidance or recommended use cases for AI tools within the course, such as examples of effective prompts or best practices for using AI responsibly in coding and data analysis tasks. [11] Just leverage it more and teach students how to prompt to take advantage of the model. |
| Increase accuracy | 122 (10.5%) | Wrong answers; hallucinations; calculations; reliability | [12] Give them some way to fact-check themselves , or make it so that they don’t give false information . [13] …AI could always be lying, hallucinating, or just a tiny bit off on a small piece of information. |
| Panel A. What were you trying to do with AI, and why did you think AI would help? | ||
|---|---|---|
| Theme | N (%) | Representative keywords |
| Accessible, on-demand learning support | 21 (84%) | Concept and syllabus questions; availability; accessible, low-pressure support |
| Professor [2] “I was trying to fill the gap of not being able to talk to each of them or even answer their questions about every single class. Sometimes they are embarrassed to ask things because they feel they are silly, so they ask the chatbot, because it knows all the content from the class.” | ||
| Professor [18] “We are trying to provide an additional resource for the students. Sometimes students may have difficulty finding a TA. Other times, they may be too shy to find a TA, or they face problems in the evening or over the weekend when a TA is not available. So this is quite a help for them.” | ||
| Professor [13] “I think maybe previously, if they needed to come to my office or write me emails to ask questions, and the question was not too significant, maybe they didn’t want to do it because of the hassle. But with these tools available to them, the cost of asking for feedback is lower. I’m thinking that could be one way this helps them.” | ||
| Scalable practice and personalized support | 19 (76%) | More practice problems; tailored explanations; personalized learning at scale |
| Panel B. What actually happened when you implemented it? How did students actually use or interact with the AI? Did they use it in the way you expected? Was there anything that worked differently than expected, surprised you, or caused you to change the intervention during the semester? | ||
|---|---|---|
| Theme | N (%) | Representative keywords |
| Useful AI support in practice | 19 (76%) | Useful explanations and task help; additional and varied practice; positive reception |
| Professor [1] “In the expected way, I knew what students wanted: they wanted more practice. Providing tools that offered more practice, I think, made them better prepared, or at the very least made them not complain to me about it. […] It succeeded on that front: it let students who wanted to prepare more do so, and those who didn’t just had to live with that decision.” | ||
| Professor [11] “Not all students engaged with the AI I trained. However, many of those who engaged with it found it useful. They especially used the apps to get numerous examples from different contexts. Providing examples for the students to practice with was very time-consuming before using AI. The examples were limited, and contexts were rather standard. AI made these diverse and made it possible to scale them up. There were no big surprises.” | ||
| Professor [15] “And because the way we do our notes and our exams, at least back then, were derivatives of what we had done in class, I think the AI was pretty good at handling that. So you gave it a new question that was very similar to something we had done in the notes or in the homework, and then the AI could adapt to that. It was exactly what the AI was meant to do: adapt things to a particular context with slight changes in numbers. So I think it worked really well for them to ask it to explain certain things.” | ||
| Uneven and context-dependent AI use | 13 (52%) | Uneven use; lower-than-expected uptake; optional versus required participation |
AI in Education
AI in Education studies how artificial intelligence can support teaching, learning, assessment, and educational decision-making.
See all 270 papers