Universities are alarmed by recent evidence that artificial intelligence (AI) can undermine learning, with students prescribed generic chatbots or centrally designed AI tools performing worse. In a pre-registered randomized controlled trial across 7,200 students at 19 universities over three semesters, we tested whether AI designed by frontline instructors can improve learning. We encouraged instructors to design AI solutions they believed would best help their students. Drawing on their unique knowledge of their classes and students, the instructors created a wide array of innovations. We show that these instructor-designed AI innovations improved student learning, measured by AI-proof assessments (proctored, in-person examinations). We found no change in student engagement, measured by course activity records, or satisfaction, measured by course evaluations. Our findings demonstrate that AI can improve student learning when its adoption is frontline-driven, with instructors empowered to design and refine AI solutions for their own classrooms.
Figures & tables
Figure 1: RCT participation. A class is a course–instructor pair; a class-term is one term of a class. In a , dots mark the semesters in which a university took part (S25, spring 2025; F25, fall 2025; S26, spring 2026) and bars show enrollment summed over class-terms. In b , totals count by university, instructor, course and class across semesters; enrollment sums semesters. Treatment and control counts refer to new courses or classes. † Eight courses were assigned to control in Spring 2025; two classes taught by the same instructor withdrew before receiving the assignment. ‡ Nine courses were assigned to control in Fall 2015; one class withdrew before receiving its assignment. § Includes five new course–instructor pairs: the course continued from an earlier term, but with instructors who had not taken part in that term’s experiment. Therefore, total classes (55) exceed new classes (50).
Figure 2: Instructor-designed AI innovations by pedagogical function. Legend on the next page.
Figure 3: Effects of instructor-designed AI innovations on learning, engagement, and satisfaction. a , Student-level analysis; b , class-level analysis. Points are OLS estimates of the contemporaneous AI-innovation effect ( β0 ) from the primary specification (Methods), measured in within-class standard deviation units, with CR2 standard errors clustered by class-term in parentheses. Thick bars show 90% and thin lines 95% confidence intervals. N is the number of student observations in a and of class-terms, including historical terms, in b .
Figure 4: Student Survey Responses - Structured Questions Results are based on an anonymous end-of-course survey: 2,019 out of 4,898 students enrolled in treatment classes responded.
Figure 5: Student Survey Responses - Open-Ended Questions. Legend on the next page.
Figure 6: Instructor interviews: themes. Share of the 25 instructor interviews featuring each theme, with representative keywords and illustrative quotations. a , What instructors were trying to do with AI, and why they thought AI would help. b , What happened when they implemented it. Each interview counts once per theme; themes may overlap.
Figure ED1: Inclusion and exclusion criteria for the analysis sample
Figure ED2: Estimates across alternative empirical specifications. Points show effect estimates for student learning, engagement, and satisfaction, expressed in within-class standard deviation units; thick bars show 90% and thin lines 95% confidence intervals. Panel a reports student-level estimates; panel b reports class-level estimates. Both panels report contemporaneous effects in the first term of AI implementation and lagged effects associated with having implemented AI previously. Only three class-terms in the eligible analysis sample contributed observations from second or third AI implementations, limiting the precision of lagged estimates. For each outcome, twelve specifications combine three estimators (ordinary least squares [OLS], the general instrumental-variable [IV] estimator, and the simple IV estimator), two aggregation levels (student and class), and two confidence-interval methods (conventional and CR2). OLS analyses exclude the two treatment-assigned class-terms that did not implement their innovations; both IV analyses include them. The IV specifications are described in Methods, Empirical Specifications. Student-level aggregation was pre-registered, but restricting learning and engagement data to consenting students may introduce consent bias (average class-level consent rate 79.4%, with a median of 84.6%). Class-level learning and engagement summaries include all enrolled students, including those who did not consent to individual data use; satisfaction summaries reflect anonymous course-evaluation respondents that do not require consent. Both interval methods use Student- t distributions and standard errors clustered by class-term. Conventional intervals use G−1 degrees of freedom, where G is the number of class-term clusters; CR2 intervals use bias-reduced cluster-robust variance estimates and Satterthwaite degrees of freedom.
Theme
N (%)
Representative keywords
Sample quotes
Practice more
696 (49.2%)
Practice questions; exam preparation; study; review; repetition
[1] …give me practice problems and conceptual questions close to the ones I could expect on the exam. [2] It allowed me to have endless problems and ensure that I wasn’t just memorizing the practice problems but actually knew the steps in a new problem
Better recall, comprehension, and problem solving
466 (33.0%)
Recall; understand; clarify; explain; guided steps; problem solving
[3] Help me to figure out some formula that i don’t remember . [4] I liked how the AI tools helped break down complex topics into simpler explanations [5] I liked that the AI tools gave clear, step-by-step explanations for Excel functions and quantitative concepts, which made the material much easier to understand . They also helped me check my work quickly and reinforced what we learned in class … [6] I liked that the AI learning aids made it easier to understand complex coding and modeling concepts, especially when working with large datasets and machine learning models.
[7] They were tailored to the class and built around the class content—it increased my trust . [8] .. referred to the correct vocabulary that was in the course… [9] It helped me understand in the same learning approach that the professor used. [10]… trained on class content , often more accurate than ChatGPT
Save time
205 (14.5%)
Quick; instant; convenient; anytime; accessible; less waiting
[11]… the AI helped me understand the conceptual things well and made it where I didn’t attend office hours for every single minuscule question [12] I didn’t have to go around and spend too much time finding information. [13] I also didn’t have to wait a long time like for the professor to respond to an email to get a basic explanation.
[14] I enjoyed the conversational testing nature. It felt like it was engaging a different part of my brain while learning. [15] I have never taken a class that utilized a specialized AI in the way Professor Nils did. It was a pretty cool experience
AI literacy
40 (2.8%)
Prompting; AI models; building agents; responsible use; future skills
[16] Learning how to properly program an AI model and how different prompts impact outcomes. [17] … taught me how to use AI in smarter ways and adapt with it. [18] It made it relatable to what we are likely to see in the future especially with programming our own agent
Table ED1: Students’ positive perceptions of AI learning components
[1] … implementing it into the course a little more frequently instead of just midterm/final [2] Incorporate checkpoints after every unit where we can engage with the AI.
Increase practice difficulty and variety
263 (22.7%)
Harder questions; variety; more practice; repetition; conceptual practice
[3] Wider variety of practice problems, with harder questions available. [4] Maybe try to make it closer to the actual exam or harder then it . [5] Help AI create more challenging ideas .
[6] …more aligned with class content and adding more examples that match our assignments and lectures. [7] …standardized explanations that follow the same approach taught in lectures… tailored to our specific assignments.
[8] … more in-depth responses that explain what I may be doing wrong and provide step-by-step instructions. [9] Using it to aid in the excel assignments, not giving the answers but guiding us to understand why certain functions in excel were used
AI literacy
129 (11.1%)
Instructions; demonstrations; prompting; appropriate use
[10] One possible improvement would be to provide more structured guidance or recommended use cases for AI tools within the course, such as examples of effective prompts or best practices for using AI responsibly in coding and data analysis tasks. [11] Just leverage it more and teach students how to prompt to take advantage of the model.
[12] Give them some way to fact-check themselves , or make it so that they don’t give false information . [13] …AI could always be lying, hallucinating, or just a tiny bit off on a small piece of information.
Table ED2: Students’ criticisms and suggestions for improving AI learning aids
Panel A. What were you trying to do with AI, and why did you think AI would help?
Theme
N (%)
Representative keywords
Accessible, on-demand learning support
21 (84%)
Concept and syllabus questions; availability; accessible, low-pressure support
Professor [2] “I was trying to fill the gap of not being able to talk to each of them or even answer their questions about every single class. Sometimes they are embarrassed to ask things because they feel they are silly, so they ask the chatbot, because it knows all the content from the class.”
Professor [18] “We are trying to provide an additional resource for the students. Sometimes students may have difficulty finding a TA. Other times, they may be too shy to find a TA, or they face problems in the evening or over the weekend when a TA is not available. So this is quite a help for them.”
Professor [13] “I think maybe previously, if they needed to come to my office or write me emails to ask questions, and the question was not too significant, maybe they didn’t want to do it because of the hassle. But with these tools available to them, the cost of asking for feedback is lower. I’m thinking that could be one way this helps them.”
Scalable practice and personalized support
19 (76%)
More practice problems; tailored explanations; personalized learning at scale
Table ED3: Faculty intentions and reasons for using AI
Panel B. What actually happened when you implemented it? How did students actually use or interact with the AI? Did they use it in the way you expected? Was there anything that worked differently than expected, surprised you, or caused you to change the intervention during the semester?
Theme
N (%)
Representative keywords
Useful AI support in practice
19 (76%)
Useful explanations and task help; additional and varied practice; positive reception
Professor [1] “In the expected way, I knew what students wanted: they wanted more practice. Providing tools that offered more practice, I think, made them better prepared, or at the very least made them not complain to me about it. […] It succeeded on that front: it let students who wanted to prepare more do so, and those who didn’t just had to live with that decision.”
Professor [11] “Not all students engaged with the AI I trained. However, many of those who engaged with it found it useful. They especially used the apps to get numerous examples from different contexts. Providing examples for the students to practice with was very time-consuming before using AI. The examples were limited, and contexts were rather standard. AI made these diverse and made it possible to scale them up. There were no big surprises.”
Professor [15] “And because the way we do our notes and our exams, at least back then, were derivatives of what we had done in class, I think the AI was pretty good at handling that. So you gave it a new question that was very similar to something we had done in the notes or in the homework, and then the AI could adapt to that. It was exactly what the AI was meant to do: adapt things to a particular context with slight changes in numbers. So I think it worked really well for them to ask it to explain certain things.”
Uneven and context-dependent AI use
13 (52%)
Uneven use; lower-than-expected uptake; optional versus required participation
Table ED4: Faculty experiences during AI implementation
Student use of Generative AI (GenAI) products in completing their classwork, with or without their professors' knowledge and/or approval, has resulted in substantial shifts in higher education. While GenAI use is widespread, its impact on student study methods, faculty course development, grade reporting, and overall learning is not well documented. This is a mixed-methods, multi-course study using retrospective quantitative analysis, instructor surveys, and anonymous student surveys at a university in the New England region of the United States. This research seeks to identify and document patterns in student and faculty perceptions of, and experiences in, the use of LLMs as a learning tool inside and outside of the university classroom. Alongside quantitative and thematic analysis of both faculty and student survey responses, historical grade data as reported to the university registrar is used to triangulate the phenomenon of learning achievement in pre- and post-LLM eras. It is hoped that this research can serve as a pilot study for a broader set of institutions. Results from this study can inform GenAI policy for professors, universities, and other educational institutions that are trying to maximize student learning in the age of AI.
AI learning tools are rapidly entering classrooms, but evidence about whether they help students learn is mixed and rests mostly on test scores. Comparatively less research addresses whether the use of AI changes students' live learning behaviors in class. Here, we report the results of a preregistered field experiment with 759 MBA students enrolled in ten sections of a course, in which each student was randomly assigned two of ten class sessions to prepare for with a purpose-built voice-based AI discussion partner. After two uses of the AI discussion partner, students made about 31% more voluntary contributions in each later class session. Students who used the AI discussion partner more also reported greater comfort speaking up and greater perceived learning, but not greater focus or motivation. These findings suggest that repeated practice with a voice-based AI partner can meaningfully increase students' engagement in class discussion, enhancing a critical intermediate learning outcome.
Dan J. Wang, Neelam Modi Jain, Vanessa Burbano +5
Management Division, Columbia Business School, Columbia University, New York, NY 10027, USA.
Artificial intelligence (AI) code assistants are rapidly being adopted in programming education, yet little is known about how students naturally use these tools when given unrestricted access. In a user study investigating AI-assisted unit test writing, we observed that a majority of students adopted a one-shot interaction strategy, asking the AI to generate a complete unit test directly from the initial requirements. While this approach is capable of efficiently producing a solution, it shifts students' focus from learning how to write effective unit tests to simply completing the task. Moreover, because students rarely engaged in critically evaluating, refining, or iterating on AI-generated output, they may be missing opportunities to develop the critical thinking skills that are essential for effective and responsible use of AI. In an effort to shift student focus from outcome to learning we also discuss how training, accountability and improved learning task design can better support a more learning-centric responsible use of AI.
Daniel Hinbest, Jeremy S. Bradbury, Michael A. Miljanovic