Generative AI and the Crisis of Higher Education Assessments: Examining 'AI-Proofing' Strategies for College-Level Assessments
Organizations: Pennsylvania State University, USA · Sungkyunkwan University, Republic of Korea
Abstract
Higher education is confronting an academic-integrity challenge unlike any in its history. Students can now outsource homework, take-home assignments, and online examinations to large language models (LLMs) that complete them in seconds. Existing research on evaluating LLM performance on academic assessments has focused largely on standardized examinations, or on questions from college-level courses drawn from a single discipline. Far less is known about how LLMs perform on college-level assessment questions spanning multiple disciplines, that instructors have purposely designed to be AI-resistant, and about which features of such questions, if any, meaningfully constrain LLM performance. We address this gap by collecting 75 questions from authentic college coursework assessments, spanning 17 disciplines, which were contributed by 37 faculty members from ten U.S. universities and one U.K. university. Of these, 61 questions were intentionally designed by faculty members to be 'LLM-proof', while the remaining 14 questions were sourced from online courses. We then (i) evaluate three state-of-the-art LLMs' ability to correctly answer these questions with and without supplementary course materials; (ii) identify five strategies, through thematic analysis, that faculty used to make their questions harder for LLMs to solve; and (iii) test whether these strategies impact LLMs' ability to answer the questions accurately. Our experimental results show that the best-performing model scored approximately 83% without supplementary course materials and 85% with them, and we observed noteworthy differences across models, answer formats, and instructor tweaking strategies. We discuss implications for assessment design, academic-integrity policy, and the broader ethics of deploying LLMs in educational settings.
Figures & tables
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Characteristic | Count (n) |
| Role | |
| Assistant Professor | 11 |
| Associate Professor | 9 |
| Other Faculty | 5 |
| Professor | 4 |
| Teaching Professor | 3 |
| Characteristic | Count (n) |
| Role | |
| Undergraduate student | 17 |
| Graduate student | 13 |
| Other | 1 |
| Department | |
| Engineering | 7 |
| Question | Online Source | Supplementary Course Material |
| MediaStudies_01 | WJEC UK Sample Assessment Materials 1 1 1 https://www.wjec.co.uk/media/bzlhh4lk/wjec-gcse-media-studies-sams-from-2017-e.pdf | The Media Student’s Book (5th ed.), ( Branston and Stafford, 2010 , Chapter 3.9) |
| MediaStudies_02 | WJEC UK Sample Assessment Materials | ( Branston and Stafford, 2010 , Chapter 2.12) |
| MediaStudies_03 | WJEC UK Sample Assessment Materials | ( Branston and Stafford, 2010 , Chapter 1.1) |
| MediaStudies_04 | WJEC UK Sample Assessment Materials | ( Branston and Stafford, 2010 , Chapter 11.1) |
| MediaStudies_05 to 10 | WJEC UK Sample Assessment Materials | ( Branston and Stafford, 2010 , Chapter 14) |
| Politics_01 | AP Central College Board Sample 1 2 2 2 https://apcentral.collegeboard.org/media/pdf/ap24-apc-us-go-po-q2-set-2.pdf | We the people: an introduction to American politics (11th ed.), ( Ginsberg et al., 2021 , Chapter 10) |
Generative AI in Education
Generative AI in education uses language and multimodal models to support teaching, learning, feedback, assessment, and educational design.
See all 301 papers