Purpose: Higher education institutions face increasing pressure to audit course designs for generative AI (GenAI) integration. This paper presents an end-to-end method for using large language models (LLMs) to scan course information sheets at scale, identify where assessments may be vulnerable to student use of GenAI tools, validate system performance through iterative refinement, and operationalise results through direct stakeholder communication and effort. Method: We developed a four-phase pipeline: (0) manual pilot sampling, (1) iterative prompt engineering with multi-model comparison, (2) full production scan of 4,684 Bachelor and Master course information sheets (Academic Year 2024-2025) from the Vrije Universiteit Brussel (VUB) with automated report generation and email distribution to teaching teams (91.4% address-matched) using a three-tier risk taxonomy (Clear risk, Potential risk, Low risk), and (3) longitudinal re-scan of 4,675 sheets after the next catalogue release. Results: Five iterations of prompt refinement achieved 87% agreement with expert labels. GPT-4o was selected for production based on superior handling of ambiguous cases involving internships and practical components. The Year 1 scan classified 60.3% of courses as Clear risk, 15.2% as Potential risk, and 24.5% as Low risk. Year 2 comparison revealed substantial shifts in risk distributions, with improvements most pronounced in practice-oriented programmes. Implications: The method enables institutions to rapidly transform heterogeneous catalogue data into structured and actionable intelligence. The approach is transferable to other audit domains (sustainability, accessibility, pedagogical alignment) and provides a template for responsible LLM deployment in higher education governance.
Figures & tables
Iteration
Agreement
Primary Error Pattern
Key Prompt Change
1
63% (19/30)
Over-classification of internships
Clarified supervision criteria
2
70% (21/30)
Misweighting practical components
Added grade contribution guidance
3
77% (23/30)
Language inconsistency
Strengthened language requirements
4
83% (25/30)
Excessive caution
Refined Low risk criteria
5
87% (26/30)
Ambiguous boundary cases
(Production stable)
Table 1: Pilot phase iteration outcomes
Category
Count
Percentage
Clear risk
2,824
60.3%
Potential risk
713
15.2%
Low risk
1,147
24.5%
Total
4,684
100%
Table 2: Year 1 risk classification distribution (AY 2024–2025, all BA/MA courses)
Category
Count
Percentage
Clear risk
2,253
48.2%
Potential risk
1,024
21.9%
Low risk
1,398
29.9%
Total
4,675
100%
Table 3: Year 2 risk classification distribution (AY 2025–2026, all BA/MA courses)
Year 2
Year 1
Clear
Potential
Low
Total
Clear risk
1,847
412
156
2,415
Potential risk
189
298
156
643
Low risk
98
156
580
834
Total
2,134
866
892
3,892
Table 4: Category transition matrix (Year 1 to Year 2)
Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.