cs.AISep 18, 2026

Reducing Barriers to Academic Support: Evaluating a Course-Specific RAG System for Addressing Help-Seeking Disparities in Higher Education

Authors: Andy Gray, Jake Hobbs

Organizations: School of Design, Bath Spa University, Bath, UK

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Access to academic support is a key determinant of student success, yet students experience it unequally: some readily seek help from lecturers or tutors, while others hesitate due to anxiety, fear of judgement, uncertainty about expectations, or low confidence in their understanding. This may be especially evident in computing education, where programming tasks are cumulative and cognitively demanding. Although students increasingly turn to general-purpose generative AI tools, these can produce responses that are inaccurate, insufficiently contextualised, or misaligned with module expectations. This study presents and evaluates Beacon, a course-specific Retrieval-Augmented Generation (RAG) system providing private, immediate, module-aligned academic support. Grounding responses in approved teaching materials, Beacon was designed to lower barriers to help-seeking while encouraging independent learning. Using a design-based research approach, Beacon was developed iteratively and evaluated via mixed methods, combining questionnaires and semi-structured interviews with students and staff at a Higher Education institution. Students described Beacon's responses as closely aligned with module content and more trustworthy than unrestricted generative AI tools, valuing its use of pseudocode and scaffolded explanations over direct solutions. Although participants remained cautious about trusting AI-generated responses without verification, they viewed the system as a valuable first point of support before consulting lecturers or official resources. The findings suggest that carefully designed course-specific AI systems may reduce barriers to academic support by occupying an intermediary space between independent study and formal support. Rather than replacing educators, educational AI may be most valuable when it broadens access to guidance while preserving the pedagogical role of lecturers.

Figures & tables

Figure 1: A visual demonstration of how RAG can help inform the LLMs with relevant material stored within a vector database ( Greer, 2024 ) . The user will ask a question, which, in turn, the smart retriever, a vector similarity searcher, will query the Vector database to retrieve the relevant information to present to the LLM as a generator. The LLM then produces an output representing the user’s question and the additional information retrieved from the vector database. In our context, lesson content and assessment details are stored in the vector database to aid the students.
Figure 2: Retrieval pipeline used to generate grounded responses. Initial questions are embedded directly for semantic retrieval, whereas follow-up questions are first rewritten into self-contained queries using the preceding conversation before retrieval is performed. The final prompt supplied to the language model combines the system prompt, user query, retrieved teaching materials, and conversation history.
Figure 3: The Version 1 web application interface. The pilot implementation displays the generated response together with the supporting source material used to produce the answer.
Figure 4: The Version 2 web application interface. The redesigned chat interface presents generated responses alongside the supporting source material used to produce each answer.
More in this topic

AI in Education

AI in Education studies how artificial intelligence can support teaching, learning, assessment, and educational decision-making.

See all 262 papers

Explore similar work

CardsList
Mar 9, 2026

AI Meets Mathematics Education: A Case Study on Supporting an Instructor in a Large Mathematics Class with Context-Aware AI

Large-enrollment university courses face persistent challenges in providing timely and scalable instructional support. While generative AI holds promise, its effective use depends on reliability and pedagogical alignment. We present a human-centered case study of AI-assisted support in a Calculus I course, implemented in close collaboration with the course instructor. We developed a system to answer students' questions on a discussion forum, fine-tuning a lightweight language model on 2,588 historical student-instructor interactions. The model achieved 75.3% accuracy on a benchmark of 150 representative questions annotated by five instructors, and in 36% of cases, its responses were rated equal to or better than instructor answers. Post-deployment student survey (N = 105) indicated that students valued the alignment of the responses with the course materials and their immediate availability, while still relying on the instructor verification for trust. We highlight the importance of hybrid human-AI workflows for safe and effective course support.

Jérémy Barghorn, Anna Sotnikova, Sacha Friedli +1

École polytechnique fédérale de Lausanne Switzerland

Mathematics EducationEducational QAGenerative AI in EducationQuestion AnsweringHuman-AI CollaborationLLM Fine-Tuning
AskPDF
Apr 8, 2026

AI Slop or AI-enhancement? Student perceptions of AI-generated media for an English for Academic Purposes course

Artificial intelligence (AI) retrieval-augmented generation (RAG) tools now enable educators to transform course materials into diverse multimedia at scale. However, it remains unclear whether such AI-generated content functions as a pedagogical scaffold or AI slop: high volume, low quality material. This innovative practice paper reports on the development, implementation, and evaluation of teacher-prompted, AI-generated supplemental materials in an English for Academic Purposes (EAP) course at a Hong Kong Community College. Using primarily Google Notebook LM, the instructor generated videos, podcasts, infographics, and individualized feedback reports from course materials and student work for 106 English as a Foreign Language learners. An explanatory sequential mixed-methods design comprising a survey, semi-structured interviews, and correlation analysis with academic scores was employed to examine students' preferences, perceptions, and learning outcomes. Findings are framed through the Technology Acceptance Model and Cognitive Load Theory. Students rated the materials highly for perceived usefulness and ease of use, and preferred assessment-linked content presented in visual and multimodal formats, particularly videos and infographics. Video preference correlated positively with academic performance; however, higher cognitive load was negatively associated with course grades, indicating that material complexity must be carefully calibrated. Notably, some lower-performing students independently adopted the materials as remedial scaffolds. The practice demonstrates that RAG tools enable scalable personalized feedback that would be less feasible through traditional methods. When aligned with student goals and cognitive principles, teacher-prompted AI generation can meaningfully enhance the EAP learning ecosystem rather than producing AI slop.

David James Woo, Deliang Wang, Kai Guo

Everwrite Limited, Hong Kong, China · Faculty of Education, The University of Hong Kong, Hong Kong, China · Faculty of Education, The Chinese University of Hong Kong, Hong Kong, China

AI in EducationGenerative AI in Education
AskPDF
Jun 11, 2026

Carolina Guide: A Multi-Agent RAG System with Institutional Guardrails for Academic Policy Assistance

University students often struggle to navigate complex academic policies, leading to advising bottlenecks and delayed access to critical information. Although large language models (LLMs) offer promise for automated assistance, their tendency toward hallucination and inability to enforce institutional constraints make them unsuitable for high-stakes policy guidance without careful architectural design. We present Carolina Guide, a retrieval-augmented generation (RAG) system for academic policy assistance at the University of South Carolina (USC). The system employs a modular multi-agent pipeline with institutional guardrails to provide citation-supported, policy-grounded answers to student queries while refusing unsafe requests such as course recommendations or personalized advising. We evaluate the system on a 90 query test set across 6 departments, achieving 98.9% retrieval success at the >= 2 threshold (genuinely relevant results) with the first relevant chunk at rank-1 for 98.9% of queries (MRR at 10 for rel >= 2 = 0.989). Through systematic baseline comparisons and ablation studies, we show that each architectural component-MMR reranking, adequate retrieval context (k=20), and citation enforcement-contributes measurable practical value despite limited statistical power at 90 queries. The evaluation of the guardrail on 30 adversarial queries demonstrates Safety F1 of 0.89, correctly refusing 86% of unsafe queries while maintaining 93% coverage of benign queries. These results show that production-ready LLM systems for institutional policy guidance require rethinking standard RAG patterns to prioritize safety, transparency, and departmental autonomy over conversational sophistication.

Ben Torsion, Jun Zhou

University of South Carolina, Columbia, South Carolina, USA · University of South Carolina, Columbia, USA

Multi-Agent LLM SystemsLLM GuardrailsAgentic RAGRAG Evaluation
AskPDF