cs.AIOct 1, 2026

On-Premises Multi-Course RAG Tutoring for Business Education: Hardware-Software Trade-offs in a Campus AI Tutor

Authors: Sidney Shapiro, Joshua Lindemann

Organizations: Dhillon School of Business, University of Lethbridge

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Campus AI tutors based on retrieval-augmented generation (RAG) must ground answers in assigned course materials while keeping textbooks and student dialogue on institutional infrastructure. We present CourseChat, an on-premises, multi-course RAG tutor for undergraduate business education, deployed behind a campus web gateway and intended for use embedded in Moodle. Six isolated course offerings, each keyed by its own course reference number (CRN), share twin-edge AI hosts running a FastAPI service, a local vector database, and a local large language model (LLM) served by Ollama. We report two generation-model bake-off rounds, a separate fixed-evidence source-fidelity comparison, and conversation and quiz audits. Several larger models failed the classroom speed gate, but a 12B model and a 7B alternative passed. A separate mixture-of-experts candidate improved some corrections while introducing new factual and continuity errors. We therefore retain the 8B production model pending a demonstrated overall improvement, rather than claiming that 8B is universally optimal. Software changes improved follow-up topic resolution while preserving course scope; 435 prebuilt questions across 65 modules decouple practice from live generation. The results support treating model choice, evidence selection, serving compatibility, and product design as a joint engineering decision. They do not establish learning gains: faculty ratings, peak-load capacity, and complete public-gateway acceptance remain separate evaluation needs.

Figures & tables

Figure 1: Application request paths. Quiz practice uses stored items and deterministic grading. Course, mixed, and dependent follow-up routes retrieve scoped evidence before generation; social and clarification routes do not retrieve course content. Responses return through the same API and UI. The dashed arrow supplies course configuration.
Figure 2: Campus access architecture and deployed twin local stacks. Solid arrows indicate request/response traffic; the dashed path transfers index snapshots. Quiz and application artifacts are deployed separately. The node audits reported here bypass the gateway; complete public course-slug acceptance remained open in the deployment snapshot.
ApproachObserved resultTrade-offDecision
Larger dense generators (first round)All passed 12/12 live probes; median completion times were 1.75–3.36 times the 8B baseline.Some answers were more complete, but every candidate exceeded the +30%+30\% classroom latency limit. Probe passes alone do not establish pedagogical quality.Retain 8B for classroom service.
Qwen 7B and Mistral Nemo 12BBoth passed 12/12 probes; latency ratios were 1.01 and 1.24. Qwen completeness was 0.93 versus 0.91 for 8B.Both met the speed gate. Speed passes and small completeness differences did not establish a faculty-reviewed tutoring advantage.Retain as candidates for paired review.
Structured Qwen mixture-of-experts responseWarm median was 3.00 s versus 3.47 s for 8B, but review found three regressions across twelve fixed-evidence cases.Faster warm responses and better corrections accompanied a new factual error and an unnecessary refusal. Model and response contract changed together.Hold activation.
Deterministic follow-up resolutionTopic checks rose from 23.3–40.0% in baseline resolver preflight to 90–100% live across five courses; all 400 scope checks passed.Better continuity required no generator change. Correct topic selection and scope do not by themselves verify answer fidelity.Deploy resolver with 8B.
Prepared quiz banks435 questions cover 65 modules. Warm practice-request medians were 4.8 and 5.8 ms on the two nodes.Inspectable evidence and deterministic grading require finite banks, maintenance after source changes, and instructor review.Serve prepared banks; faculty approval remains pending.
Table 1: Decision summary across model trials and system-design evaluations. Results come from distinct protocols and are not a common accuracy ranking. Model latency ratios use the 8B baseline from the corresponding bake-off round; all reported timings are server-local.
ModelLive evalp50 (ms)vs. 8BSpeed gate
llama3.1:8b (prod.)12/1283921.00 ×\timespass
qwen2.5:14b-instruct12/12147261.75 ×\timesfail
gemma3:12b12/12188552.25 ×\timesfail
qwen3:14b12/12281943.36 ×\timesfail
qwen3.6:35b-A3B (Q4)0/12151091.80 ×\timesfail †
Table 2: Bake-off round 1 (includes thinking-capable tags).
ModelLive evalp50 (ms)vs. 8BSpeed gate
llama3.1:8b12/1285561.00 ×\timespass
qwen2.5:7b-instruct12/1286681.01 ×\timespass
mistral-nemo:12b12/12105891.24 ×\timespass
ibm/granite3.3:8b0/12214432.51 ×\timesfail ‡
Table 3: Bake-off round 2 (non-thinking / think-disabled candidates).
Figure 3: Knowledge preparation and deployment. Automatic quiz building requires passing corpus quality gates. Independently reviewed additions provide a separate path, bound to current primary evidence, without clearing failed chat evaluations. Both paths require artifact validation. Dashed arrows denote deployment transfers; instructor pedagogical review remains separate and pending for the published banks.
OfferingBaselineUpdated liveUpdated rateScope
Course 131/6060/60100.0%80/80
Course 224/6054/6090.0%80/80
Course 322/6055/6091.7%80/80
Course 417/6058/6096.7%80/80
Course 514/6059/6098.3%80/80
Course 619/6060/60100.0%80/80
Table 4: Intended-topic follow-up checks. Course 1 has a live baseline; Courses 2–6 have resolver-only baselines. Updated results are live for all courses with the unchanged 8B generator. Scope checks also include initial turns.
More in this topic

AI in Education

AI in Education studies how artificial intelligence can support teaching, learning, assessment, and educational decision-making.

See all 261 papers

Explore similar work

CardsList
Mar 21, 2026

From 50% to Mastery in 3 Days: A Low-Resource SOP for Localizing Graduate-Level AI Tutors via Shadow-RAG

Deploying high-fidelity AI tutors in schools is often blocked by the Resource Curse -- the need for expensive cloud GPUs and massive data engineering. In this practitioner report, we present a replicable Standard Operating Procedure that breaks this barrier. Using a Vision-Language Model data cleaning strategy and a novel Shadow-RAG architecture, we localized a graduate-level Applied Mathematics tutor using only 3 person-days of non-expert labor and open-weights 32B models deployable on a single consumer-grade GPU. Our pilot study on a full graduate-level final exam reveals a striking emergence phenomenon: while both zero-shot baselines and standard retrieval stagnate around 50-60% accuracy across model generations, the Shadow Agent, which provides structured reasoning guidance, triggers a massive capability surge in newer 32B models, boosting performance from 74% (Naive RAG) to mastery level (90%). In contrast, older models see only modest gains (~10%). This suggests that such guidance is the key to unlocking the latent power of modern small language models. This work offers a cost-effective, scientifically grounded blueprint for ubiquitous AI education.

Zonglin Yang, J. -H. Xie, Lining Zhang +2

School of Mechanics and Engineering Science, Peking University, Beijing, 100871, China · Readraft Intelligent Technology Co., Ltd., China · Graduate School of Education, Peking University, Beijing, 100871, China +1

Generative AI in EducationIntelligent Tutoring SystemsRetrieval-Augmented GenerationLLM Reasoning
AskPDF
May 7, 2026

From Surface Learning to Deep Understanding: A Grounded AI Tutoring System for Moodle

This demo paper describes the development of the AI Teaching & Learning Assistant, a modular Moodle plugin that leverages Retrieval-Augmented Generation (RAG) to deliver high-quality, hallucination-free education. The system employs a dual-centric design, providing students with interactive, Socratic-based tutoring and educators with a "human-in-the-loop" workspace for supervised content generation. By grounding Large Language Model (LLM) responses in teacher-provided materials, the assistant addresses the risks of misinformation while encouraging deep conceptual mastery. Evaluation via the Ragas (LLM-as-a-Judge) framework and a preliminary user study confirms its effectiveness, achieving faithfulness scores up to 0.97 and a 4.00/5.00 recommendation rate.

Anna Ostrowska, Michał Kukla, Gabriela Majstrak +4

Faculty of Mathematics and Information Sciences, Warsaw University of Technology, Warsaw, Poland

Intelligent Tutoring SystemsAI in EducationGenerative AI in EducationSocratic TutoringHuman-in-the-Loop AI
AskPDF
Jul 15, 2026

Learning Engagement Assistant (LEA): Cross-Course Scalability and Classroom Evaluation of an Agentic AI Tutoring System

This paper is an extension of a paper presented at the ICAART 2026 conference, which introduced LEA (Learning Engagement Assistant), an adaptive AI tutoring agent combining course-specific Retrieval-Augmented Generation (RAG) with structured Knowledge Component (KC) models across integrated Chat, Tutor, and Quiz modes. That prior work validated LEA on a single STEM course (CMP511) exclusively through simulation, using synthetic learner agents. This paper extends that work by reporting the first classroom deployment of LEA with real students (n = 8, CMP511) and the first empirical test of its cross-course scalability, deploying the system across three courses spanning two academic levels and two disciplinary domains. The study reveals a divergence from simulation predictions across modes, showing that synthetic evaluation alone cannot anticipate all aspects of real deployment. A RAGAS-based cross-course scalability evaluation (660 questions) finds Answer Relevancy and Context Precision broadly stable across courses (0.88-0.94 and 0.88-0.90 respectively), while Faithfulness declines with curriculum distance from the system's original course (0.69 to 0.50), a preliminary finding that may reflect generation logic tuned to the system's original subject rather than a scalability limitation. These findings suggest that while the orchestration layer requires no modification, full course-agnosticism of all downstream components requires further investigation.

Teri Rumble, Javad Zarrin, P. George Lovell +1

Faculty of Design, Informatics and Business, Abertay University, Dundee, United Kingdom.

Retrieval-Augmented GenerationIntelligent Tutoring SystemsAI in EducationLLM Agent EvaluationCross-Domain Generalization
AskPDF