Large Language Models have been increasingly employed in the creation of Virtual Assistants due to their ability to generate human-like text and handle complex inquiries. While these models hold great promise, challenges such as hallucinations, missing information, and the difficulty of providing accurate and context-specific responses persist, particularly when applied to highly specialized content domains. In this paper, we focus on addressing these challenges by developing a virtual assistant designed to support students at Maastricht University in navigating project-specific regulations. We propose a virtual assistant based on a Retrieval-Augmented Generation system that enhances the accuracy and reliability of responses by integrating up-to-date, domain-specific knowledge. Through a robust evaluation framework and real-life testing, we demonstrate that our virtual assistant can effectively meet the needs of students while addressing the inherent challenges of applying Large Language Models to a specialized educational context. This work contributes to the ongoing discourse on improving LLM-based systems for specific applications and highlights areas for further research.
Figures & tables
Figure 1: The VA architecture consists of retrieval, generation, and self-reflection parts. Retrieval \raisebox{-.9pt} {1}⃝ collects the relevant information to the user question. Generation \raisebox{-.9pt} {2}⃝ processes the retrieval information and structures it to generate the response. Self-reflection \raisebox{-.9pt} {3}⃝ is the fallback mechanism that ensures the correctness of the response and understanding of the user question.
Metric
GPT3.5
Gemini1.0Pro
Context precision
88%
89%
Context recall
42%
41%
Answer relevancy
57%
37%
Faithfulness
43%
32%
Customized precision
77%
77%
Table 1: Evaluation metrics for different generative LLMs
Scenarios
Correct Answers
“I Don’t Know”
Without VA
With VA
Without VA
With VA
1
32.4%
46.7%
29.4%
16.7%
2
17.6%
26.7%
32.4%
6.7%
3
32.4%
56.7%
50.0%
16.7%
4
2.9%
26.7%
41.4%
26.7%
5
53.3%
35.3%
30.0%
2.9%
Table 2: Impact of VA on Correct Answers and “I Don’t Know” Responses showing improvements in accuracy and reducing uncertainty in student responses across most scenarios.
Scenarios
On a scale of 1 to 5, how helpful was the VA?
1
2
3
4
5
1
6.67%
16.67%
30%
20%
26.67%
2
10%
16.67%
33.33%
26.67%
13.33%
3
10%
16.67%
13.33%
26.67%
33.33%
4
6.67%
16.67%
40%
30%
6.67%
5
11.76%
5.88%
8.82%
29.41%
44.12%
Table 3: Distribution of Helpfulness Scores of VA for each Scenario
Figure 2: Overall student satisfaction with the VA system showing a generally positive reception, based on students’ rating
Topic
Representative Words
Counts
0
[Missed pre-examination and final presentation consequences]
97
1
[Dealing with Inactive Group Members in a Project]
23
2
[help, context, new update]
12
3
[Force Majeure and Attendance Issues]
9
4
[Factors affecting individual grades in projects]
8
5
[Communication with Tutors and Examiners]
4
Table 4: The table categorizes the most asked topics related to group projects, revealing the frequency of specific concerns among students. It identifies the most discussed issues, focusing on areas that may require more attention.
The advent of Generative Artificial Intelligence (GenAI), and in particular Large Language Models (LLMs), is reshaping educational practice, while intensifying ethical debate about its adoption. To date, the dominant paradigm remains cloud-based and text-only chatbot: a centralized service that offers limited pedagogical control, weak transparency over knowledge sources, and non-trivial risks for privacy and regulatory compliance. This model also presumes continuous connectivity and recurring API costs, creating structural barriers for many institutions, reinforcing existing digital divides. At the same time, educational interaction with LLM can benefit from multimodal cues and embodied presence, requiring interfaces that move beyond text-only tutoring. In this work, we propose ELEVATE (Efficient LLM Education with Virtual Avatar Teaching Engine), a framework to develop efficient GenAI-driven avatar tutors governed by epistemic infrastructures. ELEVATE integrates LLM-driven dialogue with embodied 3D avatars for multimodal interaction and adopts a local-first execution model enabling deployment on consumer-grade hardware. The framework formalizes a three-stratum design that separates (i) a student-facing virtual avatar interaction layer, (ii) a local GenAI execution and multimodal synthesis core, and (iii) a teacher-facing governance layer. We implemented and evaluated a working prototype deployed in a real-world educational curriculum. The system runs on standard PCs and smartphones, and we provide system-level performance evidence to show responsive interaction under realistic hardware constraints. Finally, we discuss sociotechnical and pedagogical implications for responsible adoption, positioning ELEVATE as a scalable pathway for privacy-preserving and inclusive GenAI tutoring across heterogeneous school environments.
Lorenzo Stacchio, Michele Giordano, Daniele Berardini +2
Educational virtual laboratories can make experimental training more scala-ble, adaptive, and accessible, especially when students have limited access to physical laboratory facilities. However, authoring new simulated laboratory procedures remains costly: educators must describe new equipment, define how instruments and materials interact, and specify valid procedural flows that can be executed or assessed inside the virtual environment. Large lan-guage models can assist in this authoring process by generating detailed ex-perimental procedures, but their output should not be treated as directly exe-cutable plans. They may omit necessary actions, arrange steps in the wrong order, or produce instructions that are logically incorrect or incompatible with the laboratory equipment. This paper presents a prototype framework for managing uncertainty in LLM-generated procedural knowledge for virtu-al laboratory planning. The framework aims to reduce procedural uncertainty by using structured domain representations and uncertain LLM-generated state-transition samples to extract candidate procedural rules, transform them into explicit and inspectable constraints, and use them to repair uncertain procedural steps. Although the motivating domain refers to educational vir-tual laboratories, the underlying problem is more general: managing uncer-tain procedural knowledge for action planning in structured interactive envi-ronments. We illustrate the approach in a virtual laboratory domain involving laboratory instruments, containers, tools, and material-transfer actions.
Polychronis Karpodinis, Dimitris Kalles
School of Science and Technology, Hellenic Open University, Patras 26335, Greece
The rapid emergence of Large Language Models (LLMs) presents both opportunities and challenges for programming education. While students increasingly use generative AI tools, direct access often hinders the learning process by providing complete solutions rather than pedagogical hints. Concurrently, educators face significant workload and scalability challenges when providing timely, personalized feedback. This study investigates the capabilities of LLMs to safely and effectively assist educators in answering student questions within a CS1 programming course. To achieve this, we established a rigorous, reproducible evaluation process by curating a benchmark dataset of 170 authentic student questions from a learning management system, paired with ground-truth responses authored by subject matter experts. Because traditional text-matching metrics are insufficient for evaluating open-ended educational responses, we developed and validated a custom LLM-as-a-Judge metric optimized for assessing pedagogical accuracy. Our findings demonstrate that models, such as Gemini 3 flash, can surpass the quality baseline of typical educator responses, achieving high alignment with expert pedagogical standards. To mitigate persistent risks like hallucination and ensure alignment with course-specific context, we advocate for a "teacher-in-the-loop" implementation. Finally, we abstract our methodology into a task-agnostic evaluation framework, advocating for a shift in the development of educational LLM tools from ad-hoc, post-deployment testing to a quantifiable, pre-deployment validation process.