Most questions that reach an HPC help desk have already been answered in the center's user guides or in a ticket someone else filed last semester; users still file new tickets because the guides are hard to navigate and the ticket history is not theirs to search. We present an assistant that answers from both sources: an orchestrator calls two tools over the Model Context Protocol, retrieval-augmented generation (RAG) over the live documentation and a Llama-3-8B model fine-tuned with QLoRA on 11,244 query-resolution pairs mined from historical tickets, and merges their answers. Every model runs on-premises or on the university's hosted LLM service because ticket data may not leave the institution. On 400 held-out tickets from Purdue's Rosen Center for Advanced Computing, disjoint from training and free of near-duplicates, and evaluated by two judges from model families that generated none of the answers, fine-tuning on resolutions rewritten into self-service guidance is the clear winner in agreement with RCAC's resolutions (BERTScore F1 improves from 0.06 to 0.27 over the base model) and is judged most correct on average. The end-to-end system does not improve on its tools, although they are complementary: choosing the better answer per question would exceed either tool (judged correctness 3.41 versus 3.20, an oracle bound), while the merge follows the documentation tool in 78% of answers. Returning one tool's answer instead recovers the fine-tuned tool's quality (3.19 versus 2.66), but neither a learned selector nor an LLM verifier beats that tool alone.
Figures & tables
System
Period
Tickets
Used
Purdue Footprints (legacy)
2016–2023
19,714
No
Purdue TeamDynamix
2023–present
9,630
Yes
Anvil Jira
2023–present
5,732
Yes
Table 1: Historical ticket sources.
Figure 1: Architecture. The orchestrator classifies each message; HPC-support queries invoke Tool 1 (RAG over user guides) and Tool 2 (model fine-tuned on resolved tickets) in parallel over MCP, and a synthesis step merges their answers with source links. If the answer does not resolve the issue, the assistant directs the user to the help desk, where they file a ticket as before.
Setting
Value
Base model
Llama-3-8B-Instruct (4-bit)
Data
11,244 train / 600 validation pairs
Adaptation
QLoRA; r=16 , α=16 , dropout 0.05,
all projection layers
Max sequence length
2048
Batch
2 × 4 grad. accumulation (effective 8)
Table 2: Fine-tuning configuration.
Figure 2: The deployed interface. A mode selector ( Agentic / RAG / Fine-tuned ) lets a user, or an evaluator, pose the same question to the full system or to either tool alone; the suggested prompts cover common kinds of support request.
Metric
Base
Fine-tuned
Ticket-RAG
BERTScore F1
0.061
0.269
0.118
ROUGE-1 F1
0.281
0.435
0.341
ROUGE-2 F1
0.082
0.165
0.115
ROUGE-L F1
0.156
0.271
0.191
Cosine similarity
0.704
0.736
0.698
Judge: correctness
2.87
3.20
2.86
Table 3: Tool 2 candidates on the same 400 held-out test tickets (higher is better; judge scores 1–5, mean of two judges). Bold: best in row. For every reference and judge metric, the difference between the fine-tuned model and each other candidate has a 95% paired-bootstrap interval that excludes zero.
Metric
Tool 1
Tool 2
Agentic
Best of two
BERTScore F1
0.021
0.269
0.041
0.225
ROUGE-1 F1
0.302
0.435
0.313
0.416
ROUGE-2 F1
0.079
0.165
0.082
0.159
ROUGE-L F1
0.165
0.271
0.175
0.257
Cosine similarity
0.659
0.736
0.649
0.734
Judge: correctness
2.78
3.20
2.66
3.41
Table 4: Answer quality for Tool 1 (RAG on docs), Tool 2 (fine-tuned), and the end-to-end agentic system on the same 400 held-out test tickets (judge scores 1–5, mean of two judges). Bold: best in row among the three systems. Differences from Tool 2 have 95% paired-bootstrap intervals that exclude zero, except Tool 1’s completeness ( −0.05 , [−0.13,0.04] ). Best of two is an oracle, not a system: for each question it takes whichever tool answer the judges rated more correct (Tool 1 on 104 questions, Tool 2 otherwise, ties to Tool 2) and scores that answer; because it selects on the same judge scores, its judge rows are an optimistic upper bound for choosing between the tools.
Qwen2.5-72B
Gemma-3-27B
System
Mean
Tool 2 − system
Mean
Tool 2 − system
Base
3.34
−0.13[−0.22,−0.05]
2.40
+0.79[+0.66,+0.91]
Tool 2 (fine-tuned)
3.21
—
3.19
—
Tool 1 (docs RAG)
3.18
+0.03[−0.06,+0.12]
2.38
+0.81[+0.69,+0.93]
Ticket-RAG
3.15
+0.05[−0.04,+0.14]
2.58
+0.62[+0.49,+0.74]
Agentic
3.00
+0.21[+0.11,+0.31]
2.32
+0.87[+0.76,+0.99]
Table 5: Judged correctness (1–5) by each judge, and the paired difference Tool 2 (fine-tuned) minus each system with its 95% bootstrap interval; n=400 per system. Agreement between the judges over all 2,000 answers (Spearman ρ ): correctness 0.63, completeness 0.62, helpfulness 0.54.
Picks
Correctness
Compl.
Help.
Correctness by
Strategy
docs
Mean
Δ vs. Tool 2
Qwen
Gemma
Merge (agentic)
—
2.66
−0.54[−0.64,−0.45]
2.63
3.20
3.00
2.32
Tool 2 alone
0%
3.20
—
2.85
3.57
3.21
3.19
Learned selector
6%
3.19
−0.01[−0.02,+0.01]
2.87
3.57
3.22
3.16
LLM verifier
40%
3.08
−0.11[−0.17,−0.05]
2.98
3.54
3.26
2.91
Oracle
26%
3.41
+0.21[+0.17,+0.26]
2.99
3.68
3.49
3.33
Table 6: Choosing one tool answer per question instead of merging, on the 400 test tickets (judge scores 1–5; Δ : paired difference from Tool 2 alone, 95% bootstrap CI). Both selectors were developed on the 600-ticket validation split only and see the question and the two answers, never a reference. Picks docs : share of questions answered with Tool 1’s answer. The oracle row repeats Table 4 ’s upper bound.
Question answering studies how AI systems interpret questions and produce accurate, evidence-grounded answers across text, dialogue, and multimodal data.