AskALCF: A Self-Improving Agentic System for Intelligent HPC User Support
Authors: Jingyan Jiang, Akhilesh Bondapalli, Krishna Rauniyar, Murat Keceli, Yasaman Ghadar, Haritha Siddabathuni Som, Venkatram Vishwanath, Michael E. Papka, Rong Ge, Huihuo Zheng
Organizations: Argonne National Laboratory Lemont, IL, USA · Clemson University Clemson, SC, USA · University of Louisiana at Lafayette Lafayette, LA, USA
AI-based question-answering systems are increasingly used for user support, including at high-performance computing (HPC) facilities. However, existing systems lack a systematic, deployment-oriented approach for evaluating final-answer quality and attributing failures to specific components. They also remain largely static after deployment, apart from periodic knowledge-base updates. We present AskALCF, an agentic log-analysis and improvement pipeline for HPC user support. AskALCF evaluates production interactions at a fine-grained level, identifies whether failures arise from missing knowledge, retrieval, prompting, model behavior, or tool execution, and routes each failure to an actionable repair. Its diagnostic core combines answer-quality assessment with a two-scope sufficiency test that distinguishes information missing from the documentation corpus from information present but not retrieved. We further introduce a controlled harness-training process that uses expert references, null edits, and explicit ship, revert, or decline decisions to improve the diagnostic pipeline reliably. Across 2,174 support turns, AskALCF reduced unattributable failures from 41.4% to 14.2%, while harness training improved expert agreement from 108/150 to 123/150 turns. These results demonstrate a governed form of self-evolution in which production evidence drives targeted system improvements. By turning production interactions into fine-grained diagnoses and actionable repairs, AskALCF provides a systematic foundation for continuously improving deployed AI support systems beyond knowledge-base updates alone.
Figures & tables
Fig. 1: AskALCF connects two support channels to a governed improvement loop. A user query enters Pillar 1 (1), whose trace is analyzed by Pillar 3 (2). Confirmed knowledge gaps go to Pillar 2 (3), while other failures go to their component owners (3b). A ticket enters Pillar 4 (0), which drafts a reply for staff review (4) and sends reusable fixes to Pillar 2 (5). Pillar 2 opens a pull request (6); after human approval, the merge trigger re-indexes the corpus (7). That final trigger has not yet fired.
Fig. 2: The diagnostic workflow and its expert-review loop. Layers B and A produce retrieval and answer evidence, respectively. The ordered rule tree (3) assigns the initial route, and the agent judge (4) audits that decision. A human expert reviews the evidence and both rationales (5); those labels support harness changes to the rule tree and judge skills (6). The lower panels show the review interface for a turn on which the two routes disagree.
Outcome
What it asserts, and the evidence that decides it
Repaired by
Failure modes — the turn failed, and the outcome names the component that can repair it
knowledge
No authoritative document exists anywhere in the corpus: the production result is insufficient and a wide corpus probe fails to surface one either ( R= F, C= F). The fix is to write the missing page.
documentation
retrieval
A sufficient document does exist, but production retrieval did not return it ( R= F, C= T). Internal conditions separate ranking depth, indexing, and query formulation, which are three different fix sites.
retrieval
prompt
A deterministic answer-policy check fires ahead of the sufficiency pivot: the answer addresses the wrong facility or the wrong system, or a critical user detail was missing and no clarifying question was asked.
prompt policy
model
Retrieval was sufficient and generation still failed: the answer cites a path or module that exists nowhere in the corpus, contradicts its own supplied context, or stays faithful to that context yet reaches a wrong conclusion.
model serving
tools_mcp
A tool call or backend service failed — error, schema mismatch, permission, or outage — so the turn never had a fair chance to succeed. Checked first, so an outage is never scored as a bad answer.
platform
TABLE I: The seven attribution outcomes the rule tree can emit, and the evidence that decides each. R is retrieval sufficiency at the production result and C is sufficiency over the whole corpus (§ III-B2 ); T and F are their truth values. Every scored turn receives exactly one outcome.
Fig. 3: Human-guided harness training. Each proposed change is evaluated as one arm with pre-registered target and control sets. A change is accepted only when the target cases improve and the controls remain stable. Evaluation cases are not used to propose, tune or select an arm.
Fig. 4: The same 2,174 turns scored before and after harness training, on identical input. Turns the router could not attribute fall from 900 (41.4%) to 309 (14.2%) because the retrieval test now returns a verdict where it previously returned nothing.
Fig. 5: Failure-mode distribution under two counting rules, as a share of each total. Raw is every scored turn (2,174); deduplicated keeps one session per distinct opening question (1,768), collapsing the 33-fold repeats so one defect is counted once. The ordering barely changes; the shares do. knowledge is the class a raw count understates, because documentation gaps repeat less often than live-state questions.
Arm
Change
training (98)
eval. (52)
all
—
baseline, gate closed
73
35
108
A1
rule-conditioned judge gate
80
37
117
A2
sufficiency + clarification
81
39
120
A3
stricter clarification rules
83
40
123
TABLE II: The accepted harness changes, scored against the frozen expert reference. Evaluation turns were not used to propose or select an arm.
Fig. 6: The rule tree and the agent judge on the same 2,169 turns. Neither bar is the shipped label : the judge overturns only under seven named rules (§ V-D ), so what ships lies between them. The disagreement is concentrated on the two classes a support desk cares most about: turns blamed on the model, and turns nobody could attribute.
policy
train (99)
eval (45)
spurious knowledge
A
rule tree alone, no judge
73%
64%
5
B
judge alone, no gate
80%
71%
1
C
judge if confidence ≥0.8
77%
67%
2
D
judge if the tree fired a weak rule (shipped)
81%
71%
5
E
D, and confidence ≥0.7
82%
71%
5
TABLE III: Gate policies on the same frozen records, measured earlier in the campaign, hence n=99/45 . “Spurious knowledge ” counts turns wrongly sent to documentation authoring. Bold marks the best value in a column. D is the policy we ship.
Fig. 7: One pass of the production loop. Only the first two fix kinds have been run; a retrieval-side repair is designed and not yet exercised. An effect is read as fix against null , never fix against before : selecting a target set on a stochastic symptom makes the before-arm rate 1.0 by construction. Numbers are in § V-E .
claudeopus48
gpt5.6
turns, of 288
before
null
fix
before
null
fix
clarifying question asked
145
128
101
69
68
37
under -clarification
3
4
9
15
17
24
neither defect
140
156
178
204
203
227
net, null → fix
−27 asked / +5 under
−31 asked / +7 under
p (asked / under)
0.006 / 0.18
9.3×10−6 / 0.039
TABLE IV: Turn counts over all 288 questions. The three rows are mutually exclusive and sum to 288. Row 1 counts every clarifying question asked, necessary or not. Under-clarification is the opposite defect and is necessity-aware: a question naming no system that the assistant answered instead of asking about. The effect is read null against fix. p is exact McNemar on the turns that changed.
target ( n=4 )
control ( n=64 )
dimension
gen
before
after
null
before
after
null
completeness
opus
0.118
0.299
0.127
0.437
0.411
0.447
completeness
gpt
0.098
0.173
0.089
0.348
0.370
0.365
faithfulness
opus
0.691
0.857
0.695
0.716
0.734
0.721
faithfulness
gpt
0.594
0.762
0.512
0.613
0.614
0.586
correctness
opus
0.700
0.773
0.722
0.622
0.638
0.617
TABLE V: Target is the four questions the new page answers, control the other 64 it cannot reach. The null arm is the before arm re-run through a cold cache. The judge is cross-family to both generation arms and was chosen before any result.
Most questions that reach an HPC help desk have already been answered in the center's user guides or in a ticket someone else filed last semester; users still file new tickets because the guides are hard to navigate and the ticket history is not theirs to search. We present an assistant that answers from both sources: an orchestrator calls two tools over the Model Context Protocol, retrieval-augmented generation (RAG) over the live documentation and a Llama-3-8B model fine-tuned with QLoRA on 11,244 query-resolution pairs mined from historical tickets, and merges their answers. Every model runs on-premises or on the university's hosted LLM service because ticket data may not leave the institution. On 400 held-out tickets from Purdue's Rosen Center for Advanced Computing, disjoint from training and free of near-duplicates, and evaluated by two judges from model families that generated none of the answers, fine-tuning on resolutions rewritten into self-service guidance is the clear winner in agreement with RCAC's resolutions (BERTScore F1 improves from 0.06 to 0.27 over the base model) and is judged most correct on average. The end-to-end system does not improve on its tools, although they are complementary: choosing the better answer per question would exceed either tool (judged correctness 3.41 versus 3.20, an oracle bound), while the merge follows the documentation tool in 78% of answers. Returning one tool's answer instead recovers the fine-tuned tool's quality (3.19 versus 2.66), but neither a learned selector nor an LLM verifier beats that tool alone.
Elham J. Barezi, Primus C Kabuo, Niveditha Nerella
Rosen Center for Advanced Computing, Purdue University, West Lafayette, IN, USA
Enterprise IT support interactions are fundamentally diagnostic: effective resolution requires iterative evidence gathering from ambiguous user reports to identify an underlying root cause. While retrieval-augmented generation (RAG) provides grounding through historical cases, standard multi-turn RAG systems lack explicit diagnostic state and therefore struggle to accumulate evidence and resolve competing hypotheses across turns. We introduce DQA, a diagnostic question-answering framework that maintains persistent diagnostic state and aggregates retrieved cases at the level of root causes rather than individual documents. DQA combines conversational query rewriting, retrieval aggregation, and state-conditioned response generation to support systematic troubleshooting under enterprise latency and context constraints. We evaluate DQA on 150 anonymized enterprise IT support scenarios using a replay-based protocol. Averaged over three independent runs, DQA achieves a 78.7% success rate under a trajectory-level success criterion, compared to 41.3% for a multi-turn RAG baseline, while reducing average turns from 8.4 to 3.9.
Agentic retrieval-augmented generation (RAG) interleaves retrieval, reasoning, and answer generation across multiple hops. A retrieval error at hop 1 can surface only as a wrong answer at hop 3, while later retrieval can also repair the trajectory. This paper introduces AgenticRAG-FP, an interventional benchmark for causal failure attribution in agentic RAG. The benchmark injects a certified fault at a specified hop, re-executes the downstream trajectory, and evaluates diagnosers against the known intervention site. Its central question is whether a post-hoc trace still identifies the injected hop after the suffix changes. In the dense Claude Haiku 4.5 structural sweep on three-hop MuSiQue questions, coverage-based diagnosis is 0.91 at hop 1 and 0.00 at hops 2 and 3 (n=43,36,21 failed trajectories), but it predicts hop 1 in 96% of failures, an earliest-hop rule; at hops 2 and 3, an LLM judge and frozen-hop repair localize only 0.25 to 0.48 of failures. In larger content-corruption runs, depth-2 exact-hop accuracy is 0.00 for coverage, 0.49 for an LLM judge, 0.59 for frozen-hop repair, and 0.14 for suffix regeneration (n=51); the judge and frozen-hop repair do not significantly differ (exact McNemar p=0.33).