AskALCF: A Self-Improving Agentic System for Intelligent HPC User Support
Authors: Jingyan Jiang, Akhilesh Bondapalli, Krishna Rauniyar, Murat Keceli, Yasaman Ghadar, Haritha Siddabathuni Som, Venkatram Vishwanath, Michael E. Papka, Rong Ge, Huihuo Zheng
Organizations: Argonne National Laboratory Lemont, IL, USA · Clemson University Clemson, SC, USA · University of Louisiana at Lafayette Lafayette, LA, USA
AI-based question-answering systems are increasingly used for user support, including at high-performance computing (HPC) facilities. However, existing systems lack a systematic, deployment-oriented approach for evaluating final-answer quality and attributing failures to specific components. They also remain largely static after deployment, apart from periodic knowledge-base updates. We present AskALCF, an agentic log-analysis and improvement pipeline for HPC user support. AskALCF evaluates production interactions at a fine-grained level, identifies whether failures arise from missing knowledge, retrieval, prompting, model behavior, or tool execution, and routes each failure to an actionable repair. Its diagnostic core combines answer-quality assessment with a two-scope sufficiency test that distinguishes information missing from the documentation corpus from information present but not retrieved. We further introduce a controlled harness-training process that uses expert references, null edits, and explicit ship, revert, or decline decisions to improve the diagnostic pipeline reliably. Across 2,174 support turns, AskALCF reduced unattributable failures from 41.4% to 14.2%, while harness training improved expert agreement from 108/150 to 123/150 turns. These results demonstrate a governed form of self-evolution in which production evidence drives targeted system improvements. By turning production interactions into fine-grained diagnoses and actionable repairs, AskALCF provides a systematic foundation for continuously improving deployed AI support systems beyond knowledge-base updates alone.
Figures & tables
Fig. 1: AskALCF connects two support channels to a governed improvement loop. A user query enters Pillar 1 (1), whose trace is analyzed by Pillar 3 (2). Confirmed knowledge gaps go to Pillar 2 (3), while other failures go to their component owners (3b). A ticket enters Pillar 4 (0), which drafts a reply for staff review (4) and sends reusable fixes to Pillar 2 (5). Pillar 2 opens a pull request (6); after human approval, the merge trigger re-indexes the corpus (7). That final trigger has not yet fired.
Fig. 2: The diagnostic workflow and its expert-review loop. Layers B and A produce retrieval and answer evidence, respectively. The ordered rule tree (3) assigns the initial route, and the agent judge (4) audits that decision. A human expert reviews the evidence and both rationales (5); those labels support harness changes to the rule tree and judge skills (6). The lower panels show the review interface for a turn on which the two routes disagree.
Outcome
What it asserts, and the evidence that decides it
Repaired by
Failure modes — the turn failed, and the outcome names the component that can repair it
knowledge
No authoritative document exists anywhere in the corpus: the production result is insufficient and a wide corpus probe fails to surface one either ( R= F, C= F). The fix is to write the missing page.
documentation
retrieval
A sufficient document does exist, but production retrieval did not return it ( R= F, C= T). Internal conditions separate ranking depth, indexing, and query formulation, which are three different fix sites.
retrieval
prompt
A deterministic answer-policy check fires ahead of the sufficiency pivot: the answer addresses the wrong facility or the wrong system, or a critical user detail was missing and no clarifying question was asked.
prompt policy
model
Retrieval was sufficient and generation still failed: the answer cites a path or module that exists nowhere in the corpus, contradicts its own supplied context, or stays faithful to that context yet reaches a wrong conclusion.
model serving
tools_mcp
A tool call or backend service failed — error, schema mismatch, permission, or outage — so the turn never had a fair chance to succeed. Checked first, so an outage is never scored as a bad answer.
platform
TABLE I: The seven attribution outcomes the rule tree can emit, and the evidence that decides each. R is retrieval sufficiency at the production result and C is sufficiency over the whole corpus (§ III-B2 ); T and F are their truth values. Every scored turn receives exactly one outcome.
Fig. 3: Human-guided harness training. Each proposed change is evaluated as one arm with pre-registered target and control sets. A change is accepted only when the target cases improve and the controls remain stable. Evaluation cases are not used to propose, tune or select an arm.
Fig. 4: The same 2,174 turns scored before and after harness training, on identical input. Turns the router could not attribute fall from 900 (41.4%) to 309 (14.2%) because the retrieval test now returns a verdict where it previously returned nothing.
Fig. 5: Failure-mode distribution under two counting rules, as a share of each total. Raw is every scored turn (2,174); deduplicated keeps one session per distinct opening question (1,768), collapsing the 33-fold repeats so one defect is counted once. The ordering barely changes; the shares do. knowledge is the class a raw count understates, because documentation gaps repeat less often than live-state questions.
Arm
Change
training (98)
eval. (52)
all
—
baseline, gate closed
73
35
108
A1
rule-conditioned judge gate
80
37
117
A2
sufficiency + clarification
81
39
120
A3
stricter clarification rules
83
40
123
TABLE II: The accepted harness changes, scored against the frozen expert reference. Evaluation turns were not used to propose or select an arm.
Fig. 6: The rule tree and the agent judge on the same 2,169 turns. Neither bar is the shipped label : the judge overturns only under seven named rules (§ V-D ), so what ships lies between them. The disagreement is concentrated on the two classes a support desk cares most about: turns blamed on the model, and turns nobody could attribute.
policy
train (99)
eval (45)
spurious knowledge
A
rule tree alone, no judge
73%
64%
5
B
judge alone, no gate
80%
71%
1
C
judge if confidence ≥0.8
77%
67%
2
D
judge if the tree fired a weak rule (shipped)
81%
71%
5
E
D, and confidence ≥0.7
82%
71%
5
TABLE III: Gate policies on the same frozen records, measured earlier in the campaign, hence n=99/45 . “Spurious knowledge ” counts turns wrongly sent to documentation authoring. Bold marks the best value in a column. D is the policy we ship.
Fig. 7: One pass of the production loop. Only the first two fix kinds have been run; a retrieval-side repair is designed and not yet exercised. An effect is read as fix against null , never fix against before : selecting a target set on a stochastic symptom makes the before-arm rate 1.0 by construction. Numbers are in § V-E .
claudeopus48
gpt5.6
turns, of 288
before
null
fix
before
null
fix
clarifying question asked
145
128
101
69
68
37
under -clarification
3
4
9
15
17
24
neither defect
140
156
178
204
203
227
net, null → fix
−27 asked / +5 under
−31 asked / +7 under
p (asked / under)
0.006 / 0.18
9.3×10−6 / 0.039
TABLE IV: Turn counts over all 288 questions. The three rows are mutually exclusive and sum to 288. Row 1 counts every clarifying question asked, necessary or not. Under-clarification is the opposite defect and is necessity-aware: a question naming no system that the assistant answered instead of asking about. The effect is read null against fix. p is exact McNemar on the turns that changed.
target ( n=4 )
control ( n=64 )
dimension
gen
before
after
null
before
after
null
completeness
opus
0.118
0.299
0.127
0.437
0.411
0.447
completeness
gpt
0.098
0.173
0.089
0.348
0.370
0.365
faithfulness
opus
0.691
0.857
0.695
0.716
0.734
0.721
faithfulness
gpt
0.594
0.762
0.512
0.613
0.614
0.586
correctness
opus
0.700
0.773
0.722
0.622
0.638
0.617
TABLE V: Target is the four questions the new page answers, control the other 64 it cannot reach. The null arm is the before arm re-run through a cold cache. The judge is cross-family to both generation arms and was chosen before any result.