REPORT-OPS: Hybrid Evidence-Guided Reporting with Human-Supervised Refinement for HPC

Mah Kadidia Konate; Mayur Akewar; Janki Bhimani; Steven Farrell

cs.AIOct 13, 2026

REPORT-OPS: Hybrid Evidence-Guided Reporting with Human-Supervised Refinement for HPC

Authors: Mah Kadidia Konate, Mayur Akewar, Janki Bhimani, Steven Farrell

Organizations: NERSC, Lawrence Berkeley National Laboratory, Berkeley, California, USA · Florida International University, Miami, Florida, USA

Abstract

Operational reporting for high-performance computing (HPC) facilities requires combining information from monitoring services, schedulers, incident and maintenance records, storage measurements, periodic reports, and operator investigations. Producing a reliable report requires more than retrieving relevant records: the evidence must refer to the correct system, metric definition, time period, and source of authority. We present REPORT-OPS, a hybrid reporting agent in which an LLM interprets requests, plans evidence collection, evaluates possible explanations, and drafts reports, while author-seeded operational policies and automated checks determine whether the available evidence supports the requested conclusion. Depending on the evidence state, the system answers the request, reports a qualified conclusion, or withholds an unsupported claim. During offline refinement, an LLM critic proposes bounded policy changes that a human approves or rejects without editing; accepted candidates are promoted only after automated target, held-out, and regression testing. We evaluate REPORT-OPS on 68 sanitized reporting queries constructed from 50 base scenarios in a frozen HPC operational corpus, using three agent models and two LLM judges. REPORT-OPS achieves 68.63% Pass@80, compared with 7.02% for the strongest retrieval baseline. Evidence gating increases claim-support precision from 40.58% to 90.06%. On the 13-query explanatory subset, explanation accuracy reaches 55.54%. Across five rounds of human-supervised refinement, localized policy updates correct 91.17% of targeted failures and improve performance on related held-out cases by 9.33 percentage points, with no observed regression breaks under release gating. These results demonstrate improved evidence-grounded reporting and within-environment policy reuse; they do not establish unrestricted causal discovery or cross-facility generalization.

Figures & tables

Panel of Figure 1 — Fig. 1: Reporting fitness converts heterogeneous operational artifacts into claim-specific completion decisions. Topical relevance alone is insufficient: the source role, semantic scope, time scope, and data health must match the requested claim.

Element	Role and ownership
LLM agent	Interprets requests, plans retrieval, proposes explanations, links claims to records, and drafts reports.
Author-seeded policy	Defines ontology terms, evidence requirements, acceptable source roles, fallbacks, workload relations, and completion conditions; it is versioned and inspectable.
Automated verifier	Checks schema, entity, metric, unit, temporal scope, source authority, contradictions, and evidence state before permitting a claim.
Offline refinement	An LLM proposes a bounded policy patch; a human approves or rejects the unchanged proposal, and automated target, held-out, and regression tests control promotion.

TABLE I: Division of responsibility in REPORT-OPS .

Panel of Figure 3 — Fig. 2: REPORT-OPS online workflow under frozen policy $P_{k}$ . The agent interprets the request, retrieves operational evidence, evaluates candidate explanations, and verifies the report before selecting Answer , Qualify , or Block . Every phase contributes to an auditable trace. Human review is outside the online path and occurs only when an offline policy patch is proposed.

Field	Complete-evidence example	Missing-evidence example
Request	What was Perlmutter scheduled availability in October 2023?	What exact NERSC username was behind the faulty client in the January 8–9, 2026 scratch outage?
Evidence	A reviewed metrics report records Perlmutter scheduled availability as 98.3% for October 2023.	Reviewed incident evidence links the outage to a faulty accounting-service client but contains no username.
Requirement	An authoritative value with matching system, metric definition, unit, and reporting month.	An authoritative record explicitly identifying the requested user. Context identifying the client type is insufficient.
Expected state	Answer .	Block .
Reference behavior	Report 98.3%, including the system, metric, and month.	State that the username cannot be identified from the available evidence; do not infer a person or account.

TABLE II: Sanitized benchmark examples showing the request, evidence state, requirements, and reference behavior.

Refinement property	Observed value
Sequential rounds / agent models	5 / 3
Agent–round promotion opportunities	15
Promoted updates across rounds	10
Previous policy retained	5
Permitted typed patch families	7
Human edits to patch contents	0

TABLE III: Observed policy evolution over five rounds.

RQ1: Evidence validity and selective completion
Configuration	Pass@80 $\uparrow$	State F1 $\uparrow$	FCR $\downarrow$	CSP $\uparrow$
Direct LLM	$4.43$	45.06	75.99	$24.86$
Generic RAG	$6.43$	$27.46$	$97.44$	$23.55$
RAG + verification	$7.02$	$30.55$	$89.75$	$25.61$
REPORT-OPS, no evidence gate	70.42	$32.86$	$82.05$	$40.58$
Full REPORT-OPS	$68.63$	$33.58$	$76.92$	90.06

TABLE IV: Principal evaluation results. Arrows indicate the preferred direction for each metric. Bold denotes the best value in each column; ties are all bolded. RQ2 is evaluated on the 13 explanatory queries.

Panel of Figure 7 — Fig. 3: Evaluation summary across three agent models. Panel (a) shows completion accuracy under controlled evidence states. Panel (b) shows target-set and round-held-out Pass@80, regression breaks, and agent-specific promotions after the human acceptance and automated release gates; labels such as $3/3$ indicate promotion for all three agent models.

Evidence-Grounded Generation

Evidence-grounded generation produces answers and documents whose claims are supported by traceable sources.

See all 148 papers

Jul 27, 2026

DeepFaith: Evidence-Grounded LLMs for Faithful Incident Reporting in Multi-Stage APT Defense

Advanced Persistent Threats (APTs) are difficult to detect and interpret due to their multi-stage and stealthy nature. While recent autonomous defense systems leverage provenance graphs and learning-based models for detection and mitigation, their outputs remain largely machine-oriented and difficult for analysts to interpret. Large language models (LLMs) offer a promising interface for report generation, but often produce hallucinated or weakly grounded content. In this paper, we propose DeepFaith, an evidence-grounded framework for faithful incident reporting in multi-stage APT defense. DeepFaith transforms structured outputs from autonomous defense and explainability modules into natural-language reports that are explicitly aligned with underlying system evidence. The framework integrates a unified evidence representation, evidence-grounded prompting, faithfulness-aware generation, and post-generation verification to ensure that all generated statements are supported. Experiments in a realistic enterprise testbed demonstrate that DeepFaith improves faithfulness from 0.68 to 0.92, reduces unsupported claims from 0.32 to 0.08, and increases temporal consistency from 0.6 to 0.88, while maintaining concise reports and lower error rates than existing template-based and LLM-based solutions. These results show that evidence-grounded generation enables reliable, interpretable, and actionable reporting for security operations centers.

Trung V. Phan, Tri Gia Nguyen, Thomas Bauschert

Chair of Communication Networks, Technische Universität Chemnitz, 09126 Chemnitz, Germany · Department of Information Assurance, FPT University, Da Nang 50509, Vietnam

Cybersecurity LLMs for Cybersecurity Evidence-Grounded Generation

Ask PDF

Oct 21, 2026

Before the Ticket: Grounding HPC User Support in Documentation and Resolved Tickets with an Agentic LLM Assistant

Most questions that reach an HPC help desk have already been answered in the center's user guides or in a ticket someone else filed last semester; users still file new tickets because the guides are hard to navigate and the ticket history is not theirs to search. We present an assistant that answers from both sources: an orchestrator calls two tools over the Model Context Protocol, retrieval-augmented generation (RAG) over the live documentation and a Llama-3-8B model fine-tuned with QLoRA on 11,244 query-resolution pairs mined from historical tickets, and merges their answers. Every model runs on-premises or on the university's hosted LLM service because ticket data may not leave the institution. On 400 held-out tickets from Purdue's Rosen Center for Advanced Computing, disjoint from training and free of near-duplicates, and evaluated by two judges from model families that generated none of the answers, fine-tuning on resolutions rewritten into self-service guidance is the clear winner in agreement with RCAC's resolutions (BERTScore F1 improves from 0.06 to 0.27 over the base model) and is judged most correct on average. The end-to-end system does not improve on its tools, although they are complementary: choosing the better answer per question would exceed either tool (judged correctness 3.41 versus 3.20, an oracle bound), while the merge follows the documentation tool in 78% of answers. Returning one tool's answer instead recovers the fine-tuned tool's quality (3.19 versus 2.66), but neither a learned selector nor an LLM verifier beats that tool alone.

Elham J. Barezi, Primus C Kabuo, Niveditha Nerella

Rosen Center for Advanced Computing, Purdue University, West Lafayette, IN, USA

Question Answering Document QA LLM Tool Use High-Performance Computing Retrieval-Augmented Generation

Ask PDF

Jul 27, 2026

A corrective agentic hybrid RAG and an operations-grounded evaluation for a scientific facility

Scientific user facilities accumulate decades of operational knowledge that no single search index covers: electronic logbooks, technical documents, internal wikis, operations chat messages, maintenance records, and live control-system data. We present APS-RAG, Advanced Photon Source Retrieval Augmented Generation, a deployed platform that makes the institutional knowledge at the Advanced Photon Source (APS) accessible to staff through natural-language queries, along with an operations-grounded evaluation. The retrieval engine fuses dense, sparse, and knowledge-graph (KG) channels with query-type-adaptive reciprocal-rank fusion, adds a corrective agentic loop, and runs a native-tool ReAct executor over a Model Context Protocol (MCP) tooling layer. We construct APS-Bench, a 50-question, question-answering (QA) dataset with auditable gold answers. Every retrieval-augmented variant numerically improves strict vital-nugget recall over a naive BM25 baseline (63.8%), with the full corrective Agentic GraphRAG scoring (70.3%). The cross-encoder reranker contributes significantly to answer quality: removing it and allowing the LLM to score relevance drastically reduces strict vital recall by 32.8%. The graph channel and corrective loop contribute positively as expected, but the performance gains are marginal. Additionally, we also compare the performance of open-source and closed-source LLMs in final answer synthesis. We release the APS-Bench construction methodology, the six-layer evaluation harness, and the underlying codebase, along with the '/aps-rag' retrieval agent skill framework, to support reproduction and adoption at other facilities. Together, the deployed platform and its operations-grounded evaluation present a promising workflow for trustworthy, statistically grounded AI assistance in facility operations, transferable to other large scientific instruments.

Rajat Sainju, Dariusz Jarosz, Hairong Shang +5

Advanced Photon Source, Argonne National Laboratory, Lemont, Illinois 60439, USA

Graph Retrieval-Augmented Generation Scientific QA Agentic RAG RAG Evaluation

Ask PDF

Abstract

Evidence-Grounded Generation

Explore similar work

DeepFaith: Evidence-Grounded LLMs for Faithful Incident Reporting in Multi-Stage APT Defense

Before the Ticket: Grounding HPC User Support in Documentation and Resolved Tickets with an Agentic LLM Assistant

A corrective agentic hybrid RAG and an operations-grounded evaluation for a scientific facility