REPORT-OPS: Hybrid Evidence-Guided Reporting with Human-Supervised Refinement for HPC

Mah Kadidia Konate; Mayur Akewar; Janki Bhimani; Steven Farrell

cs.AIOct 13, 2026

REPORT-OPS: Hybrid Evidence-Guided Reporting with Human-Supervised Refinement for HPC

Authors: Mah Kadidia Konate, Mayur Akewar, Janki Bhimani, Steven Farrell

Organizations: NERSC, Lawrence Berkeley National Laboratory, Berkeley, California, USA · Florida International University, Miami, Florida, USA

Abstract

Operational reporting for high-performance computing (HPC) facilities requires combining information from monitoring services, schedulers, incident and maintenance records, storage measurements, periodic reports, and operator investigations. Producing a reliable report requires more than retrieving relevant records: the evidence must refer to the correct system, metric definition, time period, and source of authority. We present REPORT-OPS, a hybrid reporting agent in which an LLM interprets requests, plans evidence collection, evaluates possible explanations, and drafts reports, while author-seeded operational policies and automated checks determine whether the available evidence supports the requested conclusion. Depending on the evidence state, the system answers the request, reports a qualified conclusion, or withholds an unsupported claim. During offline refinement, an LLM critic proposes bounded policy changes that a human approves or rejects without editing; accepted candidates are promoted only after automated target, held-out, and regression testing. We evaluate REPORT-OPS on 68 sanitized reporting queries constructed from 50 base scenarios in a frozen HPC operational corpus, using three agent models and two LLM judges. REPORT-OPS achieves 68.63% Pass@80, compared with 7.02% for the strongest retrieval baseline. Evidence gating increases claim-support precision from 40.58% to 90.06%. On the 13-query explanatory subset, explanation accuracy reaches 55.54%. Across five rounds of human-supervised refinement, localized policy updates correct 91.17% of targeted failures and improve performance on related held-out cases by 9.33 percentage points, with no observed regression breaks under release gating. These results demonstrate improved evidence-grounded reporting and within-environment policy reuse; they do not establish unrestricted causal discovery or cross-facility generalization.

Figures & tables

Panel of Figure 1 — Fig. 1: Reporting fitness converts heterogeneous operational artifacts into claim-specific completion decisions. Topical relevance alone is insufficient: the source role, semantic scope, time scope, and data health must match the requested claim.

Element	Role and ownership
LLM agent	Interprets requests, plans retrieval, proposes explanations, links claims to records, and drafts reports.
Author-seeded policy	Defines ontology terms, evidence requirements, acceptable source roles, fallbacks, workload relations, and completion conditions; it is versioned and inspectable.
Automated verifier	Checks schema, entity, metric, unit, temporal scope, source authority, contradictions, and evidence state before permitting a claim.
Offline refinement	An LLM proposes a bounded policy patch; a human approves or rejects the unchanged proposal, and automated target, held-out, and regression tests control promotion.

TABLE I: Division of responsibility in REPORT-OPS .

Panel of Figure 3 — Fig. 2: REPORT-OPS online workflow under frozen policy $P_{k}$ . The agent interprets the request, retrieves operational evidence, evaluates candidate explanations, and verifies the report before selecting Answer , Qualify , or Block . Every phase contributes to an auditable trace. Human review is outside the online path and occurs only when an offline policy patch is proposed.

Field	Complete-evidence example	Missing-evidence example
Request	What was Perlmutter scheduled availability in October 2023?	What exact NERSC username was behind the faulty client in the January 8–9, 2026 scratch outage?
Evidence	A reviewed metrics report records Perlmutter scheduled availability as 98.3% for October 2023.	Reviewed incident evidence links the outage to a faulty accounting-service client but contains no username.
Requirement	An authoritative value with matching system, metric definition, unit, and reporting month.	An authoritative record explicitly identifying the requested user. Context identifying the client type is insufficient.
Expected state	Answer .	Block .
Reference behavior	Report 98.3%, including the system, metric, and month.	State that the username cannot be identified from the available evidence; do not infer a person or account.

TABLE II: Sanitized benchmark examples showing the request, evidence state, requirements, and reference behavior.

Refinement property	Observed value
Sequential rounds / agent models	5 / 3
Agent–round promotion opportunities	15
Promoted updates across rounds	10
Previous policy retained	5
Permitted typed patch families	7
Human edits to patch contents	0

TABLE III: Observed policy evolution over five rounds.

RQ1: Evidence validity and selective completion
Configuration	Pass@80 $\uparrow$	State F1 $\uparrow$	FCR $\downarrow$	CSP $\uparrow$
Direct LLM	$4.43$	45.06	75.99	$24.86$
Generic RAG	$6.43$	$27.46$	$97.44$	$23.55$
RAG + verification	$7.02$	$30.55$	$89.75$	$25.61$
REPORT-OPS, no evidence gate	70.42	$32.86$	$82.05$	$40.58$
Full REPORT-OPS	$68.63$	$33.58$	$76.92$	90.06

TABLE IV: Principal evaluation results. Arrows indicate the preferred direction for each metric. Bold denotes the best value in each column; ties are all bolded. RQ2 is evaluated on the 13 explanatory queries.

Panel of Figure 7 — Fig. 3: Evaluation summary across three agent models. Panel (a) shows completion accuracy under controlled evidence states. Panel (b) shows target-set and round-held-out Pass@80, regression breaks, and agent-specific promotions after the human acceptance and automated release gates; labels such as $3/3$ indicate promotion for all three agent models.

Evidence-Grounded Generation

Evidence-grounded generation produces answers and documents whose claims are supported by traceable sources.

See all 148 papers

DeepFaith: Evidence-Grounded LLMs for Faithful Incident Reporting in Multi-Stage APT Defense
Jul 27, 2026Trung V. Phan, Tri Gia Nguyen, Thomas BauschertCybersecurity LLMs for Cybersecurity
Chair of Communication Networks, Technische Universität Chemnitz, 09126 Chemnitz, Germany · Department of Information Assurance, FPT University, Da Nang 50509, Vietnam
Before the Ticket: Grounding HPC User Support in Documentation and Resolved Tickets with an Agentic LLM Assistant
Oct 21, 2026Elham J. Barezi, Primus C Kabuo, Niveditha NerellaQuestion Answering Document QA
Rosen Center for Advanced Computing, Purdue University, West Lafayette, IN, USA
A corrective agentic hybrid RAG and an operations-grounded evaluation for a scientific facility
Jul 27, 2026Rajat Sainju, Dariusz Jarosz, Hairong Shang +5Graph Retrieval-Augmented Generation Scientific QA
Advanced Photon Source, Argonne National Laboratory, Lemont, Illinois 60439, USA

Jul 27, 2026J/K move · Enter open · S save

DeepFaith: Evidence-Grounded LLMs for Faithful Incident Reporting in Multi-Stage APT Defense

Trung V. Phan, Tri Gia Nguyen, Thomas Bauschert

Chair of Communication Networks, Technische Universität Chemnitz, 09126 Chemnitz, Germany · Department of Information Assurance, FPT University, Da Nang 50509, Vietnam

Cybersecurity LLMs for Cybersecurity Evidence-Grounded Generation

Open paper AskPDF arXiv

Advanced Persistent Threats (APTs) are difficult to detect and interpret due to their multi-stage and stealthy nature. While recent autonomous defense systems leverage provenance graphs and learning-based models for detection and mitigation, their outputs remain largely machine-oriented and difficult for analysts to interpret. Large language models (LLMs) offer a promising interface for report generation, but often produce hallucinated or weakly grounded content. In this paper, we propose DeepFaith, an evidence-grounded framework for faithful incident reporting in multi-stage APT defense. DeepFaith transforms structured outputs from autonomous defense and explainability modules into natural-language reports that are explicitly aligned with underlying system evidence. The framework integrates a unified evidence representation, evidence-grounded prompting, faithfulness-aware generation, and post-generation verification to ensure that all generated statements are supported. Experiments in a realistic enterprise testbed demonstrate that DeepFaith improves faithfulness from 0.68 to 0.92, reduces unsupported claims from 0.32 to 0.08, and increases temporal consistency from 0.6 to 0.88, while maintaining concise reports and lower error rates than existing template-based and LLM-based solutions. These results show that evidence-grounded generation enables reliable, interpretable, and actionable reporting for security operations centers.

Abstract

Evidence-Grounded Generation

Explore similar work

DeepFaith: Evidence-Grounded LLMs for Faithful Incident Reporting in Multi-Stage APT Defense

Before the Ticket: Grounding HPC User Support in Documentation and Resolved Tickets with an Agentic LLM Assistant

A corrective agentic hybrid RAG and an operations-grounded evaluation for a scientific facility