Technical operations teams resolve large volumes of incidents by synthesizing fragmented evidence from ticket text, historical cases, system logs, and technical documentation. Existing automation often relies on monolithic generation without explicit evidence modeling or provenance, making outputs difficult to verify when critical signals are sparse across sources. We propose ASTRA, an agentic system for ticket resolution in which a central orchestrator coordinates three specialist information-gathering agents and drives a judge-orchestrator refinement loop to produce evidence-backed troubleshooting reports. TicketSimilarityAgent retrieves relevant historical precedents through dense retrieval and LLM reranking; LogAgent distills hundreds of thousands of log lines into structured, quote-grounded findings using deterministic filtering and constrained LLM analysis; and DomainKnowledgeAgent retrieves relevant technical knowledge via the Model Context Protocol (MCP). Their outputs are transformed into a claim-evidence representation linking each claim to a verbatim source passage, assigning a support level, and preventing cross-attribution. A JudgeAgent scores the report on five criteria, while the OrchestratorAgent converts low scores into targeted follow-up queries for bounded iterative refinement. Evaluated on 987 real-world telecom fault tickets across seven product lines, ASTRA achieves a mean quality score of 4.13/5.0, with 59.9% of reports identifying the fault area at the component-family level or better. Relevance and Clarity scores are 4.88 and 4.94, respectively, while fabricated technical details remain below 3% of error cases. Stratification by fault type reveals that hardware faults remain substantially harder than software or configuration faults (Cohen's d=0.80), pointing to a fundamental limitation of text-based evidence channels for hardware fault diagnosis.
Figures & tables
Figure 1. ASTRA overview. Starting from a fault ticket and preserved logs, ASTRA invokes three specialized agents (TicketSimilarityAgent, DomainKnowledgeAgent, and LogAgent) to gather complementary evidence. Their outputs are converted into a claim–evidence intermediate representation with support labels, which is then used to generate a structured report. A Judge/Orchestrator scores each draft against rubric criteria and issues targeted follow-up queries to the most relevant agents for a bounded number of refinement rounds before producing the final report.
Figure 2
Figure 4. Accuracy stratified by fault type. (a) Horizontal dot plot with 95% confidence intervals ; the dashed line marks the “useful” threshold (score 3). Hardware faults have the lowest mean (2.03) and the widest CI, with only 32.2% scoring ≥ 3. (b) Stacked bars show the proportion of tickets in each accuracy bucket. Nearly 68% of hardware tickets score ≤ 2 (red–orange), compared to ∼ 31% for Config/Requirement and ∼ 41% for Software; hardware has almost no score-5 tickets.
Figure 5. Mean Accuracy by ground-truth root cause category, sorted by mean accuracy (final refinement round, 95% CI error bars). The dashed line marks the “useful” threshold (score 3). Faults attributable to 3rd-party hardware/software are hardest (Cohen’s d = − 0.74 vs. all other categories, p< 0.001); missing requirement/specification faults are most diagnosable, as their causal evidence appears directly in ticket narratives and documentation. Kruskal–Wallis H =65.1, p< 0.001.
Failure Mode
Count
%
Plausible but incorrect component
271
68.4
Wrong component or subsystem
108
27.3
Wrong fault domain (Acc ≤ 1)
69
17.4
Generic/boilerplate analysis
22
5.5
Fabricated technical details
< 10
< 3.0
Table 1. Error taxonomy for low-accuracy predictions (Accuracy ≤ 2, n =396, 40.1% of reports). Categories are non-exclusive.
Round
N
Acc
Comp
Rel
Clar
TD
Overall
R0
674
2.72
4.26
4.94
4.91
4.36
4.24
R1
674
2.65
4.34
4.90
4.93
3.86
4.14
R2
674
2.64
4.37
4.90
4.94
3.91
4.15
R3
674
2.62
4.36
4.86
4.94
3.90
4.14
R4
674
2.63
4.36
4.84
4.93
3.83
4.12
Table 2. Mean evaluation scores by refinement round (paired analysis, N =674 tickets that completed all five rounds).
Fault diagnosis identifies, localizes, and explains failures in engineered and computational systems using sensor data, logs, models, and domain knowledge.
Effective troubleshooting agents in enterprise customer support depend on retrieving actionable guidance from similar historical cases, yet existing retrieval-augmented generation (RAG) systems treat support cases as static documents and overlook their multi-stage, stateful nature. We introduce RAFT (Retrieval-Augmented Framework for Troubleshooting Agents), a stateful RAG framework that abstracts each closed historical case into a directed chain of timeline entries and retrieves at the entry level, surfacing cases whose intermediate states match the active case and returning the parent-case trajectory anchored at the matched state; an optional case-level graph links cases through a configurable similarity representation. We evaluate this retrieval layer directly, which, unlike evaluating a full agent system, requires no production deployment. Because public multi-stage troubleshooting data is extremely rare, we pair a synthetic benchmark built from Microsoft Learn Windows Server documentation with real Apache Jira issues carrying human-created duplicate labels. RAFT improves Case Hit over vanilla RAG and GraphRAG baselines at every stage of case progress, with statistically significant gains over the strongest baseline; the Jira results provide directional evidence that the advantage transfers to real case histories. We release our benchmark, implementation, and the Apache Jira evaluation set.
Modern software teams frequently encounter delays in resolving recurring or related issues due to fragmented knowledge scattered across JIRA tickets, developer discussions, and GitHub pull requests (PRs). To address this challenge, we propose a Retrieval-Augmented Generation (RAG) framework that integrates Sentence-Transformers for semantic embeddings with FAISS-based vector search to deliver context-aware ticket resolution recommendations. The approach embeds historical JIRA tickets, user comments, and linked PR metadata to retrieve semantically similar past cases, which are then synthesized by a Large Language Model (LLM) into grounded and explainable resolution suggestions. The framework contributes a unified pipeline linking JIRA and GitHub data, an embedding and FAISS indexing strategy for heterogeneous software artifacts, and a resolution generation module guided by retrieved evidence. Experimental evaluation using precision, recall, resolution time reduction, and developer acceptance metrics shows that the proposed system significantly improves resolution accuracy, fix quality, and knowledge reuse in modern DevOps environments.
Most questions that reach an HPC help desk have already been answered in the center's user guides or in a ticket someone else filed last semester; users still file new tickets because the guides are hard to navigate and the ticket history is not theirs to search. We present an assistant that answers from both sources: an orchestrator calls two tools over the Model Context Protocol, retrieval-augmented generation (RAG) over the live documentation and a Llama-3-8B model fine-tuned with QLoRA on 11,244 query-resolution pairs mined from historical tickets, and merges their answers. Every model runs on-premises or on the university's hosted LLM service because ticket data may not leave the institution. On 400 held-out tickets from Purdue's Rosen Center for Advanced Computing, disjoint from training and free of near-duplicates, and evaluated by two judges from model families that generated none of the answers, fine-tuning on resolutions rewritten into self-service guidance is the clear winner in agreement with RCAC's resolutions (BERTScore F1 improves from 0.06 to 0.27 over the base model) and is judged most correct on average. The end-to-end system does not improve on its tools, although they are complementary: choosing the better answer per question would exceed either tool (judged correctness 3.41 versus 3.20, an oracle bound), while the merge follows the documentation tool in 78% of answers. Returning one tool's answer instead recovers the fine-tuned tool's quality (3.19 versus 2.66), but neither a learned selector nor an LLM verifier beats that tool alone.
Elham J. Barezi, Primus C Kabuo, Niveditha Nerella
Rosen Center for Advanced Computing, Purdue University, West Lafayette, IN, USA