Organizations: School of Systems Science and Engineering, Sun Yat-sen University, Guangzhou, China · School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China · Alibaba Cloud Computing, Hangzhou, China
In the field of software operations, Large Language Models (LLMs) have attracted increasing attention. However, existing research has not yet achieved efficient and effective endto-end intelligent operations due to low-quality data, fragmented knowledge and insufficient learning. To explore the potential of LLMs in software operations, we propose OpsLLM, a domainspecific LLM that supports both knowledge-based question answering (QA) and root cause analysis (RCA). Moreover, we disclose the detailed workflow for building LLMs specifically in the software operations domain. First, a Human-in-the-Loop mechanism is introduced to curate high-quality data from a large collection of operational data and construct a fine-tuning dataset. Then, based on the data, supervised fine-tuning is conducted to achieve a base model. Furthermore, we introduce a domain process reward model (DPRM) during the reinforcement learning stage to optimize the accuracy and reliability of the fine-tuned model on RCA tasks. Experimental results on the tasks with diverse difficulties demonstrate that OpsLLMs effectively learns and aligns with the operational domain knowledge infused, outperforming existing open-source and closed-source LLMs in accuracy with improvements of 0.2%~11.9% on QA tasks and 8.5%~70.3% on RCA tasks, while exhibiting strong transferability. Moreover, we will open-source three versions of OpsLLM with 7B, 14B and 32B parameters, along with a 15K fine-tuning dataset.
Figures & tables
Fig. 1: (a) shows the accuracy of different LLMs on OpsEval. (b) shows the RCA accuracy of different LLMs on 150 fault injection cases from OnlineBoutique. OpsLLM-14B SFT and OpsLLM-32B SFT are fine-tuned models based on Qwen2.5 models.
Fig. 2: Across different LLMs, more than 80% of responses ranked in the top 20% by ROUGE-L/BERTScore still contain detail errors. This indicates that high similarity scores do not guarantee correctness.
Fig. 3: The workflow of model construction and evaluation.
Fig. 4: Ops fine-tuning dataset construction workflow.
Fig. 5: Workflow of QA pairs to single-choice MCQ.
LLM Category
LLM
ACC (%)
QA
RCA
Easy
Mid
Hard
Easy
Mid
Hard
Base LLM
Qwen2.5-7B-Instruct
96.8
97.0
66.1
8.1
4.0
4.4
Qwen2.5-14B-Instruct
97.6
97.2
73.3
9.7
4.3
4.7
Qwen2.5-32B-Instruct
98.0
98.8
76.8
11.3
8.7
6.0
Open-source LLM
Moonshot-Kimi-K2-Instruct
98.6
98.6
83.8
11.3
11.7
7.7
TABLE I: Performance comparison across different LLMs.
Task
Easy †
Mid ‡
Hard
QA
SFT data (500)
HITL-filtered (500)
OpsEval [ 46 ] (1945)
RCA
SFT data (62)
Fault injection (300)
AIOps2025 [ 47 ] (365)
TABLE II: Evaluation dataset for different difficulty-levels.
Fig. 6: Comparison of practical RCA reasoning examples across different LLMs.
Fig. 7: Inference Latency Across QA and RCA Datasets of Different Difficulty Levels. Error bars represent Mean ± Standard Deviation (SD), i.e., the range [Mean − SD, Mean + SD].
Fig. 8: Response quality assessment of OpsLLM-14B on six datasets using GLM-5.2 as an LLM judge. The solid bars denote the percentage of responses judged as high quality, and the hatched regions indicate the proportion of high-confidence judgments within the high-quality responses. (RCA-Hard samples exceeding the model context length were excluded.)
Large language model post-training adapts pretrained models into more useful, reliable, and specialized assistants through supervised and preference-based learning.
Automated root cause analysis (RCA) with large language models (LLMs) has drawn growing attention. Today, SREs typically automate RCA with LLMs in one of two ways: directly using a general-purpose agent (e.g., Codex or Claude Code) for diagnosis, or building a specialized RCA agent from scratch. As mainstream general agents grow more capable and iterate quickly, our quantitative study finds that the former now often surpasses the latter. Its accuracy, however, still falls short of production needs, and this gap stems mainly from the external adaptation layer outside the agent's general capabilities, namely the harness. We therefore argue that LLM-based RCA should focus on this external harness, reusing the strong general capabilities of a modern agent rather than rebuilding an agent from scratch. A key capability of such a harness is to self-evolve, accumulating system-specific experience from past diagnoses so that it gets better the more it is used. We introduce OpsHarness, a self-evolving RCA harness that turns diagnosis experience into reusable expertise. Its data plane combines layered operational knowledge with an idea-card tool library, while its control plane coordinates setup, diagnosis, evolution, and verification. During evolution, OpsHarness contrasts successful and failed trajectories, converts their evidence into atomic proposals, and admits updates only through a dual-gate verification process designed to prevent overfitting and regression. Across two public benchmarks and an industrial deployment, OpsHarness achieves 59.0% top-1 accuracy, improving over a bare general agent by 63.4% and over baseline RCA agents by 4.02×.
Haiyu Huang, Jiewei Lyu, Zhihan Jiang +5
The Chinese University of Hong Kong · Individual Researcher · ByteDance
We study whether Large Language Models (LLMs) can perform feature model analysis operations (AOs) directly on semi-formal textual blueprints, i.e., concise constrained-language descriptions of feature hierarchies and constraints, enabling early validation in Software Product Line scoping. Using 12 state-of-the-art LLMs and 16 standard AOs, we compare their outputs against the solver-based oracle FLAMA. Results show that reasoning-optimized models (e.g., Grok 4 Fast Reasoning, Gemini 2.5 Pro) achieve 88-89% average accuracy across all evaluated blueprints and operations, approaching solver correctness. We identify systematic errors in structural parsing and constraint reasoning, and highlight accuracy-cost trade-offs that inform model selection. These findings position LLMs as lightweight assistants for early variability validation.
Viet-Man Le, Thi Ngoc Trang Tran, Sebastian Lubos +2
Operations Research (OR) provides a rigorous framework for high-stakes decision-making, but effective OR modeling requires substantial domain knowledge, mathematical abstraction, and solver expertise. Recent LLM-based systems automate parts of this pipeline, yet remain limited by low accuracy on complex problems, opaque outputs, and narrow solver support. We propose COOPA (COoperative OPerations Agent), a modular LLM-agent architecture for interpretable and scalable OR decision support. It combines three components: iterative confidence-based modeling, which generates multiple candidate formulations, self-evaluates them across modeling dimensions, and selects one using a max-min confidence criterion; element-level provenance and confidence explanations, which link variables, parameters, constraints, and objectives to quoted source text and provide an audit trail for human verification; and multi-solver routing to specialized optimizer agents for different OR problem classes. Across three OR benchmarks, eight LLM backbones, and four baselines under identical conditions, COOPA achieves the best macro-average accuracy on six of eight backbones and improves over the strongest baseline by up to 6.7 percentage points. A within-system ablation isolates the contribution of iterative confidence-based modeling, while additional analyses and case studies illustrate the value of source traceability and multi-solver dispatch.
Chuanhao Li, Xiaoan Xu, Dirk Bergemann +3
1Tsinghua University · 2Duke University · 3Yale University