Organizations: National University of Singapore, Singapore · Indian Institute of Information Technology Dharwad, India · IBM Software, India · Indian Association for the Cultivation of Science, India
LLMs now support IT operations (ITOps), but benchmarks rarely show why a model fails. We present ORBIT, a benchmark of ITOps questions in three formats, namely multiple-choice, open-ended, and troubleshooting, with each question accompanied by a reference answer and a recorded source. An audit found template defects in about a tenth of the generated items, so we drop them and rebuild the troubleshooting format from real practitioner threads with accepted answers. Under one independent judge, five LLMs answer most multiple-choice items correctly, between a third and just over half of the open-ended items, and a minority of the troubleshooting items, even under a more lenient root-cause rubric. When models judged their own answers, they accepted many answers to defective items, so we use one independent judge for every model. Word overlap with the question finds the correct option for more than half of the multiple-choice items. ORBIT is a diagnostic aid for selecting and supervising assistants rather than a deployment test.
Figures & tables
Benchmark
Set.
OG
ME
ET
RA
SWE-bench ( Jimenez et al., 2024 )
E
✗
✓
✓
✗
ITBench ( Jha et al., 2025 )
E
✓
✓
✗
✗
AIOpsLab ( Chen et al., 2025 )
E
✓
✓
✗
✗
SREGym ( Clark et al., 2026 )
E
✓
✓
✗
✗
Cloud-OpsBench ( Wang et al., 2026 )
E
✓
✓
✗
✗
OpenRCA ( Xu et al., 2025 )
T
✓
✓
✗
✗
Table 1: How ORBIT compares with other ITOps benchmarks on operational grounding, multi-entity reasoning, evidence traceability, and reasoning annotations.
Model
Overall
MCQ
OE
TS
( n=3,019 )
( n=2,513 )
( n=298 )
( n=208 )
Llama 3.3 70B
78.0 (76.5–79.5)
88.1 (86.8–89.3)
35.6 (30.4–41.2)
16.8 (12.4–22.5)
Mistral Small 3.1
79.2 (77.7–80.6)
88.4 (87.1–89.6)
47.7 (42.0–53.3)
13.5 (9.5–18.8)
Llama 4 Maverick
77.8 (76.3–79.2)
85.8 (84.4–87.1)
45.3 (39.7–51.0)
26.9 (21.4–33.3)
Granite 4.0 H
74.1 (72.5–75.6)
81.5 (79.9–82.9)
54.7 (49.0–60.3)
12.5 (8.7–17.7)
Mistral Medium
80.3 (78.8–81.6)
89.0 (87.7–90.2)
50.7 (45.0–56.3)
16.8 (12.4–22.5)
Table 2: Zero-shot accuracy (%) of each model on the evaluation set, with Wilson confidence intervals.
Model
Standard ( n=2,384 )
Hard ( n=129 )
Δ
Llama 3.3 70B
89.7 (88.4–90.8)
59.7 (51.1–67.8)
30.0
Mistral Small 3.1
90.1 (88.8–91.2)
56.6 (48.0–64.8)
33.5
Llama 4 Maverick
87.3 (85.9–88.6)
58.1 (49.5–66.3)
29.2
Granite 4.0 H
83.1 (81.5–84.5)
51.2 (42.6–59.6)
31.9
Mistral Medium
90.6 (89.3–91.7)
60.5 (51.8–68.5)
30.1
Table 3: Multiple-choice accuracy (%) on standard and hard items, with Wilson confidence intervals.
Multiple-choice
Generative
Model
Diagnostic
Other
OE multi-hop
TS
( n=362 )
( n=2,151 )
( n=235 )
( n=208 )
Llama 3.3 70B
59.1
93.0
41.3
16.8
Mistral Small 3.1
60.8
93.0
54.5
13.5
Llama 4 Maverick
59.1
90.3
53.2
26.9
Granite 4.0 H
54.1
86.1
63.0
12.5
Table 4: Accuracy (%) on diagnosis items and on other items within each format.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 1: Taxonomy of ORBIT by question type, source type, and reasoning type.
Figure 2: The ORBIT pipeline, from collecting and indexing sources, through evidence retrieval and LLM-assisted item generation, to validation and consensus checks before release.
Metric
Value
Unit
Question length
14.28±5.56
whitespace tokens
Technical terms per question
3.19±1.72
count
Answer length (option text for MCQ)
13.72±15.66
whitespace tokens
Reference-solution length
27.56±16.82
words
Evidence context length
135.22
tokens
Grounding references per item
0.98±0.46
count
Appendix
Table 5: Structural statistics of ORBIT, as mean ± standard deviation.
Model
Strict
Root cause
Llama 3.3 70B
16.8 (12.4–22.5)
28.8 (23.1–35.3)
Mistral Small 3.1
13.5 (9.5–18.8)
21.2 (16.2–27.2)
Llama 4 Maverick
26.9 (21.4–33.3)
40.9 (34.4–47.7)
Granite 4.0 H
12.5 (8.7–17.7)
18.3 (13.6–24.1)
Mistral Medium
16.8 (12.4–22.5)
33.2 (27.1–39.8)
Appendix
Table 6: Troubleshooting accuracy (%) on the rebuilt items under the strict prompt and under the root-cause rubric, with Wilson confidence intervals.
inst-101
DP-00029
DP-00041
DP-00025
Category
Troubleshooting (MCQ)
Product Understanding
Schema Extraction
Documentation Navigation
Domain
Observability
APM (Instana)
AWS (Cloud Provider)
GCP (Cloud Provider)
Reasoning type
Abductive root cause
Semantic interpretation
Factual / schema retrieval
Semantic interpretation
Grounding source
Community Discussion
Product Documentation
API Reference
Product Documentation
Question
, PHP-FPM requests intermittently hang for 20–60 seconds after output generation. What monitoring-related root cause was identified?
Within the Instana documentation hierarchy, what functional purpose does .instana serve?
What is the complete field structure of AwsEc2Instance ?
Based on the documentation hierarchy, what type of entry is .gcp ?
Table 7: Representative ORBIT items across domains and reasoning tiers, with shortened questions, options, and reasoning summaries.
Field
DP-02007 (troubleshooting format, multi-hop cohort, abductive root cause, hard)
Question
Given the following production scenario, perform root cause analysis. Scenario: A Kubernetes pod running a Java application is continuously OOMKilled. The node has 8GB of RAM. The pod has memory requests and limits set to 256Mi. The Java application is started with -Xmx512m -Xms128m .
Reference solution
The pod’s hard memory limit (256Mi) is lower than the JVM’s maximum heap size ( -Xmx512m ). Either increase the Kubernetes memory limit above 512Mi or reduce the JVM heap to fit within 256Mi.
Reasoning trace (excerpt)
(1) The node has ample RAM, but Kubernetes enforces memory at the container level through cgroups. (2) limits.memory of 256Mi is a hard ceiling, and exceeding it triggers the OOM killer. (3) The JVM is allowed to grow its heap to 512MB. (4) The configured heap exceeds the container limit, so the container is killed.
Appendix
Table 8: A full troubleshooting item as shown to models, with its reference solution and reasoning trace.
Comparison
Δ (pts)
pHolm
h
Format
Llama 3.3 70B, MC → OE
52.6
<10−111
1.16
Llama 3.3 70B, MC → TS
71.3
<10−148
1.59
Llama 3.3 70B, OE → TS
18.7
<10−5
0.43
Mistral Small 3.1, MC → OE
40.7
<10−72
0.92
Mistral Small 3.1, MC → TS
74.9
<10−163
1.69
Appendix
Table 9: Significance tests for the comparisons behind the main findings.
Comparison
Δ (pts)
pHolm
h
Model pairs, overall accuracy
L3.3 vs M-Small
− 1.2
0.816
− 0.03
L3.3 vs L4-Mav
0.3
0.816
0.01
L3.3 vs Granite
4.0
0.002
0.09
L3.3 vs M-Med
− 2.2
0.169
− 0.05
M-Small vs L4-Mav
1.4
0.712
0.03
Appendix
Table 10: Significance tests between each pair of models on the evaluation set.
Model
Retrieval
Semantic
Reasoning-heavy
( n=1,986 )
( n=144 )
( n=383 )
Llama 3.3 70B
97.0 (96.2–97.7)
47.2 (39.2–55.3)
57.4 (52.4–62.3)
Mistral Small 3.1
97.1 (96.2–97.7)
49.3 (41.3–57.4)
58.0 (53.0–62.8)
Llama 4 Maverick
94.2 (93.1–95.2)
43.1 (35.3–51.2)
58.5 (53.5–63.3)
Granite 4.0 H
90.4 (89.1–91.6)
31.2 (24.2–39.2)
53.8 (48.8–58.7)
Mistral Medium
96.8 (95.9–97.5)
50.7 (42.6–58.7)
63.2 (58.2–67.9)
Appendix
Table 11: Zero-shot multiple-choice accuracy (%) by tier of the LLM-assigned primary reasoning type, with Wilson confidence intervals.
Open-ended
Troubleshooting
Model
Claude
GPT-5.5
Claude
GPT-5.5
Llama 3.3 70B
35.6
64.1
16.8
26.0
using Mistral Small 3.1
47.7
69.8
13.5
20.2
Llama 4 Maverick
45.3
67.4
26.9
38.9
Granite 4.0 H
54.7
74.8
12.5
16.3
Mistral Medium
50.7
75.8
16.8
27.4
Appendix
Table 12: Open-ended and troubleshooting accuracy (%) under two judges from different vendors, and how often the judges agree.
OE ( n=532 )
TS ( n=133 )
Defective ( n=287 )
Model
Self
Ind.
Self
Ind.
Self
Ind.
Mistral Medium
51.5
33.1
36.8
9.8
7.3
4.2
Mistral Small 3.1
46.4
31.6
24.8
11.3
10.8
5.2
Llama 4 Maverick
42.5
30.3
29.3
10.5
53.7
4.5
Llama 3.3 70B
50.9
23.7
39.9
10.5
56.8
3.5
Granite 4.0 H
42.5
34.8
34.6
8.3
74.6
3.5
Appendix
Table 13: Accuracy (%) on all generated open-ended and troubleshooting items, and the share of answers accepted on defective items, under each judge.
Model
Overall
MCQ
OE
TS
( n=3,178 )
( n=2,513 )
( n=532 )
( n=133 )
Llama 3.3 70B
78.82 (77.37–80.21)
86.79 (85.41–88.06)
50.94 (46.70–55.17)
39.85 ( 31.93–48.34 )
Mistral Small 3.1
78.70 (77.24–80.09)
88.38 (87.07–89.58)
46.43 (42.23–50.68)
24.81 (18.25–32.79)
Llama 4 Maverick
77.38 (75.89–78.80)
87.31 (85.95–88.55)
42.48 (38.35–46.72)
29.32 (22.25–37.55)
Granite 4.0 H
74.92 (73.38–76.40)
83.92 (82.44–85.31)
42.48 (38.35–46.72)
34.59 (27.04–43.00)
Mistral Medium
80.11 ( 78.69–81.46 )
88.46 ( 87.15–89.65 )
51.50 ( 47.26–55.72 )
36.84 (29.12–45.30)
Appendix
Table 14: Earlier self-judged pass, with zero-shot accuracy (%) on all generated items and Wilson confidence intervals.
Model
Standard ( n=2,438 )
Hard ( n=740 )
Δ
Zero-shot
Llama 3.3 70B
89.38 (88.09–90.54)
51.22 (47.62–54.80)
38.16
Mistral Small 3.1 (24B)
89.70 (88.44–90.85)
46.49 (42.92–50.09)
43.21
Granite 4.0 H Small
83.18 (81.65–84.62)
41.89 (38.39–45.48)
41.29
Mistral Medium (2505)
89.99 (88.74–91.12)
52.43 (48.83–56.01)
37.56
Few-shot
Appendix
Table 15: Earlier self-judged pass, with accuracy (%) on standard and hard items and Wilson confidence intervals.
Long-context
Multi-hop
Diagnostic
Model
localization
causal inference
reasoning
( n=6 )
( n=253 )
( n=680 )
Zero-shot
Llama 3.3 70B Instruct
66.67 (30.00–90.32)
73.12 (67.34–78.21)
47.21 (43.48–50.96)
Mistral Small 3.1 (24B)
66.67 (30.00–90.32)
67.59 (61.60–73.06)
42.65 (38.98–46.40)
Granite 4.0 H Small
50.00 (18.76–81.24)
52.57 (46.42–58.64)
41.32 (37.68–45.06)
Appendix
Table 16: Earlier self-judged pass, with accuracy (%) by failure-mode cohort and Wilson confidence intervals.
Metric
(1,2)
(2,3)
(3,1)
(1,2,3)
Weighted Cohen’s κ
0.76
0.64
0.70
–
Unweighted Cohen’s κ
0.61
0.49
0.55
–
ICC(2,1)
0.78
0.65
0.72
0.72
ICC(2, k )
0.88
0.79
0.84
0.89
Exact Agreement (%)
76
65
70
58–60
Within ± 1 Agreement (%)
98
91
95
90
Appendix
Table 17: Agreement among human annotators on grades of model responses to randomly sampled items.
Metric
C–D
C–G
D–G
Weighted Cohen’s κ
0.65
0.80
0.78
Unweighted Cohen’s κ
0.36
0.54
0.53
Exact Agreement (%)
55.24
68.57
69.52
Within ± 1 Agreement (%)
94.29
100.00
100.00
Spearman Correlation
0.64
0.80
0.81
Bland-Altman Bias
0.10
0.03
-0.08
Appendix
Table 18: Agreement among LLM graders on the sampled items.
Metric
Claude
Gemini
DeepSeek
Ensemble
Weighted Cohen’s κ
0.71
0.75
0.68
0.80
Exact Agreement (%)
71
75
68
81
Within ± 1 Agreement (%)
96
98
95
99
ICC(2,1)
0.73
0.77
0.70
0.82
Appendix
Table 19: Agreement between human graders and LLM graders on the sampled items.
Metric
Value
Stack Overflow hit rate (overall)
75.05%
Hit rate (non-documentation items)
100.00%
Top-1 title match rate
54.69%
Zero-hit items
793 (24.95%)
Documentation-derived among zero-hit
100.00%
Appendix
Table 20: Practitioner-topic coverage of ORBIT using the OpsEval Stack Overflow retrieval method.
Question Type
Hit Rate
Multiple Choice
68.44%
Open-ended
100.00%
Troubleshooting
100.00%
Appendix
Table 21: Practitioner-topic coverage by question type.
Method
ATS ↑
ACE ↓
AUROC ↑
arith-agg
0.291±0.007
0.493±0.001
0.529±0.002
spec-ecc
0.225±0.010
0.423±0.006
0.462±0.008
bayes-beta
0.291±0.008
0.500±0.000
0.529±0.002
clf-pairs
0.320±0.016
0.506±0.000
0.560±0.008
clf-mean+gen
0.334±0.007
0.488±0.002
0.585±0.008
Appendix
Table 22: How well surface cues in the options recover the correct answer on multiple-choice items.
Chain-of-thought traces from large reasoning models can span tens of thousands of tokens, yet we lack a vocabulary for describing their internal structure. Previous methods developed to analyze chain-of-thought traces are either too rigid or not expressive enough, failing to capture features across domains and models. To remedy this, we develop ReasonOps, an unsupervised, expressive method for annotating chain-of-thought traces, providing succinct universal operators. Using ReasonOps, we analyze 44,662 traces from 12 thinking LLMs spanning 6 families across 8 reasoning benchmarks and discover that they share a common compositional structure: 7 recurring reasoning operators -- discourse-level moves such as backtracking, inferring, and hypothesizing -- that emerge from unsupervised clustering of sentence-initial 3-token pivots. These operators appear across every model family and benchmark domain, confirmed by three independent LLM judges who classify held-out samples at 70 -76% accuracy. We analyze the structure of operators on easy vs. hard problems, revealing that reflective operators are more helpful on hard problems and harm performance on easy problems. Operator sequences are highly model-identifying: a classifier trained on operator distributions alone recovers the source model with macro-AUC, revealing that each model family has a distinctive reasoning fingerprint. Structural operator features predict within-problem answer correctness well above baselines. Classifiers built on these operators reach WP-AUC and on AIME specifically. ReasonOps further enables early quality estimation well before the trace completes: we predict at WP-AUC for only 50% of the trace. The ReasonOps pipeline is unsupervised and annotation-free, enabling deep insights into LLM reasoning traces as well as strong downstream results on model identification and correctness prediction.
LLMs can execute every step of chain-of-thought reasoning correctly and still produce wrong final answers. We introduce the Novel Operator Test, a benchmark that separates operator logic from operator name, enabling rigorous distinction between genuine reasoning and pattern retrieval. By evaluating Boolean operators under unfamiliar names across depths 1-10 on five models (up to 8,100 problems each), we demonstrate a reasoning-output dissociation that existing benchmarks cannot detect. At Claude Sonnet 4's depth 7, all 31 errors have verifiably correct reasoning yet wrong declared answers; 17/19 errors in mixed-operator chains exhibit the same pattern. The benchmark reveals two failure types: strategy failures at depth 2, where models attempt terse retrieval (+62pp from scaffolding), and content failures at depth 7, where models reason fully but err systematically (+8-30pp, 0/300 errors post-intervention). A Trojan operator (XOR's truth table under a novel name) confirms name alone does not gate reasoning (p >= 0.49), while Llama's novelty gap widens to 28pp at depth 8-9 with the Trojan at 92-100%, isolating genuine difficulty with novel logic from name unfamiliarity.
Can large language model (LLM) agents deliver expert-level operations research (OR) solutions directly from original client materials, without prior expert reformulation? Existing OR benchmarks typically begin with self-contained descriptions or predefined optimization tasks, underrepresenting the translation of original business materials into professionally validated solutions. We introduce OR-License, an end-to-end benchmark of 31 instances across 10 industrial scenarios from paid OR engagements totaling USD 3.68 million. It preserves original requirements, input data, deliverable formats, and information boundaries after de-identification. To study information acquisition, we introduce ORSim, a controlled client simulator that discloses business facts without supplying OR expertise. Hidden evaluation checks feasibility and compares objective quality against original expert solutions accepted by clients before benchmark construction. Deployment-Adjusted Quality (DAQ) averages expert-relative quality across instances, assigning zero to infeasible deliveries. Across 11 frontier LLMs using coding-agent harnesses, several exceed the expert reference in conditional objective quality, but none matches its DAQ; the strongest primary Claude Code configuration achieves 88.39% strict feasibility and a DAQ of 0.87 against the expert reference of 1. Client information acquisition extends beyond clarification into solution development and validation, with some later stages showing higher confirmed issue-resolution proportions. Trajectory analysis identifies formulation and implementation failures, while task-equalized research-chain completion rates range from 24% to 45% across models on evaluable tasks. These findings motivate sustained information-gap resolution, faithful problem formulation, and evidence-linked self-improvement for reliable long-horizon agents.