Orbit: An Evidence-Grounded Benchmark for Evaluating LLMs on IT Operations Reasoning
Organizations: National University of Singapore, Singapore · Indian Institute of Information Technology Dharwad, India · IBM Software, India · Indian Association for the Cultivation of Science, India
Abstract
LLMs now support IT operations (ITOps), but benchmarks rarely show why a model fails. We present ORBIT, a benchmark of ITOps questions in three formats, namely multiple-choice, open-ended, and troubleshooting, with each question accompanied by a reference answer and a recorded source. An audit found template defects in about a tenth of the generated items, so we drop them and rebuild the troubleshooting format from real practitioner threads with accepted answers. Under one independent judge, five LLMs answer most multiple-choice items correctly, between a third and just over half of the open-ended items, and a minority of the troubleshooting items, even under a more lenient root-cause rubric. When models judged their own answers, they accepted many answers to defective items, so we use one independent judge for every model. Word overlap with the question finds the correct option for more than half of the multiple-choice items. ORBIT is a diagnostic aid for selecting and supervising assistants rather than a deployment test.
Figures & tables
| Benchmark | Set. | OG | ME | ET | RA |
|---|---|---|---|---|---|
| SWE-bench ( Jimenez et al., 2024 ) | E | ✗ | ✓ | ✓ | ✗ |
| ITBench ( Jha et al., 2025 ) | E | ✓ | ✓ | ✗ | ✗ |
| AIOpsLab ( Chen et al., 2025 ) | E | ✓ | ✓ | ✗ | ✗ |
| SREGym ( Clark et al., 2026 ) | E | ✓ | ✓ | ✗ | ✗ |
| Cloud-OpsBench ( Wang et al., 2026 ) | E | ✓ | ✓ | ✗ | ✗ |
| OpenRCA ( Xu et al., 2025 ) | T | ✓ | ✓ | ✗ | ✗ |
| Model | Overall | MCQ | OE | TS |
| ( ) | ( ) | ( ) | ( ) | |
| Llama 3.3 70B | 78.0 (76.5–79.5) | 88.1 (86.8–89.3) | 35.6 (30.4–41.2) | 16.8 (12.4–22.5) |
| Mistral Small 3.1 | 79.2 (77.7–80.6) | 88.4 (87.1–89.6) | 47.7 (42.0–53.3) | 13.5 (9.5–18.8) |
| Llama 4 Maverick | 77.8 (76.3–79.2) | 85.8 (84.4–87.1) | 45.3 (39.7–51.0) | 26.9 (21.4–33.3) |
| Granite 4.0 H | 74.1 (72.5–75.6) | 81.5 (79.9–82.9) | 54.7 (49.0–60.3) | 12.5 (8.7–17.7) |
| Mistral Medium | 80.3 (78.8–81.6) | 89.0 (87.7–90.2) | 50.7 (45.0–56.3) | 16.8 (12.4–22.5) |
| Model | Standard ( ) | Hard ( ) | |
|---|---|---|---|
| Llama 3.3 70B | 89.7 (88.4–90.8) | 59.7 (51.1–67.8) | 30.0 |
| Mistral Small 3.1 | 90.1 (88.8–91.2) | 56.6 (48.0–64.8) | 33.5 |
| Llama 4 Maverick | 87.3 (85.9–88.6) | 58.1 (49.5–66.3) | 29.2 |
| Granite 4.0 H | 83.1 (81.5–84.5) | 51.2 (42.6–59.6) | 31.9 |
| Mistral Medium | 90.6 (89.3–91.7) | 60.5 (51.8–68.5) | 30.1 |
| Multiple-choice | Generative | |||
| Model | Diagnostic | Other | OE multi-hop | TS |
| ( ) | ( ) | ( ) | ( ) | |
| Llama 3.3 70B | 59.1 | 93.0 | 41.3 | 16.8 |
| Mistral Small 3.1 | 60.8 | 93.0 | 54.5 | 13.5 |
| Llama 4 Maverick | 59.1 | 90.3 | 53.2 | 26.9 |
| Granite 4.0 H | 54.1 | 86.1 | 63.0 | 12.5 |
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Metric | Value | Unit |
|---|---|---|
| Question length | whitespace tokens | |
| Technical terms per question | count | |
| Answer length (option text for MCQ) | whitespace tokens | |
| Reference-solution length | words | |
| Evidence context length | tokens | |
| Grounding references per item | count |
| Model | Strict | Root cause |
|---|---|---|
| Llama 3.3 70B | 16.8 (12.4–22.5) | 28.8 (23.1–35.3) |
| Mistral Small 3.1 | 13.5 (9.5–18.8) | 21.2 (16.2–27.2) |
| Llama 4 Maverick | 26.9 (21.4–33.3) | 40.9 (34.4–47.7) |
| Granite 4.0 H | 12.5 (8.7–17.7) | 18.3 (13.6–24.1) |
| Mistral Medium | 16.8 (12.4–22.5) | 33.2 (27.1–39.8) |
| inst-101 | DP-00029 | DP-00041 | DP-00025 | |
|---|---|---|---|---|
| Category | Troubleshooting (MCQ) | Product Understanding | Schema Extraction | Documentation Navigation |
| Domain | Observability | APM (Instana) | AWS (Cloud Provider) | GCP (Cloud Provider) |
| Reasoning type | Abductive root cause | Semantic interpretation | Factual / schema retrieval | Semantic interpretation |
| Grounding source | Community Discussion | Product Documentation | API Reference | Product Documentation |
| Question | , PHP-FPM requests intermittently hang for 20–60 seconds after output generation. What monitoring-related root cause was identified? | Within the Instana documentation hierarchy, what functional purpose does .instana serve? | What is the complete field structure of AwsEc2Instance ? | Based on the documentation hierarchy, what type of entry is .gcp ? |
| Options | (A) UDP metric corruption, (B) TCP metric transmission contention, (C) OOM-killed collector, (D) Missing OTel correlation | (A) IBM Instana observability platform, (B) Deprecated parameter, (C) Monitoring latency field, (D) Authentication schema | (A) 3 fields ( instance_id , …), (B) 7 fields ( aws_resource , …), (C) 4 fields ( image_id , …), (D) 5 fields ( aws_resource , monitoring , private_ip , tenancy , virtual_machine ) | (A) Category page, (B) Deprecated parameter, (C) Monitoring component, (D) Google Cloud Platform infrastructure resources |
| Field | DP-02007 (troubleshooting format, multi-hop cohort, abductive root cause, hard) |
|---|---|
| Question | Given the following production scenario, perform root cause analysis. Scenario: A Kubernetes pod running a Java application is continuously OOMKilled. The node has 8GB of RAM. The pod has memory requests and limits set to 256Mi. The Java application is started with -Xmx512m -Xms128m . |
| Reference solution | The pod’s hard memory limit (256Mi) is lower than the JVM’s maximum heap size ( -Xmx512m ). Either increase the Kubernetes memory limit above 512Mi or reduce the JVM heap to fit within 256Mi. |
| Reasoning trace (excerpt) | (1) The node has ample RAM, but Kubernetes enforces memory at the container level through cgroups. (2) limits.memory of 256Mi is a hard ceiling, and exceeding it triggers the OOM killer. (3) The JVM is allowed to grow its heap to 512MB. (4) The configured heap exceeds the container limit, so the container is killed. |
| Comparison | (pts) | ||
| Format | |||
| Llama 3.3 70B, MC OE | 52.6 | 1.16 | |
| Llama 3.3 70B, MC TS | 71.3 | 1.59 | |
| Llama 3.3 70B, OE TS | 18.7 | 0.43 | |
| Mistral Small 3.1, MC OE | 40.7 | 0.92 | |
| Mistral Small 3.1, MC TS | 74.9 | 1.69 | |
| Comparison | (pts) | ||
|---|---|---|---|
| Model pairs, overall accuracy | |||
| L3.3 vs M-Small | 1.2 | 0.816 | 0.03 |
| L3.3 vs L4-Mav | 0.3 | 0.816 | 0.01 |
| L3.3 vs Granite | 4.0 | 0.002 | 0.09 |
| L3.3 vs M-Med | 2.2 | 0.169 | 0.05 |
| M-Small vs L4-Mav | 1.4 | 0.712 | 0.03 |
| Model | Retrieval | Semantic | Reasoning-heavy |
|---|---|---|---|
| ( ) | ( ) | ( ) | |
| Llama 3.3 70B | 97.0 (96.2–97.7) | 47.2 (39.2–55.3) | 57.4 (52.4–62.3) |
| Mistral Small 3.1 | 97.1 (96.2–97.7) | 49.3 (41.3–57.4) | 58.0 (53.0–62.8) |
| Llama 4 Maverick | 94.2 (93.1–95.2) | 43.1 (35.3–51.2) | 58.5 (53.5–63.3) |
| Granite 4.0 H | 90.4 (89.1–91.6) | 31.2 (24.2–39.2) | 53.8 (48.8–58.7) |
| Mistral Medium | 96.8 (95.9–97.5) | 50.7 (42.6–58.7) | 63.2 (58.2–67.9) |
| Open-ended | Troubleshooting | |||
| Model | Claude | GPT-5.5 | Claude | GPT-5.5 |
| Llama 3.3 70B | 35.6 | 64.1 | 16.8 | 26.0 |
| using Mistral Small 3.1 | 47.7 | 69.8 | 13.5 | 20.2 |
| Llama 4 Maverick | 45.3 | 67.4 | 26.9 | 38.9 |
| Granite 4.0 H | 54.7 | 74.8 | 12.5 | 16.3 |
| Mistral Medium | 50.7 | 75.8 | 16.8 | 27.4 |
| OE ( ) | TS ( ) | Defective ( ) | ||||
|---|---|---|---|---|---|---|
| Model | Self | Ind. | Self | Ind. | Self | Ind. |
| Mistral Medium | 51.5 | 33.1 | 36.8 | 9.8 | 7.3 | 4.2 |
| Mistral Small 3.1 | 46.4 | 31.6 | 24.8 | 11.3 | 10.8 | 5.2 |
| Llama 4 Maverick | 42.5 | 30.3 | 29.3 | 10.5 | 53.7 | 4.5 |
| Llama 3.3 70B | 50.9 | 23.7 | 39.9 | 10.5 | 56.8 | 3.5 |
| Granite 4.0 H | 42.5 | 34.8 | 34.6 | 8.3 | 74.6 | 3.5 |
| Model | Overall | MCQ | OE | TS |
| ( ) | ( ) | ( ) | ( ) | |
| Llama 3.3 70B | 78.82 (77.37–80.21) | 86.79 (85.41–88.06) | 50.94 (46.70–55.17) | 39.85 ( 31.93–48.34 ) |
| Mistral Small 3.1 | 78.70 (77.24–80.09) | 88.38 (87.07–89.58) | 46.43 (42.23–50.68) | 24.81 (18.25–32.79) |
| Llama 4 Maverick | 77.38 (75.89–78.80) | 87.31 (85.95–88.55) | 42.48 (38.35–46.72) | 29.32 (22.25–37.55) |
| Granite 4.0 H | 74.92 (73.38–76.40) | 83.92 (82.44–85.31) | 42.48 (38.35–46.72) | 34.59 (27.04–43.00) |
| Mistral Medium | 80.11 ( 78.69–81.46 ) | 88.46 ( 87.15–89.65 ) | 51.50 ( 47.26–55.72 ) | 36.84 (29.12–45.30) |
| Model | Standard ( ) | Hard ( ) | |
|---|---|---|---|
| Zero-shot | |||
| Llama 3.3 70B | 89.38 (88.09–90.54) | 51.22 (47.62–54.80) | 38.16 |
| Mistral Small 3.1 (24B) | 89.70 (88.44–90.85) | 46.49 (42.92–50.09) | 43.21 |
| Granite 4.0 H Small | 83.18 (81.65–84.62) | 41.89 (38.39–45.48) | 41.29 |
| Mistral Medium (2505) | 89.99 (88.74–91.12) | 52.43 (48.83–56.01) | 37.56 |
| Few-shot | |||
| Long-context | Multi-hop | Diagnostic | |
| Model | localization | causal inference | reasoning |
| ( ) | ( ) | ( ) | |
| Zero-shot | |||
| Llama 3.3 70B Instruct | 66.67 (30.00–90.32) | 73.12 (67.34–78.21) | 47.21 (43.48–50.96) |
| Mistral Small 3.1 (24B) | 66.67 (30.00–90.32) | 67.59 (61.60–73.06) | 42.65 (38.98–46.40) |
| Granite 4.0 H Small | 50.00 (18.76–81.24) | 52.57 (46.42–58.64) | 41.32 (37.68–45.06) |
| Metric | (1,2) | (2,3) | (3,1) | (1,2,3) |
| Weighted Cohen’s | 0.76 | 0.64 | 0.70 | – |
| Unweighted Cohen’s | 0.61 | 0.49 | 0.55 | – |
| ICC(2,1) | 0.78 | 0.65 | 0.72 | 0.72 |
| ICC(2, ) | 0.88 | 0.79 | 0.84 | 0.89 |
| Exact Agreement (%) | 76 | 65 | 70 | 58–60 |
| Within 1 Agreement (%) | 98 | 91 | 95 | 90 |
| Metric | C–D | C–G | D–G |
|---|---|---|---|
| Weighted Cohen’s | 0.65 | 0.80 | 0.78 |
| Unweighted Cohen’s | 0.36 | 0.54 | 0.53 |
| Exact Agreement (%) | 55.24 | 68.57 | 69.52 |
| Within 1 Agreement (%) | 94.29 | 100.00 | 100.00 |
| Spearman Correlation | 0.64 | 0.80 | 0.81 |
| Bland-Altman Bias | 0.10 | 0.03 | -0.08 |
| Metric | Claude | Gemini | DeepSeek | Ensemble |
|---|---|---|---|---|
| Weighted Cohen’s | 0.71 | 0.75 | 0.68 | 0.80 |
| Exact Agreement (%) | 71 | 75 | 68 | 81 |
| Within 1 Agreement (%) | 96 | 98 | 95 | 99 |
| ICC(2,1) | 0.73 | 0.77 | 0.70 | 0.82 |
| Metric | Value |
|---|---|
| Stack Overflow hit rate (overall) | 75.05% |
| Hit rate (non-documentation items) | 100.00% |
| Top-1 title match rate | 54.69% |
| Zero-hit items | 793 (24.95%) |
| Documentation-derived among zero-hit | 100.00% |
| Question Type | Hit Rate |
|---|---|
| Multiple Choice | 68.44% |
| Open-ended | 100.00% |
| Troubleshooting | 100.00% |
| Method | ATS | ACE | AUROC |
|---|---|---|---|
| arith-agg | |||
| spec-ecc | |||
| bayes-beta | |||
| clf-pairs | |||
| clf-mean+gen |