cs.LGNov 11, 2026

Orbit: An Evidence-Grounded Benchmark for Evaluating LLMs on IT Operations Reasoning

Authors: Sathwik Narkedimilli, Rakshit Verma, Rahul Soni, Ankit Kushwaha, Balaji Ganesan, Kaustabha Ray, Sunil Saumya, Pratibha Moogi

Organizations: National University of Singapore, Singapore · Indian Institute of Information Technology Dharwad, India · IBM Software, India · Indian Association for the Cultivation of Science, India

Abstract

LLMs now support IT operations (ITOps), but benchmarks rarely show why a model fails. We present ORBIT, a benchmark of ITOps questions in three formats, namely multiple-choice, open-ended, and troubleshooting, with each question accompanied by a reference answer and a recorded source. An audit found template defects in about a tenth of the generated items, so we drop them and rebuild the troubleshooting format from real practitioner threads with accepted answers. Under one independent judge, five LLMs answer most multiple-choice items correctly, between a third and just over half of the open-ended items, and a minority of the troubleshooting items, even under a more lenient root-cause rubric. When models judged their own answers, they accepted many answers to defective items, so we use one independent judge for every model. Word overlap with the question finds the correct option for more than half of the multiple-choice items. ORBIT is a diagnostic aid for selecting and supervising assistants rather than a deployment test.

Figures & tables

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. ReasonOps: Operator Segmentation for LLM Reasoning Traces

    May 28, 2026Daniel Lee, Owen Queen, James ZouLLM ReasoningReasoning Trace Analysis

  2. OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners

    Oct 16, 2026Peisong Wang, Feixue Liu, Yize Meng +16Operations ResearchLLM Agent Evaluation