Autonomous Multi-modal Agents for ITOps Observability
Organizations: IBM Software, India
Abstract
ITOps observability products must integrate data from different tools and systems. We present ITOps Observability Studio, a multi-modal multi-agent system that orchestrates observability workflows through collaborative AI agents. Its three-tier architecture features inter-agent communication, specialized LLM-powered agents for data access, schema operations and visualization, and unified tool integration. The system supports natural language to GraphQL translation, GraphQL schema generation, schema mapping, trace data lifecycle management and multi-tool workflows for reliability engineering. Its agents are instrumented with agent analytics, so that their model and tool calls can be traced. We evaluate each agent on its own benchmark, scored against gold answers where they exist and by a model judge for schema mapping and chart quality. Query generation on a public benchmark and routing work well with a current open model, and the data lifecycle management agent solves most of its scenarios. The evaluation exposed defects that only appear when prompts, parsers and serving stacks are tested together, and we report them together with their fixes.
Figures & tables
| Agent | Benchmark | Model | Metric | Result | |
| Orchestrator (routing) | ITOps requests | 351 | Llama-3.3-70B | accuracy / macro-F1 (LLM) | 0.943 / 0.827 |
| GraphQL query | NL-to-GraphQL test | 1,164 | Llama-3.3-70B | schema-valid / exact (zero-shot) | 85.7% / 58.4% |
| Schema generation | OpenAPI GraphQL | 3 products | Llama-3.3-70B | field F1 / type F1 | 0.67 / 0.27 |
| Schema mapping | product schema pairs | 2,439 mappings | LLM judge | joinable / equivalent | 45.51% / 24.56% |
| Visualization | SRE queries | 30 | Llama-3.3-70B | VER / legality (gen., spec.) | 0% / 97.8%, 62.3% |
| DLM (trace pruning) | constructed scenarios | 30 | Llama-3.3-70B | task accuracy | 0.733 |
| Measure | Current agent | Earlier output |
| Parses | 0.87 | 0.42 |
| Builds (strict / lenient) | 0.28 / 0.87 | 0.00 / 0.19 |
| Type P / R / F1 | 0.29 / 0.26 / 0.27 | 0.01 / 0.04 / 0.02 |
| Field P / R / F1 | 0.63 / 0.72 / 0.67 | 0.20 / 0.17 / 0.18 |
Multimodal Agents
Multimodal agents perceive and reason across text, images, audio, and video to use tools, interact with interfaces, and act in the world.
See all 414 papers