ITOps observability products must integrate data from different tools and systems. We present ITOps Observability Studio, a multi-modal multi-agent system that orchestrates observability workflows through collaborative AI agents. Its three-tier architecture features inter-agent communication, specialized LLM-powered agents for data access, schema operations and visualization, and unified tool integration. The system supports natural language to GraphQL translation, GraphQL schema generation, schema mapping, trace data lifecycle management and multi-tool workflows for reliability engineering. Its agents are instrumented with agent analytics, so that their model and tool calls can be traced. We evaluate each agent on its own benchmark, scored against gold answers where they exist and by a model judge for schema mapping and chart quality. Query generation on a public benchmark and routing work well with a current open model, and the data lifecycle management agent solves most of its scenarios. The evaluation exposed defects that only appear when prompts, parsers and serving stacks are tested together, and we report them together with their fixes.
Figures & tables
Figure 1. Example workflow in ITOps Observability Block diagram showing the data flow between the UI chat agent, the coordination layer, and the tool-integration layer.
Figure 2. The three-tier reference architecture of ITOps Observability Studio, with the tools the agents call and the data sources behind them Layered diagram. Requests from an SRE and system alerts reach the UI-Orchestrator chat agent in the presentation tier, which talks over A2A to five LangGraph agents in the agent coordination tier, grouped into data access and data consumption agents. Below them, MCP servers in the tool integration tier connect to the APIs and databases of observability, automation and cost products.
Agent
Benchmark
n
Model
Metric
Result
Orchestrator (routing)
ITOps requests
351
Llama-3.3-70B
accuracy / macro-F1 (LLM)
0.943 / 0.827
GraphQL query
NL-to-GraphQL test
1,164
Llama-3.3-70B
schema-valid / exact (zero-shot)
85.7% / 58.4%
Schema generation
OpenAPI → GraphQL
3 products
Llama-3.3-70B
field F1 / type F1
0.67 / 0.27
Schema mapping
product schema pairs
2,439 mappings
LLM judge
joinable / equivalent
45.51% / 24.56%
Visualization
SRE queries
30
Llama-3.3-70B
VER / legality (gen., spec.)
0% / 97.8%, 62.3%
DLM (trace pruning)
constructed scenarios
30
Llama-3.3-70B
task accuracy
0.733
Table 1. Summary of the per-agent evaluation
Figure 3. Routing of the ITOps requests to the specialized agents, overall and by agent Bar charts of accuracy and macro-F1 for the rule, LLM and hybrid routers, and their recall on schema mapping, GraphQL query and visualization requests.
Figure 4. Natural language to GraphQL generation on the test split Bar chart of the share of generated queries that parse, validate against the schema and match the gold query, for two models with and without examples.
Measure
Current agent
Earlier output
Parses
0.87
0.42
Builds (strict / lenient)
0.28 / 0.87
0.00 / 0.19
Type P / R / F1
0.29 / 0.26 / 0.27
0.01 / 0.04 / 0.02
Field P / R / F1
0.63 / 0.72 / 0.67
0.20 / 0.17 / 0.18
Table 2. Generated schemas against the hand-written schemas
Figure 5. A sample interaction with the DLM agent Chat transcript. The user asks for the observability cost of the last 24 hours, and the agent reports total and observability cost with two threshold alerts. The user asks to evaluate the sampling methods, and the agent returns a table of four methods and recommends windowed stratified entropy-max sampling.
Figure 6. DLM agent accuracy and latency by query type, in the re-run and in the original system Bar charts of task accuracy, tool-sequence accuracy and latency for five query types, comparing the re-run with the original system.
Figure 7. Visualization agent performance across query types for both current models and the original system Bar charts of visualization error rate, legality and layout in percent, and judge scores for scale and ticks, readability and usability, for generic and specific queries.
Figure 8. Time-series visualization of application call volumes, indicating normal operational behavior Line chart of application call volume over time.
Enterprise multi-agent AI systems produce thousands of inter-agent interactions per hour, yet existing observability tools capture these dependencies without enforcing anything. OpenTelemetry and Langfuse collect telemetry but treat governance as a downstream analytics concern, not a real-time enforcement target. The result is an "observe-but-do-not-act" gap where policy violations are detected only after damage is done. We present Governance-Aware Agent Telemetry (GAAT), a reference architecture that closes the loop between telemetry collection and automated policy enforcement for multi-agent systems. GAAT introduces (1) a Governance Telemetry Schema (GTS) extending OpenTelemetry with governance attributes; (2) a real-time policy violation detection engine using OPA-compatible declarative rules under sub-200 ms latency; (3) a Governance Enforcement Bus (GEB) with graduated interventions; and (4) a Trusted Telemetry Plane with cryptographic provenance.
AI agents increasingly orchestrate long-running activities with users, tools, and system resources for days and weeks. To improve agent quality, safety, and cost efficiency, developers need to determine where failures happen, what triggers unsafe effects, and which tasks consume the most budget, then optimize those tasks. In systems software, profiling answers similar questions by aggregating resource consumption and attributing it to responsible code paths to identify hotspots. Yet existing agent observability tools focus on per-execution debugging and tracing rather than cross-run, long term profiling, making these questions difficult to answer at scale. Agent observability needs profiling, not only debugging, but profiling agents is challenging: the responsible entities are task intent like diagnose authentication, compare branches rather than code paths, and lack stable identifiers for aggregation. We propose a semantic operation stack model that adapts profiling to agent trajectories. Uniform operations represent all activities, and operation stacks replace the runtime call stack, enabling hierarchical attribution at different granularities. We observe that an agent's task occupies a contiguous span and decomposes into subtasks, so we introduce recursive operation segmentation, which recursively splits trajectories at task boundaries. AgentPProf is a profiler that aggregates agent trajectories into pprof-compatible profiles, enabling flame graph visualization and analysis. AgentPProf reaches 0.764 B3 F1 against human annotations on CodeTraceBench. On three problem-localization benchmarks, the profile raises MAP by up to 56%, demonstrating that it effectively attributes resources, locates problems, and helps optimize token cost at practical profiling cost. AgentPProf is available at https://github.com/eunomia-bpf/agentsight.
Modern distributed microservice deployments generate massive volumes of heterogeneous, multi-format observability logs across physically and logically partitioned infrastructure. Isolating root causes during cascading failures remains an arduous task for Site Reliability Engineers (SREs), bottlenecked by manual keyword filtering and disjointed temporal correlation across monitoring dashboards. We introduce Semantic Observability, an autonomous multi-agent framework that unifies disparate log streams into a dynamic, queryable Property Graph Database. By modeling physical infrastructure topologies, logical software dependencies, and chronological event sequences as attributed nodes and directed relations, it bridges the semantic gap inherent in flat, unstructured log repositories. Building on this unified substrate, we deploy a stateful multi-agent system orchestrated through Directed Cyclic Graphs (DCGs). The system autonomously navigates the topology, applying causal graph algorithms and temporal back-traversal to isolate the precise blast radius of anomalies. The pruned subgraph is then interpreted by a Large Language Model (LLM) reasoning engine to generate contextualized, code-level Root Cause Analyses (RCA) and actionable remediations. This paper establishes the foundational system architecture, formal relational ontology, and multi-agent control loops of Semantic Observability. We show conceptually how the framework eliminates ambient telemetry noise, reduces Mean Time to Resolution (MTTR), compresses LLM token consumption, and prevents structural hallucinations. We also specify an open-source validation framework on standard benchmarks (LogHub and Train-Ticket), laying the groundwork for a forthcoming companion paper on exhaustive quantitative evaluation and production benchmarking.