cs.AIMay 26, 2026

Discovery Agents for Real-Time Analytics: Toward Proactive Insight Systems

Authors: Gaetano RossielloDharmashankar Subramanian

Organizations: IBM · New York City, New York, USA · Yorktown Heights, New York, USA

Abstract

Modern analytics systems are fundamentally reactive, requiring users to define queries over increasingly complex and continuously evolving data. In real-time streaming environments, this paradigm breaks down, as the space of potential insights becomes too large to enumerate manually. We present a multi-agent architecture for autonomous insight discovery over real-time data streams. The system implements a continuous discovery loop in which agents generate hypotheses, compile them into executable analytics, validate generated artifacts, and produce visualizations and deployable applications. The architecture leverages Apache Kafka for event-driven coordination, Apache Flink for stream processing, and large language models to implement specialized agents. A key contribution is a contract-driven design based on typed intermediate artifacts, enabling modularity, observability, lineage, and safer execution of dynamically generated analytics. Through use cases in retail, finance, and public data, we show how this architecture supports a shift from query-driven analytics to proactive, discovery-driven systems.

Explore similar work

CardsList
  1. Data Agents: Agentic Data Systems

    Sep 21, 2026Guoliang Li, Peiyao Zhou, Xuanhe Zhou +3Agentic SystemsData Lakes