cs.DBOct 30, 2026

SpBench: An Agent-Native Framework for Cross-Engine Lakehouse Benchmarking

Authors: Lyndon Shi, Nic Crouch, Yi Hu, Zihao Ye, Choden Konigsmark, Eric Vandenberg, Dionysios Logothetis, Kuba Niemiec, 

Organizations: Snowflake Inc.

Abstract

SpBench is an agent-native framework for cross-engine lakehouse benchmarking, in production use for benchmarking Snowpark, Snowpark Connect for Apache Spark, and other execution environments across standardized suites -- such as TPCx-BB and TPC-DI -- and custom benchmarks distilled from production workload patterns. It runs continuously across a broad portfolio of workloads, guiding the performance roadmap and detecting regressions for production-scale data engines. Benchmarking has historically been an expensive investment requiring continuous cross-functional collaboration and domain expertise, making it difficult to obtain actionable performance insights. SpBench provides (i) a typed, code-generated benchmark substrate that compiles platform-agnostic benchmark definitions into fair, comparable platform-specific implementations; (ii) a structured-telemetry contract recording every run as queryable fact and dimension data, letting LLM agents query results deterministically; and (iii) an agent-native harness of three composable skills to expedite building benchmarks and analyzing results. This agent-native architecture enabled the rapid development of more than 400 benchmark queries that run on a regular cadence to detect performance regressions; catching these has directly led to improvements in production.

Figures & tables

Explore similar work

CardsList
  1. DataClawEval: A Benchmark for Data Engineering Agents in Real Industrial Harness

    Jul 30, 2026Debin Meng, Jiaming Yang, Zefang Zong +4LLM Agent EvaluationAutomated Evaluation

  2. Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows

    Oct 1, 2026Gabriel Tomitsuka, Arman Raayatsanati, Emma Xing +2Enterprise AnalyticsAI Agent Benchmarks