Towards an AI Software Factory for Data Systems

Anna Pavlenko; Bogdan Crivat; Brandon Haynes; Carlo Curino; Fotis Psallidas; Jaro Slawinski; Johannes Freischuetz; Laura Pereira Sanchez

cs.AISep 28, 2026

Towards an AI Software Factory for Data Systems

Authors: Anna Pavlenko, Bogdan Crivat, Brandon Haynes, Carlo Curino, Fotis Psallidas, Jaro Slawinski, Johannes Freischuetz, Laura Pereira Sanchez, +18 more

Organizations: Microsoft · University of Wisconsin-Madison · GitHub

Abstract

AI-assisted coding tools deliver significant acceleration of coding, but only limited impact across the end-to-end software development lifecycle (SDLC)--an Amdahl's law effect! In this paper, we discuss our progress towards building an AI SW Factory that accelerates all the stages of SDLC-Targeting, Coding, Reviewing, and Ops. The AI SW Factory produces a metadata exhaust that enables self-improvement by fine-tuning model weights and updating our World Model (a rich data substrate). We focus on Data Systems and the important class of Evolutionary Coding Tasks (i.e., those with a measurable objective to hill-climb) and report on 1) scaled deployments at Microsoft (tens of repositories) leading to 3x engineering efficiency above agentic coding and up to 22x token efficiency, and 2) several open challenges.

Figures & tables

Figure 1. A vision for an AI-enhanced software lifecycle.

Figure 2. World model architecture.

Figure 3. Retrieval context needed for one decision

Codebase	Code Exp.	Darwin Exp.	Gains	Inv.
Dataverse	High	None	2 $\times$ throughput 4 $\times$ mem reduction	2–4 h
Fabric Data Agent	High	Low	10% accuracy boost	3 h
Fabric Data Warehouse	Low	High	2.6 $\times$ GPU kernel speedup	4 h
Xbox	None	High	2 $\times$ code gen speed-up	16 h
Copilot CLI	None	High	9% token reduction 12% reduced cost	40 h
NL2DAX	None	High	63% prompt size reduction 7% accuracy boost	4 h

Table 1. Darwin Anecdotal Evidence

Figure 4. Darwin improvements of OSS projects.

Figure 5. Production Evidence for Autoperf.

Explore similar work

SWE-chat: Coding Agent Interactions From Real Users in the Wild
Apr 22, 2026Joachim Baumann, Vishakh Padmakumar, Xiang Li +3Coding Agents Code Generation
Stanford University
MirrorCode: AI can rebuild entire programs from behavior alone
Jun 29, 2026Tom Adamczewski, David Owen, David Rein +4Coding Agents Agent Behavior Modeling
1Epoch AI · 2METR · 3Prime Intellect +2
Socratic-SWE: Self-Evolving Coding Agents via Trace-Derived Agent Skills
Jun 5, 2026Chuan Xiao, Zhengbo Jiao, Shaobo Wang +5Self-Evolving Agents Coding Agents
2Shanghai Jiao Tong University · 1AI Data, Alibaba Group

Apr 22, 2026, cs.AIJ/K move · Enter open · S save

SWE-chat: Coding Agent Interactions From Real Users in the Wild

Joachim Baumann, Vishakh Padmakumar, Xiang Li, John Yang+2

Stanford University

Coding Agents Code Generation Real-World Dataset

Open paper AskPDF arXiv

AI coding agents are being adopted at scale, yet we lack empirical evidence on how people actually use them and how much of their output is useful in practice. We present SWE-chat, the first large-scale dataset of real coding agent sessions collected from open-source developers in the wild. The dataset currently contains almost 18,000 sessions, comprising more than 229,000 user prompts and 2 million agent tool calls. SWE-chat is a living dataset; our collection pipeline automatically and continually discovers and processes sessions from public repositories. Leveraging SWE-chat, we provide an initial empirical characterization of real-world coding agent usage and failure modes. We find that coding patterns are bimodal: in 41% of sessions, agents author virtually all committed code ("vibe coding"), while in 25%, humans write all code themselves. Despite rapidly improving capabilities, coding agents remain inefficient in natural settings. Only 59% of all agent-produced code survives into user commits, and agent-written code introduces more security vulnerabilities than code authored by humans. Furthermore, users push back against agent outputs - through corrections, failure reports, and interruptions - in 50% of all turns. By capturing complete interaction traces with human vs. agent code authorship attribution, SWE-chat provides an empirical foundation for moving beyond curated benchmarks towards an evidence-based understanding of how AI agents perform in real developer workflows.

Abstract

Explore similar work

SWE-chat: Coding Agent Interactions From Real Users in the Wild

MirrorCode: AI can rebuild entire programs from behavior alone

Socratic-SWE: Self-Evolving Coding Agents via Trace-Derived Agent Skills