Automating Feature Flag Cleanup at Scale with Multi-Agent LLM Systems: An Industrial Experience at DoorDash

Atharv Chandratre; Jai Datar

cs.SEOct 22, 2026

Automating Feature Flag Cleanup at Scale with Multi-Agent LLM Systems: An Industrial Experience at DoorDash

Authors: Atharv Chandratre, Jai Datar

Organizations: DoorDash, Inc., San Francisco, CA, USA

Abstract

Feature flags are runtime configuration mechanisms used for gradual feature rollouts, experimentation, and kill switches. Once a feature is fully rolled out, the flag and its gated code become technical debt and can introduce operational risk through inadvertently enabling dormant code paths. At DoorDash, over 56,000 feature flags span 623 repositories, with 1,659 currently stale as of May 2026, with internal estimates of one to two hours per manual cleanup. Existing rule-based cleanup tools proved insufficient in DoorDash's codebases. Our contribution is an end-to-end automated LLM-based feature flag cleanup system that integrates querying live feature flag rollout data using MCP, Jira ticket processing, and end-to-end pull request creation. Our system successfully produced engineer-approved flag removal PRs for 45 (90%) of the 50 stale flags it was evaluated on, proving the system's operational viability at scale with each cleanup averaging only $4.79 and 13.8 minutes. We evaluated the system's quality and reliability boundary based on success rates, cost per cleanup and time metrics. Furthermore, we discuss our lessons learned when architecting this system, to provide practical guidance for developing similar multi-agent systems.

Figures & tables

Fig. 1: System architecture. The orchestrator agent (dashed rectangle) manages the cleanup lifecycle in two phases. In Phase 1, it retrieves stale DVs from Jira, queries the experimentation platform for rollout state, and searches the codebase for all references. The resulting DV report is reviewed by an engineer who confirms target values. In Phase 2, the orchestrator spawns parallel cleanup agents in isolated git worktrees, each editing code, running tests, verifying coverage, and linting before creating a pull request.

Metric	Value
DV cleanups analyzed	50
Accepted (first-shot)	31 (62%)
Accepted with revision	14 (28%)
Failed	5 (10%)
Overall cleanup rate	45/50 (90%)
CI first-pass rate	44/50 (88%)

TABLE I: Evaluation results on 50 stale DVs.

Panel of Figure 3 — Fig. 2: Cleanup outcomes by complexity level. Simple DVs achieve 100% first-shot acceptance; complex DVs show higher revision and failure rates but still achieve 85% overall cleanup.

Complexity	$n$	Avg. time	Med. time	Avg. cost	Med. cost
Simple	6	7.5 min	8.2 min	$2.69	$2.50
Medium	18	10.4 min	9.9 min	$3.46	$3.42
Complex	26	17.7 min	14.4 min	$6.20	$4.50

TABLE II: Time and cost by DV complexity.

Panel of Figure 5 — Fig. 3: Agent cost vs. wall-clock time for each DV, colored by complexity. Failed DVs are marked with $\times$ . Simple and medium DVs cluster in the low-cost region; complex DVs show higher variance.

Automated Software Engineering

Automated Software Engineering uses AI, search, and verification methods to automate tasks across the software development lifecycle.

See all 82 papers

Memory-Augmented LLM-based Multi-Agent System for Automated Feature Generation on Tabular Data
Apr 22, 2026Fengxian Dong, Zhi Zheng, Xiao Han +5Tabular ML Multi-Agent LLMs
University of Science and Technology of China · Zhejiang University of Technology · Meituan
SIGMA: SHAP-Guided Implicit-Trajectory Generation for Metadata-Free LLM-Based AutoFE
Aug 18, 2026Xuan Zheng, Kento Uchida, Shinichi ShirakawaLarge Language Model-Based Feature Engineering Automated Feature Engineering
Yokohama National University
Intelligent Elastic Feature Fading: Enabling Model Retrain-Free Feature Efficiency Rollouts at Scale
May 1, 2026Jieming Di, Xiaoyu Chen, Ying She +21Efficient Inference Cost-Aware Inference
Meta · USA

Apr 22, 2026J/K move · Enter open · S save

Memory-Augmented LLM-based Multi-Agent System for Automated Feature Generation on Tabular Data

Fengxian Dong, Zhi Zheng, Xiao Han, Wei Chen+4

University of Science and Technology of China · Zhejiang University of Technology · Meituan

Tabular ML Multi-Agent LLMs Automated Feature Engineering Memory-Augmented Language Model Agents Large Language Model-Based Feature Engineering

Open paper AskPDF arXiv

Automated feature generation extracts informative features from raw tabular data without manual intervention and is crucial for accurate, generalizable machine learning. Traditional methods rely on predefined operator libraries and cannot leverage task semantics, limiting their ability to produce diverse, high-value features for complex tasks. Recent Large Language Model (LLM)-based approaches introduce richer semantic signals, but still suffer from a restricted feature space due to fixed generation patterns and from the absence of feedback from the learning objective. To address these challenges, we propose a Memory-Augmented LLM-based Multi-Agent System (MALMAS) for automated feature generation. MALMAS decomposes the generation process into agents with distinct responsibilities, and a Router Agent activates an appropriate subset of agents per iteration, further broadening exploration of the feature space. We further integrate a memory module comprising procedural memory, feedback memory, and conceptual memory, enabling iterative refinement that adaptively guides subsequent feature generation and improves feature quality and diversity. Extensive experiments on multiple public datasets against state-of-the-art baselines demonstrate the effectiveness of our approach. The code is available at https://github.com/fxdong24/MALMAS

Abstract

Automated Software Engineering

Explore similar work

Memory-Augmented LLM-based Multi-Agent System for Automated Feature Generation on Tabular Data

SIGMA: SHAP-Guided Implicit-Trajectory Generation for Metadata-Free LLM-Based AutoFE

Intelligent Elastic Feature Fading: Enabling Model Retrain-Free Feature Efficiency Rollouts at Scale