cs.AINov 1, 2026

EMBERS: Empirically Motivated Robustness Testing of Deep Research Systems via Defect-Guided Fuzzing

Authors: Shuochuan Li, Zan Wang, Dong Wang, Yuhang Jiang, Huajin Cui, Chen Zhang, Gang Wang, Junjie Chen

Organizations: Tianjin University, China · Huawei Technologies Co., Ltd., China

Abstract

Deep Research (DR) systems extend large language models into autonomous agents that decompose complex information needs, iteratively search the web, and synthesize evidence to answer challenging questions. However, their multi-stage execution may introduce robustness risks, as variations in user questions or intermediate search inputs may propagate across research stages and lead to incorrect answers. Existing evaluations focus on task-solving capabilities or robustness against external evidence disturbances, leaving intrinsic weaknesses in DR execution insufficiently explored. Thus, we conduct an empirical study of 1,932 execution trajectories from three representative DR systems and identify nine recurring defect categories across subquestion formulation, online search, and final reasoning. Guided by these findings, we propose EMBERS, an empirically motivated, defect-guided fuzzing framework for robustness testing of DR systems. Starting from correctly answered seed questions, EMBERS generates stage-targeted test inputs using defect-guided mutation, web-search-assisted fidelity assurance, and diversity-aware seed maintenance. Experiments across 16 framework--model--dataset settings show that EMBERS achieves 3.21 times the failure-discovery throughput of the strongest baseline under identical testing budgets. Human and LLM-assisted evaluations demonstrate that most generated inputs preserve the original information need and necessary contextual constraints. Moreover, model fine-tuning experiments show consistent-answer performance improvement, highlighting their value for robustness testing and model improvement.

Figures & tables

Explore similar work

CardsList
  1. DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports

    Jan 13, 2026Ruizhe Li, Mingxuan Du, Benfeng Xu +3Deep Research AgentsLLM Agent Evaluation

  2. From Intent to Evidence: A Categorical Approach for Structural Evaluation of Deep Research Agents

    Mar 26, 2026Shuoling Liu, Zhiquan Tan, Kun Yi +6Category TheoryDeep Research Agents

  3. TestifAI: Tomography-Based Testing for Deep Learning Systems

    Aug 19, 2026Arooj Arif, Tobias Hartung, Elena Botoeva +1DNN TestingNeural Network Robustness