Deep research agents (DRA) are commonly used for open-ended synthesis tasks where closed-world QA benchmarks are poorly suited to characterize relevant aspects of performance. We study the self-consistency of one commercial deep research system under repeated identical prompting, representing each report with the set of named entities it mentions, categorised by a custom schema, and the set of citations it retrieves. Across repeated runs of a single open-ended cybersecurity related prompt, we quantify pairwise overlap to characterize how stable the retrieved and synthesized output structure is under repeated identical prompts. In ten runs, the pairwise entity Jaccard results in a median of 0.12 and the pairwise citation Jaccard is below 0.02, with discovery curves that do not visibly saturate. The magnitude and structure of (dis)agreement between entity sets varies by entity class, which suggests that self-consistency in a DRA is better understood as a function of the underlying entity space than as a stable property of the system.
Figures & tables
Figure 1: Analysis pipeline . We query Google Deep Research ten times with the same cybersecurity-related prompt and treat each generated report as one run. For each run, we extract entities and citation URLs using langextract, canonicalize entity strings by class using a three-way voting procedure with gabriel, and compute pairwise Jaccard similarities and discovery curves over the resulting canonical entity and citation sets.
Figure 2: Pairwise Jaccard similarity across ten repeated runs. Each point represents one unordered pair of reports, for 45 run pairs in total. Rows show Jaccard similarity over canonical entities within each class, resolved citation URLs, and the overall canonical entity set. Boxplots summarize the distribution across run pairs. Overall entity overlap is low, with median Jaccard approximately 0.12, while citation overlap is much lower, with median citation Jaccard just over 0.01. Entity–citation edge overlap is omitted from the plot because its median Jaccard is similarly near zero and visually indistinguishable from the citation row.
Figure 3: Discovery curves for canonical entity coverage across repeated runs. For each entity class, we randomly order the ten reports and compute the cumulative proportion of distinct canonical entities observed after k runs. Curves that flatten quickly would indicate convergence towards a stable observed entity set; the mostly near-linear curves indicate that additional runs continue to surface previously unseen entities.