Organizations: Department of Medicine, Massachusetts General Hospital, Boston, MA, USA · Department of Neurology, Massachusetts General Hospital, Boston, MA, USA
Autonomous agentic workflows that iteratively refine their own behavior hold considerable promise, yet their failure modes remain poorly characterized. We investigate optimization instability, a phenomenon in which continued autonomous improvement paradoxically degrades classifier performance, using Pythia, an open-source framework for automated prompt optimization. Evaluating three clinical symptoms with varying prevalence (shortness of breath at 23%, chest pain at 12%, and Long COVID brain fog at 3%), we observed that validation sensitivity oscillated between 1.0 and 0.0 across iterations, with severity inversely proportional to class prevalence. At 3% prevalence, the system achieved 95% accuracy while detecting zero positive cases, a failure mode obscured by standard evaluation metrics. We evaluated two interventions: a guiding agent that actively redirected optimization, amplifying overfitting rather than correcting it, and a selector agent that retrospectively identified the best-performing iteration successfully prevented catastrophic failure. With selector agent oversight, the system outperformed expert-curated lexicons on brain fog detection by 331% (F1) and chest pain by 7%, despite requiring only a single natural language term as input. These findings characterize a critical failure mode of autonomous AI systems and demonstrate that retrospective selection outperforms active intervention for stabilization in low-prevalence classification tasks.
Figures & tables
Figure 1: Pythia multi-agent architecture. The optimization loop begins with a clinical note and prompt Pt passed to the Specialist Agent, which produces binary classification y^∈{0,1} . The system computes sensitivity ( σt ), specificity ( τt ), and F1 score. If the convergence criteria are met, the system outputs the final prompt P∗ . Otherwise, Error Analysis Agents (orange) generate natural language critiques for false negatives ( FN ) or false positives ( FP ), which Synthesis Agents (green) aggregate into a refined prompt Pt+1 . The loop iterates until convergence or reaches the maximum number of iterations.
Condition
Prevalence
Iter 1 F1
Iter 7 F1
Δ
Brain fog
3%
0.25
0.09
− 64%
Chest pain
12%
0.72
0.60
− 17%
SOB
23%
0.75
0.41
− 45%
Table 1: Validation performance degradation from first to final iteration. Continued autonomous optimization reduced F1 scores across all conditions, with severity inversely proportional to class prevalence.
Condition
Dev F1
Val F1
Gap
Brain fog
0.47
0.00
0.47
Chest pain
0.73
0.64
0.09
SOB
0.70
0.75
− 0.05
Table 2: Guiding agent performance. The intervention reduced oscillation on the development set but amplified overfitting, as evidenced by the gap between development and validation performance.
Condition
Prev.
Pythia F1
Lexicon F1
Δ
Brain fog
3%
0.25
0.058
+331%
Chest pain
12%
0.72
0.67
+7%
SOB
23%
0.75
0.79
− 5%
Table 3: Selector agent compared to expert curated lexicon baselines on the validation set. Pythia outperformed lexicons on brain fog (+331%) and chest pain (+7%) while achieving comparable performance on SOB ( − 5%).
Condition
Selected
Val F1
Optimal
Val F1
Brain fog
Iter 1
0.25
Iter 2
0.44
Chest pain
Iter 1
0.72
Iter 1
0.72
SOB
Iter 1
0.75
Iter 6
0.76
Table 4: Selected versus optimal iteration on the validation set. Development set performance did not reliably predict validation generalization, particularly at low prevalence.
Figure 2: Optimization dynamics and comparative performance. (A) Validation sensitivity across seven optimization iterations. Brain fog (3% prevalence) exhibited catastrophic collapse, reaching zero sensitivity at iterations 3 and 6 (marked with X). Higher prevalence conditions showed oscillation but avoided complete failure. (B) Validation F1 scores comparing Pythia with selector agent versus expert curated lexicon baselines. Pythia outperformed lexicons on brain fog (+331%) and chest pain (+7%), while achieving comparable performance on SOB ( − 5%), despite requiring only a single natural-language term as input.
Clinical notes contain many of the signs and symptoms that bring patients to care, yet this information rarely reaches structured fields. Existing extraction approaches either rely on context-insensitive rules that generate false positives or on supervised models that require substantial fine-tuning. We present Pythia, a multi-agent system that autonomously writes and optimizes extraction prompts for clinical concepts without manual prompt engineering or fine-tuning. Running on a locally hosted open-weights model, Pythia keeps clinical notes on local infrastructure and selects prompts using development-set sensitivity and specificity. We compared Pythia with a curated lexicon across 72 signs and symptoms from 400 clinical notes representing 387 patients. Development (n=300) and validation (n=100) sets were partitioned independently for each concept. Pythia achieved mean sensitivity of 0.76 and specificity of 0.95, compared with 0.82 and 0.76 for the lexicon, and matched or exceeded the lexicon on both metrics for 20 of 62 directly comparable concepts. For 14 concepts where the lexicon labeled every note positive, Pythia recovered mean specificity of 0.97 by requiring a present-tense, patient-attributed finding rather than any textual mention of a term. Specificity transferred from development to validation with minimal degradation across prevalences, whereas sensitivity transfer weakened below 5% prevalence, reaching a mean gap of 0.25 below 2% prevalence. A BERT classifier fine-tuned per concept on the same development set achieved mean sensitivity of 0.23 and collapsed to zero sensitivity for concepts below roughly 5% prevalence. These findings suggest that autonomous, fine-tuning-free prompt optimization can produce symptom extraction prompts that generalize effectively from development to validation while remaining deployable on local infrastructure.
Cameron Cagan, Pedram Fard, Jiazi Tian +3
Department of Medicine, Massachusetts General Hospital, Boston, MA, USA · Department of Biomedical Informatics and Medical Education, University of Washington, WA, USA
Agentic AI systems are increasingly being integrated into scientific workflows, yet their behavior under realistic conditions remains insufficiently understood. We evaluate CMBAgent across two workflow paradigms and eighteen astrophysical tasks. In the One-Shot setting, access to domain-specific context yields an approximately ~6x performance improvement (0.85 vs. ~0 without context), with the primary failure mode being silent incorrect computation - syntactically valid code that produces plausible but inaccurate results. In the Deep Research setting, the system frequently exhibits silent failures across stress tests, producing physically inconsistent posteriors without self-diagnosis. Overall, performance is strong on well-specified tasks but degrades on problems designed to probe reasoning limits, often without visible error signals. These findings highlight that the most concerning failure mode in agentic scientific workflows is not overt failure, but confident generation of incorrect results. We release our evaluation framework to facilitate systematic reliability analysis of scientific AI agents.
Shivam Rawat, Lucie Flek
Bonn-Aachen International Center for Information Technology, University of Bonn, Germany; Lamarr Institute for Machine Learning and Artificial Intelligence, Germany.
We introduce a comprehensive validation framework for LLM-based agentic systems that provides systematic diagnosis and improvement of reliability failures. The framework includes fifteen failure-detection tools and two root-cause analysis modules that jointly uncover weaknesses across input handling, prompt design, and output generation. It integrates lightweight rule-based checks with LLM-as-a-judge assessments to support structured incident detection, classification, and repair. We applied the framework to IBM CUGA, evaluating its performance on the AppWorld and WebArena benchmarks. The analysis revealed recurrent planner misalignments, schema violations, brittle prompt dependencies, and more. Based on these insights, we refined both prompting and coding strategies, maintaining CUGA's benchmark results while enabling mid-sized models such as Llama 4 and Mistral Medium to achieve notable accuracy gains, substantially narrowing the gap with frontier models. Beyond quantitative validation, we conducted an exploratory study that fed the framework's diagnostic outputs and agent description into an LLM for self-reflection and prioritization. This interactive analysis produced actionable insights on recurring failure patterns and focus areas for improvement, demonstrating how validation itself can evolve into an agentic, dialogue-driven process. These results show a path toward scalable, quality assurance, and adaptive validation in production agentic systems, offering a foundation for more robust, interpretable, and self-improving agentic architectures.