Optimization Instability in Autonomous Agentic Workflows for Clinical Symptom Detection

Cameron Cagan; Pedram Fard; Jiazi Tian; Jingya Cheng; Shawn N. Murphy; Hossein Estiri

cs.AIFeb 17, 2026

Optimization Instability in Autonomous Agentic Workflows for Clinical Symptom Detection

Authors: Cameron Cagan, Pedram Fard, Jiazi Tian, Jingya Cheng, Shawn N. Murphy, Hossein Estiri

Organizations: Department of Medicine, Massachusetts General Hospital, Boston, MA, USA · Department of Neurology, Massachusetts General Hospital, Boston, MA, USA

Abstract

Autonomous agentic workflows that iteratively refine their own behavior hold considerable promise, yet their failure modes remain poorly characterized. We investigate optimization instability, a phenomenon in which continued autonomous improvement paradoxically degrades classifier performance, using Pythia, an open-source framework for automated prompt optimization. Evaluating three clinical symptoms with varying prevalence (shortness of breath at 23%, chest pain at 12%, and Long COVID brain fog at 3%), we observed that validation sensitivity oscillated between 1.0 and 0.0 across iterations, with severity inversely proportional to class prevalence. At 3% prevalence, the system achieved 95% accuracy while detecting zero positive cases, a failure mode obscured by standard evaluation metrics. We evaluated two interventions: a guiding agent that actively redirected optimization, amplifying overfitting rather than correcting it, and a selector agent that retrospectively identified the best-performing iteration successfully prevented catastrophic failure. With selector agent oversight, the system outperformed expert-curated lexicons on brain fog detection by 331% (F1) and chest pain by 7%, despite requiring only a single natural language term as input. These findings characterize a critical failure mode of autonomous AI systems and demonstrate that retrospective selection outperforms active intervention for stabilization in low-prevalence classification tasks.

Figures & tables

Figure 1: Pythia multi-agent architecture. The optimization loop begins with a clinical note and prompt $P_{t}$ passed to the Specialist Agent, which produces binary classification $\hat{y}\in\{0,1\}$ . The system computes sensitivity ( $\sigma_{t}$ ), specificity ( $\tau_{t}$ ), and F1 score. If the convergence criteria are met, the system outputs the final prompt $P^{*}$ . Otherwise, Error Analysis Agents (orange) generate natural language critiques for false negatives ( $\mathcal{FN}$ ) or false positives ( $\mathcal{FP}$ ), which Synthesis Agents (green) aggregate into a refined prompt $P_{t+1}$ . The loop iterates until convergence or reaches the maximum number of iterations.

Condition	Prevalence	Iter 1 F1	Iter 7 F1	$\Delta$
Brain fog	3%	0.25	0.09	$-$ 64%
Chest pain	12%	0.72	0.60	$-$ 17%
SOB	23%	0.75	0.41	$-$ 45%

Table 1: Validation performance degradation from first to final iteration. Continued autonomous optimization reduced F1 scores across all conditions, with severity inversely proportional to class prevalence.

Condition	Dev F1	Val F1	Gap
Brain fog	0.47	0.00	0.47
Chest pain	0.73	0.64	0.09
SOB	0.70	0.75	$-$ 0.05

Table 2: Guiding agent performance. The intervention reduced oscillation on the development set but amplified overfitting, as evidenced by the gap between development and validation performance.

Condition	Prev.	Pythia F1	Lexicon F1	$\Delta$
Brain fog	3%	0.25	0.058	+331%
Chest pain	12%	0.72	0.67	+7%
SOB	23%	0.75	0.79	$-$ 5%

Table 3: Selector agent compared to expert curated lexicon baselines on the validation set. Pythia outperformed lexicons on brain fog (+331%) and chest pain (+7%) while achieving comparable performance on SOB (

-

5%).

Condition	Selected	Val F1	Optimal	Val F1
Brain fog	Iter 1	0.25	Iter 2	0.44
Chest pain	Iter 1	0.72	Iter 1	0.72
SOB	Iter 1	0.75	Iter 6	0.76

Table 4: Selected versus optimal iteration on the validation set. Development set performance did not reliably predict validation generalization, particularly at low prevalence.

Figure 2: Optimization dynamics and comparative performance. (A) Validation sensitivity across seven optimization iterations. Brain fog (3% prevalence) exhibited catastrophic collapse, reaching zero sensitivity at iterations 3 and 6 (marked with X). Higher prevalence conditions showed oscillation but avoided complete failure. (B) Validation F1 scores comparing Pythia with selector agent versus expert curated lexicon baselines. Pythia outperformed lexicons on brain fog (+331%) and chest pain (+7%), while achieving comparable performance on SOB ( $-$ 5%), despite requiring only a single natural-language term as input.

LLM Agent Workflow Optimization

LLM agent workflow optimization studies how to design and run multi-step agents for better task success, reliability, latency, and cost.

See all 256 papers

A Multi-Agent System for Autonomous, Fine-Tuning-Free Clinical Symptom Detection: Development and Validation Study
Jul 14, 2026Cameron Cagan, Pedram Fard, Jiazi Tian +3Multi-Agent LLM Systems Medical Diagnosis
Department of Medicine, Massachusetts General Hospital, Boston, MA, USA · Department of Biomedical Informatics and Medical Education, University of Washington, WA, USA
Plausible but Wrong: A case study on Agentic Failures in Astrophysical Workflows
Apr 28, 2026Shivam Rawat, Lucie FlekAgent Failure Analysis AI for Science
Bonn-Aachen International Center for Information Technology, University of Bonn, Germany; Lamarr Institute for Machine Learning and Artificial Intelligence, Germany.
AgentFixer: From Failure Detection to Fix Recommendations in LLM Agentic Systems
Feb 18, 2026Hadar Mulian, Sergey Zeltyn, Ido Levy +3LLM Agent Reliability LLM Agents
IBM Research, Israel

Jul 14, 2026J/K move · Enter open · S save

A Multi-Agent System for Autonomous, Fine-Tuning-Free Clinical Symptom Detection: Development and Validation Study

Cameron Cagan, Pedram Fard, Jiazi Tian, Jingya Cheng+2

Department of Medicine, Massachusetts General Hospital, Boston, MA, USA · Department of Biomedical Informatics and Medical Education, University of Washington, WA, USA

Multi-Agent LLM Systems Medical Diagnosis Clinical Information Extraction Automatic Prompt Optimization Clinical NLP

Open paper AskPDF arXiv

Clinical notes contain many of the signs and symptoms that bring patients to care, yet this information rarely reaches structured fields. Existing extraction approaches either rely on context-insensitive rules that generate false positives or on supervised models that require substantial fine-tuning. We present Pythia, a multi-agent system that autonomously writes and optimizes extraction prompts for clinical concepts without manual prompt engineering or fine-tuning. Running on a locally hosted open-weights model, Pythia keeps clinical notes on local infrastructure and selects prompts using development-set sensitivity and specificity. We compared Pythia with a curated lexicon across 72 signs and symptoms from 400 clinical notes representing 387 patients. Development (n=300) and validation (n=100) sets were partitioned independently for each concept. Pythia achieved mean sensitivity of 0.76 and specificity of 0.95, compared with 0.82 and 0.76 for the lexicon, and matched or exceeded the lexicon on both metrics for 20 of 62 directly comparable concepts. For 14 concepts where the lexicon labeled every note positive, Pythia recovered mean specificity of 0.97 by requiring a present-tense, patient-attributed finding rather than any textual mention of a term. Specificity transferred from development to validation with minimal degradation across prevalences, whereas sensitivity transfer weakened below 5% prevalence, reaching a mean gap of 0.25 below 2% prevalence. A BERT classifier fine-tuned per concept on the same development set achieved mean sensitivity of 0.23 and collapsed to zero sensitivity for concepts below roughly 5% prevalence. These findings suggest that autonomous, fine-tuning-free prompt optimization can produce symptom extraction prompts that generalize effectively from development to validation while remaining deployable on local infrastructure.

Abstract

LLM Agent Workflow Optimization

Explore similar work

A Multi-Agent System for Autonomous, Fine-Tuning-Free Clinical Symptom Detection: Development and Validation Study

Plausible but Wrong: A case study on Agentic Failures in Astrophysical Workflows

AgentFixer: From Failure Detection to Fix Recommendations in LLM Agentic Systems