Failure-Driven Workflow Refinement
Organizations: Sun Yat-sen University · X-Era AI Lab
Abstract
Optimizing LLM-based workflows is typically formulated as a global search, where candidate workflows are evaluated based on a scalar metric. This paradigm, however, suffers from a critical flaw: information collapse. By reducing rich, multi-step execution traces to simple success/failure signals, existing methods are rendered blind to the underlying structure of failures, fundamentally preventing them from modeling the workflow's failure distribution. We reconceptualize this challenge as a distributional problem. We propose a new paradigm where the optimization goal is not to maximize a scalar score, but to directly minimize a workflow's Expected Failure Mass, i.e., the integral of its failure probability density function defined over a high-dimensional Failure Signature Space (FSS). This distributional lens allows us to move from inefficient, zero-order optimization to a principled, gradient-like descent on the failure landscape itself. We introduce CE-Graph, a framework that operationalizes this paradigm through a novel, failure-driven refinement process. CE-Graph approximates the failure distribution from a pool of counterexamples, identifies its densest regions as recurring failure modes, and applies targeted, operator-constrained graph edits via a Propose-and-Verify mechanism to greedily reduce the failure mass. On math, code, and QA benchmarks, our CE-Graph achieves higher robustness at a significantly lower cost than strong baselines. This suggests that a system's reliability emerges not from avoiding failures, but from systematically learning and reshaping the geometric structure of its failure distributions.
Figures & tables
| Method | GSM8K | MATH | MultiArith | HumanEval | MBPP | Avg. |
|---|---|---|---|---|---|---|
| Single Agent Execution | ||||||
| Vanilla (Baseline) | 87.45 | 46.29 | 96.85 | 87.08 | 71.83 | 77.50 |
| CoT ( Wei et al., 2022 ) | 87.10 | 46.40 | 96.31 | 88.13 | 71.83 | 77.95 |
| ComplexCoT ( Kojima et al., 2022 ) | 86.89 | 46.53 | 96.70 | 87.49 | 72.36 | 78.00 |
| SC ( Wang et al., 2023 ) (CoT 5) | 87.57 | 47.91 | 96.58 | 88.60 | 73.60 | 78.85 |
| Hand-craft Multi-agent Systems |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | HumanEval | MATH | ||||
|---|---|---|---|---|---|---|
| LLM Backbone | gpt-4o-mini | Qwen-2.5-72b | llama-3.1-70b | gpt-4o-mini | Qwen-2.5-70b | llama-3.1-70b |
| Vanilla | 87.08 | 85.60 | 80.06 | 46.29 | 63.80 | 31.93 |
| + CE-Graph | 94.26 | 91.12 | 86.21 | 55.91 | 72.15 | 46.98 |
| Transfer Scenario | MATH→GSM8K | GSM8K→MATH | HumanEval→MATH |
|---|---|---|---|
| AFlow | 91.95 | 49.39 | 47.15 |
| MaAS | 92.80 | 51.02 | 50.27 |
| CE-Graph (Ours) | 93.82 | 52.89 | 52.17 |
| Embedding Model | ARI (vs. BERT-base) | Final Accuracy (%) |
|---|---|---|
| BERT-base-uncased (Default) | 1.00 | 55.91 |
| SentenceTransformers | 0.87 | 54.75 |
| text-embedding-ada-002 | 0.92 | 55.34 |
| Library | Operators Added | Example | LoC Added | Expert Time |
|---|---|---|---|---|
| Generic Baseline | (Base Set) | RevisePrompt | - | - |
| Math Reasoning | +2 Specific | AddVerifierNode | ~30 | ~0.5 hr |
| Code Generation | +3 Specific | AddExceptionHandler | ~50 | ~1.0 hr |
LLM Agent Workflow Optimization
LLM agent workflow optimization studies how to design and run multi-step agents for better task success, reliability, latency, and cost.
See all 283 papers