We present Verified Multi-Agent Orchestration (VMAO), a framework that coordinates specialized LLM-based agents through a verification-driven iterative loop. Given a complex query, our system decomposes it into a directed acyclic graph (DAG) of sub-questions, executes them through domain-specific agents in parallel, verifies result completeness via LLM-based evaluation, and adaptively replans to address gaps. The key contributions are: (1) dependency-aware parallel execution over a DAG of sub-questions with automatic context propagation, (2) verification-driven adaptive replanning that uses an LLM-based verifier as an orchestration-level coordination signal, and (3) configurable stop conditions that balance answer quality against resource usage. On 25 expert-curated market research queries, VMAO improves answer completeness from 3.1 to 4.2 and source quality from 2.6 to 4.1 (1-5 scale) compared to a single-agent baseline, demonstrating that orchestration-level verification is an effective mechanism for multi-agent quality assurance.
Figures & tables
Figure 1: (a) VMAO framework architecture showing the iterative Plan-Execute-Verify-Replan loop. (b) Agent taxonomy organized by functional tier with information flow from data gathering through analysis to output generation.
Field
Description
id
Unique identifier (e.g., sq_001)
question
Specific, answerable question text
agent_type
Agent from taxonomy to handle this question
dependencies
IDs of sub-questions that must complete first
priority
Execution priority (1–10); higher = more important
context_from_deps
Whether to include dependency results in prompt
Table 1: Sub-question structure generated by the QueryPlanner
Figure 2: (a) DAG execution: independent sub-questions execute in Wave 1; dependent questions in subsequent waves. (b) Verification-driven iteration: Iteration 1 identifies incomplete results, triggering replanning; Iteration 2 achieves sufficient completeness for synthesis.
Condition
Threshold
Rationale
Ready for Synthesis
80% complete
Sufficient sub-questions answered
High Confidence
75% conf, 50% complete
High reliability despite partial coverage
Diminishing Returns
< 5% improvement
Further iteration yields minimal gain
Token Budget
1M tokens
Hard cost limit
Max Iterations
3 iterations
Hard iteration limit
Table 2: Stop conditions for orchestration termination
Tier
Agent
Tools
Key Capabilities
1: Data
RAG
13
Semantic, keyword, and hybrid retrieval; metadata filtering
Web Search
4
General and AI-powered search, news retrieval
Financial
7
Stock quotes, technical indicators, fundamentals
Competitor
11
Market positioning, benchmarks, competitor news
2: Analysis
Analysis
20
Survey analytics, financial and competitor analysis
Reasoning
24
Cross-domain reasoning with RAG, web, and financial tools
Table 3: Agent taxonomy with tool allocation across MCP servers (42 unique tools total)
Figure 3: (a) Token usage breakdown by orchestration phase for a typical query. Execution dominates at 61%, while verification and synthesis remain efficient. (b) Completeness scores by query category across methods. VMAO shows consistent improvements, with largest gains on Strategic Assessment (+53%).
Method
Completeness
Source Quality
Avg Tokens
Avg Time (s)
Single-Agent
3.1
2.6
100K
165
Static Pipeline
3.5
3.2
350K
420
VMAO (Ours)
4.2
4.1
850K
900
Table 4: Comparison of orchestration methods on market research tasks. Completeness and Source Quality are co-scored by LLM and human evaluators (1–5 scale, higher is better).
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Parameter
Default
Description
max_iterations
3
Maximum replanning iterations
token_budget
1M
Maximum tokens before stopping
ready_threshold
0.8
Completeness ratio for synthesis
high_confidence
0.75
Confidence threshold for early stop
diminishing_returns
0.05
Minimum improvement to continue
max_concurrent
3
Parallel agent executions
Appendix
Table 5: Configuration parameters for VMAO orchestration
AI agents are increasingly used to solve complex, multi-step tasks, but existing multi-agent frameworks remain brittle as workflows grow in scale and depth. Small errors at intermediate stages can propagate through agent interactions, while insufficient grounding and weak verification mechanisms further limit reliability. We present Meta-Agent, a two-phase framework that automatically constructs and executes specialized multi-agent systems from natural-language task descriptions. In the construction phase, a task planner decomposes a problem into a directed acyclic graph of agent specifications with explicit input/output contracts and verification criteria. A web search module grounds each specification with external evidence, and a code generation module produces system prompts and tool configurations. A construction-time verification stage then validates generated artifacts and triggers targeted regeneration when failures are detected. In the execution phase, a coordinator dispatches subtasks across the agent graph while execution-time verification gates intermediate outputs. We further introduce a three-level error attribution mechanism that distinguishes local, upstream, and structural failures, enabling targeted recovery strategies ranging from localized retries to partial re-execution and re-decomposition. We evaluate Meta-Agent across coding, contextual learning, and open-ended reasoning tasks. Experiments against strong multi-agent baselines and ablation studies demonstrate consistent improvements in task success rate, error recovery, and workflow stability. The results highlight the importance of tightly integrating planning, grounding, and verification for building reliable multi-agent systems.
Large language model (LLM) agents have demonstrated remarkable capabilities in tool use, reasoning, and code generation, yet single-agent systems exhibit fundamental limitations when confronted with complex research tasks demanding multi-source synthesis, adversarial verification, and personalized delivery. We present DOVA (Deep Orchestrated Versatile Agent), a multi-agent platform introducing three key innovations: (1) deliberation-first orchestration, where explicit meta-reasoning precedes tool invocation, informed by a persistent user model and entity-aware conversation context; (2) hybrid collaborative reasoning, a composable three-phase pipeline unifying ensemble diversity, blackboard transparency, and iterative refinement; and (3) adaptive multi-tiered thinking, a six-level token-budget allocation scheme that reduces inference cost by 40-60% on simple tasks while preserving deep reasoning capacity. We formalize the core algorithms, present an architectural ablation study across seven system configurations, and analyze the contribution of each component to answer confidence, source coverage, and token efficiency.
Aaron Shen, Alfred Shen
University of California, Berkeley, USA · Amazon Web Services, USA
Large language models (LLMs) provide a flexible foundation for multi-agent systems, but their effectiveness and computational cost depend critically on orchestration design. Across different tasks, role design, capacity assignment, and dependency construction jointly affect both solution quality and execution efficiency. Existing approaches automate parts of this design process, yet they often optimize these decisions partially or sequentially, and rely on execution-level feedback that provides limited credit assignment for local orchestration decisions. We propose LEMON (Learning Executable Multi-Agent Orchestration via Counterfactual Reinforcement Learning), an LLM-based orchestrator that learns to design efficient multi-agent orchestration. Given a task, LEMON designs a unified orchestration specification that composes customized agent duties, capacity levels, and dependency relations. To train the orchestrator, we augment the orchestration-level Group Relative Policy Optimization (GRPO) objective with a localized counterfactual credit signal that edits role, capacity, or dependency fields and applies the resulting reward contrast only to the edited spans. Experiments on six reasoning and coding benchmarks, including MMLU, GSM8K, AQuA, MultiArith, SVAMP, and HumanEval, show that LEMON achieves the best average performance among evaluated multi-agent orchestration methods while improving the accuracy-token trade-off.