Conservative Meta-Agent Search: Architecture Adaptation under Non-Stationarity

Ali Atiah Alzahrani

cs.AIOct 22, 2026

Conservative Meta-Agent Search: Architecture Adaptation under Non-Stationarity

Authors: Ali Atiah Alzahrani

Abstract

Automated agentic-system design is typically evaluated on static task suites with immediate scalar rewards. Deployed systems face persistent shift, delayed dependent feedback, correlated component failures, and costly architecture changes. We formalize meta-agent architecture search as sequential incumbent replacement with switching costs and hard risk constraints and introduce Conservative Meta-Agent Search (CMAS). CMAS retrieves archived architectures and proposes typed graph edits, then scores each against the incumbent by paired shadow replay under common random numbers, with interventional component credit. Promotion requires an empirically calibrated, variance-penalized simultaneous score above a pre-specified margin, and is followed by a simulated canary stage and pre-specified rollback. Since backtests cannot certify meta-level skill, we introduce MetaInvest-Bench, a regime-switching simulator whose specified worker competence makes a surrogate-optimal architecture computable, with leakage-controlled replay, adversarial governance tracks, and five negative controls. Financial streams serve as a risk-sensitive testbed for meta-level generalization; we claim no alpha. Across 96 streams, CMAS cuts normalized mean surrogate dynamic-oracle regret from 0.116 to 0.104 and raises held-out utility from 0.44 to 0.48 over the strongest adaptive baseline; when extra workers add no value, it contracts to near-single-agent teams. Its deployment inference cost is ~6% lower, though its search-inclusive token cost is the highest of the methods compared. The observed per-update false-promotion rate is 3.4% against a nominal 5% target, and 4.8% in a sealed rerun. The calibration is empirical: it concerns an update-level replay statistic on an author-designed benchmark, and guarantees neither full-utility improvement, nor absence of simulated canary-stage harm, nor sequence-level error control.

Figures & tables

Panel of Figure 1 — Figure 1: Illustrative schematic of architecture as a sequential decision variable; annotations are conceptual and do not report experimental results. A frozen workflow may miss persistent shifts; a fixed committee stays expensive in every regime; Cmas expands, rewires, contracts, retains, or rolls back under fixed external controls, after reversible calibrated promotion.

Panel of Figure 2 — Figure 2: Cmas as a governed adaptation loop for a multi-agent investment workflow. The risk projection, authorization, accounting and data cutoffs sit outside the LLM-editable boundary. Left: the archive $\mathcal{D}_{k}$ is operational memory of prior workflow behavior, storing regimes, architectures, traces, utility and credit estimates, and failure labels; the foundation-model weights are never updated. Bottom (inner loop, weekly): incumbent $h_{k}$ runs typed specialists whose outputs $o_{i,t}$ are aggregated by $g$ , after which the immutable projection $\Pi_{\mathcal{C}_{t}}$ enforces the financial control boundary and the trace $\tau_{t}$ is logged back to the archive. Top (outer loop, monthly): ❶ retrieve parents by regime similarity and novelty; ❷ propose a compile-checked, auditable typed diff with hypothesis; ❸ paired shadow replay of candidate against incumbent under common random numbers, plus red-teaming; ❹ interventional credit $\phi_{i}$ fed back; ❺ promote only if the empirically calibrated simultaneous score exceeds $\epsilon$ ; ❻ simulated canary stage with rollback, after which the promoted $h_{k+1}$ (or the retained $h_{k}$ ) returns to the inner loop.

Method	Regret $\downarrow$	OOD utility $\uparrow$	False promo. $\downarrow$	Team size	Cost $\downarrow$
Single agent + SC	0.226 $\pm$ .022	0.28 $\pm$ .05	n/a	1.0	1.00
Fixed committee	0.183 $\pm$ .017	0.32 $\pm$ .03	n/a	6.0	2.40
Rule router	0.158 $\pm$ .016	0.38 $\pm$ .03	0.071	3.5	1.45
AFlow-style	0.137 $\pm$ .014	0.41 $\pm$ .05	0.093	4.7	2.05
MaAS/AutoMaAS-style	0.116 $\pm$ .013	0.44 $\pm$ .03	0.082	4.0	1.83
GEPA prompt-only	0.166 $\pm$ .019	0.36 $\pm$ .03	0.063	4.0	1.55

Table 1: Primary meta-level results (Track A; 96 streams;

\pm

95% stream-level block-bootstrap CIs). Search rows are our common-budget “-style” reimplementations. Cost is deployment inference only and is not a net saving: Cmas is heaviest on search-inclusive tokens (Table 9 ). “n/a”

=

never updates, so false promotion is undefined, not zero.

Panel of Figure 3 — Figure 3: Headline diagnostics. Left: regret around evaluator-known regime transitions, 95% bands; Cmas recovers in $\sim$ 2.7 steps. Right: utility–cost frontier on deployment-only cost ( Cmas is heaviest on search-inclusive tokens; Table 9 ). Bubble size is average team size. Color shows false-promotion rate for updating methods; static methods make no promotion decisions, so they are drawn as hollow grey squares outside the color scale.

Figure 4: Promotion calibration and Goodhart drift. Held-out final test (2,112 decisions, 96 streams); all plotted parameters frozen at validation (Table 16 ), so neither panel is post-hoc fitted. Left: realized false promotion vs. pre-specified

\delta

, observed below the nominal target on the tested distributions. This is empirical calibration; the score is not a valid confidence bound (Remark 1). Right: one objective-misspecification probe illustrating a failure mode.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Term	Meaning
Certified statistic $\tilde{J}$	additive block statistic in Eq. ( 3 ); excludes variance/CVaR/MDD
Full utility $J$	complete scalarization incl. CVaR/MDD penalties; risk terms act as eligibility filters
Promotion	candidate replaces incumbent after $\mathrm{Score}_{k}(c)>\epsilon$ and eligibility pass
Canary stage	simulated post-promotion, limited-authority run alongside the incumbent with pre-specified rollback triggers
Rollback	reversion to the incumbent when a trigger fires
False promotion	promoted candidate whose simulator-evaluated mean improvement in $\tilde{J}$ is $\leq\epsilon$ ; near-neutral, and not necessarily a negative- $\tilde{J}$ promotion

Appendix

Table 2: Terminology used for promotion outcomes.

Dimension	Pre-specified values
Workers	any subset of six roles; per-role backbone is a separate model-assignment field (two open-weight options; twelve role–backbone instances)
Graph	DAG; $\leq 10$ edges; no self-loops; permissions constrain edge types
Aggregation	parallel merge, confidence-weighted, sequential, debate, critic-then-aggregate
Models	one small open model, one stronger open model, one hosted reasoning model
Budget	$1\times$ , $2\times$ , $4\times$ token/tool budgets with a hard per-episode ceiling
Prompt mutation	one typed field per candidate; no hidden instruction changes

Appendix

Table 3: Pre-specified search space.

Method	Search object / proposal	Adaptation scope	Primary distinction from Cmas
ADAS / AFlow / AgentSquare [ 3 , 19 , 14 ]	executable programs or workflows	offline/static or per task	no repeated paired incumbent replacement
MaAS / AutoMaAS [ 20 , 9 ]	architecture and resources	query-level	query routing rather than deployment-time promotion calibration
GEPA / Darwin Gödel Machine [ 1 , 21 ]	prompts or executable code	iterative/open-ended	population fitness rather than paired replay and rollback
Adaptive Auto-Harness [ 43 ]	harness tree and stateful evolver	open-ended non-stationary streams	solve-time routing/human steering; no explicit update-level false-promotion target
Continual Harness / AutoRAS [ 41 , 42 ]	online or robustness-aware harnesses	continual/robust	no explicit paired incumbent gate with measured false-promotion error
Cmas	typed architecture and archive-conditioned edits	sequential deployment	paired replay, calibrated score, switching cost, canary stage, rollback

Appendix

Table 4: Closest search and adaptation methods. Entries summarize first-class mechanisms in the cited descriptions.

Method	First-class safety mechanism	Relation to Cmas
Gated semantic QD [ 39 ]	deterministic candidate-quality gate	no paired incumbent replay, rollback, or update-level calibration
HARBOR [ 40 ]	posterior chance-constrained safety check	no repeated incumbent replacement or rollback
HarnessAudit [ 44 ]	full-trajectory boundary and information-flow audit	complementary audit surface; does not optimize sequential promotion
SafeHarness [ 45 ]	lifecycle verification, privilege control, adaptive degradation, rollback	complementary execution security; no false-promotion estimand
Cmas	paired calibrated promotion, hard eligibility, canary stage, rollback	focuses on whether and when to replace the incumbent under drift

Appendix

Table 5: Closest safety and deployment-control methods. Related ingredients are acknowledged; Cmas contributes their deployment-level combination.

Method	Normalized mean surrogate dynamic-oracle regret $\downarrow$	OOD utility $\uparrow$
LinUCB menu bandit	0.142	0.38

Appendix

Table 6: Partially logged LinUCB menu baseline. Only the two metrics below were retained; it is therefore excluded from the fully populated primary comparison in Table 1 .

Proposer / config. (fixed evaluator+gate)	OOD util. $\uparrow$	Dyn. regret $\downarrow$	False promo. $\downarrow$
LLM proposer (full Cmas )	0.48	0.104	0.034
Best adaptive baseline $+$ Cmas gate	0.45	0.113	0.035
Evolutionary candidates	0.42	0.129	0.033
Random candidates	0.38	0.151	0.032

Appendix

Table 7: Proposer-value and gate decomposition. Proposer varied; evaluator, credit, and gate fixed. “

+

Cmas gate” routes that baseline’s proposals through our gate. Hierarchical (schedule-clustered) 95% CIs on OOD utility, top to bottom:

[.46,.50]

[.43,.47]

[.40,.44]

[.36,.40]

. Proposer-component ablations: Appendix C .

Method	Adaptation delay $\downarrow$	Credit $\rho$ $\uparrow$	Notes
Single agent + SC	7.3 $\pm$ .8	–	static
Fixed committee	6.2 $\pm$ .6	0.11 $\pm$ .06	static
Rule router	4.3 $\pm$ .5	0.18 $\pm$ .05	hand-routed
AFlow-style	3.5 $\pm$ .4	0.25 $\pm$ .05	search
MaAS/AutoMaAS-style	3.1 $\pm$ .3	0.32 $\pm$ .04	strongest adaptive baseline
GEPA prompt-only	4.6 $\pm$ .5	0.18 $\pm$ .05	fixed graph

Appendix

Table 8: Secondary Track A metrics , held out of the headline table for readability. Adaptation delay is right-censored at the eight-update event-study horizon; exact architecture matching is used for credit/selection diagnostics, while regret remains the primary oracle-relative metric.

Method	Tokens/update	A100-hours/update
Cmas	850k	4.2
AFlow-style	820k	6.5
MaAS-style	780k	5.1
GEPA-style	450k	1.8
Single agent	50k	0.2

Appendix

Table 9: Realized search-inclusive cost per update (mean over streams; budget cap 0.85M tokens/update for all search methods).

Figure 5: Selection vs. regime (RQ2). Mean worker-selection frequency by latent regime over the 96 streams; rotation with the hidden regime (macro triples from Growth to Inflation, tail-risk peaks in Stress) is evidence consistent with regime-conditioned selection, and the shuffled-regime control (Table 10 ) removes it.

Variant	OOD utility $\uparrow$	False promo. $\downarrow$	Cost $\downarrow$	Clone team size
Full Cmas	0.48	0.034	1.72	1.6
No regime conditioning	0.44	0.040	1.70	3.2
No paired evaluation	0.45	0.061	1.77	3.4
No interventional credit	0.46	0.046	1.75	3.5
No parsimony penalty	0.47	0.037	2.02	4.6
No simultaneous correction	0.48	0.084	1.73	3.4

Appendix

Table 10: Key ablations and controls (Track A held-out streams). The rows above the rule are one-component-removed ablations; the two rows below the rule are negative controls run on full Cmas . Clone team size is each variant’s mean team size under the clone-collapse control; for the clone-control row itself this is by construction the same quantity as Full Cmas ’s entry (1.6), and its OOD utility of 0.28 matches the single agent, as expected under identical workers. 95% hierarchical block-bootstrap CIs (clustered by regime schedule, seeds within) widen slightly relative to stream-level intervals (e.g., no-paired-evaluation OOD utility

[0.42,0.48]

), but every utility and false-promotion degradation relative to Full Cmas remains assessed by paired stream-level differences with Holm correction (full matrix omitted for space); the marginal CIs shown are descriptive, and the “No parsimony” and “No adversarial tests” false-promotion deltas are near-neutral.

Panel of Figure 6 — Figure 6: MetaInvest-Bench triangulation. Track A measures controlled meta-level capability against a computable surrogate dynamic oracle; Track B provides leakage-controlled external validity on markets (18 replay windows); Track C measures governance under a budgeted meta-adversary (576 episodes). All tracks feed a shared meta-level metric suite.

Method	Selection accuracy $\uparrow$	Credit $\rho$ $\uparrow$
Best adaptive baseline	59.1%	0.32
Cmas (ours)	63.8%	0.41

Appendix

Table 11: RQ2 interventional fidelity (Track A). Architecture-selection accuracy is the fraction of outer-loop updates matching the architecture selected by the surrogate dynamic oracle at that update; credit

\rho

is the rank correlation between estimated interventional credit and surrogate-model component value.

Method	CEQ % $\uparrow$	Sharpe $\uparrow$	MDD $\downarrow$	CVaR $\downarrow$	$\alpha$ % $\uparrow$	Turnover $\downarrow$	Violations $\downarrow$
Risk parity	4.2	0.68	19.2	2.24	0.2	0.18	0.000
Single agent + SC	5.1	0.84	17.8	2.05	0.9	0.31	0.034
Fixed committee	5.8	0.95	18.4	2.12	1.2	0.44	0.049
Rule router	6.7	1.04	15.6	1.88	1.6	0.32	0.026
Best adaptive basel.	7.4	1.12	14.5	1.79	2.0	0.41	0.031
Cmas (ours)	7.8	1.15	14.1	1.68	2.2	0.35	0.021

Appendix

Table 12: Track B external validity (

N=18

date-controlled replay windows; net of 10 bps costs; “best adaptive baseline” is MaAS/AutoMaAS-style selection). This track tests whether Track A conclusions survive contact with market data under leakage controls; with

N=18

it is underpowered to separate adaptive methods , differences are not tested for significance, and no row should be read as evidence of deployable alpha. Factor

\alpha

is the residual after market/style attribution. Worker-compromise recovery is a governance quantity and is reported only in Table 14 (Track C) to avoid conflating the two tracks. Because

N=18

, we report the sign pattern only: Cmas CEQ dispersion is

\pm 1.1

pp (across-window standard deviation; mean

7.8\%

, min

6.8\%

, max

11.2\%

; the window distribution is right-skewed), so its worst windows overlap the baseline mean, and no cell should be read as a significance claim. Units: CEQ and factor

\alpha

in annualized %; Sharpe dimensionless; MDD and CVaR %; turnover annualized fraction; violations fraction of decisions.

Held-out factor	Cmas	Best baseline	Relative
Unseen regime mixtures	83.5%	78.5%	$+5.0$ pp
Held-out worker library (2 new)	81.2%	–	recovers in 4 updates
Backbone switch (OOD util. cost)	7.8%	11.8% (AFlow-style)	$1.5\times$ smaller
Held-out violation rate (vs. 2.1% ID)	1.8%	–	safety clause met (no increase)
Held-out false promotion (vs. 3.4% ID)	4.1%	–	strict clause not met ( $+0.7$ pp)

Appendix

Table 13: Meta-level transfer (Track A held-out streams; retention

=

held-out

/

in-distribution utility). “Best baseline” is MaAS/AutoMaAS-style selection except in the backbone-switch row, where the relevant competitor is the search-based AFlow-style method (noted inline). The held-out-library baseline cell is “–” because MaAS-style selection cannot ingest new worker types without re-running search.

Attack family	Cmas recovery	( $k/n$ )	Best baseline	Committee
Instruction-corrupted worker	0.81	78/96	0.74	0.54
Anti-correlated worker	0.79	76/96	0.72	0.53
Stale/missing data	0.77	74/96	0.70	0.51
Inference-cost shock	0.76	73/96	0.69	0.50
Worker dropout	0.80	77/96	0.73	0.52
Multi-attack combination	0.75	72/96	0.68	0.49

Appendix

Table 14: Governance suite by attack family (Track C; 576 episodes nested in 96 streams; recovery within three updates). Aggregate Wilson 95% interval

[0.75,0.81]

; per-family Wilson and cluster-by-stream intervals are omitted for space. Five single-attack families plus one multi-attack combination; the aggregate spans all six scenarios. Aggregate rollback precision/recall

0.85/0.80

Figure 7: Ablation sensitivity (visualizes Table 10 ). Regime conditioning, paired evaluation, and interventional credit dominate utility; paired evaluation and the simultaneous correction dominate promotion safety; the parsimony penalty prevents architecture bloat under clone-collapse.

Figure 8: Credit fidelity (RQ2). Estimated interventional credit shows moderate positive rank agreement with surrogate-model component value (

\rho{=}0.41

n{=}16

component types pooled across streams; regression line shown), strongest among high-credit components. The pooled scatter is illustrative only; the statistic we rely on is the per-stream distribution of

\rho

over the 96 streams: median

\rho=0.41

, 95% CI

[0.35,0.47]

, excluding zero.

Step	Time / object	Information boundary
1	archive cutoff $t_{k}$	build $\mathcal{D}_{k}$ only from outcomes and credits released before $t_{k}$
2	candidate generation	freeze $\mathcal{H}_{k}$ ; no access to outcomes in the designated $W_{k}$
3	paired replay $W_{k}$	run candidate and incumbent on identical packets, shocks, seeds, and tool responses
4	gate decision	apply eligibility filters and $\mathrm{Score}_{k}$ ; retain incumbent or enter simulated canary stage
5	canary interval	candidate has limited simulated authority alongside the incumbent until the next update; rollback restores the incumbent architecture and state snapshot
6	archive release	replay labels, canary outcomes, and credits become available only to $\mathcal{D}_{k+1}$

Appendix

Table 15: Information chronology at update

k

. This distinguishes the overlapping protocol from the sealed rerun and makes the proposal/evaluation boundary explicit.

Stage	Used for	Frozen before subsequent stage
Development/training	proposer/archive development; utility normalizers	search space, typed edit interface, normalizers
Validation/calibration	score form and sensitivity analysis	block length 4, 99th-percentile winsorization, $\epsilon=0.02$ , candidate pool 8, rollback thresholds
Final Track A test	96 streams; 2,112 update decisions	no tuning or candidate-pool adaptation to final outcomes
Sealed rerun	non-overlapping empirical calibration check	reuses the same score, margin, and eligibility rules

Appendix

Table 16: Calibration and evaluation stages. The source records the roles and frozen choices below; it does not contain a public preregistration timestamp or anonymous commit hash, so none is claimed.

Symbol	Meaning	Value
$\|\mathcal{H}_{k}\|$	candidate-pool size per update	8
$K$	outer-loop updates per stream	$\approx$ 22 (monthly)
$\epsilon$	promotion margin (normalized utility)	0.02
$\delta$	family-wise false-promotion budget (per update)	0.05
$W_{k}$	paired replay-window length	26 weeks
block length	circular block bootstrap	4 weeks

Appendix

Table 17: Configuration used for the reported results. These values are the source of record for the reported results. Proposer and red-team/critic model: Llama-3-70B-Instruct (cutoff Dec 2023; temperature 0.2, top-

p

0.9); candidate utilities are computed by deterministic replay code. Workers: Mixtral-8x7B-Instruct-v0.1 (vendor cutoff undisclosed; replay restricted to post–January 2024 data; temperature 0.4) and Llama-3-8B-Instruct (cutoff Mar 2023; temperature 0.3). Hosted reasoning model (backbone-transfer test): Claude 3 Opus via API (cutoff Aug 2023; temperature 0.1). Seeds: 42 (sampling), 1337 (simulation), 2024 (proposal).

Multi-Agent System Optimization

Multi-Agent System Optimization studies how to improve agent teams’ coordination, performance, resource use, and reliability.

See all 162 papers

MetaShare: Failure-Aware Collective Self-Improvement for Open-Ended Agents
Oct 15, 2026Vishal Bhutani, Prasang Gupta, Sumanth Chundru +1Self-Improving Agents Self-Evolving Agents
PricewaterhouseCoopers (PwC)
Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer
Jul 19, 2026Jingjie Ning, Xiaochuan Li, Shanshan Zhong +2Automated Algorithm Discovery Automated Evaluation
School of Computer Science, Carnegie Mellon University · 2DP Technology
ArchAgent v2: A Case Study with the Data Prefetching Championship
Aug 10, 2026Abraham Gonzalez, Raghav Gupta, Akanksha Jain +12Automated Algorithm Discovery Automated Heuristic Design
Google · University of California, Berkeley · Google DeepMind

Oct 15, 2026J/K move · Enter open · S save

MetaShare: Failure-Aware Collective Self-Improvement for Open-Ended Agents

Vishal Bhutani, Prasang Gupta, Sumanth Chundru, Kevin Paul

PricewaterhouseCoopers (PwC)

Self-Improving Agents Self-Evolving Agents Multi-Agent Collaboration

Open paper AskPDF arXiv

Humans often learn fastest not by repeating success, but by remembering which failures not to repeat. Open-ended self-improving agents can share experience across lineages, as in GEA, or edit their own improvement mechanisms, as in DGM-Hyperagents, yet failed improvement attempts are rarely retained as reusable cross-branch knowledge, and search can collapse onto redundant strategies. We introduce MetaShare, a failure-aware framework that records successful and unsuccessful agent modifications together with signed performance deltas. Before each generation, MetaShare retrieves diverse high-performing strategies alongside high-confidence failures from other lineages, letting agents reuse effective mechanisms while avoiding known dead ends. A task-blind auditor periodically injects absent or underrepresented mechanism categories, while embedding-based de-ranking limits redundancy during retrieval. Multiple meta-agents propose modifications in parallel at heterogeneous temperatures behind a deterministic selector that gates applicability, scope, compilability, and size. Experiments across paper review, coding, math grading, robotics reward design, and financial audit-risk assessment show that MetaShare's median matches or exceeds DGM-Hyperagents' final performance and reaches its own best-performing agent in 1.7--2.9

\times

fewer evolution iterations.

Abstract

Multi-Agent System Optimization

Explore similar work

MetaShare: Failure-Aware Collective Self-Improvement for Open-Ended Agents

Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer

ArchAgent v2: A Case Study with the Data Prefetching Championship