When Does GEPA Merge Help? A Mechanism-Level Diagnostic

Chong Zhao; Ruyi Yang; Changming Liu; Alice Zhao

cs.CLOct 25, 2026

When Does GEPA Merge Help? A Mechanism-Level Diagnostic

Authors: Chong Zhao, Ruyi Yang, Changming Liu, Alice Zhao

Organizations: Harvard University, Cambridge, MA, USA.

Abstract

GEPA is among the strongest reflective prompt optimizers for language-model programs, but its system-aware merge operator is unstable across models: merge improves gpt-4.1-mini on average while degrading Qwen3-8B, an asymmetry noted but left unexplained by the GEPA paper. We investigate this instability across five datasets and two models. Our nine-configuration sweep over merge timing and merge operator shows that merge can improve both models when configured appropriately. This suggests that merge instability is better explained by configuration sensitivity than by an inherent flaw in the merge operator. We then test whether prompt-level features predict GEPA's merge-accept decisions and find that they do not, ruling out static-feature gating. As a proof of concept, a behavior-based probe built from these diagnostics avoids large merge failures in our multi-seed runs by abstaining on high-risk tasks, although it does not consistently outperform NoMerge. Code is available as GEPA-MergeProbe at https://github.com/ChongZZZZ/GEPA_MERGE .

Figures & tables

Figure 1: Test-score change from adding merge to GEPA (GEPA+Merge $-$ GEPA), computed from Tables 1–2 of Agrawal et al. (2026) . Merge helps gpt-4.1-mini but hurts Qwen3-8B; dashed lines mark the six-task aggregate change ( $+1.14$ and $-2.45$ pp).

Figure 2: Overview of the GEPA optimization loop, reproduced from Agrawal et al. (2026) . Parents are drawn from a Pareto-filtered candidate pool (bottom); each iteration proposes a child with either reflective prompt mutation or system-aware merge (right), and keeps it only if it improves on a minibatch and then on the full validation set (top). Our study varies when the merge branch may fire and how it resolves conflicting modules (§ 4 ).

Case	Condition	Resolution
A	Exactly one descendant diverged from ancestor	Take the diverged descendant
B	Both diverged differently	Variant-dependent (§ 4 )
C	Both descendants agree ( $p_{1}=p_{2}$ )	Take either

Table 1: Case classification in the GEPA merge operator.

Dataset	Task type
HotpotQA	multi-hop QA
HoVer	fact verification
IFBench	instruction following
MuSiQue	compositional multi-hop QA
2WikiMultiHopQA	entity-relation multi-hop QA

Table 2: The five benchmarks evaluated. Per-benchmark retrieval modes are reported in Appendix A .

Figure 3: Configuration sweep: $\Delta$ test score relative to NoMerge. Columns denote benchmarks and rows denote $(\text{operator}\!\times\!\text{timing})$ configurations. The left panel shows Qwen3-8B; the right panel shows gpt-4.1-mini. Per-benchmark winners are shown in boldface, and catastrophic cells with $\Delta\!\leq\!-20$ pp appear as dark-red outliers. The paper-default setting (original $\times$ immediate) is rarely the best configuration.

Task	GPT-4.1-mini		Qwen3-8B
	mean	$\sigma$	mean	$\sigma$
HotpotQA	$58.2$	$4.0$	$51.9$	$4.0$
IFBench	$49.6$	$0.8$	$34.4$	$2.2$
HoVer	$46.2$	$2.1$	$44.4$	$3.0$
MuSiQue	$48.1$	$4.7$	$47.2$	$5.1$
2WikiMHQA	$83.5^{\dagger}$	$0.2^{\dagger}$	$82.6$	$2.2$

Table 3: Per-task NoMerge test scores across

3

seeds, showing the seed-variance noise floor for interpreting merge-cell deltas in Figure 3 .

\dagger

has only

n\!=\!2

seeds.

	Qwen3-8B		gpt-4.1-mini
Task	Reflect	Merge	Reflect	Merge
HotpotQA	$-0.005$	$-0.008$	$+0.018$	$-0.006$
IFBench	$+0.039$	$+0.009$	$+0.012$	$-0.001$
HoVer	$+0.020$	$+0.015$	$+0.001$	$-0.004$
MuSiQue	$+0.018$	$+0.001$	$+0.021$	$-0.005$
2WikiMHQA	$+0.045$	$-0.011$	$+0.040$	$-0.008$

Table 4: Mean per-step

\Delta\mathrm{val}

by operator. Reflection exceeds merge on every task for both models.

Task	Qwen $r$	GPT $r$
HotpotQA	$+0.89$	$+0.71$
MuSiQue	$+0.40$	$+0.83$
IFBench	$-0.01$	$+0.00$
HoVer	$-0.06$	$+0.63$
2WikiMHQA	$+0.99$	$+0.98$

Table 5: Per-task validation/test Pearson correlation across the nine merge configurations and NoMerge baseline (

n=10

per model–task pair).

Figure 4: Overview of the behavioral probe. Merge is unlocked only after a score plateau and GEPA’s original validity checks; parent pairs are filtered by strength and maturity balance, then routed to summarize-before under bloat risk or high complementarity, and to the original rule otherwise.

Figure 5: Behavioral probe vs. NoMerge across seeds. Each tile reports mean $\bar{\Delta}$ and sample standard deviation $\sigma$ .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Retrieval	Context
HotpotQA	BM25 / Wiki	top- $k$ retrieved
HoVer	BM25 / Wiki	top- $k$ retrieved
IFBench	none	instruction only
MuSiQue	per-example	20 paragraphs (gold + distractors)
2WikiMultiHopQA	per-example	10 paragraphs (gold + distractors)

Appendix

Table 6: Per-benchmark retrieval mode. BM25/Wiki retrieves from a shared Wikipedia abstracts corpus; per-example uses each dataset example’s packaged context.

Qwen3-8B
Task	Reflect mean	Merge mean	Reflect $\sigma$	Merge $\sigma$
HotpotQA	$-0.005$	$-0.008$	$0.047$	$0.027$
IFBench	$+0.039$	$+0.009$	$0.074$	$0.023$
HoVer	$+0.020$	$+0.015$	$0.027$	$0.018$
MuSiQue	$+0.018$	$+0.001$	$0.048$	$0.015$
2WikiMHQA	$+0.045$	$-0.011$	$0.121$	$0.022$

Appendix

Table 7: Per-step

\Delta\mathrm{val}

mean and standard deviation by operator.

gpt-4.1-mini
Task	$\Delta_{s=0}$	$\Delta_{s=1}$	$\Delta_{s=2}$	$\bar{\Delta}$	$\sigma$
HotpotQA	$+3.66$	$-2.66$	$+0.00$	$+0.33$	$3.17$
IFBench	$+1.36$	$+0.34$	$-1.02$	$+0.23$	$1.19$
HoVer	$+1.66$	$+1.66$	$+5.00$	$+2.77$	$1.93$
MuSiQue	$+1.67$	$+1.66$	$+6.34$	$+3.22$	$2.70$
2WikiMHQA	$+1.00$	$+0.33$	—	$+0.66$	$0.47$

Appendix

Table 8: Per-task probe

\Delta

vs. NoMerge across

3

seeds.

Automatic Prompt Optimization

Automatic prompt optimization uses model or task feedback to revise instructions for stronger, more reliable language-model performance.

See all 103 papers

Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training
May 26, 2026Wenjie Zhou, Bohan Wang, Hongtao Zhang +3Language Model Pretraining Neural Network Optimization
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China · Alibaba Group, China +2
Predicting and Repairing Merge Collapse in Large Language Models
Oct 2, 2026Jungseob Lee, Seungyoon Lee, Sugyeong Eo +3Language Model Merging Task Vector Merging
Korea University · Yonsei University Mirae Campus · Sookmyung Women’s University +1
ContraPrompt: Contrastive Prompt Optimization via Dyadic Reasoning Trace Analysis
Apr 20, 2026Rishav Rishav, Pushpak Pujari, Pushpendre RastogiPairwise Comparison CoT Reasoning
Vizops AI

May 26, 2026J/K move · Enter open · S save

Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training

Wenjie Zhou, Bohan Wang, Hongtao Zhang, Chenxi Jia+2

State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China · Alibaba Group, China+2

Language Model Pretraining Neural Network Optimization Weight-Space Analysis Language Model Merging

Open paper AskPDF arXiv

Model merging has emerged as a lightweight paradigm for enhancing Large Language Models (LLMs), yet its underlying mechanisms remain poorly understood. In this work, we analyze late-stage pre-training trajectories and uncover a Rank-1 Subspace phenomenon: while raw optimization steps oscillate violently, consecutive merged checkpoints collapse onto a stable, approximately one-dimensional linear manifold. We theoretically ground this observation in a river-valley landscape analysis: averaging acts as a geometric low-pass filter that dampens high-curvature noise to reveal the optimal descent direction. Capitalizing on this insight, we propose Extra-Merge, a training-free strategy that extrapolates along this subspace to minimize loss without additional gradient updates. Extensive experiments across GPT-2 and LLaMA families (124M to 2B) demonstrate that Extra-Merge consistently outperforms standard merging baselines. Notably, it yields consistent zero-shot accuracy gains on Pythia-12B downstream tasks and generalizes effectively to the Muon optimizer \citep{jordan2024muon}.

Abstract

Automatic Prompt Optimization

Explore similar work

Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training

Predicting and Repairing Merge Collapse in Large Language Models

ContraPrompt: Contrastive Prompt Optimization via Dyadic Reasoning Trace Analysis