Combee: Scaling Prompt Learning for Self-Improving Language Model Agents

Hanchen Li; Runyuan He; Qizheng Zhang; Changxiu Ji; Qiuyang Mang; Xiaokun Chen; Lakshya A Agrawal; Wei-Liang Liao

cs.AIApr 5, 2026

Combee: Scaling Prompt Learning for Self-Improving Language Model Agents

Authors: Hanchen Li, Runyuan He, Qizheng Zhang, Changxiu Ji, Qiuyang Mang, Xiaokun Chen, Lakshya A Agrawal, Wei-Liang Liao,

Organizations: UC Berkeley · Stanford University · Independent · Gradient Network

Abstract

Recent advances in prompt learning allow large language model agents to acquire task-relevant knowledge from inference-time context without parameter changes. For example, existing methods (like ACE or GEPA) can learn system prompts to improve accuracy based on previous agent runs. However, these methods primarily focus on single-agent or low-parallelism settings. This fundamentally limits their ability to efficiently learn from a large set of collected agentic traces. It would be efficient and beneficial to run prompt learning in parallel to accommodate the growing trend of learning from many agentic traces or parallel agent executions. Yet without a principled strategy for scaling, current methods suffer from quality degradation with high parallelism. To improve both the efficiency and quality of prompt learning, we propose Combee, a novel framework to scale parallel prompt learning for self-improving agents. Combee speeds up learning and enables running many agents in parallel while learning from their aggregate traces without quality degradation. To achieve this, Combee leverages parallel scans and employs an augmented shuffle mechanism; Combee also introduces a dynamic batch size controller to balance quality and delay. Evaluations on AppWorld, Terminal-Bench, Formula, and FiNER demonstrate that Combee achieves up to 17x speedup over previous methods with comparable or better accuracy and equivalent cost.

Figures & tables

Figure 1: Summary of improvement snapshot. Combee achieves close-to-optimal quality with significantly reduced training time by increasing the content in the prompt learnt under high parallelism. Experiments with DeepSeek-V3.1 on AppWorld.

Figure 2: Context overload from naive scaling. As batch size increases, the aggregator LLM produces monotonically fewer and less useful context updates, directly degrading final accuracy across benchmarks.

Figure 3: Overall design of Combee. Map-Shuffle-Reduce pipeline: parallel reflection (Map), augmented shuffling (Shuffle), and parallel scan aggregation (Reduce). Combee plugs into existing methods (e.g., ACE or GEPA) between reflection and update; naive scaling aggregates all reflections in one pass.

Method	Batch	Playbook	Training	Training	Test-Normal
	Size	Size (tokens)	Time (min)	Cost	TGC $\uparrow$	SGC $\uparrow$	Avg $\uparrow$
ReAct	–	–	0	$0	63.7	42.9	53.3
ReAct + ACE	1	1,578	86	$1.62	66.1	50.0	58.1
Parallel Prompt Learning
ReAct + ACE	5	4,697	30	$1.68	70.2	57.1	63.7
ReAct + ACE	10	2,329	19	$1.50	72.0	58.9	65.4

Table 1: Parallel prompt learning results with ReAct agent for AppWorld.

Parallel Prompt Learning
Method	Batch	Playbook	Training	Training	Average
	Size	Size (tokens)	Time (min)	Cost	Accuracy @ 1
Terminus-2	–	–	0	$0	32.2%
Terminus-2 + ACE	1	9,067	42.4	$0.24	37.9%
Terminus-2 + ACE	5	4,983	10.2	$0.17	29.9%
Terminus-2 + ACE	10	3,967	5.6	$0.15	33.3%

Table 2: Parallel prompt learning results with Terminus-2 agent for Terminal-Bench 2.0. We trained on existing open-source traces instead of generating trajectories on the fly. We report the average accuracy over three runs.

Panel of Figure 4 — Figure 4: Combee achieves superior quality-delay trade off on GEPA for finance benchmarks.

Panel of Figure 5 — Figure 5: Combee achieves superior quality-delay trade off on ACE for finance benchmarks.

	Formula		FiNER
Duplication Level $p$	GEPA	ACE	GEPA	ACE
1	0.803 (407s)	0.825 (3856s)	0.710 (487s)	0.765 (3295s)
2 (default)	0.857 (470s)	0.855 (3895s)	0.742 (503s)	0.770 (3501s)
3	0.848 (445s)	0.830 (4073s)	0.740 (510s)	0.756 (4514s)
4	0.834 (483s)	0.820 (4057s)	0.747 (517s)	0.749 (4326s)

Table 3: Duplication-level ablation for augmented shuffling. Each cell reports accuracy (completion time). The batch size is fixed to the one selected by Combee in Figures 4 and 5 .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Score (mean $\pm$ std)	Epoch time (mean $\pm$ std, s)
Combee + GEPA	0.857 $\pm$ 0.005	459.3 $\pm$ 33.3
GEPA (bs 80)	0.771 $\pm$ 0.005	408.0 $\pm$ 14.0
GEPA (bs 50)	0.807 $\pm$ 0.013	461.7 $\pm$ 43.2
GEPA (bs 25)	0.855 $\pm$ 0.004	862.0 $\pm$ 39.2
GEPA (bs 10)	0.857 $\pm$ 0.007	2071.3 $\pm$ 284.9
GEPA (bs 5)	0.857 $\pm$ 0.005	3380.3 $\pm$ 404.6

Appendix

Table 4: Formula dataset: mean

\pm

standard deviation over three runs.

Method	Score (mean $\pm$ std)	Epoch time (mean $\pm$ std, s)
Combee + GEPA	0.748 $\pm$ 0.006	503.3 $\pm$ 14.5
GEPA (bs 80)	0.684 $\pm$ 0.011	429.5 $\pm$ 24.7
GEPA (bs 50)	0.714 $\pm$ 0.008	472.7 $\pm$ 32.3
GEPA (bs 25)	0.757 $\pm$ 0.007	733.0 $\pm$ 32.0
GEPA (bs 10)	0.755 $\pm$ 0.005	1719.0 $\pm$ 327.6
GEPA (bs 5)	0.744 $\pm$ 0.014	3325.7 $\pm$ 518.5

Appendix

Table 5: FiNER dataset: mean

\pm

standard deviation over three runs.

	Formula		FiNER
Method	Accuracy	Time (s)	Accuracy	Time (s)
Base LLM	0.640	–	0.666	–
Summarization + GEPA (bs 50)	0.660	–	0.675	–
TopK + GEPA (bs 50)	0.720	–	0.685	–
Hierarchical Summarization + GEPA (bs 50)	0.650	–	0.663	–
GEPA (bs 5)	0.855	2981	0.731	2892

Appendix

Table 6: Exact values plotted in Figure 4 (GEPA on the finance benchmarks). Each entry is the single representative run shown in the figure; Tables 4 and 5 report means and standard deviations over three runs for the corresponding configurations.

	Formula		FiNER
Method	Accuracy	Time (s)	Accuracy	Time (s)
Base LLM	0.640	–	0.666	–
Summarization + ACE (bs 50)	0.755	–	0.744	–
TopK + ACE (bs 50)	0.690	–	0.725	–
Hierarchical Summarization + ACE (bs 60)	0.725	–	0.734	–
ACE (bs 5)	0.820	9464	0.737	5162

Appendix

Table 7: Exact values plotted in Figure 5 (ACE on the finance benchmarks).

Framework	Configuration	Accuracy	Time (s)
ACE	Normal parallel (bs 5)	0.820	9464
	Normal parallel (bs 10)	0.810	7546
	Normal parallel (bs 20)	0.755	6285
	Normal parallel (bs 50)	0.745	4297
	Normal parallel (bs 60)	0.715	2576
	Fixed bs + shuffle/scan (bs 20)	0.850	5450

Appendix

Table 8: Exact values plotted in Figure 7 (dynamic batch size controller ablation on Formula). “Fixed bs + shuffle/scan” runs use augmented shuffling (

p=2

) and parallel scan with mini-batch size

\sqrt{bs}

, but a fixed batch size instead of the dynamic controller.

		Mini-batch size
Framework	Variant	6	8	10	15	20
ACE	w/o augmented shuffling	0.835	0.810	0.755	0.800	0.785
ACE	w/ augmented shuffling	0.845	0.850	0.815	0.840	0.810
GEPA	w/o augmented shuffling	0.839	0.827	0.845	0.825	0.837
GEPA	w/ augmented shuffling	0.845	0.849	0.853	0.847	0.846

Appendix

Table 9: Exact values plotted in Figure 7 (augmented shuffling ablation on Formula, batch size 60).

Framework	Configuration	Accuracy	Time (s)
ACE	Seed baseline	0.700	–
	Normal parallel (bs 5)	0.860	27906
	Normal parallel (bs 10)	0.870	8704
	Normal parallel (bs 20)	0.805	5955
	Normal parallel (bs 50)	0.730	3141
	Normal parallel (bs 100)	0.740	1955

Appendix

Table 10: Exact values plotted in Figure 8 (GPT-OSS 120B on Formula).

Method	Coverage	Specificity
ACE (batch 1, sequential)	90.4 $\pm$ 3.6	96.0 $\pm$ 1.2
ACE (batch 60)	59.6 $\pm$ 13.0	90.6 $\pm$ 3.1
ACE (batch 125)	71.4 $\pm$ 2.2	73.8 $\pm$ 1.8
Combee + ACE	83.6 $\pm$ 5.0	93.8 $\pm$ 1.6

Appendix

Table 11: LLM-as-judge evaluation of playbook quality on FiNER (mean

\pm

std over 5 judging runs with Gemini-3.5-Flash).

		Time (s)		Cost ($)		Reflect Tokens		Curate Tokens
Method (Batch Size)	LLM Calls	Reflect	Curate	Reflect	Curate	Input	Output	Input	Output
Combee (bs 30)	84	65.68	96.12	0.1086	0.0659	266,314	33,365	158,954	20,906
ACE (bs 1)	120	1465.52	1580.42	0.1087	0.1405	266,332	33,470	366,772	37,692
ACE (bs 30)	72	64.66	87.65	0.1087	0.0279	266,338	33,411	61,542	10,228

Appendix

Table 12: Per-stage cost and latency breakdown on Terminal-Bench 2.0 (ACE as the base prompt learning method).

LLM Input Optimization

See all 22 papers

Apr 13, 2026

Agentic Aggregation for Parallel Scaling of Long-Horizon Agentic Tasks

We study parallel test-time scaling for long-horizon agentic tasks such as agentic search and deep research, where multiple rollouts are generated in parallel and aggregated into a final response. While such scaling has proven effective for chain-of-thought reasoning, agentic tasks pose unique challenges: trajectories are long, multi-turn, and tool-augmented, and outputs are often open-ended. Aggregating only final answers discards rich information from trajectories, while concatenating all trajectories exceeds the model's context window. To address this, we propose AggAgent, an aggregation agent that treats parallel trajectories as an environment. We equip it with lightweight tools to inspect candidate solutions and search across trajectories, enabling it to navigate and synthesize information on demand. Across six benchmarks and three model families (GLM-4.7, Qwen3.5, MiniMax-M2.5), AggAgent outperforms all existing aggregation methods-by up to 5.3% absolute on average and 10.3% on two deep research tasks-while adding minimal overhead, as the aggregation cost remains bounded by a single agentic rollout. Our findings establish agentic aggregation as an effective and cost-efficient approach to parallel test-time scaling.

Yoonsang Lee, Howard Yen, Xi Ye +1

Princeton Language and Intelligence, Princeton University

Test-Time Scaling Long-Horizon Agent Tasks Long-Horizon LLM Agents

Ask PDF

Aug 27, 2026

Naive Prompt Optimization: Rethinking the Need for Complex Prompt Search

Efficiently improving autonomous agents across diverse tasks is central to accelerating recursive self-improvement (RSI) in agentic AI, with prompt optimization emerging as a promising approach capable of delivering performance gains comparable to those achieved by fine-tuning model weights, while reducing computational costs in both optimization and serving. However, recent developments increasingly favor unnecessarily complex prompt optimizers. We introduce Naive Prompt Optimization (NPO), a lightweight single-lineage method that iteratively revises prompts using a teacher model with rollout feedback. NPO achieves comparable or better performance than GEPA with fewer rollouts, and its advantage increases with stronger teacher models, suggesting that stronger teacher reasoning can partially substitute for optimizer-side search complexity. In interactive games, NPO remains broadly competitive with GEPA, while GRPO performs better on some tasks less amenable to prompt optimization. We also show that NPO-optimized prompts elicit similar performance improvements when applied verbatim to other student models, especially across models within the same family. Overall, our preliminary results show that simple, linear prompt optimization can rival substantially more sophisticated and complex search procedures.

Yuan Chang, Xiaoqi Chen

Purdue University

Automatic Prompt Optimization Prompt Optimization Large Language Model-Guided Optimization LLM Agent Self-Improvement

Ask PDF

Jun 9, 2026

EEVEE: Towards Test-time Prompt Learning in the Real World for Self-Improving Agents

In this paper, we propose EEVEE, the first multi-dataset test-time prompt learning framework for LLM agents, enabling test-time prompt learning under real-world task streams. Existing methods are largely designed for single-dataset settings, while real-world applications require models to handle heterogeneous input streams drawn from multiple datasets, domains, and task distributions, limiting their practical applicability. To mitigate cross-dataset interference, EEVEE introduces a router that partitions incoming inputs into task clusters and assigns them to suitable prompt configurations. This design is optimized via a router-prompt co-evolution strategy, which employs interleaved router and prompt learning phases to address their mutual dependency. Experiments across multiple datasets demonstrate that the framework improves robustness under heterogeneous data streams while maintaining single-benchmark learning capability and efficiency. Specifically, EEVEE improves average multi-benchmark scores by 10.38 and 24.32 points over Qwen3-4B-Instruct and DeepSeek-V3.2, surpassing SOTA methods GEPA and ACE by up to 37.2% and 48.2%.

Weixian Xu, Shilong Liu, Mengdi Wang

Shanghai Jiao Tong University · Princeton University

Continual Learning for LLM Agents LLM Agent Self-Improvement LLM Agent Training Test-Time Prompt Tuning

Ask PDF

Abstract

LLM Input Optimization

Explore similar work

Agentic Aggregation for Parallel Scaling of Long-Horizon Agentic Tasks

Naive Prompt Optimization: Rethinking the Need for Complex Prompt Search

EEVEE: Towards Test-time Prompt Learning in the Real World for Self-Improving Agents