AdaEvolve: Adaptive LLM Driven Zeroth-Order Optimization

Recent work such as AlphaEvolve has shown that combining LLM-driven optimization with evolutionary search can effectively improve programs, prompts, and algorithms across domains. In this paradigm, previously evaluated solutions are reused to guide the model toward new candidate solutions. Crucially, the effectiveness of this evolution process depends on the search strategy: how prior solutions are selected and varied to generate new candidates. However, most existing methods rely on fixed search strategies with predefined knobs (e.g., explore-exploit ratios) that remain static throughout execution. While effective in some settings, these approaches often fail to adapt across tasks, or even within the same task as the search space changes over time. We introduce EvoX, an adaptive evolution method that optimizes its own evolution process. EvoX jointly evolves candidate solutions and the search strategies used to generate them, continuously updating how prior solutions are selected and varied based on progress. This enables the system to dynamically shift between different search strategies during the optimization process. Across nearly 200 real-world optimization tasks, EvoX outperforms existing AI-driven evolutionary methods including AlphaEvolve, OpenEvolve, GEPA, and ShinkaEvolve on the majority of tasks.

LLM-driven program evolution can discover high-quality programs, but high API cost and run-to-run variance hinder reliable progress. We propose TurboEvolve, a multi-island evolutionary framework that improves sample efficiency and stability under matched API-cost budgets. Inspired by multiple-offspring strategies in evolutionary algorithms, TurboEvolve introduces Verbalized Sampling, which elicits K diverse candidates in one LLM call while using self-assigned probabilities only as generation signals, together with an online scheduler that adapts K to expand exploration under stagnation and reduce redundant generation during steady progress. To exploit existing solution pools, TurboEvolve further introduces island-aware initialization, which clusters seed programs into mode-specific subpopulations and applies controlled cross-cluster mixing with elitist preservation to balance diversity, sharing, and refinement. We also curate a reusable cross-task solution-pool dataset to support warm-start initialization studies and matched-resource comparisons. Across mathematical discovery and GPU-kernel optimization tasks, TurboEvolve achieves stronger cost-quality trade-offs and more stable performance than controlled baselines, while reaching competitive reported scores.

Recent LLM-guided evolutionary search methods have shown that iterative program mutation can discover strong algorithms, but they typically optimize each task independently, even when related tasks share reusable structure. We study evolutionary multitask optimization (EMO) for LLM-guided program discovery and propose EMO-STA (Shared-Then-Adapt), a two-stage framework that first evolves a shared archive of executable programs across a task family and then adapts selected shared candidates to each target task. Within EMO-STA, we explore multiple adaptation strategies, including warm-starting from the shared archive, adapting the best average shared program, and adapting the shared program that performs best on each target task. Across eight task families spanning continuous optimization, geometric construction, modeling, and algorithmic optimization, EMO-STA improves over single-task evolution under matched evolutionary-iteration budgets in most settings, with STA Best-Local providing the strongest in-distribution adaptation and STA Best-Shared yielding robust transfer to unseen tasks. Iteration-allocation experiments show that devoting a substantial fraction of the family-level iteration budget to shared evolution is effective, with roughly balanced shared and aggregate adaptation budgets often yielding the best performance. ARC and time-series feature-engineering case studies further suggest that shared evolution can mitigate overfitting in low-evidence settings by favoring programs that perform well across related tasks rather than exploiting brittle task-specific artifacts.