Helix: Evolutionary Reinforcement Learning for Open-Ended Scientific Problem Solving

Chang Su; Zhongkai Hao; Zhizhou Zhang; Zeyu Xia; Youjia Wu; Hang Su; Jun Zhu

cs.LGMar 8, 2026

Helix: Evolutionary Reinforcement Learning for Open-Ended Scientific Problem Solving

Authors: Chang Su, Zhongkai Hao, Zhizhou Zhang, Zeyu Xia, Youjia Wu, Hang Su, Jun Zhu

Organizations: Bosch (China) Investment Co., Ltd. · Tsinghua-Bosch Joint Center for ML, Tsinghua University · Dept. of Comp. Sci. & Techn., Tsinghua University · Dept. of EE, Tsinghua University

Abstract

Large language models (LLMs) with reasoning abilities have demonstrated growing promise for tackling complex scientific problems. Yet such tasks are inherently domain-specific, unbounded and open-ended, demanding exploration across vast and flexible solution spaces. Existing approaches, whether purely learning-based or reliant on carefully designed workflows, often suffer from limited exploration efficiency and poor generalization. To overcome these challenges, we present HELIX -- a Hierarchical Evolutionary reinforcement Learning framework with In-context eXperiences. HELIX introduces two key novelties: (i) a diverse yet high-quality pool of candidate solutions that broadens exploration through in-context learning, and (ii) reinforcement learning for iterative policy refinement that progressively elevates solution quality. This synergy enables the discovery of more advanced solutions. On the circle packing task, HELIX achieves state-of-the-art result with a sum of radii of 2.63598308 using only a 14B model. Across standard machine learning benchmarks, HELIX further surpasses GPT-4o with a carefully engineered pipeline, delivering an average F1 improvement of 5.95 points on the Adult and Bank Marketing datasets.

Figures & tables

Panel of Figure 1 — Figure 1: The figure demonstrates how our framework progressively discovers new insights and refines solutions over iterations. (a): Reward curve for the housing dataset optimization, where improvements are achieved through iterative adoption of better models, parameter tuning, and feature engineering, with the final reward of 1.758 corresponding to an RMSE of 1.747. (b): Reward curves for the beam and inductor design tasks, where the algorithm explores novel geometries and combines favorable structural features to enhance performance.

Figure 2: Illustration of HELIX framework. The workflow begins with a dataset containing task descriptions and a pool of initial solutions, which are taken by LLM as inputs. The LLM will modify and update the original solution and generate a new one, represented as descendants in lineage tree. After the evaluation pipeline, the resulting reward-labeled solutions will be used to update policy parameters via reinforcement learning. Those samples will also be selected by NSGA-II algorithm to construct promising yet diverse candidate solutions for population evolution.

		Task Specific Methods		Direct Prompt		Open Evolve		Ours
Machine Learning	Tasks	LightGBM	RRL	Qwen	GPT-4o	Qwen	GPT-4o	-
	Adult Income $\uparrow$	80.36	80.72	73.72	76.91	76.90	72.27	82.07
	Bank Marketing $\uparrow$	75.28	76.32	0.00	76.91	75.66	78.54	80.65
	Boston Housing $\downarrow$	3.258	3.966	3.149	3.031	2.937	2.937	1.747
	Transparent Conductors $\downarrow$	0.060	NA	0.060	0.059	0.059	0.056	0.049
Physics Simulation	Tasks	Parameter Scan	Topology Opt	Qwen	GPT-4o	Qwen	GPT-4o	-

Table 1: Results of main experiments. All values correspond to the best outcome obtained across all attempts. We use

\uparrow

to indicate that larger values correspond to better performance, and

\downarrow

represents the opposite. We highlighted the best results in each task in bold . "NA" denotes non-convergence or unsuitability for given case.

Panel of Figure 3 — Figure 3: Convergence analysis on the Inductor and Adult tasks. The curves show the progressive improvement of average reward and validity during training, demonstrating that our framework effectively leverages reinforcement learning feedback and evolutionary dynamics to produce increasingly valid and high-quality solutions.

Panel of Figure 4 — Figure 4: Ablation analysis of framework components. (a): Maximum reward achieved by different ablation variants. (b): Curve of epoch-wise maximum reward on the Circle Packing task, highlighting the critical role of balancing diversity and quality for stable optimization. (c): Curve of epoch-wise maximum reward on the Boston Housing task, showing the necessity of combining reinforcement learning with evolutionary guidance.

Panel of Figure 5 — Figure 5: Scaling analysis of model parameter scale on (a): magnetic torque maximization and (b): inductor design tasks.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: The RMS pressure field of an acoustic demultiplexer at frequency level 7500 Hz. The RMS pressure field in log scale is proportional to the acoustic power.

Figure 7: The magnetic flux density field generated by an iron core subject to a uniform magnetic field boundary condition. The distorted magnetic flux density field then applies a torque on the iron core.

Figure 8: The von Mises stress field generated by applying bending moment, shear force, and twisting moment on a cantilever beam cross section design.

Figure 9: Temperature distribution of the meta-material under three loading conditions. The effective properties are calculated based on temperature distributions according to the homogenization theory.

Figure 10: The magnetic flux density norm field generated by an inductor. The copper coils are excited by a $500$ A, $1000$ Hz sinusoidal current.

Figure 11: The landscape and global minimum point of Eggholder function with constraints $-512\leq x,y\leq 512$

Figure 12: The landscape and global minimum point of Mishra’s Bird function with constraints $x_{1}\in[-10,0]$ , $x_{2}\in[-6.5,0]$ and $x_{1}^{2}+x_{2}^{2}\geq 25$ .

Figure 13: The landscape and global minimum point of the 2-D Keanes Bump function. The feasible region, a small part of the search space, is the only area with finite function values.

Method	Score (Sum of Radii)
Direct Prompt (Qwen 14B)	1.673
OpenEvolve (Qwen 14B)	1.586
OpenEvolve (Qwen 32B)	1.956
HELIX (Qwen 14B)	2.636

Appendix

Table 2: Performance Comparison on Circle Packing Task

Algorithm	Dynamics Equation	Drift $D(v)$	Selection $w(v)$
Evolve	$v^{\prime}=v+\sigma\xi$	$\mathbf{0}$	$\exp(\alpha r(v))$
HELIX	$v^{\prime}=v+\eta\nabla r(v)+\sigma\xi$	$\eta\nabla r(v)$	$1$ ( $\alpha=0$ )

Appendix

Table 3: Comparison of Algorithm Dynamics

Algorithm	Gaussian Form $p^{*}(\mathbf{v})$	Variance $\Sigma^{2}$	Scaling vs. $\sigma$
HELIX	$\displaystyle\propto\exp\left(-\frac{\eta k}{\sigma^{2}}\\|v\\|^{2}\right)$	$\displaystyle\Sigma^{2}_{Helix}=\frac{\sigma^{2}}{2\eta k}$	$\propto\sigma^{2}$ (Sharper)
Evolve	$\displaystyle\propto\exp\left(-\frac{\sqrt{\alpha k}}{2\sigma}\\|v\\|^{2}\right)$	$\displaystyle\Sigma^{2}_{Evo}=\frac{\sigma}{\sqrt{\alpha k}}$	$\propto\sigma$ (Broader)

Appendix

Table 4: Comparison of Stationary Distributions under Quadratic Reward

	$\displaystyle q_{i}=\mathrm{ConstructPrompt}($	$\displaystyle\{p\}\cup\{s_{t,i},R(s_{t,i}),F(s_{t,i})\}\cup{}$
		$\displaystyle\{f^{(k)}(s_{t,i}),R(f^{(k)}(s_{t,i})),F(f^{(k)}(s_{t,i}))\}_{k=1}^{n-1}).$

Appendix

Algorithm 1 HELIX Framework

Figure 14: Acoustic pressure distribution in the optimal acoustic demultiplexer design obtained by our framework, with reward 14.260.

Figure 15: Magnetic flux density norm field for the optimized iron core configuration identified by our framework, achieving reward 11.045.

Figure 16: Von Mises stress distribution of the optimized beam design obtained by our framework, with reward 17.298.

Panel of Figure 17 — Figure 17: Optimized temperature fields of the meta-material designed by our framework under two different load conditions, achieving reward 1.278.

Figure 18: Magnetic flux density norm field of the best inductor configuration identified by our framework, achieving reward 9.609.

Figure 19: Arrangement of 26 circles within a square obtained by our framework, achieving a record-breaking sum of radii of 2.63598308.

Figure 20: Optimized circle packing of 26 disks within a unit disk by our model, yielding a sum of radii of 4.664465.

Evolutionary RL

See all 13 papers

Evolution Fine-Tuning: Learning to Discover Across 371 Optimization Tasks
Jun 27, 2026Young-Jun Lee, Seungone Kim, Minki Kang +5Large Language Model-Guided Optimization Evolutionary Optimization
University of Minnesota · Carnegie Mellon University · KAIST +3
LLM-Guided Program Evolution for Circle Packing: Breaking 10 Packomania Records for $28
Sep 4, 2026Wes SanderAutomated Algorithm Discovery Large Language Model-Guided Optimization
Practical Systems Mason, Ohio, USA
CausalEvolve: Towards Open-Ended Discovery with Causal Scratchpad
Mar 15, 2026Yongqiang Chen, Chenxi Liu, Zhenhao Chen +3AI Agents for Scientific Discovery Large Language Model-Guided Scientific Discovery
MBZUAI · Carnegie Mellon University · TMLR Group, Hong Kong Baptist University +1

Jun 27, 2026J/K move · Enter open · S save

Evolution Fine-Tuning: Learning to Discover Across 371 Optimization Tasks

Young-Jun Lee, Seungone Kim, Minki Kang, Alistair Cheong Liang Chuen+4

University of Minnesota · Carnegie Mellon University · KAIST+3

Large Language Model-Guided Optimization Evolutionary Optimization LLM Fine-Tuning Supervised Fine-Tuning Learning to Optimize Large Language Model-Guided Scientific Discovery+1

Open paper AskPDF arXiv

Would experience designing faster GPU kernels also help close in on a long-standing open mathematical conjecture? Large Language Models (LLMs) integrated into evolutionary search have recently produced state-of-the-art solutions on optimization tasks, including open mathematical conjectures, GPU kernel design, scientific law discovery, and combinatorial puzzles. To achieve this, prior work applied search scaffolds to one target task at a time, so every new problem is approached from scratch and the experience accumulated during search is discarded once the model finishes its attempt. This leaves the capability of iteratively evolving a solution (e.g., knowing which part to mutate and how, deciding when to backtrack) entirely in the scaffold rather than in the model itself. Whether the model itself could acquire this capability and reuse it across different tasks has been largely unexamined. To address this, we introduce Evolution Fine-Tuning (EFT), a mid-training paradigm that teaches LLMs to evolve solutions across tasks by converting evolutionary search trajectories into supervision. We construct Finch Collection, a 156K-trajectory dataset spanning 10 domains and 371 optimization tasks, and fine-tune open-source LLMs from 2B to 9B parameters. Empirically, EFT confers cross-task generalization: across 22 held-out tasks, our models surpass their base counterparts by 10.22% on average. Furthermore, when paired with test-time RL, our model matches state-of-the-art performance on two circle-packing tasks and outperforms its base-model counterpart on the Erdős minimum-overlap problem. EFT thus serves as a "practice phase" for general-purpose discovery agents that do not solve new problems from scratch.

Abstract

Evolutionary RL

Explore similar work

Evolution Fine-Tuning: Learning to Discover Across 371 Optimization Tasks

LLM-Guided Program Evolution for Circle Packing: Breaking 10 Packomania Records for $28

CausalEvolve: Towards Open-Ended Discovery with Causal Scratchpad