Helix: Evolutionary Reinforcement Learning for Open-Ended Scientific Problem Solving
Authors: Chang Su, Zhongkai Hao, Zhizhou Zhang, Zeyu Xia, Youjia Wu, Hang Su, Jun Zhu
Organizations: Bosch (China) Investment Co., Ltd. · Tsinghua-Bosch Joint Center for ML, Tsinghua University · Dept. of Comp. Sci. & Techn., Tsinghua University · Dept. of EE, Tsinghua University
Large language models (LLMs) with reasoning abilities have demonstrated growing promise for tackling complex scientific problems. Yet such tasks are inherently domain-specific, unbounded and open-ended, demanding exploration across vast and flexible solution spaces. Existing approaches, whether purely learning-based or reliant on carefully designed workflows, often suffer from limited exploration efficiency and poor generalization. To overcome these challenges, we present HELIX -- a Hierarchical Evolutionary reinforcement Learning framework with In-context eXperiences. HELIX introduces two key novelties: (i) a diverse yet high-quality pool of candidate solutions that broadens exploration through in-context learning, and (ii) reinforcement learning for iterative policy refinement that progressively elevates solution quality. This synergy enables the discovery of more advanced solutions. On the circle packing task, HELIX achieves state-of-the-art result with a sum of radii of 2.63598308 using only a 14B model. Across standard machine learning benchmarks, HELIX further surpasses GPT-4o with a carefully engineered pipeline, delivering an average F1 improvement of 5.95 points on the Adult and Bank Marketing datasets.
Figures & tables
Figure 1: The figure demonstrates how our framework progressively discovers new insights and refines solutions over iterations. (a): Reward curve for the housing dataset optimization, where improvements are achieved through iterative adoption of better models, parameter tuning, and feature engineering, with the final reward of 1.758 corresponding to an RMSE of 1.747. (b): Reward curves for the beam and inductor design tasks, where the algorithm explores novel geometries and combines favorable structural features to enhance performance.
Figure 2: Illustration of HELIX framework. The workflow begins with a dataset containing task descriptions and a pool of initial solutions, which are taken by LLM as inputs. The LLM will modify and update the original solution and generate a new one, represented as descendants in lineage tree. After the evaluation pipeline, the resulting reward-labeled solutions will be used to update policy parameters via reinforcement learning. Those samples will also be selected by NSGA-II algorithm to construct promising yet diverse candidate solutions for population evolution.
Task Specific Methods
Direct Prompt
Open Evolve
Ours
Machine Learning
Tasks
LightGBM
RRL
Qwen
GPT-4o
Qwen
GPT-4o
-
Adult Income ↑
80.36
80.72
73.72
76.91
76.90
72.27
82.07
Bank Marketing ↑
75.28
76.32
0.00
76.91
75.66
78.54
80.65
Boston Housing ↓
3.258
3.966
3.149
3.031
2.937
2.937
1.747
Transparent Conductors ↓
0.060
NA
0.060
0.059
0.059
0.056
0.049
Physics Simulation
Tasks
Parameter Scan
Topology Opt
Qwen
GPT-4o
Qwen
GPT-4o
-
Table 1: Results of main experiments. All values correspond to the best outcome obtained across all attempts. We use ↑ to indicate that larger values correspond to better performance, and ↓ represents the opposite. We highlighted the best results in each task in bold . "NA" denotes non-convergence or unsuitability for given case.
Figure 3: Convergence analysis on the Inductor and Adult tasks. The curves show the progressive improvement of average reward and validity during training, demonstrating that our framework effectively leverages reinforcement learning feedback and evolutionary dynamics to produce increasingly valid and high-quality solutions.
Figure 4: Ablation analysis of framework components. (a): Maximum reward achieved by different ablation variants. (b): Curve of epoch-wise maximum reward on the Circle Packing task, highlighting the critical role of balancing diversity and quality for stable optimization. (c): Curve of epoch-wise maximum reward on the Boston Housing task, showing the necessity of combining reinforcement learning with evolutionary guidance.
Figure 5: Scaling analysis of model parameter scale on (a): magnetic torque maximization and (b): inductor design tasks.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: The RMS pressure field of an acoustic demultiplexer at frequency level 7500 Hz. The RMS pressure field in log scale is proportional to the acoustic power.
Figure 7: The magnetic flux density field generated by an iron core subject to a uniform magnetic field boundary condition. The distorted magnetic flux density field then applies a torque on the iron core.
Figure 8: The von Mises stress field generated by applying bending moment, shear force, and twisting moment on a cantilever beam cross section design.
Figure 9: Temperature distribution of the meta-material under three loading conditions. The effective properties are calculated based on temperature distributions according to the homogenization theory.
Figure 10: The magnetic flux density norm field generated by an inductor. The copper coils are excited by a 500 A, 1000 Hz sinusoidal current.
Figure 11: The landscape and global minimum point of Eggholder function with constraints −512≤x,y≤512
Figure 12: The landscape and global minimum point of Mishra’s Bird function with constraints x1∈[−10,0] , x2∈[−6.5,0] and x12+x22≥25 .
Figure 13: The landscape and global minimum point of the 2-D Keanes Bump function. The feasible region, a small part of the search space, is the only area with finite function values.
Method
Score (Sum of Radii)
Direct Prompt (Qwen 14B)
1.673
OpenEvolve (Qwen 14B)
1.586
OpenEvolve (Qwen 32B)
1.956
HELIX (Qwen 14B)
2.636
Appendix
Table 2: Performance Comparison on Circle Packing Task
Algorithm
Dynamics Equation
Drift D(v)
Selection w(v)
Evolve
v′=v+σξ
0
exp(αr(v))
HELIX
v′=v+η∇r(v)+σξ
η∇r(v)
1 ( α=0 )
Appendix
Table 3: Comparison of Algorithm Dynamics
Algorithm
Gaussian Form p∗(v)
Variance Σ2
Scaling vs. σ
HELIX
∝exp(−σ2ηk∥v∥2)
ΣHelix2=2ηkσ2
∝σ2 (Sharper)
Evolve
∝exp(−2σαk∥v∥2)
ΣEvo2=αkσ
∝σ (Broader)
Appendix
Table 4: Comparison of Stationary Distributions under Quadratic Reward
Would experience designing faster GPU kernels also help close in on a long-standing open mathematical conjecture? Large Language Models (LLMs) integrated into evolutionary search have recently produced state-of-the-art solutions on optimization tasks, including open mathematical conjectures, GPU kernel design, scientific law discovery, and combinatorial puzzles. To achieve this, prior work applied search scaffolds to one target task at a time, so every new problem is approached from scratch and the experience accumulated during search is discarded once the model finishes its attempt. This leaves the capability of iteratively evolving a solution (e.g., knowing which part to mutate and how, deciding when to backtrack) entirely in the scaffold rather than in the model itself. Whether the model itself could acquire this capability and reuse it across different tasks has been largely unexamined. To address this, we introduce Evolution Fine-Tuning (EFT), a mid-training paradigm that teaches LLMs to evolve solutions across tasks by converting evolutionary search trajectories into supervision. We construct Finch Collection, a 156K-trajectory dataset spanning 10 domains and 371 optimization tasks, and fine-tune open-source LLMs from 2B to 9B parameters. Empirically, EFT confers cross-task generalization: across 22 held-out tasks, our models surpass their base counterparts by 10.22% on average. Furthermore, when paired with test-time RL, our model matches state-of-the-art performance on two circle-packing tasks and outperforms its base-model counterpart on the Erdős minimum-overlap problem. EFT thus serves as a "practice phase" for general-purpose discovery agents that do not solve new problems from scratch.
Young-Jun Lee, Seungone Kim, Minki Kang +5
University of Minnesota · Carnegie Mellon University · KAIST +3
We present Discovery Loop, a lightweight system that uses a large language model (LLM) to iteratively evolve optimization algorithms. Starting from a simple seed solver, the LLM proposes algorithmic improvements guided by a scoreboard of results and a history of prior ideas. Each candidate is evaluated against an independent verifier; improvements are kept and failures discarded. Applied to the Packomania circle-packing benchmark (csqv: maximize the sum of radii of N variable-radius circles in the unit square), the system improved the best known solutions for 10 values of N in the range 101-114, with gains of 2.4%-5.4% over prior records, all within 15 iterations and at a total LLM cost of $27.72. These results have been independently accepted by Packomania. We describe the method, analyze cost-efficiency dynamics including an adaptive plateau-detection mechanism, and discuss implications for democratizing automated scientific discovery.
Evolve-based agent such as AlphaEvolve is one of the notable successes in using Large Language Models (LLMs) to build AI Scientists. These agents tackle open-ended scientific problems by iteratively improving and evolving programs, leveraging the prior knowledge and reasoning capabilities of LLMs. Despite the success, existing evolve-based agents lack targeted guidance for evolution and effective mechanisms for organizing and utilizing knowledge acquired from past evolutionary experience. Consequently, they suffer from decreasing evolution efficiency and exhibit oscillatory behavior when approaching known performance boundaries. To mitigate the gap, we develop CausalEvolve, equipped with a causal scratchpad that leverages LLMs to identify and reason about guiding factors for evolution. At the beginning, CausalEvolve first identifies outcome-level factors that offer complementary inspirations in improving the target objective. During the evolution, CausalEvolve also inspects surprise patterns during the evolution and abductive reasoning to hypothesize new factors, which in turn offer novel directions. Through comprehensive experiments, we show that CausalEvolve effectively improves the evolutionary efficiency and discovers better solutions in 4 challenging open-ended scientific tasks.
Yongqiang Chen, Chenxi Liu, Zhenhao Chen +3
MBZUAI · Carnegie Mellon University · TMLR Group, Hong Kong Baptist University +1