From Hand-Crafted to LLM-Based Variation Operators in Metaheuristics: A Tutorial

Camilo Chacón Sartori; Guillem Rodríguez-Corominas; Christian Blum

cs.NESep 13, 2026

From Hand-Crafted to LLM-Based Variation Operators in Metaheuristics: A Tutorial

Authors: Camilo Chacón Sartori, Guillem Rodríguez-Corominas, Christian Blum

Organizations: Apeiron Intelligence, Barcelona, Spain · Artificial Intelligence Research Institute (IIIA-CSIC), Bellaterra, Spain · Universitat Politècnica de Catalunya (UPC), Barcelona, Spain

Abstract

Large language models (LLMs) are increasingly being employed as variation operators in metaheuristics, generating or modifying candidate solutions, heuristics, or programs inside iterative search loops. This shift reframes variation as a model call conditioned on different types of information. We introduce an operator-level framework with two descriptors: (1) the type of prompt-conditioning information at variation time (Numeric, Symbolic, Linguistic), and (2) artifact persistence, identifying what survives the model call (Transient, Amortized, Transfer). The tutorial shows how to classify, build, and select these operators through a worked build template, a method survey, an evidence table, and a cost-aware decision guide.

Figures & tables

Work	Type of work	Organizing axis	After reading, you can …
LLM4AD ( Liu et al., 2026b ) (CSUR)	survey	the LLM’s role : optimizer / predictor / extractor / designer	locate a method by the role the LLM plays
Da Ros et al. ( Da Ros et al., 2026 ) (CSUR)	systematic review (PRISMA)	task, architecture, dataset, application	find the census of LLM-for-CO (combinatorial optimization) studies
Wu et al. ( Wu et al., 2025 )	survey + roadmap	direction of enhancement: LLM-enhanced EA vs. EA-enhanced LLM	see how evolutionary computation (EC) and LLMs enhance each other
Zhang et al. ( Zhang et al., 2025b )	survey	integration stage: modeling $\to$ solving	trace the modeling-to-solving pipeline
This tutorial	tutorial	what the operator is conditioned on : Numeric / Symbolic / Linguistic	build, classify, and choose an operator

Table 1. Complementary positioning of this tutorial relative to recent surveys.

Panel of Figure 1 — Figure 1. Token-level view of an LLM call. A prompt containing the representation $r$ is converted into a completion by repeatedly sampling a next token conditioned on the prompt and the tokens already generated. The completed text is then parsed as a candidate artifact $x^{\prime}$ . A prompt representation is tokenized, passed through an LLM, decoded into a completion, and parsed into a candidate artifact.

Panel of Figure 2 — Figure 2. The operator as a proposal distribution $q(x^{\prime}\mid r)$ . Left: a classical fixed kernel produces a narrow proposal centred on the current state $x$ . Right: an LLM prompt can induce a different, sometimes broader distribution over candidate artifacts. Two schematic proposal distributions compare a classical neighborhood around a current state with an LLM-induced distribution shaped by a prompt representation.

Panel of Figure 3 — Figure 3. The conditioning lens made concrete: one TSP operator (“propose the next construction heuristic”) conditioned on numeric, symbolic code, or linguistic content. These alternatives change the prompt representation $r$ supplied at the variation step and therefore shape the induced proposal distribution shown in Fig. 2 .

Panel of Figure 4 — Figure 4. What persists, as a descriptive attribute. Transient : the LLM remains inside the loop. Amortized : an offline call emits an artifact that later runs at near-zero runtime cost. Transfer : a source-domain artifact or principle is reused in a new domain without rediscovering it.

Panel of Figure 5 — Figure 5. Anatomy of an LLM variation operator: prompt, sample, parse-and-validate (with a repair path for invalid output), and loop integration with an optional feedback channel.

Panel of Figure 6 — Figure 6. An illustrative human–operator exchange: the designer authors the prompt, and the LLM operator samples a candidate that passes validation and improves the objective value.

Panel of Figure 7 — Figure 7. Representative method classifications on the conditioning–persistence map (the lens) of Section 3 . The horizontal axis records what conditions the LLM call; the vertical axis records what survives it. The $\dagger$ symbol marks prompt-optimization boundary cases whose placement depends on prompt design.

Panel of Figure 8 — Figure 8. Decision guide for choosing whether and how to use an LLM variation operator. The verification bar summarizes the required reporting checks.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Channels present	Dominance (ablation/source)	Drop-channel reasoning
OPRO	(solution, score) trajectory	Numeric (boundary)	drop the score trajectory and no signal steers the next proposal
EvoPrompt	parent prompt text, scores	Numeric (boundary)	scored parents drive selection, while parent text is the artifact varied; boundary label retained
LMX	parent solutions (few-shot)	Numeric	drop the parents and there is no crossover to perform
FunSearch	sampled high-scoring programs, scores	Symbolic	drop the program text and there is nothing to mutate
LLaMEA	algorithm code, score, error note	Symbolic	drop the code and no algorithm remains; the error note is annotated
MCTS-AHD	program, tree-search state, scores	Symbolic	drop the code and the tree explores nothing executable

Appendix

Table 3. Drop-channel classification audit for the methods of Table 2 . The table column Dominance is the source-described operative channel, audited where possible by removing or neutralizing that signal ( Numeric , Symbolic , Linguistic ); the other channels present are listed but not ranked.

Method	Dominant Conditioning	Persistence	Conditions on $\rightarrow$ emits
EvoLLM ( Lange et al., 2024 )	Numeric	Transient	fitness-ranked solution vectors $\rightarrow$ distribution/candidate update
LMEA ( Liu et al., 2024d )	Numeric	Transient	parent tours + tour lengths (TSP instantiation) $\rightarrow$ offspring tour (LLM crossover/mutation)
DiscoPOP ( Lu et al., 2024 )	Symbolic	Amortized	prior objective/loss code + eval metrics $\rightarrow$ preference-optimization loss; held-out-task reuse
ELM ( Lehman et al., 2023 )	Symbolic	Transient	parent code + edit instruction (diff model) $\rightarrow$ program; adjacent background anchor (Section 5 )
EvoPrompting ( Chen et al., 2023 )	Symbolic	Amortized	parent architecture code + fitness $\rightarrow$ NN architecture code
GI-LLM ( Brownlee et al., 2023 )	Symbolic	Transient	source code + edit type $\rightarrow$ code patch (GI mutation)

Appendix

Table 4. Extended coverage: LLM variation operators by dominant conditioning channel and persistence. Entries are read against their primary sources. Channel and persistence assignments follow Section 3 ; this is not a systematic census.

Large Language Model-Guided Optimization

Large Language Model-Guided Optimization uses language models to propose, steer, and refine solutions to hard search and design problems.

See all 605 papers

Large Language Models as Optimizers: A Survey of Direct vs. Tool-Augmented Approaches and Their Performance Frontiers
Apr 9, 2026Roko Peran, Luka Hobor, Mihael Kovac +1Large Language Model-Guided Optimization Natural Language to Optimization Modeling
Student at University of Zagreb Faculty of Electrical Engineering and Computing, Zagreb, Croatia · University of Zagreb Faculty of Electrical Engineering and Computing, Zagreb, Croatia
How to Guide LLM Generation: Dual-Surrogate Guided Search for Automated Heuristic Design
Jul 15, 2026Yuhan Wang, Chaoda Peng, Xingyu Wu +2Surrogate-Assisted Optimization Automated Heuristic Design
College of Mathematics and Informatics, South China Agricultural University · Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University · College of Artificial Intelligence, Nankai University
Breaking Validity-Induced Boundaries to Expand Algorithm Search Space: A Two-Stage AST-Based Operator for LLM-Driven Automated Heuristic Evolution
Apr 3, 2026Sun Shengming, Shi JialongAutomated Heuristic Design Large Language Model-Guided Optimization
School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, China

Apr 9, 2026J/K move · Enter open · S save

Large Language Models as Optimizers: A Survey of Direct vs. Tool-Augmented Approaches and Their Performance Frontiers

Roko Peran, Luka Hobor, Mihael Kovac, Mario Brcic

Student at University of Zagreb Faculty of Electrical Engineering and Computing, Zagreb, Croatia · University of Zagreb Faculty of Electrical Engineering and Computing, Zagreb, Croatia

Large Language Model-Guided Optimization Natural Language to Optimization Modeling Automated Algorithm Discovery

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly involved in complex mathematical optimization, even if the pragmatic user who triggers them is unaware of it. After all, many real-world problems reduce to the search for better or the best solutions. The field of LLM-as-optimizer has three paradigms: direct optimization, tool-augmented optimization, and tool-creating optimization. Direct optimization uses iterative prompting and heuristic generation to navigate solution spaces. Tool-augmented optimization translates natural language problems into formal specifications and orchestrates external solvers. Tool-creating optimization goes further, using LLMs to discover reusable algorithms or heuristics that can be deployed at zero marginal LLM cost. We describe current performance frontiers based on the benchmarks from the literature. We identify the critical reasoning gap in current architectures and argue for trade-offs between the future potential of direct optimization and the auditability of tool-augmented optimization. Even future, more powerful models might opt for tool-making to improve operational efficiency for repetitive families of problems.

Abstract

Large Language Model-Guided Optimization

Explore similar work

Large Language Models as Optimizers: A Survey of Direct vs. Tool-Augmented Approaches and Their Performance Frontiers

How to Guide LLM Generation: Dual-Surrogate Guided Search for Automated Heuristic Design

Breaking Validity-Induced Boundaries to Expand Algorithm Search Space: A Two-Stage AST-Based Operator for LLM-Driven Automated Heuristic Evolution