Combining LLMs and Genetic Search for ARC-AGI-2

Val Dyachenko

cs.NESep 23, 2026

Combining LLMs and Genetic Search for ARC-AGI-2

Authors: Val Dyachenko

Abstract

LLMs can generate programs for ARC-AGI-2 tasks, but the provided compute only allows a small number of attempts to generate, debug and validate solutions. Genetic algorithms can search and test many more programs, but random search rarely starts in a useful neighborhood of the solution space. We combine the two methods through a compact domain specific language (DSL). First, a quantized Qwen3.5-4B LLM generates an initial set of programs for each ARCAGI-2 task. Then, we use those programs to seed an initial population of starting programs, and use genetic algorithms to evolve these programs towards a solution to the given task. The DSL is designed such that every mutated program remains valid and can be executed. The initial programs proposed by the LLM solve 2 (3.3%) of the first 60 tasks of the ARC-2 public evaluation set. The genetic algorithm solves an additional 4, giving 6 correct test outputs in total (10.0%). If we try using evolving solutions without this LLM seeding, we do not arrive at any solutions at all. The results show that genetic search can improve programs generated by LLMs and produce additional correct solutions.

Figures & tables

Panel of Figure 1 — Figure 1: The LLM creates the initial programs. The GA then creates new programs, and the Machine executes and scores them. The LLM is not used during genetic search.

Tensor	Shape	Contents
State	$[P,N,21,30,30]$	Grid channels for $P$ programs and $N$ task examples, including OUT, color masks, objects, and scratch grids.
Registers	$[P,N,176]$	Shared perceived facts plus per-program scratch registers.
Genes	$[P,20,6]$	Twenty fixed-width genes per chromosome.

Table 1: Machine tensors. Each island uses

P=4{,}000

chromosomes.

GA parameter	Value
Islands	2
Population per island	4,000
Chromosome length	20 genes
LLM-derived / random initialization	40% / 60%
Tournament size	2
Gene mutation probability	0.09%

Table 2: Genetic-search parameters used in the reported run.

Condition	Test solves	Solve rate	Interpretation
LLM seed pool only	2/60	3.3%	Correct program already present before evolution.
LLM seed pool + GA	6/60	10.0%	Evolution adds four correct solutions!

Table 3: Separating results from LLM and GA on the same 60 public-evaluation tasks.

Task	Seed programs	Source	Generation	Test output
08ed6ac7	93	LLM	-	Correct
0b17323b	69	LLM	-	Correct
1190e5a7	46	LLM	-	Incorrect
017c7c7b	73	GA	22	Correct
0c786b71	18	GA	35	Correct
0c9aba6e	45	GA	50	Correct

Table 4: All exact training fits. Only rows marked “Correct” are counted as solved.

Task	Combined system	Unseeded GA best after 200 generations
0b17323b	Exact at generation 0	79.73%
017c7c7b	Exact at generation 22	93.50%
0c786b71	Exact at generation 35	83.18%

Table 5: Three illustrative exploratory runs with an unseeded, core-operation-only GA.

Figure 2: Best fitness with LLM seeding and in three random seeded runs. Unfortunately, the random runs do not even reach an exact training task fit.

Figure 3: Best and mean fitness for three solved and three unsolved tasks selected from the 60 task run. The examples illustrate the different convergence behaviors.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Model	Qwen3.5-4B, Q4_K_M GGUF
Topology	One llama-server per GPU; --split-mode none ; all layers on the selected GPU
Parallelism	10 conversation slots per GPU in the production pipeline (12gb VRAM)
Context	35,000 tokens per slot; 350,000-token total server context at 10 slots
KV cache	Q8_0 keys and values; Flash Attention enabled
Decoding mode	Thinking disabled with reasoning budget 0 and chat-template enable_thinking=false

Appendix

Table 6: LLM inference configuration used in the reported run.

#	Operation	Tier	Function
0	NOP	core	No-Op; fills unused chromosome gene positions.
1	COPY	core	Copy one channel (color or object) to another.
2	TRANSLATE	core	Shift content by a row and column offset.
3	RAY_STEP	core	Extend content repeatedly in a given direction.
4	BOOL_OR	core	Union of two channel channel grids.
5	BOOL_AND	core	Intersection of two channel grids.

Appendix

Table 7: The 41-operation DSL: 27 core operations and 14 LLM-gated operations.

Explore similar work

LLM-as-Code: Agentic Programming for Agent Harness
Jun 14, 2026Junjia Qi, Zichuan Fu, Jingtong Gao +4Large Language Model Agents Agent Harness
City University of Hong Kong · Tencent Jarvis Lab
LLM Program Optimization via Retrieval Augmented Search
Jan 31, 2025Sagnik Anupam, Alexander Shypula, Osbert BastaniCode Optimization Parameter-Efficient Adaptation
University of Pennsylvania
optimize_anything: A Universal API for Optimizing any Text Parameter
May 19, 2026Lakshya A Agrawal, Donghyun Lee, Shangyin Tan +11Prompt Optimization Optimization Modeling
UC Berkeley USA · MIT USA

Jun 14, 2026, cs.AIJ/K move · Enter open · S save

LLM-as-Code: Agentic Programming for Agent Harness

Junjia Qi, Zichuan Fu, Jingtong Gao, Wenlin Zhang+3

City University of Hong Kong · Tencent Jarvis Lab

Large Language Model Agents Agent Harness Orchestration

Open paper AskPDF arXiv

Every major LLM agent framework gives the LLM the role of orchestrator; the model decides what to do next, when to call tools, and when to stop. We argue that token explosion, control-flow hallucination, and unreliable completion are not implementation bugs but architectural consequences of assigning the deterministic work of looping, branching, and sequencing to a probabilistic system. A better prompt or a stronger model cannot guarantee the reliability of the LLM agent. We therefore propose Agentic Programming, in which the program governs all control flow, and the LLM is itself part of it, an adaptive component we call LLM-as-Code and invoke only where a task calls for reasoning or generation. Within each call the model keeps full flexibility, but it cannot alter the program's execution path. With control in the program, the LLM's context is built from the execution history's call tree and forms a directed acyclic graph (DAG). Each call's context length is then determined by its call depth rather than by accumulation over steps. A case study of computer-use agents shows that the design is practical, not just a theoretical stance, substantially improving the stability of long visual operation sequences.

Abstract

Explore similar work

LLM-as-Code: Agentic Programming for Agent Harness

LLM Program Optimization via Retrieval Augmented Search

optimize_anything: A Universal API for Optimizing any Text Parameter