Evolutionary Context Search for Automated Skill Acquisition

Qi Sun; Stefan Nielsen; Rio Yokota; Yujin Tang

cs.NEFeb 18, 2026

Evolutionary Context Search for Automated Skill Acquisition

Authors: Qi Sun, Stefan Nielsen, Rio Yokota, Yujin Tang

Organizations: Sakana AI, Japan · Institute of Science Tokyo, Japan

Abstract

Large Language Models cannot reliably acquire new knowledge post-deployment -- even when relevant text resources exist, models fail to transform them into actionable knowledge without retraining. Retrieval-Augmented Generation attempts to bridge this gap by surfacing relevant documents at inference time, yet similarity-based retrieval often fails to identify context that actually improves task performance. We introduce Evolutionary Context Search (ECS), an evolutionary method that searches context combinations using accuracy on a small development set, requiring only inference calls without weight updates. ECS moves beyond semantic similarity to discover non-obvious context pairings that significantly boost performance. Our empirical results show that ECS improves BackendBench by 27% and $τ$ -bench airline by 7%. The evolved contexts are model-agnostic, as those evolved with Gemini-3-Flash transfer effectively to Claude Sonnet and DeepSeek. This suggests that ECS opens a path toward automated context discovery for skill acquisition -- an efficient alternative to manual prompt engineering or costly fine-tuning.

Figures & tables

Figure 1: Evolutionary Context Search. Our method takes a population of text resources and evolves optimized contexts that confer the knowledge required to perform tasks in unseen domains. Each successive generation accumulates task-dependent, knowledge-rich information, effectively searching the corpora to obtain token-efficient contexts that enable novel skill acquisition in LLMs.

Figure 2: Performance comparison on Backend Bench. Our method (ECS) achieves a 27% relative improvement over AST + Hybrid. Detailed result are in Appendix A.2 .

Figure 3: Fitness score during the evolutionary search process.

Method	Pass 1	Pass 2	Pass 3
Plain LLM	0.622	0.489	0.400
Random	0.650	0.556	0.500
BM25	0.641	0.550	0.483
Full	0.717	0.628	0.550
ECS (Ours)	0.767	0.710	0.683

Table 1: Comparison on

\tau^{2}

-Bench. ECS significantly outperforms baselines across all Pass metrics.

Figure 4: Transferability to Unseen Models (BackendBench). Contexts evolved by ECS transfer effectively to models not used during evolution. On DeepSeek-V3.2, where standard retrieval (AST+Dense) fails to provide meaningful gains, ECS unlocks significant capability, yielding a significant improvement.

	Claude-4.5-Sonnet			DeepSeek-V3.2
Metric	Plain	Full	ECS	Plain	Full	ECS
Pass 1	0.600	0.678	0.678	0.583	0.633	0.600
Pass 2	0.506	0.561	0.617	0.478	0.500	0.472
Pass 3	0.450	0.500	0.583	0.417	0.417	0.417
# Tokens	0	5491	848	0	5491	848

Table 2: Transferability on

\tau^{2}

-Bench. With Claude-4.5-Sonnet, ECS matches the Full baseline on Pass 1 and significantly outperforms it on stricter metrics (Pass 2 , Pass 3 ). Notably, ECS achieves comparable performance to the Full baselines with substantially fewer tokens, reducing usage from 5491 to 848, a 6

\times

reduction.

(a) Evolved Context	(b) RAG Baseline	(c) With Evolved Context
hopper/fmha.py (lines 1372–1384)	DeepSeek generates wrong API	DeepSeek generalizes correctly
⬇ 1 for j in range_constexpr (...): 2 acc_qk_mn [ i , j ] = cute.math.exp2 ( 3 scale_softmax_log2 4 * acc_qk_mn [ i , j ]	⬇ 1 input_val = gA [ tidx ] 2 # WRONG: ArithValue is symbolic! 3 result = math.atan ( input_val )	⬇ 1 input_val = gA [ tidx ] 2 # CORRECT: DSL intrinsic 3 result = cute.math.atan ( input_val )
cute.math.* pattern	$\times$ Fails: stdlib on symbolic value	$\checkmark$ Works: DSL intrinsic

Figure 6: Cross-model context transfer from Gemini Flash to DeepSeek V3.2 on BackendBench. Top: (a) Evolved context demonstrates the cute.math.* pattern. (b) Without this context, DeepSeek incorrectly uses Python’s math.atan() on symbolic values, causing JIT compilation failures. (c) With evolved context, DeepSeek correctly generalizes to DSL intrinsics. Bottom: Quantitative results show evolved context improves pass rate from 6.5% to 22.3% (

3.4\times

); e.g., div.Tensor passes 16 out of 18 test cases.

Figure 7: Ablation study. We evaluate variants by removing components. Fitness , Mutation are universally critical. Refinement is domain-dependent: it is essential for resolving conflicting agentic policies (

\tau^{2}

-Bench) but has negligible impact on BackendBench.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Operator	Signature	Description
Trigonometric Functions
	acos	$f:[-1,1]\rightarrow[0,\pi]$	Computes element-wise inverse cosine (arccosine) of the input tensor.
	acosh	$f:[1,\infty)\rightarrow[0,\infty)$	Computes element-wise inverse hyperbolic cosine: $\ln(x+\sqrt{x^{2}-1})$ .
	asin	$f:[-1,1]\rightarrow[-\frac{\pi}{2},\frac{\pi}{2}]$	Computes element-wise inverse sine (arcsine) of the input tensor.
	asinh	$f:\mathbb{R}\rightarrow\mathbb{R}$	Computes element-wise inverse hyperbolic sine: $\ln(x+\sqrt{x^{2}+1})$ .
	atan	$f:\mathbb{R}\rightarrow(-\frac{\pi}{2},\frac{\pi}{2})$	Computes element-wise inverse tangent (arctangent) of the input tensor.

Appendix

Table 3: PyTorch operators used in BackendBench evaluation. We evaluate 20 operators across three categories: trigonometric functions (8 ops), arithmetic operations (4 ops), and linear algebra primitives (8 ops).

Operator	Plain	Random	Chunk	AST	AST	AST	ECS
	LLM		+Dense	+Dense	+BM25	+Hybrid	(Ours)
acos.default	0.333	0.444	0.444	0.778	0.667	0.222	0.778
acosh.default	0.333	0.444	0.778	0.667	0.667	0.667	0.778
addmm.default	0.333	0.000	0.000	0.000	0.000	0.000	0.000
addmv.default	0.167	0.000	0.167	0.000	0.000	0.000	0.222
addbmm.default	0.000	0.000	0.000	0.000	0.000	0.000	0.000

Appendix

Table 4: Comparison between ECS and various RAG baselines in BackendBench

Statistic	Value
Total training samples	117
Average sequence length	15,940 tokens
Samples truncated (to 14K tokens)	24
Source Python files	62
Source Jupyter notebooks	12
By GPU Architecture

Appendix

Table 6: SFT Dataset Statistics

Hyperparameter	Value
Base Model	Qwen3-8B
Fine-tuning Method	LoRA (rank=8, $\alpha$ =32)
Trainable Parameters	21.8M (0.27%)
Learning Rates	{1e-6, 5e-6, 1e-5, 5e-5, 1e-4}
Global Batch Size	64
Epochs	15

Appendix

Table 7: SFT Training Configuration

Retrieval-Augmented ICL

Retrieval-augmented in-context learning (ICL) selects relevant examples or knowledge to help models adapt without updating their weights.

See all 45 papers

Context Training with Active Information Seeking
May 13, 2026Zeyu Huang, Adhiguna Kuncoro, Qixuan Feng +4Knowledge Augmentation for Language Models LLM Prompting
The University of Edinburgh · Google DeepMind
SearchSkill: Teaching LLMs to Use Search Tools with Evolving Skill Banks
May 9, 2026Jinchao Hu, Meizhi Zhong, Kehai Chen +1Tool-Augmented Language Model Agents Agentic Search
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen · Independent Researcher Shenzhen / Beijing, China
From Context to Skills: Can Language Models Learn from Context Skillfully?
Apr 30, 2026Shuzheng Si, Haozhe Zhao, Yu Lei +10In-Context Learning Agent Skill Learning
THU · UIUC · DeepLang AI +2

May 13, 2026J/K move · Enter open · S save

Context Training with Active Information Seeking

Zeyu Huang, Adhiguna Kuncoro, Qixuan Feng, Jiajun Shen+3

The University of Edinburgh · Google DeepMind

Knowledge Augmentation for Language Models LLM Prompting

Open paper AskPDF arXiv

Most existing large language models (LLMs) are expensive to adapt after deployment, especially when a task requires newly produced information or niche domain knowledge. Recent work has shown that, by manipulating and optimizing their context, LLMs can be tailored to downstream tasks without updating their weights. However, most existing methods remain closed-loop, relying solely on the model's intrinsic knowledge. In this paper, we equip these context optimizers with Wikipedia search and browser tools for active information seeking. We show that naively adding these tools to a standard sequential context optimization pipeline can actually degrade performance compared to baselines. However, when paired with a search-based training procedure that maintains and prunes multiple candidate contexts, active information seeking delivers consistent and substantial gains. We demonstrate these improvements across diverse domains, including low-resource translation (Flores+), health scenarios (HealthBench), and reasoning-heavy tasks (LiveCodeBench and Humanity's Last Exam). Furthermore, our method proves to be data-efficient, robust across different hyperparameters, and capable of generating effective textual contexts that generalize well across different models.

Abstract

Retrieval-Augmented ICL

Explore similar work

Context Training with Active Information Seeking

SearchSkill: Teaching LLMs to Use Search Tools with Evolving Skill Banks

From Context to Skills: Can Language Models Learn from Context Skillfully?