CURIO: Curiosity-Driven Test-Time Learning for Open-Ended Discovery

Tao Feng; Fangxu Yu; Zijie Lei; Jiaru Zou; Changjiang Jiang; Yi Yan; Jiaxuan You; Pan Lu

cs.LGOct 4, 2026

CURIO: Curiosity-Driven Test-Time Learning for Open-Ended Discovery

Authors: Tao Feng, Fangxu Yu, Zijie Lei, Jiaru Zou, Changjiang Jiang, Yi Yan, Jiaxuan You, Pan Lu

Organizations: University of Illinois Urbana-Champaign · University of Maryland, College Park · Stanford University

Abstract

Open-ended discovery requires learning from repeated attempts while continuing to explore directions whose value is not yet apparent. Search with a frozen large language model (LLM) can reuse previous solutions in context, but cannot update the model from its successes and failures on the test problem. Reinforcement learning (RL) enables such adaptation; however, strongly favoring high-reward trajectories may suppress low-reward yet potentially promising directions too early. We introduce CURIO, a curiosity-driven test-time learning framework that complements task feedback with an Intrinsic Curiosity World Model (ICWM). The ICWM learns transitions in the policy's hidden-state representation and supplies prediction-error bonuses at sampled tokens outside the policy's top-k choices. Epoch normalization and an annealed weight regulate their contribution to the policy update. On six mathematical discovery tasks and single-cell denoising with Qwen3 backbones from 8B to 235B, three-run means improve over a matched task-only RL control on five mathematical objectives, match the best reported performance on Circle Packing, and improve denoising Score and mean squared error (MSE) on both held-out corpora at every tested scale. Relative gains reach 18.3% on Hadamard and 10.8% on denoising Score. Code-diversity measurements show greater structural variation among generated programs, supporting curiosity as a complementary exploration signal for learning in open-ended discovery.

Figures & tables

Figure 1: Motivating exploration beyond task rewards. On Erdős and Circle Packing, CURIO has higher code diversity than task-only RL (TTT-Discover) at every displayed stage (metric: Appendix F ).

Figure 2: Overview of CURIO . A policy expands archive parents into rollouts. The ICWM learns token-level transitions in encoded policy hidden states, and its prediction residual informs the curiosity bonus. The signal is retained at sampled tokens outside the policy’s top- $k$ choices and combined with the task coefficient. The states $s_{t},s_{t+1}$ and action $a_{t}$ are token-level features and embeddings. The diagram is schematic; Equations ( 5 )–( 12 ) specify the representations, normalization, and masking.

Panel of Figure 3 — Figure 3: Code-level solution diversity across six discovery tasks. Panels are ordered left to right, top to bottom: Erdős, AC1, AC2, AC3, Circle Packing, and HadamardMatrix. Each panel displays measured code diversity $D$ over epochs for CURIO and TTT-Discover, using OpenEvolve’s island code diversity metric ( Sharma, 2025 ) (Appendix F ); higher $D$ indicates greater code-level structural variation.

Scale	Method	Erdős $(\downarrow)$	AC1 $(\downarrow)$	AC2 $(\uparrow)$	AC3 $(\downarrow)$	Circle Packing $(\uparrow)$	HadamardMatrix $(\uparrow)$
8B	AlphaEvolve	0.381002	1.50561	0.8947	1.5062	2.63585	0.5589
	ThetaEvolve	0.380999	1.50649	0.9449	1.4942	2.63597	0.5748
	ShinkaEvolve	0.381026	1.51493	0.8501	1.5074	2.63597	0.5661
	Best-of-25600	0.381091	1.51032	0.9318	1.5081	2.63570	0.5554
	SimpleTES	0.381006	1.50628	0.9458	1.4951	2.63597	0.5776
	TTT-Discover	0.380995	1.50546	0.9463	1.4935	2.63598	0.5809

Table 1: Test-time discovery across mathematical and combinatorial tasks. Results are grouped by policy-backbone scale. All results are means over three runs; variability is omitted for space. Arrows denote improvement direction. Within each scale block, best is bold and second-best is underlined. Ties at the displayed precision share the same marking. CURIO improves over TTT-Discover on five objectives and matches the best reported performance on Circle Packing across model scales, with particularly strong gains on HadamardMatrix.

Scale	Method	PBMC			Tabula
Scale	Method	Score $(\uparrow)$	MSE $(\downarrow)$	Poisson $(\downarrow)$	Score $(\uparrow)$	MSE $(\downarrow)$	Poisson $(\downarrow)$
8B	AlphaEvolve	0.6738	0.1817	0.0554	0.6785	0.1716	0.0323
	ThetaEvolve	0.6981	0.1588	0.0494	0.7207	0.1441	0.0302
	ShinkaEvolve	0.6787	0.1764	0.0535	0.6881	0.1669	0.0312
	Best-of-25600	0.6274	0.2071	0.0607	0.6382	0.1968	0.0354
	SimpleTES	0.6958	0.1641	0.0503	0.7118	0.1514	0.0305

Table 2: Single-cell denoising. Results on PBMC and Tabula scRNA-seq corpora, grouped by policy-backbone scale. All results are means over three runs; variability is omitted for space. Arrows denote improvement direction. Within each scale block, best is bold and second-best is underlined. Ties at the displayed precision share the same marking. CURIO consistently improves over TTT-Discover across model scales.

Panel of Figure 4 — Figure 4: Ablation studies of CURIO on the Qwen3-14B backbone. We report direction-aware normalized performance across six discovery tasks, with larger values indicating better performance. For each panel, the reference configuration is fixed at 50 on every task, so values above (below) 50 indicate improvement (degradation) relative to the reference. (a) Curiosity weight : we vary the initial curiosity weight $\eta_{0}\in\{0,0.3,0.5,0.8\}$ , using $\eta_{0}=0$ (TTT-DISCOVER) as the reference. (b) ICWM architecture : we compare the ICWM’s default multilayer perceptron (MLP) forward predictor with gated recurrent unit (GRU) and Transformer variants, using CURIO (MLP) as the reference. (c) Curiosity representation : we compare the ICWM’s policy-coupled input representation with a frozen-actor representation and a frozen external-LLM representation, using CURIO (Coupled) as the reference. Single-cell averages the PBMC and Tabula results.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Discovery loop
Epochs $E$	50
Target parent groups per epoch $N$	8
Rollouts per parent $G$	64
Configured generation budget $ENG$	25,600
PUCT exploration constant $c$	1.0

Appendix

Table 3: Reported hyperparameters. The discovery loop and policy optimization follow § 2 and the task coefficients of § 3.1 ; the ICWM-based curiosity block is specific to CURIO . Setting

\eta_{0}=0

removes the intrinsic reward.

Task	Objective	Dir.	Limit	CPUs	Reward from raw score $r$
Erdős’	overlap constant $C_{5}$	$\downarrow$	530 s	1	$1/(10^{-8}+r)$
AC1	1st autocorrelation bound	$\downarrow$	1100 s	2	$1/(10^{-8}+r)$
AC2	2nd autocorrelation bound	$\uparrow$	1100 s	2	$r$
AC3	3rd autocorrelation bound	$\downarrow$	1100 s	2	$3\,\ell^{3}$ , $\ell$ linear in $-r$
Circle Packing	sum of radii, $n{=}26$	$\uparrow$	530 s	1	$r$
HadamardMatrix	$\|\det M\|/\det_{\mathrm{ref}}$ , $n{=}29$	$\uparrow$	700 s	2	$r$

Appendix

Table 6: Discovery tasks. “Limit” is the hard wall-clock limit the sandbox enforces per candidate; the prompt separately advertises a soft compute budget to the model, and where the two differ the hard limit binds. “Reward” is the map from the raw objective

r

to the scalar the RL loop consumes; a candidate that fails the task’s feasibility check receives reward

0

irrespective of

r

Test-Time RL

Test-Time Reinforcement Learning adapts policies at deployment using interaction or self-generated feedback, without a separate task-specific training phase.

See all 48 papers

Curiosity as Linguistic Intervention: Using LLM Tutoring Dialogues to Influence Exploratory Learning Behavior
Jun 21, 2026Gevindu Ganganath, Pasindu Bolonghege, Qianru Lyu +2Intelligent Tutoring Systems Intrinsic Motivation
School of Computing and Information Systems, Singapore Management University · Human-Computer Interaction Institute, Carnegie Mellon University
Endogenous Exploration in Reinforcement Learning with Intrinsic Curiosity
Sep 4, 2026Armando VieiraRL Exploration Intrinsic Reward Methods
University of Tartu
Instruction-Conditioned Exploration for Reinforcement Learning with Self-Distillation to an Unconditioned Policy
Aug 3, 2026Jim Dilkes, Vahid Yazdanpanah, Sebastian SteinRL for Language Model Reasoning RL Post-Training
University of Southampton Southampton United Kingdom

Jun 21, 2026J/K move · Enter open · S save

Curiosity as Linguistic Intervention: Using LLM Tutoring Dialogues to Influence Exploratory Learning Behavior

Gevindu Ganganath, Pasindu Bolonghege, Qianru Lyu, Pradeep Varakantham+1

School of Computing and Information Systems, Singapore Management University · Human-Computer Interaction Institute, Carnegie Mellon University

Intelligent Tutoring Systems Intrinsic Motivation Causal Interventions in Language Models

Open paper AskPDF arXiv

Large Language Models (LLMs) provide a new opportunity to study how language shapes exploratory cognition because conversational strategies can be systematically manipulated at inference time. We introduce CURIOBOT, a framework that operationalizes Berlyne's collative variables, novelty, complexity, conflict, and uncertainty, as adaptive linguistic interventions for conversational tutoring. Across 270 tutoring conversations spanning multiple model families, domains, and topic complexity levels, curiosity-oriented interventions consistently increased exploratory learner behaviors, producing up to 2.4x more conversational turns under fixed time budgets. To measure these effects, we further introduce a learner-centered evaluation framework capturing exploratory questioning, conversational agency, productive struggle, and observable curiosity. Learner-side gains persisted even when tutor-side instructional quality remained unchanged, suggesting that curiosity functions as a partially independent interaction-level mechanism. More broadly, our results demonstrate that LLM-mediated dialogue can serve as a scalable experimental framework for studying how language shapes exploratory learning behavior.

Abstract

Test-Time RL

Explore similar work

Curiosity as Linguistic Intervention: Using LLM Tutoring Dialogues to Influence Exploratory Learning Behavior

Endogenous Exploration in Reinforcement Learning with Intrinsic Curiosity

Instruction-Conditioned Exploration for Reinforcement Learning with Self-Distillation to an Unconditioned Policy