Open-ended discovery requires learning from repeated attempts while continuing to explore directions whose value is not yet apparent. Search with a frozen large language model (LLM) can reuse previous solutions in context, but cannot update the model from its successes and failures on the test problem. Reinforcement learning (RL) enables such adaptation; however, strongly favoring high-reward trajectories may suppress low-reward yet potentially promising directions too early. We introduce CURIO, a curiosity-driven test-time learning framework that complements task feedback with an Intrinsic Curiosity World Model (ICWM). The ICWM learns transitions in the policy's hidden-state representation and supplies prediction-error bonuses at sampled tokens outside the policy's top-k choices. Epoch normalization and an annealed weight regulate their contribution to the policy update. On six mathematical discovery tasks and single-cell denoising with Qwen3 backbones from 8B to 235B, three-run means improve over a matched task-only RL control on five mathematical objectives, match the best reported performance on Circle Packing, and improve denoising Score and mean squared error (MSE) on both held-out corpora at every tested scale. Relative gains reach 18.3% on Hadamard and 10.8% on denoising Score. Code-diversity measurements show greater structural variation among generated programs, supporting curiosity as a complementary exploration signal for learning in open-ended discovery.
Figures & tables
Figure 1: Motivating exploration beyond task rewards. On Erdős and Circle Packing, CURIO has higher code diversity than task-only RL (TTT-Discover) at every displayed stage (metric: Appendix F ).
Figure 2: Overview of CURIO . A policy expands archive parents into rollouts. The ICWM learns token-level transitions in encoded policy hidden states, and its prediction residual informs the curiosity bonus. The signal is retained at sampled tokens outside the policy’s top- k choices and combined with the task coefficient. The states st,st+1 and action at are token-level features and embeddings. The diagram is schematic; Equations ( 5 )–( 12 ) specify the representations, normalization, and masking.
Figure 3: Code-level solution diversity across six discovery tasks. Panels are ordered left to right, top to bottom: Erdős, AC1, AC2, AC3, Circle Packing, and HadamardMatrix. Each panel displays measured code diversity D over epochs for CURIO and TTT-Discover, using OpenEvolve’s island code diversity metric ( Sharma, 2025 ) (Appendix F ); higher D indicates greater code-level structural variation.
Scale
Method
Erdős (↓)
AC1 (↓)
AC2 (↑)
AC3 (↓)
Circle Packing (↑)
HadamardMatrix (↑)
8B
AlphaEvolve
0.381002
1.50561
0.8947
1.5062
2.63585
0.5589
ThetaEvolve
0.380999
1.50649
0.9449
1.4942
2.63597
0.5748
ShinkaEvolve
0.381026
1.51493
0.8501
1.5074
2.63597
0.5661
Best-of-25600
0.381091
1.51032
0.9318
1.5081
2.63570
0.5554
SimpleTES
0.381006
1.50628
0.9458
1.4951
2.63597
0.5776
TTT-Discover
0.380995
1.50546
0.9463
1.4935
2.63598
0.5809
Table 1: Test-time discovery across mathematical and combinatorial tasks. Results are grouped by policy-backbone scale. All results are means over three runs; variability is omitted for space. Arrows denote improvement direction. Within each scale block, best is bold and second-best is underlined. Ties at the displayed precision share the same marking. CURIO improves over TTT-Discover on five objectives and matches the best reported performance on Circle Packing across model scales, with particularly strong gains on HadamardMatrix.
Scale
Method
PBMC
Tabula
Score (↑)
MSE (↓)
Poisson (↓)
Score (↑)
MSE (↓)
Poisson (↓)
8B
AlphaEvolve
0.6738
0.1817
0.0554
0.6785
0.1716
0.0323
ThetaEvolve
0.6981
0.1588
0.0494
0.7207
0.1441
0.0302
ShinkaEvolve
0.6787
0.1764
0.0535
0.6881
0.1669
0.0312
Best-of-25600
0.6274
0.2071
0.0607
0.6382
0.1968
0.0354
SimpleTES
0.6958
0.1641
0.0503
0.7118
0.1514
0.0305
Table 2: Single-cell denoising. Results on PBMC and Tabula scRNA-seq corpora, grouped by policy-backbone scale. All results are means over three runs; variability is omitted for space. Arrows denote improvement direction. Within each scale block, best is bold and second-best is underlined. Ties at the displayed precision share the same marking. CURIO consistently improves over TTT-Discover across model scales.
Figure 4: Ablation studies of CURIO on the Qwen3-14B backbone. We report direction-aware normalized performance across six discovery tasks, with larger values indicating better performance. For each panel, the reference configuration is fixed at 50 on every task, so values above (below) 50 indicate improvement (degradation) relative to the reference. (a) Curiosity weight : we vary the initial curiosity weight η0∈{0,0.3,0.5,0.8} , using η0=0 (TTT-DISCOVER) as the reference. (b) ICWM architecture : we compare the ICWM’s default multilayer perceptron (MLP) forward predictor with gated recurrent unit (GRU) and Transformer variants, using CURIO (MLP) as the reference. (c) Curiosity representation : we compare the ICWM’s policy-coupled input representation with a frozen-actor representation and a frozen external-LLM representation, using CURIO (Coupled) as the reference. Single-cell averages the PBMC and Tabula results.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Discovery loop
Epochs E
50
Target parent groups per epoch N
8
Rollouts per parent G
64
Configured generation budget ENG
25,600
PUCT exploration constant c
1.0
Appendix
Table 3: Reported hyperparameters. The discovery loop and policy optimization follow § 2 and the task coefficients of § 3.1 ; the ICWM-based curiosity block is specific to CURIO . Setting η0=0 removes the intrinsic reward.
Task
Objective
Dir.
Limit
CPUs
Reward from raw score r
Erdős’
overlap constant C5
↓
530 s
1
1/(10−8+r)
AC1
1st autocorrelation bound
↓
1100 s
2
1/(10−8+r)
AC2
2nd autocorrelation bound
↑
1100 s
2
r
AC3
3rd autocorrelation bound
↓
1100 s
2
3ℓ3 , ℓ linear in −r
Circle Packing
sum of radii, n=26
↑
530 s
1
r
HadamardMatrix
∣detM∣/detref , n=29
↑
700 s
2
r
Appendix
Table 6: Discovery tasks. “Limit” is the hard wall-clock limit the sandbox enforces per candidate; the prompt separately advertises a soft compute budget to the model, and where the two differ the hard limit binds. “Reward” is the map from the raw objective r to the scalar the RL loop consumes; a candidate that fails the task’s feasibility check receives reward 0 irrespective of r .
Test-Time Reinforcement Learning adapts policies at deployment using interaction or self-generated feedback, without a separate task-specific training phase.
Large Language Models (LLMs) provide a new opportunity to study how language shapes exploratory cognition because conversational strategies can be systematically manipulated at inference time. We introduce CURIOBOT, a framework that operationalizes Berlyne's collative variables, novelty, complexity, conflict, and uncertainty, as adaptive linguistic interventions for conversational tutoring. Across 270 tutoring conversations spanning multiple model families, domains, and topic complexity levels, curiosity-oriented interventions consistently increased exploratory learner behaviors, producing up to 2.4x more conversational turns under fixed time budgets. To measure these effects, we further introduce a learner-centered evaluation framework capturing exploratory questioning, conversational agency, productive struggle, and observable curiosity. Learner-side gains persisted even when tutor-side instructional quality remained unchanged, suggesting that curiosity functions as a partially independent interaction-level mechanism. More broadly, our results demonstrate that LLM-mediated dialogue can serve as a scalable experimental framework for studying how language shapes exploratory learning behavior.
We propose a reinforcement learning framework in which exploration is driven by intrinsic curiosity, designed for scenarios where environments are non-stationary and rewards are sparse, delayed, uninformative, or absent. In our model, action selection is guided by a combination of external rewards and an epistemic motivation mechanism that biases the agent toward structured exploratory directions. The central hypothesis is that effective exploration emerges at intermediate levels of incoherence, while performance degrades under both overly rigid and overly disordered dynamics. To test this idea, we implement the framework on top of a Liquid State Machine (LSM) substrate and evaluate it on two standard benchmarks: the discrete-action LunarLanderv2 and the continuous-control BipedalWalkerv3. The proposed method achieves competitive performance on both tasks relative to established deep RL algorithms, including Proximal Policy Optimization (PPO) and Intrinsic Curiosity Module (ICM). We further show that the curiosity window is not recovered in Active Inference agents under the same analysis, suggesting that the proposed dynamics capture a distinct exploration regime
Post-training Large Language Models (LLMs) with Reinforcement Learning (RL) has become an important tool for improving model capabilities, but the LLM action-space structure introduces challenges distinct from classical RL, with implications for inducing exploration. New methods are required that leverage the broad knowledge and flexibility of pre-trained LLMs to deliberately generate diverse experience at training time. We propose Instruction-Conditioned Exploration (ICE), which appends one of a small fixed set of instructions to task prompts during training, using the same set for every problem, increasing the coverage of behaviours attempted. To facilitate ICE, we combine RL on the instruction-conditioned policy with self-distillation of its correct rollouts into the unconditioned test-time policy. ICE with this objective improves Qwen3-1.7B held-out pass@1 performance at 4K response length on mathematical reasoning tasks by 5.0% relative to training with DAPO, with improvement persisting at a longer 8K context. The improvement does not appear for Qwen3-4B at 4K, where the instructions do not expand base-model coverage.
Jim Dilkes, Vahid Yazdanpanah, Sebastian Stein
University of Southampton Southampton United Kingdom