BabyTheorist: A Benchmark for Learning to Theorize the World from Observation Alone

Doojin Baek; Junyeob Baek; Mingyu Jo; Hosung Lee; Taegu Kang; Gyubin Lee; Sungjin Ahn

cs.AINov 5, 2026

BabyTheorist: A Benchmark for Learning to Theorize the World from Observation Alone

Authors: Doojin Baek, Junyeob Baek, Mingyu Jo, Hosung Lee, Taegu Kang, Gyubin Lee, Sungjin Ahn

Organizations: KAIST

Abstract

How would we know whether a machine truly understands the world? We argue that one key capability is observational theory learning: acquiring reusable theories from pure observations and applying them to new situations. Current visual reasoning benchmarks focus primarily on the latter, presenting each task as a few-shot puzzle grouped by a common rule. We introduce BabyTheorist, a benchmark for controlled training and evaluation of observational theory learning. BabyTheorist generates continual streams of before-after visual observations from hidden compositional programs; during training, learners see only the resulting transitions, with no program labels, task identities, or support sets. To support theory learning and evaluation, BabyTheorist controls the program language, a curriculum over program complexity, recurring frequent subprograms, and a continual train-test protocol with level skipping. Seven representative baselines plateau well below ceiling, with the gap widening as the curriculum advances---evidence that BabyTheorist isolates a capability current program-learning approaches do not yet address.

Figures & tables

Figure 1: Observational theory learning. Humans learn about the world through theory acquisition and theory application. For example, observations of falling objects can lead to a theory of gravity, which can then be applied to explain orbital motion.

Figure 2: Overview of continual observational theory learning. A useful theory system should continually evolve as a learner receives a stream of observations. During theory acquisition (blue arrows), the learner discovers new theories when existing ones cannot explain an observation ( Mirror ) and abstracts recurring compositions into reusable units ( Up + Rotate ). Theory application combines them to explain new observations.

Figure 3: BabyTheorist program language and task generation. (a) Each program step pairs a selector, which identifies the objects to transform, with a primitive, which specifies the transformation. (b) The illustrated depth-6 program combines frequently recurring subprograms of depths 1, 3, and 2. Learners that discover such subprograms can reuse them to efficiently explain more complex phenomena.

Mode	# Objects	Selectors	# Primitives	Primitive Expansion	Program space
Single-A	1	–	8	–	$8^{d}$
Single-B	1	–	11	✓(8 $\rightarrow$ 11)	$11^{d}$
Multi-A	2	by_index	11	–	$22^{d}$
Multi-B	3	all selectors	14	✓(11 $\rightarrow$ 14)	$216^{d}$

Table 1: BabyTheorist benchmark modes. The Single/Multi axis varies scene structure and relational reference, while the A/B axis varies whether the primitive vocabulary expands across levels. Program space counts nominal depth-

d

step sequences, including object/selector and primitive choices, after all primitives become available (Appendix A.4 ).

Figure 4: Observations across modes and levels. Each pair shows one before–after observation from Levels 1, 2, and 8 of each mode. Observations become more complex through longer hidden programs, more objects, relational structure, and primitive expansion.

Figure 5: Continual train–test protocol. At the start of level $\ell$ , the learner is evaluated before training. Passing (at least $80\%$ accuracy) enables level skipping (red arrow): the learner advances without that level’s training observations, rewarding transfer to increasingly complex levels.

Figure 6: Score and highest curriculum level attempted in each benchmark mode. No baseline progresses beyond Lv2. Score sums $100\times$ accuracy over attempted levels, so only models that reach Lv2 can exceed 100. Trained baselines show the mean $\pm$ one population standard deviation over three seeds; LLMs are evaluated once with greedy decoding on the first 100 tasks at each attempted level. Cont-Mono-OC is evaluated only on Multi-A and Multi-B. Labels above the bars give the highest level attempted.

Figure 7: Transfer failures despite exact support fit. Five failure patterns on Lv1 tasks: (a) incorrect displacement (AR), (b) incorrect flip axis (Cont-Mono), (c) two color shifts instead of one (Cont-Mono), (d) a rotation-only output when rotation followed by a flip is required (Disc-Mono), and (e) the correct color change applied to the wrong object (NEO; dashed boxes mark the changed object). Rows show Input, Target, and Prediction. Both supports are predicted correctly (one shown), but the query is incorrect; green/red borders mark correct/incorrect predictions. Examples (a–d) use Single-A/B and (e) uses Multi-A.

Figure 8: Step-wise inference succeeds where joint inference fails. Joint predictions are wrong despite exact support fit. Given ground-truth intermediate supports, inferring a theory for each program step separately ( Separate ) yields correct intermediate and final outputs. Each step’s theory is inferred from its first support pair. This diagnostic adds supervision and changes inference, so it is not a baseline comparison.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Selector types. Each row shows a source scene and the objects selected by different selector types. Global selection applies to all objects; attribute-based selectors choose objects by properties such as color or hollowness; spatial selectors choose objects by position; and index-based selectors use a fixed raster-scan ordering.

Figure 10: Footprint-preserving transformation primitives. Representative before-after examples for primitives that transform selected objects while preserving their footprint, including rotation, reflection, and color change.

Figure 11: Footprint-changing transformation primitives. Representative before-after examples for object-shape primitives that change the interior or boundary of selected objects, including fill_interior , hollow , and add_border .

Figure 12: Movement primitives. Representative before-after examples for primitives that translate selected objects up, down, left, or right.

Figure 13: Relational primitives. Representative before-after examples for primitives that use a target object, including color swapping and row or column alignment.

Figure 14: Level depth distributions. Each level defines a distribution over hidden program depths rather than a single depth. As the level increases, the distribution shifts toward longer programs while still mixing observations of different complexity within the same level.

Target depth	Samples	Validated	Shortcuts	Reliability
1	1,000	1,000	0	100.0%
2	1,000	998	2	99.8%
3	1,000	997	3	99.7%
4	1,000	987	13	98.7%
Total	4,000	3,982	18	99.55%

Appendix

Table 2: Independent oracle validation of the depth filter on the Multi-B setting at level 4. We sampled 1,000 accepted tasks at each target depth and searched for shorter programs. Reliability is the fraction of accepted tasks for which the oracle found no shorter explanation.

Hyperparameter	Value
Backbone	Causal Transformer with RoPE
Hidden size	$384$
Number of blocks	$12$
Number of heads	$8$
Sequence length	$451$ ( $225+1+225$ )
Vocabulary size	$12$

Appendix

Table 3: AR baseline hyperparameters.

	Single Lv1	Multi-A Lv1	Multi-B Lv1
Seeds	41, 42, 43	7, 13, 42	41, 42, 43
Max transition length	1	1	1
Codebook size	—	—	—
Length-control coeff	0.9	0.9	0.9
Latent VAE $\beta$	0.05	0.10	0.05
Recon loss weight	1.0	1.0	1.0

Appendix

Table 4: Cont-Mono training hyperparameters.

	Single Lv1	Multi-A Lv1	Multi-B Lv1
Seeds	41, 42, 43	42, 43, 44	42, 43, 44
Max transition length	1	1	1
Codebook size	40	80	320
Recon loss weight	1.0	1.0	1.0
Latent VQ loss weight	1.0	1.0	1.0
Grounding loss weight	0.1	0.1	0.1

Appendix

Table 5: Disc-Mono training hyperparameters.

	Multi-A Lv1	Multi-B Lv1
Seeds	7, 13, 2024	7, 13, 2024
Max transition length	1	1
Codebook size	—	—
Length-control coeff	0.1	0.1
Latent VAE $\beta$	0.012	0.008
Recon loss weight	1.0	1.0

Appendix

Table 6: Cont-Mono-OC finetuning hyperparameters. Batch size is per GPU.

	Lv1 (Single A & B)		Lv2
	Phase 1 (mtl=2)	Phase 2 (mtl=4)	Single-A	Single-B
Seeds	40, 41, 42	40, 41, 42	40, 42	40, 42
Max transition length	2	4	4	4
Codebook size	30	30	30	30
Length-control coeff	0.9	1.0	1.2	1.2
Recon loss weight	1.0	1.0	1.0	1.0

Appendix

Table 7: NEO single-object hyperparameters.

	Multi-A		Multi-B
	Phase 1 (mtl=2)	Phase 2 (mtl=3)	Phase 1 (mtl=2)	Phase 2 (mtl=3)
Seeds	42, 43, 44	42, 43, 44	42, 43, 44	42, 43, 44
Max transition length	2	3	2	3
Codebook size	40	40	160	160
Length-control coeff	0.9	0.9	0.9	0.9
Recon loss weight	1.0	1.0	1.0	1.0

Appendix

Table 8: NEO multi-object hyperparameters.

Model	Single-A	Single-B	Multi-A	Multi-B
LLM (GPT-OSS 120B, high)	59	52	15	15
LLM (Qwen 3.5 27B, thinking-off)	8	8	4	6
AR	$6.7\pm 0.6$	$7.3\pm 0.5$	$7.7\pm 0.6$	$9.3\pm 1.0$
Cont-Mono	$15.9\pm 1.4$	$16.7\pm 1.5$	$10.5\pm 5.7$	$5.6\pm 0.7$
Disc-Mono	$52.5\pm 7.0$	$54.5\pm 6.9$	$16.1\pm 2.9$	$8.5\pm 0.6$
Cont-Mono-Opt	$16.8\pm 1.4$	$17.9\pm 1.6$	$8.7\pm 4.9$	$5.6\pm 0.7$

Appendix

Table 9: Score on the four BabyTheorist benchmark modes. Score sums per-level exact-match accuracy across reached levels. Higher is better. Mean over

n=3

seeds (population std); LLM rows are single evaluations on the first 100 tasks per level.

Setting	Value
CNN channels / hidden width	$(64,128,128)$ / $512$
Action-component embedding / action feature width	$64$ / $256$
Optimizer / learning rate	Adam / $1.5\times 10^{-4}$
Adam $\epsilon$ / gradient-norm clipping	$10^{-5}$ / $0.5$
Rollout batch / episode minibatch size	$512$ / $128$
PPO epochs / clipping coefficient	$4$ / $0.2$

Appendix

Table 10: Training and inference settings for PPO with Known DSL. The policy head differs between Single and Multi modes as described in the text.

Mode	Score	Highest level passed	First failed level
Single-A	$680.2\pm 2.5$	8	–
Single-B	$500.4\pm 0.8$	5	6
Multi-A	$331.8\pm 1.5$	3	4
Multi-B	$52.5\pm 5.9$	0	1

Appendix

Table 11: PPO with Known DSL on the four BabyTheorist modes. Score is reported as mean

\pm

population standard deviation over three seeds. Level progression is identical across seeds; zero passed levels means failure at Level 1. Settings were selected using the public evaluation set, as described below.

Mode	Level 1	Level 2	Level 3	Level 4	Level 5	Level 6	Level 7	Level 8
Single-A	58	68	58	55	47	44	37	31
Single-B	64	69	60	57	51	39	45	40
Multi-A	51	48	42	50	42	42	31	30
Multi-B	52	48	51	53	40	29	35	33

Appendix

Table 12: GPT-5.6 Sol exact-match accuracy (%) on 100 tasks per mode and level. Levels 2–8 are evaluated independently for diagnostic purposes.

Dataset	Method	$S$ (%)	$Q$ (%)	$Q\mid S$ (%)
Single-A/B	Cont-Mono	21.93 $\pm$ 1.81	16.66 $\pm$ 1.24	25.98 $\pm$ 2.32
	Cont-Mono-Opt	23.84 $\pm$ 1.54	16.97 $\pm$ 1.23	25.51 $\pm$ 1.90
	Disc-Mono	50.19 $\pm$ 5.33	54.48 $\pm$ 5.66	84.86 $\pm$ 5.50
	NEO	87.01 $\pm$ 4.08	85.81 $\pm$ 4.17	92.11 $\pm$ 2.77
	AR	99.95 $\pm$ 0.03	7.32 $\pm$ 0.45	7.31 $\pm$ 0.44
Multi-A	Cont-Mono	10.64 $\pm$ 3.98	10.45 $\pm$ 4.61	22.74 $\pm$ 9.31

Appendix

Table 13: Support fit and query transfer at Lv1. Mean

\pm

SD across three seeds (%).

S

: every support correct;

Q

: query correct;

Q\mid S

: query accuracy conditional on

S

. Each dataset contains 10,000 tasks per seed; Single-A/B share the same Lv1 data. See Appendix H.1 for protocols.

Method	$N$	$S$ (%)	$Q$ (%)	$S\cap\neg Q$ (%)
Cont-Mono	2944	34.69 $\pm$ 2.73	26.18 $\pm$ 2.01	23.51 $\pm$ 1.04
Cont-Mono-Opt	2944	36.16 $\pm$ 2.20	26.51 $\pm$ 1.97	24.63 $\pm$ 0.79
Disc-Mono	2944	73.97 $\pm$ 7.36	75.51 $\pm$ 8.09	8.48 $\pm$ 3.36
NEO	2944	97.54 $\pm$ 2.29	96.49 $\pm$ 2.69	2.36 $\pm$ 1.41
AR	2944	99.95 $\pm$ 0.02	13.07 $\pm$ 0.78	86.92 $\pm$ 0.77

Appendix

Table 14: Single-A/B transfer at Lv1, primitive depth one. Mean

\pm

SD across three seeds (%).

S

: every support correct;

Q

: query correct.

S\cap\neg Q

uses all

N

tasks as its denominator. Inference follows Appendix H.1 .

Method	$N$	$S$ (%)	$Q$ (%)	$S\cap\neg Q$ (%)
Cont-Mono	5017	20.01 $\pm$ 1.83	15.19 $\pm$ 1.15	15.51 $\pm$ 0.93
Cont-Mono-Opt	5017	22.27 $\pm$ 1.30	15.59 $\pm$ 1.11	17.34 $\pm$ 0.56
Disc-Mono	5017	56.19 $\pm$ 6.70	61.64 $\pm$ 6.80	9.27 $\pm$ 2.74
NEO	5017	89.44 $\pm$ 4.75	88.22 $\pm$ 4.78	6.54 $\pm$ 2.24
AR	5017	99.95 $\pm$ 0.03	5.97 $\pm$ 0.33	93.98 $\pm$ 0.36

Appendix

Table 15: Single-A/B transfer at Lv1, primitive depth two. Mean

\pm

SD across three seeds (%).

N

: tasks per seed;

S

: every support correct;

Q

: query correct.

S\cap\neg Q

uses all

N

tasks as its denominator. Inference follows Table 14 , including step-10 trajectories for Cont-Mono-Opt. Depth is the number of primitives in the generating program.

Method	Denominator	Single	Multi-A	Multi-B
		One constituent	Wrong target	Wrong target
Cont-Mono	Eligible	15.05 $\pm$ 1.15	8.82 $\pm$ 4.89	4.96 $\pm$ 0.73
	Full set	7.55 $\pm$ 0.58	1.92 $\pm$ 1.07	1.05 $\pm$ 0.16
Cont-Mono-Opt	Eligible	15.16 $\pm$ 0.94	8.91 $\pm$ 5.22	5.23 $\pm$ 0.85
	Full set	7.60 $\pm$ 0.47	1.94 $\pm$ 1.14	1.11 $\pm$ 0.18
Disc-Mono	Eligible	3.70 $\pm$ 0.94	14.91 $\pm$ 4.29	5.78 $\pm$ 0.22

Appendix

Table 16: Output-pattern frequencies. Mean

\pm

SD across three seeds (%). Eligible

n

: Single 5,017; Multi-A 2,177; Multi-B 2,117; full

n

: 10,000. Patterns are non-exclusive. Opt uses 2,944 depth-one tasks; full-set rescue/harm:

1.12\pm 0.06

0.99\pm 0.05

%. Within-trajectory net gain:

0.43\pm 0.11

pp.

Figure 15: AR: support fit without query transfer. Each column shows a different depth-one Single-B task from this 256-task subset (seed 42). The adapted model fits both supports but fails on the query.

Figure 16: Transfer errors on depth-1 programs. Cont-Mono (a–c) and Disc-Mono (d–f) fit both supports but make reflection-axis, color-change, and translation-distance errors, respectively (seed 42). Color predictions match two shifts instead of one. The Disc-Mono translation error is a rare illustrative case.

Figure 17: Support fitting can damage a correct query. Three Cont-Mono-Opt cases change from correct at step 0 to incorrect at step 10. Both supports remain exact at recorded steps 0, 1, 5, and 10; support loss need not be constant.

Figure 18: Correct transformation, wrong target. Cont-Mono, Disc-Mono, and NEO fit both supports but change the wrong query object in Multi-A (seed 42). Each prediction exactly matches the correct primitive applied to another target. Dashed outlines identify the affected objects.

Figure 19: Re-encoding corrects some NEO continuations. The first rotation is exact, but the following translation moves 3, 3, and 4 cells instead of 4, 5, and 5. Re-encoding the decoded image before applying the same second code corrects these cases.

Figure 20: Re-encoding also breaks correct NEO continuations. Under the same intervention, three previously correct queries become incorrect. Together with Figure 19 , these cases show state-dependent execution; re-encoding is not a general remedy.

Figure 21: Continuous latent representations of Cont-Mono. Each point represents the latent inferred from a task’s first support pair. Colors identify the primitives that generated the source tasks.

Figure 22: Pre-quantization representations and codebook vectors of Disc-Mono. Points represent continuous vectors before quantization, colored by the primitives that generated the source tasks. Hollow diamonds mark selected codebook entries, and numeric labels denote code IDs within each checkpoint.

Figure 23: Pre-quantization representations and codebook vectors of NEO. Points represent continuous vectors before quantization, colored by the primitives that generated the source tasks. Hollow diamonds mark selected codebook entries, and numeric labels denote code IDs within each checkpoint.

Visually Grounded Reasoning

Visually grounded reasoning links visual evidence to multi-step conclusions and actions, aiming to make multimodal systems accurate, auditable, and robust.

See all 349 papers

Learning to Theorize the World from Observation
May 5, 2026Doojin Baek, Gyubin Lee, Junyeob Baek +2Computational Cognitive Modeling World Model Learning
KAIST
BabyVision: Visual Reasoning Beyond Language
Jan 10, 2026Liang Chen, Weichu Xie, Yiyan Liang +27VLM Evaluation Visual Reasoning
1UniPat AI · 6Peking University · 3Alibaba Group +8
TACIT Benchmark: A Programmatic Visual Reasoning Benchmark for Generative and Discriminative Models
Feb 27, 2026Daniel Nobrega MedeirosVisual Reasoning Spatial Reasoning Benchmarks
Independent Researcher

May 5, 2026J/K move · Enter open · S save

Learning to Theorize the World from Observation

Doojin Baek, Gyubin Lee, Junyeob Baek, Hosung Lee+1

KAIST

Computational Cognitive Modeling World Model Learning Program Synthesis Neurosymbolic Learning

Open paper AskPDF arXiv

What does it mean to understand the world? Contemporary world models often operationalize understanding as accurate future prediction in latent or observation space. Developmental cognitive science, however, suggests a different view: human understanding emerges through the construction of internal theories of how the world works, even before mature language is acquired. Inspired by this theory-building view of cognition, we introduce Learning-to-Theorize, a learning paradigm for inferring explicit explanatory theories of the world from raw, non-textual observations. We instantiate this paradigm with the Neural Theorizer (NEO), a World Theory Model, that induces latent programs as a learned Language of Thought and executes them through a shared transition model. In NEO, a theory is represented as an executable, compositional program whose learned primitives can be systematically recombined to explain novel phenomena. Experiments show that this formulation enables explanation-driven generalization, allowing observations to be understood in terms of the programs that generate them.

Abstract

Visually Grounded Reasoning

Explore similar work

Learning to Theorize the World from Observation

BabyVision: Visual Reasoning Beyond Language

TACIT Benchmark: A Programmatic Visual Reasoning Benchmark for Generative and Discriminative Models