How would we know whether a machine truly understands the world? We argue that one key capability is observational theory learning: acquiring reusable theories from pure observations and applying them to new situations. Current visual reasoning benchmarks focus primarily on the latter, presenting each task as a few-shot puzzle grouped by a common rule. We introduce BabyTheorist, a benchmark for controlled training and evaluation of observational theory learning. BabyTheorist generates continual streams of before-after visual observations from hidden compositional programs; during training, learners see only the resulting transitions, with no program labels, task identities, or support sets. To support theory learning and evaluation, BabyTheorist controls the program language, a curriculum over program complexity, recurring frequent subprograms, and a continual train-test protocol with level skipping. Seven representative baselines plateau well below ceiling, with the gap widening as the curriculum advances---evidence that BabyTheorist isolates a capability current program-learning approaches do not yet address.
Figures & tables
Figure 1: Observational theory learning. Humans learn about the world through theory acquisition and theory application. For example, observations of falling objects can lead to a theory of gravity, which can then be applied to explain orbital motion.
Figure 2: Overview of continual observational theory learning. A useful theory system should continually evolve as a learner receives a stream of observations. During theory acquisition (blue arrows), the learner discovers new theories when existing ones cannot explain an observation ( Mirror ) and abstracts recurring compositions into reusable units ( Up + Rotate ). Theory application combines them to explain new observations.
Figure 3: BabyTheorist program language and task generation. (a) Each program step pairs a selector, which identifies the objects to transform, with a primitive, which specifies the transformation. (b) The illustrated depth-6 program combines frequently recurring subprograms of depths 1, 3, and 2. Learners that discover such subprograms can reuse them to efficiently explain more complex phenomena.
Mode
# Objects
Selectors
# Primitives
Primitive Expansion
Program space
Single-A
1
–
8
–
8d
Single-B
1
–
11
✓(8 → 11)
11d
Multi-A
2
by_index
11
–
22d
Multi-B
3
all selectors
14
✓(11 → 14)
216d
Table 1: BabyTheorist benchmark modes. The Single/Multi axis varies scene structure and relational reference, while the A/B axis varies whether the primitive vocabulary expands across levels. Program space counts nominal depth- d step sequences, including object/selector and primitive choices, after all primitives become available (Appendix A.4 ).
Figure 4: Observations across modes and levels. Each pair shows one before–after observation from Levels 1, 2, and 8 of each mode. Observations become more complex through longer hidden programs, more objects, relational structure, and primitive expansion.
Figure 5: Continual train–test protocol. At the start of level ℓ , the learner is evaluated before training. Passing (at least 80% accuracy) enables level skipping (red arrow): the learner advances without that level’s training observations, rewarding transfer to increasingly complex levels.
Figure 6: Score and highest curriculum level attempted in each benchmark mode. No baseline progresses beyond Lv2. Score sums 100× accuracy over attempted levels, so only models that reach Lv2 can exceed 100. Trained baselines show the mean ± one population standard deviation over three seeds; LLMs are evaluated once with greedy decoding on the first 100 tasks at each attempted level. Cont-Mono-OC is evaluated only on Multi-A and Multi-B. Labels above the bars give the highest level attempted.
Figure 7: Transfer failures despite exact support fit. Five failure patterns on Lv1 tasks: (a) incorrect displacement (AR), (b) incorrect flip axis (Cont-Mono), (c) two color shifts instead of one (Cont-Mono), (d) a rotation-only output when rotation followed by a flip is required (Disc-Mono), and (e) the correct color change applied to the wrong object (NEO; dashed boxes mark the changed object). Rows show Input, Target, and Prediction. Both supports are predicted correctly (one shown), but the query is incorrect; green/red borders mark correct/incorrect predictions. Examples (a–d) use Single-A/B and (e) uses Multi-A.
Figure 8: Step-wise inference succeeds where joint inference fails. Joint predictions are wrong despite exact support fit. Given ground-truth intermediate supports, inferring a theory for each program step separately ( Separate ) yields correct intermediate and final outputs. Each step’s theory is inferred from its first support pair. This diagnostic adds supervision and changes inference, so it is not a baseline comparison.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Selector types. Each row shows a source scene and the objects selected by different selector types. Global selection applies to all objects; attribute-based selectors choose objects by properties such as color or hollowness; spatial selectors choose objects by position; and index-based selectors use a fixed raster-scan ordering.
Figure 10: Footprint-preserving transformation primitives. Representative before-after examples for primitives that transform selected objects while preserving their footprint, including rotation, reflection, and color change.
Figure 11: Footprint-changing transformation primitives. Representative before-after examples for object-shape primitives that change the interior or boundary of selected objects, including fill_interior , hollow , and add_border .
Figure 12: Movement primitives. Representative before-after examples for primitives that translate selected objects up, down, left, or right.
Figure 13: Relational primitives. Representative before-after examples for primitives that use a target object, including color swapping and row or column alignment.
Figure 14: Level depth distributions. Each level defines a distribution over hidden program depths rather than a single depth. As the level increases, the distribution shifts toward longer programs while still mixing observations of different complexity within the same level.
Target depth
Samples
Validated
Shortcuts
Reliability
1
1,000
1,000
0
100.0%
2
1,000
998
2
99.8%
3
1,000
997
3
99.7%
4
1,000
987
13
98.7%
Total
4,000
3,982
18
99.55%
Appendix
Table 2: Independent oracle validation of the depth filter on the Multi-B setting at level 4. We sampled 1,000 accepted tasks at each target depth and searched for shorter programs. Reliability is the fraction of accepted tasks for which the oracle found no shorter explanation.
Hyperparameter
Value
Backbone
Causal Transformer with RoPE
Hidden size
384
Number of blocks
12
Number of heads
8
Sequence length
451 ( 225+1+225 )
Vocabulary size
12
Appendix
Table 3: AR baseline hyperparameters.
Single Lv1
Multi-A Lv1
Multi-B Lv1
Seeds
41, 42, 43
7, 13, 42
41, 42, 43
Max transition length
1
1
1
Codebook size
—
—
—
Length-control coeff
0.9
0.9
0.9
Latent VAE β
0.05
0.10
0.05
Recon loss weight
1.0
1.0
1.0
Appendix
Table 4: Cont-Mono training hyperparameters.
Single Lv1
Multi-A Lv1
Multi-B Lv1
Seeds
41, 42, 43
42, 43, 44
42, 43, 44
Max transition length
1
1
1
Codebook size
40
80
320
Recon loss weight
1.0
1.0
1.0
Latent VQ loss weight
1.0
1.0
1.0
Grounding loss weight
0.1
0.1
0.1
Appendix
Table 5: Disc-Mono training hyperparameters.
Multi-A Lv1
Multi-B Lv1
Seeds
7, 13, 2024
7, 13, 2024
Max transition length
1
1
Codebook size
—
—
Length-control coeff
0.1
0.1
Latent VAE β
0.012
0.008
Recon loss weight
1.0
1.0
Appendix
Table 6: Cont-Mono-OC finetuning hyperparameters. Batch size is per GPU.
Lv1 (Single A & B)
Lv2
Phase 1 (mtl=2)
Phase 2 (mtl=4)
Single-A
Single-B
Seeds
40, 41, 42
40, 41, 42
40, 42
40, 42
Max transition length
2
4
4
4
Codebook size
30
30
30
30
Length-control coeff
0.9
1.0
1.2
1.2
Recon loss weight
1.0
1.0
1.0
1.0
Appendix
Table 7: NEO single-object hyperparameters.
Multi-A
Multi-B
Phase 1 (mtl=2)
Phase 2 (mtl=3)
Phase 1 (mtl=2)
Phase 2 (mtl=3)
Seeds
42, 43, 44
42, 43, 44
42, 43, 44
42, 43, 44
Max transition length
2
3
2
3
Codebook size
40
40
160
160
Length-control coeff
0.9
0.9
0.9
0.9
Recon loss weight
1.0
1.0
1.0
1.0
Appendix
Table 8: NEO multi-object hyperparameters.
Model
Single-A
Single-B
Multi-A
Multi-B
LLM (GPT-OSS 120B, high)
59
52
15
15
LLM (Qwen 3.5 27B, thinking-off)
8
8
4
6
AR
6.7±0.6
7.3±0.5
7.7±0.6
9.3±1.0
Cont-Mono
15.9±1.4
16.7±1.5
10.5±5.7
5.6±0.7
Disc-Mono
52.5±7.0
54.5±6.9
16.1±2.9
8.5±0.6
Cont-Mono-Opt
16.8±1.4
17.9±1.6
8.7±4.9
5.6±0.7
Appendix
Table 9: Score on the four BabyTheorist benchmark modes. Score sums per-level exact-match accuracy across reached levels. Higher is better. Mean over n=3 seeds (population std); LLM rows are single evaluations on the first 100 tasks per level.
Setting
Value
CNN channels / hidden width
(64,128,128) / 512
Action-component embedding / action feature width
64 / 256
Optimizer / learning rate
Adam / 1.5×10−4
Adam ϵ / gradient-norm clipping
10−5 / 0.5
Rollout batch / episode minibatch size
512 / 128
PPO epochs / clipping coefficient
4 / 0.2
Appendix
Table 10: Training and inference settings for PPO with Known DSL. The policy head differs between Single and Multi modes as described in the text.
Mode
Score
Highest level passed
First failed level
Single-A
680.2±2.5
8
–
Single-B
500.4±0.8
5
6
Multi-A
331.8±1.5
3
4
Multi-B
52.5±5.9
0
1
Appendix
Table 11: PPO with Known DSL on the four BabyTheorist modes. Score is reported as mean ± population standard deviation over three seeds. Level progression is identical across seeds; zero passed levels means failure at Level 1. Settings were selected using the public evaluation set, as described below.
Mode
Level 1
Level 2
Level 3
Level 4
Level 5
Level 6
Level 7
Level 8
Single-A
58
68
58
55
47
44
37
31
Single-B
64
69
60
57
51
39
45
40
Multi-A
51
48
42
50
42
42
31
30
Multi-B
52
48
51
53
40
29
35
33
Appendix
Table 12: GPT-5.6 Sol exact-match accuracy (%) on 100 tasks per mode and level. Levels 2–8 are evaluated independently for diagnostic purposes.
Dataset
Method
S (%)
Q (%)
Q∣S (%)
Single-A/B
Cont-Mono
21.93 ± 1.81
16.66 ± 1.24
25.98 ± 2.32
Cont-Mono-Opt
23.84 ± 1.54
16.97 ± 1.23
25.51 ± 1.90
Disc-Mono
50.19 ± 5.33
54.48 ± 5.66
84.86 ± 5.50
NEO
87.01 ± 4.08
85.81 ± 4.17
92.11 ± 2.77
AR
99.95 ± 0.03
7.32 ± 0.45
7.31 ± 0.44
Multi-A
Cont-Mono
10.64 ± 3.98
10.45 ± 4.61
22.74 ± 9.31
Appendix
Table 13: Support fit and query transfer at Lv1. Mean ± SD across three seeds (%). S : every support correct; Q : query correct; Q∣S : query accuracy conditional on S . Each dataset contains 10,000 tasks per seed; Single-A/B share the same Lv1 data. See Appendix H.1 for protocols.
Method
N
S (%)
Q (%)
S∩¬Q (%)
Cont-Mono
2944
34.69 ± 2.73
26.18 ± 2.01
23.51 ± 1.04
Cont-Mono-Opt
2944
36.16 ± 2.20
26.51 ± 1.97
24.63 ± 0.79
Disc-Mono
2944
73.97 ± 7.36
75.51 ± 8.09
8.48 ± 3.36
NEO
2944
97.54 ± 2.29
96.49 ± 2.69
2.36 ± 1.41
AR
2944
99.95 ± 0.02
13.07 ± 0.78
86.92 ± 0.77
Appendix
Table 14: Single-A/B transfer at Lv1, primitive depth one. Mean ± SD across three seeds (%). S : every support correct; Q : query correct. S∩¬Q uses all N tasks as its denominator. Inference follows Appendix H.1 .
Method
N
S (%)
Q (%)
S∩¬Q (%)
Cont-Mono
5017
20.01 ± 1.83
15.19 ± 1.15
15.51 ± 0.93
Cont-Mono-Opt
5017
22.27 ± 1.30
15.59 ± 1.11
17.34 ± 0.56
Disc-Mono
5017
56.19 ± 6.70
61.64 ± 6.80
9.27 ± 2.74
NEO
5017
89.44 ± 4.75
88.22 ± 4.78
6.54 ± 2.24
AR
5017
99.95 ± 0.03
5.97 ± 0.33
93.98 ± 0.36
Appendix
Table 15: Single-A/B transfer at Lv1, primitive depth two. Mean ± SD across three seeds (%). N : tasks per seed; S : every support correct; Q : query correct. S∩¬Q uses all N tasks as its denominator. Inference follows Table 14 , including step-10 trajectories for Cont-Mono-Opt. Depth is the number of primitives in the generating program.
Method
Denominator
Single
Multi-A
Multi-B
One constituent
Wrong target
Wrong target
Cont-Mono
Eligible
15.05 ± 1.15
8.82 ± 4.89
4.96 ± 0.73
Full set
7.55 ± 0.58
1.92 ± 1.07
1.05 ± 0.16
Cont-Mono-Opt
Eligible
15.16 ± 0.94
8.91 ± 5.22
5.23 ± 0.85
Full set
7.60 ± 0.47
1.94 ± 1.14
1.11 ± 0.18
Disc-Mono
Eligible
3.70 ± 0.94
14.91 ± 4.29
5.78 ± 0.22
Appendix
Table 16: Output-pattern frequencies. Mean ± SD across three seeds (%). Eligible n : Single 5,017; Multi-A 2,177; Multi-B 2,117; full n : 10,000. Patterns are non-exclusive. Opt uses 2,944 depth-one tasks; full-set rescue/harm: 1.12±0.06 / 0.99±0.05 %. Within-trajectory net gain: 0.43±0.11 pp.
Figure 15: AR: support fit without query transfer. Each column shows a different depth-one Single-B task from this 256-task subset (seed 42). The adapted model fits both supports but fails on the query.
Figure 16: Transfer errors on depth-1 programs. Cont-Mono (a–c) and Disc-Mono (d–f) fit both supports but make reflection-axis, color-change, and translation-distance errors, respectively (seed 42). Color predictions match two shifts instead of one. The Disc-Mono translation error is a rare illustrative case.
Figure 17: Support fitting can damage a correct query. Three Cont-Mono-Opt cases change from correct at step 0 to incorrect at step 10. Both supports remain exact at recorded steps 0, 1, 5, and 10; support loss need not be constant.
Figure 18: Correct transformation, wrong target. Cont-Mono, Disc-Mono, and NEO fit both supports but change the wrong query object in Multi-A (seed 42). Each prediction exactly matches the correct primitive applied to another target. Dashed outlines identify the affected objects.
Figure 19: Re-encoding corrects some NEO continuations. The first rotation is exact, but the following translation moves 3, 3, and 4 cells instead of 4, 5, and 5. Re-encoding the decoded image before applying the same second code corrects these cases.
Figure 20: Re-encoding also breaks correct NEO continuations. Under the same intervention, three previously correct queries become incorrect. Together with Figure 19 , these cases show state-dependent execution; re-encoding is not a general remedy.
Figure 21: Continuous latent representations of Cont-Mono. Each point represents the latent inferred from a task’s first support pair. Colors identify the primitives that generated the source tasks.
Figure 22: Pre-quantization representations and codebook vectors of Disc-Mono. Points represent continuous vectors before quantization, colored by the primitives that generated the source tasks. Hollow diamonds mark selected codebook entries, and numeric labels denote code IDs within each checkpoint.
Figure 23: Pre-quantization representations and codebook vectors of NEO. Points represent continuous vectors before quantization, colored by the primitives that generated the source tasks. Hollow diamonds mark selected codebook entries, and numeric labels denote code IDs within each checkpoint.
Visually grounded reasoning links visual evidence to multi-step conclusions and actions, aiming to make multimodal systems accurate, auditable, and robust.
What does it mean to understand the world? Contemporary world models often operationalize understanding as accurate future prediction in latent or observation space. Developmental cognitive science, however, suggests a different view: human understanding emerges through the construction of internal theories of how the world works, even before mature language is acquired. Inspired by this theory-building view of cognition, we introduce Learning-to-Theorize, a learning paradigm for inferring explicit explanatory theories of the world from raw, non-textual observations. We instantiate this paradigm with the Neural Theorizer (NEO), a World Theory Model, that induces latent programs as a learned Language of Thought and executes them through a shared transition model. In NEO, a theory is represented as an executable, compositional program whose learned primitives can be systematically recombined to explain novel phenomena. Experiments show that this formulation enables explanation-driven generalization, allowing observations to be understood in terms of the programs that generate them.
While humans develop core visual skills long before acquiring language, contemporary Multimodal LLMs (MLLMs) still rely heavily on linguistic priors to compensate for their fragile visual understanding. We uncovered a crucial fact: state-of-the-art MLLMs consistently fail on basic visual tasks that humans, even 3-year-olds, can solve effortlessly. To systematically investigate this gap, we introduce BabyVision, a benchmark designed to assess core visual abilities independent of linguistic knowledge for MLLMs. BabyVision spans a wide range of tasks, with 388 items divided into 22 subclasses across four key categories. Empirical results and human evaluation reveal that leading MLLMs perform significantly below human baselines. Gemini3-Pro-Preview scores 49.7, lagging behind 6-year-old humans and falling well behind the average adult score of 94.1. These results show despite excelling in knowledge-heavy evaluations, current MLLMs still lack fundamental visual primitives. Progress in BabyVision represents a step toward human-level visual perception and reasoning capabilities. We also explore solving visual reasoning with generation models by proposing BabyVision-Gen and automatic evaluation toolkit. Our code and benchmark data are released at https://github.com/UniPat-AI/BabyVision for reproduction.
Liang Chen, Weichu Xie, Yiyan Liang +27
1UniPat AI · 6Peking University · 3Alibaba Group +8
Existing visual reasoning benchmarks predominantly rely on natural language prompts, evaluate narrow reasoning modalities, or depend on subjective scoring procedures such as LLM-as-judge. We introduce the TACIT Benchmark, a programmatic visual reasoning benchmark comprising 10 tasks across 6 reasoning domains: spatial navigation, abstract pattern completion, causal simulation, logical constraint satisfaction, graph theory, and topology. The benchmark provides dual-track evaluation: a generative track in which models must produce solution images verified through deterministic computer-vision pipelines, and a discriminative track offering five-way multiple choice with structurally plausible near-miss distractors. Each distractor violates exactly one structural constraint, requiring models to reason about fine-grained visual differences rather than exploit superficial cues. Version 0.1.0 distributes 6,000 puzzles (108,000 PNG images across three resolutions) with fully deterministic seeded generation and reproducible verification. The dataset, generation code, and evaluation harness are released under the Apache 2.0 license on HuggingFace (DOI: 10.57967/hf/7904).