Open-world AI: a Tale of Three Requirements

Jianyu Zhang; Léon Bottou

cs.AIOct 14, 2026

Open-world AI: a Tale of Three Requirements

Authors: Jianyu Zhang, Léon Bottou

Organizations: FAIR, Meta∗ · Flatiron Institute

Abstract

The closed world asks AI for specialized competence and lets it manufacture the iid data its theory assumes: a fixed criterion licenses collecting examples until the task is covered. The open world asks for versatility --- taking up tasks a machine was not built for, from few examples and with little prior from a human designer --- and offers data that is grouped, not shuffled. Its tasks are not known in advance, so a machine must do inference-time learning (1st requirement). Modeling tasks as sharing latent factors and decomposing the training objective, we derive two more: a rich representation, retaining more than iid generalization requires, and a predictive disentanglement that aligns inputs and outputs into matched low-dimensional groups. The analysis further predicts the need for a consistent inner learner, and the benefit of long contexts spanning related tasks. We test these claims on language models up to 8B, asking whether a model has a property (qualitative) rather than how much it wins (quantitative). Models that meet the requirements improve as data from related tasks accumulates; the rest degrade, and neither scale nor layer-type interleaving repairs this.

Figures & tables

Figure 1: Latent factors, task, and related tasks.

Figure 2: Model architecture groups. Architectures within each group differ only in the dark box. (a) Basic : the dark box is one of

\{

Mosaics, attention, DeltaNet

\}

. (b) Modern hybrid :

\{

DeltaNet, sliding window attention ( swa )

\}

, interleaved with full attention ( nope ). (c)

+

\{

Mosaics, attention

\}

; the dark box skips the

s

most recent positions, the swa below it uses a small window

w

, and the leaky average in Mosaics is removed. (d) Two-stage :

\{

Mosaics, attention

\}

; as (c), but the target task and the related tasks are handled separately.

Figure 3

$l^{\prime}$	Transformer				DeltaNet				Hybrid swa				Hybrid DeltaNet
$l$	4k	8k	16k	32k	4k	8k	16k	32k	4k	8k	16k	32k	4k	8k	16k	32k
4k	48.40	—	—	—	47.40	—	—	—	48.19	—	—	—	48.57	—	—	—
8k	48.48	48.70	—	—	47.49	47.48	—	—	47.88	47.80	—	—	48.49	48.70	—	—
16k	48.05	48.27	48.60	—	47.49	47.43	47.58	—	47.97	47.98	48.01	—	48.15	48.33	48.57	—
32k	47.72	48.07	48.44	48.49	47.27	47.23	47.31	47.15	47.65	47.60	47.65	47.53	46.49	46.51	46.52	46.37
64k	47.51	47.85	48.23	48.39	47.35	47.30	47.40	47.21	47.54	47.55	47.60	47.46	46.24	46.26	46.28	46.13

Table 2: Top-1 accuracy (%) on SimpleWiki for architectures whose inner learners lack consistency, under the same protocol as Table 1 . Red:

l=l^{\prime}

(trained at the evaluation length). Blue:

l=64

k. A longer max training length

l

does not help these models learn new tasks; in almost every case it hurts them.

Figure 4: GSM8K accuracy (%) after fine-tuning on math, against training length

l

, for the four architectures of Table 2 . Accuracy shows no systematic gain as

l

grows, unlike Memory Mosaics (Figure 3 ). We do not compare accuracy quantitatively across architectures, since it is easily altered by small engineering details. For example, Memory Mosaics has 25% fewer learning-module parameters 14 ; filling this gap by adjusting its heads raises its accuracy to 28.73% at

l=4

k and 30.78% at

l=64

k, while its trend with

l

is unchanged (Appendix D ).

	SimpleWiki top-1 (%), $l^{\prime}=4$ k						GSM8K accuracy (%), fine-tuned
	Number of groups $G$ (heads)						Number of groups $G$ (heads)
Model	4	8	16	32	64	$\Delta>0$	4	8	16	32	64	$\Delta>0$
Memory Mosaics	48.43	48.63	48.97	49.07	49.02	✔	21.53	25.32	25.78	28.43	27.60	✔
Transformer	48.31	48.38	48.05	47.92	47.08	✘	28.81	26.76	27.52	26.84	24.18	✘
Hybrid swa	48.00	47.95	47.97	47.67	46.85	✘	26.16	27.67	27.98	25.55	22.82	✘

Table 3: Effect of the number of groups

G

(heads) at fixed model width: raising the number of heads raises

G

and shrinks each group. All models are pre-trained at length

l=16

k. Left: top-1 accuracy on SimpleWiki, evaluated at length

l^{\prime}=4

k. Right: gsm8k accuracy after fine-tuning as in Figure 3 .

\Delta

is ( 64 heads

-

4 heads ).

Figure 5: Left: Transformer vs Transformer

+

. Right: Memory Mosaics vs Memory Mosaics

+

x

-axis: evaluation per-document length. The gap widens with length.

Figure 6: Accuracy as the number of related tasks in the context grows. Training length

l=16

k. At

1.5

B: 1) RNN and hybrid architectures; 2) Transformer versus two-stage Transformer

+

; 3) Memory Mosaics versus two-stage Memory Mosaics

+

. Panel 4) repeats the comparison at

8

b .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

	$1.5$ B	$8$ B
model dimension $d$	2 048	4 096
layers	24	32
head dimension	128	128
heads	16	32
ffn hidden dimension	5 632 $(2.75\,d)$	14 336 $(3.5\,d)$
vocabulary (Mistral-7B-v0.1)	32 000	32 000

Appendix

Table 4: Architecture settings shared by all models at each scale. The feed-forward width follows the Llama SwiGLU convention (see text), and the input and output embeddings are not tied.

Source	% of documents
Cosmopedia ( Ben Allal et al., 2024 )	$71.17$
NemotronCC-high ( Su et al., 2025 )	$22.89$
open-web-math ( Azerbayev et al., 2024 )	$4.77$
arXiv ( Azerbayev et al., 2024 )	$0.88$
ShareGPT ( Vicuna, 2023 )	$0.21$
Gutenberg books ( Manu, 2023 )	$0.07$

Appendix

Table 5: Training mixture, as a share of documents.

Figure 7: Memory Mosaics and two-stage Memory Mosaics

+

on all

50

Simple English Wikipedia clusters, at

1.5

B. As Figure 6 , without the cluster selection.

Figure 8: Accuracy as the number of related tasks in the context grows, at

1.5

B. As Figure 6 , but the whole of Simple English Wikipedia is treated as related tasks, with no clustering. 1) RNN and hybrid architectures; 2) Transformer versus two-stage Transformer

+

; 3) Memory Mosaics versus two-stage Memory Mosaics

+

Model	4k	64k	$\Delta>0$
	Training length $l$
Memory Mosaics ( $16~\text{heads}\times 128~\text{dim}$ )	25.93	28.58	✔
Memory Mosaics ( $43~\text{heads}\times 64~\text{dim}$ )	28.73	30.78	✔

Appendix

Table 6: gsm8k accuracy (%) after fine-tuning as in Figure 3 , at 1.5B.

\Delta

is (64k

-

4k).

Figure 9: As Figure 5 , at

1.5

B, with a third curve: Transformer

+

whose long-range attention also carries rope .

$l^{\prime}$	Hybrid swa -4k				Hybrid swa -2k
$l$	4k	8k	16k	32k	4k	8k	16k	32k
4k	47.94	—	—	—	48.19	—	—	—
8k	47.97	47.90	—	—	47.88	47.80	—	—
16k	47.91	47.91	48.00	—	47.97	47.98	48.01	—
32k	47.68	47.69	47.71	47.62	47.65	47.60	47.65	47.53
64k	47.54	47.54	47.64	47.49	47.54	47.55	47.60	47.46

Appendix

Table 7: Top-1 accuracy (%) on SimpleWiki for Hybrid swa with a sliding-window length of 4k vs 2k, under the same protocol as Table 1 . The window length does not change the conclusion: for both, a longer max training length

l

does not help the model learn new tasks.

Figure 10: As Figure 5 , at

1.5

B, sweeping the short-memory implementation in Memory Mosaics

+

: attention with rope , a Mosaics module with the leaky-average key extractor, and DeltaNet.

Model	4	8	16	32	64	$\Delta>0$
	Number of groups $G$ (heads)
Memory Mosaics	48.43	48.63	48.97	49.07	49.02	✔
Memory Mosaics [no per-head gate]	48.15	48.29	48.46	48.52	48.47	✔
Transformer	48.31	48.38	48.05	47.92	47.08	✘
Transformer [ nope ]	45.27	45.44	45.36	45.37	44.40	✘
Hybrid swa	48.00	47.95	47.97	47.67	46.85	✘

Appendix

Table 8: Top-1 accuracy (%) on SimpleWiki, evaluated at length

l^{\prime}=4

k; all models trained at length

l=16

k. As Table 3 , with two ablations added: Memory Mosaics without the per-head gate, and Transformer without position encoding [ nope ]. Model width is fixed, so raising the number of heads raises the number of groups

G

and shrinks each group. Row maxima in bold.

\Delta

is ( 64 heads

-

4 heads ).

Figure 11: Output magnitude of the target-task and related-task parts of the long memory in two-stage Memory Mosaics

+

1.5

B, against evaluation length.

Continual Learning for LLMs

Continual learning for large language models studies how models absorb new knowledge and skills while retaining previously learned capabilities.

See all 231 papers

Language Models Struggle to Use Representations Learned In-Context
Feb 4, 2026Michael A. Lepori, Tal Linzen, Ann Yuan +1In-Context Learning LLM World Models
Brown University · Google Research · New York University +1
Do World Models Learn Global Understanding?
Sep 28, 2026Alexander Detkov, Matt ThomsonLLM World Models World Model Learning
California Institute of Technology Pasadena, CA 91125, USA
Data World: Scaling Data Environments from Scratch
Oct 15, 2026Zhanli Li, Zeyuan Xu, Chaoxu Pang +16Synthetic Data Generation for LLM Agents Data Analysis Agents
Longcat Interaction Team, Meituan · Zhongnan University of Economics and Law · Hunan University +7

Feb 4, 2026J/K move · Enter open · S save

Language Models Struggle to Use Representations Learned In-Context

Michael A. Lepori, Tal Linzen, Ann Yuan, Katja Filippova

Brown University · Google Research · New York University+1

In-Context Learning LLM World Models

Open paper AskPDF arXiv

Though large language models (LLMs) have enabled great success across a wide variety of tasks, they still appear to fall short of one of the loftier goals of artificial intelligence research: creating an artificial system that can adapt its behavior to radically new contexts upon deployment. One important step towards this goal is to create systems that can induce rich representations of data that are seen in-context, and then flexibly deploy these representations to accomplish goals. Recently, Park et al. (2024) demonstrated that current LLMs are indeed capable of inducing such representation from context (i.e., in-context representation learning). The present study investigates whether LLMs can use these representations to complete simple downstream tasks. We first assess whether open-weights LLMs can use in-context representations for next-token prediction, and then probe models using a novel task, adaptive world modeling. In both tasks, we find evidence that open-weights LLMs struggle to deploy representations of novel semantics that are defined in-context, even if they encode these semantics in their latent representations. Furthermore, we assess closed-source, state-of-the-art reasoning models on the adaptive world modeling task, demonstrating that even the most performant LLMs cannot reliably leverage novel patterns presented in-context. Overall, this work seeks to inspire novel methods for encouraging models to not only encode information presented in-context, but to do so in a manner that supports flexible deployment of this information.

Abstract

Continual Learning for LLMs

Explore similar work

Language Models Struggle to Use Representations Learned In-Context

Do World Models Learn Global Understanding?

Data World: Scaling Data Environments from Scratch