Entity tracking emerges in sub-billion parameter language models and exceeds human performance in naturalistic narratives

Karolina Drożdż; Micha Heilbron

cs.CLJun 4, 2026

Entity tracking emerges in sub-billion parameter language models and exceeds human performance in naturalistic narratives

Authors: Karolina Drożdż, Micha Heilbron

Organizations: IDEAS Research Institute; Warsaw, Poland · Max Planck Institute for Psycholinguistics; Nijmegen, Netherlands · University of Amsterdam, Amsterdam Brain and Cognition; Amsterdam, Netherlands

Abstract

Understanding language requires tracking entities across discourse - i.e., knowing where things are and how they change, even when not explicitly stated. Whether language models perform such tracking in a human-like fashion remains unclear, in part because existing evaluations rely on artificial tasks, far removed from natural language comprehension, and lack comparisons to humans. Here, we evaluate entity tracking in both language models and humans (N = 48) using naturalistic narratives at multiple levels of complexity. In humans, we find that entity tracking degrades specifically with narrative complexity, not narrative length. In language models, we find that human-level entity tracking is already present at 410 million parameters - well below the multi-billion parameter, code-specialised models identified by prior work - and improves with scale, with contemporary models far exceeding human performance. Together, these results demonstrate that entity tracking, a core component of language understanding, emerges at model scales far smaller than previously thought.

Figures & tables

Figure 1: Conceptual overview. Left: language understanding requires tracking entities in an internal model of discourse context, as a narrative unfolds. Right: in humans (top), entity tracking accuracy decreases with situational complexity; in language models (bottom), we find a similar pattern, and an increase in capability with scale.

Figure 2: Overview of the experimental design. The top panel displays a simplified narrative at complexity 3, with two target objects (F0, F1). The bottom panels show the two evaluation formats: an explicit open recall task (left) requiring generation of the correct final location of the evaluated object, and an implicit task (right) administered as a forced-choice selection for human participants and as a probability read-out comparing correct versus incorrect continuations for language models. For full example narratives at multiple complexity levels, see Appendix E .

Figure 3: Humans show declining accuracy with increasing complexity while contemporary 70B LMs (Llama 3.3 and Qwen 2.5) maintain ceiling performance. (A) Explicit entity tracking task. (B) Implicit entity tracking task. Error bars represent 95% confidence intervals.

Figure 4: Model scaling effects on implicit entity tracking accuracy. (A) Pythia models (70M to 12B parameters). (B) OLMo 2 base models (1B to 32B parameters). Larger models generally show improved performance, with the exception of OLMo 32B-base.

Figure 5: Instruction tuning effects on OLMo 2 models (A) Explicit task. (B) Implicit task. Instruction tuning substantially improves explicit task performance, with gains increasing with model size. In contrast, instruction tuning has mixed effects on implicit task performance.

Figure 6: (A) Examples of standard, pseudoword, and improbable object types used in the stimuli. (B) Explicit task accuracy. (C) Implicit task accuracy. Object type does not affect model performance.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: OLMo 2 Instruct models on the explicit (generation) task. Instruction-tuned models show substantial scaling effects, with accuracy improving from 55% (1B) to 94% (32B) overall. The 32B model maintains high accuracy even at C5, while smaller instruction-tuned models show steep performance declines with increasing complexity.

Template	Objects
Standard	ring, key, postcard, pen, bookmark, book, wallet, watch, coin, photo, scarf, letter
Pseudoword	rint, kel, pothlard, pes, boolmarm, bood, gallel, wawed, coft, knolo, scact, lunter
Improbable
Rare	kidney, squid, brick, urn
Phys. Impos.	volcano, mountain, skyscraper, cathedral
Abstract	happiness, silence, justice, peace

Appendix

Table 1: Object words used in each template condition. The conditions include: 1) common & possible objects (standard template;

N=12

), 2) pseudowords derived from the standard objects (

N=12

), 3a) rare but possible objects (

N=4

), 3b) rare and physically impossible objects (size violation;

N=4

), and 3c) rare and logically impossible objects (abstract violation;

N=4

Figure 8: Distribution of human accuracy across complexity levels on the explicit (A; $N=41$ ) and implicit (B; $N=43$ ) tasks. Half-violins show kernel density estimates of the between-participant distribution, boxes show the median and interquartile range, diamonds indicate group means.

Explore similar work

Do Language Models Track Entities Across State Changes?
May 28, 2026Zilu Tang, Qiao Zhao, Gabriel Franco +4Entities State-Tracking
Department of Computer Science, Boston University, Boston, USA · Department of Data Science, Monash University, Indonesia · Faculty of Computer Science, University of Vienna, Austria +1
NarrativeTrack: Evaluating Entity-Centric Reasoning for Narrative Understanding
Jan 3, 2026Hyeonjeong Ha, Jinjin Ge, Bo Feng +2Video Multimodal Large Language Models Narratives
Apple · University of Illinois Urbana-Champaign
Graded Entity-Familiarity Readouts in Language Models: Polish Adaptation, Cross-Language Robustness, and Refusal Steering
Jul 15, 2026Grzegorz BrzezinkaReadout Representational Capacity
Prosit AS

May 28, 2026, cs.CLJ/K move · Enter open · S save

Do Language Models Track Entities Across State Changes?

Zilu Tang, Qiao Zhao, Gabriel Franco, Derry Wijaya+3

Department of Computer Science, Boston University, Boston, USA · Department of Data Science, Monash University, Indonesia · Faculty of Computer Science, University of Vienna, Austria+1

Entities State-Tracking Transformer Architectures

Open paper AskPDF arXiv

Entity tracking (ET), the ability to keep track of states, is a fundamental skill that underlies complex reasoning. An increasing amount of work investigates how transformer language models (LMs) solve entity binding

\textit{without}

state changes. However, there is limited understanding of how non-toy LMs address ET problems of realistic difficulties expressed in natural language. To this end, we investigate the mechanisms underlying ET in more complex scenarios featuring multiple state-changing operations. We find that LMs do not incrementally track world states across tokens or query-relevant states across layers, but simply aggregate relevant information in parallel at the last token when the query becomes evident. We further investigate mechanisms of individual operations (

\texttt{PUT}

\texttt{REMOVE}

\texttt{MOVE}

) to characterize this non-incremental ET mechanism. Surprisingly, LMs implement the

\texttt{REMOVE}

operation with a fragile global suppression tag; this global removal mechanism predicts various failure modes that we confirm behaviorally. We provide a mechanistic solution of nullifying this tag to partially address this issue. Overall, our findings reveal that LMs solve a fundamentally sequential task using a non-sequential strategy. More broadly, our work illustrates how behavioral and mechanistic analyses can fruitfully interact. Behavioral results inform mechanistic hypotheses, and insights from mechanistic analyses help build stronger behavioral evaluations by predicting failure modes missing from existing evaluations.

Abstract

Explore similar work

Do Language Models Track Entities Across State Changes?

NarrativeTrack: Evaluating Entity-Centric Reasoning for Narrative Understanding

Graded Entity-Familiarity Readouts in Language Models: Polish Adaptation, Cross-Language Robustness, and Refusal Steering