Entity tracking emerges in sub-billion parameter language models and exceeds human performance in naturalistic narratives
Authors: Karolina Drożdż, Micha Heilbron
Organizations: IDEAS Research Institute; Warsaw, Poland · Max Planck Institute for Psycholinguistics; Nijmegen, Netherlands · University of Amsterdam, Amsterdam Brain and Cognition; Amsterdam, Netherlands
Understanding language requires tracking entities across discourse - i.e., knowing where things are and how they change, even when not explicitly stated. Whether language models perform such tracking in a human-like fashion remains unclear, in part because existing evaluations rely on artificial tasks, far removed from natural language comprehension, and lack comparisons to humans. Here, we evaluate entity tracking in both language models and humans (N = 48) using naturalistic narratives at multiple levels of complexity. In humans, we find that entity tracking degrades specifically with narrative complexity, not narrative length. In language models, we find that human-level entity tracking is already present at 410 million parameters - well below the multi-billion parameter, code-specialised models identified by prior work - and improves with scale, with contemporary models far exceeding human performance. Together, these results demonstrate that entity tracking, a core component of language understanding, emerges at model scales far smaller than previously thought.
Figures & tables
Figure 1: Conceptual overview. Left: language understanding requires tracking entities in an internal model of discourse context, as a narrative unfolds. Right: in humans (top), entity tracking accuracy decreases with situational complexity; in language models (bottom), we find a similar pattern, and an increase in capability with scale.
Figure 2: Overview of the experimental design. The top panel displays a simplified narrative at complexity 3, with two target objects (F0, F1). The bottom panels show the two evaluation formats: an explicit open recall task (left) requiring generation of the correct final location of the evaluated object, and an implicit task (right) administered as a forced-choice selection for human participants and as a probability read-out comparing correct versus incorrect continuations for language models. For full example narratives at multiple complexity levels, see Appendix E .
Figure 3: Humans show declining accuracy with increasing complexity while contemporary 70B LMs (Llama 3.3 and Qwen 2.5) maintain ceiling performance. (A) Explicit entity tracking task. (B) Implicit entity tracking task. Error bars represent 95% confidence intervals.
Figure 4: Model scaling effects on implicit entity tracking accuracy. (A) Pythia models (70M to 12B parameters). (B) OLMo 2 base models (1B to 32B parameters). Larger models generally show improved performance, with the exception of OLMo 32B-base.
Figure 5: Instruction tuning effects on OLMo 2 models (A) Explicit task. (B) Implicit task. Instruction tuning substantially improves explicit task performance, with gains increasing with model size. In contrast, instruction tuning has mixed effects on implicit task performance.
Figure 6: (A) Examples of standard, pseudoword, and improbable object types used in the stimuli. (B) Explicit task accuracy. (C) Implicit task accuracy. Object type does not affect model performance.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: OLMo 2 Instruct models on the explicit (generation) task. Instruction-tuned models show substantial scaling effects, with accuracy improving from 55% (1B) to 94% (32B) overall. The 32B model maintains high accuracy even at C5, while smaller instruction-tuned models show steep performance declines with increasing complexity.
Table 1: Object words used in each template condition. The conditions include: 1) common & possible objects (standard template; N=12 ), 2) pseudowords derived from the standard objects ( N=12 ), 3a) rare but possible objects ( N=4 ), 3b) rare and physically impossible objects (size violation; N=4 ), and 3c) rare and logically impossible objects (abstract violation; N=4 ).
Figure 8: Distribution of human accuracy across complexity levels on the explicit (A; N=41 ) and implicit (B; N=43 ) tasks. Half-violins show kernel density estimates of the between-participant distribution, boxes show the median and interquartile range, diamonds indicate group means.
Entity tracking (ET), the ability to keep track of states, is a fundamental skill that underlies complex reasoning. An increasing amount of work investigates how transformer language models (LMs) solve entity binding without state changes. However, there is limited understanding of how non-toy LMs address ET problems of realistic difficulties expressed in natural language. To this end, we investigate the mechanisms underlying ET in more complex scenarios featuring multiple state-changing operations. We find that LMs do not incrementally track world states across tokens or query-relevant states across layers, but simply aggregate relevant information in parallel at the last token when the query becomes evident. We further investigate mechanisms of individual operations (PUT, REMOVE, MOVE) to characterize this non-incremental ET mechanism. Surprisingly, LMs implement the REMOVE operation with a fragile global suppression tag; this global removal mechanism predicts various failure modes that we confirm behaviorally. We provide a mechanistic solution of nullifying this tag to partially address this issue. Overall, our findings reveal that LMs solve a fundamentally sequential task using a non-sequential strategy. More broadly, our work illustrates how behavioral and mechanistic analyses can fruitfully interact. Behavioral results inform mechanistic hypotheses, and insights from mechanistic analyses help build stronger behavioral evaluations by predicting failure modes missing from existing evaluations.
Zilu Tang, Qiao Zhao, Gabriel Franco +4
Department of Computer Science, Boston University, Boston, USA · Department of Data Science, Monash University, Indonesia · Faculty of Computer Science, University of Vienna, Austria +1
Multimodal large language models (MLLMs) have achieved impressive progress in vision-language reasoning, yet their ability to understand temporally unfolding narratives in videos remains underexplored. True narrative understanding requires grounding who is doing what, when, and where, maintaining coherent entity representations across dynamic visual and temporal contexts. We introduce NarrativeTrack, the first benchmark to evaluate narrative understanding in MLLMs through fine-grained entity-centric reasoning. Unlike existing benchmarks limited to short clips or coarse scene-level semantics, we decompose videos into constituent entities and examine their continuity via a Compositional Reasoning Progression (CRP), a structured evaluation framework that progressively increases narrative complexity across three dimensions: entity existence, entity changes, and entity ambiguity. CRP challenges models to advance from temporal persistence to contextual evolution and fine-grained perceptual reasoning. A fully automated entity-centric pipeline enables scalable extraction of temporally grounded entity representations, providing the foundation for CRP. Evaluations of state-of-the-art MLLMs reveal that models fail to robustly track entities across visual transitions and temporal dynamics, often hallucinating identity under context shifts. Open-source general-purpose MLLMs exhibit strong perceptual grounding but weak temporal coherence, while video-specific MLLMs capture temporal context yet hallucinate entities' contexts. These findings uncover a fundamental trade-off between perceptual grounding and temporal reasoning, indicating that narrative understanding emerges only from their integration. NarrativeTrack provides the first systematic framework to diagnose and advance temporally grounded narrative comprehension in MLLMs.
Can a language model estimate its familiarity with an entity before generating an answer? We study activations at the final prompt token in twelve instruction-tuned models from the Bielik, PLLuM, Gemma-4, and Qwen3 families, using a new dataset of 1,440 Polish entities spanning four domains and ten Wikipedia-pageview deciles, plus fabricated controls. Familiarity-probe scores separate real from fabricated entities in every family; in the Polish-adapted Bielik and PLLuM families they additionally track entity popularity (model-mean Spearman ρ 0.28-0.57, versus at most 0.11 in Gemma-4 and Qwen3), a pattern more strongly associated with Polish adaptation than with parameter count in this model sample. In a paired experiment on two families, probes retain 96-101% of within-language AUROC when the Polish question stem is replaced with an English one around unchanged entity names, showing robustness to prompt language in this setting. In Gemma-4-12B, the only model that natively refuses, adding a one-dimensional familiarity direction at a single layer moves refusal rates monotonically in both directions (0.24 to 1.00 on well-known entities; 0.73 to 0.00 on unknown ones). Finally, a calibrated familiarity probe is competitive among pre-generation abstention gates, although post-generation detectors better predict behavioral error on average. These results support a graded pre-generation entity-familiarity readout, and a separation between representational familiarity and the policy that converts it into abstention.