State Tracking in Language Models

Follow this topic Ask about this topic

Latest papers

8 papers3 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

A Fine-Grained Hierarchy and Benchmark for State Tracking in Language Models
Nov 2, 2026Jackson Petty, William Merrill, Ashish SabharwalState Tracking in Language Models LLM Evaluation
Allen Institute for AI · New York University · Toyota Technological Institute at Chicago
Agent Memory as Rate-Limited Filtering: Explicit State Outlasts Long Context Beyond a Reader-Dependent Horizon
Nov 2, 2026Tianhao Guo, Chongyao GuoState Tracking in Language Models Agent Memory Management
Self-Assessment Reinforcement Learning for Search-State Awareness in Retrieval-Augmented Large Language Models
Oct 21, 2026Taewhoo Lee, Minju Song, Jungwoo Park +6State Tracking in Language Models RL for RAG
Korea University · Amazon · AIGEN Sciences +2
LEAPS: Long-context Evaluation of Adaptive Pattern Learning and State Tracking
Oct 19, 2026Howard Yen, Nitya Mani, Alex Renda +1State Tracking in Language Models Long-Context Language Model Evaluation
Princeton Language and Intelligence, Princeton University · Jane Street
What Can't Jev Decide? Probing the Limits of One-Pass Decision Models
Oct 18, 2026Wang Yang, Xinpeng Li, Yanyan Zhang +8State Tracking in Language Models LLM Decision-Making
Case Western Reserve University · NII LLMC (Japan) · Georgia Institute of Technology +1
Comparing Transformers and Hybrid Models at the Token Level
Jun 18, 2026Yanhong Li, William MerrillLLM Evaluation Transformer
Allen Institute for AI
Why We Need World Models for AGI: Where LLMs Fail and How World Models May Outperform
May 13, 2026Feisal Alaswad, Batoul Aljaddouh, Maher Alrahhal +2Latent Dynamics Modeling Long-Horizon Agent Evaluation
Department of Computing Technologies SRM Institute of Science and Technology Kattankulathur 603203, Tamil Nadu, India · Bio-Sensing and Bio-Sensors Group Smart Automation and Communication Technologies Research Institute of Sciences and Engineering University of Sharjah, UAE · Department of Computer Engineering College of Computing and Informatics University of Sharjah, UAE
Reasoning Primitives in Hybrid and Non-Hybrid LLMs: Do Architectural Differences Yield Advantages in State-Tracking and Recall?
Apr 23, 2026Shivam Rawat, Lucie Flek, Florian Mai +1CoT Reasoning LLM Memory
Lamarr Institute for Machine Learning and Artificial Intelligence · Rheinische Friedrich-Wilhelms-Universität Bonn

Nov 2, 2026J/K move · Enter open · S save

A Fine-Grained Hierarchy and Benchmark for State Tracking in Language Models

Jackson Petty, William Merrill, Ashish Sabharwal

Allen Institute for AI · New York University · Toyota Technological Institute at Chicago

State Tracking in Language Models LLM Evaluation Recurrent Language Models

Open paper AskPDF arXiv

Theoretical limitations of transformers on state tracking---the ability to track information about entities or variables over long contexts---have motivated new recurrent language modeling (LM) architectures such as Gated DeltaNet (GDN). These architectures have, in particular, been benchmarked against the

S_5

word problem, a theoretically hard state-tracking task that captures abilities such as code evaluation under arbitrary variable assignment. It, however, remains open whether other, simpler forms of state tracking might both be more relevant in practice and already explain observed differences between these architectures. We propose evaluating state tracking using a hierarchy of word problems with four increasing levels of difficulty: r-trivial, corresponding to immutable assignment to variables in programming; aperiodic, where assignments are mutable; soluble, where assignments are self-referential; and general finite state tracking. Theoretically, we show that the class of problems transformers are thought to length-generalize on does not even contain all aperiodic state tracking. On the other hand, we prove that GDN with positive eigenvalues expresses all aperiodic problems, and the regular languages recognized by certain transformers are precisely all soluble ones, thus providing a fine-grained picture of state tracking capabilities of various architectures. Taking these ideas to practice, we construct StateBench, a 4-tier long-horizon state-tracking benchmark. Our systematic direct training experiments across architectures largely align with theoretical expectation, albeit with interesting observations at very large scale length-generalization. Additionally, we also evaluate pretrained models---both open hybrid and closed frontier ones---on StateBench, finding that none of them robustly solves even soluble state tracking, suggesting a direction for future improvement.