cs.CLNov 17, 2026

Tracking Representations in LLM Reasoning Traces for Iterated Games

Authors: Chaipat Tirapongprasert, Rajarshi Das, Parijat Dube

Organizations: Columbia University · MQube Cognition

Abstract

Large language model (LLM) agents achieve state-of-the-art performance across domains through chain-of-thought (CoT) reasoning. Despite their ubiquitous use, little is understood about the relationship between the externalized trace and the model's actual computation. We address this by investigating how LLM agents make decisions in iterative games with incomplete information. Using a series of mechanistic interpretability techniques, we find that the model forms and iteratively revises an internal representation of the game throughout the CoT. Once the model prematurely commits to an incorrect belief, it rarely recovers. This failure is manifested in both the literal trace and the layer activations. Ultimately, as multi-agent systems are more prevalent, our work opens the door to a more focused diagnosis of their strategic reasoning errors. For illustrative purposes, we offer an interactive visualizer of our methodology on selected sample traces at https://interp-self.github.io/gametrace/

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Reasoning or Pattern Exploitation? Mechanistic Insights from RL-Trained Language Models

    Oct 29, 2026Aishik Nagar, Cheston TanRL for Language Model ReasoningLLM Interpretability

  2. Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and Actions

    Apr 30, 2026Jan Sobotka, Mustafa O. Karabag, Ufuk TopcuImperfect-Information GamesLLM Interpretability