Large language models (LLMs) exhibit failures on elementary symbolic tasks such as character counting in a word, despite excelling on complex benchmarks. Although this limitation has been noted, the internal reasons remain unclear. We use character counting (e.g., "How many p's are in apple?") as a minimal, controlled probe that isolates token-level reasoning from higher-level confounds. Using this setting, we uncover a consistent phenomenon across modern architectures, including LLaMA, Qwen, and Gemma: models often compute the correct answer internally yet fail to express it at the output layer. Through mechanistic analysis combining probing classifiers, activation patching, logit lens analysis, and attention head tracing, we show that character-level information is encoded in early and mid-layer representations. However, this information is attenuated by a small set of components in later layers, especially the penultimate and final layer MLP. We identify these components as negative circuits: subnetworks that downweight correct signals in favor of higher-probability but incorrect outputs. Our results lead to two contributions. First, we show that symbolic reasoning failures in LLMs are not due to missing representations or insufficient scale, but arise from structured interference within the model's computation graph. This explains why such errors persist and can worsen under scaling and instruction tuning. Second, we provide evidence that LLM forward passes implement a form of competitive decoding, in which correct and incorrect hypotheses coexist and are dynamically reweighted, with final outputs determined by suppression as much as by amplification. These findings carry implications for interpretability and robustness: simple symbolic reasoning exposes weaknesses in modern LLMs, underscoring need for design strategies that ensure information is encoded and reliably used.
Figures & tables
Figure 1: Overview of how character-level information flows through an LLM during the task.
Prompt Templates
The number of <target_letter> ’s in <count_subject> is <count>
Counting the letter <target_letter> in <count_subject> gives <count>
The number of <target_letter> ’s found in <count_subject> is <count>
The <target_letter> count for <count_subject> equals <count>
The total number of <target_letter> ’s in <count_subject> is <count>
Table 1: Prompt templates for probing character count representations. For example: “The number of p’s in apple is 2”, where p is the target letter and apple is the count_subject.
Corruption Type
Clean Prompt
Corrupted Prompt
Word Corruption
The number of a’s in apple is
The number of a’s in ada is
Letter Corruption
The number of a ’s in apple is
The number of p ’s in apple is
Word+Letter Corruption
The number of a ’s in apple is
The number of p ’s in ada is
Table 2: Examples of clean and corrupted prompts used in activation patching. Each corruption alters the expected letter count while maintaining the prompt structure.
Figure 2: Accuracy of predicted letter counts. The dotted line indicates the random baseline accuracy. We calculate the proportion of samples where deviation is zero
Figure 3: Layerwise probe accuracy
Figure 4: Logit difference across layers for the correct count token, averaged over approximately 100 samples. Higher values indicate layers where suppression of the correct answer occurs. The final label final denotes the final hidden state after all layers. Results are computed over samples where the model produces an incorrect prediction, corresponding to the “always predict 1” strategy for Qwen2.5-3B and the near-uniform random strategy for LLaMA3.2-3B.
Figure 5: Activation patching heatmaps when corrupting the word, letter or both letter and word for Qwen2.5-3B attention heads averaged over 1000 samples. Green regions indicate components where clean activations restore the correct behavior, helping identify localized circuits for character counting. The Red regions indicate components which diminish performance when patched in.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Confusion matrices for predicted versus true letter counts. The two models represent the two dominant failure modes observed across all model families: a random strategy (LLaMA) and a constant prediction strategy (Qwen).
Figure 7: Distributions of absolute deviations between the correct letter count and the predicted count ( ∣predicted count−correct count∣ ). The y-axis indicates the proportion of predictions at each deviation. Every stacked bar represents a distribution of deviations. Bars labelled with values above zero reflect over- or under-counting errors relative to the true letter count and the bars labelled deviation=0 represent the accuracy (no deviation). The number on top of every bar denotes how many instances of the category were present.
Figure 8: Layerwise probe accuracy heatmap for individual target letters at the final token of the word for Qwen2.5-3B. Each cell represents the probe accuracy for a specific layer and target letter.
Figure 9: Prompt templates used for the character counting task. Each template was evaluated under the same experimental setting.
Figure 10: Accuracy of predicted letter counts for instruction-tuned model variants. The dotted line indicates the random baseline accuracy for comparison.
Figure 11: Activation patching heatmaps for Llama Models averaged over 100 samples. Green regions indicate components where clean activations restore the correct behavior, helping identify localized circuits for character counting.
Figure 12: Activation patching heatmaps when corrupting the word, letter or both letter and word for Llama attention heads averaged over 100 samples. Green regions indicate components where clean activations restore the correct behavior, helping identify localized circuits for character counting. The Red regions indicate components which diminish performance when patched in.
Figure 13: Activation patching heatmaps for Gemma Models averaged over 100 samples. Green regions indicate components where clean activations restore the correct behavior, helping identify localized circuits for character counting.
Figure 14: Activation patching heatmaps when corrupting the word, letter or both letter and word for Gemma attention heads averaged over 100 samples. Green regions indicate components where clean activations restore the correct behavior, helping identify localized circuits for character counting. The Red regions indicate components which diminish performance when patched in.
Figure 15: Activation patching heatmaps for Qwen Models averaged over 100 samples. Green regions indicate components where clean activations restore the correct behavior, helping identify localized circuits for character counting.
Figure 16: Activation patching heatmaps when corrupting the word, letter or both letter and word for Llama attention heads averaged over 100 samples. Green regions indicate components where clean activations restore the correct behavior, helping identify localized circuits for character counting. The Red regions indicate components which diminish performance when patched in.
Large language models (LLMs) exhibit failure modes on seemingly trivial tasks. We propose a formalisation of LLM interaction using a deterministic multi-tape Turing machine, where each tape represents a distinct component: input characters, tokens, vocabulary, model parameters, activations, probability distributions, and output text. The model enables precise localisation of failure modes to specific pipeline stages, revealing, e.g., how tokenisation obscures character-level structure needed for counting tasks. The model clarifies why techniques like chain-of-thought prompting help, by externalising computation on the output tape, while also revealing their fundamental limitations. This approach provides a rigorous, falsifiable alternative to geometric metaphors and complements empirical scaling laws with principled error analysis.
Large language models fail at counting repeated tokens despite strong performance on broader reasoning benchmarks. These failures are commonly attributed to limitations in internal count tracking. We show this attribution is wrong. Linear probes on the residual stream decode the correct count with near-perfect accuracy at every post-embedding layer, across all model depths. This holds even at the exact layers where the wrong answer crystallizes while the model simultaneously outputs an incorrect count. Attention patterns show no evidence of collapse over repeated tokens and tokenization artifacts account for none of the failure. Instead, a format-triggered multi-layer perceptron (MLP) block overwrites the correctly-encoded count with a fixed wrong answer at roughly 88--93,% network depth. This prior fires for repeated word-tokens in space-separated list format and is absent for repeated digit-tokens. It is suppressed by comma-separated delimiters in larger models but persists in smaller ones. The finding holds across Llama-3.2 (1B and 3B) and Qwen2.5 (1.5B, 3B and 7B) at consistent relative depth. Counting failure is a failure of routing not of representation and the two require different interventions.
Large language models (LLMs) trained with canonical tokenization exhibit surprising robustness to non-canonical inputs such as character-level tokenization, yet the mechanisms underlying this robustness remain unclear. We study this phenomenon through mechanistic interpretability and identify a core process we term word recovery. We first introduce a decoding-based method to detect word recovery, showing that hidden states reconstruct canonical word-level token identities from character-level inputs. We then provide causal evidence by removing the corresponding subspace from hidden states, which consistently degrades downstream task performance. Finally, we conduct a fine-grained attention analysis and show that in-group attention among characters belonging to the same canonical token is critical for word recovery: masking such attention in early layers substantially reduces both recovery scores and task performance. Together, our findings provide a mechanistic explanation for tokenization robustness and identify word recovery as a key mechanism enabling LLMs to process character-level inputs.
Zhipeng Yang, Shu Yang, Lijie Hu +1
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Provable Responsible AI and Data Analytics (PRADA) Lab · King Abdullah University of Science and Technology