Autoregressive large language models (LLMs) generate text token-by-token, requiring n forward passes to produce a sequence of length n. Recent work, Exploring the Latent Capacity of LLMs for One-Step Text Reconstruction (Mezentsev and Oseledets), shows that frozen LLMs can reconstruct hundreds of tokens from only two learned proto-tokens in a single forward pass, suggesting a path beyond the autoregressive paradigm. In this paper, we study what information these proto-tokens encode and how they behave under reconstruction and controlled constraints. We perform a series of experiments aimed at disentangling semantic and syntactic content in the two proto-tokens, analyzing stability properties of the e-token, and visualizing attention patterns to the e-token during reconstruction. Finally, we test two regularization schemes for "imposing" semantic structure on the e-token using teacher embeddings, including an anchor-based loss and a relational distillation objective. Our results indicate that the m-token tends to capture semantic information more strongly than the e-token under standard optimization; anchor-based constraints trade off sharply with reconstruction accuracy; and relational distillation can transfer batch-level semantic relations into the proto-token space without sacrificing reconstruction quality, supporting the feasibility of future non-autoregressive seq2seq systems that predict proto-tokens as an intermediate representation.
Figures & tables
Figure 1: Attention to the e -token averaged over heads (examples).
Figure 2: Layer-wise heatmaps of attention to the e -token (averaged over heads).
Figure 3: Head-level attention to the e -token for selected layers/heads (examples).
Figure 4: Reconstruction accuracy as a function of noise intensity α for different noise types.
Figure 5: t-SNE visualization of optimized e -token embeddings for original, lexical, and semantic augmentations.
Figure 6: t-SNE visualization of optimized m -token embeddings for original, lexical, and semantic augmentations.
Figure 7: Reconstruction token accuracy vs. target length under anchor regularization with a small weight.
Figure 8: Cosine similarity between e and the teacher embedding over optimization steps (anchor).
Figure 9: Cosine similarity between e and the teacher embedding over optimization steps for a larger anchor weight.
Figure 10: Token accuracy vs. target length for a larger anchor weight, showing degraded reconstruction.
Figure 11: Distributions of semantic-matrix correlation per step for relational distillation, with and without a shared m -token.
Figure 12: Token-accuracy distributions with and without the relational distillation constraint.
Figure 13: Improved semantic-matrix correlation distributions after applying the relational distillation objective.
Large language models (LLMs) generate text by auto-regressively sampling the next token. This inherently leads to a many-to-many mapping between prompts and responses, complicating the task of inferring prompts from observed outputs. Prior work on LLM inversion frames prompt recovery as a semantic reconstruction task. They rely on fine-tuning pretrained sequence-to-sequence models on large external datasets--and requiring access to model weights or logits--to generate semantically plausible prompts. In contrast, we present a functional approach to inverting a given LLM in a black-box setting, without auxiliary aids. We train an explicit inverse language model entirely from scratch on data synthetically generated from the target LLM itself. Analogous to forward next-token prediction, our inverse model is trained using previous-token prediction, establishing a generative link between the forward and inverse processes that enables faithful prompt reconstruction. Moreover, it naturally supports diverse prompt reconstructions through sampling, whereby all such prompts induce similar responses under the forward, target LLM. Our approach generalises across datasets and exhibits transferability in reconstructing prompts from responses generated by different LLMs. Further, across the set of token based evaluation metrics for prompt and response reconstructions, our approach outperforms prior work.
Pirzada Suhail, Nagasai Saketh Naidu, Atanu R Sinha +1
Large language models represent text as hidden numerical patterns, but what information these representations actually contain remains a mystery. Can we recover the original text from just one of them? In this paper, we introduce Rep2Text, a method designed to answer this question by reconstructing text from a single hidden representation. We find that these representations preserve much more information than we might expect. For short texts of about 16 words, Rep2Text can recover roughly half of the original words while preserving their overall meaning. As the text gets longer, recovering the exact wording becomes harder, but the main ideas can still be recovered. These findings offer a new way to understand what language models store internally and highlight potential privacy risks, as hidden representations may reveal parts of the original input.
Haiyan Zhao, Zirui He, Yiming Tang +4
New Jersey Institute of Technology · National University of Singapore · Wake Forest University +2
Large language models (LLMs) trained with canonical tokenization exhibit surprising robustness to non-canonical inputs such as character-level tokenization, yet the mechanisms underlying this robustness remain unclear. We study this phenomenon through mechanistic interpretability and identify a core process we term word recovery. We first introduce a decoding-based method to detect word recovery, showing that hidden states reconstruct canonical word-level token identities from character-level inputs. We then provide causal evidence by removing the corresponding subspace from hidden states, which consistently degrades downstream task performance. Finally, we conduct a fine-grained attention analysis and show that in-group attention among characters belonging to the same canonical token is critical for word recovery: masking such attention in early layers substantially reduces both recovery scores and task performance. Together, our findings provide a mechanistic explanation for tokenization robustness and identify word recovery as a key mechanism enabling LLMs to process character-level inputs.
Zhipeng Yang, Shu Yang, Lijie Hu +1
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Provable Responsible AI and Data Analytics (PRADA) Lab · King Abdullah University of Science and Technology