On the Semantic and Syntactic Information Encoded in Proto-Tokens for One-Step Text Reconstruction

Ivan Bondarenko; Egor Palkin; Fedor Tikunov

cs.LGFeb 20, 2026

On the Semantic and Syntactic Information Encoded in Proto-Tokens for One-Step Text Reconstruction

Authors: Ivan Bondarenko, Egor Palkin, Fedor Tikunov

Abstract

Autoregressive large language models (LLMs) generate text token-by-token, requiring n forward passes to produce a sequence of length n. Recent work, Exploring the Latent Capacity of LLMs for One-Step Text Reconstruction (Mezentsev and Oseledets), shows that frozen LLMs can reconstruct hundreds of tokens from only two learned proto-tokens in a single forward pass, suggesting a path beyond the autoregressive paradigm. In this paper, we study what information these proto-tokens encode and how they behave under reconstruction and controlled constraints. We perform a series of experiments aimed at disentangling semantic and syntactic content in the two proto-tokens, analyzing stability properties of the e-token, and visualizing attention patterns to the e-token during reconstruction. Finally, we test two regularization schemes for "imposing" semantic structure on the e-token using teacher embeddings, including an anchor-based loss and a relational distillation objective. Our results indicate that the m-token tends to capture semantic information more strongly than the e-token under standard optimization; anchor-based constraints trade off sharply with reconstruction accuracy; and relational distillation can transfer batch-level semantic relations into the proto-token space without sacrificing reconstruction quality, supporting the feasibility of future non-autoregressive seq2seq systems that predict proto-tokens as an intermediate representation.

Figures & tables

Panel of Figure 1 — Figure 1: Attention to the $e$ -token averaged over heads (examples).

Panel of Figure 2 — Figure 2: Layer-wise heatmaps of attention to the $e$ -token (averaged over heads).

Panel of Figure 3 — Figure 3: Head-level attention to the $e$ -token for selected layers/heads (examples).

Figure 4: Reconstruction accuracy as a function of noise intensity $\alpha$ for different noise types.

Figure 5: t-SNE visualization of optimized $e$ -token embeddings for original, lexical, and semantic augmentations.

Figure 6: t-SNE visualization of optimized $m$ -token embeddings for original, lexical, and semantic augmentations.

Figure 7: Reconstruction token accuracy vs. target length under anchor regularization with a small weight.

Figure 8: Cosine similarity between $e$ and the teacher embedding over optimization steps (anchor).

Figure 9: Cosine similarity between $e$ and the teacher embedding over optimization steps for a larger anchor weight.

Figure 10: Token accuracy vs. target length for a larger anchor weight, showing degraded reconstruction.

Panel of Figure 11 — Figure 11: Distributions of semantic-matrix correlation per step for relational distillation, with and without a shared $m$ -token.

Panel of Figure 12 — Figure 12: Token-accuracy distributions with and without the relational distillation constraint.

Panel of Figure 13 — Figure 13: Improved semantic-matrix correlation distributions after applying the relational distillation objective.

Non-Autoregressive Text Generation

Non-autoregressive text generation produces multiple tokens in parallel to reduce decoding time while maintaining coherent, useful outputs.

See all 31 papers

PTP: Previous-Token Prediction based LLM Inversion for Near-Exact Prompt Reconstruction
Jul 31, 2026Pirzada Suhail, Nagasai Saketh Naidu, Atanu R Sinha +1LLM Prompting Next-Token Prediction
IIT Bombay · Adobe Research
Rep2Text: Decoding Full Text from a Single LLM Token Representation
Nov 9, 2025Haiyan Zhao, Zirui He, Yiming Tang +4LLM Interpretability LLM Information Extraction
New Jersey Institute of Technology · National University of Singapore · Wake Forest University +2
Word Recovery in Large Language Models Enables Character-Level Tokenization Robustness
Mar 11, 2026Zhipeng Yang, Shu Yang, Lijie Hu +1Language Model Robustness Tokenization in Language Models
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Provable Responsible AI and Data Analytics (PRADA) Lab · King Abdullah University of Science and Technology

Jul 31, 2026J/K move · Enter open · S save

PTP: Previous-Token Prediction based LLM Inversion for Near-Exact Prompt Reconstruction

Pirzada Suhail, Nagasai Saketh Naidu, Atanu R Sinha, Amit Sethi

IIT Bombay · Adobe Research

LLM Prompting Next-Token Prediction Sequence Modeling

Open paper AskPDF arXiv

Large language models (LLMs) generate text by auto-regressively sampling the next token. This inherently leads to a many-to-many mapping between prompts and responses, complicating the task of inferring prompts from observed outputs. Prior work on LLM inversion frames prompt recovery as a semantic reconstruction task. They rely on fine-tuning pretrained sequence-to-sequence models on large external datasets--and requiring access to model weights or logits--to generate semantically plausible prompts. In contrast, we present a functional approach to inverting a given LLM in a black-box setting, without auxiliary aids. We train an explicit inverse language model entirely from scratch on data synthetically generated from the target LLM itself. Analogous to forward next-token prediction, our inverse model is trained using previous-token prediction, establishing a generative link between the forward and inverse processes that enables faithful prompt reconstruction. Moreover, it naturally supports diverse prompt reconstructions through sampling, whereby all such prompts induce similar responses under the forward, target LLM. Our approach generalises across datasets and exhibits transferability in reconstructing prompts from responses generated by different LLMs. Further, across the set of token based evaluation metrics for prompt and response reconstructions, our approach outperforms prior work.

Abstract

Non-Autoregressive Text Generation

Explore similar work

PTP: Previous-Token Prediction based LLM Inversion for Near-Exact Prompt Reconstruction

Rep2Text: Decoding Full Text from a Single LLM Token Representation

Word Recovery in Large Language Models Enables Character-Level Tokenization Robustness