We show that large language models (LLMs) have an astonishing ability to recover meaning from severely degraded English texts. Texts in which content words have been randomly substituted by nonsense strings, e.g., "At the ghybe of the swuint, we are haiveed to Wourge Phrear-gwurr, who sproles into an ghitch flount with his crurp", can be translated to conventional English that is, in many cases, close to the original text, e.g., "At the start of the story, we meet a man, Chow, who moves into an apartment building with his wife." These results show that structural cues (e.g., morphosyntax, closed-class words) constrain lexical meaning to a much larger degree than imagined. Although the abilities of LLMs to make sense of "Jabberwockified" English are clearly superhuman, they are highly relevant to understanding linguistic structure and suggest that efficient language processing either in biological or artificial systems likely benefits from very tight integration between syntax, lexical semantics, and general world knowledge.
Figures & tables
Original text
Jabberwockified text
Translation of the Jabberwockified Text
In the United States, federal law has priority over any particular state’s law. If a state law conflicts with federal law, the state law is invalidated and the federal law is followed. For example, one federal law specifies what information should be included on the labels of medical devices. When Massachusetts passed a law that required additional information on hearing aid labels, federal courts barred Massachusetts from enforcing this law.
In the Staught Splunk, sprarb phlaint has phlol over any bredge strith’s phlaint. If a strith phlaint lyss with sprarb phlaint, the strith phlaint is whoaped and the sprarb phlaint is glorked. For glauge, stroothe sprarb phlaint jymes what pruib should be phalped on the thwirrs of scrorch psugs. When Splisk gninched a phlaint that strirled troor pruib on whess slinn thwirrs, sprarb nynchs knelveed Splisk from gwiveing this phlaint.
In the United States, federal law has supremacy over any conflicting state law. If a state law conflicts with federal law, the state law is invalidated [or preempted], and the federal law prevails. For example, federal law determines what warnings must be placed on the labels [or packaging] of cigarettes [or specific regulated products]. When [a State] enacted a law that required different [or conflicting] warnings on its own labels, federal courts [or authorities] prevented [that State] from enforcing this law.
Table 1: An example of LLM translation of a Jabberwockified text
A very severely degraded text
Translation
In the BLANK BLANK, BLANK BLANK has BLANK over any BLANK BLANK’s BLANK. If a BLANK BLANK BLANKs with BLANK BLANK, the BLANK BLANK is BLANKed and the BLANK BLANK is BLANKed. For BLANK, BLANK BLANK BLANK BLANKs what BLANK should be BLANKed on the BLANKs of BLANK BLANKs. When BLANK BLANKed a BLANK that BLANKed BLANK BLANK on BLANK BLANK BLANKs, BLANK BLANKs BLANKed BLANK from BLANKing this BLANK.
In the United States, federal law has precedence over any state government’s authority. If a state law conflicts with federal law, the federal law is upheld and the state law is overturned. For example, federal law strictly regulates what warnings should be placed on the labels of cigarette packages. When Massachusetts passed a law that required graphic warnings on cigarette package labels, federal courts blocked Massachusetts from enforcing this legislation.
Table 2: Successful translation of an even more degraded text
Condition Name
Replacement type
Words retained
Standard Jabberwocky
Standard
Standard NLTK stop-words
No modal verbs
Standard
Auxiliary verbs removed
No verbs or prepositions
Standard
Auxiliary verbs & prepositions removed
Lowercase-no-numbers
Text lowercased and numerals removed
Standard NLTK stop-words
No punctuation
End of sentence markers changed to periods; other punctuation removed
Standard NLTK stop-words
BLANKs
All non-stop-word lemmas replaced with the string “BLANK”
Standard NLTK stop-words
Table 3: Summary of manipulations
Figure 1: Comparison of translation accuracy by genre. Gray dots show similarity between the passage and a random non-target passage from the same genre
Figure 2: Translation accuracy for content-matched passages that were guaranteed not to be in the pretraining ( offline ) or were in the pretraining ( online ).
Figure 3: A comparison of meaning recovery of the 150-passage dataset when degraded through the standard Jabberwocky procedure and five variants (see Table 3 for details). Error bars show 95% within-passage CI.
Figure 4: Meaning recovery for a subset of passages that were tested incrementally, one sentence at a time, along with their original provenance ( S =Screenplay; F =Fiction excerpt).
Original text
Translation with just this context ( sim=.37 )
Translation with fuller context (225 content words) ( sim=.64 )
The sunniest place upon the hillside was the little pasture in which the old mare was grazing, moving slowly about and nipping at the short grass as if that which lay directly under her nose could not be nearly as choice as that which she could obtain by constant perambulation.
The nearest place around was the small room in which the woman was pacing, tossing things about and shouting at the main door, as if what lay under her feet were not as good as what she had left earlier by the big door.
The nearest spot by the path was the kind of place where the young person was playing, running quickly about and reaching at the small edge, as if what stirred under her feet might not be as good as what she could get by some other means.
Table 4: An example of how subsequent context improves translation of a sentence. Only a Jabberwockified version of the text was shown to the LLM.
Language model robustness studies how reliably language models preserve correct, safe behavior under input changes, distribution shifts, and adversarial pressure.
Large language models (LLMs) have excelled in processing multiple languages through pre- and post-training on multilingual data, even though English dominates the training data. Prior work focusing on token representations has revealed how those LLMs process non-English text. Although these analyses have provided insightful findings, they fail to capture a structural view, which is an inherent property of language. In this study, we explore the multilinguality of LLMs through representational structural analysis. Our findings reveal that low-resource languages are structurally more different from English than high- and mid-resource languages, and that language-specific post-training alters their structures while preserving inter-language relationships.
Large language models (LLMs) excel at semantic understanding, yet their ability to reconstruct internal structure from scrambled inputs remains underexplored. Sentence-level restoration is difficult to evaluate automatically because scrambled sentences often admit multiple valid reorderings. We introduce OrderProbe, a deterministic benchmark for structural reconstruction using fixed four-character expressions in Chinese, Japanese, and Korean, which have a unique canonical order and thus support exact-match scoring. We further propose a diagnostic framework that evaluates models beyond recovery accuracy, including Semantic Accuracy, Logical Validity, Structural Consistency, Robustness, and Information Density. Experiments on twelve widely used LLMs show that structural reconstruction remains difficult even for frontier systems: zero-shot recovery frequently falls below 35%. We also observe a consistent gap between meaning-oriented generation and exact structural reconstruction, suggesting that structural robustness is not an automatic byproduct of semantic competence.
Zhaolu Kang, Yingjie He, Kehan Jiang +18
Hunyuan, Tencent · Peking University · Tsinghua University +8
We aim to examine the extent to which Large Language Models (LLMs) can 'talk much' about grammar modules, providing evidence from syntax core properties translated by ChatGPT into Arabic. We collected 44 terms from generative syntax previous works, including books and journal articles, as well as from our experience in the field. These terms were translated by humans, and then by ChatGPT-5. We then analyzed and compared both translations. We used an analytical and comparative approach in our analysis. Findings unveil that LLMs still cannot 'talk much' about the core syntax properties embedded in the terms under study involving several syntactic and semantic challenges: only 25% of ChatGPT translations were accurate, while 38.6% were inaccurate, and 36.4.% were partially correct, which we consider appropriate. Based on these findings, a set of actionable strategies were proposed, the most notable of which is a close collaboration between AI specialists and linguists to better LLMs' working mechanism for accurate or at least appropriate translation.
Mohammed Q. Shormani, Yehia A. AlSohbani
Ibb University, Yemen · Arab Open University, Saudia Arabia