We show that large language models (LLMs) have an astonishing ability to recover meaning from severely degraded English texts. Texts in which content words have been randomly substituted by nonsense strings, e.g., "At the ghybe of the swuint, we are haiveed to Wourge Phrear-gwurr, who sproles into an ghitch flount with his crurp", can be translated to conventional English that is, in many cases, close to the original text, e.g., "At the start of the story, we meet a man, Chow, who moves into an apartment building with his wife." These results show that structural cues (e.g., morphosyntax, closed-class words) constrain lexical meaning to a much larger degree than imagined. Although the abilities of LLMs to make sense of "Jabberwockified" English are clearly superhuman, they are highly relevant to understanding linguistic structure and suggest that efficient language processing either in biological or artificial systems likely benefits from very tight integration between syntax, lexical semantics, and general world knowledge.
Figures & tables
Original text
Jabberwockified text
Translation of the Jabberwockified Text
In the United States, federal law has priority over any particular state’s law. If a state law conflicts with federal law, the state law is invalidated and the federal law is followed. For example, one federal law specifies what information should be included on the labels of medical devices. When Massachusetts passed a law that required additional information on hearing aid labels, federal courts barred Massachusetts from enforcing this law.
In the Staught Splunk, sprarb phlaint has phlol over any bredge strith’s phlaint. If a strith phlaint lyss with sprarb phlaint, the strith phlaint is whoaped and the sprarb phlaint is glorked. For glauge, stroothe sprarb phlaint jymes what pruib should be phalped on the thwirrs of scrorch psugs. When Splisk gninched a phlaint that strirled troor pruib on whess slinn thwirrs, sprarb nynchs knelveed Splisk from gwiveing this phlaint.
In the United States, federal law has supremacy over any conflicting state law. If a state law conflicts with federal law, the state law is invalidated [or preempted], and the federal law prevails. For example, federal law determines what warnings must be placed on the labels [or packaging] of cigarettes [or specific regulated products]. When [a State] enacted a law that required different [or conflicting] warnings on its own labels, federal courts [or authorities] prevented [that State] from enforcing this law.
Table 1: An example of LLM translation of a Jabberwockified text
A very severely degraded text
Translation
In the BLANK BLANK, BLANK BLANK has BLANK over any BLANK BLANK’s BLANK. If a BLANK BLANK BLANKs with BLANK BLANK, the BLANK BLANK is BLANKed and the BLANK BLANK is BLANKed. For BLANK, BLANK BLANK BLANK BLANKs what BLANK should be BLANKed on the BLANKs of BLANK BLANKs. When BLANK BLANKed a BLANK that BLANKed BLANK BLANK on BLANK BLANK BLANKs, BLANK BLANKs BLANKed BLANK from BLANKing this BLANK.
In the United States, federal law has precedence over any state government’s authority. If a state law conflicts with federal law, the federal law is upheld and the state law is overturned. For example, federal law strictly regulates what warnings should be placed on the labels of cigarette packages. When Massachusetts passed a law that required graphic warnings on cigarette package labels, federal courts blocked Massachusetts from enforcing this legislation.
Table 2: Successful translation of an even more degraded text
Condition Name
Replacement type
Words retained
Standard Jabberwocky
Standard
Standard NLTK stop-words
No modal verbs
Standard
Auxiliary verbs removed
No verbs or prepositions
Standard
Auxiliary verbs & prepositions removed
Lowercase-no-numbers
Text lowercased and numerals removed
Standard NLTK stop-words
No punctuation
End of sentence markers changed to periods; other punctuation removed
Standard NLTK stop-words
BLANKs
All non-stop-word lemmas replaced with the string “BLANK”
Standard NLTK stop-words
Table 3: Summary of manipulations
Figure 1: Comparison of translation accuracy by genre. Gray dots show similarity between the passage and a random non-target passage from the same genre
Figure 2: Translation accuracy for content-matched passages that were guaranteed not to be in the pretraining ( offline ) or were in the pretraining ( online ).
Figure 3: A comparison of meaning recovery of the 150-passage dataset when degraded through the standard Jabberwocky procedure and five variants (see Table 3 for details). Error bars show 95% within-passage CI.
Figure 4: Meaning recovery for a subset of passages that were tested incrementally, one sentence at a time, along with their original provenance ( S =Screenplay; F =Fiction excerpt).
Original text
Translation with just this context ( sim=.37 )
Translation with fuller context (225 content words) ( sim=.64 )
The sunniest place upon the hillside was the little pasture in which the old mare was grazing, moving slowly about and nipping at the short grass as if that which lay directly under her nose could not be nearly as choice as that which she could obtain by constant perambulation.
The nearest place around was the small room in which the woman was pacing, tossing things about and shouting at the main door, as if what lay under her feet were not as good as what she had left earlier by the big door.
The nearest spot by the path was the kind of place where the young person was playing, running quickly about and reaching at the small edge, as if what stirred under her feet might not be as good as what she could get by some other means.
Table 4: An example of how subsequent context improves translation of a sentence. Only a Jabberwockified version of the text was shown to the LLM.
Language model robustness studies how reliably language models preserve correct, safe behavior under input changes, distribution shifts, and adversarial pressure.