cs.CLSep 28, 2026
SaveLLMs are not stochastic parrots: Evidence for meaning-mediated abstraction from conlang-like tasks
Organizations: Computational Story Lab · Computational Ethics Lab · Vermont Complex Systems Institute University of Vermont Burlington, VT 05405, USA
Abstract
The strong version of the stochastic parrot argument claims that, although large language models (LLMs) may exceed rote regurgitation, they cannot move beyond statistical pattern matching into abstraction or reasoning, remaining ontologically near the lower bound of pattern reuse despite producing alluringly fluent text. We test this hypothesis using conlang-like tasks. Several LLMs are given only natural-language descriptions of fictional languages that subvert prominent superficial patterns in training data by combining statistically uncommon and unattested features. Crucially, no example outputs are given. We argue that if the models exhibit rule-following behaviour, they cannot be relying solely on superficial statistical patterns; such patterns often work against the correct output. Instead, successful performance requires representations of the constraints specified in the prompt. Across three complementary task families, models systematically move in the meaning-predicted direction: they distinguish prompt exposure from instructed use, alter semantic relationships in response to novel constraints, and sometimes produce exact matches to complex translation answer keys. Although performance varies across the spectrum of models used, these results provide evidence for meaning-mediated abstraction in LLMs and refute the strong stochastic parrot hypothesis. Our work shows that, under appropriate architectural and contextual constraints, statistical learning can produce meaning-mediated abstractions, although generation remains strongly constrained by superficial plausibility. We discuss implications for model development and for understanding how increasingly abstract representations may emerge from plausible-text-generation objectives.
Figures & tables
Figure 1: This figure shows prompts describing the translation family of rules (translate-1 and dan-1), their answer keys, and sample outputs from each model, to illustrate what we mean by separating meaning-mediated plausibility from statistical plausibility (although we do that in different ways across the rule families). It is apparent from these responses that these tasks are hard, that some models do much better than others, and that dan-1 is more difficult than translate-1. However, it is also apparent that models can produce outputs exactly matching or near to the answer keys, behaviour which is better explained by meaning plausibility than statistical plausibility.
Figure 2: Comparison of rare bigram usage across conditions, for the 3 models (Qwen (with thinking), GPT-4o, OLMo) with cleaned output.
| Comparison | Fisher statistic ( ) | ||
| comparison-2 vs sidequest-1 | 226.794 | 3.700e-46 | 3 |
| sidequest-1 vs comparison-1 | 187.022 | 1.093e-37 | 3 |
| comparison-2 vs comparison-1 | 243.473 | 1.017e-49 | 3 |
Table 2: Fisher-combined -values across models (Qwen (T), Olmo, GPT-4o (no SP)) for each contrast; cleaned outputs.
Figure 3: Illustrative summary of double verb family results from our poster presentation ( Institut des sciences cognitives, Université du Québec à Montréal, 2026 ) , Fig 30 .
Figure 4: Doublev-1 Rule Similarity Scores and Verb Counts by Model. Histogram of similarity scores (cosine similarity between marked and closest verb, where marked verb is appropriately terminated with bigram -qw ) per model, as well as the per-run count of correctly marked -qw verbs and count of correct pairs of -qw verbs and closest verb. Total number of runs with verbs reported for each model ( runs indicates each run had at least one instance of a correctly assigned -qw verb). Cosine similarity scores closer to 0 indicate dissimilar words. Counts by model that are closer to total clauses reported by model indicate higher performance by model.
Figure 5: Doublev-2 Rule Similarity Scores and Verb Counts by Model. Histogram of similarity scores (cosine similarity between marked and closest verb, where marked verb is appropriately terminated with bigram -qw ) per model for all verbs paired in all runs, as well as the per-run count of correctly marked -qw verbs and count of correct pairs of -qw verbs and closest verb. Total number of runs with verbs reported for each model ( runs indicates each run had at least one instance of a correctly assigned -qw verb). Cosine similarity scores closer to 0 indicate dissimilar words. Counts by model that are closer to total clauses reported by model indicate higher performance by model.
Figure 6: Translate-1 Distributions by Model and Metric. Histogram of string distance scores (Levenshtein distance, normalized Levenshtein distance, Jaccard similarity, and character n-gram cosine similarity) comparing the output for Translate-1 rule versus the correct translated response. Scores closer to 0 for Levenshtein distances, and scores closer to 1 for Jaccard and cosine similarities indicate better rule-following/more accurate translation. Qwen (thinking) performs best for Translate-1 rule-following based on median and mean scores for all string similarity measures. For both Wilcoxon Rank-Sum and Kolmogorov-Smirnov tests, all models perform better than the random strings variants achieving statistically significantly better median scores (reported in Tables 31 and 32 ).
Figure 7: Dan-1 Distributions by Model and Metric. Histogram of string distance scores (Levenshtein distance, normalized Levenshtein distance, character n-gram cosine similarity) comparing the output for Dan-1 rule versus the correct translated response. Scores closer to 0 for Levenshtein distances, and scores closer to 1 for cosine similarities indicate better rule-following/more accurate translation. Qwen (thinking) performs best for Dan-1 rule-following based on median and mean scores for all string similarity measures. Median OLMo and Llama3-70B scores against random string (select characters) median cosine similarity scores are not significantly different for the Wilcoxon Rank-Sum Tests (reported in Table 33 ). Likewise, for the Kolmogorov-Smirnov test, which examines the distributional shapes of the cosine similarity scores, neither OLMo, nor Llama3-70B show statistically different scores than the random string (select characters). The distribution of OLMo cosine similarity scores additionally shows no statistical difference with the random string (all characters) using the Kolmogorov-Smirnov test (reported in Table 34 ).
Figure 8: We joined rule-level performance with linguistic rule features and architectural features for exploratory correlation analysis (Spearman, n=9 rules / n=5 models). To examine the relationship between (this representation of) rule complexity and performance, we collapsed dan-1 and translate-1 to single rows (averaging perfect match and vicinity scores), ranked rules within each model, and computed per-model Spearman r against rule complexity rank.
Figure 9: Heavily normalized performance subset. Comparison-1 is a baseline, asking the model to generate a short story in English. All models perform well. For non-thinking models, cleaned and uncleaned outputs are nearly identical. For Qwen (thinking), the uncleaned rare bigram rate is suppressed by the CoT trace, which is in standard English. Levenshtein scores are normalized twice here. All performance metrics are reoriented so that higher values indicate better performance, then normalized to in two versions for comparison: top-only , where each row is divided by its maximum observed value (best model , others scaled relative to it); and both-ends , where each row is min-max normalized (best , worst ). Results are visualized as a heatmap with models as columns and metrics as rows, with missing values (models lacking cleaned output data) grayed out. The former gives a clearer picture of overall model performance (for each rule, which models were comparatively able to do it), while the latter gives a clearer picture of relative model performance (for each rule, which model did best and worst). The comparatively un-normalized version of this performance heatmap is included in the Appendix in Figure 28 .
| Qwen3(Thinking) (val) | Qwen3(Thinking) (rank) | GPT-4 (val) | GPT-4 (rank) | Qwen3 (No Thinking) (val) | Qwen3 (No Thinking) (rank) | OLMo-2 (val) | OLMo-2 (rank) | Llama-3 (val) | Llama-3 (rank) | |
| rule | ||||||||||
| comparison-1 | 0.9998 | 7 | 0.9998 | 7 | 0.9998 | 7 | 0.9997 | 7 | 0.9998 | 7 |
| comparison-2 | 0.5904 | 3 | 0.5127 | 3 | 0.3800 | 3 | 0.6900 | 3 | 0.0910 | 2 |
| sidequest-1 | 0.9467 | 6 | 0.9828 | 6 | 0.9341 | 6 | 0.9919 | 6 | 0.9580 | 6 |
| doublev-1 | 0.9063 | 5 | 0.9073 | 5 | 0.9016 | 5 | 0.9034 | 5 | 0.8705 | 5 |
| doublev-2 | 0.8331 | 4 | 0.7680 | 4 | 0.7800 | 4 | 0.8220 | 4 | 0.6590 | 4 |
Table 6: Raw performance values and within-model ranks (1 = worst performance/ hardest rule, 7 = best performance/ easiest rule) for each rule after collapsing dan-1 and translate-1 to single rows (average of perfect match and vicinity scores).
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Searching for Google results similar to wordorderalternation-3 .
Figure 11: Tokenizer fragmentation of example answers by rule (log 2 ratio relative to comparison-1). Rules whose correct outputs most strongly deviate from superficial English statistical patterns (dan-1, wordorderalternation-1, comparison-2) show substantially higher fragmentation ratios, indicating that the required outputs are not well-supported by the orthographic building blocks of any tokenizer’s training distribution. Rules with more English-like correct outputs (sidequest-1, comparison-1) cluster near or below the baseline. All values are log 2 -scaled relative to the comparison-1 (standard English story) answer. These are “point-in-time” counts for a single example answer per rule; Part 2 will extend this analysis to all model outputs. Speculatively, we wonder if we will observe more prosaic output in sidequest-1 than comparison-1, and maybe in doublev-1 versus doublev-2, perhaps as a sign of cognitive tradeoff (in other words, the writing may suffer as the models allocate more resources to rule compliance).
Figure 12: Spearman rank correlations of rule fragmentation across 13 tokenizers. Correlations are uniformly high (all ; see Table 11 ), indicating that the ranking of rules by tokenizer fragmentation is robust across diverse model families and vocabulary designs. The consistent ordering of rules by fragmentation is therefore not an artifact of any single tokenizer’s vocabulary, but better explained as a reflection of underlying statistical patterns in the corpora.
Figure 13: Raw characters per token by rule, excluding dan-1. Rules requiring more non-English-like output (wordorderalternation-1, wordorderalternation-2, comparison-2) have fewer characters per token, consistent with their correct answers containing more character sequences not found in standard tokenizer vocabularies (thus necessitating more, shorter tokens). Rules whose outputs remain closer to English (sidequest-1, comparison-1) tokenize into longer, more familiar stems. dan-1 is excluded here because its purely numeric and punctuation-based character set make it a bit unclear to interpret alongside the other rules, which share some characteristics (with each other and English) such as spaces for word boundaries.
Figure 14: Raw tokens per character by rule (excluding dan-1). The inverse of Figure 13 ; higher values indicate greater tokenizer fragmentation. We included both figures in case some readers find tokens-per-character more intuitive than characters-per-token, but the information is identical.
Figure 15: Mean recovered corpus prefix length by rule. Rules whose correct outputs utilize larger structures borrowed from standard English (wordorderalternation-3, comparison-1) have longer recovered prefixes, indicating that their local character sequences are more frequently attested in training corpora. Rules with highly non-English-like outputs (dan-1, wordorderalternation-1) have shorter recovered prefixes, consistent with the tokenization fragmentation results (Figure 13 ). Prefix length is computed on a single gold-standard example answer per rule (see Section A.4 for interpretation caveats), and therefore only a “point-in-time” proxy.
Figure 16: Scatter plot of mean characters-per-token (CPT) versus mean recovered corpus prefix length, by rule. A moderate positive correlation is observed (Pearson , ; Spearman , ; see Table 12 ), consistent with the interpretation that both measures reflect the degree to which a rule’s required output aligns with the statistical structure of the training corpus. Rules that deviate more strongly from English (low CPT/ high TPC) also tend to fall outside corpus -gram patterns (short recovered prefixes).
Figure 17: Rank-rank plot of mean characters-per-token versus mean recovered corpus prefix length, by rule. The positive correlation observed in Figure 16 is preserved at the rank level (Spearman , ).
| correlation | coefficient | p_value | n_rules |
| Pearson | 0.626000 | 0.012600 | 15 |
| Spearman | 0.514800 | 0.049600 | 15 |
Table 12: Correlation between mean characters-per-token (CPT) and mean recovered corpus prefix length. N rules encompasses the suite of rules, but including both hyphenated and non-hyphenated translate-1 answers, and excluding translate-2 because its answer key is identical to that of translate-1). Both correlations are statistically significant, supporting the interpretation that CPT and prefix length at least partially reflect the same signal.
| Search expression | Retrieved excerpt |
| 0 is the word boundary OR 0 is the space symbol | “…use a vertical flip (embPattern_flip)…0x2 0 is the space symbol 0, so when padding either 0 or space is preferred …” |
| 0 is the word boundary OR 0 is the space symbol | “…i.e. use only the C Standard Library…0x2 0 is the space symbol 0, so when padding either 0 or space is preferred …” |
| 0 is the word boundary OR 0 is the space symbol | “…the working answer for me, tested in chrome, IE and firefox… window.location.href = ’mailto:address@dmail.com?subject=Hello there&body=This is the body’; …%2 0 is the space symbol 0 that should be used, but it worked for me as well with normal space.” |
Table 13: Example documents returned by searches for superficially similar phrases. The retrieved examples concern an embroidery-pattern convention in which zero or a blank space is used for padding, and a forum question about mailto links, rather than a constructed-language rule. The matched search string is a paraphrase of that part of the rule, not a verbatim match to any part of the prompt. Document sources: http://embroidermodder.org/docs/manual/ , https://stackoverflow.com/questions/3868315/invoke-click-a-mailto-link-with-jquery-javascript , retrieved from the InfiniGram UI. There are at most 4 matches per corpus to this n-gram search string, across all the corpora, and every example surfaced by the InfiniGram UI matched one of these two use cases.
Figure 18: To what extent is any part of the correct continuation among the top predicted tokens? For each context, we list all top-probability tokens for each corpus in descending probability order, and compare against the correct continuation. Correct answers were split into meaningful chunks (allowing for minor ordering variation) and both hyphenated and unhyphenated translate-1 forms were accepted. We split continuations to give greater context or to allow greater freedom in acceptable responses (making positive matches easier). We counted the hyphenated or unhyphenated correct answers for translate-1 as equally valid/ correct (we were flexible on punctuation and casing).
| Family/ Gloss | Rule | Base Prompt | |
|---|---|---|---|
| doublev-1 | doublev | 1/2 | Your goal is to generate text in a new language with strict grammatical rules. In this new version of English, speakers include an extra verb in each clause. In other words, everywhere there would normally be a single verb, there are now two consecutive verbs. One of the verbs is irrelevant to the meaning of the sentence. The irrelevant verb could either be first or second, and it is marked with the terminal morpheme -qw. The irrelevant verb should result in no other changes to the sentence. Otherwise, this language is identical to English. Please write a short story (20 sentences) in this language as if you speak this language: |
| doublev-2 | doublev (emphasized irrelevance) | 2/2 | Your goal is to generate text in a new language with strict grammatical rules. In this new version of English, speakers include an extra verb in each clause. In other words, everywhere there would normally be a single verb, there are now two consecutive verbs. One of the verbs is irrelevant to the meaning of the sentence. It doesn’t just fail to add to the meaning of the sentence, but it ought to feel “random” in the context of the sentence. In other words, the two verbs should not be closely related, synonymous, or antonymous. The irrelevant verb could either be first or second, and it is always marked with the terminal morpheme -qw. The irrelevant verb should result in no other changes to the sentence. Otherwise, this language is identical to English. Please write a short story (20 sentences) in this language as if you speak this language: |
| wordorderalternation-1 | wordorderalternation (restricted lexicon) | 1/3 | Your goal is to generate text in a new language with strict grammatical rules. In this new language, speakers follow a word order parity rule. They strictly alternate between OSV (Object–Subject–Verb) and VSO (Verb–Subject–Object) clauses. When a clause is left incomplete, e.g. due to interruption, the new clause flips to the other word order. Verbs are inflected for tense with suffixes: -qw indicates past tense, -jx indicates present tense, and -kz indicates future tense. All verbs must be inflected with one of these suffixes. The tense suffixes -qw, -jx, and -kz are always the final morphemes in the verb. Those morphemes only have one meaning (indicating verb tense); they are not polysemous and have no homonyms. Adverbs always appear in phrases before the verb when word order is OSV, and after the verb when word order is VSO. In other words, for OSV clauses, the verb is strictly final, and for VSO clauses, the verb is strictly first. Besides the tense suffixes, the rest of the lexicon is created from this set of graphemes, optionally joined with each other: VF, PX, ZD, QT, ZP, QO, DX, HJ, GV, JW, QC, JY, GJ, QB, PQ, JG, BZ, MX, QM, MZ, QF, WJ, ZQ, XR, ZK, CX, FX, FV, BX, VW, VJ, MQ, QV, ZF, QE, YJ, GX, KX, XG, QD, XJ, SX, VZ, VX, WV, YQ, BQ, GQ, VK, ZJ, XK, QP, HX, FZ, QH, QJ, JZ, VQ, KQ, XD, QX, WX, FQ, XZ, ZX, A, E, I, O, U. Please write a short story (20 sentences) in this language as if you speak this language: |
| wordorderalternation-2 | wordorderalternation (any lexicon, with verb marking) | 2/3 | Your goal is to generate text in a new language with strict grammatical rules. In this new language, speakers follow a word order parity rule. They strictly alternate between OSV (Object–Subject–Verb) and VSO (Verb–Subject–Object) clauses. When a clause is left incomplete, e.g. due to interruption, the new clause flips to the other word order. Verbs are inflected for tense with suffixes: -qw indicates past tense, -jx indicates present tense, and -kz indicates future tense. All verbs must be inflected with one of these suffixes. The tense suffixes -qw, -jx, and -kz are always the final morphemes in the verb. Those morphemes only have one meaning (indicating verb tense); they are not polysemous and have no homonyms. Adverbs always appear in phrases before the verb when word order is OSV, and after the verb when word order is VSO. In other words, for OSV clauses, the verb is strictly final, and for VSO clauses, the verb is strictly first. Please write a short story (20 sentences) in this language as if you speak this language: |
| wordorderalternation-3 | wordorderalternation (English lexicon) | 3/3 | Your goal is to generate text in a new language with strict grammatical rules. In this new language, which is derived from English, speakers follow a word order parity rule. They strictly alternate between OSV (Object–Subject–Verb) and VSO (Verb–Subject–Object) clauses. When a clause is left incomplete, e.g. due to interruption, the new clause flips to the other word order. Adverbs always appear in phrases before the verb when word order is OSV, and after the verb when word order is VSO. In other words, for OSV clauses, the verb is strictly final, and for VSO clauses, the verb is strictly first. Besides these differences in syntax, the language is identical to present day English. Please write a short story (20 sentences) in this language as if you speak this language: |
| testingenglish-1 | testingenglish (replace content words) | 1/4 | Your goal is to generate text in a new language with strict grammatical rules. In this new language, speakers follow similar rules to English. The function words (prepositions, articles, pronouns, auxiliary verbs, etc.) and inflectional morphemes (verb tense, plural markers, adverb and adjective affixes, etc.) are all from English. However, the content words — nouns, adjectives, adverbs, and main verbs — come from this lexicon: VF, PX, ZD, QT, ZP, QO, DX, HJ, GV, JW, QC, JY, GJ, QB, PQ, JG, BZ, MX, QM, MZ, QF, WJ, ZQ, XR, ZK, CX, FX, FV, BX, VW, VJ, MQ, QV, ZF, QE, YJ, GX, KX, XG, QD, XJ, SX, VZ, VX, WV, YQ, BQ, GQ, VK, ZJ, XK, QP, HX, FZ, QH, QJ, JZ, VQ, KQ, XD, QX, WX, FQ, XZ, ZX. English auxiliary verbs (e.g., be, have, do, will, can) remain in the language; all English main verbs are replaced. Besides the novel lexical items that replace English content words, the language is identical to English. Please write a short story (20 sentences) in this language as if you speak this language: |
Table 15: Columns taken from our prompt spreadsheet; prompt variations not included.
| model | rule | count | mean | std | min | 25% | 50% | 75% | max |
| gpt4o (no SP) | comparison-1 | 50 | 0.000192 | 0.000409 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.001784 |
| gpt4o (no SP) | comparison-2 | 50 | 0.512722 | 0.140408 | 0.207241 | 0.379942 | 0.565855 | 0.635927 | 0.721683 |
| gpt4o (no SP) | sidequest-1 | 50 | 0.017194 | 0.021529 | 0.000000 | 0.000000 | 0.001982 | 0.037612 | 0.065900 |
| olmo | comparison-1 | 50 | 0.000253 | 0.000371 | 0.000000 | 0.000000 | 0.000000 | 0.000478 | 0.001654 |
| olmo | comparison-2 | 50 | 0.689975 | 0.107207 | 0.000000 | 0.678283 | 0.706837 | 0.728323 | 0.781534 |
| olmo | sidequest-1 | 50 | 0.008060 | 0.009513 | 0.000000 | 0.001993 | 0.005576 | 0.010312 | 0.044701 |
Table 17: Summary statistics for mean proportion of rare bigram presence (cleaned) for rules comparison-1, comparison-2, sidequest-1. The mean of the proportion of rare bigrams to total bigrams for all runs ( ) of each rule, presented for cleaned outputs from three models (GPT4o, OLMo, and Qwen (thinking).
Figure 19: Comparison of rare bigram usage across conditions (for 5 models; Qwen (thinking), Qwen (no thinking), GPT-4o, OLMo, Llama), using all model output (not cleaned).
Figure 20: Histogram of Similarity Scores for Doublev-1, all Models (combined). Histogram of cosine similarity between correctly marked -qw verb and adjacent verb for all models, as well as the combined histogram of counts of correctly marked -qw verbs by run by model and the sum of counts of correctly marked -qw verbs and appropriately placed adjacent verbs.
Figure 21: Histogram of Similarity Scores for Doublev-2, all Models (combined). Histogram of cosine similarity between correctly marked -qw verb and adjacent verb for all models, as well as the combined histogram of counts of correctly marked -qw verbs by run by model and the sum of counts of correctly marked -qw verbs and appropriately placed adjacent verbs.
| Model | Med doublev-1 | doublev-1 | Med doublev-2 | doublev-2 | D stat | p (raw) | p (adj.) | Sig.? |
| OLMo | 0.9034 | 1285 | 0.1780 | 1186 | 0.9480 | 0.0000 | 0.0000 | Yes |
| Qwen (no thinking) | 0.9016 | 1079 | 0.2200 | 1178 | 0.8571 | 0.0000 | 0.0000 | Yes |
| Qwen | 0.9063 | 1209 | 0.1669 | 1045 | 0.9310 | 0.0000 | 0.0000 | Yes |
| GPT-4o (no system prompt) | 0.9073 | 1085 | 0.2320 | 594 | 0.9113 | 0.0000 | 0.0000 | Yes |
| Llama3-70b | 0.8705 | 68 | 0.3410 | 131 | 0.8284 | 0.0000 | 0.0000 | Yes |
| All models (pooled) | 0.9042 | 4726 | 0.1970 | 4134 | 0.9075 | 0.0000 | — | Yes |
Table 27: Within-Model Two-Sample Kolmogorov-Smirnov Tests: doublev-1 vs. doublev-2 on Cosine Similarity (where qw_is_verb = True , Benjamini-Hochberg correction, ; composite row uses no correction). If models appropriately follow these rules, cosine similarity will be lower (closer to ) for doublev-2 which emphasizes irrelevance between the two verbs used in each clause. All models display this behavior, indicated by significant differences in median Cosine Similarity scores.
Figure 22: Sentence-level translation quality across five language models (GPT-4o, Llama 3, OLMo, Qwen, and Qwen without thinking) evaluated using BLEU, chrF, TER, METEOR, and a composite score . Each histogram shows the distribution of metric scores over 50 full (non-truncated) LLM-generated responses to the translation prompt, whose source sentence is five words (41 characters) long. The orange triangle denotes the model mean, and the dotted vertical line denotes the upper-bound score obtained when the hypothesis exactly matches the reference (BLEU = 100.0, chrF = 100.0, TER = 0.0, METEOR = 99.6, composite = 99.9). Higher values indicate better performance for BLEU, chrF, METEOR, and the composite score, whereas lower values indicate better performance for TER. The composite score is computed as the unweighted mean of BLEU, chrF, METEOR, and .
Figure 23: Sentence-level translation quality across five language models (GPT-4o, Llama 3, OLMo, Qwen, and Qwen without thinking) evaluated using BLEU, chrF, TER, METEOR, and a composite score. Each histogram shows the distribution of metric scores over 50 LLM-generated responses to the translation prompt, with each response truncated to its first 41 characters to match the length of the source sentence ("Mary gjyqazkjx pxmqiMary. pxmqiMary qtojx"). The orange triangle denotes the model mean, and the dotted vertical line denotes the upper-bound score obtained when the hypothesis exactly matches the reference (BLEU = 100.0, chrF = 100.0, TER = 0.0, METEOR = 99.6, composite = 99.9). Higher values indicate better performance for BLEU, chrF, METEOR, and the composite score, whereas lower values indicate better performance for TER. The composite score is computed as the unweighted mean of BLEU, chrF, METEOR, and .
Figure 24: Histograms of Similarity Scores for Dan-1, all Models (combined).
Figure 25: Histograms of Similarity Scores for Translate-1, all Models (combined).
| Qwen-3 (T) | GPT-4 | Qwen-3 (No T) | OLMo-2 | Llama-3 | ||
| Composite scores | Straight mean | 0.960 | 0.697 | 0.677 | 0.723 | 0.459 |
| Family-weighted mean | 0.975 | 0.655 | 0.661 | 0.654 | 0.437 |
Table 35: Normalized performance scores per model and metric (all scores oriented: higher better, top-normalized per row). Metrics are grouped into four families, each contributing equally ( ) to the family-weighted composite. Bigram metrics use cleaned outputs where available (GPT-4, OLMo-2, Qwen3 Thinking), uncleaned otherwise. Levenshtein distances are log-normalized prior to top-normalization to reduce compression from the Llama-3 dan-1 outlier. Metrics are grouped into four rule families: the bigram family (comparison-1, comparison-2, sidequest-1), the dan-1 family , the translate-1 family , and the doublev family (doublev-1, doublev-2), each contributing equally ( ) to the family-weighted composite. Elsewhere, we discuss dan-1 and translate-1 as both being part of the translation rules family, but performance here is anchored to each independently, rather than relative (as it is with the story rules and doublev rules), which is why we count them separately. However, again, reasonable variations are totally possible.
Figure 26: Exploratory look at model performance and features of the conlang rules.
Figure 27: Exploratory look at model performance and architectural features.
Figure 28: Comparatively un-normalized performance subset. Comparison-1 is a baseline, asking the model to generate a short story in English. All models perform well. For non-thinking models, cleaned and uncleaned results are broadly similar. However, for Qwen (Thinking), the uncleaned bigram ratios are suppressed by the CoT trace.
Figure 29: An illustration about patterns and abstractions. How do patterns become abstraction? We know there are plenty of patterns in the training data. If we consider patterns at any level of abstraction, it becomes impossible to argue that these models never saw a comparable pattern to those underlying these rules. For example, a model may have seen many question-answer pairs, and task-solution pairs, that follow a general, very abstract pattern that might be called “instruction following”. However, we are not trying to say the models do not learn from patterns, and do not see instructive patterns during training. In fact, the exact opposite! Our contention is that the patterns that play out at the level of the symbols perceptible to the model are not a sufficient explanation for the rule-following behaviour we see. We have to invoke patterns at a more abstract level than co-occurrences between symbols in order to explain the rule-following observations we elicited (which are varying degrees of implausible at more concrete levels of pattern observation). Because language and abstractions are so ubiquitous for us, it is easy to overlook a subconscious step along the ladder of abstraction. But in this experiment, it is important to try to separate them out. For example, even if a model learned to follow certain instructions, that does not imply the ability to follow any instruction: understanding and executing a specific instruction is distinguishable from what looks like general instruction-following. The authors, for example, can follow instructions in general, but also wouldn’t be able to execute instructions in an alien language without some way of internally representing those instructions in accessible form.
Figure 30: The poster we presented at ISC Summer School 2026 (UQAM). The poster summarizes our results up to that point. This pre-print includes some additional work, including work undertaken after conversation with other ISC Summer School participants. In the poster, the flower illustrates the idea that form and function are not truly separable (in language, in cognition; for us, for machines). Architecture is an epistemic choice. All cognition is, in some way, embodied, because it must be physically instantiated. Locality is the core constraint on problem-solving because of our physical reality. More discussion in Zimmerman [2025] . The gravitational inset illustrates PTGA. For LLMs, the resolution at which they sense the world is determined in large part by their training objective, and the many other, though less obvious, ways in which tokens are central to their architecture. Because they only “sense” the world through training data, and they only “see” that data as 1D relationships between tokens, and they learn what is salient via their objective(s), that process is sort of like their “eyes”.