cs.CLOct 21, 2026
SaveWhy is AI-generated language so hard to understand?
Organizations: 1New York University · 2The University of Texas at Austin
Abstract
There is a widespread sense that AI-generated text has recently become harder to understand. In this study, we investigate a set of psycholinguistic features that are known to correlate with human processing difficulty (e.g., object relative clauses, long syntactic dependencies, low-frequency n-grams). Using a novel corpus of AI-generated text across diverse domains, we find that AI models use these difficult features more often than paraphrases generated by round-trip translation. Compared with human writing, all nine frontier and near-frontier models use more inanimate subjects with animate verbs and more low-frequency n-grams, and most also use longer dependencies, more stacked nouns, and more reduced object relatives in conversation; we also demonstrate substantial variation across model families. We propose and test a hypothesis as to why: while human language is optimized for successful communication, AI-generated text is increasingly optimized for other goals like math, code, and reasoning, in which there is a pressure for information density rather than successful communication. We provide a proof-of-concept for this hypothesis by fine-tuning a language model under a length penalty, which leads to a divergence from human-like linguistic behavior. Our findings highlight the axes of misalignment between linguistic processing demands in humans and LLMs, and point toward possible remedies.
Figures & tables
Figure 1: Overview of models studied, data collection procedure, and linguistic features under analysis.
Figure 2: Change in linguistic features under round-trip translation (NLLB-200 via Japanese), by model, in standard deviations of the original values across models ( Section B.4 ). Bars: 95% bootstrap intervals; a dash marks a setting in which the model was not run.
Figure 3: Summary of changes to each linguistic feature across applicable models. (a) Changes across the chat, coding, and essay settings. (b) Comparison of round-trip translation vs. student essays as the reference.
Figure 4: Summary of changes to each linguistic feature across applicable models across the following model families: base pre-frontier models, post-trained pre-frontier models, near-frontier open-weight models, GPT, and Claude. Each “cell” denotes a combination of model and setting (e.g. chat, coding, or essay). Not all models are compatible with all settings (e.g. base models only produced essays, not chat or coding results). Model preferences for objective relatives with and without a relativizer tend to point in opposite directions, suggesting a consistent preference for the no-relativizer alternative.
Figure 5: Comparison of linguistic features across models and settings. Diamonds: each model’s pooled value; bars: 95% bootstrap intervals. Dashed lines: human baseline from OpenSubtitles (chat), StackExchange (coding), and PERSUADE 2.0 student essays (essays). Dotted lines: Wikipedia. Details and panels without a human baseline: Section B.4 . Summary figures for by-model and by-feature comparison are provided in Figures 2 , 4 and 3
Figure 6: Comparison of linguistic features across models and settings, relative to a parallel corpus generated with round-trip translation via Japanese (NLLB-200). Stars: paired test of the difference ( Section B.4 ). For most features and most models, responses over-use the feature relative to the round-trip translation.
Figure 7: Effect of length penalty during RL on linguistic preferences. (A) Example item. (B) Share of correct sampled outputs using the short variant, on unseen items. (C, D) Forced-choice probability of the short variant. Points: mean of three seeds (faint: single seeds); dashed: base model; orange: (no KL); blue: .
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Price | Cost (USD) | ||||
| Model | In | Out | Chat | Code | Essay |
| GPT-5.6 Luna | 0.22 | 1.32 | 0.64 | 0.53 | 0.04 |
| GPT-5.6 Terra | 2.20 | 13.20 | 1.67 | 3.32 | 0.36 |
| GPT-5.6 Sol | 4.40 | 22.00 | 2.42 | 6.49 | 0.73 |
| Kimi K2.5 | 0.60 | 3.00 | 0.74 | 12.36 | 0.09 |
| GLM 5 | 1.00 | 3.20 | 0.98 | 29.59 | 0.11 |
Table 1: API cost of the frontier models’ runs in the dataset, by model and setting. Price: USD per million input and output tokens on Amazon Bedrock at the time of collection. Chat: ten conversations per model. Code: five sessions per model. Essay: 70 essays per model. Chat and code costs include the simulated user. GLM 5 and Kimi K2.5 have no prompt caching on Bedrock, so the full context of a code session is billed as new input on every call.
| Measure | Chat | Coding | Essay |
|---|---|---|---|
| Object relative clause without that | Kimi That’s not “trip to Vegas” money, that’s “leave the job you hate ” money. Opus 5.5 That’s great for a beginner, honestly one of the best genres you could pick . | Opus 5.5 Three fail on the old code; the others check behaviour the old code never reached . GLM The case/when fix is technically more impressive, but the unclosed-tag errors are the papercut people feel day-to-day . | Terra Some students may feel that required service takes time away from activities they care about . Opus 5 I am writing to share my thoughts on the cell phone policy you are considering . |
| Denominal verb | Opus 5 And Austin has unusually high inventory, which cushions price spikes better than supply-constrained markets. Sonnet Genuinely unsolicited advice: don’t try to get her to categorize movies as speed-runnable or not. | Luna The last_when fallback is therefore not dead code; it is masking that inconsistency. Sol Prevented Git dry runs from installing hooks or staging changes. | Fable In each case, the logic is the same: pooling independent judgments filters out individual errors. GLM This process mirrors real-world challenges much more closely than a packet of predefined questions. |
| Animate-subject verb, inanimate subject | Sol Sometimes chaos has a 1099; sometimes chaos expects you to remember it independently. Terra A song doesn’t ask you to turn melancholy into a calendar item. | Fable The docstring promises they’re skipped when they “appear anywhere in the source_dir path”; e.g. backing up a dotfolder like ~/.cache/foo was not skipped. Sonnet No new branch, no change in control flow — just closing the gap in an existing check that covered str but forgot its sibling scalar-like type bytes . | Opus 5.5 These are the exact skills the worksheet hoped to teach, but she has learned them because she wanted an answer. Kimi Second, the time objection misunderstands how service actually functions. |
| Dependency length | Sonnet The guitar sitting in the corner because you’re still “deciding” teaches you zero chords. Luna Has the building had water, mold, structural, or insurance claims ? | Fable All five expression shapes — attribute access, subscript, calls, nested calls, string literals — leave current on NAME(’b’) , and the full template renders correctly. Terra A lattice constructed with duplicate points, or a SymbolSet with colliding labels/names, can overwrite internal dictionary or attribute mappings. | Sol Projects completed on weekends, during school events, or over the summer could all count . Sonnet Class time , where a total blackout policy would apply under Policy 1, remains protected and focused. |
| Stacked nouns | GLM Summer park district classes are a whole thing in Chicago and people actually do them. Opus 5 Online banks (Ally, Marcus, Capital One 360, Discover, SoFi) or brokerage money market funds (Fidelity’s SPAXX, Vanguard’s VMFXX) are the standard options. | Luna Fixed absolute dotfile backup path handling while preserving the : marker used during reinstall. Opus 5 Every long line in that list is pre-existing (long error-message string literals , which black leaves alone since it doesn’t split strings without --preview ). | Kimi I acknowledge concerns about distraction, but Policy 1’s class-time shutdown requirement directly addresses this. Fable Instead of a requirement, our school could offer incentives like extra credit, special recognition at assemblies, or a community service honor cord at graduation. |
| Unattested word pair | Kimi Color might be relatively “protected” because it’s so grounded in early perceptual processing. Fable That’s a really legitimate concern, and honestly it’s the part most internet advice glosses over. | Luna as_documents: true previously mutated the tag instance permanently, causing later enrichments to return DocumentsList unexpectedly. Terra The most substantive unfixed item is the existing xfailed lazy-intermediate Lookup behavior; the others are primarily compatibility, validation, or API-policy decisions. | GLM The benefits of attending classes from home outweigh the potential drawbacks, provided it is implemented thoughtfully. Luna Choosing a topic teaches independence, creativity, and responsibility—qualities that cannot be developed through rigid worksheets . |
Table 2: Example sentences for each measure, by setting, quoted from model replies. Typewriter type is inline code.
| Alternation | Short variant | Long variant | tokens |
| Trained | |||
| Dative | gave his wife a gift | gave a gift to his wife | 1 |
| Object relative | the book the professor wrote | the book that the professor wrote | 1 |
| Complement clause | knew the answer was wrong | knew that the answer was wrong | 1 |
| Particle (control) | picked up the book | picked the book up | 0 |
| Held out | |||
Table 3: Alternations. Top: the four alternations in the training items. Bottom: the six held-out alternations, never targeted in training and used only to evaluate the trained models. Passive: 2 tokens for 132 of 150 items, 3 for 15, 1 for 3.
Figure 8: Full prompt for one dative evaluation item (verb held out of training). Only the highlighted Event: line changes between items.
| Setting | Value |
|---|---|
| Base model | Qwen2.5-1.5B-Instruct |
| Fine-tuning | LoRA (PEFT 0.21.0), , , on all linear layers |
| Precision | bf16; gradient checkpointing on |
| Optimizer | Transformers default (AdamW) |
| Learning rate | , linear decay to 0 over 500 steps, no warmup |
| Training steps | 500 (4,000 prompts, 1,000 per alternation; 3,750 distinct) |
Table 4: Training configuration, identical for all 24 runs except , and seed.
| ID | Topic | Register | Opening user message |
|---|---|---|---|
| t01 | Picking up a new hobby as an adult | Everyday life | okay hear me out. i’m 40 and i want to learn guitar. is that a stupid idea or should i just go for it? where would i even start |
| t02 | Renting vs. buying a home | Opinion | My lease is up in a few months and everyone keeps telling me I’m “throwing money away” renting. Is that actually true right now, or is that just something people say? |
| t03 | Finding a briefcase of cash | Hypothetical | ok random question for you. if you found a briefcase with like 500k in unmarked cash just sitting on a park bench, no one around, what would you actually do |
| t04 | A possible career change | Advice-seeking | I’ve been at my job for six years and I’m just… bored. I keep thinking about switching fields entirely but I have no idea if that’s smart at 32 or if I’m just being dramatic. Can I talk it through with you? |
| t05 | Collaborative improvised storytelling | Storytelling | wanna make up a story together? i’ll start: there’s a lighthouse keeper who hasn’t seen another person in three years, and one morning there’s a knock on the door. your turn, take it wherever you want |
| t06 | Everyday phenomena (color of the sky) | Factual curiosity | Random thing I’ve always wondered — why does the sky turn red/orange at sunset but it’s blue the rest of the day? I feel like I should know this by now. |
Table 5: Stimuli for the chat and coding settings. (a) The ten chat topics, each with the first user message of the conversation, reproduced verbatim. Topics were drawn from public lists of conversation starters. (b) The five code repositories, all public on GitHub ( github.com/ repository ), written in Python and MIT-licensed. LOC: lines of source code, excluding tests and vendored or generated files. Stars: GitHub stars when the repository was selected. Tests: tests passing offline at the pinned commit. Commit: first seven characters of the pinned commit hash.
Figure 9: N-gram statistics of open-ended responses to 40 prompts unrelated to the training task, from the base model and the 24 fine-tuned models, scored against Wikipedia. Points: individual seeds; dashed line: base model.
Figure 10: Mean words per response: original and round trip. Rows: translation model and pivot language; the nine frontier models only. Response: all analyzed text of one reply or one essay (code blocks dropped). Word: a token that is not punctuation or an inline-code placeholder. Lines whose round trip is under half or over twice the source’s length are left out of both versions. Error bars: 95% cluster-bootstrap intervals; whole conversations resampled in chat (10 per model) and coding (5 per model), essays within their prompt (70 per model). * / : the 95% interval of the mean paired difference (round trip minus original, per response) is above / below zero.
More in this topic
Language Model Generation Evaluation
Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.
See all 396 papers