cs.CLOct 14, 2026
SaveKoAGE: Do Large Language Models Know, Learn, and Coin Korean Neologisms?
Organizations: Department of Artificial Intelligence, Korea University, Seoul, South Korea · Department of Computer Science and Engineering, Korea University, Seoul, South Korea
Abstract
Korean neologisms arise through diverse mechanisms, a demanding test of language models beyond lexical memorization. To enable systematic evaluation, we introduce NeoTrace, an automated, rerunnable framework that collects and dates neologisms across genres. Its 4,271 entries support KoAGE, a cutoff-aware benchmark with three acquisition and two generation tasks---the latter, to our knowledge, the first to compare model coinages with attested neologisms under their original materials and community context. Across 21 models, lexical recall diverges from contextual inference: in a five-model temporal analysis, models define only 13-30% of cutoff-year words, yet can infer meaning from one usage. However, structural understanding remains limited: even correct definitions barely improve formation analysis of pre-cutoff neologisms. Given word-building materials, single models often follow human formation mechanisms, whereas multi-agent coinage favors abbreviation and depends on interaction framing. KoAGE thus separates what models know, learn from use, and coin.
Figures & tables
Figure 1: Overview of the taxonomy of the KoAGE lexicon: formation (FORM), multiword expression (MWE), borrowing (BORROW), and coinage (COIN).
| Benchmark | Lang. | #Neologisms | #Genres | Cutoff split | Gen. |
| NEO-BENCH | EN | 2,505 | 3 | ✗ | |
| SLANG | EN | 408 | 1 | ✗ | |
| NewTerm | EN | 600 | 1 | ✗ | |
| CNeo-Bench | ZH | 4,759 | 1 | ✗ | ✗ |
| KoNeoBench | KO | 1,785 | 1 | ✗ | ✗ |
| KoAGE (ours) | KO | 4,271 | 6 | ✓ | ✓ |
Table 1: Neologism benchmarks. #Genres: counted among news, social media, messenger chat, video, spoken dialogue and lexicographic references; Cutoff split: results separated into pre- and post-cutoff items per model ( = cutoffs shown or post-cutoff items only, no per-model split); Gen.: coinage evaluated.
Figure 2: The NeoTrace framework: candidate extraction and automated verification over the three collection tracks. Validation is described in § 3.3 ; details in Appendix A .
Figure 3: (a) Neologism rate by candidate rank: every candidate up to 2K verified, 100 sampled per window beyond. (b) Share of 200 sampled entries whose earliest sense-matched attestation lies within the top results per channel.
Figure 4: (a) Blind human validation of accepted Korean items. (b) Criterion-wise web verification of accepted items from Reddit and news.
Figure 5: The KoAGE tasks with one example each: input and expected output for the three acquisition tasks (A1–A3) and the two generation tasks (G1–G2).
| Acquisition (accuracy, %; pre- / post-cutoff) | G1 coinage (exact, %) | G2 community (%) | ||||||||||
| Model | Cutoff | A1-W | A1-WU | A2 | A3-W | A3-WD | M | MD | MDC | Formed | Reprod. | Mech. |
| Proprietary (API) | ||||||||||||
| GPT-3.5 Turbo | 2021-09 | 0.9 / 0.1 | 6.7 / 5.3 | 71.1 / 63.1 | 23.6 / 21.2 | 20.5 / 19.0 | 15.5 | 16.7 | 17.4 | 0.0 | 0.0 | 0.0 |
| GPT-4o | 2023-10 | 22.1 / 7.1 | 45.2 / 31.6 | 90.5 / 84.4 | 51.6 / 48.9 | 55.6 / 56.4 | 29.0 | 32.9 | 36.7 | 20.0 | 0.0 | 12.0 |
| GPT-5 | 2024-09 | 51.2 / 21.2 | 70.9 / 49.4 | 97.1 / 92.2 | 67.9 / 59.0 | 74.5 / 77.6 | 44.7 | 47.1 | 45.8 | 86.4 | 22.7 | 22.7 |
| GPT-5.4 mini | 2025-08 | 32.1 / 15.4 | 57.3 / 32.0 | 89.7 / 78.0 | 61.0 / 56.6 | 66.0 / 70.7 | 39.2 | 36.7 | 42.0 | 65.2 | 13.0 | 26.1 |
Table 2: KoAGE results for 21 models across acquisition and generation tasks. Acquisition cells report accuracy on items dated before (pre-cutoff) and after (post-cutoff) each model’s training cutoff; item counts are provided in Table 11 . Cutoff denotes the documented training cutoff; r indicates the release month when no cutoff is reported.
Figure 6: Word-only definition accuracy by year of first attestation. The shaded band spans the accuracy of the five models on words from their own cutoff year.
Figure 7: Acquisition performance by domain. Top: distribution of A1 items in each domain by formation type. Bottom: accuracy across the four acquisition regimes for 13 proprietary models.
Figure 8: Formation mechanism of the same word by the single agent given its materials (G1-M), by the humans, and by the multi-agent community (G2). Pooled over 225 formed sessions of 21 backbones (Appendix E.2 ).
Figure 9: (a) Adding the two protocol devices cumulatively on three backbones. (b) Formation under the full protocol vs. cutoff (bubble size: share of sessions reproducing the human word).
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| FORM (Morphological) | MWE (Multiword) | BORROW | COIN | Total | |||||||
| Source | CPD | DER | BLD | ABB | SYN | NSEQ | MOD | CLS | |||
| Literature | 285 | 355 | 562 | 435 | 20 | 1,078 | 71 | 14 | 120 | 14 | 2,954 |
| Online corpus | 54 | 37 | 64 | 37 | 7 | – | 1 | 1 | 34 | 4 | 239 |
| Web API | 158 | 97 | 244 | 141 | 10 | 234 | 29 | 28 | 126 | 11 | 1,078 |
| Total | 497 | 489 | 870 | 613 | 37 | 1,312 | 101 | 43 | 280 | 29 | 4,271 |
Table 7: Composition of the KoAGE lexicon (4,271 entries) by collection track and taxonomy leaf. CPD: compounding; DER: derivation; BLD: blending (incl. splinters); ABB: abbreviation (initial-syllable and clipping); SYN: syntactic fusion; NSEQ: noun sequence; MOD: modifier construction; CLS: clausal expression. “–” denotes zero.
| Name in tables | API identifier / HF repository | Release | Cutoff | Decoding | Weights | A1 judge | Ref. |
| Proprietary (API) | |||||||
| GPT-3.5 Turbo | gpt-3.5-turbo-0125 | 2023-03 | 2021-09 | vendor default | API | Claude Opus 5 (low) | ( Ouyang et al., 2022 ) |
| GPT-4o | gpt-4o-2024-05-13 | 2024-05 | 2023-10 | vendor default | API | Claude Opus 5 (low) | ( OpenAI, 2024 ) |
| GPT-5 | gpt-5-2025-08-07 | 2025-08 | 2024-09 | vendor default , effort low | API | Claude Opus 5 (low) | ( OpenAI, 2025 ) |
| GPT-5.4 mini | gpt-5.4-mini-2026-03-17 | 2026-03 | 2025-08 | vendor default , effort low | API | Claude Opus 5 (low) | ( OpenAI, 2026b ) |
| GPT-5.6 sol | gpt-5.6-sol | 2026-07 | 2026-02 | vendor default , effort low | API | Claude Opus 5 (low) | ( OpenAI, 2026a ) |
Table 8: Evaluated models: API identifier or Hugging Face repository, release and cutoff months (“–” = unpublished; release month used), decoding, weights, A1 judge and reference.
| Task | Items | Admitted items |
| A1-W | 2,504 | evaluation pool: entries first attested 2020 or later |
| A1-WU | 1,819 | pool entries with a reviewed usage sentence |
| A2 | 1,230 | usage masks without leaking the answer; all three distractors pass the adversarial check |
| A3 | 1,332 | FORM entries plus controls from BORROW and COIN |
| G1 | 839 | pool entries whose word is not a plain concatenation of its materials |
| G2 | 25 | cards with a verified origin (community, medium, situation) |
Table 9: Item counts per task and the admission rules.
| Judgment | Sample | Agree | ||
| A1 definition match (0/1/2) | ||||
| Judge vs. reference | known-answer pairs (156 approved + 410 controls) | 566 | 96–97 | 0.91–0.93 (bin.) |
| Judge vs. judge (Opus 5 vs. GPT-5.6) | stratified Sonnet 5 outputs | 300 | 79–81 | 0.68–0.72 |
| Judge vs. human | random, stratified by judge score, all models | 150 | 81.3 | 0.72 (lin. 0.79) |
| binary (exact vs. not) | same | 150 | 89.3 | 0.77 |
| G1 mechanism label (four of the five classes) | ||||
Table 10: Judge validation against one human rater: agreement (%) and Cohen’s (lin. = linearly weighted, bin. = exact vs. not). The near-boundary row checks one judge label, so is undefined.
| A1-W | A1-WU | A2 | A3-W | A3-WD | G2 | |||||||
| Model | pre | post | pre | post | pre | post | pre | post | pre | post | cards | Formed CI |
| GPT-3.5 Turbo | 931 ( 1) | 1,474 ( 0) | 703 ( 2) | 1,075 ( 1) | 484 ( 4) | 713 ( 4) | 415 ( 4) | 838 ( 3) | 415 ( 4) | 838 ( 3) | 25 | 0–13 |
| GPT-4o | 1,832 ( 2) | 590 ( 2) | 1,383 ( 3) | 411 ( 4) | 969 ( 2) | 244 ( 5) | 913 ( 3) | 360 ( 5) | 913 ( 3) | 360 ( 5) | 25 | 9–39 |
| GPT-5 | 2,174 ( 2) | 241 ( 5) | 1,641 ( 2) | 164 ( 8) | 1,159 ( 1) | 64 ( 7) | 1,107 ( 3) | 156 ( 8) | 1,107 ( 3) | 156 ( 6) | 22 | 67–95 |
| GPT-5.4 mini | 2,335 ( 2) | 149 ( 6) | 1,703 ( 2) | 100 ( 9) | 1,184 ( 2) | 41 ( 12) | 1,221 ( 3) | 99 ( 10) | 1,221 ( 3) | 99 ( 9) | 23 | 45–81 |
| GPT-5.6 sol | 2,422 ( 2) | 82 ( 8) | 1,766 ( 2) | 53 ( 13) | 1,208 ( 1) | 22 ( 7) | 1,280 ( 2) | 52 ( 12) | 1,280 ( 2) | 52 ( 10) | 19 | 83–100 |
Table 11: Item-level 95% confidence intervals behind Table 2 : for each acquisition cell, the number of items first attested before (pre) or after (post) the model’s cutoff (boundary-year items without a month excluded) and, in parentheses, the half-width of the 95% Wilson interval of its accuracy in points. The G2 columns give the cards unknown to the backbone and the card-level 95% Wilson interval of the formation rate.
Figure 10: A1 accuracy on pre-cutoff items by age of the word at cutoff (solid: word only; dotted: with one usage).
| Neologism reports | Web-collected | |||||||||||
| Model | 20 | 21 | 22 | 23 | 24 | 20 | 21 | 22 | 23 | 24 | 25 | 26 |
| GPT-5.6 sol | 71 | 60 | 62 | 46 | 41 | 69 | 64 | 56 | 56 | 57 | 42 | 26 |
| Claude Sonnet 5 | 59 | 40 | 42 | 35 | 28 | 58 | 42 | 34 | 32 | 30 | 26 | 16 |
| Gemini 3 Flash | 75 | 65 | 66 | 54 | 48 | 75 | 80 | 74 | 67 | 62 | 35 | 26 |
| GPT-5 | 67 | 51 | 55 | 40 | 31 | 65 | 58 | 46 | 35 | 28 | 23 | 24 |
| Solar Pro 4 | 48 | 35 | 31 | 21 | 14 | 42 | 39 | 26 | 21 | 18 | 15 | 5 |
Table 12: A1-W accuracy (%) by first-attestation year for items from the Neologisms 2020–2024 series and from our web collection; model cutoffs as in Table 2 .
Figure 11: A1-W accuracy on pre-cutoff items by cumulative corpus mentions up to the model’s cutoff year (log bins), for three word ages at cutoff; dots = median exposure; five 2025–2026 models pooled.
Figure 12: A1-WU exact accuracy by taxonomy leaf. Abbreviations are the hardest leaf for every model; multiword expressions and borrowings the easiest.
| High transparency | Low transparency | Single-word only | |||||||
| Model | Inst. | Pop. | Coll. | Inst. | Pop. | Coll. | Inst. | Pop. | Coll. |
| GPT-4o | 30 | 33 | 39 | 0 | 0 | 2 | 24 | 22 | 18 |
| GPT-5 | 74 | 69 | 59 | 21 | 22 | 14 | 55 | 48 | 35 |
| GPT-5.6 sol | 77 | 72 | 78 | 37 | 27 | 30 | 57 | 52 | 51 |
| Claude Sonnet 4.5 | 44 | 34 | 35 | 0 | 6 | 2 | 26 | 23 | 16 |
| Claude Sonnet 5 | 62 | 51 | 57 | 8 | 7 | 5 | 42 | 32 | 25 |
Table 13: A1-W accuracy (%) on pre-cutoff items by domain (Inst. = institutional, Pop. = popular culture, Coll. = colloquial).
| A1 exact (partial) | ||||
| Model | W | WU | +search | |
| each model’s own post-cutoff items | ||||
| GPT-5.6 sol | 53 | 23 (15) | 42 (28) | 70 (23) |
| Claude Sonnet 5 | 55 | 11 (15) | 42 (27) | 65 (20) |
| Gemini 3.5 Flash-Lite | 46 | 15 (7) | 17 (26) | 52 (17) |
| items post-cutoff for all three | ||||
Table 14: Retrieval ablation on post-cutoff items. Mean queries per five-item request: 3.2 (GPT-5.6 sol), 5.8 (Sonnet 5), 1.8 (Gemini).
| exact on known items | |||||
| Model | unk. | known | M | MD | MDC |
| Proprietary (API) | |||||
| GPT-3.5 Turbo | 830 | 3 | – | – | – |
| GPT-4o | 699 | 86 | 46.5 | 58.1 | 61.5 |
| GPT-5 | 490 | 277 | 62.8 | 66.1 | 72.6 |
| GPT-5.4 mini | 561 | 207 | 64.3 | 60.4 | 69.4 |
Table 15: G1 exact match (%) for words already defined correctly in A1-W. Counts of unknown (unk.) and known evaluation items are shown for comparison; results on unknown words are in Table 2 . Cells with fewer than 20 known items are suppressed. Items receiving partial A1-W credit are excluded from both splits, so counts need not sum to 839.
| Backbone | Form. | Single-word |
| Proprietary (API) | ||
| GPT-3.5 Turbo | 0 | 0 |
| GPT-4o | 20 | 12 |
| GPT-5 | 86 | 77 |
| GPT-5.4 mini | 65 | 65 |
| GPT-5.6 sol | 100 | 89 |
Table 16: G2 formation under the judge criterion (label shared by 2 agents) and under a stricter single-word criterion (no space or punctuation, 8 characters), in percent. Reproduction is reported in Table 2 .
| Community | Form. | Repro. | Exact | Abbr. |
| Mixed (4 backbones, 2 seeds) | 97 | 53 | 25 | 42 |
| GPT-5.6 sol alone | 100 | 47 | 26 | 42 |
| Gemini 3 Flash alone | 100 | 44 | 17 | 56 |
| Claude Sonnet 5 alone | 96 | 29 | 17 | 26 |
| Solar Pro 4 alone | 79 | 12 | 0 | 26 |
Table 17: Mixed vs. homogeneous communities under the full protocol (%; unknown cards only: 32 mixed sessions over two seeds, 18–24 homogeneous sessions). Form. = shared name; Repro. = exact or near reproduction; Abbr. = abbreviations among formed names.
| Model | Cutoff | pre | post | cut.+1 | recorded | month | band |
| GPT-3.5 Turbo | 2021-09 | 0.9 | 0.1 | 0.1 | 0.2 | 0.2 | 0.5 |
| GPT-4o | 2023-10 | 22.1 | 7.1 | 5.8 | 7.8 | 7.3 | 12.6 |
| GPT-5 | 2024-09 | 51.2 | 21.2 | 20.4 | 23.9 | 21.3 | 29.9 |
| GPT-5.4 mini | 2025-08 | 32.1 | 15.4 | 12.6 | 16.7 | 15.4 | 15.7 |
| GPT-5.6 sol | 2026-02 | 57.5 | 19.5 | – | – | 19.5 | 22.1 |
| Claude Sonnet 4.5 | 2025-07 | 30.0 | 10.8 | 7.4 | 11.1 | 10.8 | 15.7 |
Table 18: Temporal sensitivity of A1-W (exact, %) for models with a documented cutoff: pre-cutoff items; all post-cutoff items; items first attested at least one year after the cutoff; the recorded-basis subset of those; month-dated items strictly after the cutoff month; and items from the cutoff year (band). “–”: no item yet.
| A2 accuracy (%) | Distractor chosen in errors (%) | |||||
| Model | all | unknown | known | same type | hard neg. | near syn. |
| GPT-3.5 Turbo | 66.1 | 65.7 | 80.0 | 33.3 | 14.9 | 51.8 |
| GPT-4o | 89.4 | 85.5 | 94.8 | 46.2 | 10.0 | 43.8 |
| GPT-5 | 96.7 | 92.5 | 99.5 | 40.0 | 2.5 | 57.5 |
| GPT-5.4 mini | 89.2 | 83.4 | 95.0 | 42.9 | 7.5 | 49.6 |
| GPT-5.6 sol | 97.9 | 95.0 | 99.0 | 53.8 | 7.7 | 38.5 |
Table 19: A2 accuracy on all items and conditional on A1-W outcome (unknown = wrong or abstained; known = exact), and which distractor the model chose when wrong (share of errors; unparsed answers omitted).
| pre-cutoff | post-cutoff | |||||
| Model | gain | loss | stay wrong | gain | loss | stay wrong |
| GPT-3.5 Turbo | 9.6 | 12.8 | 66.7 | 10.7 | 13.0 | 68.0 |
| GPT-4o | 13.8 | 9.7 | 34.6 | 16.9 | 9.4 | 34.2 |
| GPT-5 | 14.5 | 7.9 | 17.6 | 23.1 | 4.5 | 17.9 |
| GPT-5.4 mini | 15.4 | 10.4 | 23.6 | 19.2 | 5.1 | 24.2 |
| GPT-5.6 sol | 8.4 | 5.5 | 12.1 | 17.3 | 1.9 | 15.4 |
Table 20: A3 item-level transitions when the definition is added (W WD): share of items that become correct (gain), become wrong (loss), or stay wrong, pre- vs post-cutoff. Means are over the 13 proprietary models.
| pre-cutoff | post-cutoff | |||||||
| Model | W | WD | WDM | fixed | W | WD | WDM | fixed |
| GPT-5.6 sol | 83.0 | 85.7 | 86.6 | 20 | 76.7 | 88.4 | 86.0 | 25 |
| Claude Sonnet 5 | 71.0 | 80.9 | 70.6 | 27 | 62.2 | 84.4 | 73.3 | 0 |
| Gemini 3 Flash | 78.6 | 80.2 | 83.6 | 31 | 73.7 | 78.9 | 84.2 | 39 |
| Solar Pro 4 | 55.1 | 57.8 | 61.2 | 32 | 39.5 | 74.4 | 65.1 | 30 |
Table 21: A3 with gold constituents added to definition (WDM) on the 1,203 FORM items that have materials: accuracy (%) under W, WD and WDM, and the share of items still wrong after WD that WDM corrects (fixed).
| Formed (%) by agents agreeing | Persistence (% of formed) | Mech. agreement formed | G1-M mech. match | ||||||||
| Backbone | cards | 2 | 3 | 4 | 2 agents reuse | 3 later ep. | % | 95% CI | % | 95% CI | concept-only |
| GPT-3.5 Turbo | 25 | 0.0 | 0.0 | 0.0 | – | – | – | – | 54.5 | 32–77 | 52 |
| GPT-4o | 25 | 20.0 | 12.0 | 4.0 | 100.0 | 100.0 | 60.0 | 20–100 | 72.7 | 55–91 | 36 |
| GPT-5 | 22 | 86.4 | 68.2 | 54.5 | 100.0 | 89.5 | 52.6 | 32–74 | 86.4 | 73–100 | 45 |
| GPT-5.4 mini | 23 | 65.2 | 47.8 | 34.8 | 100.0 | 86.7 | 60.0 | 33–80 | 81.8 | 64–95 | 39 |
| GPT-5.6 sol | 19 | 100.0 | 100.0 | 94.7 | 100.0 | 100.0 | 68.4 | 47–89 | 81.8 | 64–95 | 58 |
Table 22: G2 robustness on the main set (full protocol, one seed, cards unknown to the backbone). Formation under stricter consensus thresholds; persistence of the shared name in the chat log (reused by at least two agents before the exit interview; reappearing in at least three later episodes); mechanism agreement with the human coinage among formed sessions with a session-level bootstrap 95% interval; G1-M mechanism match on the same words (22 of the 25 are G1 items); and mechanism match of a single-model concept-only coinage control (same concept text, no materials, no community).
| A1-W | A1-WU | ||||
| Slice | items | prop. | open | prop. | open |
| Provenance | |||||
| Neologism reports | 1671 | 35.7 | 12.3 | 51.4 | 31.6 |
| Web API track | 569 | 31.6 | 7.3 | 45.2 | 26.5 |
| Online corpus track | 162 | 37.3 | 9.2 | 59.4 | 33.2 |
| Formation type | |||||
Table 23: A1 accuracy (%, pre-cutoff items; proprietary and open-weight means) by lexicon provenance, formation type, verification confidence of web-collected entries, and number of attested usages in the lexicon. Slices with fewer than 20 items are omitted.
More in this topic
Language Modeling
Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.
See all 522 papers