KoAGE: Do Large Language Models Know, Learn, and Coin Korean Neologisms?
Authors: SungHo Kim, Eojin Jeon, SangKeun Lee
Organizations: Department of Artificial Intelligence, Korea University, Seoul, South Korea · Department of Computer Science and Engineering, Korea University, Seoul, South Korea
Korean neologisms arise through diverse mechanisms, a demanding test of language models beyond lexical memorization. To enable systematic evaluation, we introduce NeoTrace, an automated, rerunnable framework that collects and dates neologisms across genres. Its 4,271 entries support KoAGE, a cutoff-aware benchmark with three acquisition and two generation tasks---the latter, to our knowledge, the first to compare model coinages with attested neologisms under their original materials and community context. Across 21 models, lexical recall diverges from contextual inference: in a five-model temporal analysis, models define only 13-30% of cutoff-year words, yet can infer meaning from one usage. However, structural understanding remains limited: even correct definitions barely improve formation analysis of pre-cutoff neologisms. Given word-building materials, single models often follow human formation mechanisms, whereas multi-agent coinage favors abbreviation and depends on interaction framing. KoAGE thus separates what models know, learn from use, and coin.
Figures & tables
Figure 1: Overview of the taxonomy of the KoAGE lexicon: formation (FORM), multiword expression (MWE), borrowing (BORROW), and coinage (COIN).
Benchmark
Lang.
#Neologisms
#Genres
Cutoff split
Gen.
NEO-BENCH
EN
2,505
3
△
✗
SLANG
EN
408
1
△
✗
NewTerm
EN
600
1
△
✗
CNeo-Bench
ZH
4,759
1
✗
✗
KoNeoBench
KO
1,785
1
✗
✗
KoAGE (ours)
KO
4,271
6
✓
✓
Table 1: Neologism benchmarks. #Genres: counted among news, social media, messenger chat, video, spoken dialogue and lexicographic references; Cutoff split: results separated into pre- and post-cutoff items per model ( △ = cutoffs shown or post-cutoff items only, no per-model split); Gen.: coinage evaluated.
Figure 2: The NeoTrace framework: candidate extraction and automated verification over the three collection tracks. Validation is described in § 3.3 ; details in Appendix A .
Figure 3: (a) Neologism rate by candidate rank: every candidate up to 2K verified, 100 sampled per window beyond. (b) Share of 200 sampled entries whose earliest sense-matched attestation lies within the top k results per channel.
Figure 4: (a) Blind human validation of accepted Korean items. (b) Criterion-wise web verification of accepted items from Reddit and news.
Figure 5: The KoAGE tasks with one example each: input and expected output for the three acquisition tasks (A1–A3) and the two generation tasks (G1–G2).
Acquisition (accuracy, %; pre- / post-cutoff)
G1 coinage (exact, %)
G2 community (%)
Model
Cutoff
A1-W
A1-WU
A2
A3-W
A3-WD
M
MD
MDC
Formed
Reprod.
Mech.
Proprietary (API)
GPT-3.5 Turbo
2021-09
0.9 / 0.1
6.7 / 5.3
71.1 / 63.1
23.6 / 21.2
20.5 / 19.0
15.5
16.7
17.4
0.0
0.0
0.0
GPT-4o
2023-10
22.1 / 7.1
45.2 / 31.6
90.5 / 84.4
51.6 / 48.9
55.6 / 56.4
29.0
32.9
36.7
20.0
0.0
12.0
GPT-5
2024-09
51.2 / 21.2
70.9 / 49.4
97.1 / 92.2
67.9 / 59.0
74.5 / 77.6
44.7
47.1
45.8
86.4
22.7
22.7
GPT-5.4 mini
2025-08
32.1 / 15.4
57.3 / 32.0
89.7 / 78.0
61.0 / 56.6
66.0 / 70.7
39.2
36.7
42.0
65.2
13.0
26.1
Table 2: KoAGE results for 21 models across acquisition and generation tasks. Acquisition cells report accuracy on items dated before (pre-cutoff) and after (post-cutoff) each model’s training cutoff; item counts are provided in Table 11 . Cutoff denotes the documented training cutoff; r indicates the release month when no cutoff is reported.
Figure 6: Word-only definition accuracy by year of first attestation. The shaded band spans the accuracy of the five models on words from their own cutoff year.
Figure 7: Acquisition performance by domain. Top: distribution of A1 items in each domain by formation type. Bottom: accuracy across the four acquisition regimes for 13 proprietary models.
Figure 8: Formation mechanism of the same word by the single agent given its materials (G1-M), by the humans, and by the multi-agent community (G2). Pooled over 225 formed sessions of 21 backbones (Appendix E.2 ).
Figure 9: (a) Adding the two protocol devices cumulatively on three backbones. (b) Formation under the full protocol vs. cutoff (bubble size: share of sessions reproducing the human word).
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
FORM (Morphological)
MWE (Multiword)
BORROW
COIN
Total
Source
CPD
DER
BLD
ABB
SYN
NSEQ
MOD
CLS
Literature
285
355
562
435
20
1,078
71
14
120
14
2,954
Online corpus
54
37
64
37
7
–
1
1
34
4
239
Web API
158
97
244
141
10
234
29
28
126
11
1,078
Total
497
489
870
613
37
1,312
101
43
280
29
4,271
Appendix
Table 7: Composition of the KoAGE lexicon (4,271 entries) by collection track and taxonomy leaf. CPD: compounding; DER: derivation; BLD: blending (incl. splinters); ABB: abbreviation (initial-syllable and clipping); SYN: syntactic fusion; NSEQ: noun sequence; MOD: modifier construction; CLS: clausal expression. “–” denotes zero.
Name in tables
API identifier / HF repository
Release
Cutoff
Decoding
Weights
A1 judge
Ref.
Proprietary (API)
GPT-3.5 Turbo
gpt-3.5-turbo-0125
2023-03
2021-09
vendor default T
API
Claude Opus 5 (low)
( Ouyang et al., 2022 )
GPT-4o
gpt-4o-2024-05-13
2024-05
2023-10
vendor default T
API
Claude Opus 5 (low)
( OpenAI, 2024 )
GPT-5
gpt-5-2025-08-07
2025-08
2024-09
vendor default T , effort low
API
Claude Opus 5 (low)
( OpenAI, 2025 )
GPT-5.4 mini
gpt-5.4-mini-2026-03-17
2026-03
2025-08
vendor default T , effort low
API
Claude Opus 5 (low)
( OpenAI, 2026b )
GPT-5.6 sol
gpt-5.6-sol
2026-07
2026-02
vendor default T , effort low
API
Claude Opus 5 (low)
( OpenAI, 2026a )
Appendix
Table 8: Evaluated models: API identifier or Hugging Face repository, release and cutoff months (“–” = unpublished; release month used), decoding, weights, A1 judge and reference.
Task
Items
Admitted items
A1-W
2,504
evaluation pool: entries first attested 2020 or later
A1-WU
1,819
pool entries with a reviewed usage sentence
A2
1,230
usage masks without leaking the answer; all three distractors pass the adversarial check
A3
1,332
FORM entries plus controls from BORROW and COIN
G1
839
pool entries whose word is not a plain concatenation of its materials
G2
25
cards with a verified origin (community, medium, situation)
Appendix
Table 9: Item counts per task and the admission rules.
Judgment
Sample
n
Agree
κ
A1 definition match (0/1/2)
Judge vs. reference
known-answer pairs (156 approved + 410 controls)
566
96–97
0.91–0.93 (bin.)
Judge vs. judge (Opus 5 vs. GPT-5.6)
stratified Sonnet 5 outputs
300
79–81
0.68–0.72
Judge vs. human
random, stratified by judge score, all models
150
81.3
0.72 (lin. 0.79)
binary (exact vs. not)
same
150
89.3
0.77
G1 mechanism label (four of the five classes)
Appendix
Table 10: Judge validation against one human rater: agreement (%) and Cohen’s κ (lin. = linearly weighted, bin. = exact vs. not). The near-boundary row checks one judge label, so κ is undefined.
A1-W
A1-WU
A2
A3-W
A3-WD
G2
Model
pre
post
pre
post
pre
post
pre
post
pre
post
cards
Formed CI
GPT-3.5 Turbo
931 ( ± 1)
1,474 ( ± 0)
703 ( ± 2)
1,075 ( ± 1)
484 ( ± 4)
713 ( ± 4)
415 ( ± 4)
838 ( ± 3)
415 ( ± 4)
838 ( ± 3)
25
0–13
GPT-4o
1,832 ( ± 2)
590 ( ± 2)
1,383 ( ± 3)
411 ( ± 4)
969 ( ± 2)
244 ( ± 5)
913 ( ± 3)
360 ( ± 5)
913 ( ± 3)
360 ( ± 5)
25
9–39
GPT-5
2,174 ( ± 2)
241 ( ± 5)
1,641 ( ± 2)
164 ( ± 8)
1,159 ( ± 1)
64 ( ± 7)
1,107 ( ± 3)
156 ( ± 8)
1,107 ( ± 3)
156 ( ± 6)
22
67–95
GPT-5.4 mini
2,335 ( ± 2)
149 ( ± 6)
1,703 ( ± 2)
100 ( ± 9)
1,184 ( ± 2)
41 ( ± 12)
1,221 ( ± 3)
99 ( ± 10)
1,221 ( ± 3)
99 ( ± 9)
23
45–81
GPT-5.6 sol
2,422 ( ± 2)
82 ( ± 8)
1,766 ( ± 2)
53 ( ± 13)
1,208 ( ± 1)
22 ( ± 7)
1,280 ( ± 2)
52 ( ± 12)
1,280 ( ± 2)
52 ( ± 10)
19
83–100
Appendix
Table 11: Item-level 95% confidence intervals behind Table 2 : for each acquisition cell, the number of items first attested before (pre) or after (post) the model’s cutoff (boundary-year items without a month excluded) and, in parentheses, the half-width of the 95% Wilson interval of its accuracy in points. The G2 columns give the cards unknown to the backbone and the card-level 95% Wilson interval of the formation rate.
Figure 10: A1 accuracy on pre-cutoff items by age of the word at cutoff (solid: word only; dotted: with one usage).
Neologism reports
Web-collected
Model
20
21
22
23
24
20
21
22
23
24
25
26
GPT-5.6 sol
71
60
62
46
41
69
64
56
56
57
42
26
Claude Sonnet 5
59
40
42
35
28
58
42
34
32
30
26
16
Gemini 3 Flash
75
65
66
54
48
75
80
74
67
62
35
26
GPT-5
67
51
55
40
31
65
58
46
35
28
23
24
Solar Pro 4
48
35
31
21
14
42
39
26
21
18
15
5
Appendix
Table 12: A1-W accuracy (%) by first-attestation year for items from the Neologisms 2020–2024 series and from our web collection; model cutoffs as in Table 2 .
Figure 11: A1-W accuracy on pre-cutoff items by cumulative corpus mentions up to the model’s cutoff year (log bins), for three word ages at cutoff; dots = median exposure; five 2025–2026 models pooled.
Figure 12: A1-WU exact accuracy by taxonomy leaf. Abbreviations are the hardest leaf for every model; multiword expressions and borrowings the easiest.
High transparency
Low transparency
Single-word only
Model
Inst.
Pop.
Coll.
Inst.
Pop.
Coll.
Inst.
Pop.
Coll.
GPT-4o
30
33
39
0
0
2
24
22
18
GPT-5
74
69
59
21
22
14
55
48
35
GPT-5.6 sol
77
72
78
37
27
30
57
52
51
Claude Sonnet 4.5
44
34
35
0
6
2
26
23
16
Claude Sonnet 5
62
51
57
8
7
5
42
32
25
Appendix
Table 13: A1-W accuracy (%) on pre-cutoff items by domain (Inst. = institutional, Pop. = popular culture, Coll. = colloquial).
A1 exact (partial)
Model
n
W
WU
+search
each model’s own post-cutoff items
GPT-5.6 sol
53
23 (15)
42 (28)
70 (23)
Claude Sonnet 5
55
11 (15)
42 (27)
65 (20)
Gemini 3.5 Flash-Lite
46
15 (7)
17 (26)
52 (17)
items post-cutoff for all three
Appendix
Table 14: Retrieval ablation on post-cutoff items. Mean queries per five-item request: 3.2 (GPT-5.6 sol), 5.8 (Sonnet 5), 1.8 (Gemini).
n
exact on known items
Model
unk.
known
M
MD
MDC
Proprietary (API)
GPT-3.5 Turbo
830
3
–
–
–
GPT-4o
699
86
46.5
58.1
61.5
GPT-5
490
277
62.8
66.1
72.6
GPT-5.4 mini
561
207
64.3
60.4
69.4
Appendix
Table 15: G1 exact match (%) for words already defined correctly in A1-W. Counts of unknown (unk.) and known evaluation items are shown for comparison; results on unknown words are in Table 2 . Cells with fewer than 20 known items are suppressed. Items receiving partial A1-W credit are excluded from both splits, so counts need not sum to 839.
Backbone
Form.
Single-word
Proprietary (API)
GPT-3.5 Turbo
0
0
GPT-4o
20
12
GPT-5
86
77
GPT-5.4 mini
65
65
GPT-5.6 sol
100
89
Appendix
Table 16: G2 formation under the judge criterion (label shared by ≥ 2 agents) and under a stricter single-word criterion (no space or punctuation, ≤ 8 characters), in percent. Reproduction is reported in Table 2 .
Community
Form.
Repro.
Exact
Abbr.
Mixed (4 backbones, 2 seeds)
97
53
25
42
GPT-5.6 sol alone
100
47
26
42
Gemini 3 Flash alone
100
44
17
56
Claude Sonnet 5 alone
96
29
17
26
Solar Pro 4 alone
79
12
0
26
Appendix
Table 17: Mixed vs. homogeneous communities under the full protocol (%; unknown cards only: 32 mixed sessions over two seeds, 18–24 homogeneous sessions). Form. = shared name; Repro. = exact or near reproduction; Abbr. = abbreviations among formed names.
Model
Cutoff
pre
post
≥ cut.+1
recorded
month
band
GPT-3.5 Turbo
2021-09
0.9
0.1
0.1
0.2
0.2
0.5
GPT-4o
2023-10
22.1
7.1
5.8
7.8
7.3
12.6
GPT-5
2024-09
51.2
21.2
20.4
23.9
21.3
29.9
GPT-5.4 mini
2025-08
32.1
15.4
12.6
16.7
15.4
15.7
GPT-5.6 sol
2026-02
57.5
19.5
–
–
19.5
22.1
Claude Sonnet 4.5
2025-07
30.0
10.8
7.4
11.1
10.8
15.7
Appendix
Table 18: Temporal sensitivity of A1-W (exact, %) for models with a documented cutoff: pre-cutoff items; all post-cutoff items; items first attested at least one year after the cutoff; the recorded-basis subset of those; month-dated items strictly after the cutoff month; and items from the cutoff year (band). “–”: no item yet.
A2 accuracy (%)
Distractor chosen in errors (%)
Model
all
unknown
known
same type
hard neg.
near syn.
GPT-3.5 Turbo
66.1
65.7
80.0
33.3
14.9
51.8
GPT-4o
89.4
85.5
94.8
46.2
10.0
43.8
GPT-5
96.7
92.5
99.5
40.0
2.5
57.5
GPT-5.4 mini
89.2
83.4
95.0
42.9
7.5
49.6
GPT-5.6 sol
97.9
95.0
99.0
53.8
7.7
38.5
Appendix
Table 19: A2 accuracy on all items and conditional on A1-W outcome (unknown = wrong or abstained; known = exact), and which distractor the model chose when wrong (share of errors; unparsed answers omitted).
pre-cutoff
post-cutoff
Model
gain
loss
stay wrong
gain
loss
stay wrong
GPT-3.5 Turbo
9.6
12.8
66.7
10.7
13.0
68.0
GPT-4o
13.8
9.7
34.6
16.9
9.4
34.2
GPT-5
14.5
7.9
17.6
23.1
4.5
17.9
GPT-5.4 mini
15.4
10.4
23.6
19.2
5.1
24.2
GPT-5.6 sol
8.4
5.5
12.1
17.3
1.9
15.4
Appendix
Table 20: A3 item-level transitions when the definition is added (W → WD): share of items that become correct (gain), become wrong (loss), or stay wrong, pre- vs post-cutoff. Means are over the 13 proprietary models.
pre-cutoff
post-cutoff
Model
W
WD
WDM
fixed
W
WD
WDM
fixed
GPT-5.6 sol
83.0
85.7
86.6
20
76.7
88.4
86.0
25
Claude Sonnet 5
71.0
80.9
70.6
27
62.2
84.4
73.3
0
Gemini 3 Flash
78.6
80.2
83.6
31
73.7
78.9
84.2
39
Solar Pro 4
55.1
57.8
61.2
32
39.5
74.4
65.1
30
Appendix
Table 21: A3 with gold constituents added to definition (WDM) on the 1,203 FORM items that have materials: accuracy (%) under W, WD and WDM, and the share of items still wrong after WD that WDM corrects (fixed).
Formed (%) by agents agreeing
Persistence (% of formed)
Mech. agreement ∣ formed
G1-M mech. match
Backbone
cards
≥ 2
≥ 3
4
≥ 2 agents reuse
≥ 3 later ep.
%
95% CI
%
95% CI
concept-only
GPT-3.5 Turbo
25
0.0
0.0
0.0
–
–
–
–
54.5
32–77
52
GPT-4o
25
20.0
12.0
4.0
100.0
100.0
60.0
20–100
72.7
55–91
36
GPT-5
22
86.4
68.2
54.5
100.0
89.5
52.6
32–74
86.4
73–100
45
GPT-5.4 mini
23
65.2
47.8
34.8
100.0
86.7
60.0
33–80
81.8
64–95
39
GPT-5.6 sol
19
100.0
100.0
94.7
100.0
100.0
68.4
47–89
81.8
64–95
58
Appendix
Table 22: G2 robustness on the main set (full protocol, one seed, cards unknown to the backbone). Formation under stricter consensus thresholds; persistence of the shared name in the chat log (reused by at least two agents before the exit interview; reappearing in at least three later episodes); mechanism agreement with the human coinage among formed sessions with a session-level bootstrap 95% interval; G1-M mechanism match on the same words (22 of the 25 are G1 items); and mechanism match of a single-model concept-only coinage control (same concept text, no materials, no community).
A1-W
A1-WU
Slice
items
prop.
open
prop.
open
Provenance
Neologism reports
1671
35.7
12.3
51.4
31.6
Web API track
569
31.6
7.3
45.2
26.5
Online corpus track
162
37.3
9.2
59.4
33.2
Formation type
Appendix
Table 23: A1 accuracy (%, pre-cutoff items; proprietary and open-weight means) by lexicon provenance, formation type, verification confidence of web-collected entries, and number of attested usages in the lexicon. Slices with fewer than 20 items are omitted.
Large language models (LLMs) are typically evaluated on static benchmarks, even though natural language constantly evolves through newly emerging words and meanings. Existing Korean benchmarks are centered on established vocabulary and therefore provide limited coverage of such recent lexical change, and their English-oriented design makes it difficult to assess the typological properties of Korean, in which content words combine productively with functional morphemes. In this paper, we introduce KoNeoBench, a benchmark for evaluating LLMs' understanding of Korean neologisms. KoNeoBench is built on 1,785 Korean neologisms attested in online news since 2020 and curated through expert lexicographic review. Each entry provides usage examples, word-formation analyses, and dictionary-style definitions. Based on this resource, we define four tasks and report results on recent models, together with a human baseline. Our experiments show that current LLMs exhibit clear limitations in recovering source components, distinguishing semantic categories, and generating accurate definitions. These results reveal specific aspects of recent Korean lexical change that remain challenging for current LLMs. KoNeoBench is available at https://github.com/bcmilab/ko-neobench/ .
Soha Lee, Soojin Lee, Heesung Yang +8
School of Computer Science and Engineering, Kyungpook National University · International Exchange Department, Kyungpook National University · Dept. of Korean Language and Literature Education, Kyungpook National University +4
Chinese neologisms exploit diverse and unique linguistic mechanisms, such as phonetic substitution (e.g., 886 for ``bye-bye'') and visual character decomposition that are rare in other languages. We introduce CNeo-Bench, a benchmark of 4,759 such neologisms with reference definitions, organized into five top-level categories and nine subcategories by the linguistic mechanism behind each expression. CNeo-Bench is paired with a two-tier evaluation framework that separates whether a model can describe a neologism from whether it can operate on its underlying mechanism. Evaluating 18 LLMs, we find that Chinese neologisms remain an open challenge; most models fall below 40% on definition generation, and on several subcategories a systematic recognition-manipulation gap emerges: models describe neologisms correctly but, in source-form restoration tasks, substitute a semantic equivalent (paraphrase) for the source form rather than producing the source form itself. A few-shot analysis on 1,058 hard items shows that in-context examples can solve many difficult cases, but leave a noticeable portion of errors remaining, indicating challenges beyond prompting alone can address.
Kaiyan Zhao, Zhongtao Miao, Zheyong Xie +2
The University of Tokyo · Xiaohongshu Inc. · Tsinghua University
Although negation is known to challenge large language models (LLMs), benchmarks for evaluating negation understanding-especially in Korean-are scarce. We conduct a corpus-based analysis of Korean negation and show that LLM performance degrades under negation. We then introduce Thunder-KoNUBench, a sentence-level negation understanding benchmark that reflects the empirical distribution of Korean negation phenomena. Evaluating 47 LLMs on Thunder-KoNUBench, we analyze the effects of model size and instruction tuning, and perform error analysis to better understand model behavior. We further show that fine-tuning on Thunder-KoNUBench improves negation understanding and broader contextual comprehension in Korean.
Sungmok Jung, Yeonkyoung So, Joonhak Lee +3
Graduate School of Data Science, Seoul National University · Dept. of Computer Science and Engineering, Seoul National University