Large language models (LLMs) achieve impressive results in terms of fluency in text generation, yet the nature of their linguistic knowledge - in particular the human-likeness of their internal lexicon - remains uncertain. This study compares human and LLM-generated word associations to evaluate how accurately models capture human lexical patterns. Using English cue-response pairs from the SWOW dataset and newly generated associations from three LLMs (Mistral-7B, Llama-3.1-8B, and Qwen-2.5-32B) across multiple temperature settings, we examine (i) the influence of lexical factors such as word frequency and concreteness on cue-response pairs, and (ii) the variability and typicality of LLM responses compared to human responses. Results show that all models mirror human trends for frequency and concreteness but differ in response variability and typicality. Larger models such as Qwen tend to emulate a single "prototypical" human participant, generating highly typical but minimally variable responses, while smaller models such as Mistral and Llama produce more variable yet less typical responses. Temperature settings further influence this trade-off, with higher values increasing variability but decreasing typicality. These findings highlight both the similarities and differences between human and LLM lexicons, emphasizing the need to account for model size and temperature when probing LLM lexical representations.
Figures & tables
Figure 1: Distribution of log-transformed response/cue frequency ratios across respondents, considering responses provided by humans and LLMs at Rank 1, with LLM temperature set to 1.
Figure 2: Average relative frequency of R1 responses across cue frequency bins (LLM temperature = 1). The number of cues included in each bin is indicated below or above the bars.
Figure 3: Distribution of response/cue concreteness ratios across respondents, considering responses provided by humans and LLMs at Rank 1, with LLM temperature set to 1.
Figure 4: Average relative concreteness of R1 responses across cue concreteness bins (LLM temperature = 1). The number of cues included in each bin is indicated above the bars.
Variability
Typicality
Resp.
Temp.
avg#R1
tot#R1
tok-SS1
typ-SS1
Human
-
38.4 {38.6}
9,175 {56,625}
1.67 (0.60) {1.66 (0.57)}
0.00 (0.00)
Mistral
0.3
2.9
963
1.62 (1.89)
1.36 (1.81)
1.0
14.5
3,801
1.50 (1.38)
0.14 (0.55)
1.5
24.6
6,577
0.97 (1.15)
-0.18 (0.32)
2.0
33.8
9,749
0.63 (0.87)
-0.34 (0.24)
Table 1: Measures of variability ( avg#R1 : average number of unique R1 per cue; tot#R1 : total number of unique R1 responses) and typicality ( tok-SS1 and typ-SS1 , averaged over the 490 cues) for all respondents at different temperatures. Standard deviations are indicated in parentheses. Results for the entire set of SWOW-EN cues are presented in curly brackets.
Figure 5: Distribution of variability ( #R1 ) and typicality ( tok-SS1 ) in human and LLM datasets at temperature 1. Grey boxplots (right y-axis) represent variability, while blue boxplots (left y-axis) represent typicality.
Figure 6: Variability ( #R1 ) vs. typicality ( tok-SS1 ), both averaged over the 490 cues, for all respondents and temperatures. Temperatures are indicated as dot labels.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Mistral: log relative frequency across temperatures w.r.t. humans.
Figure 8: Llama: log relative frequency across temperatures w.r.t. humans.
Figure 9: Qwen: log relative frequency across temperatures w.r.t. humans.
Model
F
p
η2
Mistral
449.82
.000
.035
Llama
64.54
.000
.007
Qwen
10.21
.000
.009
Appendix
Table 2: ANOVA results for the effect of temperature on relative frequencies.
Figure 10: Mistral: relative concreteness across temperatures w.r.t. humans.
Figure 11: Llama: relative concreteness across temperatures w.r.t. humans.
Figure 12: Qwen: relative concreteness across temperatures w.r.t. humans.
Model
F
p
η2
Mistral
26.17
.000
.003
Llama
2.26
.079
-
Qwen
.06
.980
-
Appendix
Table 3: ANOVA results for the effect of temperature on relative concreteness.
Systematicity, the probabilistic mapping of form to meaning, permeates language at all levels, and sublexical cues have been shown to govern human pseudoword processing. Yet whether LLMs exhibit comparable sensitivity to these cues remains unclear. We tested five LLMs on two Italian two-alternative forced-choice pseudoword experiments and compared their responses with a human behavioural baseline. LLMs aligned more reliably with humans when real-word options provided a lexical familiarity cue than in the pseudoword-only condition, where they fell substantially below fastText, a character-n-gram model. In addition, the sublexical cosine-similarity cue that reliably drove human--fastText agreement did not consistently transfer to human--LLM alignment, and reasoning-token expenditure bore no consistent relation to human processing difficulty. These findings suggest that LLMs do not necessarily share the sublexical cues that govern human pseudoword processing; we discuss tokenization and training-data coverage as candidate explanations.
Jing Chen, Giulia Loca, Simona Amenta +1
Department of Psychology, University of Milano-Bicocca, Piazza dell’Ateneo Nuovo 1, 20126 Milano, Italy · Department of Informatics, Systems and Communication – DISCo, University of Milano-Bicocca, Milano, Italy
Large language models (LLMs) often produce judgments similar to those of humans, but their interpretation of figurative language remains unclear. We compared human participants and four instruction-tuned LLMs (GPT-4, Gemma-2-9B, Llama-3.2, and Mistral-7B) on their ratings of 240 dialogue-based sentences spanning six linguistic traits: conventionality, sarcasm, funny, emotion, idiomacy, and Gen Z slang. While humans and LLMs were strongly aligned at the surface level, differences emerged in representational similarity patterns, particularly in idiomacy and Gen Z slang. GPT-4 showed the closest alignment to humans, though all models differed at the representational level, suggesting that similar outputs may arise from different underlying language representations. We also examined the relationship between model scale and representational similarity using the Gemma model series (Gemma-2-9B-IT, Gemma-3-27B-IT, and Gemma-4-31B-IT), and found that representational similarity patterns were largely unaffected by model size.
While factual correctness and task-performance have been in focus of Large Language Model (LLM) research for a long time, the fundamental question of how human-like generated texts are on a linguistic level has been underexplored. From a corpus-linguistic perspective, language production is inherently context-dependent, with distinct communicative contexts giving rise to differences in frequencies and co-occurrence patterns of linguistic features. A text failing to adhere to these patterns can be content-wise correct, but still be unfavorable to human readers. In this work, we propose a context-aware evaluation framework in which human-likeness is assessed using a two-sample problem between the linguistic feature distribution of a human reference corpus for a given register and a corresponding LLM-generated corpus. We implement this framework using the Maximum Mean Discrepancy (MMD) and the 67 lexico-grammatical features introduced by Biber, which are commonly applied in corpus linguistics. In our experiments, we compare seven instruction-tuned, open-source models across five English-language datasets spanning distinct registers against a human baseline. While across all tested setups, LLMs deviate from the human baseline, which models are closest to human language depends on the register and is not dictated by model size.
Björn Nieth, Marianna Gracheva, Michaela Mahlberg +2
Department Artificial Intelligence in Biomedical Engineering (AIBE), FAU Erlangen-Nürnberg, Germany · Department of Digital Humanities and Social Studies (DHSS), FAU Erlangen-Nürnberg, Germany · University of Birmingham, United Kingdom +2