Pseudowords as probes: Large Language Models show little of the sublexical sensitivity that governs human pseudoword processing

Jing Chen; Giulia Loca; Simona Amenta; Marco Marelli

cs.CLOct 6, 2026

Pseudowords as probes: Large Language Models show little of the sublexical sensitivity that governs human pseudoword processing

Authors: Jing Chen, Giulia Loca, Simona Amenta, Marco Marelli

Organizations: Department of Psychology, University of Milano-Bicocca, Piazza dell’Ateneo Nuovo 1, 20126 Milano, Italy · Department of Informatics, Systems and Communication – DISCo, University of Milano-Bicocca, Milano, Italy

Abstract

Systematicity, the probabilistic mapping of form to meaning, permeates language at all levels, and sublexical cues have been shown to govern human pseudoword processing. Yet whether LLMs exhibit comparable sensitivity to these cues remains unclear. We tested five LLMs on two Italian two-alternative forced-choice pseudoword experiments and compared their responses with a human behavioural baseline. LLMs aligned more reliably with humans when real-word options provided a lexical familiarity cue than in the pseudoword-only condition, where they fell substantially below fastText, a character-n-gram model. In addition, the sublexical cosine-similarity cue that reliably drove human--fastText agreement did not consistently transfer to human--LLM alignment, and reasoning-token expenditure bore no consistent relation to human processing difficulty. These findings suggest that LLMs do not necessarily share the sublexical cues that govern human pseudoword processing; we discuss tokenization and training-data coverage as candidate explanations.

Figures & tables

Figure 1: Distribution of item-level response entropy (

H_{i}

, bits) across the 50 items in each experiment. Coloured vertical lines mark the per-experiment mean (

\bar{H}_{i}

); the dashed grey line marks the theoretical maximum (

H=1

bit).

Figure 2: Split-half reliability across participants. Each panel shows the distribution of Spearman-Brown-corrected split-half correlations (

\rho

) across 1,000 random participant splits for EXP1 and EXP2. Dark bars show observed

\rho

values; light bars show the shuffled null distribution, in which response labels were permuted before splitting.

Model	Prompt	EXP1	EXP2
GPT-4o	Baseline	32%	36%
	Role	60%	44%
DeepSeek-V3	Baseline	52%	40%
	Role	42%	40%
DeepSeek-R1	Baseline	52%	42%
	Role	40%	32%

Table 1: Within-model response consistency: proportion of items receiving unanimous responses across three iterations.

	EXP 1						EXP 2
	Base			Role			Base			Role
Model	$N$	%	$p$	$N$	%	$p$	$N$	%	$p$	$N$	%	$p$
GPT-4o	2,975	50.2	.947	2,975	53.8	.268	2,997	56.4	.055	2,997	59.7	.002
DeepSeek-V3	2,975	54.3	.203	2,975	52.2	.520	2,877	58.9	.007	2,997	55.5	.099
DeepSeek-R1	2,975	48.5	.665	2,975	52.8	.405	2,997	55.5	.102	2,997	58.3	.009
o4-mini	2,975	56.1	.066	2,975	54.2	.213	2,997	57.8	.016	2,997	58.5	.008

Table 2: Trial-level human-model alignment estimated from intercept-only GLMMs.

N

= number of trials entering each fit; values are back-transformed alignment probabilities (%). Reduced

N

for Minerva-7B and DeepSeek-V3 (EXP2 Base) reflects exclusion of trials with invalid responses. Bold =

p<.05

		EXP 1		EXP 2
Model	Prompt	%	$p$	%	$p$
GPT-4o	Base	52.8	.479	58.6	.022
	Role	50.1	.980	60.8	.003
DeepSeek-V3	Base	50.7	.869	60.1	.004
	Role	47.4	.634	56.1	.116
DeepSeek-R1	Base	46.7	.393	56.4	.091

Table 3: Human-model alignment on the shared valid item set (items with valid responses from all five models). EXP1:

N_{\text{items}}=35

N=2{,}082

trials; EXP2:

N_{\text{items}}=39

N=2{,}337

trials. Values are back-transformed alignment probabilities (%). Bold =

p<.05

EXP 1
	fastText		DeepSeek-R1		DeepSeek-V3		GPT-4o		Minerva-7B		o4-mini
	$\hat{\beta}$	$p$	$\hat{\beta}$	$p$	$\hat{\beta}$	$p$	$\hat{\beta}$	$p$	$\hat{\beta}$	$p$	$\hat{\beta}$	$p$
$D_{\text{cos}}$	0.20	.041	$-0.08$	$.602$	$\phantom{-}0.12$	$.480$	$-0.17$	$.282$	$\phantom{-}0.14$	$.395$	$-0.13$	$.355$
$D_{L}$	$\phantom{-}0.31$	$.057$	$-0.01$	$.952$	$-0.01$	$.983$	$\phantom{-}0.11$	$.656$	$-0.10$	$.708$	$\phantom{-}0.34$	$.146$
$D_{\text{LD}}$	$-0.32$	$.054$	$\phantom{-}0.12$	$.594$	$-0.15$	$.624$	$-0.12$	$.661$	$\phantom{-}0.13$	$.661$	$-0.26$	$.285$
EXP 2

Table 4: Item-level sublexical predictors of human–model alignment. LLM estimates pool across prompt conditions. Bold =

p<.05

; all predictors

z

-scored.

EXP 1
Model	$r\ (p)$	$r_{\text{part}}\ (p)$
DeepSeek-R1 base	$-.10$ (.494)	$-.12$ (.424)
DeepSeek-R1 role	$\phantom{-}.21$ (.147)	$\phantom{-}.20$ (.170)
o4-mini base	.28 ( .046 )	.29 ( .039 )
o4-mini role	$\phantom{-}.23$ (.116)	$\phantom{-}.26$ (.065)
EXP 2

Table 5: Pearson correlations between LLM reasoning tokens and mean item-level human response time.

Figure 3: Association between model reasoning tokens and human response entropy in EXP1 (top) and EXP2 (bottom). Each point represents one of the 50 items, with the x-axis showing log mean reasoning tokens and the y-axis showing human response entropy (

H_{i}

). Lines show best-fitting linear regression for each model and prompt condition.

Explore similar work

Production and Perception in LLMs: A Token Probability Approach
Jul 13, 2026Anna Marklová, Jiří Milička, Martina Vokáčová +1Single-Token Output Distributions Linguistics
Faculty of Arts, Charles University, Prague · Faculty of Mathematics and Physics, Charles University, Prague
Tracing the ongoing emergence of human-like reasoning in Large Language Models
May 20, 2026Paolo Morosi, Nikoleta Pantelidou, Fritz Günther +2LLM Reasoning Strategies Emergence
Departament de Filologia Catalana, Universitat Autònoma de Barcelona, Barcelona 08193, Spain. · Institut für Psychologie, Humboldt-Universitat zu Berlin, Berlin 10099, Germany. · Institució Catalana de Recerca i Estudis Avançats (ICREA), Barcelona 08010, Spain
Dual Alignment Between Language Model Layers and Human Sentence Processing
Apr 20, 2026Tatsuki Kuribayashi, Alex Warstadt, Yohei Oseki +1Natural Language Surprisal
MBZUAI · Tohoku University · UC San Diego +2

Jul 13, 2026, cs.CLJ/K move · Enter open · S save

Production and Perception in LLMs: A Token Probability Approach

Anna Marklová, Jiří Milička, Martina Vokáčová, Rudolf Rosa

Faculty of Arts, Charles University, Prague · Faculty of Mathematics and Physics, Charles University, Prague

Single-Token Output Distributions Linguistics Perceptually Token-Level Uncertainty Perception Probability

Open paper AskPDF arXiv

The asymmetry between language production and perception has been well-documented in psycholinguistics. Whether large language models (LLMs) exhibit a functionally analogous distinction remains an open question, particularly given that LLMs rely on the same underlying mechanism (next-token prediction) for both input and output processing. In this exploratory study, we operationalize the production-perception distinction through direct token probability measurements rather than metalinguistic prompting. Using the base Llama-3.1-8B model, we generated poems under a production prompt and re-scored the same tokens under both rephrased production prompts and perception-oriented prompts. Across an extended experiment with four production and three perception prompts, production-perception distances consistently and substantially exceeded production-production distances, with non-overlapping ranges across conditions and an overall average ratio of approximately 1.8. Near-ceiling correlations in the production-production control confirm that the effect is specific to communicative framing rather than prompt surface variation, and we show the effect replicates across five open-weight models (Llama-3.1-8B, EuroLLM-9B, gemma-2-9b-it, Mistral-7B-Instruct-v0.3, and Qwen2.5-7B-Instruct), spanning both base and instruction-tuned variants. Temporal analysis revealed that the perception prompt exerts its strongest influence at the beginning of the sequence, with divergence decaying as generated context accumulates, though the specific shape of this decay varies across prompt pairs. These findings suggest that prompt framing alone induces a production-perception distinction in LLM probability distributions, even within a decoder-only architecture.

Abstract

Explore similar work

Production and Perception in LLMs: A Token Probability Approach

Tracing the ongoing emergence of human-like reasoning in Large Language Models

Dual Alignment Between Language Model Layers and Human Sentence Processing