Not Just an Evaluation: A Case Study of Discourse Particle Sensitivity in Language Models

William Sheffield; Jiahui Chen; Amy Zhang; Alex Warstadt; Junyi Jessy Li; Kanishka Misra

cs.CLOct 9, 2026

Not Just an Evaluation: A Case Study of Discourse Particle Sensitivity in Language Models

Authors: William Sheffield, Jiahui Chen, Amy Zhang, Alex Warstadt, Junyi Jessy Li, Kanishka Misra

Organizations: The University of Texas at Austin · University of California, San Diego

Abstract

Discourse particles subtly shape the flow of discourse by reflecting conversational context and interlocutor attitudes, often leaving the truth-conditional meaning untouched. They are also often polyfunctional, with distinct senses which partially overlap with other particles; for example, the exclusive use of English "just" and "only" ("John only/just drinks water"). Despite the frequency of these words in natural language, they have gone largely unstudied due to the subtle nature of their effects. We investigate LLM sensitivity to "just" and "only" in natural conversations using behavioral and mechanistic analyses. First, we find that LLMs and humans similarly prefer the presence of these particles in context, and both are more sensitive to "just" compared to "only". Second, when sampling from LLMs with and without these particles, we find LLMs consistently rate generations as more likely, given their source context, demonstrating their particle sensitivity. In both experiments, instruction tuning increases sensitivity. Finally, we localize particle-sensitive units within LLMs and find the overlap between "just" and "only" is asymmetric: "only" units transfer selectively to exclusive "just", while all "just" units transfer to "only" equally well regardless of sense, suggesting LLM lexical-level particle representations are partially organized by sense. We release our code, sense-labeled dialogue stimuli, and human preference annotations at https://github.com/JiahuiKChen/Not_Just_an_Evaluation.

Figures & tables

Figure 1: An example stimulus from our data, involving the context

C

, target sentence with particle

S

and counterfactual sentence without particle

S^{\prime}

, ground-truth follow-up

F

, and LLM follow-ups generated with the particle

F^{*}_{CS}

and without

F^{*}_{CS^{\prime}}

Figure 2: Top: Z-scored LLM and Human preferences on the gold data (distractors excluded). LLM preference is defined in Eq. ( 1 ), Human preference is rated on a 7-pt Likert scale, with 7 being strongly prefer the original passage with the word (

CSF

). Stars indicate significance on a Wilcoxon signed-rank test with Bonferroni correction. Bottom: Spearman’s

\rho

between human and LLM ratings, reported per word as well as for all words.

Figure 3: Own-context advantage by model in the generation experiment. We construct paired prompt contexts

S

and

S^{\prime}

that differ only in the target word, sample next-turn responses from both prompts, and then rescore each sampled response under both contexts.

\Delta_{\mathrm{tok}}

is the mean per-token difference (Equation 2 ) Positive values indicate an own-context advantage.

	Context $C$ and utterance $S$ / $S^{\prime}$	Gold follow-up $F$	Sampled from $CS$ ( $F^{*}_{CS}$ )	Sampled from $CS^{\prime}$ ( $F^{*}_{CS^{\prime}}$ )
(a)	C: …I’m looking for a black, leather bag. / How’s this? S: I’m (just) looking. Human: 6.2	If you need any help, let me know.	Understood! No pressure.	I understand! I need a little more information to help you find the perfect black leather bag.
			Gemma-2 9B IT ; $\Delta_{\mathrm{tok}}$ : $0.75$ ( $CS$ ), $-0.03$ ( $CS^{\prime}$ )
(b)	C: Did you already wash your clothes today? S: I’ve (only) washed my white clothes. Human: 6.4	Could you wash some of my dark clothes with yours?	Oh, okay! Are you planning on washing the rest of your clothes later?	Good job!
			Gemma-2 9B IT ; $\Delta_{\mathrm{tok}}$ : $0.17$ ( $CS$ ), $0.51$ ( $CS^{\prime}$ )
(c)	C: Hello! Are you shopping here today? Haven’t I seen you in some flea markets and thrift shops a couple of times? S: I’m (just) comparing prices. Human: 5.9	You should! This thrift shop is really cheap, though, you have to pay cash here.	A savvy shopper! Comparing prices is a great way to make sure you’re getting the best deal. Are you looking for something in particular or just browsing?	A savvy shopper! Comparing prices is a great way to make sure you’re getting the best deal. Are you looking for something in particular or just browsing to see what catches your eye?
			Llama 3 8B Instruct ; $\Delta_{\mathrm{tok}}$ : $0.05$ ( $CS$ ), $0.02$ ( $CS^{\prime}$ )

Table 1: Follow-ups sampled from instruction-tuned LLMs for three gold items. The target word appears in parentheses in

S

and is removed in

S^{\prime}

; “/” separates turns in

C

. Human is the mean rating on the 7-point scale (7: strongly prefer

CSF

). Each sampled follow-up is selected for illustration from the 50 samples per history;

\Delta_{\mathrm{tok}}

is the LLM’s mean own-context advantage (Equation 2 ) over all 50 samples from each history.

		Localization: $\Delta_{\text{patch}}$ ( $\downarrow$ = stronger)						Transfer: $\Delta_{\text{sense}}$ ( $+=J_{E}$ stronger)
Component	top- $k$	not		just		only		just $\rightarrow$ only	only $\rightarrow$ just
Component	top- $k$	localized	random	localized	random	localized	random	just $\rightarrow$ only	only $\rightarrow$ just
	1	-0.47	-0.06	-0.22	-0.01	-0.34	-0.12	-0.36	0.01
	3	-2.15	-0.41	-1.09	-0.07	-2.10	-0.13	-0.78	0.18
	5	-6.59	-0.35	-1.72	-0.12	-3.29	-0.12	-0.60	0.52
Attn.	10	-15.88	-0.34	-2.98	-0.10	-5.51	-0.18	0.29	1.15

Table 2: Component-localization and cross-particle-transfer results (

\times 10^{-2}

, per token) for Llama 3 8B Instruct, by component and number of patched component–layer sites

k

. Left: mean change in log-probability under the necessity intervention,

\Delta_{\text{patch}}

. Localized sites are compared with the mean of four equally sized random-site sets. More negative values indicate stronger necessity effects. Boldface indicates a significantly larger necessity effect for localized than random sites (

p<.05

, Bonferroni correction). Right: the non-exclusive -minus- exclusive difference in necessity effects under localized-site patching,

\Delta_{\text{sense}}

, for each transfer direction. Positive values indicate a stronger effect for exclusive just , and negative values indicate a stronger effect for non-exclusive just . Boldface indicates a significant difference (two-tailed

p<.05

, Bonferroni correction).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Annotation interface. Annotators were tasked to rate which version of the conversation they preferred by using the slider.

Model	Part.	% inc.	Median $\Delta_{\mathrm{rem}}$	95% CI	N
Llama 3 8B	just	$69.7$	$0.057$	[0.050,0.064]	$1121$
	only	$73.8$	$0.071$	[0.061,0.084]	$347$
	not	$97.1$	$0.373$	[0.358,0.388]	$1269$
OLMo-2 7B	just	$72.0$	$0.060$	[0.054,0.069]	$1112$
	only	$77.1$	$0.098$	[0.079,0.118]	$349$
	not	$98.0$	$0.415$	[0.390,0.433]	$1273$

Appendix

Table 3: Sensitivity to particle removal, measured by the per-token removal delta

\Delta_{\mathrm{rem}}=\log P(CSF)-\log P(CS^{\prime}F)

. % inc. is the percentage of examples with positive

\Delta_{\mathrm{rem}}

N

is the number of non-outlier examples retained after IQR-based outlier filtering.

Model	Part.	% inc.	Median $\Delta_{\mathrm{rem}}$	95% CI	N
Llama 3 8B Instruct	just	$85.3$	$0.189$	[0.176,0.207]	$1112$
	only	$80.3$	$0.152$	[0.129,0.172]	$350$
	not	$97.3$	$0.528$	[0.502,0.559]	$1253$
OLMo-2 7B Instruct	just	$80.5$	$0.150$	[0.132,0.167]	$1136$
	only	$78.1$	$0.159$	[0.138,0.199]	$351$
	not	$96.7$	$0.536$	[0.511,0.555]	$1281$

Appendix

Table 4: Instruction-tuned LLM sensitivity to particle removal, measured by the per-token removal delta

\Delta_{\mathrm{rem}}=\log P(CSF)-\log P(CS^{\prime}F)

. % inc. is the percentage of examples with positive

\Delta_{\mathrm{rem}}

N

is the number of non-outlier examples retained after IQR filtering.

		not		just		only
		$S$	$S^{\prime}$	$S$	$S^{\prime}$	$S$	$S^{\prime}$
Llama3	Base	$0.0554\pm 0.00242$	$0.0598\pm 0.00334$	$0.0266\pm 0.00207$	$0.0164\pm 0.00165$	$0.0202\pm 0.00141$	$0.0161\pm 0.00124$
Llama3	Instruct	$0.207\pm 0.005$	$0.299\pm 0.00733$	$0.0297\pm 0.000966$	$0.0297\pm 0.00117$	$0.054\pm 0.00175$	$0.0481\pm 0.00145$
OLMo2	Base	$0.0402\pm 0.00365$	$0.084\pm 0.0044$	$0.0126\pm 0.00168$	$0.0224\pm 0.00181$	$0.0231\pm 0.00158$	$0.0197\pm 0.00152$
OLMo2	Instruct	$0.215\pm 0.00508$	$0.264\pm 0.00561$	$0.0392\pm 0.00214$	$0.0324\pm 0.00162$	$0.0405\pm 0.00185$	$0.044\pm 0.00168$
Gemma2	Base	$0.0599\pm 0.00222$	$0.0659\pm 0.00312$	$0.0166\pm 0.00106$	$0.0146\pm 0.00125$	$0.0189\pm 0.00158$	$0.0168\pm 0.00112$

Appendix

Table 5: Mean candidate-level own-context advantage per token (

\pm

standard error), by LLM, particle, and response source (

S

: with word,

S^{\prime}

: word removed.)

	Context $C$ and utterance $S$ / $S^{\prime}$	Gold follow-up $F$	Sampled from $CS$ ( $F^{*}_{CS}$ )	Sampled from $CS^{\prime}$ ( $F^{*}_{CS^{\prime}}$ )
(d)	C: Excuse me, here is my claim tag, but I can’t find my baggage. / How many pieces of baggage have you lost? S: (Just) one piece. Human: 4.2	Can you tell me the features of your baggage?	I’m sorry to hear that. Since you only have one piece missing, we can start by immediately locating it.	Okay, I have noted that one piece of baggage is missing.
			Qwen3.5 9B CT ; $\Delta_{\mathrm{tok}}$ : $0.01$ ( $CS$ ), $0.02$ ( $CS^{\prime}$ )
			Thank you for providing this information. If you have lost just one piece of baggage, here are the steps you should take:	If you have lost one piece of baggage, please follow these steps:
			OLMo-2 7B Instruct ; $\Delta_{\mathrm{tok}}$ : $-0.01$ ( $CS$ ), $0.03$ ( $CS^{\prime}$ )
(e)	C: Did you ever get a parking ticket? S: (Only) a few. Human: 4.1	What’s a few?	Only a few?	Oh no! Those are the worst. What were they for? Parking too long? In a restricted zone? Something else? …
			Gemma-2 9B IT ; $\Delta_{\mathrm{tok}}$ : $0.53$ ( $CS$ ), $-0.24$ ( $CS^{\prime}$ )

Appendix

Table 6: Additional follow-ups sampled from instruction-tuned LLMs, formatted as in Table 1 . Emoji in sampled follow-ups are omitted, and “…” marks a truncated follow-up.

Family	Size	Base checkpoint	Instruction-tuned checkpoint
Llama 3	8B	meta-llama/Meta-Llama-3-8B	meta-llama/Meta-Llama-3-8B-Instruct
OLMo-2	7B	allenai/OLMo-2-1124-7B	allenai/OLMo-2-1124-7B-Instruct
Qwen3.5	9B	Qwen/Qwen3.5-9B-Base	Qwen/Qwen3.5-9B
Gemma-2	9B	google/gemma-2-9b	google/gemma-2-9b-it

Appendix

Table 7: Model checkpoints used in Experiments 1 and 2.

Family	Architecture	Pretraining	Post-training	Open data
Llama 3 8B	Dense decoder-only Transformer; grouped-query attention	Over 15T tokens from publicly available sources	SFT and RLHF	No
OLMo-2 7B	Dense decoder-only Transformer; multi-head attention	Two stages: 3.90T tokens of general pretraining, then a curated mid-training mix (4.05T total)	Tülu 3 recipe: SFT, DPO, and RL with verifiable rewards	Yes
Gemma-2 9B	Dense decoder-only Transformer; grouped-query attention, alternating local sliding-window and global attention layers	8T tokens, trained with knowledge distillation from a larger teacher instead of next-token prediction	SFT (largely on teacher-generated responses), RLHF with a conversation-oriented reward model, and model merging	No
Qwen3.5 9B	Hybrid: 24 Gated DeltaNet (linear attention) and 8 Gated Attention layers; vision encoder	Early-fusion multimodal pretraining on “trillions” of tokens; exact count not reported	Large-scale RL across agentic environments; details not reported	No

Appendix

Table 8: Documented differences between the model families in Experiments 1 and 2, drawn from AI@Meta (2024) , OLMo et al. (2024) , Team et al. (2024) , and Qwen Team (2026) ; layer counts are from the released checkpoint configurations. For Llama 3 we follow the model card of the original release rather than Grattafiori et al. (2024) , whose results are for the later Llama 3.1 models.

Explore similar work

Can Large Language Models Handle Discourse Particles? A Case Study of Colloquial Malay
May 27, 2026Mariah Al Giptiah Binte Yusoff, Jakin Tan, Bocheng Chen +2Multilingual Language Model Evaluation Pragmatic Reasoning in Language Models
Nanyang Technological University · University of Mississippi · Indiana University
Focus particles and scalar inferences across humans and language models
Aug 8, 2026Catherine M. Brousse, Nelu D. RadpourLLM Evaluation Pragmatic Reasoning in Language Models
Department of Psychology, Florida State University
It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief
Jul 20, 2026Kevin Du, Clara Kümpel, Michelle Wastl +1LLM Evaluation Knowledge Conflicts in Language Models
ETH Zürich · University of Zurich · UC San Diego

May 27, 2026, cs.CLJ/K move · Enter open · S save

Can Large Language Models Handle Discourse Particles? A Case Study of Colloquial Malay

Mariah Al Giptiah Binte Yusoff, Jakin Tan, Bocheng Chen, Guangliang Liu+1

Nanyang Technological University · University of Mississippi · Indiana University

Multilingual Language Model Evaluation Pragmatic Reasoning in Language Models Language Modeling Low-Resource Language Processing

Open paper AskPDF arXiv

Discourse particles, such as well and kind of, are crucial components that enable LLMs to "speak" more like humans. They are used to convey emotions, intentions, and interpersonal attitudes. However, existing studies have not yet built a comprehensive understanding of LLMs' capabilities in handling discourse particles. Moreover, the limited number of research focuses primarily on high-resource languages such as English, with little attention paid to Southeast Asian languages. In this paper, we (1) propose MalayPrag, a benchmark designed to systematically evaluate and analyze LLMs' capabilities in handling discourse particles in colloquial Malay; (2) introduce five attributes that provide a theoretically grounded, unified framework for interpreting pragmatic functions of discourse particles. Applying these two, we prompt ten off-the-shelf LLMs to perform three prediction tasks. The experimental results reveal substantial challenges for current LLMs to accurately connect discourse particles and their pragmatic functions in Malay. The provision of the five attributes designed in this study is found to significantly improve the connections, highlighting the need for structured scaffolding for models' pragmatic competence.

Abstract

Explore similar work

Can Large Language Models Handle Discourse Particles? A Case Study of Colloquial Malay

Focus particles and scalar inferences across humans and language models

It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief