Semantic Variability of Replies Across LLMs: Implications for Designing Conversation-Based Assessment

Jiangang Hao

cs.CLAug 13, 2026

Semantic Variability of Replies Across LLMs: Implications for Designing Conversation-Based Assessment

Authors: Jiangang Hao

Organizations: ETS Research Institute Princeton, New Jersey, USA

Abstract

This study examines whether LLM-generated replies remain semantically consistent when the underlying LLM changes. Using messages from real collaborative conversations, we compared the semantic similarity of generated replies across LLMs under two conditions: with and without preceding chat history. Results show that model choice and conversational context both affect response similarity and alignment with human replies. These findings indicate that prompting and conversational context alone may not be sufficient to preserve response consistency across LLMs, highlighting the need for infrastructure and design strategies that can maintain stable and comparable responses amid the rapid and continuous evolution of LLMs.

Figures & tables

Figure 1: Semantic similarity by model and context condition.

Figure 2: Average within- and between-LLM similarity by context condition.

Figure 3: Relationship between similarity of focal messages and that of LLM generated replies by model and context condition.

Figure 4: Distribution of similarity from messages generated with and without chat history.

Term	Estimate	Std. Error	$z$	$p$	Sig.
(Intercept)	0.7267	0.0150	48.485	$<.001$
GPT-5.4	0.0226	0.0152	1.492	0.136
GPT-5.4 mini	-0.0014	0.0150	-0.095	0.924
GPT-5.4 nano	0.0054	0.0150	0.359	0.720
With History	-0.0415	0.0151	-2.748	0.006	**
GPT-5.4 $\times$ With History	0.0534	0.0212	2.523	0.012

Table 1: Mixed-effects results for mean pairwise similarity among LLM-generated replies. Reference categories are GPT-4o mini and No History. Significance codes: ***

p<.001

, **

p<.01

, *

p<.05

, .

p<.10

Term	Estimate	Std. Error	$z$	$p$	Sig.
(Intercept)	0.1224	0.0068	17.926	$<.001$
GPT-5.4	0.0202	0.0064	3.159	0.002	**
GPT-5.4 mini	0.0084	0.0063	1.328	0.184
GPT-5.4 nano	-0.0372	0.0063	-5.913	$<.001$
With History	0.0162	0.0064	2.553	0.011
GPT-5.4 $\times$ With History	-0.0169	0.0089	-1.902	0.057	.

Table 2: Mixed-effects results for variation in pairwise similarity among LLM-generated replies. Reference categories are GPT-4o mini and No History. Significance codes: ***

p<.001

, **

p<.01

, *

p<.05

, .

p<.10

Explore similar work

Paraphrase-Induced Output-Mode Collapse: When LLMs Break Character Under Semantically Equivalent Inputs
May 6, 2026Aofan Liu, Jingxiang MengPrompt Sensitivity LLM Evaluation
1Peking University. · University of Chicago.
Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility
Aug 10, 2026Siyang Wu, Yibo Jiang, Bryon AragamPrompt Sensitivity LLM Evaluation
Data Science Institute University of Chicago Chicago, USA · Department of Computer Science University of Chicago Chicago, USA · Booth School of Business University of Chicago Chicago, USA
Post-Training Recipe, More Than Model Family, Shapes Multi-Agent LLM Conversational Behavior
May 30, 2026Luyang Zhang, Jialu Wang, Fei Xue +1Language Model Post-Training Multi-LLM Collaboration
Carnegie Mellon University · University of California, Santa Cruz · Independent Researcher

May 6, 2026J/K move · Enter open · S save

Paraphrase-Induced Output-Mode Collapse: When LLMs Break Character Under Semantically Equivalent Inputs

Aofan Liu, Jingxiang Meng

1Peking University. · University of Chicago.

Prompt Sensitivity LLM Evaluation LLM Reliability Structured Output Generation Language Model Robustness

Open paper AskPDF arXiv

When the substantive content of a request is rewritten, do large language models still answer in the format the original task asked for? We find that they often do not, even at temperature zero. On a 150-query evaluation over five compact 2025-era LLMs and four task types, we observe a systematic failure mode we call prompt-variant output-mode collapse: when a closed-form prompt asks for a bare label or a single choice token, content-preserving prompt variants can push the model into conversational prose, the requested format dissolves, and exact-match evaluation pipelines silently misjudge the result. To make this measurable, we release PARACONSIST, a 900-prompt benchmark of 150 base queries with five lexical, syntactic, and semantic-expansion prompt variants each, and a Semantic Consistency Score that decomposes prompt-variant robustness into answer consistency, sentence-BERT semantic similarity, and length stability. Under a whole-word answer-set match, only ~22% of closed-form variant responses preserve the ground-truth label inside their output, while ~78% drift away from the answer space entirely. In our pool, the dominant predictor of collapse is task structure rather than model identity, with model differentiation jointly carried by answer consistency and length stability. Robustness audits should therefore track response-mode preservation as a first-class reliability target alongside answer accuracy.

Abstract

Explore similar work

Paraphrase-Induced Output-Mode Collapse: When LLMs Break Character Under Semantically Equivalent Inputs

Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility

Post-Training Recipe, More Than Model Family, Shapes Multi-Agent LLM Conversational Behavior