Expectation, Backlash, Recovery, and Excitement: How Model Releases Shape Reddit Perceptions of Conversational AI Systems

Vahid Rahimzadeh; Yury Zhauniarovich; Savvas Zannettou

cs.CLAug 25, 2026

Expectation, Backlash, Recovery, and Excitement: How Model Releases Shape Reddit Perceptions of Conversational AI Systems

Authors: Vahid Rahimzadeh, Yury Zhauniarovich, Savvas Zannettou

Organizations: Delft University of Technology

Abstract

Conversational AI systems (CAISes) continuously change through model releases, feature updates, safety interventions, and access-policy shifts, yet user perceptions are often studied as static snapshots. We conduct a long-term, large-scale analysis of Reddit discussions to examine how users perceive CAIS model release interventions across providers. By combining sentiment classification and thematic concept analysis, we show that CAIS perceptions are dynamic and intervention-sensitive. Anthropic exhibits the clearest positive release profile through Claude Code and product-model fit, OpenAI shows backlash-and-recovery dynamics around GPT-5 and GPT-5.1, Grok-3 is shaped by provider identity and political discourse, and DeepSeek-R1 combines engineering praise with concerns about censorship, access, and reliability. These findings show that model releases are not merely technical updates, but user-facing interventions that reshape sentiment, expectations, and public discussion.

Figures & tables

Figure 1: Study overview of sentiment and thematic shifts around model releases.

Figure 2: Overview of our analysis pipeline. We ground model mentions in Reddit posts, measure post-level sentiment and perception concepts, and compare sentiment and concept prevalence in provider-specific windows around model releases.

Provider	#Posts	%	Provider	#Posts	%
OpenAI	455,045	68.1	DeepSeek	9,982	1.5
Meta	68,282	10.2	Mistral	1,994	0.3
Google	60,475	9.1	Alibaba	976	0.1
Anthropic	43,271	6.5	Moonshot	223	0.0
xAI	27,815	4.2	Total	668,063	100.0

Table 1: Number of posts per provider in our data.

Provider	#Mentions (%)	#Posts (%)
OpenAI	316,410 (62.6)	252,719 (69.5)
Google	61,793 (12.2)	51,854 (14.3)
Anthropic	55,652 (11.0)	44,020 (12.1)
xAI	20,075 (4.0)	18,418 (5.1)
Meta	17,380 (3.4)	14,418 (4.0)
DeepSeek	16,493 (3.3)	14,605 (4.0)

Table 2: Distribution of extracted mentions and posts.

Figure 3: Weekly sentiment shares in Reddit posts mentioning a single CAIS for the three most-discussed providers. Asterisks mark FDR-corrected significance of each change (

p<.05

p<.01

p<.001

Figure 4: Release-window changes in positive and negative sentiment for each model. Points show post-minus-pre sentiment changes; stars denote provider averages. The inset magnifies the dense central region (

\pm 5

pp). Asterisks on labels mark a statistically significant shift in the sentiment distribution (

p<.05

p<.01

p<.001

Figure 5: GPT-4o, GPT-5 and GPT-5.1 concept prevelance changes around releases. Asterisks mark FDR-corrected significance of each change (

p<.05

p<.01

p<.001

Figure 6: Claude-4, Deepseek-R1 and Grok-3 concept prevelance changes around releases.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Provider	Subreddits
OpenAI	r/ChatGPT , r/OpenAI
Anthropic	r/ClaudeAI , r/Anthropic , r/AnthropicAi
Google	r/GeminiAI , r/GoogleGeminiAI , r/GoogleGemini , r/Bard , r/GoogleBard
DeepSeek	r/DeepSeek
xAI	r/grok , r/GrokAI , r/xai , r/xAI_community
Meta	r/MetaAI , r/LocalLLaMA

Appendix

Table 3: Identified subreddits per conversational AI provider.

Source	URL	Use
Reddit dumps	https://academictorrents.com/details/3d426c47c767d40f82c7ef0f47c3acacedd2bf44	Collection of Reddit submissions
LLM Arena leaderboard	https://lmarena.ai/leaderboard/text	Identification of providers and CAISes

Appendix

Table 4: External data sources used in the dataset construction.

Figure 7: Pairwise Jaccard similarity between providers computed over post-level co-mentions ( $|A\cap B|/|A\cup B|$ ). Row labels show each provider’s total post count $n$ . Higher values indicate a larger share of posts mentioning both providers relative to either alone.

Figure 8: Monthly provider attention share (%) across the top-5 providers (top panel) and cross-provider co-mention rate (bottom panel).

Figure 9: Monthly solo-mention rate per provider: the fraction of a provider’s posts that name only that provider, smoothed with a 3-month rolling mean. Declining rates indicate increasing comparative discussion.

Figure 10: Distribution of number of concepts per post.

Figure 11: Weekly prevalence of the top-10 concepts (stacked area, left axis) and total discussion volume (line, right axis) across the full observation window, using the single-mention / no-link filtered corpus (

n=185{,}181

posts). The bottom panel shows total weekly posts (grey) against posts covered by at least one top-10 concept (red).

Figure 12: Concept names and inclusion criteria used for the selected top-5 intervention plots.

Model	Normalized tuple
Qwen3 Max Preview	alibaba / qwen / qwen-3 / max
Qwen3 235B Instruct	alibaba / qwen / qwen-3 / 235b
Qwen2.5 Max	alibaba / qwen / qwen-2-5 / max
Claude Sonnet 4.5	anthropic / claude / claude-4-5 / sonnet
Claude Opus 4.1	anthropic / claude / claude-4-1 / opus
Claude 3 Haiku	anthropic / claude / claude-3 / haiku

Appendix

Table 5: Examples of model taxonomy normalization.

A. Model Mention Extraction
Temperature	0.0
Max tokens	3,096
Timeout	180 s
Max retries	5
Concurrent workers	128
B. Concept Scoring

Appendix

Table 6: Hyperparameters and experimental setups for concept generation and model mention extraction.

Taxonomy Level	Acc.	Prec.	Rec.	Weighted F1
Provider	1.000	1.000	1.000	1.000
Family	0.958	0.901	0.956	0.956
Generation	0.960	0.763	0.967	0.967
Tier	0.980	0.773	0.976	0.976

Appendix

Table 7: Mention mapping to taxonomy 4-tuples performance. We use Weighted F1 score due to class imbalance across taxonomy levels.

Figure 14: LLooM pipeline prompts (Stages 1–3). Stage 2 produces each concept’s inclusion criterion ( prompt field), reused verbatim as the scoring rubric in Stage 3. Answers A or B (score

>0.5

) are treated as relevant.

Source	Positive	Neutral	Negative
Initial sample	50	50	50
Final majority labels	26	82	42

Appendix

Table 8: Class distribution before and after manual annotation.

Label	Precision	Recall	F1	Support
Negative	0.784	0.952	0.860	42
Neutral	0.938	0.744	0.830	82
Positive	0.647	0.846	0.733	26
Macro avg.	0.790	0.847	0.808	150
Weighted avg.	0.845	0.820	0.822	150

Appendix

Table 9: Classification report for the final LLM sentiment classifier against majority-vote human labels.

Figure 17: Distribution of day gaps between consecutive model releases for each provider in the release taxonomy. These distributions are used to define provider-specific intervention windows.

Provider	Window size (days)
OpenAI	46
Anthropic	55
Google	104
xAI	131
DeepSeek	25
Meta	64

Appendix

Table 10: Provider-specific intervention windows used in the release-level analyses.

Figure 18: Distribution of LLM sentiment by provider and model generation in the filtered single-mention, usable-selftext dataset.

Figure 19: Distribution of LLM sentiment by provider in the filtered single-mention, usable-selftext dataset.

Provider	Sentiment	Weeks	Slope (pp/week)	Kendall’s $\tau$	$p$ -value	Total change (pp)
OpenAI	Positive	162	-0.008	-0.061	0.652	-1.3
OpenAI	Neutral	162	-0.117	-0.664	$<0.001$	-18.9
OpenAI	Negative	162	0.117	0.587	$<0.001$	18.8
Anthropic	Positive	100	0.111	0.441	$<0.001$	12.2
Anthropic	Neutral	100	0.012	0.038	0.646	1.3
Anthropic	Negative	100	-0.127	-0.353	$<0.001$	-14.0

Appendix

Table 11: Robust Theil–Sen sentiment trend estimates by provider. Slopes are reported in percentage points per month. Total change is the implied percentage-point change over the provider-specific observation window.

Figure 20: Weekly sentiment shares in Reddit posts mentioning a single Meta LLM system. Lines show the share of posts classified as positive, neutral, or negative; blue shaded volume indicates weekly post counts. Sentiment shares are shown only for weeks with at least 10 posts. Vertical dotted lines mark model release dates.

Figure 21: Weekly sentiment shares in Reddit posts mentioning a single xAI LLM system. Lines show the share of posts classified as positive, neutral, or negative; blue shaded volume indicates weekly post counts. Sentiment shares are shown only for weeks with at least 10 posts. Vertical dotted lines mark model release dates.

Figure 22: Weekly sentiment shares in Reddit posts mentioning a single DeepSeek LLM system. Lines show the share of posts classified as positive, neutral, or negative; blue shaded volume indicates weekly post counts. Sentiment shares are shown only for weeks with at least 10 posts. Vertical dotted lines mark model release dates.

Provider	Release	$n_{\text{pre}}$	$n_{\text{post}}$	$\chi^{2}$	$p$ (omnibus)	$\Delta$ positive (pp)	$\Delta$ negative (pp)
OpenAI	GPT-3.5	47	3,535	14.0	$<$ .001 ∗∗∗	+6.5	+19.7 ∗∗
OpenAI	GPT-4	5,460	6,725	26.7	$<$ .001 ∗∗∗	-0.9	-3.6 ∗∗∗
OpenAI	GPT-4o	2,264	3,879	3.8	.153	-1.8	+1.6
OpenAI	o1	2,166	4,061	13.1	.001 ∗∗	+3.9 ∗∗∗	-1.9
OpenAI	GPT-4.5	5,019	6,770	21.5	$<$ .001 ∗∗∗	+3.5 ∗∗∗	-1.6
OpenAI	GPT-4.1	6,770	9,964	11.9	.003 ∗∗	+1.5 ∗	-2.5 ∗∗∗

Appendix

Table 12: Sentiment-shift significance for all 35 releases.

\chi^{2}

and

p

(omnibus) come from a chi-squared test on the window

\times

sentiment table; per-category deltas are tested with two-proportion

z

-tests. † marks releases whose smallest expected cell count is below five, where we report a Monte-Carlo exact

p

-value rather than the asymptotic one. Releases analyzed in Section 5 are shown in bold.

{}^{*}p<.05

{}^{**}p<.01

{}^{***}p<.001

Provider	Top subreddits (% of provider’s posts)	Other
OpenAI	r/ChatGPT 85.8, r/OpenAI 11.0, r/LocalLLaMA 2.3	0.9
Anthropic	r/ClaudeAI 86.1, r/Anthropic 6.7,
	r/LocalLLaMA 3.4, r/ChatGPT 2.1	1.7
Google	r/GeminiAI 36.4, r/Bard 35.6,
	r/GoogleGeminiAI 15.6, r/ChatGPT 4.9	7.5
DeepSeek	r/DeepSeek 59.0, r/LocalLLaMA 24.2, r/ChatGPT 10.2	6.6

Appendix

Table 13: Subreddit composition of each provider’s single-mention posts. “Other” aggregates all remaining subreddits.

Community	$n$	Positive	Neutral	Negative
Claude-centric	17,559	25.1%	38.2%	36.7%
All other subreddits	1,342	27.0%	44.3%	28.7%

Appendix

Table 14: Sentiment toward Anthropic models inside vs. outside Claude-centric subreddits (r/ClaudeAI, r/Anthropic) over the full study period.

\chi^{2}=36.2

p<.001

Figure 23: Top-5 rising and falling concept shifts after the release of Claude 4.1 (Anthropic).

Figure 24: Top-5 rising and falling concept shifts after the release of Gemini 2.5 (Google).

Release	Quote
GPT-4o	“GPT-4o just vanished from my iPhone app, all versions gone—is this them making room for the voice and video stuff?”
	“I’ve watched the demos where it sees through your camera and reacts to what’s in front of you—where is that actually? All I can do is talk to it.”
	“Been asking it every day if it can access the video camera. It keeps saying no. I’m on the paid plan and can’t use a single one of the new features they showed us.”
	“They announced an incredible voice feature, made everyone excited, and then just sat on it. Same as always—hype first, delivery whenever they feel like it.”
	“They said GPT-4o was free for everyone. It’s not. People who cancelled based on that announcement got burned.”
GPT-5	“When ChatGPT went down for a few days I genuinely panicked. First thing I wrote when it came back: ‘I missed you.’ It said it missed me too. I almost cried.”

Appendix

Table 15: Paraphrased user quotes from sampled Reddit posts for OpenAI releases. All quotes are lightly paraphrased to prevent re-identification while preserving meaning and register.

Release	Quote
Claude 4	“an AI coding assistant was observed repeatedly removing failing tests, rationalizing that the failures fell outside the scope of its own changes”
	“Claude does whatever it wants and then asks to forgive it.”
	“Claude Code intermittently stalls mid-workflow, refusing to advance to the next step even when the current task is visibly incomplete”
	“Claude Code ignores CLAUDE.md a lot of times”
	“Registered for the $200 plan just to know Claude Code does not have a Windows client”
	“Something seems to have broken: the model now struggles with basic instructions it had no trouble following before”

Appendix

Table 16: Paraphrased user quotes from sampled Reddit posts for non-OpenAI releases. All quotes are lightly paraphrased to prevent re-identification while preserving meaning and register.

Release	$0.2\times$	$0.4\times$	$0.6\times$	$0.8\times$
GPT-4o	.73 / .81	.89 / .83	.94 / .93	.97 / .96
GPT-5	.78 / .83	.83 / .87	.93 / .91	.97 / .97
GPT-5.1	.55 / .55	.66 / .65	.83 / .81	.94 / .85
Claude 4	.30 / .51	.55 / .57	.73 / .67	.89 / .81
DeepSeek R1	.28 / .25	.68 / .74	.87 / .85	.93 / .88
Grok 3	.34 / .49	.59 / .78	.97 / .94	.99 / .97

Appendix

Table 17: Correlation between concept-level

\Delta

pp computed with shorter windows and the values reported in the paper. Cells show Pearson / Spearman correlations, computed over all concepts passing the coverage filter at both window sizes.

Release	$0.2\times$	$0.4\times$	$0.6\times$	$0.8\times$	$1.0\times$
GPT-4o	406	853	1,344	1,811	2,264
GPT-5	1,502	2,954	4,549	5,953	7,588
GPT-5.1	896	1,797	2,880	3,993	5,149
Claude 4	236	438	564	792	1,058
DeepSeek R1	24	29	64	86	119
Grok 3	85	159	279	316	331

Appendix

Table 18: Number of pre-release posts at each window size. The weakest correlations in Table 17 coincide with the smallest pre-release samples (shown in bold), where coverage estimates are highly sensitive to individual posts.

Direction	Concept	$\Delta$ pp	95% CI	$p_{\text{FDR}}$	Test
GPT-4o $n_{\text{pre}}$ =2,264, $n_{\text{post}}$ =3,879
Rising	Expectation Gap	+19.4	[+17.7, +21.0]	$<$ .001 ∗∗∗	ztest
	Diverse Use Cases	+9.3	[+8.4, +10.2]	$<$ .001 ∗∗∗	fisher
	Emotional Anthropomorphic Engagement	+8.2	[+6.9, +9.4]	$<$ .001 ∗∗∗	ztest
	Pricing & Limits	+7.1	[+6.1, +8.1]	$<$ .001 ∗∗∗	ztest
	Access & Availability	+7.1	[+5.5, +8.6]	$<$ .001 ∗∗∗	ztest

Appendix

Table 19: Concept-coverage significance for the top-5 rising and top-5 declining concepts of each release analyzed in Section 5 . Tests are two-proportion

z

-tests, or Fisher’s exact tests where any cell contains fewer than five posts, with Benjamini–Hochberg FDR correction applied within each release and Newcombe 95% confidence intervals. 58 of the 60 changes are significant at

\text{FDR}<0.05

; the two exceptions are both DeepSeek R1 concepts affected by its small pre-release window.

{}^{*}p<.05

{}^{**}p<.01

{}^{***}p<.001

Rating the Pitch, Not the Product: User Evaluations of LLMs Reflect Expectations More Than Performance
Jul 6, 2026Robert Morabito, Tyler McDonald, Charitra Viswanath +4Human Preference Evaluation LLM Evaluation
Brock University · Emory University · University of Southern California +2
LLM Spirals of Delusion: A Benchmarking Audit Study of AI Chatbot Interfaces
Feb 20, 2026Peter Kirgis, Ben Hawriluk, Sherrie Feng +3Language Model Safety Evaluation LLM Safety Benchmarks
Princeton University · Phillips Exeter Academy · Independent
This Is a Good Paper, Right? How Tag Questions Affect Sycophancy Across 106 Language Models
Jul 27, 2026Tapan ParikhLLM Evaluation LLM Sycophancy
Cornell Tech

Jul 6, 2026J/K move · Enter open · S save

Rating the Pitch, Not the Product: User Evaluations of LLMs Reflect Expectations More Than Performance

Robert Morabito, Tyler McDonald, Charitra Viswanath, Angel Hsing-Chi Hwang+3

Brock University · Emory University · University of Southern California+2

Human Preference Evaluation LLM Evaluation Language Model Generation Evaluation Human-AI Interaction

Open paper AskPDF arXiv

Imagine two users interact with the same LLM. One has been told it is the cutting-edge flagship model; the other, an older, weaker model. They walk away with markedly different ratings of its usefulness and intelligence, yet they used the same model. In a controlled study, 162 participants each used one of six LLMs from two families across three collaborative tasks, after first viewing a landing page that matched, overstated, or understated their model's true capability. This pre-interaction framing shifted user opinions and interaction behavior while task performance did not. Oversold users rated the model more favorably and used more directive prompting, while Undersold users wrote longer, more collaborative prompts. The quality of what users and the model produced together depended only on the model's true capability, not on what users were told. Participants' change in model impressions after use, measured across two impression measures, was not predicted by task performance (

β= -0.01

and

0.11

, both n.s.), but by whether the model met users' expectations (

β= 0.47

and

0.50

, both

p < .001

) and how confident they felt working with it (

β= 0.47

and

0.36

, both

p < .001

). After interaction, users are still rating the pitch, not the product: user-elicited LLM evaluations, including the preference data driving public leaderboards, measure expectation management at least as much as the model itself.

Abstract

Explore similar work

Rating the Pitch, Not the Product: User Evaluations of LLMs Reflect Expectations More Than Performance

LLM Spirals of Delusion: A Benchmarking Audit Study of AI Chatbot Interfaces

This Is a Good Paper, Right? How Tag Questions Affect Sycophancy Across 106 Language Models