Investigating Vaccine Buyer's Remorse: Post-Vaccination Decision Regret in COVID-19 Social Media Using Politically Diverse Human Annotation

Miles Stanley; Soumyajit Datta; Ashutosh Kumar; Ashiqur R. KhudaBukhsh

cs.CYMar 18, 2026

Investigating Vaccine Buyer's Remorse: Post-Vaccination Decision Regret in COVID-19 Social Media Using Politically Diverse Human Annotation

Authors: Miles Stanley, Soumyajit Datta, Ashutosh Kumar, Ashiqur R. KhudaBukhsh

Organizations: University of Washington, Seattle Washington · Rochester Institute of Technology Rochester, NY 14623

Abstract

A significant gap exists in datasets regarding post-COVID-19 vaccination experiences, particularly ``vaccine buyer's remorse''. Understanding the prevalence and nature of vaccine regret, whether based on personal or vicarious experiences, is vital for addressing vaccine hesitancy and refining public health communication. In this paper, we curate a novel dataset from a large YouTube news corpus capturing COVID-19 vaccination experiences, and construct a benchmark subset focused on vaccine regret, annotated by a politically diverse panel to account for the subjective and often politicized nature of the topic. We utilize large language models (LLMs) to identify posts expressing vaccine regret, analyze the reasons behind this regret, and quantify its occurrence in both first and second-person accounts. This paper aims to (1) quantify the prevalence of vaccine regret; (2) identify common reasons for this sentiment; (3) analyze differences between first-person and vicarious experiences; and (4) assess potential biases introduced by different LLMs. We find that while vaccine buyer's remorse appears in only $<2\%$ of public discourse, it is disproportionately concentrated in vaccine-skeptic influencer communities and is predominantly expressed through first-person narratives citing adverse health events.

Figures & tables

Example Comment	Regret Label	Subject Label
I got my first shot, then tried to get a mamogram and they will not do it until 4 weeks after my second shot. apparently the limp nods have a chance of swelling. I wouldn’t have gotten the shot if i knew that. they are not telling us everything.	Positive	First-Person
i didn’t vote for world shut down. i can do the research myself and have decided this particular vaccine is not safe for many reasons. my own doctor is no longer recommending it and regrets taking it himself.	Positive	Third-Party
coming home after my booster shot, i just crossed paths with the coroner as he was removing a body from my apt. building - covid is everywhere. get vaxed!	Negative	First-Person
isn’t dying the activity you do after the vaccine	Negative	Unspecified

Table 1: Examples of manually annotated comments from the dataset.

Model	Zero-Shot						Few-Shot
Model	Subject		Vaccinated		Regret		Subject		Vaccinated		Regret
	F1	Accuracy	F1	Accuracy	F1	Accuracy	F1	Accuracy	F1	Accuracy	F1	Accuracy
mistral-small3.2:24b	0.734	0.746	0.824	0.828	0.739	0.757	0.695	0.702	0.836	0.837	0.790	0.816
gpt-4o-mini	0.672	0.674	0.842	0.843	0.803	0.855	0.647	0.650	0.839	0.839	0.804	0.853
mixtral:8x22b	0.745	0.769	0.770	0.783	0.806	0.842	0.733	0.743	0.833	0.836	0.808	0.846
llama3.1:70b	0.744	0.769	0.795	0.803	0.775	0.807	0.709	0.717	0.840	0.841	0.800	0.843

Table 2: Performance comparison across models in zero-shot and few-shot settings on the held-out test set

Model	Fine-Tuned Performance						Inference Speed
Model	Subject		Vaccinated		Regret		Inference Speed
	F1 (macro)	Accuracy	F1 (macro)	Accuracy	F1 (macro)	Accuracy	(sec/400 comments)
llama3.1:70b	0.77	80.25%	0.87	87.00 %	0.83	87.00 %	650
llama3.1:8b	0.72	76.00%	0.86	85.75%	0.81	84.50%	164
mixtral:8x7b	0.79	81.50 %	0.86	85.75%	0.83	87.25%	2,703

Table 3: Performance and inference speed comparison across fine-tuned models on the held-out test set.

Figure 1: Percentage of regret comments across news sources and influencer categories.

Figure 2: Overall Distribution of Relationships in Vicarious Regret Comments

Figure 3: Distribution of Regret Reasons by Source Type.

Task	Fleiss’ $\kappa$	Dem vs Rep $\kappa$	Dem vs Ind $\kappa$	Rep vs Ind $\kappa$
Subject	0.5089	0.5093	0.5140	0.5046
Vaccinated	0.4272	0.4380	0.4286	0.4330
Regret	0.4480	0.4246	0.4967	0.3677

Table 4: Inter-Annotator Agreement by Task

Task	Chi-square ( $\chi^{2}$ )	p-value
Subject	22.8515	0.0001
Vaccinated	1.6875	0.4301
Regret	3.1468	0.2073

Table 5: Systematic Differences in Annotation by Political Affiliation

Figure 4: Temporal Distribution of Regret Comments over time across zones

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

News Channel	# of Videos	# of Comments
Fox News	23902	31340801
CNN	12490	22139179
MSNBC	29494	26827950
Total	65886	80307930

Appendix

Table 6: Summary of the unfiltered data collected from news organization channels.

Vaccine Opinion	Influencer	# of Videos	# of Comments
Pro-vaccine	David Pakman	412	204,410
	The Young Turks	177	151,068
	Secular Talk	88	57,175
	Hasan Piker	14	10,961
	Destiny	10	4,915
	Pro-vaccine Subtotal	701	428,529

Appendix

Table 7: Summary of data collected from individual influencers, grouped by vaccine opinion.

Task	Metric/Class	Precision	Recall	F1	Support
Subject	Accuracy: 81.50%
	other	0.86	0.74	0.80	181
	self	0.91	0.94	0.93	139
	unspecified	0.60	0.76	0.67	80
	macro avg	0.79	0.82	0.80	400
Vaccinated	Accuracy: 87.25%

Appendix

Table 8: Performance Metrics of the Classification Pipeline

Category	Precision	Recall	F1-Score	Support
Spouse or Partner	1.00	0.90	0.95	10
Family Member	1.00	1.00	1.00	32
Friend	1.00	0.91	0.95	11
Health Care Provider	1.00	0.50	0.67	2
Public Figure	1.00	0.83	0.91	24
Other Acquaintance	0.69	0.95	0.80	19

Appendix

Table 9: Performance of Zero-Shot Model for Relationship Classification

Category	Precision	Recall	F1-Score	Support
Adverse Health Event	0.96	0.93	0.95	58
Perceived Coercion	1.00	0.85	0.92	13
Lack of Efficacy	0.94	0.89	0.91	18
Shift in Beliefs	0.67	1.00	0.80	6
Vague or Unspecified	0.75	1.00	0.86	6
Weighted Avg	0.93	0.92	0.92	101

Appendix

Table 10: Performance of Zero-Shot Model for Reason for Regret Extraction

Figure 5: Distribution of Regret Reasons by Narrative Perspective.

Hyperparameter	Value
Learning Rate	2e-4
Learning Rate Scheduler	cosine
Batch Size	1
Gradient Accumulation Steps	4
Number of Epochs	3
Weight Decay	0.001

Appendix

Table 11: Fine-tuning Hyperparameters for Llama-3.1 70b

Comment	Subject	Vaccinated	Regret
i am 69 years old. i was vaccinated twice. i am vaccine injured. i cannot take the booster. i have not been out in public since july of 2021 without a p100 respirator. no one is going to tell me to get another risky vaxx. no one! eric speaks for me. so all you high and mighty anti trumper’s can stick it!	self	1	1
don’t believe anything don lemon says cnn lies you can pay anybody to say anything the right amount of money will make anybody say anything! best friends vaccinated mother has got a story to tell two and regrets ever taking this vaccine	other	1	1
it hurts my heart knowing my great state of michigan is climbing in cases. the biggest b.s is that we the smart vaccinated ones if we end up in the hospital we aren’t eligible for the infusion but unvaccinated people are. how is this even fair!?!? how can the non-vaccinated people be put above people who actually care for more then just themselves!? i don’t like to talk ill about people but if your that ignorant not to get vaccinated and spread this crap because your scared the government is trying to spy on you. p.s you are already being tracked by your dang smart phones, smart cars and smart devices in your home duh! they don’t need to try and track you with a dang vaccine that is only trying g to get us back some what of normalcy!	self	1	0
i wonder what’s in the vaccine that they’re pushing it so hard	unspecified	0	0

Appendix

Table 12: Examples of Pipeline Classifications

Comment	Dem.	Rep.	Ind.	Pipeline
i caught covid wit da vaccine	1	0	0	0
if people don’t want the vaccine, then they don’t have to get it. two weeks after my second dose, i had to be rushed to the emergency room. i was diagnosed with a heart condition called sinus tachycardia. im only 27, in good health, and have no family history of any heart conditions whatsoever. so i know why people are iffy about the vaccine, we do not know the long term effects	0	0	1	1

Appendix

Table 13: Examples of Comments with Annotator Disagreement for Regret Classification

Source Category	Relevant Comments	Regret Rate
Fox News	56,822	0.7%
CNN	57,883	0.7%
MSNBC	43,968	0.6%
Pro-Vaccine Influencers	43,527	1.0%
Vaccine-Skeptic Influencers	41,347	2.9%
Total	243,547	1.1%

Appendix

Table 14: Distribution of Vaccine Regret Sentiment Across Source Categories

Source Category	Total Comments	Relevance Rate
Fox News	100,000	56.8%
CNN	100,000	58.9%
MSNBC	100,000	44.0%
Subtotal News	300,000	52.9%
Pro-Vaccine Influencers	150,000	29.0%
Vaccine-Skeptic Influencers	150,000	27.6%

Appendix

Table 15: Results of NLI Relevance Filtering by Source Category

Comment Text	True Label			Predicted Label
	Subject	Vaccinated	Regret	Subject	Vaccinated	Regret	Relevance
"not a big deal"- joe biden after the first balloon was shot down."	other	0	0	unspecified	0	0	0
hey johnson, here is religion for you: “and the lord god formed man of the dust of the ground, and breathed into his nostrils the breath of life; and man became a living soul.” - genesis 2:7 so according to your bible, a baby is only a living soul after the first breath."	other	0	0	unspecified	0	0	0
we can be sure dawn bancroft is only one of many of the rioters at the capitol on jan. 6 that wishes they hadn’t shot video of themselves responding to djt’s incitement to violence. if all criminals were as stupid as she is, we wouldn’t need law enforcement any more. just round up the cell phones, watch the videos, and throw them in jail.	other	0	0	unspecified	0	0	0

Appendix

Table 16: Examples of NLI Errors

Comment Text	True Label			Predicted Label
	Subject	Vaccinated	Regret	Subject	Vaccinated	Regret
blah blah blah bu blah bu bu blah…guess who i am? i’m a cuflu shot recipient! gessssshhh when are going to recall all this crap and accept the fraud…	self	1	0	self	1	1
i bet he regrets getting the clot shot and the 30 boosters following.	other	1	0	other	1	1
crony liars and misinforms, million of people after the booster got tested positive to covid 19, does not protect from getting the virus, i am one of them , the main question is why we need the mandate	self	1	0	self	1	1

Appendix

Table 17: Examples of False Positives (Regret)

Comment Text	True Label			Predicted Label
	Subject	Vaccinated	Regret	Subject	Vaccinated	Regret
i give kudo’s to this man .. i am in the same position forced to get the shot..got the shot still ended up getting covid..my job said no shot no paycheck	self	1	1	self	1	0
no one’s died from the vaccine can you really be that ignorant and say that with a straight face marvin haggler there was a dr developed a rare blood disease and died he had a elite group of drs attempting to save his life but they could not and he said right before he died and this man is a dr who was a proponent for vaccines and he told his wife that it was no doubt the vaccine that killed him and there are plenty more to to go find	other	1	1	other	1	0
o.k. so, another booster, on top of the booster i got after my first booster six months after my initial 2 shot vaccination. what’s next? wake me when the "bumblebee" variant shows up. c’mon man.	self	1	1	self	1	0

Appendix

Table 18: Examples of False Negatives (Regret)

Comment Text	True Label			Predicted Label
	Subject	Vaccinated	Regret	Subject	Vaccinated	Regret
soooo many vaccinated realizing they’ve made a huge mistake	unspecified	0	0	other	0	0
stupid parents almost kill their child with a covid jab. parents get your head out of your asses!	other	1	0	other	0	0
how about the vaccine caused his med. problems	other	1	0	other	0	0

Appendix

Table 19: Subject and Vaccinated misclassification examples

Mar 4, 2026

Who Shapes Brazil's Vaccine Debate? Semi-Supervised Modeling of Stance and Polarization in YouTube's Media Ecosystem

Vaccination remains a cornerstone of global public health, yet the COVID-19 pandemic exposed how online misinformation, political polarization, and declining institutional trust can undermine immunization efforts. Most of the prior computational studies that analyzed vaccine discourse on social platforms focus on English-language data, specific vaccines, or short time windows, impairing our understanding of long-term dynamics in high-impact, non-English contexts like Brazil, home to one of the world's most comprehensive immunization systems. We here present the largest longitudinal study of Brazil's vaccine discourse on YouTube, leveraging a semi-supervised stance detection framework that combines self-labeling and self-training to classify nearly 1.4 million comments. By integrating stance with temporal patterns, engagement metrics, and channel taxonomy (legacy media, science communicators, digital-native outlets), we map how pro- and anti-vaccine narratives evolve and circulate within a hybrid media ecosystem. Our results show that semi-supervised learning substantially improves stance classification robustness, enabling fine-grained tracking of public attitudes across Brazil's full immunization schedule. Polarization spikes during epidemiological crises, especially COVID-19, but becomes fragmented across vaccines and interaction patterns in the post-pandemic period. Notably, science communication and digital-native channels emerge as the primary loci of both supportive and oppositional engagement, revealing structural vulnerabilities in contemporary health communication. Thus, our work advances computational methods for large-scale stance modeling while offering actionable evidence for public health agencies, platform governance, and online information ecosystems.

Geovana S. de Oliveira, Ana P. C. Silva, Fabricio Murai +1

Universidade Federal de Ouro Preto, João Monlevade, Brazil · Universidade Federal de Minas Gerais, Belo Horizonte, Brazil · Worcester Polytechnic Institute, Worcester, United States +1

Social Media Analysis Stance Detection Scientific Communication

Ask PDF

Mar 25, 2026

WhatsApp Vaccine Discourse (WhaVax): An Expert-Annotated Dataset and Benchmark for Health Misinformation Detection

We introduce WhaVax, a new expert-annotated dataset of vaccine-related WhatsApp messages collected from large Brazilian public groups spanning multiple pandemic years. The dataset was constructed through a rigorous, carefully designed pipeline that integrates keyword-based data collection, semantic deduplication to remove near-duplicate content, and a multi-stage annotation protocol conducted by medical specialists. This process produced a high-quality gold-standard corpus, characterized by substantial inter-annotator agreement and strong reliability for downstream analysis. Additionally, we provide a detailed characterization of WhatsApp misinformation, revealing distinctive linguistic, structural, lexical, temporal, and group-level patterns, as well as a meaningful layer of ambiguous cases that reflect the complexity of health discourse in private messaging. We also benchmark classical models, fine-tuned Small Language Models, and zero- or few-shot Large Language Models under realistic data-scarcity constraints, demonstrating that strong embeddings and LLM approaches perform competitively, while domain alignment and data availability remain critical factors. This study provides a rare, high-quality resource to support misinformation research and computational modeling in encrypted communication environments.

Jônatas H. dos Santos, Julio C. S. Reis, Philipe Melo +9

Universidade Federal de Minas Gerais (UFMG), Brasil · Universidade Federal de Viçosa (UFV), Brasil

Health Misinformation Detection Fake News Detection Automated Fact-Checking

Ask PDF

Jun 9, 2026

Leveraging Social Media Data for COVID-19 Studies

Nowadays, social media networks have become widely preferred sources of information. Especially during the time of the Coronavirus disease 2019 COVID 19 pandemic, social media has been one of the most used platforms to get the latest news and information related to COVID 19. Social media are popular because they offer free access to their registered users and allow them to do posting, disseminate information, and respond to others postings. With almost 4.6 billion social media users worldwide, it is not surprising the significant amount of information shared through these platforms could affect how people perceive and cope with the pandemic that we are facing right now. With decent use, social media can be a beneficial digital tool to spread reliable news and public awareness for patients, clinicians, and society. Specifically, this chapter describes linguistic, visual, and emotional indicators expressed in user disclosures. Thus, in this chapter, the related studies of social media platforms usage during the COVID 19 pandemic are explored and discussed in detail. This chapter also categorizes social media data used, introduces different deployed machine learning, feature engineering, natural language processing, and survey methods, and outlines directions for future research.

Nur Hafieza Ismail, Nur Shazwani Kamarudin, Nurol Husna Che Rose

Social Media Analysis Computational Social Science

Ask PDF

Abstract

Explore similar work

Who Shapes Brazil's Vaccine Debate? Semi-Supervised Modeling of Stance and Polarization in YouTube's Media Ecosystem

WhatsApp Vaccine Discourse (WhaVax): An Expert-Annotated Dataset and Benchmark for Health Misinformation Detection

Leveraging Social Media Data for COVID-19 Studies