WhatsApp Vaccine Discourse (WhaVax): An Expert-Annotated Dataset and Benchmark for Health Misinformation Detection
Authors: Jônatas H. dos Santos, Julio C. S. Reis, Philipe Melo, João F. H. Olivetti, Thales H. Silva, Matheus Gontijo Guimaraes, Glaucio de Souza, Marcos A. Gonçalves, Fabricio Benevenuto, Filipe B. B. Zanovello, Marco A. G. Rodrigues, Cristiano X. Lima
Organizations: Universidade Federal de Minas Gerais (UFMG), Brasil · Universidade Federal de Viçosa (UFV), Brasil
We introduce WhaVax, a new expert-annotated dataset of vaccine-related WhatsApp messages collected from large Brazilian public groups spanning multiple pandemic years. The dataset was constructed through a rigorous, carefully designed pipeline that integrates keyword-based data collection, semantic deduplication to remove near-duplicate content, and a multi-stage annotation protocol conducted by medical specialists. This process produced a high-quality gold-standard corpus, characterized by substantial inter-annotator agreement and strong reliability for downstream analysis. Additionally, we provide a detailed characterization of WhatsApp misinformation, revealing distinctive linguistic, structural, lexical, temporal, and group-level patterns, as well as a meaningful layer of ambiguous cases that reflect the complexity of health discourse in private messaging. We also benchmark classical models, fine-tuned Small Language Models, and zero- or few-shot Large Language Models under realistic data-scarcity constraints, demonstrating that strong embeddings and LLM approaches perform competitively, while domain alignment and data availability remain critical factors. This study provides a rare, high-quality resource to support misinformation research and computational modeling in encrypted communication environments.
Figures & tables
Dataset
Description
Labels
Annotation Method
( Cui and Lee 2020 )
COVID-19 healthcare misinformation dataset with news articles, social media posts, and user engagements.
Fake vs. real + metadata labels.
Fact-check verification.
( Hossain et al. 2020 )
6.6k tweets annotated for stance regarding curated COVID-19 misconceptions.
Agree/Disagree/No Stance per misconception.
Misconceptions curated by medical experts + human annotation.
( Weinzierl and Harabagiu 2021 )
Tweets mapped to specific COVID-19 vaccine misinformation targets; includes stance labels.
Target labels; stance: Agree/Disagree/No Stance.
Curated misinformation targets + human expert annotation.
( Chen et al. 2021 )
Multimodal repository with news and tweets (text + images + temporal data) focused on COVID-19 vaccines.
Media credibility rankings + manual stance/credibility annotation.
( Hayawi et al. 2022 )
Large Twitter dataset ( 15M tweets) with 15k manually annotated for COVID-19 vaccine misinformation.
Binary: misinformation vs. general vaccine tweet.
Human annotation + expert validation.
( Crupi et al. 2022 )
Twitter dataset (16.2M tweets) of the Italian COVID-19 vaccination debate (665k unique users, Sep 2019–Nov 2021).
Stance: Supporter/Hesitant/Other/Pets.
Manual annotation propagated via hierarchical clustering on retweet networks.
Table 1: Datasets related to vaccine misinformation.
keyword
#Occurrences
Unique users
Unique groups
vacina
65373
10710
876
vacinas
32476
6835
768
vacinação
17773
4450
695
vacinados
7813
2521
533
vacinar
5675
2458
532
vacinado
3504
1758
473
Table 2: Key words for the vaccine filter.
Figure 1: Message distribution of annotators agreements.
Figure 2: Message size distribution.
Class
#Characters
#Words
#Exclamations
#Question Marks
#Capital letters
Non Misinformation
210.92
22.43
0.22
0.29
12.57
Misinformation
290.39
39.62
0.59
0.46
35.67
Table 3: Textual characteristics.
Figure 3: Concentration of misinformation by group.
Figure 4: Temporal distribution of messages.
Model
Average macro-F1
Bertimbau Embedding
Support Vector Machine
0.724 ± 0.030
Logistic Regression
0.747 ± 0.036
Random Forest
0.656 ± 0.041
Multi-Layer Perceptron
0.738 ± 0.031
XGBOOST
0.687 ± 0.034
Table 4: Macro F1 - Classifications models.
Figure 5: Impact of few-shots on performance.
Portuguese
English (Translation)
Correct Label
N Incorrect Models
CENTENAS DE MILHARES ESTÃO MORTOS E ENTERRADOS PORQUE O RECUSOU-SE A COMPRAR VACINAS! POR CAUSA DISSO A CONTAMINAÇÃO AUMENTOU E EXPLODIU!!! É UM ASSASSINO MILICIANO GENOCIDA!
HUNDREDS OF THOUSANDS ARE DEAD AND BURIED BECAUSE REFUSED TO BUY VACCINES! BECAUSE OF THIS, THE CONTAMINATION INCREASED AND EXPLODED!!! IS A GENOCIDAL MILITIA MURDERER!
0
18
BÉLGICA EM FÚRIA Bruxelas sob domínio da Guerra Os holandeses e agora os Belgas. A humanidade está saindo do ostracismo das manifestações pacíficas e idiotas para a verdadeira reação contra a tirania sanitária. O alvo são as cabeças dos políticos, dos governantes e também dos jornalistas corruptos que promovem a tirania das vacinas genocidas e patrocinam a ditadura dos passaportes sanitários. Parece que a fúria foi despertada. Agora a Humanidade está acordando para a realidade: de que não há conquistas sem guerras. Bruxelas amanheceu debaixo da ira da população que atacou o sistema com pedras, porretes e muita gasolina. A fúria e a raiva chegarão nos responsáveis, que serão caçados, trucidados e enviados de volta para o inferno. NÃO RECUEM! NÃO SE DOBREM! ARREBEBENTEM COM TUDO!
BELGIUM IN FURY Brussels under the dominion of War The Dutch and now the Belgians. Humanity is emerging from the ostracism of peaceful and idiotic demonstrations into a true reaction against sanitary tyranny. The target is the heads of politicians, rulers, and also corrupt journalists who promote the tyranny of genocidal vaccines and sponsor the dictatorship of health passports. It seems that fury has been awakened. Now Humanity is waking up to the reality: that there are no conquests without wars. Brussels awoke under the wrath of the population that attacked the system with stones, clubs, and plenty of gasoline. The fury and rage will reach those responsible, who will be hunted down, slaughtered, and sent back to hell. DO NOT RETREAT! DO NOT BEND! BREAK EVERYTHING DOWN!
0
17
Pessoal, percam uma horinha de vocês e vejam esta reportagem. É estarrecedora e muito esclarecedora. São dois cientistas, sendo que um deles é umas maiores sumidades sobre vacinas, responsável pelo design de vacinas em grandes órgãos públicos e empresas privadas. Veja o que ele fala sobre a vacinação. É muito esclarecedor.
Folks, take an hour of your time and watch this report. It’s shocking and very enlightening. It features two scientists, one of whom is a leading authority on vaccines, responsible for vaccine design in major public agencies and private companies. Watch what he says about vaccination. It’s very insightful.
0
17
China admite que eficácia de suas vacinas contra Covid-19 não é alta
China admits that the effectiveness of its Covid-19 vaccines is not high.
1
16
Vai saber se foi alguma vacina que deu errado
Who knows if it was some vaccine that went wrong?
1
16
Table 5: Examples incorrectly classified by most models.
Universidade Federal de Ouro Preto, João Monlevade, Brazil · Universidade Federal de Minas Gerais, Belo Horizonte, Brazil · Worcester Polytechnic Institute, Worcester, United States +1