Authors: Philipp Meier, Katarina Boland, Laura Kallmeyer, Stefan Dietze
Organizations: Heinrich-Heine-University Düsseldorf, Germany · Heinrich-Heine-University & GESIS - Leibniz Institute for the Social Sciences, Cologne, Germany Düsseldorf, Germany
With the growing importance of science-related discourse on the Web and the erosion of the classical knowledge order, it is important to identify different user groups, such as scientists, automatically. This work proposes an approach for identifying scientists and non- scientists on X/Twitter based on their user biographies and tweets. We show that we are able to classify accounts as scientists and non- scientists on two different datasets, reaching an F1 score of up to 0.88 using Random Forests with linguistic features and up to 0.96 using a contrastively fine-tuned DeBERTa model in an ensemble setup. Furthermore, we provide two datasets with X users labeled as scientists or non scientists and their respective tweets and user biographies.
Figures & tables
Figure 1. Log scaled distribution of tweets per user for scientists for the Scholar corpus
Figure 2. Log scaled distribution of tweets per user for non-scientists for the Scholar corpus
Data
Tweets of scientists
Scientists
Tweets of non-scientists
Non- scientists
Orcid
21,443
578
6737
508
Scholar
189,852
23,785
189,852
23,785
Table 1. Dataset statistics for Orcid and Scholar datasets.
Tweet
Biography
Label
Election Expert Richard L. Hasen on Voters’ Distrust [URL]
Professor of Law and Political Science at UC Irvine; Election Law Blogger. Coming 2/4/20: Election Meltdown (Preorders [URL])
Scientist
#AcademicTwitter - is there any advantage to editing a special issue of a journal when you otherwise were going to publish your relevant work in other journals of much higher impact factor?
Spatial is my speciality. If you are a coral, your location is a matter of life or death - especially when a cyclone comes! Views my own. #TeamHB3
Scientist
@USER Here’s just one of many articles that debunk these stupid claims. Just Google it. [URL]
Software Engineer. Anything retweeted is not adopted as a statement of fact.
Non-Scientist
Anyone used to writing qualitative research papers? Writing this way is just not natural to me and I’m struggling so if anyones familiar pls help me get an A
keepin the fun in dysfunctional
Non-Scientist
REPLY: @USER @USER That’s my pick too,,, he would of been so perfect.
Just a very cool Jim Morrison fan …..Cowboys Fan/ addicted to movies & creating art, especially digital :: I Stand with Ukraine & President Biden
Unknown
Table 2. Example Tweets and user biographies from accounts with the label of the account. Users are anonymized with @USER, Urls with [URL].
Classifier
Accuracy
Precision
Recall
F1
SVM
0.84
0.85
0.82
0.84
Random Forest
0.84
0.85
0.82
0.84
Ada Boost
0.83
0.84
0.81
0.83
Logistic Reg.
0.83
0.84
0.82
0.83
SVM + KW
0.88
0.90
0.85
0.87
Random Forest + KW
0.88
0.89
0.86
0.88
Table 3. Results on a 10-fold cross validation on the Scholar dataset. ’KW’ stands for keyword features (Keyword+Regex), described in Section 4.1 . Recall, Precision and F1 Scores are macro averaged.
Classifier
Accuracy
Precision
Recall
F1
SVM
0.77
0.81
0.74
0.77
Random Forest
0.77
0.79
0.78
0.78
Ada Boost
0.76
0.78
0.77
0.77
Logistic Reg.
0.78
0.80
0.79
0.79
SVM + KW
0.81
0.84
0.79
0.81
Random Forest + KW
0.80
0.83
0.79
0.81
Table 4. Results on a 10-fold cross validation on the Orcid dataset. ’KW’ stands for keyword features (Keyword+Regex), described in Section 4.1 . Recall, Precision and F1 Scores are macro averaged.
Model
Accuracy
Precision
Recall
F1
Random Forest
0.84
0.85
0.82
0.84
Random Forest only KW
0.76
0.90
0.57
0.71
Random Forest + KW
0.88
0.89
0.86
0.88
DeBERTa Ensemble
0.94
0.94
0.95
0.94
DeBERTa Tweet
0.89
0.90
0.89
0.89
DeBERTa Bio
0.92
0.92
0.91
0.92
Table 5. Results of the contrastively fine-tuned models on the 10-fold cross validation on the Scholar dataset. Precision, Recall and F1 are macro averaged. ’KW’ stands for keyword features (Keyword+Regex).
Figure 3. Top 10 most important features for Random Forest for the Orcid dataset and the Scholar dataset with keyword features
Figure 4. Top 10 most important features for Random Forest for the Orcid dataset and the Scholar dataset without keyword features
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Fold
Triplet Count Training
Triplet Count Validation
Removed Users with duplicate texts
1
34,243
8,561
9
2
34,242
8,561
10
3
34,242
8,561
10
4
34,243
8,561
9
5
34,247
8,562
4
6
34,242
8,561
10
Appendix
Table 6. Triplet summary over 10 folds for Tweet data. Each fold contains 41,653 instances in total. This fold is split into 80% Train and 20% Validation
Fold
Contrastive Pairs Training
Positive Pairs
Negative Pairs
Validation Pairs
Removed Users with duplicate texts
1
34,243
17,121
17,122
8,561
9
2
34,242
17,121
17,121
8,561
10
3
34,242
17,121
17,121
8,561
10
4
34,243
17,121
17,122
8,561
9
5
34,247
17,123
17,124
8,561
4
6
34,242
17,121
17,121
8,561
10
Appendix
Table 7. Contrastive summary over 10 folds for Tweet data. Each fold contains 41,653 instances in total. This fold is split into 80% Train and 20% Validation. The created dataset is balanced,
Fold
Non-Scientists
Scientists
0
21,369
21,444
1
21,377
21,436
2
21,454
21,359
3
21,428
21,385
4
21,343
21,470
5
21,408
21,405
Appendix
Table 8. Statistics for each fold. Folds will be split (not stratified) into training (80%) and validation set (20%).
Fold
Triplet Count Training
Triplet Count Validation
Removed Users with duplicate texts
1
30,891
7,723
67
2
30,907
7,727
69
3
30,869
7,718
62
4
30,871
7,718
69
5
30,892
7,723
68
6
30,857
7,715
76
Appendix
Table 9. Triplet summary over 10 folds for biography . Each fold contains 41,653 instances in total. This fold is split into 80% Train and 20% Validation
Fold
Contrastive Pairs Training
Positive Pairs
Negative Pairs
Validation Pairs
Removed Users with duplicate texts
1
30,891
15,445
15,446
7,723
67
2
30,907
15,453
15,454
7,727
69
3
30,869
15,434
15,435
7,718
62
4
30,871
15,435
15,436
7,718
69
5
30,892
15,446
15,446
7,723
68
6
30,857
15,428
15,429
7,715
76
Appendix
Table 10. Contrastive summary over 10 folds for biography data . Each fold contains 41,653 instances in total. This fold is split into 80% Train and 20% Validation.
Bot accounts and coordinated influence operations are often discovered via heuristic methods, leaving a dearth of reliable ground-truth labels for training detection systems. To address this challenge, we study a natural question: can we generate labels to assist in learning embeddings in which bots and accounts from the same coordinated operation are close? We present FISSION, a method to generate labels by splitting each account's activity into positively labeled sub-accounts. Given this label source, we train detection models which preserve behavioral regularities recurring across positive sub-accounts. We evaluate FISSION and show it outperforms prior methods in detecting Wikipedia sockpuppets and Twitter/X bots.
Scientific papers follow rhetorical structures that organize content into sections such as Introduction, Methods, Results, and Discussion. Automatically identifying these sections at scale enables granular analysis of scientific writing patterns. We present a dataset of section-level annotations for millions of scientific papers from the Semantic Scholar Open Research Corpus (S2ORC). Using a rule-based classification algorithm, we identified and labeled major sections across 15.6 million papers after quality filtering. The dataset covers primarily STEM disciplines, with strong representation in medicine and biology. We provide comprehensive human and LLM-based validation showing that classifier agreement with human annotators is on par with human inter-annotator agreement. This dataset enables large-scale computational studies of scientific discourse and writing patterns.
Daniel Verdi, Jacob Aarup Dalsgaard, Roberta Sinatra
Graduate School of Education, Stanford University, United States · Center for Social Data Science (SODAS), University of Copenhagen, Denmark · Networks, Data, and Society (NERDS), IT University of Copenhagen, Denmark +2
On social media, many users actively push back against false claims. Understanding who pushes back and how they do so matters, as this corrective activity is central to how misinformation is contested. We study this counter-misinformation ecosystem at scale: applying a domain-specific NLI model from our prior work to a large corpus of COVID-19 tweets, we classify 264,737 posts as supporting or opposing false claims and compare 23 user- and text-level features across the two groups. Contrary to the dominant assumption that negative emotion is a signature of falsehood, we find that anti-misinformation posts are more emotionally negative than pro-misinformation posts, with higher levels of anger, disgust, and sadness. These differences are modest in magnitude but consistent in direction across the negative emotions. We also find that posts opposing misinformation tend to come from more established users, i.e., older accounts, more followers, and higher listed counts.
Eun Cheol Choi, Emilio Ferrara
University of Southern California, Annenberg School of Communication, Los Angeles, CA, United States · University of Southern California, Thomas Lord Department of Computer Science, Los Angeles, CA, United States