Organizations: Department of Electrical and Electronics Engineering, Faculty of Engineering, C¸ anakkale Onsekiz Mart University, C¸ anakkale, 17110, Türkiye · Department of Public Relations and Publicity, Faculty of Communication, Ege University, Izmir, 35100, Türkiye · Department of Media and Communication, Faculty of Communication, Ege University, Izmir, 35100, Türkiye · Department of Business Administration, Faculty of Economics and Administrative Sciences, Ege University, Izmir, 35100, Türkiye · Department of Statistics, Yildiz Technical University, Istanbul, 34349, Türkiye
Studies of online political discourse commonly infer public opinion through stance analysis while implicitly assuming that observed user participation is authentic. Meanwhile, studies focusing on detecting manipulative participation are typically modeling it as a binary phenomenon, overlooking the varying degrees to which users may engage in coordinated or inauthentic behavior. This study proposes a continuous modeling approach for manipulative participation and examines its integration with stance analysis. Using online discourse on the 2024 Street Animal Law in Türkiye as a case study, we collected a dataset comprising 7,320,574 tweets from 537,994 users. From this corpus, 301 users were selected through a high-information sampling strategy and manually annotated by domain experts using a 10-criterion rubric that yields user-level continuous manipulative participation scores. Evaluation under a unified user-level cross-validation protocol showed that the proposed framework accurately estimates both manipulative participation and user stance, with gradient boosting methods providing the best manipulative participation estimates and BERTurk achieving the highest stance classification performance. External validation on an independently annotated user set demonstrated good generalization performance. The findings support modeling manipulative participation as a continuous construct rather than a binary label, providing a more informative basis for computational analyses of the relationship between manipulative participation and stance in online political discourse.
Figures & tables
Figure 1: Various statistical distributions within the data collection.
Figure 2: Distributions underlying the high-information sampling strategy.
Criteria
Nusers
1
2
3
4
5
6
7
8
9
10
24 hours
102
96
96
96
96
97
95
96
96
98
98
tweet/retweet ratio
101
96
96
96
96
96
96
96
96
96
96
Additional
115
112
112
112
112
112
112
112
112
112
112
Table 1: Number of user accounts processed during expert-annotation, and their distribution counts per criterion.
Figure 3: Expert annotation distributions of the manipulative participation scores and stance labels as pro and anti-amendment.
Figure 4: The unified framework developed for stance detection and manipulative participation filtering.
Figure 5: The distribution of token sizes of tweets and retweets in the dataset, together with their mean token sizes.
Setting
Ridge
SVR
HistGBT
XGBoost
LightGBM
BERTurk
XLM-R
Text only
0.2019
0.2061
0.1995
0.2021
0.1984
0.2002
0.2034
± 0.0099
± 0.0115
± 0.0117
± 0.0146
± 0.0134
± 0.0309
± 0.0144
Text and metadata
0.1990
0.2103
0.2207
0.2252
0.2284
0.2162
0.2312
± 0.0140
± 0.0160
± 0.0273
± 0.0264
± 0.0208
± 0.0252
± 0.0186
Table 2: 5-fold cross-validation mean and standard deviation of MAE for predicted continuous estimation of manipulative participation.
Text only
Text and metadata
Model
Precision
Recall
F1
Precision
Recall
F1
LR
0.9285
0.7920
0.8311
0.9213
0.7645
0.8031
± 0.0379
± 0.0487
± 0.0481
± 0.0404
± 0.0612
± 0.0598
SVM
0.9140
0.7728
0.8064
0.9108
0.7380
0.7734
± 0.0469
± 0.0769
± 0.0746
± 0.0535
± 0.0723
± 0.0782
HistGBT
0.9199
0.7721
0.8091
0.8766
0.7613
0.7910
Table 3: 5-fold cross-validation performance for user-level stance detection using textual features alone and textual features combined with metadata. Mean scores are reported in the first row and standard deviations in the second row.
Task
Retweets
Performance
Manipulative participation
Included
MAE = 0.2606
Excluded
MAE = 0.3073
Stance classification
Included
Precision = 0.8555, Recall = 0.6911, F1 = 0.7234
Excluded
Precision = 0.8723, Recall = 0.6812, F1 = 0.7283
Table 4: Generalization performance of the proposed unified framework on both tasks.
Figure 6: Predicted results of the complete collection and their distributions per two tasks.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Precision
Recall
F1
Ridge
0.7596 ± 0.0606
0.6812 ± 0.0362
0.6730 ± 0.0477
SVR
0.7678 ± 0.0672
0.6892 ± 0.0367
0.6825 ± 0.0441
HistGBT
0.7740 ± 0.0616
0.6902 ± 0.0431
0.6831 ± 0.0549
XGBoost
0.7334 ± 0.0683
0.6719 ± 0.0550
0.6654 ± 0.0657
LightGBM
0.7595 ± 0.0477
0.6851 ± 0.0291
0.6771 ± 0.0416
BERTurk
0.6704 ± 0.2188
0.6533 ± 0.0924
0.6140 ± 0.1563
Appendix
Table 5: 5-fold cross-validation mean and standard deviation of binary manipulative participation detection after thresholding regression predictions (text only).
Model
Precision
Recall
F1
Ridge
0.7635 ± 0.0444
0.7015 ± 0.0473
0.6959 ± 0.0622
SVR
0.6890 ± 0.0714
0.6534 ± 0.0726
0.6416 ± 0.0846
HistGBT
0.6193 ± 0.0688
0.5961 ± 0.0608
0.5875 ± 0.0616
XGBoost
0.6074 ± 0.0540
0.5966 ± 0.0467
0.5919 ± 0.0447
LightGBM
0.6142 ± 0.0361
0.6019 ± 0.0312
0.5979 ± 0.0307
BERTurk
0.6240 ± 0.1898
0.6317 ± 0.0772
0.5955 ± 0.1384
Appendix
Table 6: 5-fold cross-validation mean and standard deviation of binary manipulative participation detection after thresholding regression predictions (text and metadata).
Information and Computer Science Department, KFUPM, Saudi Arabia · SDAIA–KFUPM Joint Research Center for Artificial Intelligence, Saudi Arabia · HUMAIN, Saudi Arabia