Avoiding Over-smoothing in Social Media Rumor Detection with Pre-trained Propagation Tree Transformer
Organizations: Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, Beijing Jiaotong University, Beijing 100044, China
Abstract
Deep learning techniques for rumor detection typically utilize Graph Neural Networks (GNNs) to analyze post relations. These methods, however, falter due to over-smoothing issues when processing rumor propagation structures, leading to declining performance. Our investigation into this issue reveals that over-smoothing is intrinsically tied to the structural characteristics of rumor propagation trees, in which the majority of nodes are 1-level nodes. Furthermore, GNNs struggle to capture long-range dependencies within these trees. To circumvent these challenges, we propose a Pre-Trained Propagation Tree Transformer (P2T3) method based on pure Transformer architecture. It extracts all conversation chains from a tree structure following the propagation direction of replies, utilizes token-wise embedding to infuse connection information and introduces necessary inductive bias, and pre-trains on large-scale unlabeled datasets. Experiments indicate that P2T3 surpasses previous state-of-the-art methods in multiple benchmark datasets and performs well under few-shot conditions. P2T3 not only avoids the over-smoothing issue inherent in GNNs but also potentially offers a large model or unified multi-modal scheme for future social media research.
Figures & tables
| Statistic | DRWeibo | Twitter15 | Twitter16 | UWeibo | UTwitter | |
| # claims | 4664 | 6037 | 1490 | 818 | 209549 | 204922 |
| # non-rumors | 2351 | 3185 | 374 | 205 | - | - |
| # false rumors | 2313 | 2852 | 370 | 205 | - | - |
| # true rumors | - | - | 372 | 207 | - | - |
| # unverified rumors | - | - | 374 | 201 | - | - |
| # avg reply | 803.5 | 61.8 | 50.2 | 49.1 | 50.5 | 82.5 |
| DRWeibo | |||||||||
| Method | Class | Acc. | Prec. | Rec. | F1 | Acc. | Prec. | Rec. | F1 |
| PLAN | R | 0.915±0.007 | 0.908 | 0.923 | 0.915 | 0.788±0.005 | 0.786 | 0.760 | 0.771 |
| N | 0.923 | 0.907 | 0.914 | 0.793 | 0.813 | 0.802 | |||
| HD-TRANS | R | 0.921±0.004 | 0.915 | 0.929 | 0.920 | 0.810±0.006 | 0.814 | 0.834 | 0.823 |
| N | 0.928 | 0.913 | 0.921 | 0.809 | 0.783 | 0.794 | |||
| BiGCN | R | 0.942±0.008 | 0.919 | 0.968 | 0.942 | 0.866±0.010 | 0.869 | 0.849 | 0.858 |
| Twitter15 | Twitter16 | |||||||||
| N | F | T | U | N | F | T | U | |||
| Method | Acc. | F1 | F1 | F1 | F1 | Acc. | F1 | F1 | F1 | F1 |
| PLAN | 0.819±0.004 | 0.839 | 0.854 | 0.817 | 0.759 | 0.843±0.005 | 0.855 | 0.851 | 0.858 | 0.805 |
| HD-TRANS | 0.810±0.008 | 0.834 | 0.841 | 0.750 | 0.808 | 0.828±0.004 | 0.836 | 0.840 | 0.843 | 0.790 |
| BiGCN | 0.844±0.005 | 0.856 | 0.844 | 0.863 | 0.809 | 0.880±0.009 | 0.793 | 0.912 | 0.947 | 0.849 |
| ClaHi-GAT | 0.851±0.004 | 0.851 | 0.870 | 0.863 | 0.816 | 0.885±0.010 | 0.798 | 0.952 | 0.917 | 0.854 |
| DRWeibo | Twitter15 | Twitter16 | ||
| P2T3 | 0.973 | 0.912 | 0.874 | 0.911 |
| w/o Token-wise Embedding | 0.921(↓0.052) | 0.836(↓0.076) | 0.803(↓0.071) | 0.843(↓0.068) |
| w/o Chain Identifier | 0.943(↓0.030) | 0.867(↓0.045) | 0.857(↓0.017) | 0.854(↓0.057) |
| w/o Depth Embedding | 0.952(↓0.021) | 0.893(↓0.019) | 0.867(↓0.007) | 0.876(↓0.035) |
| w/o Type Embedding | 0.964(↓0.009) | 0.915(↑0.003) | 0.870(↓0.004) | 0.893(↓0.018) |
| w/o Pre-training | 0.958(↓0.015) | 0.898(↓0.014) | 0.861(↓0.013) | 0.901(↓0.010) |
| Method | Setting | DRWeibo | |
| GIN | w/o Pre-training | 0.940 | 0.847 |
| w/ Pre-training | 0.943(↑0.003) | 0.854(↑0.007) | |
| GAT | w/o Pre-training | 0.927 | 0.836 |
| w/ Pre-training | 0.928(↑0.001) | 0.841(↑0.005) | |
| GCN | w/o Pre-training | 0.931 | 0.849 |
| w/ Pre-training | 0.939(↑0.008) | 0.851(↑0.002) |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Depth | Post | Rank |
| source | JUST IN: Donald Trump to @MSNBC: ‘There has to be some form of punishment’ for women who have an abortion. | - |
| 1 | UPDATE : Donald Trump advocates abortion ban and ‘some form of punishment’ for women who have an abortion. | 3 |
| 2 | NEW : Trump has released this statement following his abortion comments to @MSNBC. | 4 |
| 3 | In new statement, Trump says doctors, or anyone performing abortion, would be held legally responsible under a ban – ‘not the woman.’ | 2 |
| 4 | ok you lost me here! I’ve just rtweeted he said women shld get some punushment 4 that | 1 |
| source | Attorney: New audio reveals pause in gunfire when Michael Brown was shot. | - |