Classifying Problem and Solution Framing in Congressional Social Media
Authors: Misha Melnyk, Mitchell Dolny, Joshua D. Elkind, A. Michael Tjhin, Saisha Chebium, Blake VanBerlo, Annelise Russell, Michelle M. Buehlmann, Jesse Hoey
Organizations: David R. Cheriton School of Computer Science, University of Waterloo · Martin School of Public Policy & Admin, University of Kentucky · Nistler College of Business and Public Administration, University of North Dakota
Policy setting in the USA according to the Garbage Can'' model differentiates between problem'' and ``solution'' focused processes. In this paper, we study a large dataset of US Senator postings on Twitter (1.68m tweets in total). Our objective is to develop an automated method to label Senatorial posts as either in the problem or solution streams. Two academic policy experts labeled a subset of 3967 tweets as either problem, solution, or other (anything not problem or solution). We split off a subset of 500 tweets into a test set, with the remaining 3467 used for training. During development, this training set was further split by 60/20/20 proportions for fitting, validation, and development test sets. We investigated supervised learning methods for building problem/solution classifiers directly on the training set, evaluating their performance in terms of F1 score on the validation set, allowing us to rapidly iterate through models and hyperparameters, achieving an average weighted F1 score of above 0.8 on cross validation across the three categories using a BERTweet Base model.
Figures & tables
Category
AR TRAIN
%
MB TRAIN
%
TRAIN_AGREE
%
1
1337
39
1268
37
1156
37
2
743
21
747
21
638
21
3
1387
40
1452
42
1314
42
Table 1: Label counts and proportions for coded labels and agreed upon labels
Figure 1: Labeled data set partition used for training and evaluation. Size of subsets in graph are not indicative of set size.
Figure 2: BERT Model Details showing flow of data from initial text to output scores. Information flows upwards in this picture, starting from a sequence of words (the Tweet) at the bottom. The Tokenizer separates out the raw text into tokens which are vectors in an embedding space. These vectors are augmented with positional information and fed into the Bidirectional Transformer (shown summarized as an attention layer and a feed forward layer in blue). The outputs of this model are then fed through a dropout and a linear layer to produce the final classification into problem/solution/other.
BERT-base
BERT-large
Total Weights
110 million
340 million
Attention layers
12
24
Embedding Dimension
768
1024
Table 2: BERT base vs large characterists comparison
model_name
vinai/bertweet-base1U
The base model used
dropout_p
0.1
Dropout probability in training
trials
20
Number of different starting seeds
cross_val_folds
7
How many folds for k-fold cross validation
learning_rate
0.00003
Learning rate for optimizer
max_epochs
25
Maximum number of epochs to do before stopping
accumulate_grad_batches
1
How many batches to do before updating gradients
Table 3: . Parameters used while running the training trials.
Classifier
Logistic Regression
XGBoost Treest
BERT Base
BERT Large
[AR]
[AR]
[AR]
[MB]
[AR]
[MB]
Avg F1
0.44
0.42
0.813
0.796
0.800
0.836
Avg Weighted F1
0.47
0.45
0.806
0.823
0.810
0.845
Table 4: Performance comparison of classifier architectures on held-out evaluation data (macro F1 and weighted F1 scores).
Variant
Base
Large
agreed
disagreed
x-val
agreed
disagreed
x-val
AR m-F1
0.796 (0.0084)
0.800 (0.0089)
0.799 (0.016)
0.800 (0.011)
0.796 (0.011)
0.806 (0.02)
AR w-F1
0.806 (0.0084)
0.810 (0.0084)
0.812 (0.015)
0.810 (0.0099)
0.807 (0.01)
0.817 (0.02)
MB F1
0.813 (0.0093)
0.810 (0.011)
-
0.836 (0.014)
0.810 (0.013)
-
MB m-w-F1
0.823 (0.0091)
0.819 (0.010)
-
0.845 (0.013)
0.820 (0.011)
AR accuracy
0.789 (0.0083)
0.79 (0.0070)
0.800 (0.017)
0.790 (0.013)
0.789 (0.017)
0.804 (0.02)
Table 5: Performance of BERT model variants across training configurations and validation settings (macro F1 and weighted F1 scores denoted m-F1 and w-F1, respectively). Standard deviations across 20 random trials shown in parentheses.
Training Strategy
Accuracy
Macro F1
Weighted F1
Iterative Pseudo-Labeling
81.6%
0.807
0.818
LLM Direct Classification (no explanation)
65.8%
0.481
0.648
LLM Confidence-Based Classification (k = 5)
60.9%
0.374
0.666
Table 6: Overall scores for unsupervised or semi-supervised methods.
Figure 3: Tweets per month by party
Figure 4: Distribution of labels per month for each party.
Figure 5: Distribution of labels per month by gender and race.
tweet
human label
BERT-base
AR
MB
label
confidence
We need @Senate_GOPs to join us to help small businesses grow & create jobs
problem
problem
problem
0.96
New @HHSGov report found nearly 1 in 3 people on Medicare was prescribed opioid painkillers last year
problem
problem
problem
0.96
VIDEO: Maria visits GE Aviation in #Yakima to discuss growing WA’s aerospace industry:
problem
other
problem
0.87
#Israel faces unique threats but offers amazing opportunity.
problem
other
problem
0.96
This #MemorialDay, we honor those who gave their lives for #USA & pledge to help all vets access health care, education job opportunities.