cs.LGOct 12, 2026

Is the Platform Part of the Measurement? A Protocol and Simulation Study for Cross-Platform Equivalence of Machine-Labelled Policy Sentiment

Authors: Asad Ikram, Ella Haig

Organizations: Independent researcher. · School of Computing, Mathematics & Physics, University of Portsmouth, UK

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Comparing machine-labelled sentiment across social media platforms is now routine: differences in classifier scores between platforms are read as differences in public opinion. These comparisons rest on an assumption that the classifier measures the same construct, on the same scale, on every platform. Some studies check a classifier's validity platform by platform, but to our knowledge none has modelled the platform as a facet of the classifier's measurement. We adapt the psychometric equivalence toolkit (generalizability theory, measurement invariance and differential item functioning, and score linking) to machine-labelled policy sentiment, treating the platform as a survey mode, and specify a six-step protocol with anchor events fixed in advance, an anchor-density audit, and a two-classifier convergence control. A simulation with known ground truth illustrates what the assumption can cost in one generating configuration fixed in advance: across 500 replications, because the platforms measure through different functions, a pooled index that takes platform scores at face value reports the wrong sign of aggregate sentiment on nearly a quarter of days, whereas an index adjusted with each platform's known measurement function does so on under one day in sixty on average. A case study of the instrument from the first author's MSc dissertation, a fine-tuned BERT classifier over 255,446 UK economic-policy comments from Twitter/X, Reddit, YouTube, and Quora, motivates the design and illustrates the instrument-dependence the protocol addresses. The full four-platform application, under fresh collection with the pre-specified design, is the designated follow-up study. The protocol lets a researcher test whether pooling platforms is defensible in a given corpus, or whether platform scores must be linked before any average is taken.

Figures & tables

Figure 1: What assuming equivalence costs, in one simulation run with known ground truth (seed 20261201; parameters fixed ex ante and released with the code; Appendix D ). The naive pooled index reports the wrong sign of aggregate sentiment on 13 of 60 days in this run; the measurement-adjusted index tracks the latent truth with no wrong sign day. Here the true event shift is −0.332-0.332 ; the naive index estimates −0.369-0.369 and the adjusted index −0.326-0.326 ; the naive shift error mainly reflects the coarse three point scale rather than non-equivalence (Section 7 ). Section 7 reports 500 replications.
ModelAccuracyMacro precisionMacro recallMacro F1
BERT (fine-tuned)88.06%0.88150.87440.8778
BiLSTM (GloVe)81.51%0.81410.80990.8111
VADER baseline65.20%0.660.650.65
Logistic regression (BoW)71.40%0.720.710.71
Table 1: Dissertation-stage classifier comparison (dissertation Table 7.1, p. 35). The BERT and BiLSTM rows match the appendix confusion matrices exactly and therefore measure agreement with the VADER-seeded labels over the full classified corpus; the reference standard for the VADER and logistic regression rows is not recorded, and their accuracies are not whole-number proportions of 200, so they cannot be accuracies over the full 200-post human set. The rows therefore cannot be assumed to share a reference standard, and no row measures fine-tuned performance against the human criterion; the dissertation’s own notebook output computes the BERT figures against the corpus labels (p. 31). Values are shown as printed in the dissertation. The application will retrain BERT and the BiLSTM by the procedure the executed code records on its own corpus and score every model against both reference standards, with binomial intervals on the human-criterion figures.
Figure 2: Sentiment shares by policy category under the fine-tuned BERT classifier, as they stood at dissertation stage (Figure 7.12, p. 41). The nine keyword-assigned categories carry per-category counts that sum to 255,446, the total the surviving pipeline records establish, and the Other/Unclear category absorbs roughly a third of classified posts.
EventDate
Budget 2020 (Covid response)11 Mar 2020
Furlough scheme announced20 Mar 2020
Budget 20213 Mar 2021
Autumn Budget 202127 Oct 2021
Energy bills package and Bank rate rise3 Feb 2022
Cost-of-living package and windfall tax26 May 2022
Table 2: Anchor events fixed on 21 August 2026, before any window count was computed. Per-platform counts in the 48-hour and two-week windows are produced by the application’s density audit on its corpus; the inclusion thresholds, fixed at the same time, are 200 comments per platform-event cell in the two-week window and 50 in the 48-hour window. The audit re-verifies each date against official records before counting.
Figure 3: Dissertation-stage NRC distribution over all comments (dissertation Figure 7.8, p. 40): eight emotion categories plus positive, negative, and neutral tallies. The bars sum to the classified corpus, so each comment carried one mutually exclusive label at dissertation stage; the application replaces this single-label assignment with binary presence codes per emotion, which become the Step 3 indicators.
Figure 4: Dissertation-stage cross-tabulation of BERT and BiLSTM predictions over the classified corpus (dissertation Figure 7.15, p. 43). Both families were trained on the same VADER-seeded labels, so their disagreement illustrates instrument-dependence within a shared label heritage; the BiLSTM is not the non-lexicon control, which is specified in the text.
Figure 5: BERT-versus-BiLSTM disagreement rate by policy category at dissertation stage (Figure 7.14, p. 43). Disagreement is highest for Trade and Brexit and Welfare and Benefits, though the spread across categories is modest, roughly 10 to 13 per cent.
Figure 6: The same corpus rescored with the BiLSTM (dissertation Figure 7.13, p. 42). The broad category ordering is nearly preserved, with adjacent categories swapping rank; measured shares are not preserved. Compare Figure 2 .
Figure 7: The naive pooled index in practice: dissertation-stage daily mean sentiment pooled across platforms, with the Prophet fit and forecast band (dissertation Figure 7.7, p. 39). Several problems with the construction are visible here: days with a single post pin the daily mean at plus or minus one, no support threshold or volume weighting is applied, the uncertainty band ignores the bounded scale, and Twitter/X enters the pool only from January 2018 (dissertation p. 21), so the platform mix behind the daily mean shifts within the series. The simulation isolates the measurement component of exactly this construction.
Figure 8: Dissertation-stage Prophet decomposition of the pooled index (dissertation Figure 7.6, p. 39): a declining long-run trend and a midweek dip that we read as a candidate composition artefact, equally consistent with weekday differences in platform mix and with mood itself.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Dissertation-stage per-class report and confusion matrix, fine-tuned BERT (dissertation Figure 7.1, p. 36). Class supports total 255,446; the reference labels are the VADER-seeded corpus labels, so these diagnostics measure agreement with the seed lexicon, not accuracy against the human criterion.
Figure 10: Per-class report and confusion matrix for the BiLSTM at dissertation stage (Figure 7.2, p. 36). The reference labels are again the VADER-seeded corpus labels, not the human criterion.
Figure 11: Agreement between the two classifier families over the classified corpus, from the executed notebook: 225,010 of 255,446 comments (88.1 per cent). The dissertation text’s figure of 85 per cent is superseded.
Figure 12: Per-comment NRC lexicon match counts for one example (executed notebook). These raw counts, not the single dominant label, are the basis of the Step 3 binary emotion indicators.
More in this topic

Social Media Analysis

Social media analysis studies online posts, media, users, and interactions to understand communication, communities, and public behavior.

See all 221 papers

Explore similar work

CardsList

  1. Quantifying Political Partisanship for Cross-Platform Analyses

    Jul 23, 2026Fathima Ameen, Christopher G. HealeySocial Media AnalysisComputational Social Science

    North Carolina State University, Raleigh, NC, USA

  2. The Model as One Rater Among Several: Measuring Political Positions in Data-Sparse Regions with a Language-Model Panel

    Jun 22, 2026Tarek GaraInter-Rater ReliabilityLLM-Assisted Annotation

    *Independent researcher.

  3. Nobody Truly Agrees on Sentiment: Humans, Bespoke Tools, and LLMs Struggle with Social Media Texts

    Oct 7, 2026Himarsha R. Jayanetti, Sivakanesan Dhanushkanda, Shuai Hao +2Sentiment AnalysisSocial Media Analysis

    Old Dominion University, Norfolk, Virginia, USA