cs.SISep 20, 2026
SaveFrom Web(logs) to Web(AI): Questions, Platforms, and Methods across Twenty Editions of ICWSM
Organizations: Siebel School of Computing and Data Science University of Illinois Urbana-Champaign, Urbana, IL, USA
Abstract
Over twenty editions, the ICWSM community has examined social life online as platforms, interactions, and research methods have changed. What can this body of research tell us at this critical juncture, as AI increasingly reshapes how people communicate online? We analyzed 2,139 indexed contributions from 2007 to 2026, distinguishing topics identified through nonnegative matrix factorization from problem framings captured through explicit textual cues. We find that platform mentions shift from blogs toward Twitter and, more recently, Reddit. Online community research maintains a similar topic share (10.5% to 10.0%), but governance cues within it increase from 4.0% to 34.3%. Harm-related cues also increase after restricting abstracts to a fixed length. Our review also traces advances in sampling, measurement, and causal and experimental methods. We discuss how AI-mediated interactions complicate these questions and provide a reporting checklist to support research across changing platforms.
Figures & tables
Figure 1: Selected annual topics (A) and problem-framing cues (B). Topic shares describe mean NMF mixture weights; framing-cue rates describe overlapping textual matches. Bands are 95% within-edition resampling intervals. They reflect composition sensitivity under fixed measures; they do not assess semantic validity.
Figure 2: Community research and governance framing. Panel A compares the community topic’s share of all topic weight with the fraction of community-topic weight attached to papers containing governance cues; these series have different denominators. Panel B repeats the conditional rate after removing governance-matching features before projection onto the fixed topic basis. Bands describe composition sensitivity under the corresponding fixed representation.
| Measure | 2007–2011 | 2012–2016 | 2017–2021 | 2022–2026 |
|---|---|---|---|---|
| Contributions ( ) | 386 | 503 | 488 | 762 |
| Networks / diffusion | 16.6 | 13.2 | 10.1 | 7.8 |
| Politics / polarization | 3.8 | 4.8 | 8.7 | 11.8 |
| Communities / participation | 10.5 | 9.2 | 10.0 | 10.0 |
| Health / social support | 2.8 | 4.4 | 6.2 | 6.3 |
| Prediction / classification cues | 36.3 | 42.9 | 47.7 | 49.1 |
Table 1: Selected measures across the same four five-edition periods. All values except are percentages. The first four rows of measures are mean topic weights; the next six are overlapping cue prevalences. The last row is conditional on community-topic weight. Headline resampling intervals are reported in the text.
Figure 3: Platform mentions in titles and abstracts for selected long-running contexts, with 95% composition-resampling bands. Counts can overlap across platforms. The full dictionary contains twenty named families plus weblog terms (Appendix A ).
| Design | ICWSM Example | Requires | Leaves open |
| Randomized intervention | Reconsideration prompts on Twitter ( Katsaros, Yang, and Fratamico 2022 ) | Random assignment supports a comparison of eligible users assigned to receive the prompt. | Classifier-defined eligibility and outcomes; transfer beyond English replies and the study setting. |
| Quasi-random assignment | Civil communication on Roblox ( Liu et al. 2024 ) | Residual stochasticity in matchmaking assignment acts as a quasi-randomization mechanism for the analyzed server comparisons. | Other server properties may co-vary with civility; validity and transfer of the local comparison. |
| Matching and difference-in-differences | Fringe-platform participation ( Russo et al. 2023 ) | Comparable untreated trends after matching; no differential concurrent changes explaining the contrast. | Selective migration, cross-platform linkage, and changing composition or exposure. |
| Regression discontinuity / interrupted time series | Conversation threading after a platform interface change ( Aragón, Gómez, and Kaltenbrunner 2017 ) | A defined intervention point supports comparison of outcomes before and after the platform change. | Concurrent changes and time-varying factors may also explain the observed discontinuity. |
| Text-adjustment evaluation | Known-effect benchmark tasks ( Weld et al. 2022 ) | Effects are known under the benchmark’s construction, enabling comparisons of estimators. | Performance under the benchmark need not transfer to an unknown real-world assignment process. |
| Stratified propensity-score matching | Support language ( De Choudhury and Kıcıman 2017 ) , college alcohol mentions ( Kıcıman, Counts, and Gasser 2018 ) , psychiatric medication use ( Saha et al. 2019 ) , and algorithmic rank movement ( Chan et al. 2026 ) | Conditional exchangeability, overlap, consistent exposure definitions, and appropriate timing. | Unmeasured circumstances and linguistic proxies; balance alone cannot establish exchangeability. |
Table 2: Causal-inference designs in ICWSM research, with the conditions each design requires and the questions it leaves open. Examples illustrate each design’s use.
| Reporting item | Information to provide |
|---|---|
| Target population | Who or what is the study about? Describe observed users, communities, languages, dates, and exclusions; distinguish the sample from the target population. |
| Question and estimand | State the descriptive quantity or causal contrast, unit of analysis, exposure, outcome, and time horizon. Specify whose effect is estimated. |
| Constructs and measures | Explain why each trace or label measures the intended construct. Report validation, disagreements, missingness, and relevant subgroup or period differences. |
| Collection and provenance | Record interfaces, sampling rules, access restrictions, deletion or attrition, and data and model versions. Document content-production processes where known. |
| Assignment and timing | Describe how exposure arises, the comparison group, and the ordering of covariates, treatment, and outcomes. Identify concurrent events and possible spillovers. |
| Adjustment and overlap | Justify pretreatment covariates, including text features. Report balance, overlap, exclusions, and any change in the target population after adjustment. |
Table 3: A reporting checklist for studies using social media observations. Items should be applied to the study’s question and design, with non-applicable items explained. The checklist is a reporting aid, not a quality score or a substitute for validation.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1: Annual contributions in the retrieved corpus.
| ID | Interpretive label | Twenty highest-weight terms |
|---|---|---|
| T00 | Networks, diffusion, recommendation | diffusion, influence, link, graph, structure, recommendation, nodes, problem, prediction, links, cascades, node, systems, graphs, real, items, spread, popularity, link prediction, algorithms |
| T01 | Conventional classification | learning, classification, features, detection, machine, machine learning, performance, task, accuracy, supervised, art, state art, training, state, framework, labeled, reviews, f1, problem, existing |
| T02 | News and journalism | news, articles, news articles, article, outlets, sources, news outlets, fake, news sources, coverage, fake news, stories, news article, comments, misinformation, bias, source, readers, sharing, local news |
| T03 | Language modeling and generative AI | llms, language, llm, ai, human, language llms, generated, gpt, tasks, fine, performance, text, reasoning, english, languages, framework, bias, biases, natural language, detection |
| T04 | Pandemics and vaccination | covid, covid 19, 19, pandemic, 19 pandemic, vaccine, anti, misinformation, vaccination, vaccines, related, impact, health, stance, concerns, narratives, public, understand, population, conspiracy |
| T05 | Politics and polarization | political, election, accounts, elections, polarization, discourse, party, politicians, partisan, campaigns, presidential, public, parties, leaning, engagement, right, political discourse, campaign, topics, candidates |
Table A1: NMF components and top terms, ordered by component identifier.
Figure A2: Complete annual topic shares and framing-cue prevalences, ordered by endpoint-period change. Topic shares sum to 100%; framing categories overlap. No historical phase boundaries are imposed.
| Category | Definition and scope limitation | Example cues |
|---|---|---|
| P1 Structure and diffusion | Explicit framing of relational structure, diffusion, contagion, social influence, collective dynamics, or community formation. Does not identify every explanatory study of collective behavior. | diffusion; cascades; social ties; homophily |
| P2 Prediction and classification | Explicit prediction, forecasting, classification, detection, or recommendation terminology. May match a prediction or detection task that a paper critiques rather than performs. | prediction; forecasting; classification; detection |
| P3 Measurement and validity | Explicit validation, sampling, representation, reliability, ground truth, reproducibility, or measurement-error terminology. Routine uses of the verb measure alone do not qualify. Validation and reliability can refer to routine evaluation or background discussion, not a central measurement contribution. | validity; sampling; ground truth; reliability |
| P4 Explanation and intervention | Explicit causation, intervention, recognizable experimental-design terminology, causal/social mechanisms, or theory testing. Generic mechanism and theoretical implication mentions are excluded. Theory, mechanism, or causal language does not establish a valid causal design. | causal; intervention; randomized; propensity score |
| P5 Harm and information integrity | Explicit information-integrity, abuse, discrimination, privacy-risk, spam, or other harm framing. Includes descriptive and technical work about harm, without inferring the authors’ normative commitments. | misinformation; spam; harassment; privacy; harm |
| P6 Governance and participation | Explicit moderation, governance, censorship, enforcement, platform rules, or community norms. Does not capture every study of participation; moderation can also be mentioned as background. | moderation; governance; censorship; community rules |
Table A2: Definitions and example lexical cues for the eight overlapping framing cues.
| Measure | 2007–11 | 2012–16 | 2017–21 | 2022–26 |
|---|---|---|---|---|
| Harm: full title/abstract | 6.2 | 12.3 | 27.0 | 38.1 |
| Harm: title + first 100 abstract words | 5.7 | 9.5 | 22.7 | 31.9 |
| Harm: first 100 abstract words only | 5.7 | 8.7 | 21.9 | 31.0 |
| Harm: historical union | 6.2 | 12.3 | 27.9 | 38.3 |
| Harm: mean matches / 100 words | 0.10 | 0.23 | 0.54 | 0.78 |
| Governance within community topic | 4.0 | 1.2 | 14.4 | 34.3 |
Table A3: Sensitivity estimates. All rows except cue density are percentages.
| Topic | P1 | P2 | P3 | P4 | P5 | P6 | P7 | P8 |
|---|---|---|---|---|---|---|---|---|
| T00 | 2 | 3 | 0 | 0 | 0 | 0 | 0 | 0 |
| T01 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 0 |
| T02 | 0 | 0 | 0 | 0 | 2 | 0 | 0 | 0 |
| T03 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 |
| T04 | 0 | 0 | 0 | 0 | 2 | 0 | 7 | 0 |
| T05 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
Table A4: Counts of top-twenty topic terms matched by each framing lexicon. These are literal term matches, not measures of semantic independence.
| Boundary start | Baseline 200; min. 3 | New 200; min. 3 | Same new 200; min. 2 |
|---|---|---|---|
| 2009 | – | – | 21.5 |
| 2010 | 39.5 | 39.0 | 33.0 |
| 2011 | 56.5 | 59.0 | 54.5 |
| 2012 | 3.5 | 1.5 | 28.0 |
| 2013 | 19.5 | 20.5 | 5.0 |
| 2014 | 16.5 | 13.0 | 8.0 |
Table A5: Exact-year boundary recurrence (percent) under five-segment joint profiles. A dash marks a boundary disallowed by the minimum-length rule. The two new columns use identical resampled paper compositions.