Beyond Effectiveness: A Multi-Criteria Framework for Comparing Practical Socio-Technical Interventions

Catherine King; Lynnette Hui Xian Ng; Kathleen M. Carley

cs.AIAug 21, 2026

Beyond Effectiveness: A Multi-Criteria Framework for Comparing Practical Socio-Technical Interventions

Authors: Catherine King, Lynnette Hui Xian Ng, Kathleen M. Carley

Organizations: Carnegie Mellon University Pittsburgh, USA

Abstract

Designers and policymakers in sociotechnical domains like content moderation, privacy interfaces, recommender systems and beyond, must choose among a growing menu of proposed interventions, but typically lack a principled basis for comparing them. Prior work tends to evaluate interventions individually and mostly along the effectiveness criteria, while implementation constraints such as cost, effort and feasibility are often considered separately. We present a multi-criteria framework for evaluating sociotechnical interventions. This framework is instantiated through the case of misinformation, a domain of intense focus for proposed countermeasures. We survey $N=39$ researchers on 40 operationalized interventions across five evaluative criteria: political feasibility, effectiveness, user acceptance, cost, and implementation effort. We find that the interventions that experts judge to be the most effective are not always the most acceptable to the public or the most feasible to implement. We also discuss how this tension has implications for the design of sociotechnical interventions beyond misinformation, and offer a decision framework for practitioners navigating the trade-offs of sociotechnical interventions.

Figures & tables

Figure 1. The information lifecycle ( King et al., 2026 ) and corresponding intervention categories. The information lifecycle and corresponding intervention categories. The creation step is associated with account and content moderation. The spread step is associated with content moderation, distribution, and generative AI-specific interventions. The belief step is associated with content labeling and user-based measures. Prevention can affect all the three steps and includes media literacy and education. Institutional measures are external to the lifecycle.

Information Lifecycle	Intervention Category	Definition
Creation : accounts, content	Account Moderation	The moderation of user accounts such as by suspending, banning, or limiting users
Spread : viral amplification, individual sharing	Content Moderation	The moderation of content such as by removing, downranking, or correcting it
	Content Distribution	Affecting the distribution/ sharing of content such as by using redirection, accuracy prompts, or friction
	Generative AI-specific	Employing genAI to address harmful or low-quality content (e.g., using chatbots to dialogue with users, using genAI to produce fact checks)
Belief : claim verification, correction	Content Labeling	The use of labeling or disclosure to notify users, provide additional context, or verify claims
	User-based Measures	Relating to user-driven responses to the information, such as social corrections or reporting

Table 1. Categories of Socio-Technical Interventions, based on the information lifecycle ( King et al., 2026 )

Category	Response	N	%
Gender	Women	16	41.0
	Men	22	56.4
	Other / prefer not to say	1	2.6
Age	18–34	15	38.5
	35–44	12	30.8
	45+	10	25.6

Table 2. Participant Demographics (N = 39)

Figure 2. Stacked bar plot showing the percentage of responses on the effectiveness and acceptance of the general intervention categories. Stacked bar plot showing the percentage of responses on the effectiveness and acceptance of the general intervention categories. The sub-figure on the left shows the responses for effectiveness, with the percentage of experts who at least somewhat agree that category of interventions would be effective highlighted. Content moderation, account moderation, and media literacy score above 70\%, content labeling and distribution above 60\%, user-based measures at around 50\% and genAI at 28\%. The sub-figure on the right shows the responses for acceptance, with the percentage of experts who at least somewhat agree that category of interventions would be acceptable to users highlighted. Media literacy, user-based measures, and content labeling are above 70\%. Content moderation, distribution, and institutional measures are around 50\% , and account moderation and genAI are just below 40\%. — Figure 2. Stacked bar plot showing the percentage of responses on the effectiveness and acceptance of the general intervention categories. Stacked bar plot showing the percentage of responses on the effectiveness and acceptance of the general intervention categories. The sub-figure on the left shows the responses for effectiveness, with the percentage of experts who at least somewhat agree that category of interventions would be effective highlighted. Content moderation, account moderation, and media literacy score above 70%, content labeling and distribution above 60%, user-based measures at around 50% and genAI at 28%. The sub-figure on the right shows the responses for acceptance, with the percentage of experts who at least somewhat agree that category of interventions would be acceptable to users highlighted. Media literacy, user-based measures, and content labeling are above 70%. Content moderation, distribution, and institutional measures are around 50% , and account moderation and genAI are just below 40%.

Figure 3. Average effectiveness and acceptance Likert scores per general intervention category. Average effectiveness and acceptance Likert scores per general intervention category. Average effectiveness and acceptance Likert scores per general intervention category. Account moderation and user-based measures are the only categories where the 95\% confidence intervals of the average effectiveness and acceptance Likert scores do not overlap, with account moderation having higher effectiveness scores and user-based measures having higher acceptance scores. — Figure 3. Average effectiveness and acceptance Likert scores per general intervention category. Average effectiveness and acceptance Likert scores per general intervention category. Average effectiveness and acceptance Likert scores per general intervention category. Account moderation and user-based measures are the only categories where the 95% confidence intervals of the average effectiveness and acceptance Likert scores do not overlap, with account moderation having higher effectiveness scores and user-based measures having higher acceptance scores.

Figure 4. A heatmap showing the average score for each metric across general intervention categories. A heatmap showing the average score for each metric across general intervention categories. A heatmap showing the average score for each metric across general intervention categories. Most categories perform better on certain criteria than others. The most well-balanced category, and the only one that averages above a 3 on the Likert scale for each criterion, is content distribution.

Intervention Rank	Category	Score	Intervention Rank	Category	Score
1. Government Labels	Content Labeling	4.04	21. Virality Circuit Breakers	Content Moderation	3.23
2. Limit Resharing	Content Distribution	3.91	22. Ban Deepfakes	Content Moderation	3.21
3. Limit Forwarding	Content Distribution	3.91	23. Platform Alterations	Content Distribution	3.20
4. Crowdsourcing	Content Labeling	3.76	24. Data Sharing	Institutional Measures	3.20
5. AI Disclosure	Content Labeling	3.62	25. Warning Labels	Content Labeling	3.18
6. Friction	Content Distribution	3.55	26. Redirection	Content Distribution	3.16

Table 3. Interventions ranked by adjusted average score. The top-ranked intervention in each category is bolded.

	Category	Intervention	N	Feasibility	Effectiveness	Acceptance	Cheapness	Ease	Overall
Balanced high performers	Account Moderation	Demonetization	12	3.58 (1.31)	4.00 (0.60)	3.75 (1.06)	2.92 (1.51)	2.92 (1.24)	3.43 (0.52)
	Content Moderation	Algorithmic Changes	11	3.64 (1.03)	4.09 (0.54)	3.55 (1.04)	3.00 (1.26)	2.55 (0.93)	3.36 (0.54)
	Content Distribution	Accuracy Prompts	8	3.38 (1.51)	3.62 (0.92)	3.50 (1.07)	2.88 (1.13)	3.38 (1.06)	3.35 (0.75)
	Content Distribution	Limit Forwarding	13	4.08 (0.86)	3.69 (0.63)	3.54 (1.05)	4.31 (0.85)	3.92 (1.04)	3.91 (0.49)
	Content Distribution	Limit Resharing	9	4.00 (1.22)	3.78 (0.44)	3.00 (1.22)	4.56 (0.53)	4.22 (0.67)	3.91 (0.45)
	Content Labeling	Fact-check Labels	15	3.53 (1.06)	3.87 (0.74)	3.80 (0.94)	2.79 (1.12)	2.21 (1.12)	3.26 (0.59)

Table 4. Interventions sorted by cluster and category with mean scores and standard deviations (in parentheses). Cluster mean rows show the average across interventions in that cluster.

Cluster	When to choose it	Example interventions	Design considerations
1 Well-balanced, high-performing	Needs broadly viable wins without requiring unusual budget or political cover	Demonetization † ; limits on resharing/forwarding; fact-check and government labels; reporting; AI disclosure labels	Preserves users’ ability to interpret, contest, or act on information rather than removing the choice for them. Pair with at least one creation-phase measure from cluster 2 and invest in increasing that measure’s acceptance
2 Effective and affordable, but low acceptance	Has an enforcement mandate or legal cover and can absorb public pushback	Account suspension † ; deplatforming † ; shadow banning † ; downranking; content removal	Acceptance cost is addressable, not fixed. Invest in transparent criteria, reduced opacity, recourse options, and error minimization
3 Effective and acceptable, but resource-intensive	Has a long time horizon, coalition partners, or dedicated funding	Media literacy programs; journalism/media support; data sharing; regulation	Share cost across a coalition (platform + civil society + government); look for delivery channels that amortize costs
4 Emerging, straightforward, and low-cost	Needs to build public trust or political capital before bigger moves	Crowdsourcing; user control; deepfake/AI-in-ads regulation	Useful as a legitimacy-building first step or complement, designed with user agency in mind. Treat as an initial testbed to iterate and improve effectiveness, not as the primary countermeasure

Table 5. A practitioner decision framework for selecting misinformation interventions, mapped to the four empirically derived clusters in Table 4 . † marks creation-phase interventions.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Courchesne et al. (2021) ( Courchesne et al., 2021 )	Aghajari et al. (2023) ( Aghajari et al., 2023 )	Blair et al. (2024) ( Blair et al., 2024 )	Kozyreva et al. (2024) ( Kozyreva et al., 2024 )
Account Moderation	$\times$
Account Removal		$\times$
Accuracy Prompts		$\times$	$\times$	$\times$
Advertising Policy	$\times$
Content Distribution	$\times$
Content Labeling	$\times$

Appendix

Table 6. Categorizations of misinformation interventions in the literature.

Specified Intervention Implementations	Intervention Type(s)
Account Moderation
Account Suspensions - Permanently or temporarily ban certain users.	Account Suspension
Deplatforming - Coordinated efforts to ban prominent misinformation spreaders.	Account Suspension
Shadow Banning - Limit the visibility and the spread of posts from certain policy-violating accounts.	Shadow Banning
Demonetization - Remove or restrict monetization features for a user account.	Demonetization
Content Moderation

Appendix

Table 7. Operationalized misinformation interventions across the eight general categories.

Multi-Criteria Decision Analysis

Multi-Criteria Decision Analysis (MCDA) ranks or selects alternatives by balancing competing criteria, preferences, uncertainty, and constraints.

See all 35 papers

"If It Looks Like a User": Measuring Real-Time Moderation Effects via Social Media Simulation
Aug 17, 2026Enrico Verdolotti, Gianluca Nogara, Luca Luceri +1Content Moderation Social Media Analysis
Università della Svizzera italiana (USI), Lugano, Switzerland · SUPSI, DTI-ISIN, Viganello, Switzerland · USC Information Sciences Institute, Marina Del Rey, CA, USA
Not All Eval-Awareness Is Equal: Capabilities Framing Predicts Compliance
Aug 27, 2026Allison Zhuang, Santiago AranguriEvaluation Awareness in Language Models LLM Safety Evaluation
ENS Paris-Saclay · Goodfire AI
InterveneBench: Benchmarking LLMs for Intervention Reasoning and Causal Study Design in Real Social Systems
Mar 16, 2026Shaojie Shi, Zhengyu Shi, Lingran Zheng +15Causal Reasoning in Language Models Causal Inference
Fudan University · Shanghai Innovation Institute

Aug 17, 2026J/K move · Enter open · S save

"If It Looks Like a User": Measuring Real-Time Moderation Effects via Social Media Simulation

Enrico Verdolotti, Gianluca Nogara, Luca Luceri, Silvia Giordano

Università della Svizzera italiana (USI), Lugano, Switzerland · SUPSI, DTI-ISIN, Viganello, Switzerland · USC Information Sciences Institute, Marina Del Rey, CA, USA

Content Moderation Social Media Analysis Agent-Based Modeling

Open paper AskPDF arXiv

Agent-based social media simulators offer a controlled environment to study content moderation, yet their value hinges on how faithfully they reproduce real platform dynamics. We develop a calibrated extension of SimSoM, an agent-based model of information diffusion on social networks, grounded in a real-world dataset of online vaccine discourse during the COVID-19 pandemic. Our approach replaces ad-hoc parametrisations with empirically fitted distributions, optimised via CMA-ES (Covariance Matrix Adaptation Evolution Strategy) and validated against real data across temporal, distributional, and structural dimensions. Using this validated simulator, we provide three key contributions. First, we show that the calibrated model reproduces key statistical signatures of the empirical data, including activity distributions, post/reshare ratios, and temporal patterns. Second, we apply established misinformation-spreader detection and prevention methods to both empirical and simulated data, progressively removing top-ranked users and showing that the resulting decline in low-quality content is consistent across the two. Third, comparing static (retroactive) and dynamic (in-simulation) moderation across 30 network realisations, we show that static evaluation significantly overestimates the effectiveness of user bans for the most effective detectors: when moderation is applied in real time, compensatory resharing by the remaining users dampens the expected reduction in low-quality content, so static estimates should be read as an upper bound. These findings highlight the necessity of simulation-based evaluation for content moderation policies and contribute a reusable, empirically grounded simulation framework.

Abstract

Multi-Criteria Decision Analysis

Explore similar work

"If It Looks Like a User": Measuring Real-Time Moderation Effects via Social Media Simulation

Not All Eval-Awareness Is Equal: Capabilities Framing Predicts Compliance

InterveneBench: Benchmarking LLMs for Intervention Reasoning and Causal Study Design in Real Social Systems