Article and Comment Frames Shape the Quality of Online Comments

Matteo Guida; Yulia Otmakhova; Eduard Hovy; Lea Frermann

cs.CLMar 29, 2026

Article and Comment Frames Shape the Quality of Online Comments

Authors: Matteo Guida, Yulia Otmakhova, Eduard Hovy, Lea Frermann

Organizations: School of Computing and Information Systems, The University of Melbourne

Abstract

Framing theory posits that how information is presented shapes audience responses, but computational work has largely ignored audience reactions. While recent work showed that article framing systematically shapes the content of reader responses, this paper asks: does framing also affect response quality? Analyzing 1M comments across 2.7K news articles, we operationalize quality as comment health. We find that article frames significantly predict comment health while controlling for topic, and that comments that adopt the article frame are healthier than those that depart from it. Further, unhealthy top-level comments tend to generate more unhealthy responses, independent of the frame being used in the comment. Our results establish a link between framing theory and discourse quality, laying the groundwork for downstream applications. We illustrate this potential with a pro-active frame-aware LLM- based system to mitigate unhealthy discourse.

Figures & tables

Healthy	Unhealthy
"How do you spell exploitation? This is a disgusting practice that sanctions abuses of fundamental rights and human decency. We cannot continue to condone it."	"Funny how the ’free’ market is not OK in this situation. So ’not OK’ that corps lobbied the ’Harper Government’ to create a policy to circumvent the free market. How f@#ked is that? The Harper govt must go"
"Excellent article. For too long, the aboriginals’ concerns have been treated as a problem to avoid, when it should be seen as a vital part of the whole project."	"Are you kidding? The mandatory native on all the projects I did with the Feds in northern Alberta was a uniformly drunk, indolent or absent ’partner’. What a joke"

Table 1: Examples from our corpus illustrating healthy versus unhealthy comments. Note that healthy comments may include strong language or disagreement while remaining substantive and constructive; unhealthy comments may avoid explicit profanity yet undermine discourse through dismissiveness, generalisation, or bad faith, different to the widely used concept of toxicity (See Section 2.2 and Appendix B ).

	NYT	SOCC	Total
Articles	1,671	1,077	2,748
Comments	831.9K	194.8K	1.03M
Depth 0	620.9K	93.7K	714.6K
Depth 1	208.5K	69.2K	277.7K
Depth 2	2.6K	31.8K	34.4K
Depth 3+	0.05K	2.6K	2.6K

Table 2: Dataset statistics. Depth 0: top-level comments; Depth 1: replies to top-level comments; Depth 2: nested replies. Depths 3 and beyond account for only 0.25% of all comments.

Frame	Description
Economic	Costs, benefits, economic consequences, jobs, trade
Morality	Religious/ethical perspectives, moral judgments
Fairness & Equality	Equal treatment, discrimination, rights, justice
Legality & Crime ∗	Laws, constitutionality, crime, punishment
Political & Policies +	Policy prescriptions, governance, partisan framing
Security & Defense	National security, military, terrorism, border control

Table 3: Frame taxonomy. ∗ Merged from Legality & jurisprudence and Crime & punishment ; + merged from Policy prescription and Political ( Card et al., 2015 ; Guida et al., 2025 ) .

Model	Acc.	Precision	Recall	F1
LLaMA	0.59	0.50	0.50	0.49
ModernBERT	0.68	0.69	0.71	0.67
DeBERTa	0.74	0.72	0.74	0.72

Table 4: Accuracy and macro-averaged precision, recall, F1 on the re-balanced UCC test set.

Figure 1: Health by frame alignment across platforms. A clear gradient emerges where comments primary matching article frames are the healthiest, followed by selective reframing (adopting secondary frames present in the article), with complete reframing (introducing frames absent from the article) showing the lowest health. Error bars represent 95% confidence intervals.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

	Healthy	Unhealthy	Total
Train	32,848	2,655	35,503
Val	4,091	336	4,427
Test	4,105	320	4,425
Total	41,044	3,311	44,355

Appendix

Table 5: Original class distribution in UCC splits.

	Healthy	Unhealthy	Total
Train	5,298	2,649	7,947
Val	662	331	993
Test	662	331	993
Total	6,622	3,311	9,933

Appendix

Table 6: Balanced high-confidence split used for fine-tuning.

Comment Text	Toxicity	Characteristics
"A few general comments: did not read the column. Stopped reading Lady Astor long ago; interesting she did not editorialize about Chris Spence, admitted serial plagiarizer, head of TDSB; Really, who still reads her columns? A columnist who’s plagiarizes?"	0.127	Dismissive, unconstructive
"The number of lies, distortions and exaggerations in this article are many. Wonder where she got them?"	0.243	Accusatory
"America: it’s an amazing country full of a lot of fine people. But like wayward teenagers you can’t tell them anything. They know what’s best, even when it’s so obvious that their behavior is harming themselves and others. There will be no significant change in America’s gun-cult mentality. Not now, nor even if there’s 20 more shootings like Newtown. Guns are fetish objects to Americans, and they’re worshiped accordingly. Either accept that, or live and visit elsewhere."	0.188	Stereotypes, generalization
"Get with the program, Margaret, or you’ll be sent to a re-education camp. It’s ’climate change.’"	0.049	Sarcasm

Appendix

Table 7: Examples of comments classified as unhealthy (Healthy=0) but with low Perspective API toxicity scores (<0.5).

Headline	Topic	Article Frame	Secondary Frames	Comment	Comment Frame
Without Obamacare, I Will Get Sicker, Faster	Healthcare	Health	Economic, Political, Morality	My granddaughter once asked me the difference between Democrats and Republicans… ’Democrats,’ I said, ’want to help people. Republicans don’t want to help people.’	Political
The B.C. teachers’ case gets a failing grade	Education	Political and Policies	Economic, Legality and Crime	Gotta love how the second paragraph of this piece is worded in such a way that it suggests that the BC government’s attempts to illegally strip teachers of their constitutional right to bargain class size and composition actually puts it in a better light than the BCTF.	Legality and Crime

Appendix

Table 8: Examples of selective reframing where commenters adopt secondary frames present in the article.

Headline	Topic	Article Frame	Secondary Frames	Comment	Comment Frame
Trump Is Insulting Our Intelligence	Trump	Political and Policies	Legality and Crime, Morality, Cultural Identity	We keep discussing #45 as though he were sane. He is not… He has no true friends and his family don’t care… It would be tragic if he wasn’t President, but it’s simply horrifying now.	Health and Safety
Mr. Trump, Meet My Family	Immigration	Cultural Identity	Legality and Crime, Morality, Political and Policies, Security and Defense, Health and Safety, Fairness and Equality	I wonder why the folks who voted for Trump are not encouraged to move where there are jobs and retrain… Immigrants cost money from taxpayers, so why not use that money to train current Americans…	Economic

Appendix

Table 9: Examples of complete reframing where commenters introduce frames entirely absent from the article.

Topic	NYT Health	SOCC Health
Healthcare	88%	95%
Education	88%	87%
Climate Change	85%	86%
Abortion	83%	86%
Syria	83%	77%
Gun Control	79%	78%

Appendix

Table 10: Mean comment health by topic, sorted by NYT rate. Differences between topics are statistically significant (

p<0.001

) for both platforms.

Figure 2: Comment health by frame alignment (Match, Diff in Article, Never in Article) across topics for NYT (top) and SOCC (bottom). The dashed line represents overall health.

Frame Alignment (vs Match)
	NYT				SOCC
Predictor	Est.	SE	$z$	$p$	Est.	SE	$z$	$p$
(Intercept)	1.727	0.099	17.37	$<0.001$	1.862	0.173	10.77	$<0.001$
Selective	$-0.108$	0.009	$-11.94$	$<0.001$ ***	$-0.132$	0.029	$-4.51$	$<0.001$ ***
Complete	$-0.298$	0.018	$-16.31$	$<0.001$ ***	$-0.289$	0.044	$-6.62$	$<0.001$ ***
Article Topic (vs Abortion)

Appendix

Table 11: Mixed-effects logistic regression predicting comment health from frame alignment, NYT and SOCC.

Article Frame (vs Cultural)
	NYT				SOCC
Predictor	Est.	SE	$z$	$p$	Est.	SE	$z$	$p$
(Intercept)	1.608	0.108	14.90	$<0.001$	1.734	0.196	8.85	$<0.001$
Economic	$-0.104$	0.098	$-1.06$	0.288	0.185	0.097	1.92	0.055
Fairness	$-0.560$	0.275	$-2.04$	0.042*	$-0.544$	0.195	$-2.79$	0.005**
Health	0.390	0.085	4.61	$<0.001$ ***	0.051	0.122	0.42	0.673

Appendix

Table 12: Mixed-effects logistic regression predicting comment health from article frames, NYT and SOCC.

Table 15

Top Comment Health
	NYT				SOCC
Predictor	$\beta$	SE	$t$	$p$	$\beta$	SE	$t$	$p$
(Intercept)	0.706	0.019	37.39	$<0.001$	0.633	0.029	21.49	$<0.001$
Healthy (vs Unhealthy)	0.095	0.018	5.38	$<0.001$ ***	0.126	0.023	5.40	$<0.001$ ***
Top Comment Frame (vs Cultural)
Economic	0.026	0.023	1.14	0.256	0.070	0.028	2.48	0.013*

Appendix

Table 15: Linear regression predicting mean reply health from top comment health, frame, and topic, NYT and SOCC.

Component	Input / Function	Implementation Details
Article Analysis
Text Processing	Full article text	Sentence tokenization (NLTK punkt )
Frame Classification	Sentence-level frame detection	Fine-tuned RoBERTa model ( Guida et al., 2025 ) (10-frame taxonomy)
Output	Sentence-level frame labels	Each sentence annotated as [{frame, confidence}]
Comment Analysis
Text Processing	Comment text	Sentence tokenization (NLTK punkt )

Appendix

Table 16: System architecture for frame-aware comment moderation. Articles and comments are analyzed separately for framing and health, combined into a risk assessment, and used to generate context-aware interventions via an LLM.

Figure 4: Landing page with topic search functionality. Users can enter keywords or select predefined topics to retrieve 3 relevant articles from The Conversation. Article view with sentence-level frame analysis. Detected frames are displayed as tags; hovering over a frame highlights corresponding sentences in the article text.

Figure 5: Moderation output for an unhealthy comment expressing climate skepticism. The system assigns a high risk level based on healthiness score, detects frames and retention, and recommends refusing the comment. The LLM generates three ways to reformulate more constructively while preserving the user’s core concern and idea.

FramingQA: Does the Question Shape the Answer? Measuring the Compositional Framing Effect
Sep 7, 2026Hazel H. Kim, Andrew M. Bean, Guilherme Affonso Ferreira de Camargo +10LLM Evaluation LLM Reliability
Department of Computer Science, University of Oxford · Oxford Internet Institute, University of Oxford · Thomson Reuters +7
Framing by Wording, Framing by Selection: A Large-Scale Two-Dimensional Audit of French News Headlines, 2022-2025
Aug 12, 2026Amr SobhyPolitical Discourse Analysis
Le French News Lab
Recognizing Is Not Reversing: A Controlled Inversion Test of Fact-Preserving News Framing
Sep 10, 2026Yi LiuLLM Evaluation Framing Effects in Language Models
University of Science and Technology of China

Sep 7, 2026J/K move · Enter open · S save

FramingQA: Does the Question Shape the Answer? Measuring the Compositional Framing Effect

Hazel H. Kim, Andrew M. Bean, Guilherme Affonso Ferreira de Camargo, Shanyu Chauhan+9

Department of Computer Science, University of Oxford · Oxford Internet Institute, University of Oxford · Thomson Reuters+7

LLM Evaluation LLM Reliability Framing Effects in Language Models Language Model Bias Evaluation Language Model Robustness

Open paper AskPDF arXiv

We introduce FramingQA, a benchmark that measures the model sensitivity to question framing across law, medicine, finance, and robotic simulations. Large language models (LLMs) often change their responses to subtle rephrasings that align with an implied stance by users. This can leave users with advice tainted by how they happened to phrase a question rather than by the underlying facts, and the consequences are highly costly in high-stakes domains. Because in the realistic scenarios, both expert practitioners and non-expert users frequently ask LLMs questions containing incomplete or misleading assumptions, models are highly susceptible to those framings. To test this, we inject the framing bias across three nested levels: a framing-biased question phrasing (root), an injected framing-biased premise prepended to a neutral question (propositional), and a premise paired with a framing-biased question (global). Evaluating nine open models (3.8B-70B) across four families, we find that strong per-variant accuracy does not guarantee the robustness across differently phrased questions under the fixed factual information.

Abstract

Explore similar work

FramingQA: Does the Question Shape the Answer? Measuring the Compositional Framing Effect

Framing by Wording, Framing by Selection: A Large-Scale Two-Dimensional Audit of French News Headlines, 2022-2025

Recognizing Is Not Reversing: A Controlled Inversion Test of Fact-Preserving News Framing