Not Worth Mentioning? A Pilot Study on Salient Proposition Annotation

Amir Zeldes; Katherine Conhaim; Lauren Levine

cs.CLMar 28, 2026

Not Worth Mentioning? A Pilot Study on Salient Proposition Annotation

Authors: Amir Zeldes, Katherine Conhaim, Lauren Levine

Organizations: Department of Linguistics, Georgetown University

Abstract

Despite a long tradition of work on extractive summarization, which by nature aims to recover the most important propositions in a text, little work has been done on operationalizing graded proposition salience in naturally occurring data. In this paper, we adopt graded summarization-based salience as a metric from previous work on Salient Entity Extraction (SEE) and adapt it to quantify proposition salience. We define the annotation task, apply it to a small multi-genre dataset, evaluate agreement and carry out a preliminary study of the relationship between our metric and notions of discourse unit centrality in discourse parsing following Rhetorical Structure Theory (RST).

Figures & tables

Genre	Docs	Tokens	EDUs	Alignments
academic	2	1,952	250	1,250
biography	2	1,679	182	910
conversation	2	1,868	342	1,710
court	2	2,075	253	1,265
essay	2	2,359	301	1,505
fiction	2	2,029	264	1,320

Table 1: GUM V12 test set data.

Figure 1: GlowLyter annotation interface for summary-wise salient proposition alignment.

	accuracy		$\kappa$
metric	micro	macro	micro	macro
strict all	92.97	92.73	65.43	64.62
strict literal	95.08	94.78	74.07	72.74
strict match	95.96	95.73	78.05	77.57
duplicates OK	96.63	96.52	83.99	82.64

Table 2: Inter-annotator agreement

Figure 2: Distribution of salience scores.

Feature	df	AIC	LRT	Pr( $\chi^{2}$ )
$<$ none $>$		-4000.5
position	1	-3962.8	39.68	2.987e-10 ***
is_sent	1	-3935.5	67.02	2.682e-16 ***
length	1	-3906.5	95.98	< 2.2e-16 ***
relation	31	-3469.0	593.46	< 2.2e-16 ***
centrality	1	-3198.2	804.28	< 2.2e-16 ***

Table 3: LRT single term deletions for features in a mixed effects beta-binomial regression model predicting salience, ranked by AIC.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Input format for the annotation interface

Figure 4: RST centrality example. Graph distance from the root is indicated in blue, e.g. $[$ d=1 $]$ means one horizontal edge away from the root, which is unit $[$ 2 $]$ .

Figure 5: Fragment of an RST tree with units shaded by salience: a score of 5=red, 3=orange, 1=yellow.

Importance Weighting

Importance weighting adjusts the influence of samples, features, or outcomes to correct distribution differences or emphasize relevant information.

See all 30 papers

The Sum of All Summaries: PROPALIGN for Graded Proposition Salience Evaluation
Oct 14, 2026Lin Ai, Emma Thronson, Katherine Conhaim +1LLM Evaluation
Department of Linguistics, Georgetown University, USA
An Interpretable Linguistically-Grounded Analysis of Scene Saliency in Movie Screenplays
Oct 11, 2026Uday Bindal, Ishaan Karan, Manan Garg +3
IIIT Hyderabad
Preserving What Matters: Semantic Scaffolds Beyond Saturation in Summarization Evaluation
Sep 18, 2026Nikhil Reddy Pottanigari, Ramin Fahimi, Noah Bolger +2LLM Evaluation Text Summarization
ServiceNow Canada

Oct 14, 2026J/K move · Enter open · S save

The Sum of All Summaries: PROPALIGN for Graded Proposition Salience Evaluation

Lin Ai, Emma Thronson, Katherine Conhaim, Amir Zeldes

Department of Linguistics, Georgetown University, USA

LLM Evaluation

Open paper AskPDF arXiv

Identification and ranking of salient propositions in documents is indirectly involved in many AI tasks, but few papers define proposition salience scores clearly, make diverse data available, or evaluate models directly on the task. This paper proposes PROPALIGN, a multiple-summary-based approach for propositional salience score estimation, and releases PROPALIGNBANK, an accompanying dataset for the method with 138K alignments of over 35K propositions. We compare several approaches to salient proposition extraction in English across a broad range of spoken and written genres, and demonstrate that naively asking models for judgments leads to inconsistencies. By contrast, graded, summary-based metrics are more stable. We further observe that while single models perform worse than humans, an ensemble of models incorporating linguistic features approaches manual annotation quality ceilings. In addition, analyses of the model outputs suggest insights into how models can extract salient information with and without the use of summaries.

Abstract

Importance Weighting

Explore similar work

The Sum of All Summaries: PROPALIGN for Graded Proposition Salience Evaluation

An Interpretable Linguistically-Grounded Analysis of Scene Saliency in Movie Screenplays

Preserving What Matters: Semantic Scaffolds Beyond Saturation in Summarization Evaluation