Identification and ranking of salient propositions in documents is indirectly involved in many AI tasks, but few papers define proposition salience scores clearly, make diverse data available, or evaluate models directly on the task. This paper proposes PROPALIGN, a multiple-summary-based approach for propositional salience score estimation, and releases PROPALIGNBANK, an accompanying dataset for the method with 138K alignments of over 35K propositions. We compare several approaches to salient proposition extraction in English across a broad range of spoken and written genres, and demonstrate that naively asking models for judgments leads to inconsistencies. By contrast, graded, summary-based metrics are more stable. We further observe that while single models perform worse than humans, an ensemble of models incorporating linguistic features approaches manual annotation quality ceilings. In addition, analyses of the model outputs suggest insights into how models can extract salient information with and without the use of summaries.
Figures & tables
Figure 1: Asking models for salient text is more subjective and less consistent than asking what appears in a summary. Multiple summaries can compensate for summary subjectivity and yield gradient salience scores, and model judgments overlap more on the right.
Genre
Source
Docs
EDUs
Alignments
Genre
Source
Docs
EDUs
Alignments
Academic writing
Various
18
1,981
9,905
How-to guides
wikiHow
19
2,399
11,995
Biographies
Wikipedia
20
2,071
10,355
Interviews
Wikinews
19
2,408
12,040
Vlogs
YouTube
15
2,438
12,190
Letters
Various
18
2,102
10,510
Conversations
UCSB Corpus
15
3,083
15,415
News stories
Wikinews
24
1,912
9,560
Courtroom transcripts
Various
14
2,143
10,715
Podcasts
Various
14
2,256
11,280
Essays
Various
14
2,001
10,005
Political speeches
Various
15
1,913
9,565
Table 1: Overview of the PropAlignBank data from GUM .
Setting
F1
Baseline zero-shot
0.464
Baseline one-shot
0.638
Summary zero-shot
0.716
Summary one-shot
0.767
Table 2: Inter-model agreement between GPT and Claude, measured by F1 across experimental settings.
Setting
Model
P
R
F1
Baseline zero-shot
GPT
0.742
0.161
0.264
Claude
0.594
0.332
0.426
Qwen
0.720
0.089
0.159
Baseline one-shot
GPT
0.548
0.451
0.495
Claude
0.517
0.517
0.517
Qwen
0.575
0.293
0.388
Table 3: Precision (P), recall (R), and F1 across models and experimental settings, evaluated against aggregated human gold labels and output IDs.
Setting
Model
P
R
F1
MAE
RMSE
Summary zero-shot
GPT
0.401
0.717
0.515
0.755
1.38
Claude
0.568
0.639
0.601
0.466
0.996
Qwen
0.507
0.392
0.431
0.533
1.035
Summary one-shot
GPT
0.480
0.679
0.563
0.583
1.19
Claude
0.572
0.629
0.599
0.452
0.993
Qwen
0.478
0.504
0.496
0.515
0.973
Table 4: Precision (P), recall (R), F1, mean absolute error (MAE), and root mean squared error (RMSE) across experimental settings, the ensemble, and human performance for the fine-grained evaluation scheme.
Figure 2: Precision, recall and F1 for three models on written and spoken test data.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Fragment from an RST tree in the GUM corpus.
Genre
Dev example (summary)
Match
Approx.
Duplicate
Component
Types
academic
librarians (1)
7
4
1
0
3
bio
byron (2)
4
6
2
3
4
conversation
grounded (3)
4
7
4
2
4
court
loan (5)
6
4
4
3
4
essay
tools (4)
2
5
2
2
4
fiction
beast (2)
4
7
2
3
4
Appendix
Table 5: Development-set examples used for one-shot in-genre prompting on the test set. Document names omit the prefix GUM_<genre>_ . Types counts categories with a nonzero value.
Prompting
Normalized
Unnormalized
Total
Zero-shot
58
102
160
One-shot
11
149
160
Total
69
251
320
Appendix
Table 6: Qwen test-set prediction files by whether at least one predicted EDU ID required normalization.
Setting
Model
P
R
F1
Summary zero-shot
GPT
0.323
0.509
0.395
Claude
0.345
0.256
0.294
Summary one-shot
GPT
0.395
0.479
0.433
Claude
0.411
0.328
0.365
Appendix
Table 7: Precision (P), recall (R), and F1 for the subtype-aware fine-grained evaluation scheme. Extra predicted EDUs are false positives, while a salient EDU assigned the wrong subtype is counted once as a false negative.
Figure 4: Absolute error distance from gold salience by model. Qwen models have more errors than any model except GPT zero shot, but disproportionately many off-by-one errors, usually between 0 and 1, resulting in better RMSE than expected, but worse MAE. Claude one-shot is the best model by F1 score except for the ensemble.
Parameter
Search space
Number of trees
{200,300,…,800}
Max feats per split
{sqrt,log2,0.3,0.5,0.7}
Min samples per leaf
Integers in [1,20]
Imbalance strategy
{none, class weight, balanced subsample}
Positive-class weight
Log-uniform on [1,max(1.5r,2)]
Threshold
Uniform on [0.1,0.6]
Appendix
Table 8: Hyperparameter search space for the Random Forest model.
Parameter
Search space
Max depth
Integers in [3,10]
Learning rate
Log-uniform on [0.01,0.3]
Row subsample
Uniform on [0.5,1.0]
Column subsample
Uniform on [0.5,1.0]
Min child weight
Integers in [1,10]
Boosting rounds
{100,150,…,600}
Appendix
Table 9: Hyperparameter search space for the XGBoost model.
Figure 5: Variable importance for Random Forest, calculated on dev .
Figure 6: Variable importance for XGBoost, calculated on dev .
Feature
Description
position
Normalized position of the EDU within its document
depth
Normalized depth of the EDU in the discourse (RST) tree
parent_similarity
Similarity between the EDU and its parent EDU
len
Token length of the EDU
relname
RST discourse relation connecting the EDU to its parent
mention_count
Number of entity mentions whose span overlaps EDU
Appendix
Table 10: All possible features used for EDU-level salience classification. Features kept after pruning on dev are bolded.
Figure 7: Distribution of percentage of errors across EDUs in a document.
Figure 8: Distribution of percentage of errors across EDUs in a document by error type.
Despite a long tradition of work on extractive summarization, which by nature aims to recover the most important propositions in a text, little work has been done on operationalizing graded proposition salience in naturally occurring data. In this paper, we adopt graded summarization-based salience as a metric from previous work on Salient Entity Extraction (SEE) and adapt it to quantify proposition salience. We define the annotation task, apply it to a small multi-genre dataset, evaluate agreement and carry out a preliminary study of the relationship between our metric and notions of discourse unit centrality in discourse parsing following Rhetorical Structure Theory (RST).
Summarization ships in countless production systems, making model selection a routine decision that depends on measuring summary quality. Existing metrics struggle to support this: ROUGE captures only surface overlap, while LLM-as-judge scores saturate to near-identical values that fail to rank models effectively. We observe this saturation across three public datasets, two proprietary datasets, and multilingual settings. Motivated by this, we introduce Semantic Scaffold, an evaluation framework that extracts a hierarchical representation of facts, questions, and entity attributes from a source text, labeling each as a main point or supporting detail, and reusing this structure as a fixed reference for scoring summaries. From this representation, we derive three diagnostic metrics: Fact Preservation Score (FPS), Question Preservation Score (QPS), and Entity Preservation Score (EPS), designed to reward the preservation of essential information while penalizing detail overload, and position them as interpretable diagnostics that remain informative where holistic axes collapse. Finally, we analyze four recurring failure modes of ROUGE and LLM-as-judge scores, demonstrating that scaffold-based evaluation remains informative where conventional metrics collapse.
We investigate the factors affecting salience in movie screenplay by posing two research questions: 1. {\em Does Uniform Information Density (UID) predict salient scenes?} 2. {\em What other linguistic properties distinguish salient scenes from non-salient background?} We answer both questions through a controlled corpus study on the MENSA corpus, paired with a Linguistically-Grounded Salience Model (LGSM), a dual-stream classifier that fuses a frozen BERT encoder with explicit linguistic features through adaptive gating. The UID picture is layered in long-form discourse. Per-scene UID summaries carry no independent signal once length is controlled, but at the token level salient scenes match non-salient material in the bulk while specifically suppressing the rarest, hardest-to-predict tokens in the upper tail. UID therefore lives at the within-scene token level rather than as a per-scene aggregate. Salient scenes also show a moderate and sustained cast of central characters (following an inverted-U pattern), strong third-person usage, syntactically simpler language, and more abstract vocabulary about motivation and conflict. LGSM exploits this signature and improves recall on salient scenes over a strong semantic baseline while yielding better downstream summaries.