Identification and ranking of salient propositions in documents is indirectly involved in many AI tasks, but few papers define proposition salience scores clearly, make diverse data available, or evaluate models directly on the task. This paper proposes PROPALIGN, a multiple-summary-based approach for propositional salience score estimation, and releases PROPALIGNBANK, an accompanying dataset for the method with 138K alignments of over 35K propositions. We compare several approaches to salient proposition extraction in English across a broad range of spoken and written genres, and demonstrate that naively asking models for judgments leads to inconsistencies. By contrast, graded, summary-based metrics are more stable. We further observe that while single models perform worse than humans, an ensemble of models incorporating linguistic features approaches manual annotation quality ceilings. In addition, analyses of the model outputs suggest insights into how models can extract salient information with and without the use of summaries.
Figures & tables
Figure 1: Asking models for salient text is more subjective and less consistent than asking what appears in a summary. Multiple summaries can compensate for summary subjectivity and yield gradient salience scores, and model judgments overlap more on the right.
Genre
Source
Docs
EDUs
Alignments
Genre
Source
Docs
EDUs
Alignments
Academic writing
Various
18
1,981
9,905
How-to guides
wikiHow
19
2,399
11,995
Biographies
Wikipedia
20
2,071
10,355
Interviews
Wikinews
19
2,408
12,040
Vlogs
YouTube
15
2,438
12,190
Letters
Various
18
2,102
10,510
Conversations
UCSB Corpus
15
3,083
15,415
News stories
Wikinews
24
1,912
9,560
Courtroom transcripts
Various
14
2,143
10,715
Podcasts
Various
14
2,256
11,280
Essays
Various
14
2,001
10,005
Political speeches
Various
15
1,913
9,565
Table 1: Overview of the PropAlignBank data from GUM .
Setting
F1
Baseline zero-shot
0.464
Baseline one-shot
0.638
Summary zero-shot
0.716
Summary one-shot
0.767
Table 2: Inter-model agreement between GPT and Claude, measured by F1 across experimental settings.
Setting
Model
P
R
F1
Baseline zero-shot
GPT
0.742
0.161
0.264
Claude
0.594
0.332
0.426
Qwen
0.720
0.089
0.159
Baseline one-shot
GPT
0.548
0.451
0.495
Claude
0.517
0.517
0.517
Qwen
0.575
0.293
0.388
Table 3: Precision (P), recall (R), and F1 across models and experimental settings, evaluated against aggregated human gold labels and output IDs.
Setting
Model
P
R
F1
MAE
RMSE
Summary zero-shot
GPT
0.401
0.717
0.515
0.755
1.38
Claude
0.568
0.639
0.601
0.466
0.996
Qwen
0.507
0.392
0.431
0.533
1.035
Summary one-shot
GPT
0.480
0.679
0.563
0.583
1.19
Claude
0.572
0.629
0.599
0.452
0.993
Qwen
0.478
0.504
0.496
0.515
0.973
Table 4: Precision (P), recall (R), F1, mean absolute error (MAE), and root mean squared error (RMSE) across experimental settings, the ensemble, and human performance for the fine-grained evaluation scheme.
Figure 2: Precision, recall and F1 for three models on written and spoken test data.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Fragment from an RST tree in the GUM corpus.
Genre
Dev example (summary)
Match
Approx.
Duplicate
Component
Types
academic
librarians (1)
7
4
1
0
3
bio
byron (2)
4
6
2
3
4
conversation
grounded (3)
4
7
4
2
4
court
loan (5)
6
4
4
3
4
essay
tools (4)
2
5
2
2
4
fiction
beast (2)
4
7
2
3
4
Appendix
Table 5: Development-set examples used for one-shot in-genre prompting on the test set. Document names omit the prefix GUM_<genre>_ . Types counts categories with a nonzero value.
Prompting
Normalized
Unnormalized
Total
Zero-shot
58
102
160
One-shot
11
149
160
Total
69
251
320
Appendix
Table 6: Qwen test-set prediction files by whether at least one predicted EDU ID required normalization.
Setting
Model
P
R
F1
Summary zero-shot
GPT
0.323
0.509
0.395
Claude
0.345
0.256
0.294
Summary one-shot
GPT
0.395
0.479
0.433
Claude
0.411
0.328
0.365
Appendix
Table 7: Precision (P), recall (R), and F1 for the subtype-aware fine-grained evaluation scheme. Extra predicted EDUs are false positives, while a salient EDU assigned the wrong subtype is counted once as a false negative.
Figure 4: Absolute error distance from gold salience by model. Qwen models have more errors than any model except GPT zero shot, but disproportionately many off-by-one errors, usually between 0 and 1, resulting in better RMSE than expected, but worse MAE. Claude one-shot is the best model by F1 score except for the ensemble.
Parameter
Search space
Number of trees
{200,300,…,800}
Max feats per split
{sqrt,log2,0.3,0.5,0.7}
Min samples per leaf
Integers in [1,20]
Imbalance strategy
{none, class weight, balanced subsample}
Positive-class weight
Log-uniform on [1,max(1.5r,2)]
Threshold
Uniform on [0.1,0.6]
Appendix
Table 8: Hyperparameter search space for the Random Forest model.
Parameter
Search space
Max depth
Integers in [3,10]
Learning rate
Log-uniform on [0.01,0.3]
Row subsample
Uniform on [0.5,1.0]
Column subsample
Uniform on [0.5,1.0]
Min child weight
Integers in [1,10]
Boosting rounds
{100,150,…,600}
Appendix
Table 9: Hyperparameter search space for the XGBoost model.
Figure 5: Variable importance for Random Forest, calculated on dev .
Figure 6: Variable importance for XGBoost, calculated on dev .
Feature
Description
position
Normalized position of the EDU within its document
depth
Normalized depth of the EDU in the discourse (RST) tree
parent_similarity
Similarity between the EDU and its parent EDU
len
Token length of the EDU
relname
RST discourse relation connecting the EDU to its parent
mention_count
Number of entity mentions whose span overlaps EDU
Appendix
Table 10: All possible features used for EDU-level salience classification. Features kept after pruning on dev are bolded.
Figure 7: Distribution of percentage of errors across EDUs in a document.
Figure 8: Distribution of percentage of errors across EDUs in a document by error type.