Zero-shot relation extraction aims to identify relations between entity mentions using textual descriptions of novel types (i.e., previously unseen) instead of labeled training examples. Previous works often rely on unrealistic assumptions: (1) pairs of mentions are often encoded directly in the input, which prevents offline pre-computation for large scale document database querying; (2) no rejection mechanism is introduced, biasing the evaluation when using these models in a retrieval scenario where some (and often most) inputs are irrelevant and must be ignored. In this work, we study the robustness of existing zero-shot relation extraction models when adapting them to a realistic extraction scenario. To this end, we introduce a typology of existing models, and propose several strategies to build single pass models and models with a rejection mechanism. We adapt several state-of-the-art tools, and compare them in this challenging setting, showing that no existing work is really robust to realistic assumptions, but overall AlignRE (Li et al., 2024) performs best along all criteria.
Figures & tables
Figure 1: In relation extraction, the input is an utterance with two identified mentions, a head mention and a tail mention. We assume that the only targeted relation type is operating system . (left) Two input examples. In (1), the model must predict that there is a relation of this type between the two mentions. In (2), the model should reject the input, as the candidate input relation does not correspond to any type in the targeted ones. (right) Example of side-information used for zero-shot relation extraction.
Zs-Bert
LaVeEntail
Rcl
MatchPrompt
ReMatching
Cl&Cd
Emma
AlignRE
Ce-Da
GliRel
Chen and Li
Sainz et al.
Wang et al.
Wang et al.
Zhao et al.
Yang et al.
Li et al.
Li et al.
Zhang et al.
Boylan et al.
( 2021 )
( 2021 )
( 2022b )
( 2022a )
( 2023 )
( 2024 )
( 2024a )
( 2024b )
( 2025 )
( 2025 )
Neural architecture
SBert
✓
✓
✓
Single pass
✓
✓
Rej. mechanism
✓
Table 1: Typology of zero-shot relation extraction models, ordered chronologically. Row SBert indicates the use of SBert to encode side-information of relation types. The F1 row contains results as reported by original authors on FewRel using 5 unknown relation types, without rejection mechanism. (We do not include results for MatchPrompt and AlignRE as they have not been tested on this dataset) Note that there is also a late interaction variant of Emma , but it achieves significantly lower F1 scores.
Figure 2: Generic illustration of an encoder-only zero-shot RE model.
|T|
|D|
Dist. me.
Avg. len.
WikiZSL
113
94 383
77 623
24.85
FewRel
80
56 000
72 954
24.95
Table 2: Datasets statistics, where T is the set of relation types, D the set of relation quadruples including utterance, entities and relation type (s,e1,e2,t) , and the last two columns give the number of distinct entity mentions and the average length of utterances, respectively.
FewRel
WikiZSL
5
10
15
5
10
15
Emma
Off-the-shelf
98.4±1.2
84.5±4.5
79.5±6.2
88.2±7.0
67.9±8.8
62.8±6.6
Our efficient variants
↪ first
94.5±1.6
79.5±3.0
70.2±4.6
81.6±4.8
64.7±7.7
57.0±2.0
↪ projection
94.6±1.2
77.8±4.8
67.2±6.1
79.0±6.2
59.0±9.9
50.5±2.3
Table 3: Results in terms of macro F1, without rejection mechanism. We compare the off-the-shelf softwares with our efficient variants that allow on-the-fly zero-shot relation extraction with offline encoding.
FewRel
WikiZSL
5
10
15
5
10
15
Emma
Threshold
9.8±4.0
7.3±8.9
9.8±9.3
18.0±6.4
28.7±9.6
25.8±12.7
Description
42.8±16.6
42.5±28.5
24.9±15.3
61.4±14.2
49.8±16.0
47.3±9.5
Prototypes
100.0±0.0
99.8±0.3
100.0±0.0
37.1±33.4
36.0±54.5
67.5±26.4
Re-Matching
Table 4: Rejection accuracy in the rejection pass.
FewRel
WikiZSL
5
10
15
5
10
15
Emma
Threshold
52.4±1.9
31.8±6.5
29.0±11.0
54.6±6.0
47.1±8.4
47.0±8.1
Description
41.3±12.2
30.1±4.2
20.6±6.0
30.6±8.5
24.8±3.5
24.5±6.0
Prototypes
0.0±0.0
0.2±0.3
0.0±0.0
29.9±13.8
25.8±19.2
23.7±22.9
Re-Matching
Table 5: Macro F1 measure in the retention pass, computed for (non reject) relation types.