HeRE: A Novel Benchmark for Hebrew Relation Extraction via LLM-Guided Denoising of Knowledge Graph Alignments

Ron Keinan; Amir David Nissan Cohen; Reut Tsarfaty

cs.CLOct 15, 2026

HeRE: A Novel Benchmark for Hebrew Relation Extraction via LLM-Guided Denoising of Knowledge Graph Alignments

Authors: Ron Keinan, Amir David Nissan Cohen, Reut Tsarfaty

Organizations: Bar-Ilan University · Origin AI

Abstract

While LLMs demonstrate strong capabilities in English Relation Extraction (RE) tasks, a significant gap remains in assessing their RE performance in low-resource languages, due to a lack of standardized benchmarks. Creating native, language-specific annotated RE benchmarks is expensive and requires human expertise, while relying on translation yields suboptimal, inaccurate datasets that introduce English-centric biases. To tackle this, we propose a scalable, language-agnostic methodology to create RE data by denoising naturally occurring yet noisy alignments of knowledge bases and corresponding texts. We define a proper alignment via a Natural Language Inference (NLI) task, where a relation triplet is entailed by an input text. We empirically assess five denoising strategies, and demonstrate that multilingual models outperform language-specific models in the few-shot LLM setting, while Hebrew-specific encoders remain competitive under NLI fine-tuning and are strongest in the label-free cross-training setting. We further find that prompting in the native language achieves performance on par with or slightly exceeding English prompts, confirming that the pipeline operates naturally in the target language. The best open-weights model, Gemma-4-31B-it (F1,=,0.915), matches the best API model, GPT-5.4 (0.921), and reaches F1,=,0.844 on a fully held-out, dual-annotated test set. Applying the optimized pipeline, we release HeRE: a novel, massive, 1.5-million-instance silver dataset for Hebrew RE, designed to support data-intensive experimentation and model fine-tuning. HeRE bridges a critical gap in standard Hebrew RE evaluation, while providing the NLP community with a reproducible, language-agnostic pipeline for RE dataset construction in other low-resource settings.

Figures & tables

Panel of Figure 1 — Figure 1: The HeRE construction pipeline.

Statistic	Gold val.	Gold test	Silver
Examples	500	500	1,491,070
Unique documents	454	494	$\sim$ 228K
Triples per document	1.1	1.0	$\sim$ 6.5
Unique predicates	97	124	1,137
Predicates w/ $\geq$ 20 ex.	4	1	593
Positive rate	61.4%	52.0%	72.5%

Table 1: Dataset statistics.

Baseline: CROCODILE NLI filter
Model	Type	F1	Macro
XLM-R-large-XNLI (as published)	multi	0.406	0.299
XLM-R-large-XNLI (best tuned)	multi	0.784	0.595
Strategy 1: Fine-tuned NLI Encoders
AlephBERT	mono	0.618	0.560
mE5-large	multi	0.692	0.614

Table 2: Best micro-F1 and macro-F1 per model on the gold validation set (adjudicated labels; macro-F1 over predicates with at least 10 examples).

Figure 2: Best micro-F1 per model on the gold validation set with 95% bootstrap confidence intervals, by strategy. Dashed lines: the CROCODILE filter as published (0.406) and after tuning (0.784).

Training graph	Triples	MRR	Hits@10
Raw (all alignments)	1,479,292	0.161 $\pm$ 0.001	0.307 $\pm$ 0.003
Confidence-weighted	1,479,292	0.150 $\pm$ 0.003	0.289 $\pm$ 0.005
Denoised (agree3)	840,991	0.093 $\pm$ 0.002	0.189 $\pm$ 0.002
Size-matched control	840,991	0.075 $\pm$ 0.001	0.147 $\pm$ 0.002

Table 3: Link prediction on HeRE-KG. Triples are unique

(s,p,o)

: HeRE-KG’s 1,080,605 evidence-linked edges are mentions of 860,990 unique triples, of which 20,000 were drawn for the held-out set and removed from every training graph; the 14,163–14,213 scored per seed are those whose entities and relation appear in every training graph.

Evaluation set	$n$	F1
Gold validation (adjudicated)	500	0.915
Gold test (held out)	500	0.844
seen predicates (depth)	97	0.944
unseen predicates	203	0.776
silver-validation subset	200	0.885

Table 4: Gemma-4-31B-it: validation set vs. held-out test set and its subsets.

Model	Str.	F1	95% CI	Macro
Gemma-4-31B-it	S4	0.844	[0.808, 0.877]	0.619
Gemma-3-27B-it	S4	0.829	[0.794, 0.862]	0.543
DictaLM-3.0-24B-Base	S4	0.725	[0.682, 0.766]	0.426
DictaLM-3.0-24B	S2	0.745	[0.703, 0.785]	0.454
Gemma-2-9B ‡	S2	0.332	[0.271, 0.390]	0.113
Aya-Expanse-32b ‡	S2	0.159	[0.103, 0.217]	0.132

Table 5: Held-out test-set F1 of each signal deployed on the silver corpus. ‡ Deployed at score threshold 0.9. Ensembles: Table 16 .

Figure 3: HeRE-KG neighbourhood example: ten highest-confidence edges (width = mean confidence).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Encoder models
Model	Type	Dev F1	Test F1
AlephBERT	mono	0.763	0.781
mmBERT	multi	0.819	0.845
mE5-large	multi	0.845	0.850
NeoDictaBERT	mono	0.861	0.878
XLM-RoBERTa-large	multi	0.849	0.876

Appendix

Table 6: HebNLI macro-F1 of all fine-tuned models.

Model	Type	Hypothesis	F1
AlephBERT	mono	basic	0.578
		template	0.605
		llm	0.618
mE5-large	multi	basic	0.642
		template	0.692
		llm	0.680

Appendix

Table 7: Strategy 1: F1 by model and hypothesis, best over scoring formula and fixed threshold.

Model	Type	Hypothesis	F1
DictaLM-3.0-1.7B-Instruct	mono	basic	0.365
DictaLM-3.0-1.7B-Instruct	mono	template	0.429
Gemma-2-9B	multi	basic	0.794
Gemma-2-9B	multi	template	0.829
Gemma3-12B-it	multi	basic	0.606
Gemma3-12B-it	multi	template	0.623

Appendix

Table 8: Strategy 2: best soft-score F1 by model and hypothesis (constrained-prompt reruns for Gemma-4-31B, Qwen3.5-35B, Aya-Expanse-32b and DictaLM-3.0-24B).

Model	Type	K	F1	Macro	Silver pos.%
XLM-R-large	multi	3	0.729	0.613	53.5%
		5	0.713	0.622	56.8%
		7	0.718	0.622	57.4%
		10	0.714	0.621	57.9%
mmBERT	multi	3	0.732	0.614	52.1%
		5	0.733	0.619	54.9%

Appendix

Table 9: Strategy 3: ensemble F1 and macro-F1 by model and

K

; Silver pos.% is the ensemble’s positive rate on the silver corpus.

		Hebrew F1		English F1
Model	Shots	triplet	template	triplet	template
DictaLM-3.0-24B-Base	0	0.643	0.658	0.635	0.802
	2	0.822	0.818	0.723	0.811
	5	0.835	0.822	0.595	0.821
Aya-Expanse-32b	0	0.820	0.821	0.839	0.833
	2	0.823	0.847	0.824	0.835

Appendix

Table 10: Strategy 4: F1 by model, hypothesis, shots and prompt language. Bold : best per model.

Model	Lang	Shots	F1
GPT-5.4	He	0	0.893
	He	5	0.921
	En	0	0.887
	En	5	0.908
Gemini 3 Flash	He	0	0.903
	He	5	0.916

Appendix

Table 11: Strategy 5: F1 by model, prompt language and shots (triplet hypothesis). Bold : best per model.

	Validation		Test
Predicate	$n$	% pos.	$n$	% pos.
Country	79	81.0	16	75.0
Diplomatic relations	15	13.3	29	6.9
Instance of	29	93.1	9	88.9
Official language	19	0.0	17	5.9
Occupation	17	88.2	17	94.1

Appendix

Table 12: Positive rate of the most frequent predicates in the gold validation and test sets.

Predicate	$n$	F1
Country	79	0.947
Instance of	29	0.962
Administrative unit	25	0.955
Borders	21	0.857
Official language	19	n/a ∗
Admin. subdivision	18	0.903

Appendix

Table 13: Per-predicate F1 of the best configuration (GPT-5.4, Hebrew, triplet, 5-shot) on the validation set,

n\geq 10

. ∗ No positive gold examples, so F1 is undefined; the model predicts 0 and 2 false positives, respectively.

Passage length (words)	$n$	F1	95% CI
100–250	96	0.909	[0.840, 0.966]
250–500	256	0.835	[0.784, 0.880]
500+	133	0.818	[0.740, 0.882]

Appendix

Table 14: Gemma-4-31B-it test-set F1 by passage length (15 passages under 100 words omitted).

Model	Strategy	Pos.	Neg.
Gemma-4-31B-it	S4: Open-source LLM	54.0%	46.0%
Gemma-3-27B-it	S4: Open-source LLM	70.4%	29.6%
DictaLM-3.0-24B-Base	S4: Open-source LLM	66.9%	33.1%
DictaLM-3.0-24B	S2: Fine-tuned NLI	67.8%	32.2%
Gemma-2-9B	S2: Fine-tuned NLI	23.2%	76.8%
Aya-Expanse-32b	S2: Fine-tuned NLI	18.7%	81.3%

Appendix

Table 15: Positive rates of the ten deployed signals on the silver corpus; agree3 rates are within the published set.

Ensemble	F1	95% CI
agree3 (published label) ‡	0.916	[0.886, 0.944]
vote7 (majority of 7)	0.801	[0.763, 0.837]
agree2	0.770	[0.728, 0.809]
vote_all	0.680	[0.628, 0.729]
agree4	0.590	[0.531, 0.646]

Appendix

Table 16: Held-out test-set F1 of consensus rules over the silver-corpus signals: agree3 is the unanimous vote of Gemma-4-31B-it, Gemma-3-27B-it and DictaLM-3.0-24B (S2); agree2 the unanimous vote of Gemma-3-27B-it and DictaLM-3.0-24B (S2); agree4 adds NeoDictaBERT (S1) and NeoDictaBERT

K{=}3

(S3) to agree2 ; vote7 the majority of the three S4 models, DictaLM-3.0-24B (S2), NeoDictaBERT (S1) and both S3 signals; vote_all the majority of all ten. Unanimity rules abstain on split votes (abstentions counted as negatives). ‡ ; precision 0.899, recall 0.934.

KG Entity Alignment

See all 11 papers

Cross-lingual Relation Extraction with Large Language Models: Zero-Shot, Few-Shot, and Fine-Tuned Evaluation on Romanian
Jun 30, 2026Dragos-Mitrut Vasile, Elena-Simona Apostol, Stefan-Adrian Toma +2Relation Extraction LLM Fine-Tuning
National University of Science and Technology POLITEHNICA Bucharest, Splaiul Independent,ei 313, Bucures,ti 060042, Romania · Military Technical Academy ‘Ferdinand I’, Bulevardul George Cos,buc 39-49, 050141, Bucharest, Romania · Freie Universität Berlin, Arnimallee 14, Berlin, 14195, Germany +2
U4RASD at KnowledgeGraphEval-2026: Ontology-Constrained Encoder and LLM Systems for Arabic Relation Extraction
Oct 8, 2026Ahmad Abbas, Hussein Mazeh, Chadi Abou Chakra +3Relation Extraction Arabic NLP
Arab Center for Research and Policy Studies, Doha · American University of Beirut, Beirut, Lebanon
HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model
May 11, 2026Noam Kayzer, Dan Revital, Ori Bar Joseph +10Mixture-of-Experts Language Models Open-Weight Language Models
PwC Next

Jun 30, 2026J/K move · Enter open · S save

Cross-lingual Relation Extraction with Large Language Models: Zero-Shot, Few-Shot, and Fine-Tuned Evaluation on Romanian

Dragos-Mitrut Vasile, Elena-Simona Apostol, Stefan-Adrian Toma, Adrian Paschke+1

National University of Science and Technology POLITEHNICA Bucharest, Splaiul Independent,ei 313, Bucures,ti 060042, Romania · Military Technical Academy ‘Ferdinand I’, Bulevardul George Cos,buc 39-49, 050141, Bucharest, Romania · Freie Universität Berlin, Arnimallee 14, Berlin, 14195, Germany+2

Relation Extraction LLM Fine-Tuning LLM Information Extraction Low-Resource Language Processing Cross-Lingual Transfer

Open paper AskPDF arXiv

Relation extraction (RE) for low-resource languages is typically constrained by the lack of annotated corpora. We investigate the feasibility of cross-lingual RE for Romanian by combining automatic dataset translation with large language model (LLM) inference. We translate the SemEval-2010 Task 8 benchmark from English to Romanian using an LLM-based translation pipeline and evaluate Gemma 4 31B under zero-shot, few-shot, and QLoRA fine-tuned configurations, against four encoder baselines spanning 125M to 560M parameters: XLM- RoBERTa (base and large), Romanian BERT, and RoBERT- large. We assess two task formulations: relation classification with marked entities and end-to-end extraction. Our results show that Romanian incurs a 3 to 5 percentage point (pp) drop relative to English in prompt-only settings, that few-shot prompting provides marginal gains over zero-shot, and that QLoRA fine-tuning improves macro F1-Score by more than 22 percentage points in both languages while reducing the cross-lingual gap from 3.3 to 1.4pp. The encoder baselines come within 1-4pp of QLoRA Gemma on Romanian despite being 50-250 times smaller, with monolingual Romanian BERT at 125M parameters matching multilingual XLM-R at 278M. The case for using a 31B model for single-task RE on Romanian is therefore weak in deployment scenarios where compute matters. We release the translated dataset, evaluation code, and trained models.

Abstract

KG Entity Alignment

Explore similar work

Cross-lingual Relation Extraction with Large Language Models: Zero-Shot, Few-Shot, and Fine-Tuned Evaluation on Romanian

U4RASD at KnowledgeGraphEval-2026: Ontology-Constrained Encoder and LLM Systems for Arabic Relation Extraction

HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model