EMERGE: A Benchmark for Updating Knowledge Graphs with Emerging Textual Knowledge
Organizations: Aarhus University, Denmark · Amsterdam UMC, The Netherlands · Sapienza University of Rome, Italy · University of Amsterdam, The Netherlands
Abstract
Knowledge Graphs (KGs) are structured knowledge repositories containing entities and relations between them. In this paper, we study the problem of automatically updating KGs over time in response to evolving knowledge in unstructured textual sources. Addressing this problem requires identifying a wide range of update operations based on the state of an existing KG at a given time and the information extracted from text. This contrasts with traditional information extraction pipelines, which extract knowledge from text independently of the current state of a KG. To address this challenge, we propose a generic and extensible pipeline that pairs textual passages with the KG edit operations they induce on a given KG snapshot. We instantiate this pipeline on the Wikidata knowledge graph and the English Wikipedia corpus to create EMERGE, a dataset of 233K Wikipedia passages associated with a total of 1.19 million KG edits across seven yearly Wikidata snapshots from 2019 to 2025. Our experimental results highlight key challenges in updating KG snapshots based on emerging textual knowledge, particularly in integrating knowledge expressed in text with the existing KG structure. We further show that KG access substantially improves executable updates and enables triple deprecation. These findings position EMERGE as a benchmark for developing methods that keep KGs up to date with emerging textual knowledge. The dataset (https://huggingface.co/datasets/klimzaporojets/emerge-benchmark) and code (https://github.com/klimzaporojets/emerge) are publicly available.
Figures & tables
| Evolution | TKGU operations | Annotation statistics | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | KG | Text | E | A | M+A | I | D | Instances | Rel. types | Entities | Triples |
| WebNLG ( gardent2017webnlg ) | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | 38.87K | 411 | 3.62K | 115.28K |
| T-REX ( elsahar2019t ) | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | 6.20M | 642 | 4.65M | 11.00M |
| Wiki-NRE ( distiawan2019neural ) | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | 255.65K | 158 | 279.89K | 330.01K |
| DocRED ( yao2019docred ) | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | 101.87K | 96 | 782.46K | 1.51M |
| BioRel ( xing2020biorel ) | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | 533.56K | 125 | 17.82K | 578.01K |
| Model | KG | TKGU operations | ||||
| Linked | E | A | M+A | I | D | |
| REBEL ( cabot2021rebel ) | ✗ | – | – | – | – | – |
| GenIE ( josifoski2021genie ) | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ |
| KnowGL ( rossiello2023knowgl ) | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ |
| GCD ( geng2023grammar ) | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ |
| ReLiK cIE ( orlando2024relik ) | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ |
| Exists | Add | Mint+Add | Infer | Deprecate | |||||||||
| Model | C | G-R | E-R | C | G-R | E-R | C | G-R | C | G-R | C | G-R | E-R |
| EDC+ GPT-5.1 | 25.8 | 66.5 | – | 39.9 | 77.3 | – | 35.6 | 77.3 | 21.9 | 72.4 | 34.8 | 53.2 | – |
| EDC+ M-Lg | 16.2 | 53.4 | – | 29.9 | 70.4 | – | 27.9 | 71.5 | 19.9 | 68.8 | 31.7 | 53.1 | – |
| EDC+ M-Sm | 8.8 | 45.5 | – | 20.8 | 63.3 | – | 16.3 | 62.6 | 21.4 | 70.8 | 14.5 | 28.6 | – |
| KGGen GPT-5.1 | 12.7 | 62.7 | – | 15.3 | 68.1 | – | 13.3 | 67.0 | 3.3 | 59.1 | – | – | – |
| KGGen M-Lg | 11.8 | 60.5 | – | 16.1 | 67.7 | – | 13.7 | 65.9 | 2.1 | 56.9 | – | – | – |
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Evolution | Text-to-KG integration | |||||||
| Dataset | KG | Text | KG | Exists | Add | Mint+Add | Infer | Deprecate |
| Linked | ||||||||
| MUC-7 ( chinchor1998muc ) | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| CoNLL04 ( roth2004linear ) | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| ACE 2005 ( walker2006ace ) | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| SemEval 2010 ( hendrickx2010semeval ) | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Operation | Exact-match feasibility | Additional capability required | Supported by |
|---|---|---|---|
| Exists | inference-time | link both entities to | ReLiK cIE, KGA |
| Add | inference-time | link both entities to | ReLiK cIE, KGA |
| Mint+Add | post-hoc only | generate and add emerging entity to | none |
| Infer | post-hoc only | generate and resolve emerging entity; KG-side reasoning | none |
| Deprecate | inference-time | identify deprecated | KGA |
| Metric | TKGU operation | 95% CI width | Rank | ||
|---|---|---|---|---|---|
| median | max | min | mean | ||
| C | Exists | 2.9 | 3.2 | 0.87 | 0.97 |
| C | Add | 4.3 | 5.2 | 0.87 | 0.96 |
| C | Mint+Add | 3.3 | 4.6 | 0.90 | 0.97 |
| C | Infer | 1.3 | 3.4 | 0.89 | 0.98 |
| C | Deprecate | 0.0 | 5.8 | 0.99 | 1.00 |
| TKGU Operation | H-H Cohen’s | H1-LLM Cohen’s | H2-LLM Cohen’s | H+LLM Fleiss’ | H+LLM Kripp. |
|---|---|---|---|---|---|
| Exists | 0.718 | 0.649 | 0.637 | 0.668 | 0.669 |
| Add | 0.750 | 0.698 | 0.750 | 0.732 | 0.733 |
| Mint+Add | 0.680 | 0.811 | 0.863 | 0.784 | 0.785 |
| Infer | 0.880 | 0.840 | 0.761 | 0.827 | 0.827 |
| Deprecate | 0.771 | 0.675 | 0.610 | 0.687 | 0.688 |
| Overall | 0.792 | 0.765 | 0.744 | 0.767 | 0.767 |
| Complete dataset | Subsampled test set | KG statistics | |||||
|---|---|---|---|---|---|---|---|
| Snapshot | Instances | Operations | Instances | Operations | Entities | Rel. Types | Triples |
| 2019 | 37K | 173K | 500 | 3.24K | 5.96M | 5,646 | 26.35M |
| 2020 | 31K | 159K | 500 | 3.21K | 6.14M | 7,017 | 29.47M |
| 2021 | 40K | 232K | 500 | 3.82K | 6.34M | 8,216 | 31.63M |
| 2022 | 30K | 155K | 500 | 2.86K | 6.54M | 9,425 | 34.30M |
| 2023 | 26K | 126K | 500 | 2.91K | 6.67M | 10,599 | 35.93M |
| Snap. | # | Triple | Example Passage |
|---|---|---|---|
| 2021 | 793 | 2020 United States presidential election; candidate; Donald Trump | In 2020, Pletts voiced support for Donald Trump and the Republican Party in the 2020 United States presidential election and Senate elections. |
| 2021 | 787 | Donald Trump; candidacy in election; 2020 United States presidential election | Over the span of the 2020 presidential election, RSBN’s coverage of Donald Trump’s campaign rallies grossed over 127 million views on YouTube. |
| 2021 | 666 | 2020 United States presidential election; candidate; Joe Biden | Despite being divorced, she remains good friends with her ex-husband, and she supported Joe Biden and Kamala Harris in the 2020 election. |
| 2021 | 663 | Joe Biden; candidacy in election; 2020 United States presidential election | In September 2020, Kennedy Kent endorsed Republican President Donald Trump for reelection over Democratic nominee Joe Biden. |
| 2021 | 255 | midfielder; sport; association football | “Niko Rak” (born 26 July 2003) is a Croatian footballer who plays for Šibenik as a midfielder. |
| Snap. | # | Triple | Example Passage |
|---|---|---|---|
| 2021 | 597 | 2020 United States presidential election; successful candidate; Joe Biden | The winner of the 2020 election, President Joe Biden, is next in line to this purported curse. |
| 2020 | 161 | Abu Mahdi al-Muhandis; military branch; Popular Mobilization Forces | On 3 January 2020, a MQ-9 missile strike at the Baghdad International Airport killed Qasem Soleimani, the commander of Iran’s Quds Force, and Abu Mahdi al-Muhandis, the commander of the Popular Mobilization Forces. |
| 2024 | 141 | Houthi movement; country; Yemen | On 8 November 2023, Houthi rebels in Yemen shot down a US Air Force MQ-9 over the Red Sea amid the attacks on U.S. bases in Iraq and Syria. |
| 2023 | 115 | Kevin McCarthy; position held; Speaker of the United States House of Representatives | On January 3, 2023, at the beginning of the 118th Congress, Boebert voted for Jim Jordan to be the U.S. House Speaker, in rebuke of House Minority Leader Kevin McCarthy. |
| 2020 | 114 | Qasem Soleimani; place of death; Baghdad | Soleimani was assassinated in a targeted U.S. drone strike on 3 January 2020 in Baghdad, which was approved by President Donald Trump on the grounds that Soleimani posed an “imminent threat” to American lives. |
| Snap. | # | Triple | Example Passage |
|---|---|---|---|
| 2020 | 674 | Qasem Soleimani; significant event; assassination of Qasem Soleimani | He was killed by a targeted U.S. drone strike at the Baghdad International Airport on 3 January 2020, which also killed Iranian Armed Forces Major General Qasem Soleimani. |
| 2021 | 666 | January 6 United States Capitol attack ; significant person; Donald Trump | CRP reported that President Trump’s re-election campaign was financially related to the Capitol riot on January 6, 2021. |
| 2022 | 299 | Dawn FM ; performer; The Weeknd | In 2022 the group also received credit for co producing songs off The Weeknds fifth studio album Dawn FM. |
| 2025 | 296 | 2025 New Orleans truck attack ; located in the administrative territorial entity; New Orleans | 2025 New Orleans truck attack: President Joe Biden has been briefed on the attack and has been in touch with New Orleans Mayor to offer support. |
| 2023 | 72 | Flowers ; performer; Miley Cyrus | The chart’s current number one as of the issue dated January 28, 2023, is “Flowers” by Miley Cyrus |
| Snap. | # | Triple | Example Passage |
|---|---|---|---|
| 2021 | 2169 | January 6 United States Capitol attack ; located in the administrative territorial entity; Washington, D.C. | She called for the impeachment of President Donald Trump, in wake of the 2021 storming of the United States Capitol. |
| 2020 | 1065 | assassination of Qasem Soleimani ; instance of; assassination | Assassination of Qasem Soleimani: the president called for restraint and said the events in Iraq were the result of previous “terrorist acts”. |
| 2025 | 704 | Golden Age of Argentine cinema ; part of; history of film | “Huella” is a 1940 Argentine film directed by Luis Moglia Barth during the Golden Age of Argentine cinema. |
| 2024 | 270 | South Africa v. Israel ; charge; genocide | In 2023-24, he was appointed as a member of the South African legal team arguing “South Africa v. Israel” regarding the Genocide Convention. |
| 2019 | 153 | All Elite Wrestling ; instance of; business | On January 1, 2019 Cody Rhodes unveiled a new promotion; All Elite Wrestling, in which he, along with Matt and Nick Jackson, will serve as Executive Vice President. |
| Snap. | # | Triple | Example Passage |
|---|---|---|---|
| 2021 | 3332 | Proud Boys; significant event; January 6 United States Capitol attack | Trump supporters infiltrated Capitol Hill in Washington DC., 5 people killed. |
| 2022 | 434 | Dawn FM ; distribution format; LP record | In 2022 the group also received credit for co producing songs off The Weeknds fifth studio album Dawn FM. |
| 2024 | 290 | 2024 Haneda Airport runway collision ; destination point; Niigata Airport | 2024 Haneda Airport runway collision: All flights in and out of Haneda were suspended following the accident; operations currently remain suspended. |
| 2019 | 172 | All Elite Wrestling ; legal form; privately held company | On January 1, 2019 Cody Rhodes unveiled a new promotion; All Elite Wrestling, in which he, along with Matt and Nick Jackson, will serve as Executive Vice President. |
| 2019 | 31 | @world_record_egg ; country; United Kingdom | @world_record_egg is an account on social media platform Instagram, notable for holding the world records for both the most-liked Instagram post and most liked online post on any media platform in history. |
| Snap. | # | Triple | Example Passage |
|---|---|---|---|
| 2021 | 75 | Parler; distributed by; Google Play | After complaints that Parler was used to coordinate the 2021 storming of the U.S. Capitol, Apple and Google removed Parler’s mobile app from their app stores. Parler went offline on January 10, 2021 at 11:59 PM (PST) after Amazon Web Services canceled its hosting services. |
| 2021 | 75 | Mike Pence; position; Vice President of the United States | “Marlon Bundo”, also known as “Bunny of the United States” (“BOTUS”), is a rabbit, belonging to the family of Mike Pence, the 48th and former Vice President of the United States. |
| 2025 | 63 | Vice President of the United States; position holder; Kamala Harris | West is the brother-in-law of former Vice President Kamala Harris. He served as an advisor to her 2024 presidential campaign. |
| 2024 | 62 | Adam Peters ; member of sports team; San Francisco 49ers | Peters joined the Denver Broncos as a scout in 2009. He was promoted to assistant director of college scouting in July 2014 and to director of college scouting in 2016. He was a member of the team that won Super Bowl 50 in 2015. |
| 2020 | 43 | European Union; has part(s); United Kingdom | Chris Davies was the chairman (2019 – 2020) - until the United Kingdom left the European Union. |
| Passage 1. (KG Snapshot 2025) Shafique Ahmed Siddique: == Personal life == Siddique married Sheikh Rehana in 1977 in London, sister of Prime Minister Sheikh Hasina. The couple have a son and two daughters, one of whom is Tulip Rizwana Siddiq. | |||
| TKGU | Ground Truth | Predicted | BERTScore |
| Exists | Sheikh Rehana; sibling; Sheikh Hasina | Sheikh Rehana; sibling; Sheikh Hasina | 1.000 |
| Add | Shafiq Ahmed Siddiq; spouse; Sheikh Rehana | Shafique Ahmed Siddique; spouse; Sheikh Rehana | 0.956 |
| Add | Shafiq Ahmed Siddiq; child; Tulip Siddiq | Shafique Ahmed Siddique; child; Tulip Rizwana Siddiq | 0.898 |
| Add | Sheikh Rehana; spouse; Shafiq Ahmed Siddiq | Shafique Ahmed Siddique; spouse; Sheikh Rehana | 0.889 |
| Exists | Sheikh Rehana; child; Tulip Siddiq | Shafique Ahmed Siddique; child; Tulip Rizwana Siddiq | 0.781 |
| Relation | 2019 | 2020 | 2021 | 2022 | 2023 | 2024 | 2025 |
|---|---|---|---|---|---|---|---|
| significant event | 1 | 17 | 127 | 3 | 0 | 6 | 5 |
| target | 1 | 6 | 100 | 0 | 3 | 0 | 0 |
| facet of | 0 | 18 | 78 | 0 | 1 | 2 | 12 |
| significant person | 0 | 0 | 64 | 0 | 0 | 0 | 3 |
| has cause | 0 | 0 | 45 | 0 | 1 | 0 | 1 |
| has effect | 0 | 0 | 44 | 5 | 0 | 0 | 2 |
| Head | Relation | Tail | KG rank |
|---|---|---|---|
| United States Capitol | significant event | January 6 United States Capitol attack | 103 |
| Donald Trump | significant event | second impeachment of Donald Trump | 103 |
| first presidency of Donald Trump | significant event | January 6 United States Capitol attack | 103 |
| Tatmadaw | significant event | 2021 Myanmar coup d’état | 103 |
| January 6 United States Capitol attack | target | United States Congress | 453 |
| January 6 United States Capitol attack | facet of | presidential transition of Joe Biden | 86 |
| Head | Relation | Tail | KG rank |
|---|---|---|---|
| 2020 US presidential election Electoral College count | significant event | January 6 United States Capitol attack | 103 |
| Lauren Boebert | significant event | January 6 United States Capitol attack | 103 |
| Donald Trump | significant event | second impeachment of Donald Trump | 103 |
| January 6 United States Capitol attack | significant person | Donald Trump | 313 |
| second impeachment of Donald Trump | significant person | Donald Trump | 313 |
| January 6 United States Capitol attack | in opposition to | 2020 US presidential election Electoral College count | 718 |
Knowledge Graphs
Knowledge graphs represent entities and relationships as structured, queryable networks for integrating data and supporting reasoning.
See all 251 papers