cs.CLOct 12, 2026
SaveCalibrated Typed Decisions for Entity, Relation, and Event Extraction
Organizations: University of Manchester · Mayo Clinic · Shanghai University of Finance and Economics
Abstract
Information extraction models often perform well on the datasets they are trained on, but transferring to new domains and annotation guidelines remains challenging. Supporting user-specified types alone does not resolve this problem. We present CalOn, a unified 0.6B-parameter encoder for entity, relation, and event extraction across datasets and schemas. The model jointly encodes text and type definitions, then directly identifies entity spans, relations between spans, and event triggers and arguments. Its extraction decisions consider the requested types together, using their combined evidence to identify relevant candidates before assigning a type. This gives all three tasks a common probabilistic formulation while keeping every extracted mention anchored to the input text. We train on diverse sources, and supplement supervised learning with a reward that encourages correct extractions and penalises incorrect ones. Across 40 benchmarks, CalOn outperforms the evaluated baselines on most benchmarks. It leads all reported relation extraction benchmarks and achieves the highest average F1 for event triggers and arguments with gold triggers. The same model also performs strongly on aspect sentiment triplet extraction and pronoun resolution through its span-pair formulation. A correction for the probability bias introduced during training improves confidence estimates for entities, relations, and event arguments after calibration, with little change in F1.
Figures & tables
Figure 1: Overview of CalOn : (a) schema-text encoding with type-candidate interaction and bilinear scoring over the queried types and a null class, and (b) the RLCD reward on sampled typed decisions added to the supervised loss.
Figure 2: Ablation results for CalOn . Each row changes one model component. Bars show average micro F1 (%) with type descriptions over 15 NER benchmarks, 18 RE benchmarks with gold entities, and 7 EE benchmarks with gold triggers. Stars ( ) show results with type names alone on their own scales. The bottom row shows the strongest baseline for each task, averaged over the same benchmarks. InstructUIE’s training data include some of the event benchmarks.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Training mentions of the name | |||
|---|---|---|---|---|
| Rare | Middle | Frequent | ||
| CalOn | 58.8 | 54.8 | 61.3 | 0.04 |
| GLiNER2 | 50.6 | 46.7 | 52.0 | 0.01 |
| GLiFormer | 50.0 | 50.6 | 55.2 | 0.09 |
| GNER | 36.1 | 42.7 | 48.7 | 0.21 |
| InstructUIE | 38.7 | 39.5 | 44.2 | 0.10 |
Table 5: Macro F1 (%) over the 150 entity types in the 15 NER benchmarks, counting types separately for each benchmark. Types are grouped into thirds by their mention counts in CalOn ’s training data. Rare types have at most 13,400 mentions, and frequent types have more than 77,924. is the Spearman correlation between these counts and per-type F1. Evaluation settings follow Table 1 .
| F1 (%) | ECE | |||||
| Dataset | Fitted | +Corr. | +Corr.+ | Fitted | +Corr. | +Corr.+ |
| Entities | ||||||
| Validation (training sources) | 68.4 | 68.5 | 68.4 | 0.25 | 0.16 | 0.12 |
| CrossNER AI | 66.1 | 66.0 | 65.8 | 0.26 | 0.21 | 0.17 |
| CrossNER Literature | 63.5 | 63.3 | 63.1 | 0.27 | 0.22 | 0.18 |
| CrossNER Music | 72.0 | 71.7 | 71.4 | 0.18 | 0.15 | 0.10 |
Table 9: F1 and ECE for CalOn . Fitted uses the original calibration parameters without null-class correction. +Corr. applies the correction and refits thresholds with . +Corr.+ additionally refits temperature. Parameters are selected on validation data from the training sources, using REBEL for relations and event arguments, and remain fixed on evaluation datasets. Relations use gold entities, and event arguments use gold triggers. ECE uses 15 equal-width bins over returned predictions and the probability of the predicted type or role. Lowest ECE values are bold .
Figure 3: Average micro F1 as training data increase. Solid lines use type descriptions, dashed lines use type names alone, and dotted lines show GLiNER2 with descriptions. Averages cover the 15 NER, 18 RE with gold entities, and 7 EE with gold triggers benchmarks.
| Dataset | Task | Domain and genre | # Types | # Examples |
|---|---|---|---|---|
| CrossNER AI Liu et al. (2021) | Entities | AI, encyclopedia | 14 | 431 |
| CrossNER Literature Liu et al. (2021) | Entities | Literature, encyclopedia | 12 | 416 |
| CrossNER Music Liu et al. (2021) | Entities | Music, encyclopedia | 13 | 465 |
| CrossNER Politics Liu et al. (2021) | Entities | Politics, encyclopedia | 9 | 651 |
| CrossNER Science Liu et al. (2021) | Entities | Science, encyclopedia | 17 | 543 |
| MIT Movie Liu et al. (2013) | Entities | Film, search queries | 12 | 1,953 |
Table 11: The 40 evaluation benchmarks with their tasks, domains, genres, type counts, and numbers of converted evaluation inputs. Types are entity, relation, or event types according to the task. Inputs may be sentences, passages, or document chunks. All scores use micro F1.
| Dataset | Task | Domain and genre | Annotation | # Types | # Examples |
|---|---|---|---|---|---|
| Few-NERD Ding et al. (2021) | Entities | General, encyclopedia | Human | 74 | 125,839 |
| MASSIVE FitzGerald et al. (2023) | Entities (slots) | Personal assistant, voice commands | Human | 55 | 11,062 |
| FiNERWeb Golde et al. (2026) | Entities | General, web pages | LLM (released) | 3,895 | 16,798 |
| Pile-NER Zhou et al. (2024) | Entities | General, mixed text | LLM (released) | 268,736 | 73,395 |
| REBEL Huguet Cabot and Navigli (2021) | Entities and relations | General, encyclopedia | LLM (ours) | 45,074 | 399,983 |
| MAVEN-Arg Wang et al. (2024) | Events | General, encyclopedia | Human | 162 | 12,889 |
Table 12: Training sources using real texts. Human denotes the original human annotations, LLM (released) denotes released LLM annotations, and LLM (ours) denotes annotations produced by our pipeline. Type counts include names with at least one training annotation, counting relation types for REBEL and event types for event sources. Example counts refer to converted training instances, including trigger and argument instances derived from event passages. Synthetic sources are described in Table 13 and Figure 5 .
| Genres | email; text message conversation; internal business report; meeting notes; social media post; product review; short news brief; customer support ticket; invoice or order confirmation; job posting; contract clause; clinical note; research paper abstract; recipe; travel itinerary; forum discussion post; press release; resume summary; encyclopedia-style paragraph; podcast transcript excerpt; spoken request to a voice assistant; search query log; legal contract summary; government notice; patent abstract; wiki talk page comment; product specification sheet; hospital discharge summary; sports match report; obituary; museum exhibit label; travel guide paragraph; restaurant menu description; movie synopsis; tweet thread; answer on a question-and-answer site; tutorial step; software changelog entry; science news article; court judgment excerpt |
| Domains | finance and banking; healthcare; pharmaceuticals; law and courts; politics and elections; government and public policy; military and defense; sports; music; film and television; literature and publishing; visual arts; video games; fashion; food and restaurants; travel and tourism; hotels and hospitality; real estate; construction and architecture; automotive; aviation; shipping and logistics; retail and e-commerce; consumer electronics; telecommunications; software engineering; cybersecurity; artificial intelligence research; data science; physics; chemistry; biology and genetics; astronomy and space; earth science and climate; agriculture; energy and utilities; oil and gas; mining; manufacturing; human resources; education; academia and universities; religion; history; archaeology; philosophy; linguistics; journalism and media; advertising and marketing; insurance; cryptocurrency and blockchain; nonprofit and charity; environmental conservation; veterinary and animals; sports betting and gaming; social work; public transport; weddings and events; fitness and wellness; parenting and family; ophthalmology; dentistry; mental health; nutrition; tennis; association football; basketball; baseball; cricket; motorsport; cycling; the Olympic Games; chess; board games; anime and manga; theatre; dance; photography; sculpture; museums; classical antiquity; medieval history; military history; genealogy; urban planning; maritime industry; railways; space industry; robotics; semiconductors; cloud computing; open-source software; mobile apps; social networks; e-sports; tax law; intellectual property; immigration; criminal justice; human rights; international relations; diplomacy; local government; labor unions; supply chains; wine and beverages; coffee and tea; gardening; pets; zoology; botany; marine biology; ecology; geology; meteorology; oceanography; epidemiology; neuroscience; materials science; mathematics |
Table 13: Genres and domains used for synthetic generation. Each request combines one genre with one domain and asks the generator to produce a text and an extraction schema.
Figure 4: Training type frequencies. For each source, types are ranked by mention count. Both axes use logarithmic scales. Longer curves indicate more type names, and flatter curves indicate a more even distribution of mentions. The synthetic curve pools all synthetic sources, while the other curves show individual datasets. Counts use the complete training files before sampling.
Figure 5: Generation prompts for entities, relations, and events. The genre and domain values come from Table 13 . Requested text lengths are 30, 60, 120, 200, or 300 words. Types listed as absent provide explicit negative supervision after annotation checking.
Figure 6: UMAP ( McInnes et al., 2018 ) projection of sentence embeddings sampled from synthetic training sources, real-text training sources, and evaluation datasets.