Organizations: Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd · State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University
Structured semantic parsing is typically judged by whether the correct category is assigned, which leaves value recovery and instance-level associations unexamined: a system may recognize an event yet bind an argument to the wrong instance, or identify an intent yet supply an incorrect slot value. We present USP, a cross-family semantic contract that represents categories, fields, values, and instance associations through a common interface for named-entity recognition, relation extraction, event extraction, and intent-slot parsing. The contract is paired with five diagnostic matching views---category agreement, field recognition, field-value pairing, instance-bound assignment, and complete-instance recovery---that ask distinct questions about the same output. Using this contract, we normalize 61 datasets into a unified corpus with shared schemas, prompts, and matching rules, and compare six models on a fixed test subset. Our fine-tuned Xing4.0-29B-Intent attains 57.70 task-balanced Field-Value Micro-F1 (equally weighted across the four task families) and ranks first in every family and on all five views; its margin over the strongest alternative widens from 2.93 points for category matching to 20.73 points for complete-instance matching. This widening gap shows that category agreement alone understates structural deficits. USP thus offers the community a shared representation, a normalized 61-dataset corpus, and a diagnostic evaluation protocol that make structural agreement measurable and comparable across extraction and intent-slot tasks.
Figures & tables
Figure 1: An example of schema-conditioned input and structured annotation under the USP contract.
Family
Datasets
Records
Valid
Invalid
Ambig.
NER
28
252,983
95.80
4.20
0.00
RE
13
179,615
96.36
0.33
3.31
EE
9
17,738
58.63
31.24
10.13
Intent-slot
7
28,592
29.30
43.68
27.01
All
57
478,928
90.66
6.11
3.23
Table 1: Contract conformance on the evaluation splits. Percentages in the upper block are computed within each family; the lower block reports shares of all records. Conformance measures structural validity, not source-to-USP semantic fidelity.
Field-Value Micro-F1
Other matching views
Parsing
Model
EE
NER
RE
IR
Overall
Category
Field
Inst.-bound
Complete
Parse
GLM-4-9B-Chat-HF
8.19
7.29
1.81
75.08
23.09
37.61
27.42
22.92
20.42
45.21
Qwen3-8B-Base
17.17
46.30
15.40
38.94
29.45
55.02
46.56
26.88
19.58
93.34
Qwen3.7-Flash
20.90
55.62
20.36
46.65
35.88
62.15
53.28
31.90
23.56
98.72
DeepSeek-V4-Flash
23.75
55.21
23.84
54.33
39.28
64.72
57.51
35.10
24.76
98.91
DeepSeek-V4-Pro
23.35
59.35
24.02
58.38
41.28
66.38
58.56
40.10
30.15
99.98
Table 2: Main comparison on the fixed test subset.
Figure 2: Task-balanced F1 under five structural matching views. Each view evaluates the same outputs using its own matching units; the profiles do not represent sequential processing stages.
Figure 3: Per-dataset Field-Value Micro-F1 margins of Xing4.0-29B-Intent over the best alternative. Vertical offsets separate overlapping points within each task family.
Figure 4: Parser acceptance and Field-Value Micro-F1 across model–dataset pairs, grouped by task. Parser acceptance includes empty outputs.
Model
EE
NER
RE
IR
Overall
Parse
Tokens
Gemini-3.6-Flash
30.46
70.73
29.34
64.34
48.72
99.40
22.34M
Qwen3.8-Max
28.98
66.15
26.33
63.31
46.19
99.90
15.05M
Kimi-K3
28.32
67.70
24.53
60.59
45.28
99.48
23.07M
GPT-5.6-Sol
28.27
67.87
23.01
60.08
44.81
100.00
16.20M
GLM-5.2
28.70
65.48
25.04
58.01
44.31
99.90
18.98M
GPT-5.5
27.66
67.18
22.91
59.18
44.23
99.97
17.22M
Table 3: Additional model results on USP-Core-6K, ordered by Overall.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Measure
Matching unit
Matching rule
Reporting role
Category
category
Set overlap
Category selection
Field
(category, field)
Set overlap
Field selection
Field-Value
(category, field, value)
Set overlap
Primary extraction measure
Instance-Bound Field
fields within matched instances
One-to-one assignment maximizing field overlap
Instance association
Complete Instance
complete instance
Multiset overlap
Complete structure recovery
Full-event
complete instance
Same implementation as Complete Instance
Compatibility name; no additional view
Appendix
Table 4: Metric definitions and reporting roles. Structural matching views report Micro-F1 within each dataset; task and overall scores macro-average these dataset-level values. Accuracy measures are reported separately.
Model
Core
Full
∣Δ∣
MAE
Qwen3.7-Flash
36.37
35.94
0.43
2.25
DeepSeek-V4-Flash
38.84
39.14
0.30
2.63
DeepSeek-V4-Pro
40.83
41.20
0.37
2.28
Appendix
Table 5: Calibration of the reduced suite against full-benchmark results from the same broader evaluation pool. ∣Δ∣ is the absolute difference in task-balanced F1, and MAE is the mean absolute per-dataset difference. The preserved ordering is evidence for screening utility, not universal validity.
Event extraction
Xing4.0- 29B-Intent
DS-V4-P
DS-V4-F
Qwen3.7-F
Qwen3-8B
GLM-4-9B
CASIE
—
9.77
11.94
8.68
8.16
5.56
CrudeOil
48.90
8.70
6.57
9.31
4.82
2.69
DuEE
75.70
57.74
59.75
54.91
44.34
21.64
DuEE-Fin
80.55
47.00
49.64
47.46
45.82
28.06
FewFC
31.77
0.03
0.00
0.05
0.05
0.00
PHEE
72.45
41.97
41.56
37.61
32.52
7.58
Appendix
Table 6: Field-Value Micro-F1 on the event extraction datasets in the fixed test subset. Models are ordered by recorded overall Field-Value F1; bold marks row maxima among the displayed scores.
Named-entity recognition
Xing4.0- 29B-Intent
DS-V4-P
DS-V4-F
Qwen3.7-F
Qwen3-8B
GLM-4-9B
AnatEM
74.91
43.20
45.45
41.44
43.80
10.98
BC2GM
76.39
60.78
44.62
59.28
53.98
13.29
BC5CDR
82.35
73.98
68.24
67.33
50.16
2.97
BioRED
76.85
65.77
60.14
61.35
49.50
3.39
Boson
68.65
27.12
12.93
31.16
28.12
4.12
CN-AI
58.69
63.88
58.43
56.60
43.42
7.92
Appendix
Table 7: Field-Value Micro-F1 on the named-entity recognition datasets in the fixed test subset. Models are ordered by overall Field-Value F1; bold marks row maxima.
Relation extraction
Xing4.0- 29B-Intent
DS-V4-P
DS-V4-F
Qwen3.7-F
Qwen3-8B
GLM-4-9B
CoNLL04
61.92
56.17
53.38
44.48
44.08
5.15
DocRED
29.87
25.23
27.29
24.68
10.33
0.26
FewRel
14.86
17.06
17.09
15.97
13.01
0.90
FinRED
24.23
10.47
9.59
5.77
4.11
0.78
GIDS
61.95
29.22
29.89
26.88
23.07
3.99
HacRED
54.02
38.47
38.40
31.64
13.63
1.66
Appendix
Table 8: Field-Value Micro-F1 on the relation extraction datasets in the fixed test subset. Models are ordered by overall Field-Value F1; bold marks row maxima.
Intent-slot understanding
Xing4.0- 29B-Intent
DS-V4-P
DS-V4-F
Qwen3.7-F
Qwen3-8B
GLM-4-9B
ATIS
88.44
76.73
82.55
71.77
57.60
92.31
CAIS
82.10
71.01
68.66
61.76
56.49
87.48
MulWOZ
84.40
45.89
44.19
35.33
30.59
5.89
SMP19
74.13
45.63
34.78
28.61
20.23
86.15
SMP20
66.20
40.16
31.95
30.36
23.14
86.02
SNIPS
90.22
70.86
63.86
52.07
45.60
92.64
Appendix
Table 9: Field-Value Micro-F1 on the intent-slot understanding datasets in the fixed test subset. Models are ordered by overall Field-Value F1; bold marks row maxima.
Model
EE
NER
RE
IR
Overall
Xing4.0-29B-Intent
5
17
8
1
31
DeepSeek-V4-Pro
2
5
0
0
7
GLM-4-9B-Chat-HF
0
0
0
5
5
DeepSeek-V4-Flash
1
0
1
0
2
No highest-observed cell: Qwen3-8B-Base , Qwen3.7-Flash
Appendix
Table 10: Fractional observed-best counts for Field-Value Micro-F1, ordered by total. Tied maxima share the count equally. Counts include the main protocol’s failure handling and do not indicate statistical significance.
Model
Params
Track
Accuracy
95% CI
Fmt. Compl.
Parse Rate
External comparators
GLM-4-9B-Chat-HF
9 B
I
0.3286
[0.3202, 0.3369]
0.6990
0.7415
Qwen3-8B-Base
8 B
I
0.5998
[0.5911, 0.6084]
0.9704
0.9503
Reported SFT checkpoint
Xing4.0-29B-Intent
29/4 B
I
0.6290 †
[0.6204, 0.6377]
0.8974
0.9015
Appendix
Table 11: MMLU-Pro accuracy on 12,032 questions under a shared zero-shot boxed-answer protocol. Empty or unparseable answers count as incorrect; confidence intervals use paired question-level bootstrap. Format completion detects a boxed marker, while Parse Rate measures answer extraction. Results are not directly comparable with the public five-shot leaderboard.
Task
Dataset
Lang.
#Schemas
Cite
NER-en
AnatEM
en
1
Pyysalo and Ananiadou (2014)
BC2GM
en
1
Smith et al. (2008)
BC4CHEMD ∗
en
1
Krallinger et al. (2015)
NCBI-Disease ∗
en
1
Dogan et al. (2014)
BC5CDR
en
2
Li et al. (2016)
HarveyNER
en
4
Chen et al. (2022)
Appendix
Table 12: Statistics of the 29 NER datasets in USP: task subgroup, source language, per-dataset schema count from the dataset inventory, and the originating publication. NCBI-Disease is outside the fixed test subset because of documented task or split fallback; the other datasets marked ∗ are outside the test subset because of partial model coverage. Datasets marked ‡ do not have a verified origin publication. Per-split instance counts and source URLs are provided on the dataset page.
Task
Dataset
Lang.
#Schemas
Cite
RE-en
ADE Corpus
en
1
Gurulingappa et al. (2012)
GIDS
en
4
Jat et al. (2017)
CoNLL2004
en
5
Roth and Yih (2004)
SciERC
en
7
Luan et al. (2018)
Semeval-RE
en
10
Hendrickx et al. (2010)
KBP37
en
18
Zhang and Wang (2015)
Appendix
Table 13: Statistics of the 16 RE datasets in USP: task subgroup, source language, per-dataset schema count from the dataset inventory, and the originating publication. Wiki-ZSL and CMeIE are outside the fixed test subset because of documented task or split fallback; DuIE2.0, also marked ∗ , is outside the test subset because of partial model coverage. Datasets marked ‡ do not have a verified origin publication and are cited by a shared-task placeholder. DuIE2.0 has no released training split. Per-split instance counts and source URLs are provided on the dataset page.
Task
Dataset
Lang.
#Event types (#Roles)
Cite
EE-en
CASIE
en
5 (26)
Satyapanich et al. (2020)
PHEE
en
2 (16)
Sun et al. (2022)
CrudeOilNews
en
18 (104)
Lee et al. (2022b)
RAMS
en
106 (398)
Ebner et al. (2020)
WikiEvents
en
31 (81)
Li et al. (2021)
EE-zh
DuEE1.0
zh
65 (217)
Li et al. (2020)
Appendix
Table 14: Statistics of the 9 EE and 7 Intent/Slot SLU datasets in USP: task subgroup, source language, per-dataset schema count from the dataset inventory (for EE, #event types with #argument roles in parentheses; the same convention applies to SLU intents), and the originating publication. MASSIVE, marked † , was normalized into the construction inventory but excluded from the current evaluation due to size. An unavailable schema count is shown as “—” rather than inferred; per-split instance counts and source URLs are provided on the dataset page.
Table 15: USP schema and reference annotation examples across four task families. Schemas show relevant categories and fields; category and argument descriptions are reproduced in full for all displayed fields.
Semantic parsing maps natural language into structured representations, such as logical forms, graphs, or queries, that machines can execute or reason over.
A well-designed extraction schema is not necessarily ready for reliable LLM execution. When only limited verified extractions are available, manually tuning hundreds of field definitions through trial and error is costly. We frame this problem as few-instance schema calibration: adapting the operational semantics of an existing schema from a few annotated documents while preserving its structural contract. We introduce CPSE, a contract-preserving semantic extraction framework that jointly calibrates extraction prompts and field-level semantic descriptions from a few gold annotations. CPSE decomposes the schema into an invariant structural contract and mutable field semantics, and further separates identity discovery from record completion using manifest-conditioned resolution. On expert-annotated polymer-science documents, CPSE improves extraction by 9.93 points over an execution-matched baseline, with consistent gains under an independent judge and in a blinded expert audit. These results show that CPSE enables low-resource schema execution while preserving the output structure required downstream.
Zixiao Dong, Wei Yang, Zihao Liu +4
School of Computer Science and Technology, University of Science and Technology of China · State Key Laboratory of Cognitive Intelligence · University of Science and Technology of China +1
Information extraction models often perform well on the datasets they are trained on, but transferring to new domains and annotation guidelines remains challenging. Supporting user-specified types alone does not resolve this problem. We present CalOn, a unified 0.6B-parameter encoder for entity, relation, and event extraction across datasets and schemas. The model jointly encodes text and type definitions, then directly identifies entity spans, relations between spans, and event triggers and arguments. Its extraction decisions consider the requested types together, using their combined evidence to identify relevant candidates before assigning a type. This gives all three tasks a common probabilistic formulation while keeping every extracted mention anchored to the input text. We train on diverse sources, and supplement supervised learning with a reward that encourages correct extractions and penalises incorrect ones. Across 40 benchmarks, CalOn outperforms the evaluated baselines on most benchmarks. It leads all reported relation extraction benchmarks and achieves the highest average F1 for event triggers and arguments with gold triggers. The same model also performs strongly on aspect sentiment triplet extraction and pronoun resolution through its span-pair formulation. A correction for the probability bias introduced during training improves confidence estimates for entities, relations, and event arguments after calibration, with little change in F1.
Yang Wang, Hanlin Xiao, Yanan Ma +2
University of Manchester · Mayo Clinic · Shanghai University of Finance and Economics
Enterprise workflows increasingly rely on agents for schema-guided extraction: given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata. We present ExtractBench, a benchmark for schema-guided extraction and, to our knowledge, the first to score value accuracy, record completeness at scale, grounding, and measured cost together. The evaluation system contains 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, with clear tags differentiating their challenge scenarios. The scalable schema and ground-truth curation pipeline combines independent-system agreement for real documents, known values for synthetic lists, and human verification for forms. We report order-insensitive value F1 for value accuracy, plus two grounding metrics for source traceability: word- and page-level F1. Commercial VLMs perform well on short documents but often truncate record lists on long ones, while coding agents retain higher accuracy at much higher cost. LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to coding agents at a fraction of the cost. Dataset and evaluation code are available on HuggingFace and GitHub.