Xing USP: A Cross-Family Semantic Contract for Information Extraction and Intent-Slot Parsing

Zhongqiu Li; Boyang Wang; Ruiyu Fang; Shiquan Wang; Di Wu; Jiayi Liang; Feilong Zhang; Hongfeng Du

cs.CLOct 25, 2026

Xing USP: A Cross-Family Semantic Contract for Information Extraction and Intent-Slot Parsing

Authors: Zhongqiu Li, Boyang Wang, Ruiyu Fang, Shiquan Wang, Di Wu, Jiayi Liang, Feilong Zhang, Hongfeng Du,

Organizations: Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd · State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University

Abstract

Structured semantic parsing is typically judged by whether the correct category is assigned, which leaves value recovery and instance-level associations unexamined: a system may recognize an event yet bind an argument to the wrong instance, or identify an intent yet supply an incorrect slot value. We present USP, a cross-family semantic contract that represents categories, fields, values, and instance associations through a common interface for named-entity recognition, relation extraction, event extraction, and intent-slot parsing. The contract is paired with five diagnostic matching views---category agreement, field recognition, field-value pairing, instance-bound assignment, and complete-instance recovery---that ask distinct questions about the same output. Using this contract, we normalize 61 datasets into a unified corpus with shared schemas, prompts, and matching rules, and compare six models on a fixed test subset. Our fine-tuned Xing4.0-29B-Intent attains 57.70 task-balanced Field-Value Micro-F1 (equally weighted across the four task families) and ranks first in every family and on all five views; its margin over the strongest alternative widens from 2.93 points for category matching to 20.73 points for complete-instance matching. This widening gap shows that category agreement alone understates structural deficits. USP thus offers the community a shared representation, a normalized 61-dataset corpus, and a diagnostic evaluation protocol that make structural agreement measurable and comparable across extraction and intent-slot tasks.

Figures & tables

Figure 1: An example of schema-conditioned input and structured annotation under the USP contract.

Family	Datasets	Records	Valid	Invalid	Ambig.
NER	28	252,983	95.80	4.20	0.00
RE	13	179,615	96.36	0.33	3.31
EE	9	17,738	58.63	31.24	10.13
Intent-slot	7	28,592	29.30	43.68	27.01
All	57	478,928	90.66	6.11	3.23

Table 1: Contract conformance on the evaluation splits. Percentages in the upper block are computed within each family; the lower block reports shares of all records. Conformance measures structural validity, not source-to-USP semantic fidelity.

Model	EE	NER	RE	IR	Overall	Category	Field	Inst.-bound	Complete	Parse
	Field-Value Micro-F1					Other matching views				Parsing
GLM-4-9B-Chat-HF	8.19	7.29	1.81	75.08	23.09	37.61	27.42	22.92	20.42	45.21
Qwen3-8B-Base	17.17	46.30	15.40	38.94	29.45	55.02	46.56	26.88	19.58	93.34
Qwen3.7-Flash	20.90	55.62	20.36	46.65	35.88	62.15	53.28	31.90	23.56	98.72
DeepSeek-V4-Flash	23.75	55.21	23.84	54.33	39.28	64.72	57.51	35.10	24.76	98.91
DeepSeek-V4-Pro	23.35	59.35	24.02	58.38	41.28	66.38	58.56	40.10	30.15	99.98

Table 2: Main comparison on the fixed test subset.

Figure 2: Task-balanced F1 under five structural matching views. Each view evaluates the same outputs using its own matching units; the profiles do not represent sequential processing stages.

Figure 3: Per-dataset Field-Value Micro-F1 margins of Xing4.0-29B-Intent over the best alternative. Vertical offsets separate overlapping points within each task family.

Figure 4: Parser acceptance and Field-Value Micro-F1 across model–dataset pairs, grouped by task. Parser acceptance includes empty outputs.

Model	EE	NER	RE	IR	Overall	Parse	Tokens
Gemini-3.6-Flash	30.46	70.73	29.34	64.34	48.72	99.40	22.34M
Qwen3.8-Max	28.98	66.15	26.33	63.31	46.19	99.90	15.05M
Kimi-K3	28.32	67.70	24.53	60.59	45.28	99.48	23.07M
GPT-5.6-Sol	28.27	67.87	23.01	60.08	44.81	100.00	16.20M
GLM-5.2	28.70	65.48	25.04	58.01	44.31	99.90	18.98M
GPT-5.5	27.66	67.18	22.91	59.18	44.23	99.97	17.22M

Table 3: Additional model results on USP-Core-6K, ordered by Overall.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Measure	Matching unit	Matching rule	Reporting role
Category	category	Set overlap	Category selection
Field	(category, field)	Set overlap	Field selection
Field-Value	(category, field, value)	Set overlap	Primary extraction measure
Instance-Bound Field	fields within matched instances	One-to-one assignment maximizing field overlap	Instance association
Complete Instance	complete instance	Multiset overlap	Complete structure recovery
Full-event	complete instance	Same implementation as Complete Instance	Compatibility name; no additional view

Appendix

Table 4: Metric definitions and reporting roles. Structural matching views report Micro-F1 within each dataset; task and overall scores macro-average these dataset-level values. Accuracy measures are reported separately.

Model	Core	Full	$\|\Delta\|$	MAE
Qwen3.7-Flash	36.37	35.94	0.43	2.25
DeepSeek-V4-Flash	38.84	39.14	0.30	2.63
DeepSeek-V4-Pro	40.83	41.20	0.37	2.28

Appendix

Table 5: Calibration of the reduced suite against full-benchmark results from the same broader evaluation pool.

|\Delta|

is the absolute difference in task-balanced F1, and MAE is the mean absolute per-dataset difference. The preserved ordering is evidence for screening utility, not universal validity.

Event extraction	Xing4.0- 29B-Intent	DS-V4-P	DS-V4-F	Qwen3.7-F	Qwen3-8B	GLM-4-9B
CASIE	—	9.77	11.94	8.68	8.16	5.56
CrudeOil	48.90	8.70	6.57	9.31	4.82	2.69
DuEE	75.70	57.74	59.75	54.91	44.34	21.64
DuEE-Fin	80.55	47.00	49.64	47.46	45.82	28.06
FewFC	31.77	0.03	0.00	0.05	0.05	0.00
PHEE	72.45	41.97	41.56	37.61	32.52	7.58

Appendix

Table 6: Field-Value Micro-F1 on the event extraction datasets in the fixed test subset. Models are ordered by recorded overall Field-Value F1; bold marks row maxima among the displayed scores.

Named-entity recognition	Xing4.0- 29B-Intent	DS-V4-P	DS-V4-F	Qwen3.7-F	Qwen3-8B	GLM-4-9B
AnatEM	74.91	43.20	45.45	41.44	43.80	10.98
BC2GM	76.39	60.78	44.62	59.28	53.98	13.29
BC5CDR	82.35	73.98	68.24	67.33	50.16	2.97
BioRED	76.85	65.77	60.14	61.35	49.50	3.39
Boson	68.65	27.12	12.93	31.16	28.12	4.12
CN-AI	58.69	63.88	58.43	56.60	43.42	7.92

Appendix

Table 7: Field-Value Micro-F1 on the named-entity recognition datasets in the fixed test subset. Models are ordered by overall Field-Value F1; bold marks row maxima.

Relation extraction	Xing4.0- 29B-Intent	DS-V4-P	DS-V4-F	Qwen3.7-F	Qwen3-8B	GLM-4-9B
CoNLL04	61.92	56.17	53.38	44.48	44.08	5.15
DocRED	29.87	25.23	27.29	24.68	10.33	0.26
FewRel	14.86	17.06	17.09	15.97	13.01	0.90
FinRED	24.23	10.47	9.59	5.77	4.11	0.78
GIDS	61.95	29.22	29.89	26.88	23.07	3.99
HacRED	54.02	38.47	38.40	31.64	13.63	1.66

Appendix

Table 8: Field-Value Micro-F1 on the relation extraction datasets in the fixed test subset. Models are ordered by overall Field-Value F1; bold marks row maxima.

Intent-slot understanding	Xing4.0- 29B-Intent	DS-V4-P	DS-V4-F	Qwen3.7-F	Qwen3-8B	GLM-4-9B
ATIS	88.44	76.73	82.55	71.77	57.60	92.31
CAIS	82.10	71.01	68.66	61.76	56.49	87.48
MulWOZ	84.40	45.89	44.19	35.33	30.59	5.89
SMP19	74.13	45.63	34.78	28.61	20.23	86.15
SMP20	66.20	40.16	31.95	30.36	23.14	86.02
SNIPS	90.22	70.86	63.86	52.07	45.60	92.64

Appendix

Table 9: Field-Value Micro-F1 on the intent-slot understanding datasets in the fixed test subset. Models are ordered by overall Field-Value F1; bold marks row maxima.

No highest-observed cell: Qwen3-8B-Base , Qwen3.7-Flash
Model	EE	NER	RE	IR	Overall
Xing4.0-29B-Intent	5	17	8	1	31
DeepSeek-V4-Pro	2	5	0	0	7
GLM-4-9B-Chat-HF	0	0	0	5	5
DeepSeek-V4-Flash	1	0	1	0	2

Appendix

Table 10: Fractional observed-best counts for Field-Value Micro-F1, ordered by total. Tied maxima share the count equally. Counts include the main protocol’s failure handling and do not indicate statistical significance.

Model	Params	Track	Accuracy	95% CI	Fmt. Compl.	Parse Rate
External comparators
GLM-4-9B-Chat-HF	9 B	I	0.3286	[0.3202, 0.3369]	0.6990	0.7415
Qwen3-8B-Base	8 B	I	0.5998	[0.5911, 0.6084]	0.9704	0.9503
Reported SFT checkpoint
Xing4.0-29B-Intent	29/4 B	I	0.6290 †	[0.6204, 0.6377]	0.8974	0.9015

Appendix

Table 11: MMLU-Pro accuracy on

12{,}032

questions under a shared zero-shot boxed-answer protocol. Empty or unparseable answers count as incorrect; confidence intervals use paired question-level bootstrap. Format completion detects a boxed marker, while Parse Rate measures answer extraction. Results are not directly comparable with the public five-shot leaderboard.

Task	Dataset	Lang.	#Schemas	Cite
NER-en	AnatEM	en	1	Pyysalo and Ananiadou (2014)
	BC2GM	en	1	Smith et al. (2008)
	BC4CHEMD $\ast$	en	1	Krallinger et al. (2015)
	NCBI-Disease $\ast$	en	1	Dogan et al. (2014)
	BC5CDR	en	2	Li et al. (2016)
	HarveyNER	en	4	Chen et al. (2022)

Appendix

Table 12: Statistics of the 29 NER datasets in USP: task subgroup, source language, per-dataset schema count from the dataset inventory, and the originating publication. NCBI-Disease is outside the fixed test subset because of documented task or split fallback; the other datasets marked

\ast

are outside the test subset because of partial model coverage. Datasets marked

\ddagger

do not have a verified origin publication. Per-split instance counts and source URLs are provided on the dataset page.

Task	Dataset	Lang.	#Schemas	Cite
RE-en	ADE Corpus	en	1	Gurulingappa et al. (2012)
	GIDS	en	4	Jat et al. (2017)
	CoNLL2004	en	5	Roth and Yih (2004)
	SciERC	en	7	Luan et al. (2018)
	Semeval-RE	en	10	Hendrickx et al. (2010)
	KBP37	en	18	Zhang and Wang (2015)

Appendix

Table 13: Statistics of the 16 RE datasets in USP: task subgroup, source language, per-dataset schema count from the dataset inventory, and the originating publication. Wiki-ZSL and CMeIE are outside the fixed test subset because of documented task or split fallback; DuIE2.0, also marked

\ast

, is outside the test subset because of partial model coverage. Datasets marked

\ddagger

do not have a verified origin publication and are cited by a shared-task placeholder. DuIE2.0 has no released training split. Per-split instance counts and source URLs are provided on the dataset page.

Task	Dataset	Lang.	#Event types (#Roles)	Cite
EE-en	CASIE	en	5 (26)	Satyapanich et al. (2020)
	PHEE	en	2 (16)	Sun et al. (2022)
	CrudeOilNews	en	18 (104)	Lee et al. (2022b)
	RAMS	en	106 (398)	Ebner et al. (2020)
	WikiEvents	en	31 (81)	Li et al. (2021)
EE-zh	DuEE1.0	zh	65 (217)	Li et al. (2020)

Appendix

Table 14: Statistics of the 9 EE and 7 Intent/Slot SLU datasets in USP: task subgroup, source language, per-dataset schema count from the dataset inventory (for EE, #event types with #argument roles in parentheses; the same convention applies to SLU intents), and the originating publication. MASSIVE, marked

\dagger

, was normalized into the construction inventory but excluded from the current evaluation due to size. An unavailable schema count is shown as “—” rather than inferred; per-split instance counts and source URLs are provided on the dataset page.

Table 15: USP schema and reference annotation examples across four task families. Schemas show relevant categories and fields; category and argument descriptions are reproduced in full for all displayed fields.

Semantic Parsing

Semantic parsing maps natural language into structured representations, such as logical forms, graphs, or queries, that machines can execute or reason over.

See all 30 papers

Adapt Semantics, Not Structure: Few-Instance Schema Calibration for Scientific PDF Extraction
Sep 28, 2026Zixiao Dong, Wei Yang, Zihao Liu +4Document Information Extraction LLM Information Extraction
School of Computer Science and Technology, University of Science and Technology of China · State Key Laboratory of Cognitive Intelligence · University of Science and Technology of China +1
Calibrated Typed Decisions for Entity, Relation, and Event Extraction
Oct 12, 2026Yang Wang, Hanlin Xiao, Yanan Ma +2Event Extraction Document Information Extraction
University of Manchester · Mayo Clinic · Shanghai University of Finance and Economics
ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction
Jul 31, 2026Boyang Zhang, Adrian Lyjak, Eli Stewart +2LLM Grounding LLM Agent Evaluation

Sep 28, 2026J/K move · Enter open · S save

Adapt Semantics, Not Structure: Few-Instance Schema Calibration for Scientific PDF Extraction

Zixiao Dong, Wei Yang, Zihao Liu, Chenshu Li+3

School of Computer Science and Technology, University of Science and Technology of China · State Key Laboratory of Cognitive Intelligence · University of Science and Technology of China+1

Document Information Extraction LLM Information Extraction

Open paper AskPDF arXiv

A well-designed extraction schema is not necessarily ready for reliable LLM execution. When only limited verified extractions are available, manually tuning hundreds of field definitions through trial and error is costly. We frame this problem as few-instance schema calibration: adapting the operational semantics of an existing schema from a few annotated documents while preserving its structural contract. We introduce CPSE, a contract-preserving semantic extraction framework that jointly calibrates extraction prompts and field-level semantic descriptions from a few gold annotations. CPSE decomposes the schema into an invariant structural contract and mutable field semantics, and further separates identity discovery from record completion using manifest-conditioned resolution. On expert-annotated polymer-science documents, CPSE improves extraction by 9.93 points over an execution-matched baseline, with consistent gains under an independent judge and in a blinded expert audit. These results show that CPSE enables low-resource schema execution while preserving the output structure required downstream.

Abstract

Semantic Parsing

Explore similar work

Adapt Semantics, Not Structure: Few-Instance Schema Calibration for Scientific PDF Extraction

Calibrated Typed Decisions for Entity, Relation, and Event Extraction

ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction