Learning to Generate and Extract: A Multi-Agent Collaboration Framework For Zero-shot Document-level Event Arguments Extraction
Authors: Guangjun Zhang, Hu Zhang, Yazhou Han, Yue Fan, Yuhang Shao, Ru Li, Hongye Tan
Organizations: School of Computer and Information Technology, Shanxi University, Taiyuan, China · Key Laboratory of Computational Intelligence and Chinese Information Processing of Ministry of Education, Shanxi University, Taiyuan, China
Document-level event argument extraction (DEAE) is essential for knowledge acquisition, aiming to extract participants of events from documents . In the zero-shot setting, existing methods employ LLMs to generate synthetic data to address the challenge posed by the scarcity of annotated data. However, relying solely on Event-type-only prompts makes it difficult for the generated content to accurately capture the contextual and structural relationships of unseen events. Moreover, ensuring the reliability and usability of synthetic data remains a significant challenge due to the absence of quality evaluation mechanisms. To this end, we introduce a multi-agent collaboration framework for zero-shot document-level event argument extraction (ZS-DEAE), which simulates the human collaborative cognitive process of "Propose-Evaluate-Revise." Specifically, the framework comprises a generation agent and an evaluation agent. The generation agent synthesizes data for unseen events by leveraging knowledge from seen events, while the evaluation agent extracts arguments from the synthetic data and assesses their semantic consistency with the context. The evaluation results are subsequently converted into reward signals, with event structure constraints incorporated into the reward design to enable iterative optimization of both agents via reinforcement learning.In three zero-shot scenarios constructed from the RAMS and WikiEvents datasets, our method achieves improvements both in data generation quality and argument extraction performance, while the generated data also effectively enhances the zero-shot performance of other DEAE models.
Figures & tables
Figure 1: Examples of DEAE data generated by GPT-4o.
Figure 2: The overall workflow of multi-agent collaboration for ZS-DEAE
Model
RAMS2RAMS
RAMS2Wiki
Wiki2Wiki
Seen R.
Unseen R.
Overall
Seen R.
Unseen R.
Overall
Seen R.
Unseen R.
Overall
DEAE Models
PAIE
32.52
28.87
30.80
19.57
31.72
20.15
23.58
23.57
24.42
TabEAE
37.16
35.26
36.22
16.94
35.05
26.74
37.19
28.84
30.97
DEEIA
36.57
39.49
37.95
1.50
7.17
5.12
34.11
19.48
22.51
HMPEAE
35.18
37.74
36.44
16.89
32.74
25.61
38.43
27.48
30.20
Table 1: Main Results on three zero-shot settings. Seen R. refers to roles shared between seen and unseen event schemas, while Unseen R. denotes roles unique to unseen schemas. Overall indicates the F1 score calculated over all roles. We use bold text to indicate the best performance and underline to indicate the second best. For ZSTL and LLM-based methods, since they are not trained on Ds but instead make predictions directly via queries or prompts, they do not distinguish between seen and unseen roles—thus, the corresponding entries are left blank.
Model
R2R
R2W
W2W
Ours((LLaMA)
45.28
32.38
46.96
-reward
42.46
24.93
38.39
-constraint
44.72
29.03
40.98
Ours(QWEN)
44.59
31.18
47.62
-reward
40.52
27.19
43.30
-constraint
40.87
27.87
38.70
Table 2: Ablation study on the effects of reward and constraint across three datasets. R2R : RAMS2RAMS . R2W : RAMS2Wiki , W2W : Wiki2Wiki
Model
Data
RAMS2RAMS
RAMS2Wiki
Wiki2Wiki
TabEAE
Seen data
36.22
26.74
30.97
+ Synthetic data ((LLaMA 3.1)
37.37
26.17
12.48
+ Synthetic data (Ours)
44.43
28.42
33.36
Bart-Gen
Seen data
38.53
28.52
40.82
+ Synthetic data (LLaMA 3.1)
41.04
28.95
44.17
+ Synthetic data (Ours)
46.06
32.38
46.96
Table 3: Performance comparison on three transfer settings using unseen and synthetic data.
Figure 3: Proportion of empty arguments in synthetic data.
Figure 4: (a): Loss of the two agent during reinforcement learning on RAMS2RAMS . (b): F1 trend across three datasets.
Figure 5: Log-likelihood scores assigned by the evaluation agent to different types of synthetic data. Empty: randomly removing arguments. Mismatch: randomly swapping argument-role mappings. Normal: correctly generated data without perturbation.
Zero-shot information extraction (IE) with large language models (LLMs) enables adaptation to new schemas and domains without task-specific training. Existing methods mainly follow three paradigms. Monolithic prompting is efficient but prone to missed mentions, boundary errors, and type confusion. Each-type prompting improves type-level focus but may produce overlapping or conflicting predictions, while multi-agent debate can resolve such conflicts at the cost of irrelevant context, redundant interactions, and high token overhead. To address these issues, we propose SMADE-IE, a sparse and evidence-driven multi-agent framework. An Adaptive Mode Selector routes simple inputs to a lightweight Global Extraction Mode and ambiguous inputs to a Type-Centric Extraction Mode based on sample complexity and relevant types. Cross-type conflicts are resolved by an Evidence-Driven Debate module that uses Toulmin-style arguments, external evidence scoring, Beta-based confidence updates, and early stopping. Experiments on nine benchmarks covering NER, RE, and JERE show that SMADE-IE improves average Partial F1 over the strongest baselines by 11.37, 3.83, and 14.46 points, respectively. Compared with the multi-agent baseline CrossAgentIE, SMADE-IE reduces token consumption by 85.1% on DocRED and 80.1% on CrossRE, demonstrating substantially higher inference efficiency. Code is available at https://github.com/Cppys/SMADE-IE.
Kenfeng Huang, Yi Cai, Xin Wu +2
School of Software Engineering, South China University of Technology, Guangzhou, China
Event Argument Extraction (EAE) converts documents into structured event records by identifying argument spans and assigning them schema-defined roles. Document-level EAE is challenging due to long-range dependencies between triggers and arguments, cross-sentence context, and strict role constraints, which often lead to boundary errors, uncertainty in roles, and inconsistencies with restricted schemas. In this paper, we study whether mid-sized open LLMs can perform schema-constrained EAE reliably at the document level on MAVEN-ARG. Our approach combines (i) role-set injection in prompts for schema compliance, (ii) parameter-efficient supervised fine-tuning (LoRA) using the same JSON-only interface used at inference, and (iii) deterministic decoding with post-processing that validates JSON, filters invalid roles, de-duplicates arguments, and aligns spans to the document window. Under the official MAVEN-ARG evaluator, fine-tuned mid-sized open models outperform previously reported GPT baselines across mention, entity-coreference, and event-coreference evaluations; our best model (Phi-4, 14B) reaches 42.39% F1 at the event-coreference level. Code to reproduce experiments is publicly available at https://github.com/dessertlab/EAE/.
Roberto Pietrantuono, Antonio Guerriero, Pouya Sattari
University of Naples Federico II, Naples, Italy · University of Salerno, Fisciano, Italy
End-to-end event extraction remains challenging for large language models as it requires simultaneous identification of event triggers, classification of event types, and extraction of schema-grounded argument spans. We present EAGER, a reinforcement learning framework for generative event extraction that combines fine-grained verifiable rewards with Schema-Contrastive Advantage Estimation to alleviate advantage collapse under sparse binary rewards. Our reward design explicitly targets structural validity, extraction accuracy, groundedness, coverage, over-generation, and span precision. Experiments across seven benchmark datasets show that EAGER consistently outperforms prompting, supervised fine-tuning, and prior reinforcement learning baselines, achieving a substantial improvement over the strongest prior method. Results demonstrate that task-aligned verifiable rewards and contrastive advantage estimation substantially improve structured extraction.