ECHO: Event-Centric Hypergraph Operations via Multi-Agent Collaboration for Multimedia Event Extraction
Authors: Hailong Chu, Hongbing Li, Yunlong Chu, Shutai Huang, Xingyue Zhang, Tinghe Yan, Jinsong Zhang, Shuo Zhang, Lei Li
Organizations: Beijing University of Posts and Telecommunications Beijing, China · Tianjin University Tianjin, China · Chongqing University of Posts and Telecommunications Chongqing, China · Harbin Institute of Technology Harbin, China
Multimedia event extraction (M2E2) aims to predict triggers, ground arguments across text and images, and then assemble them into schema-consistent event records. Recent LLM-based approaches have shown strong potential for M2E2, but their intermediate event hypotheses often remain implicit, and event-argument linking is still tightly coupled with role binding. This leaves little opportunity to inspect or revise intermediate event hypotheses and makes predictions brittle to early errors. To bridge this gap, we present ECHO, a multi-agent framework that reframes M2E2 as iterative refinement over an explicit Multimedia Event Hypergraph (MEHG). Instead of relying on implicit linear generation, ECHO performs auditable atomic updates over a shared hypergraph, making intermediate event structures explicit and revisable. Furthermore, we introduce a Link-then-Bind strategy that decouples event-argument linking from role binding, reducing premature semantic commitment during structured prediction. Extensive experiments on the M2E2 benchmark show that ECHO consistently outperforms prior state-of-the-art approaches, achieving gains of 7.3 and 15.5 F1 points on event mention and argument role, respectively.
Figures & tables
Figure 1. Illustrative comparison of M2E2 paradigms on the same multimedia document. (a) Traditional encoder-based systems rely on implicit event-level structure and may suffer from the heterogeneity gap. (b) Directly prompted LLMs/LVLMs entangle event identification, argument linking, and role assignment in one-shot generation, which can lead to hallucinated bindings. (c) ECHO performs iterative multi-agent revision over an explicit intermediate event structure through node seeding, MEHG construction, and role binding.
Figure 2. Overview of ECHO. Given D=(T,I) , Stage I constructs the vertex inventory and initializes an edge-free MEHG; Stage II agents negotiate MEHG updates via auditable atomic operations; Stage III performs role binding and consolidation to produce schema-consistent event predictions. The purple hyperedge represents a concurrent event hypothesis generated via the same ecosystem. Inside the hyperedge, ye , tk , and ck denote the event type, trigger, and confidence score, respectively.
Model
Venue
Textual Events
Visual Events
Multimedia Events
Event Mention
Argument Role
Event Mention
Argument Role
Event Mention
Argument Role
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
Flat
ACL’20
34.2
63.2
44.4
20.1
27.1
23.1
27.1
57.3
36.7
4.3
8.9
5.8
33.9
59.8
42.2
12.9
17.6
14.9
WASE-T
ACL’20
42.3
58.4
48.2
21.4
30.1
24.9
–
–
WASE-V_att
ACL’20
–
29.7
61.9
40.1
9.1
10.2
9.6
–
WASE-V_obj
ACL’20
–
28.6
59.2
38.7
13.3
9.8
11.2
–
Table 1. Main results of ECHO on event mention and argument role across modalities. Best and second-best F1 are in bold and underlined , respectively. DeepSeek uses DeepSeek-V3.2.
Figure 3. F1 comparison of Direct prompting, Dialogue-Mediated baseline, and ECHO on M2E2 across all settings.
Figure 4. Ablation on the multimedia setting of M2E2. We report F1 for EM and AR with three backbones.
Qwen3-VL-8B Thinking
Qwen3-VL-30B-A3B Thinking
Setting
P
R
F1
P
R
F1
Qwen3-8B
Visual
61.6
60.0
60.8
63.1
62.5
62.8 (+2.0)
Multimedia
47.0
58.9
52.3
49.5
64.5
56.0 (+3.7)
Qwen3-32B
Visual
61.8
57.6
59.6
66.0
60.0
62.9 (+3.3)
Table 2. Vision tool ablation on AR. Replacing the default vision tool with a stronger one consistently improves visual and multimedia AR.
Figure 5. Negotiation budget analysis for Stage II under early stopping. (a) Proportion of samples that stop by round k (i.e., Tused≤k ) when varying the maximum budget Tmax . (b) Distribution of the number of committed atomic operations per sample. (c) Sensitivity of extraction performance to Tmax , reported as Δ F1 (percentage points) relative to Tmax=2 .
AR error type
Textual
Visual
Multimedia
Spurious
540
0
180
Span mismatch
552
0
181
Role misassignment
60
12
54
No-gold event type
768
111
208
Localization error
0
173
400
Total
1920
296
1023
Table 3. AR error counts for Qwen3-32B under ECHO on M2E2.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Event Type
Argument Role
Movement:Transport
Agent, Artifact, Vehicle, Destination, Origin
Conflict:Attack
Attacker, Target, Instru- ment, Place
Conflict:Demonstrate
Entity, Police, Instru- ment, Place
Justice:Arrest-Jail
Agent, Person, Instru- ment, Place
Contact:PhoneWrite
Entity, Instrument, Place
Contact:Meet
Participant, Place
Appendix
Table 4. Event types and corresponding argument roles in M2E2.
Method
Textual Events
Visual Events
Multimedia Events
Event Mention
Argument Role
Event Mention
Argument Role
Event Mention
Argument Role
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
X-MTL
49.7
65.7
56.6
34.6
37.6
36.0
73.1
70.3
71.7
33.2
31.3
32.2
78.3
57.3
66.2
40.3
42.6
41.4
Qwen2.5-VL
48.0
38.8
42.9
10.3
13.9
11.8
82.1
80.9
81.5
50.0
26.7
34.8
34.8
35.0
34.9
28.0
18.1
22.0
Qwen3-VL
56.9
49.4
52.9
10.3
17.1
12.9
75.1
85.1
79.8
20.6
21.8
21.2
61.7
61.7
61.7
21.1
24.6
22.7
Qwen3-8B
62.5
43.8
51.5
14.2
16.9
15.5
77.0
60.6
67.9
39.3
21.8
28.1
29.0
32.0
30.5
19.4
18.6
19.0
Appendix
Table 5. Detailed evaluation results of Direct Prompting LLMs/LVLMs. DS denotes DeepSeek models. The table reports Precision (P), Recall (R), and F1 scores across textual, visual, and multimedia settings. Bold and underlined values denote the best and second-best F1 scores in each column, respectively. All methods use the same default vision tool outputs generated by Qwen3-VL-8B-Thinking.
Method
Textual Events
Visual Events
Multimedia Events
Event Mention
Argument Role
Event Mention
Argument Role
Event Mention
Argument Role
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
X-MTL
49.7
65.7
56.6
34.6
37.6
36.0
73.1
70.3
71.7
33.2
31.3
32.2
78.3
57.3
66.2
40.3
42.6
41.4
Qwen3-8B
50.3
65.6
56.9
20.5
41.2
27.4
65.0
84.5
73.5
28.2
57.6
37.9
44.7
71.4
55.0
26.3
61.0
36.8
Qwen3-32B
51.3
67.7
58.3
21.2
43.5
28.6
67.1
83.3
74.3
46.7
52.9
49.6
48.4
77.6
59.6
43.4
42.7
43.0
DS-V3.2
59.0
69.0
63.6
21.7
39.4
28.0
74.0
84.6
78.9
52.6
50.9
51.7
60.0
83.8
69.9
30.0
55.2
38.8
Appendix
Table 6. Detailed evaluation results of the dialogue-mediated baseline. DS denotes DeepSeek models. The table reports Precision (P), Recall (R), and F1 scores across textual, visual, and multimedia settings. Bold and underlined values denote the best and second-best F1 scores in each column, respectively. This baseline uses the same default vision tool outputs generated by Qwen3-VL-8B-Thinking.
Method
Qwen3-8B
Qwen3-32B
DeepSeek-v3.2
Event Mention
Argument Role
Event Mention
Argument Role
Event Mention
Argument Role
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
Full Model
70.6
69.9
70.2
47.0
58.9
52.3
72.5
72.5
72.5
49.8
61.2
54.9
75.1
75.1
75.1
47.2
65.8
55.0
- w/o Linker
69.9
69.9
69.9
49.7
45.8
47.7
70.6
70.6
70.6
42.1
37.3
39.6
74.8
74.8
74.8
45.2
40.3
42.6
- w/o Verifier
68.9
68.9
68.9
46.0
56.8
50.8
71.8
71.8
71.8
44.8
50.8
47.6
74.8
74.8
74.8
46.6
56.4
51.0
- w/o SpanAlign
64.7
64.7
64.7
44.5
54.9
49.1
67.3
67.3
67.3
38.5
48.5
42.9
74.8
74.8
74.8
38.6
47.7
42.6
Appendix
Table 7. Full ablation results in the multimedia setting (P/R/F1). We evaluate ECHO across different backbones: Qwen3 (8B, 32B) and DeepSeek-V3.2. The ablations test the impact of removing the Linker (relevance linking), Verifier, SpanAlign (Stage III text span alignment), and Link-then-Bind. All variants use the same default vision tool outputs.
Table 8. Audit-trail excerpt for the representative transport example discussed in this subsection. Representative committed updates are shown; P / L / V / B denote Proposer, Linker, Verifier, and Role Binder, respectively, and adj. conf. denotes adjust_confidence . Additional linking steps are omitted for brevity.
AR error type
Textual
Visual
Multimedia
Spurious
540
0
180
Span mismatch
552
0
181
Role misassignment
60
12
54
No-gold event type
768
111
208
Localization error
0
173
400
Total
1920
296
1023
Appendix
Table 9. AR error counts for Qwen3-32B under ECHO on M2E2.
String relation
Textual
Multimedia
No-gold
1536
184
Exact
987
282
None
994
329
Contained-by
65
20
Contains
36
14
Overlap
3
0
Appendix
Table 10. Normalized string relations between predicted and gold text arguments for Qwen3-32B under ECHO.
Setting
Source
Most frequent misassigned roles
Textual
Text
Agent (18), Target (10), Attacker (9), Artifact (7), Place (6)
Visual
Box
Entity (5), Agent (5), Recipient(1), Artifact (1)
Multimedia
Text
Target (7), Place (6), Attacker(3), Agent (2), Entity (1)
Table 11. Most frequent roles involved in Argument Role misassignments for Qwen3-32B under ECHO.
Setting
Spurious
Missing
Trigger mismatch
Textual
469
371
129
Visual
37
37
0
Multimedia
85
85
52
Appendix
Table 12. EM error counts for Qwen3-32B under ECHO.
Textual (Gold = 1659)
Model
Pred
Matched
FP
P(%)
R(%)
OverGen
Qwen3-8B
2771
893
1878
32.2
53.8
1.67
Qwen3-14B
2620
881
1739
33.6
53.1
1.58
Qwen3-32B
2853
933
1920
32.7
56.2
1.72
DS-V3.2
2936
949
1987
32.3
57.2
1.77
GPT-5
3796
1033
2763
27.2
62.3
2.29
Appendix
Table 13. Argument Role (AR) output statistics under ECHO on M2E2. DS denotes DeepSeek models. Matched counts are computed using the same official AR matching protocol as in the main evaluation. OverGen denotes the over-generation ratio ( Pred / Gold ).
Setting
Interaction Cost (Number of Calls)
Computation Cost (Token Usage)
Dialogue-mediated
ECHO (Ours)
Dialogue-mediated
ECHO (Ours)
Main
Vis
Total
Main
Vis
Total
Input
Output
Total
Input
Output
Total
Textual
8
0
8
9
0
9
29,110
6,065
35,175
14,416
13,141
27,557
Visual
7
2
9
8
2
10
42,384
7,996
50,380
28,191
16,812
45,003
Multimedia
8
2
10
9
2
11
54,460
13,123
67,582
34,695
22,210
56,905
Appendix
Table 14. Interaction and computation cost under an idealized schedule. We assume Tmax=2 rounds with no early stopping and a single extracted event, and report (left) the number of main-model and vision-tool calls and (right) token usage (input/output/total) for ECHO and the dialogue-mediated baseline across textual, visual, and multimedia settings. This table is intended as an illustrative cost accounting under the controlled protocol rather than an average empirical runtime summary. In practice, call counts can be lower due to early stopping ( Tused<Tmax ) and vary with the number of extracted events.
Event linking associates event mentions in text with entries in a knowledge base (KB), or identifies them as out-of-KB events. Although existing methods use different architectures, candidate event acquisition can still be weakened by short ambiguous mentions, noisy arguments, and evidence that is unevenly useful for retrieval. We present MACE, a Multi-Agent Candidate Event acquisition method that refines event structure before linking. MACE uses evidence-specialized LLM agents to acquire time, location, participant, and event-type evidence, exposes intermediate queries to candidate-event lookup tools, and lets a coordinator revise the evidence set before final candidate construction. Experiments on two event linking benchmarks show that adding MACE to different event linking models consistently improves accuracy. These results show that MACE improves event linking through better candidate event acquisition without modifying the event linking model.
Ziyang Zhang, Yinan Liu, Boyi Xue +3
School of Computer Science and Engineering, Northeastern University Shenyang 110819, China
Event Argument Extraction (EAE) converts documents into structured event records by identifying argument spans and assigning them schema-defined roles. Document-level EAE is challenging due to long-range dependencies between triggers and arguments, cross-sentence context, and strict role constraints, which often lead to boundary errors, uncertainty in roles, and inconsistencies with restricted schemas. In this paper, we study whether mid-sized open LLMs can perform schema-constrained EAE reliably at the document level on MAVEN-ARG. Our approach combines (i) role-set injection in prompts for schema compliance, (ii) parameter-efficient supervised fine-tuning (LoRA) using the same JSON-only interface used at inference, and (iii) deterministic decoding with post-processing that validates JSON, filters invalid roles, de-duplicates arguments, and aligns spans to the document window. Under the official MAVEN-ARG evaluator, fine-tuned mid-sized open models outperform previously reported GPT baselines across mention, entity-coreference, and event-coreference evaluations; our best model (Phi-4, 14B) reaches 42.39% F1 at the event-coreference level. Code to reproduce experiments is publicly available at https://github.com/dessertlab/EAE/.
Roberto Pietrantuono, Antonio Guerriero, Pouya Sattari
University of Naples Federico II, Naples, Italy · University of Salerno, Fisciano, Italy
End-to-end event extraction remains challenging for large language models as it requires simultaneous identification of event triggers, classification of event types, and extraction of schema-grounded argument spans. We present EAGER, a reinforcement learning framework for generative event extraction that combines fine-grained verifiable rewards with Schema-Contrastive Advantage Estimation to alleviate advantage collapse under sparse binary rewards. Our reward design explicitly targets structural validity, extraction accuracy, groundedness, coverage, over-generation, and span precision. Experiments across seven benchmark datasets show that EAGER consistently outperforms prompting, supervised fine-tuning, and prior reinforcement learning baselines, achieving a substantial improvement over the strongest prior method. Results demonstrate that task-aligned verifiable rewards and contrastive advantage estimation substantially improve structured extraction.