ECHO: Event-Centric Hypergraph Operations via Multi-Agent Collaboration for Multimedia Event Extraction

Hailong Chu; Hongbing Li; Yunlong Chu; Shutai Huang; Xingyue Zhang; Tinghe Yan; Jinsong Zhang; Shuo Zhang

cs.CVMar 4, 2026

ECHO: Event-Centric Hypergraph Operations via Multi-Agent Collaboration for Multimedia Event Extraction

Authors: Hailong Chu, Hongbing Li, Yunlong Chu, Shutai Huang, Xingyue Zhang, Tinghe Yan, Jinsong Zhang, Shuo Zhang,

Organizations: Beijing University of Posts and Telecommunications Beijing, China · Tianjin University Tianjin, China · Chongqing University of Posts and Telecommunications Chongqing, China · Harbin Institute of Technology Harbin, China

Abstract

Multimedia event extraction (M2E2) aims to predict triggers, ground arguments across text and images, and then assemble them into schema-consistent event records. Recent LLM-based approaches have shown strong potential for M2E2, but their intermediate event hypotheses often remain implicit, and event-argument linking is still tightly coupled with role binding. This leaves little opportunity to inspect or revise intermediate event hypotheses and makes predictions brittle to early errors. To bridge this gap, we present ECHO, a multi-agent framework that reframes M2E2 as iterative refinement over an explicit Multimedia Event Hypergraph (MEHG). Instead of relying on implicit linear generation, ECHO performs auditable atomic updates over a shared hypergraph, making intermediate event structures explicit and revisable. Furthermore, we introduce a Link-then-Bind strategy that decouples event-argument linking from role binding, reducing premature semantic commitment during structured prediction. Extensive experiments on the M2E2 benchmark show that ECHO consistently outperforms prior state-of-the-art approaches, achieving gains of 7.3 and 15.5 F1 points on event mention and argument role, respectively.

Figures & tables

Figure 1. Illustrative comparison of M2E2 paradigms on the same multimedia document. (a) Traditional encoder-based systems rely on implicit event-level structure and may suffer from the heterogeneity gap. (b) Directly prompted LLMs/LVLMs entangle event identification, argument linking, and role assignment in one-shot generation, which can lead to hallucinated bindings. (c) ECHO performs iterative multi-agent revision over an explicit intermediate event structure through node seeding, MEHG construction, and role binding.

Figure 2. Overview of ECHO. Given $D=(T,I)$ , Stage I constructs the vertex inventory and initializes an edge-free MEHG; Stage II agents negotiate MEHG updates via auditable atomic operations; Stage III performs role binding and consolidation to produce schema-consistent event predictions. The purple hyperedge represents a concurrent event hypothesis generated via the same ecosystem. Inside the hyperedge, $y^{e}$ , $t_{k}$ , and $c_{k}$ denote the event type, trigger, and confidence score, respectively.

Model	Venue	Textual Events						Visual Events						Multimedia Events
		Event Mention			Argument Role			Event Mention			Argument Role			Event Mention			Argument Role
		P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Flat	ACL’20	34.2	63.2	44.4	20.1	27.1	23.1	27.1	57.3	36.7	4.3	8.9	5.8	33.9	59.8	42.2	12.9	17.6	14.9
WASE-T	ACL’20	42.3	58.4	48.2	21.4	30.1	24.9	–						–
WASE-V_att	ACL’20	–						29.7	61.9	40.1	9.1	10.2	9.6	–
WASE-V_obj	ACL’20	–						28.6	59.2	38.7	13.3	9.8	11.2	–

Table 1. Main results of ECHO on event mention and argument role across modalities. Best and second-best F1 are in bold and underlined , respectively. DeepSeek uses DeepSeek-V3.2.

Figure 3. F1 comparison of Direct prompting, Dialogue-Mediated baseline, and ECHO on M2E2 across all settings.

Figure 4. Ablation on the multimedia setting of M2E2. We report F1 for EM and AR with three backbones.

	Qwen3-VL-8B Thinking			Qwen3-VL-30B-A3B Thinking
Setting	P	R	F1	P	R	F1
Qwen3-8B
Visual	61.6	60.0	60.8	63.1	62.5	62.8 (+2.0)
Multimedia	47.0	58.9	52.3	49.5	64.5	56.0 (+3.7)
Qwen3-32B
Visual	61.8	57.6	59.6	66.0	60.0	62.9 (+3.3)

Table 2. Vision tool ablation on AR. Replacing the default vision tool with a stronger one consistently improves visual and multimedia AR.

Figure 5. Negotiation budget analysis for Stage II under early stopping. (a) Proportion of samples that stop by round

k

(i.e.,

T_{\mathrm{used}}\leq k

) when varying the maximum budget

T_{\max}

. (b) Distribution of the number of committed atomic operations per sample. (c) Sensitivity of extraction performance to

T_{\max}

, reported as

\Delta

F1 (percentage points) relative to

T_{\max}{=}2

AR error type	Textual	Visual	Multimedia
Spurious	540	0	180
Span mismatch	552	0	181
Role misassignment	60	12	54
No-gold event type	768	111	208
Localization error	0	173	400
Total	1920	296	1023

Table 3. AR error counts for Qwen3-32B under ECHO on M2E2.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Event Type	Argument Role
Movement:Transport	Agent, Artifact, Vehicle, Destination, Origin
Conflict:Attack	Attacker, Target, Instru- ment, Place
Conflict:Demonstrate	Entity, Police, Instru- ment, Place
Justice:Arrest-Jail	Agent, Person, Instru- ment, Place
Contact:PhoneWrite	Entity, Instrument, Place
Contact:Meet	Participant, Place

Appendix

Table 4. Event types and corresponding argument roles in M2E2.

Method	Textual Events						Visual Events						Multimedia Events
	Event Mention			Argument Role			Event Mention			Argument Role			Event Mention			Argument Role
	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
X-MTL	49.7	65.7	56.6	34.6	37.6	36.0	73.1	70.3	71.7	33.2	31.3	32.2	78.3	57.3	66.2	40.3	42.6	41.4
Qwen2.5-VL	48.0	38.8	42.9	10.3	13.9	11.8	82.1	80.9	81.5	50.0	26.7	34.8	34.8	35.0	34.9	28.0	18.1	22.0
Qwen3-VL	56.9	49.4	52.9	10.3	17.1	12.9	75.1	85.1	79.8	20.6	21.8	21.2	61.7	61.7	61.7	21.1	24.6	22.7
Qwen3-8B	62.5	43.8	51.5	14.2	16.9	15.5	77.0	60.6	67.9	39.3	21.8	28.1	29.0	32.0	30.5	19.4	18.6	19.0

Appendix

Table 5. Detailed evaluation results of Direct Prompting LLMs/LVLMs. DS denotes DeepSeek models. The table reports Precision (P), Recall (R), and F1 scores across textual, visual, and multimedia settings. Bold and underlined values denote the best and second-best F1 scores in each column, respectively. All methods use the same default vision tool outputs generated by Qwen3-VL-8B-Thinking.

Method	Textual Events						Visual Events						Multimedia Events
	Event Mention			Argument Role			Event Mention			Argument Role			Event Mention			Argument Role
	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
X-MTL	49.7	65.7	56.6	34.6	37.6	36.0	73.1	70.3	71.7	33.2	31.3	32.2	78.3	57.3	66.2	40.3	42.6	41.4
Qwen3-8B	50.3	65.6	56.9	20.5	41.2	27.4	65.0	84.5	73.5	28.2	57.6	37.9	44.7	71.4	55.0	26.3	61.0	36.8
Qwen3-32B	51.3	67.7	58.3	21.2	43.5	28.6	67.1	83.3	74.3	46.7	52.9	49.6	48.4	77.6	59.6	43.4	42.7	43.0
DS-V3.2	59.0	69.0	63.6	21.7	39.4	28.0	74.0	84.6	78.9	52.6	50.9	51.7	60.0	83.8	69.9	30.0	55.2	38.8

Appendix

Table 6. Detailed evaluation results of the dialogue-mediated baseline. DS denotes DeepSeek models. The table reports Precision (P), Recall (R), and F1 scores across textual, visual, and multimedia settings. Bold and underlined values denote the best and second-best F1 scores in each column, respectively. This baseline uses the same default vision tool outputs generated by Qwen3-VL-8B-Thinking.

Method	Qwen3-8B						Qwen3-32B						DeepSeek-v3.2
Method	Event Mention			Argument Role			Event Mention			Argument Role			Event Mention			Argument Role
	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Full Model	$70.6$	$69.9$	$70.2$	$47.0$	$58.9$	$52.3$	$72.5$	$72.5$	$72.5$	$49.8$	$61.2$	$54.9$	$75.1$	$75.1$	$75.1$	$47.2$	$65.8$	$55.0$
- w/o Linker	$69.9$	$69.9$	$69.9$	$49.7$	$45.8$	$47.7$	$70.6$	$70.6$	$70.6$	$42.1$	$37.3$	$39.6$	$74.8$	$74.8$	$74.8$	$45.2$	$40.3$	$42.6$
- w/o Verifier	$68.9$	$68.9$	$68.9$	$46.0$	$56.8$	$50.8$	$71.8$	$71.8$	$71.8$	$44.8$	$50.8$	$47.6$	$74.8$	$74.8$	$74.8$	$46.6$	$56.4$	$51.0$
- w/o SpanAlign	$64.7$	$64.7$	$64.7$	$44.5$	$54.9$	$49.1$	$67.3$	$67.3$	$67.3$	$38.5$	$48.5$	$42.9$	$74.8$	$74.8$	$74.8$	$38.6$	$47.7$	$42.6$

Appendix

Table 7. Full ablation results in the multimedia setting (P/R/F1). We evaluate ECHO across different backbones: Qwen3 (8B, 32B) and DeepSeek-V3.2. The ablations test the impact of removing the Linker (relevance linking), Verifier, SpanAlign (Stage III text span alignment), and Link-then-Bind. All variants use the same default vision tool outputs.

Step	Ag.	Op.	Target / Outcome
1	P	propose	$\epsilon_{\textsc{atk}}$ : Conflict:Attack
2	P	propose	$\epsilon_{\textsc{trans}}$ : Movement:Transport
3	L	link	$\epsilon_{\textsc{trans}}\leftarrow$
			{ militants , vehicles , Iraq , vehicle region ( img ) }
4	V	adj. conf.	$\epsilon_{\textsc{atk}}:\varnothing\rightarrow 0.70\rightarrow 0.60$ (insufficient multimodal support)
5	V	adj. conf.	$\epsilon_{\textsc{trans}}:\varnothing\rightarrow 0.90\rightarrow 0.95$ (strong cross-modal support)

Appendix

Table 8. Audit-trail excerpt for the representative transport example discussed in this subsection. Representative committed updates are shown; P / L / V / B denote Proposer, Linker, Verifier, and Role Binder, respectively, and adj. conf. denotes adjust_confidence . Additional linking steps are omitted for brevity.

AR error type	Textual	Visual	Multimedia
Spurious	540	0	180
Span mismatch	552	0	181
Role misassignment	60	12	54
No-gold event type	768	111	208
Localization error	0	173	400
Total	1920	296	1023

Appendix

Table 9. AR error counts for Qwen3-32B under ECHO on M2E2.

String relation	Textual	Multimedia
No-gold	1536	184
Exact	987	282
None	994	329
Contained-by	65	20
Contains	36	14
Overlap	3	0

Appendix

Table 10. Normalized string relations between predicted and gold text arguments for Qwen3-32B under ECHO.

Setting	Source	Most frequent misassigned roles
Textual	Text	Agent (18), Target (10), Attacker (9), Artifact (7), Place (6)
Visual	Box	Entity (5), Agent (5), Recipient(1), Artifact (1)
Multimedia	Text	Target (7), Place (6), Attacker(3), Agent (2), Entity (1)
Multimedia	Box	Instrument (6), Artifact (5), Target (5), Agent (4), Entity (4)

Appendix

Table 11. Most frequent roles involved in Argument Role misassignments for Qwen3-32B under ECHO.

Setting	Spurious	Missing	Trigger mismatch
Textual	469	371	129
Visual	37	37	0
Multimedia	85	85	52

Appendix

Table 12. EM error counts for Qwen3-32B under ECHO.

Textual (Gold = 1659)
Model	Pred	Matched	FP	P(%)	R(%)	OverGen
Qwen3-8B	2771	893	1878	32.2	53.8	1.67
Qwen3-14B	2620	881	1739	33.6	53.1	1.58
Qwen3-32B	2853	933	1920	32.7	56.2	1.72
DS-V3.2	2936	949	1987	32.3	57.2	1.77
GPT-5	3796	1033	2763	27.2	62.3	2.29

Appendix

Table 13. Argument Role (AR) output statistics under ECHO on M2E2. DS denotes DeepSeek models. Matched counts are computed using the same official AR matching protocol as in the main evaluation. OverGen denotes the over-generation ratio ( Pred / Gold ).

Setting	Interaction Cost (Number of Calls)						Computation Cost (Token Usage)
	Dialogue-mediated			ECHO (Ours)			Dialogue-mediated			ECHO (Ours)
	Main	Vis	Total	Main	Vis	Total	Input	Output	Total	Input	Output	Total
Textual	8	0	8	9	0	9	29,110	6,065	35,175	14,416	13,141	27,557
Visual	7	2	9	8	2	10	42,384	7,996	50,380	28,191	16,812	45,003
Multimedia	8	2	10	9	2	11	54,460	13,123	67,582	34,695	22,210	56,905

Appendix

Table 14. Interaction and computation cost under an idealized schedule. We assume

T_{\max}{=}2

rounds with no early stopping and a single extracted event, and report (left) the number of main-model and vision-tool calls and (right) token usage (input/output/total) for ECHO and the dialogue-mediated baseline across textual, visual, and multimedia settings. This table is intended as an illustrative cost accounting under the controlled protocol rather than an average empirical runtime summary. In practice, call counts can be lower due to early stopping (

T_{\mathrm{used}}<T_{\max}

) and vary with the number of extracted events.

Event Extraction

See all 15 papers

Enhancing Event Candidate Acquisition for Event Linking
Sep 12, 2026Ziyang Zhang, Yinan Liu, Boyi Xue +3
School of Computer Science and Engineering, Northeastern University Shenyang 110819, China
Schema-Constrained Document-Level Event Argument Extraction with Lightweight LLM Fine-Tuning
Jul 18, 2026Roberto Pietrantuono, Antonio Guerriero, Pouya SattariDocument Information Extraction LLM Information Extraction
University of Naples Federico II, Naples, Italy · University of Salerno, Fisciano, Italy
EAGER: Enhancing Generative Event Extraction via Reinforcement Learning with Verifiable Rewards
Sep 24, 2026Omar Adjali, Siting Liang, Omair Shahzad Bhatti +1RL for Language Models Document Information Extraction
German Research Center for Artificial Intelligence (DFKI), Germany · Carl von Ossietzky Universität Oldenburg, Germany

Sep 12, 2026J/K move · Enter open · S save

Enhancing Event Candidate Acquisition for Event Linking

Ziyang Zhang, Yinan Liu, Boyi Xue, Yingxuan Huang+2

School of Computer Science and Engineering, Northeastern University Shenyang 110819, China

Open paper AskPDF arXiv

Event linking associates event mentions in text with entries in a knowledge base (KB), or identifies them as out-of-KB events. Although existing methods use different architectures, candidate event acquisition can still be weakened by short ambiguous mentions, noisy arguments, and evidence that is unevenly useful for retrieval. We present MACE, a Multi-Agent Candidate Event acquisition method that refines event structure before linking. MACE uses evidence-specialized LLM agents to acquire time, location, participant, and event-type evidence, exposes intermediate queries to candidate-event lookup tools, and lets a coordinator revise the evidence set before final candidate construction. Experiments on two event linking benchmarks show that adding MACE to different event linking models consistently improves accuracy. These results show that MACE improves event linking through better candidate event acquisition without modifying the event linking model.

Abstract

Event Extraction

Explore similar work

Enhancing Event Candidate Acquisition for Event Linking

Schema-Constrained Document-Level Event Argument Extraction with Lightweight LLM Fine-Tuning

EAGER: Enhancing Generative Event Extraction via Reinforcement Learning with Verifiable Rewards