AILS-NTUA at SemEval-2026 Task 12: Graph-Based Retrieval and Reflective Prompting for Abductive Event Reasoning

Nikolas Karafyllis; Maria Lymperaiou; Giorgos Filandrianos; Athanasios Voulodimos; Giorgos Stamou

cs.CLMar 4, 2026

AILS-NTUA at SemEval-2026 Task 12: Graph-Based Retrieval and Reflective Prompting for Abductive Event Reasoning

Authors: Nikolas Karafyllis, Maria Lymperaiou, Giorgos Filandrianos, Athanasios Voulodimos, Giorgos Stamou

Organizations: School of Electrical and Computer Engineering, AILS Laboratory, National Technical University of Athens

Abstract

We present a winning three-stage system for SemEval 2026 Task12: Abductive Event Reasoning that combines graph-based retrieval, LLM-driven abductive reasoning with prompt design optimized through reflective prompt evolution, and post-hoc consistency enforcement; our system ranks first on the evaluation-phase leaderboard with an accuracy score of 0.95. Cross-model error analysis across 14 models (7families) reveals three shared inductive biases: causal chain incompleteness, proximate cause preference, and salience bias, whose cross-family convergence (51% cause-count reduction) indicates systematic rather than model-specific failure modes in multi-label causal reasoning.

Figures & tables

Panel of Figure 1 — Figure 1: System pipeline. Stage 1 constructs a hybrid document graph (Figure 2 ), selects dense/sparse entry points, retrieves the connected component, and filters disconnected distractors. Stage 2 performs structured analysis-before-answer prompting with self-consistency. Stage 3 applies eight post-hoc consistency heuristics.

Panel of Figure 2 — Figure 2: Hybrid document-graph retrieval in three steps. Step 1: Build a hybrid similarity graph ( $\alpha{=}0.7$ dense $+$ $0.3$ sparse); disconnected documents ( $d_{9}$ – $d_{12}$ ) are potential distractors. Step 2: At query time, pick entry points from dense and sparse signals (3 $+$ 2, deduplicated). Step 3: Retrieve the full connected component from the seeds, filter disconnected documents, and pass the selected topic context to the LLM reasoner.

Component	Description
Dense embeddings	Cohere Embed v4 (1024-dim)
Sparse retrieval	BM25+ Robertson and Zaragoza (2009) with 3 $\times$ entity boosting
Hybrid weighting	$\alpha{=}0.7$ dense $+$ $0.3$ sparse
Entry points	3 dense $+$ 2 sparse (deduplicated)
Traversal	Full connected component

Table 1: Hybrid document graph retrieval configuration.

Family	Standard	Reasoning
Claude	Haiku 4.5, Sonnet 4,	Sonnet 4 Thinking,
	Sonnet 4.5, Opus 4.0,	Sonnet 4.5 Thinking
	Opus 4.5
Gemini	—	Flash 3 Preview
OpenAI	—	GPT-5.2 (xhigh)
DeepSeek	V3.1	V3.1 Think, R1

Table 2: Model configurations evaluated. Reasoning models use extended thinking or chain-of-thought.

	Model	Score
Standard Models	Claude Haiku 4.5	0.788
	Claude Sonnet 4	0.781
	Claude Sonnet 4.5	0.790
	Claude Opus 4.0	0.770
	Claude Opus 4.5	0.801
	DeepSeek V3.1	0.709

Table 3: Zero-shot baseline results (dev set). Bold marks the best score within each group under the partial-credit metric (Section 4 ).

	Configuration	Base	+ Post-hoc
Individual Models	Claude Son. 4.5 Thinking	0.904	0.952
	GPT-5.2	0.912	0.949
	Gemini 3 Flash	0.907	0.943
Self-Consistency	SC: Sonnet 3 $\times$ ( $\theta{=}0.50$ )	0.902	0.948
	SC: Gemini 5 $\times$ ( $\theta{=}0.50$ )	0.902	0.943
Ensemble	Sonnet + GPT + Gemini	0.926	0.952

Table 4: Test set results. Post-hoc = deterministic consistency enforcement via heuristics. SC = self-consistency with majority voting;

\theta

= inclusion threshold.

System	Single	Multi	$\boldsymbol{\Delta}$
GPT-5.2	91.0	52.6	$-$ 38.4
Claude Son. 4.5 Thinking	88.6	60.5	$-$ 28.1
Gemini 3 Flash	85.2	63.7	$-$ 21.5

Table 5: Exact-match accuracy (%) by answer cardinality across three frontier systems. Multi-answer questions (190/400, 47.5%) are inherently harder due to the combinatorial requirement.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Statistic	Train	Dev	Test
Questions	1,819	400	612
Topics	36	36	24
Documents	775	775	405
Multi-answer (%)	42.7	47.5	–
Avg. Q/Topic	50.5	11.1	25.5
Avg. Doc Length	1188	1188	899

Appendix

Table 6: Dataset statistics across splits. Multi-answer percentage indicates questions with more than one correct answer.

Target Event	Candidate Options	Gold
Ontario placed a 25% tax on electricity exports to the U.S.	(A) Trump placed 25% tariffs on imports from Canada. (B) McKinley was shot by anarchist Czolgosz in 1901. (C) The 16th Amendment was ratified in 1913. (D) McKinley died on Sept. 14, 1901.	A
The Chinese government criticized Japan’s plan to release Fukushima wastewater.	(A) IAEA confirmed the release would have negligible radiological effect. (B) Japan’s cabinet approved a treated-water plan in 2021. (C) Japan announced its plan to release treated wastewater into the sea. (D) TEPCO has used water to cool nuclear fuel rods since 2011.	B,C
Facebook changed its corporate name to Meta at the annual Connect event.	(A) Zuckerberg introduced Meta at the Connect 2021 event. (B) Facebook announced Bosworth’s promotion to CTO. (C) Facebook plans to change its name to reflect metaverse focus. (D) Zuckerberg announced Facebook was betting on the metaverse.	A,C,D
Goldman Sachs estimated AI could automate 300 million jobs worldwide.	(A) OpenAI released ChatGPT Plus and later introduced GPT-4 Turbo. (B) Stack Overflow banned ChatGPT-generated answers. (C) None of the others are correct causes. (D) OpenAI launched DALL-E 2 beta in July 2022.	C

Appendix

Table 7: Representative examples illustrating each answer cardinality. Correct options are shown in bold . Option texts are abbreviated for space.

Figure 3: Dataset composition across splits.

Category	Count	Percentage
Option A selected	909	41.0%
Option B selected	857	38.6%
Option C selected	891	40.2%
Option D selected	836	37.7%
Single answer	928	41.8%
Two answers	660	29.7%

Appendix

Table 8: Answer distribution analysis on the combined train and dev sets.

Figure 4: Answer frequency by option.

Figure 5: Distribution of answer cardinality.

Figure 6: Most common multi-answer combinations.

Metric	Value
Label cardinality	1.574
Label density	0.394
Multi-label rate	43.6%

Appendix

Table 9: Multi-label characteristics of the dataset.

ID	Topic Description	Questions	Docs	Multi%	Context
19	Historic winter storm paralyzes Texas, causin…	132	24	53.0	18,289
30	Killing of George Floyd by police sparks Blac…	113	25	57.5	70,271
23	Japan begins discharging treated Fukushima nu…	111	24	45.0	38,702
24	Halloween crowd crush in Seoul’s Itaewon kill…	111	28	45.0	14,375
22	Iranian General Qasem Soleimani killed by US …	110	20	36.4	12,160
32	Taliban regains control of Afghanistan after …	109	27	55.0	24,098

Appendix

Table 10: Topic-level statistics (top 10 topics by question count). Full table in supplementary materials.

Figure 8: Topic characteristics: questions vs complexity (bubble size = document count, color = context length).

Figure 9: Distribution of document lengths (99th percentile shown).

Figure 10: Target event length distribution.

Figure 11: Frequency of structural patterns addressed by post-hoc consistency via heuristics.

Figure 12: Sibling group size distribution. Groups of 2+ questions enable cross-question consistency enforcement.

Heuristic	Name	Scope	Mechanism	Changes
R1	None Exclusivity	Local	Enforce mutual exclusivity of “None” option	6
R2	Duplicate Consistency	Local	Ensure identical options receive same prediction	13
R3	Over-selection Guard	Local	Resolve over-selection when all four options predicted	0
R4	Cross-Q Dup. Prop.	Cross-Q	Propagate duplicate consistency to sibling questions	20
R5	Triple Exclusion	Cross-Q	Exclude unique option from triple-duplicate siblings	1
R6	None Cross-Exclusion	Cross-Q	Extend None exclusivity to sibling questions	4

Appendix

Table 11: Post-hoc consistency heuristics applied iteratively until convergence (fixed-point). Scope indicates whether the heuristic operates within a single question (Local) or across sibling questions sharing the same target event (Cross-Q). Changes reported on Claude Sonnet 4.5 (thinking), dev set.

Figure 13: Cumulative accuracy improvement from post-hoc consistency heuristics on Claude Sonnet 4.5 (thinking) predictions. Background colors indicate heuristic scope: local (blue), cross-question (red), and closure (green). Black line shows per-rule improvement.

Figure 14: Document graph and query-time selection for Topic 7 (South Korean impeachment crisis). Node positions are identical across panels. (a) Document context graph: 14 documents connected by hybrid similarity edges (semantic-dominant, lexical-dominant, balanced). The densely connected component captures the core impeachment narrative; peripheral nodes represent tangentially related events. (b) Query-time selection for q-2043 (target: “Yoon’s aides offered to resign”): 6/14 documents (43%) selected via 3 dense entries, 2 sparse entries, and 1 traversal-discovered node. Grey nodes are filtered distractors.

Metric	Value
Avg. documents per topic	21.5
Avg. retrieved per question	16.6 (73%)
Avg. in topic-wide context	15.8 (73%)
Likely distractors excluded	27%
Prompt cache hit rate	91%
Overall inference cost reduction	87%

Appendix

Table 12: Retrieval statistics on the dev set (400 questions, 36 topics, 775 documents).

Model	Base	+Retr.	$\boldsymbol{\Delta}$
Haiku 3.5 †	0.440	0.530	$+$ 0.090
Sonnet 4	0.781	0.785	$+$ 0.004
Gemini 3 Flash	0.804	0.806	$+$ 0.003
GPT-5.2	0.810	0.818	$+$ 0.008
† 100 questions; others on full dev set (400).

Appendix

Table 13: Impact of distractor filtering on accuracy. Smaller models benefit substantially from context reduction Xu et al. (2024) .

Panel of Figure 15 — Figure 15: Structured zero-shot prompt for direct inference (Section 3.2 ).

QID	Topic	Gold	Consensus	Fail	Multi?	Pattern
Truly unbreakable: all 14 models score zero (all 7 families fail)
q-2261	Afghanistan	A,C	D	14/14	Yes	Under-sel. + proximal
q-2356	DeepSeek	D	C	14/14	No	Consensus wrong (7/14 C)
q-2411	DOGE/Musk	A	D	14/14	No	Consensus wrong (13/14 D)
Near-unbreakable: $\geq$ 12 models score zero
q-2341	Afghanistan	B,C,D	A	13/14	Yes	Under-sel. + wrong opt.

Appendix

Table 14: Hardest questions across 14 models (7 families). Top: 3 questions where all models score zero. Bottom: 6 near-unbreakable questions where

\geq

12 models fail. Fail = models scoring zero. Under-sel. = multi-answer under-selection; Consensus wrong = majority agrees on incorrect option; Proximal-cause bias = models prefer temporally nearest cause Joshi et al. (2024) .

ID	Domain	N	Multi%	$\boldsymbol{\kappa_{14}}$
26	California Fire	3	—	0.456
33	Amazon fires	14	43	0.457
30	George Floyd	18	72	0.458
18	DeepSeek	14	—	0.492
32	Afghanistan	23	57	0.545
12	Wagner	11	—	0.547

Appendix

Table 15: Hardest topics by Fleiss’

\kappa

across 14 models. Topics with complex causal structures and high multi-answer prevalence show the lowest agreement.

Figure 16: Per-topic Fleiss’

\kappa

across 14 models (bar labels = question counts per topic); dashed line marks the per-topic mean (

\kappa=0.697

). Topics are sorted by agreement, coloured by Landis–Koch interpretation: green = almost perfect (

\kappa\geq 0.81

), yellow = substantial (

0.61

–

0.80

), orange = moderate (

0.41

–

0.60

Metric	Value
Fleiss’ $\kappa$ Fleiss (1971)	0.690
Kripp. $\alpha$ (Jaccard) Krippendorff (2011)	0.645
Kripp. $\alpha$ (nominal)	0.613
Unanimous agreement	10.0%
Majority agreement	100%
Within-family $\bar{\kappa}$	0.794

Appendix

Table 16: Inter-annotator agreement metrics across 14 models (7 families: Claude, DeepSeek, Gemini, GPT, Kimi, Llama, Qwen) on 400 dev questions.

Figure 17: Family-level mean pairwise Cohen’s $\kappa$ across 14 models (7 families). Frontier families (Claude, Gemini, GPT, Kimi) show high mutual agreement ( $\kappa>0.77$ ); DeepSeek is dragged down by R1’s low accuracy. Numbers in parentheses = model count per family. Diagonal masked (within-family comparison undefined for single-model families).

Figure 18: Dev-set difficulty distribution across 14 models: Easy 38, Medium 225, Hard 95, Very Hard 42 (based on how many models score zero). The majority of questions are medium-difficulty, with only 42 very hard questions where

\geq

10 models fail.

Same-Model SC ( $\tau{=}1.0$ )
Configuration	$\boldsymbol{k}$	Base	+SC	$\boldsymbol{\Delta}$
Claude Sonnet 4.5	3	0.790	0.806	$+$ 1.6
Gemini 3 Flash	3	0.804	0.810	$+$ 0.6
Multi-Model Ensemble
Son. Think + GPT + Gemini	3	—	0.811	—

Appendix

Table 17: Self-consistency and multi-model ensemble on the dev set (400 questions).

k

= number of predictions aggregated via majority voting (

\theta{=}0.5

). Same-model SC provides negligible gains; multi-model combination provides modest gains through model complementarity.

Sonnet 4.5 ( $k{=}3$ , $\tau{=}1.0$ )
Strategy	$\boldsymbol{\theta}$	Dev	Test
Baseline ( $k{=}1$ )	—	0.790	0.904 †
Majority	0.50	0.806	0.902
Union	0.33	0.788	0.869
Strict Majority	0.67	0.784	0.897
Intersection	1.00	0.784	0.897

Appendix

Table 18: Self-consistency aggregation strategies on dev (400 questions) and test (612 questions).

\theta

= per-option inclusion threshold. Bold = best SC strategy per section.

Figure 19: Failure mechanism breakdown across 42 no-exact-match questions (14 models). Causal chain incompleteness, proximate cause preference, and salience bias together account for 38/42 failures, all manifestations of the single-cause default bias.

Figure 20: Failure gradient across 14 models: distribution of how many models fail each question. 88 questions (22%) are solved by all 14 models; only 3 questions (0.75%) defeat all 14. The long right tail (10-14 failures) contains 22 questions requiring systematic improvements.

Figure 21: Per-model exact-match accuracy by answer cardinality across 14 models. Every model shows a single-to-multi gap (25-56 pp); under-selections (1,389 total) vastly exceed over-selections (52), confirming conservative cause selection as the dominant shared bias.

Explore similar work

SemEval-2026 Task 12: Abductive Event Reasoning: Towards Real-World Event Causal Inference for Large Language Models
Mar 23, 2026Pengfei Cao, Mingxuan Yang, Yubo Chen +4Abductive Reasoning Causal Reasoning in Language Models
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Correct Prediction, Wrong Steps? Consensus Reasoning Knowledge Graph for Robust Chain-of-Thought Synthesis
Apr 15, 2026Zipeng Ling, Shuliang Liu, Seonil Son +4Logical Reasoning Self-Consistency Decoding
The Hong Kong University of Science and Technology (Guangzhou) · University of Alberta · Alberta Machine Intelligence Institute +3
Causal Concept Graphs in LLM Latent Space for Stepwise Reasoning
Mar 11, 2026Md Muntaqim Meherab, Noor Islam S. Mohammad, Faiza FerozLLM Reasoning with Graphs Causal Interpretability
Daffodil International University, Dhaka, Bangladesh · New York University, Brooklyn, NY, USA

Mar 23, 2026J/K move · Enter open · S save

SemEval-2026 Task 12: Abductive Event Reasoning: Towards Real-World Event Causal Inference for Large Language Models

Pengfei Cao, Mingxuan Yang, Yubo Chen, Chenlong Zhang+3

The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China

Abductive Reasoning Causal Reasoning in Language Models Causal Inference Evidence-Grounded Reasoning

Open paper AskPDF arXiv

Understanding why real-world events occur is important for both natural language processing and practical decision-making, yet direct-cause inference remains underexplored in evidence-rich settings. To address this gap, we organized SemEval-2026 Task 12: Abductive Event Reasoning (AER).\footnote{The task data is available at https://github.com/sooo66/semeval2026-task12-dataset.git} The task asks systems to identify the most plausible direct cause of a target event from supporting evidence. We formulate AER as an evidence-grounded multiple-choice benchmark that captures key challenges of real-world causal reasoning, including distributed evidence, indirect background factors, and semantically related but non-causal distractors. The shared task attracted 122 participants and received 518 submissions. This paper presents the task formulation, dataset construction pipeline, evaluation setup, and system results. AER provides a focused benchmark for abductive reasoning over real-world events and highlights challenges for future work on causal reasoning and multi-document understanding.

Abstract

Explore similar work

SemEval-2026 Task 12: Abductive Event Reasoning: Towards Real-World Event Causal Inference for Large Language Models

Correct Prediction, Wrong Steps? Consensus Reasoning Knowledge Graph for Robust Chain-of-Thought Synthesis

Causal Concept Graphs in LLM Latent Space for Stepwise Reasoning