ABAI at COLIEE 2026 Task 1: Multi-Stage Retrieval with GraphRAG-Enhanced Meta-Learning, and a Post-Hoc Study of the Cross-Validation-to-Test Gap
Authors: Minhan Cho, Soyoung Park, Daejin Choi, Jinyoung Han
Organizations: AlphaBridge Seongnam-si, Gyeonggi-do, Republic of Korea · Sungkyunkwan University Seoul, Republic of Korea · National Assembly Research Service Seoul, Republic of Korea · Ewha Womans University Seoul, Republic of Korea
We present the ABAI submission to COLIEE 2026 Task 1, case law retrieval, together with a controlled study of why it underperformed. The task suppresses the cited passages themselves, which removes much of the lexical overlap a retriever would rely on. Our pipeline answers this with four independently trained stages: multi-view BM25 over citation-context windows with reciprocal rank fusion, neural reranking, graph-based features from entity communities and a graph attention network, and a LightGBM meta-learner over 34 features. Our best run reached F1=0.177 on the official test set, against a cross-validated 0.311, and we attributed that gap to a recall ceiling, temporal distribution shift, and threshold miscalibration. We then tested all three. Under leakage-free protocols threshold transfer costs 0.007 F1, decision quality is flat across chronological quartiles, and the official test queries are not measurably farther from the training manifold than training queries are from each other, in two independent embedding spaces. Decomposing the misses instead splits them exactly evenly between candidates never retrieved and candidates retrieved but ranked below the cut. Measuring the remedies for each half, BM25 length-normalisation tuning, an event-triple view, and full-content dense fusion lift top-200 recall by three to seven points, and citation-graph features add 0.014 F1 over eight seeds once own-citation leakage is removed, while per-query cutoff rules, a zero-shot reranker swap, and a date filter do not help. We also document four evaluation artifacts, each of which reversed a result once the protocol was corrected.
Figures & tables
Figure 1. Architecture of the proposed multi-stage pipeline.
Stage
#
Feature
Type
Description
Stage 1
1
bm25_score
cont.
Raw BM25 score (full-document query)
2
bm25_rrf_score
cont.
RRF-fused score from multi-view BM25
3
n_context_matches
int
Number of citation-context windows matching c
4
max_context_bm25
cont.
Highest BM25 score among context windows for c
5
tfidf_cosine
cont.
Cosine similarity of TF-IDF vectors (50K vocab)
6
jaccard
cont.
Word-level Jaccard: ∣Tq∩Tc∣/∣Tq∪Tc∣
Table 1. Feature inventory of the LightGBM meta-learner (34 features per query–candidate pair). Type: cont. = continuous, int = integer count or rank, bin. = binary.
System
F1
Prec.
Recall
BM25 (vanilla)
0.033
0.018
0.217
TQM 2024 (LTR)
0.119
0.145
0.101
JNLP 2025 (BM25+SAILER)
0.138
0.123
0.157
Ours ( t =0.547, balanced)
0.311
0.368
0.270
Ours ( t =0.400, recall)
0.285
0.264
0.310
Ours ( t =0.700, precision)
0.273
0.472
0.192
Table 2. Internal validation results against simplified prior winners.
Rank
Team / Run
F1
Prec.
Recall
1
NOWJ / sub_2
0.422
0.424
0.421
3
JNLP / random_forest
0.413
0.434
0.393
7
SIL / sil2
0.387
0.419
0.360
35
ABAI / run2 (recall)
0.177
0.160
0.199
36
ABAI / run1 (balanced)
0.167
0.195
0.146
41
ABAI / run3 (precision)
0.131
0.220
0.093
Table 3. Official COLIEE 2026 Task 1 results (selected runs).
Table 5. GraphRAG feature distribution in gold vs. non-gold pairs (BM25 top-200, training data). Feature # refers to Table 1 .
Proposed cause
Internal proxy
Effect
Temporal shift
leave-one-quartile-out F1
0.022 spread
Threshold transfer
calibration vs oracle t
0.007 F1
Query-text shift
5-NN cosine to train
d=−0.16 / −0.00
Table 6. The three causes proposed in Section 4.5 , each re-tested under a leakage-free protocol. None reproduces on an internal proxy. Effects are in the units of their own test; the query-text row gives both embedding spaces.
Figure 3. Left: of the 1,643 gold citations in the test slice the pipeline recovers 369, and the 1,274 it misses divide into 637 that never entered the candidate pool and 637 that entered it and were ranked below the decision cut. The two failure modes are the same size; the panel reports how large each half is, not what causes it. Right: each first-stage recall lever plotted against the baseline observed in the same run (grey tick) rather than a shared baseline, because the BM25 sweep reproduced the baseline at 0.570 while the other runs measured 0.578. Full-content dense fusion gives the largest single lift and the parameter sweep the cheapest. Generated by scripts/experiments/make_fig_misses.py from the same JSON as Table 7 .
Lever
Baseline
R@200
Δ
BM25 sweep ( k1=4.5 , b=1.0 )
0.570
0.630
+0.060
Event-triple view, fused
0.578
0.608
+0.030
Corpus-graph backfill ( B=500 )
0.578
0.633
+0.056
Dense @512, fused
0.578
0.589
+0.011
Dense @4096, fused
0.578
0.649
+0.071
Table 7. First-stage recall levers, each measured against the baseline observed in the same run. Lexical tuning is the cheapest gain and full-content dense fusion the largest. We do not report a combined figure because we did not measure the combination.
Configuration
F1
Δ F1
Feature set (global threshold, 8 seeds)
Base, 34 features
0.3456
–
+ graph features, leave-one-out
0.3596
+0.0141
+ graph + lexical/semantic gap
0.3619
+0.0164
Decision rule (base features)
Global threshold
0.3453
–
Table 8. Reranking and decision levers on the three-way split, wins and failures together. Only leakage-corrected graph features survive a seed-paired test. Oracle rows peek at test labels and bound the available headroom. Feature-set rows are means over eight seeds.
Artifact
Protocol fault
Correction
Metric
Naive → clean
Score-scale mismatch
Threshold fit on out-of-fold scores from early-stopped models, then applied to a model trained on the whole slice
Fit and apply with one model
F1 penalty
0.188→0.007
Truncated document input
No end-of-sequence token for a last-token pooler, then a 512-token window on 2–5k-word judgments
Append EOS, embed the full document
recall@200
0.435→0.611
Own-citation edge
A training query’s own citation inflates the indegree of its gold candidates
Leave-one-query-out indegree
Δ F1
−0.180→+0.014
Assumed test shift
Claimed without running an experiment that could have disconfirmed it
Label-free embedding probe
Cohen’s d
asserted →−0.16
Table 9. The four evaluation artifacts, the protocol fault behind each, the correction that exposed it, and the result before and after. Each row reports a different metric and the desirable direction differs , so the last column is not comparable down the table: the calibration penalty should be near zero, recall and Δ F1 should be large and positive, and a Cohen’s d near zero means no distribution shift. The value after the arrow is the one reported elsewhere in this paper. Three of the four inflated a conclusion we already expected; the graph-feature artifact is the exception, and because it pointed against expectation it took a distribution check rather than a sanity check to find.
VNU University of Engineering and Technology Hanoi, Vietnam · Center for Juris-Informatics, ROIS-DS College of Engineering & Computer Science, VinUniversity Hanoi, Vietnam