ABAI at COLIEE 2026 Task 1: Multi-Stage Retrieval with GraphRAG-Enhanced Meta-Learning, and a Post-Hoc Study of the Cross-Validation-to-Test Gap
Authors: Minhan Cho, Soyoung Park, Daejin Choi, Jinyoung Han
Organizations: AlphaBridge Seongnam-si, Gyeonggi-do, Republic of Korea · Sungkyunkwan University Seoul, Republic of Korea · National Assembly Research Service Seoul, Republic of Korea · Ewha Womans University Seoul, Republic of Korea
We present the ABAI submission to COLIEE 2026 Task 1, case law retrieval, together with a controlled study of why it underperformed. The task suppresses the cited passages themselves, which removes much of the lexical overlap a retriever would rely on. Our pipeline answers this with four independently trained stages: multi-view BM25 over citation-context windows with reciprocal rank fusion, neural reranking, graph-based features from entity communities and a graph attention network, and a LightGBM meta-learner over 34 features. Our best run reached F1=0.177 on the official test set, against a cross-validated 0.311, and we attributed that gap to a recall ceiling, temporal distribution shift, and threshold miscalibration. We then tested all three. Under leakage-free protocols threshold transfer costs 0.007 F1, decision quality is flat across chronological quartiles, and the official test queries are not measurably farther from the training manifold than training queries are from each other, in two independent embedding spaces. Decomposing the misses instead splits them exactly evenly between candidates never retrieved and candidates retrieved but ranked below the cut. Measuring the remedies for each half, BM25 length-normalisation tuning, an event-triple view, and full-content dense fusion lift top-200 recall by three to seven points, and citation-graph features add 0.014 F1 over eight seeds once own-citation leakage is removed, while per-query cutoff rules, a zero-shot reranker swap, and a date filter do not help. We also document four evaluation artifacts, each of which reversed a result once the protocol was corrected.
Figures & tables
Figure 1. Architecture of the proposed multi-stage pipeline.
Stage
#
Feature
Type
Description
Stage 1
1
bm25_score
cont.
Raw BM25 score (full-document query)
2
bm25_rrf_score
cont.
RRF-fused score from multi-view BM25
3
n_context_matches
int
Number of citation-context windows matching c
4
max_context_bm25
cont.
Highest BM25 score among context windows for c
5
tfidf_cosine
cont.
Cosine similarity of TF-IDF vectors (50K vocab)
6
jaccard
cont.
Word-level Jaccard: ∣Tq∩Tc∣/∣Tq∪Tc∣
Table 1. Feature inventory of the LightGBM meta-learner (34 features per query–candidate pair). Type: cont. = continuous, int = integer count or rank, bin. = binary.
System
F1
Prec.
Recall
BM25 (vanilla)
0.033
0.018
0.217
TQM 2024 (LTR)
0.119
0.145
0.101
JNLP 2025 (BM25+SAILER)
0.138
0.123
0.157
Ours ( t =0.547, balanced)
0.311
0.368
0.270
Ours ( t =0.400, recall)
0.285
0.264
0.310
Ours ( t =0.700, precision)
0.273
0.472
0.192
Table 2. Internal validation results against simplified prior winners.
Rank
Team / Run
F1
Prec.
Recall
1
NOWJ / sub_2
0.422
0.424
0.421
3
JNLP / random_forest
0.413
0.434
0.393
7
SIL / sil2
0.387
0.419
0.360
35
ABAI / run2 (recall)
0.177
0.160
0.199
36
ABAI / run1 (balanced)
0.167
0.195
0.146
41
ABAI / run3 (precision)
0.131
0.220
0.093
Table 3. Official COLIEE 2026 Task 1 results (selected runs).
Table 5. GraphRAG feature distribution in gold vs. non-gold pairs (BM25 top-200, training data). Feature # refers to Table 1 .
Proposed cause
Internal proxy
Effect
Temporal shift
leave-one-quartile-out F1
0.022 spread
Threshold transfer
calibration vs oracle t
0.007 F1
Query-text shift
5-NN cosine to train
d=−0.16 / −0.00
Table 6. The three causes proposed in Section 4.5 , each re-tested under a leakage-free protocol. None reproduces on an internal proxy. Effects are in the units of their own test; the query-text row gives both embedding spaces.
Figure 3. Left: of the 1,643 gold citations in the test slice the pipeline recovers 369, and the 1,274 it misses divide into 637 that never entered the candidate pool and 637 that entered it and were ranked below the decision cut. The two failure modes are the same size; the panel reports how large each half is, not what causes it. Right: each first-stage recall lever plotted against the baseline observed in the same run (grey tick) rather than a shared baseline, because the BM25 sweep reproduced the baseline at 0.570 while the other runs measured 0.578. Full-content dense fusion gives the largest single lift and the parameter sweep the cheapest. Generated by scripts/experiments/make_fig_misses.py from the same JSON as Table 7 .
Lever
Baseline
R@200
Δ
BM25 sweep ( k1=4.5 , b=1.0 )
0.570
0.630
+0.060
Event-triple view, fused
0.578
0.608
+0.030
Corpus-graph backfill ( B=500 )
0.578
0.633
+0.056
Dense @512, fused
0.578
0.589
+0.011
Dense @4096, fused
0.578
0.649
+0.071
Table 7. First-stage recall levers, each measured against the baseline observed in the same run. Lexical tuning is the cheapest gain and full-content dense fusion the largest. We do not report a combined figure because we did not measure the combination.
Configuration
F1
Δ F1
Feature set (global threshold, 8 seeds)
Base, 34 features
0.3456
–
+ graph features, leave-one-out
0.3596
+0.0141
+ graph + lexical/semantic gap
0.3619
+0.0164
Decision rule (base features)
Global threshold
0.3453
–
Table 8. Reranking and decision levers on the three-way split, wins and failures together. Only leakage-corrected graph features survive a seed-paired test. Oracle rows peek at test labels and bound the available headroom. Feature-set rows are means over eight seeds.
Artifact
Protocol fault
Correction
Metric
Naive → clean
Score-scale mismatch
Threshold fit on out-of-fold scores from early-stopped models, then applied to a model trained on the whole slice
Fit and apply with one model
F1 penalty
0.188→0.007
Truncated document input
No end-of-sequence token for a last-token pooler, then a 512-token window on 2–5k-word judgments
Append EOS, embed the full document
recall@200
0.435→0.611
Own-citation edge
A training query’s own citation inflates the indegree of its gold candidates
Leave-one-query-out indegree
Δ F1
−0.180→+0.014
Assumed test shift
Claimed without running an experiment that could have disconfirmed it
Label-free embedding probe
Cohen’s d
asserted →−0.16
Table 9. The four evaluation artifacts, the protocol fault behind each, the correction that exposed it, and the result before and after. Each row reports a different metric and the desirable direction differs , so the last column is not comparable down the table: the calibration penalty should be near zero, recall and Δ F1 should be large and positive, and a Cohen’s d near zero means no distribution shift. The value after the arrow is the one reported elsewhere in this paper. Three of the four inflated a conclusion we already expected; the graph-feature artifact is the exception, and because it pointed against expectation it took a distribution check rather than a sanity check to find.
Retrieval-augmented generation systems for legal question answering typically retrieve passages based on semantic similarity and provide them to a language model, which then generates cited answers. Prior work assumes that highly ranked passages are most likely to be usefully cited by the model. Perturbation-based attribution methods, such as C-LIME, have been used exclusively for post-hoc explanation. However, on the AQuAECHR benchmark, semantic similarity does not correlate with passage attribution. Within a retriever's candidate pool, similarity-based ranking performs worse than random selection at surfacing gold citation paragraphs. To address this limitation, a lightweight cross-encoder is trained on continuous perturbation-based attribution scores to re-rank passages prior to generation. This approach is evaluated on the AQuAECHR benchmark, using two language models and five-fold cross-validation. The re-ranker substantially improves citation faithfulness and alignment with gold expert answers. Notably, two re-rankers trained independently on different models converge beyond their raw attribution agreement. This finding indicates that the cross-encoder reduces model-specific noise and produces a shared relevance signal that partially transfers across models, although same-model re-ranking remains more effective. These results demonstrate that perturbation-based attribution provides a practical, model-agnostic training signal for citation-aware retrieval.
This paper presents the methodologies and results of the NOWJ team's participation across all five tasks of the COLIEE 2026 competition. For Task 1 (Legal Case Retrieval), we propose a four-stage pipeline comprising candidate filtering, dense retrieval with complementary embedding models, cross-encoder reranking via fine-tuned generative rerankers and MLP-based pairwise classification, and adaptive per-query cutoff prediction. For Task 2 (Legal Case Entailment), we combine BM25 filtering, T5-based reranking, and LLM-based entailment verification with consensus ensemble. For Task 3 (Statute Law Retrieval and Entailment), we adopt a retrieval-augmented generation framework with dense retrieval, attention-based reranking, and few-shot-prompted LLM reasoning. For Task 4 (Legal Textual Entailment), we introduce a dynamic routing pipeline that classifies query difficulty and dispatches cases to either a balanced few-shot solver or a structured zero-shot chain-of-thought solver. For the Pilot Task (Legal Judgment Prediction), we combine hierarchical transformers with CRF layers, argument relation mining, and probabilistic argumentation graph reasoning.
VNU University of Engineering and Technology Hanoi, Vietnam · Center for Juris-Informatics, ROIS-DS College of Engineering & Computer Science, VinUniversity Hanoi, Vietnam
Retrieval-augmented QA can fail even when the correct answer is in the retrieved evidence. This can happen when a competing answer satisfies some, but not all, of the question's requirements and is mentioned frequently in the retrieved documents, which can make it appear to be the best-supported answer. To test whether this retrieval skew actually affects the decision, we manipulate it directly: increasing the prevalence of passages that favor the competing answer makes the model more likely to select it. This points to a decision problem that candidate-level support alone cannot resolve. We therefore introduce Joint Constraint Aggregation (JCA), which compares candidates against the same question-derived requirements using the retrieved evidence. Across three evaluation settings and three instruction-tuned backbones, JCA improves overall accuracy, and its advantage grows when retrieval favors a competing answer. Together, these results suggest that when partial matches and retrieval prominence pull models toward a competing answer, comparing candidates against the question's full requirements can make selection more reliable.
Jennifer Hsia, Zachary C. Lipton, Aarti Singh
Machine Learning Department Carnegie Mellon University Pittsburgh, PA 15213, USA