cs.IRApr 11, 2026
SaveReproduction Beyond Benchmarks: ConstBERT and ColBERT-v2 Across Backends and Query Distributions
Organizations: Missouri University of Science and Technology, Rolla, MO, USA
Abstract
Reproducibility must validate architectural robustness, not just numerical accuracy. We evaluate ColBERT-v2 and ConstBERT across five dimensions, finding that while ConstBERT reproduces within 0.05% MRR@10 on MS-MARCO, both models show a drop of 86-97% on long, narrative queries (TREC ToT 2025). Ablations prove this failure is architectural: performance plateaus at 20 words because the MaxSim operator's uniform token weighting cannot distinguish signal from filler noise. Furthermore, undocumented backend parameters create an 8-point gap due to ConstBERT's sparse centroid coverage, and fine-tuning with 3x more data actually degrades performance by up to 29%. We conclude that architectural constraints in multi-vector retrieval cannot be overcome by adaptation alone. Code: https://github.com/utshabkg/multi-vector-reproducibility.
Figures & tables
| Model | Metric | Paper | Ours | |
|---|---|---|---|---|
| ColBERT-v2 | MRR@10 | 39.70% | 39.15% | –0.55% |
| Recall@50 | — | 85.91% | — | |
| Recall@1000 | 98.40% | 90.30% | –8.10% | |
| ConstBERT | MRR@10 | 39.04% | 38.99% | –0.05% |
| Recall@50 | 85.86% | 85.35% | –0.51% | |
| Recall@1000 | 96.34% | 92.85% | –3.49% |
Table 1. Reproduction of MS-MARCO passage results. Both models successfully reproduce within acceptable variance.
| Dataset | Metric | Paper | FAISS-IVF | PLAID-4 | PLAID-16 |
|---|---|---|---|---|---|
| MS-MARCO | MRR@10 | 39.04% | 38.99% | 30.01% | 31.09% |
| R@1000 | 96.34% | 92.85% | 87.64% | 93.88% | |
| ToT 2025 | MRR@10 | — | 4.27% | — | 0.94% |
| R@1000 | — | 25.72% | — | 13.50% | |
| PLAID-4: documented defaults; PLAID-16: optimized (ncells=16, threshold=0.3) | |||||
Table 2. ConstBERT performance across retrieval backends. FAISS-IVF reproduces original effectiveness (38.99% vs. 39.04%), while PLAID exhibits systematic degradation on MS-MARCO that compounds under structural shift (ToT).
| Dataset | Paper (PLAID) | Ours (PLAID) | |
|---|---|---|---|
| ArguAna | 45.1% | 26.6% | –18.5% |
| Climate-FEVER | 14.2% | 11.0% | –3.2% |
| DBPedia-Entity | 41.8% | 30.9% | –10.9% |
| FEVER | 69.6% | 57.1% | –12.5% |
| FiQA-2018 | 31.2% | 24.7% | –6.5% |
| HotpotQA | 62.1% | 50.1% | –12.0% |
Table 3. ConstBERT on BEIR with PLAID (Reproduction Attempt): nDCG@10 showing consistent performance gaps across all 13 datasets.
| Dataset | Paper (PLAID) | Ours (FAISS-IVF) | |
|---|---|---|---|
| ArguAna | 45.1% | 35.4% | –9.7% |
| Climate-FEVER | 14.2% | 2.4% | –11.8% |
| DBPedia-Entity | 41.8% | 36.2% | –5.6% |
| FEVER | 69.6% | 11.8% | –57.8% |
| FiQA-2018 | 31.2% | 14.7% | –16.5% |
| HotpotQA | 62.1% | 29.7% | –32.4% |
Table 4. ConstBERT on BEIR with FAISS-IVF (Alternative Backend): nDCG@10 showing larger performance gaps than PLAID.
| Dataset | ColBERT-v2 (Paper) | Ours | |
|---|---|---|---|
| ArguAna | 46.3% | 33.5% | –12.8% |
| Climate-FEVER | 17.6% | 16.0% | –1.6% |
| DBPedia-Entity | 44.6% | 49.8% | +5.2% |
| FEVER | 78.5% | 74.7% | –3.8% |
| FiQA-2018 | 35.6% | 34.0% | –1.6% |
| HotpotQA | 66.7% | 66.9% | +0.2% |
Table 5. ColBERT-v2 on BEIR: Comparing our reproduction against values reported in the ColBERT-v2 paper ( Santhanam et al., 2022b ) . Close reproduction (mean –1.4% on all 13 datasets) versus ConstBERT’s –9.4%/–17.5% demonstrates reproduction challenges are partially ConstBERT-specific.
| Model | Backend | MRR@10 | R@1000 | MS-MARCO |
|---|---|---|---|---|
| BM25 | Lucene | — | 23.94% | (baseline) |
| ConstBERT | FAISS-IVF | 4.27% | 25.72% | –89% |
| ConstBERT | PLAID-16 | 0.94% | 13.50% | –97% |
| ColBERT-v2 | FAISS-IVF | 5.66% | 28.94% | –86% |
Table 6. TREC ToT 2025 TEST results—significant performance degradation on structurally different queries. MS-MARCO shows relative change from in-domain performance. BM25 baseline included for context. BM25 serves as first-stage retriever for candidate generation; MRR@10 not computed as it is not a reranking model.
| Query Length | MRR@10 | Recall@1000 | nDCG@10 |
|---|---|---|---|
| 10 words | 2.09% | 16.40% | 2.51% |
| 20 words | 4.32% | 25.24% | 5.05% |
| 40 words | 4.27% | 25.88% | 4.82% |
| 60 words | 4.27% | 25.88% | 4.82% |
| 80 words | 4.27% | 25.88% | 4.82% |
| 100 words | 4.27% | 25.88% | 4.82% |
Table 7. Query length ablation on ToT. Performance peaks at 20 words then plateaus, demonstrating MaxSim’s saturation rather than gradual degradation. ConstBERT with FAISS-IVF.
| Model | Backend | MRR@10 | Recall@1000 |
|---|---|---|---|
| ConstBERT | Exact MaxSim | 5.08% | 24.0% |
| ConstBERT | FAISS-IVF | 4.27% | 25.88% |
| ColBERT-v2 | FAISS-IVF | 5.66% | 28.94% |
Table 8. Exact MaxSim vs. approximate retrieval on ToT (100 sampled queries). Similar performance across methods confirms architectural limitations, not backend approximation, cause ToT failure.
| Model | Split | Queries | MRR@10 | nDCG@10 | R@1000 | Base |
| ColBERT-v2 | Base (pretrained) | — | 5.66% | 6.07% | 28.94% | — |
| DEV1 (fine-tuned) | 142 | 3.65% | 3.78% | 12.68% | –35.5% | |
| DEV2 (fine-tuned) | 143 | 2.23% | 2.33% | 13.29% | — | |
| DEV3 (fine-tuned) | 536 | 2.43% | 2.71% | 10.82% | — | |
| TEST (n=3) | 622 | 4.01% 0.08% | — | 21.70% 0.23% | –29.2% | |
| ConstBERT | Base (pretrained) | — | 4.27% | 4.82% | 25.72% | — |
Table 9. Fine-tuning results on ToT across all evaluation splits. Both models show degradation when fine-tuned on the small TRAIN split (143 queries)—a counterintuitive result suggesting limited training data and distributional mismatch prevent effective adaptation. TEST results are multi-seed (n=3, seeds 42/123/456); DEV splits are single-run from fine-tuned models.