CoRA-Bench: Evaluating Evidence Preservation by Context Selectors Under Token Budgets
Organizations: Stanford University · Georgia Institute of Technology · ZettaQuant
Abstract
Context selection can reduce input length during long-input inference but can remove evidence required to answer a query. We introduce CoRA-Bench, a benchmark for evaluating evidence preservation under shared token budgets. We compare head and tail truncation, BM25, dense retrieval, dense retrieval with reranking, LLMLingua-2, and Provence across BABILong, LOONG Financial Spotlight, a custom NoLiMa diagnostic, and LongMemEval. Evidence-preservation results reveal task-dependent tradeoffs: BM25 achieves 0.921 supporting-fact recall on BABILong at 128K, dense retrieval and dense reranking each achieve 0.908 needle recall on NoLiMa at 64K, and dense retrieval achieves 0.993 answer-bearing-turn recall on LongMemEval at 64K. Evidence, token retention, and complementary reader evaluations use the same selected contexts. Reader evaluations use dataset-native answer metrics and three runs on the same questions, with paired bootstrap confidence intervals. At a 32K budget, BM25 achieves 46.20% BABILong answer accuracy and dense retrieval achieves 71.13% LongMemEval accuracy. The results show that method rankings vary across evidence types and between evidence recall and downstream accuracy.
Figures & tables
| Dataset | Selection stress | N | Source tokens | Configuration |
|---|---|---|---|---|
| BABILong ( Kuratov et al., 2024 ) | Sparse supporting facts | 500 | 217.7K to 253.7K | qa1 to qa5, 100 examples each, generated at 256K with gold support occurrences |
| LOONG ( Wang et al., 2024 ) | Financial line localization | 75 | 45.2K to 326.4K | English Financial Spotlight, level 1; 2 to 9 documents; 74/69/50 evidence-evaluable at 32K/64K/128K |
| NoLiMa ( Modarressi et al., 2025 ) | Minimal lexical overlap | 870 | 128.57K to 128.74K | 58 question and needle pairs, 3 depths, 5 fixed haystacks; repeated measures |
| LongMemEval ( Wu et al., 2025 ) | Conversational retrieval | 500 | 98.4K to 107.1K | Cleaned small split; 470 answerable and 30 abstention examples |
| Method | BABILong | LOONG | NoLiMa | LongMemEval |
|---|---|---|---|---|
| Full context | 31.13 | 72.44 | 0.57 | 63.20 |
| Empty context | 15.00 | 0.00 | 0.00 | 10.27 |
| Head and tail | 21.40 | 78.67 | 0.00 | 25.27 |
| BM25 | 46.20 | 69.33 | 3.10 | 68.13 |
| Dense | 30.27 | 69.33 | 4.60 | 71.13 |
| Dense+Rerank | 32.20 | 75.56 | 5.29 | 70.60 |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | BABILong | LOONG | NoLiMa | LongMemEval | Total |
|---|---|---|---|---|---|
| Dense | 2.5 | 0.3 | 1.2 | 1.1 | 5.1 |
| Dense+Rerank | 3.8 | 0.5 | 2.4 | 1.8 | 8.5 |
| LLMLingua-2 | 4.6 | 0.4 | 2.9 | 1.4 | 9.3 |
| Provence | 6.6 | 2.4 | 11.5 | 5.1 | 25.6 |
| Total | 17.5 | 3.6 | 18.0 | 9.4 | 48.5 |
| Component | Configuration |
|---|---|
| Reader | gemini-2.5-flash-lite ; temperature 0; thinking budget 0; output cap 1,024; concurrency 8 |
| Reader repetitions | Same questions; seeds 5768, 78516, 944601 |
| BABILong metric | Pinned official task-label answer scorer and qa1-qa5 prompts |
| NoLiMa metric | Released case-sensitive contains scorer on the frozen custom diagnostic |
| LongMemEval judge | gpt-4o-2024-08-06 ; official question-type/abstention rubric; temperature 0; 10 output tokens |
| LOONG judge | gpt-5.5-2026-04-23 ; original rubric with an adapted judge model; reasoning disabled; 512 output tokens |
| Method | Budget | Coverage | Accuracy [95% CI] | Seed min-max | Cap hits |
|---|---|---|---|---|---|
| Full context | n/a | 500/500 | 31.13 [27.33, 35.07] | 30.80 to 31.60 | 0 |
| Empty context | n/a | 500/500 | 15.00 [12.00, 18.00] | 14.20 to 15.40 | 0 |
| Head and tail | 32K | 500/500 | 21.40 [18.00, 24.80] | 21.40 to 21.40 | 0 |
| Head and tail | 64K | 500/500 | 30.13 [26.33, 33.87] | 30.00 to 30.20 | 0 |
| Head and tail | 128K | 500/500 | 32.47 [28.53, 36.40] | 32.40 to 32.60 | 0 |
| BM25 | 32K | 500/500 | 46.20 [42.60, 49.80] | 46.20 to 46.20 | 0 |
| Method | Budget / policy | Coverage | Accuracy [95% CI] | Seed range | Cap hits |
|---|---|---|---|---|---|
| Full context | n/a | 75/75 | 72.44 [61.78, 82.22] | 72.00 to 73.33 | 0 |
| Empty context | n/a | 75/75 | 0.00 [0.00, 0.00] | 0.00 to 0.00 | 0 |
| Head and tail | 32K | 75/75 | 78.67 [69.78, 87.12] | 77.33 to 80.00 | 0 |
| Head and tail | 64K | 70/70 | 80.00 [70.48, 88.57] | 78.57 to 81.43 | 0 |
| Head and tail | 128K | 51/51 | 67.32 [54.90, 79.08] | 64.71 to 70.59 | 0 |
| BM25 | 32K | 75/75 | 69.33 [59.11, 79.56] | 68.00 to 70.67 | 0 |
| Method | Budget | Coverage | Accuracy [95% CI] | Seed min-max | Cap hits |
|---|---|---|---|---|---|
| Full context | n/a | 870/870 | 0.57 [0.00, 1.48] | 0.57 to 0.57 | 0 |
| Empty context | n/a | 870/870 | 0.00 [0.00, 0.00] | 0.00 to 0.00 | 0 |
| Head and tail | 32K | 870/870 | 0.00 [0.00, 0.00] | 0.00 to 0.00 | 0 |
| Head and tail | 64K | 870/870 | 0.27 [0.00, 0.88] | 0.23 to 0.34 | 0 |
| BM25 | 32K | 870/870 | 3.10 [0.65, 6.42] | 3.10 to 3.10 | 0 |
| BM25 | 64K | 870/870 | 1.69 [0.17, 4.11] | 1.61 to 1.72 | 0 |
| Method | Budget | Coverage | Accuracy [95% CI] | Seed min-max | Cap hits |
|---|---|---|---|---|---|
| Full context | n/a | 500/500 | 63.20 [59.00, 67.13] | 62.80 to 63.80 | 156 |
| Empty context | n/a | 500/500 | 10.27 [7.53, 13.00] | 10.20 to 10.40 | 3 |
| Head and tail | 32K | 500/500 | 25.27 [21.53, 29.07] | 25.00 to 25.40 | 97 |
| Head and tail | 64K | 500/500 | 44.13 [39.93, 48.20] | 44.00 to 44.20 | 121 |
| BM25 | 32K | 500/500 | 68.13 [64.00, 72.07] | 67.00 to 68.80 | 115 |
| BM25 | 64K | 500/500 | 67.87 [63.93, 71.93] | 67.60 to 68.20 | 118 |
| Method | 32K | 64K | 128K |
|---|---|---|---|
| Full context | 88.75 | 88.75 | 88.75 |
| Empty context | 6.60 | 6.60 | 6.60 |
| Head and tail | 91.21 | 90.39 | 87.19 |
| BM25 | 81.11 | 85.78 | 79.48 |
| Dense | 84.39 | 88.72 | 81.73 |
| Dense+Rerank | 89.98 | 88.67 | 82.45 |
| Evidence recall | Compression ratio | |||||
|---|---|---|---|---|---|---|
| Method | 32K | 64K | 128K | 32K | 64K | 128K |
| Head and tail | 0.238 | 0.446 | 0.659 | 0.130 | 0.260 | 0.519 |
| BM25 | 0.838 | 0.868 | 0.921 | 0.129 | 0.259 | 0.518 |
| Dense | 0.329 | 0.481 | 0.727 | 0.129 | 0.259 | 0.519 |
| Dense+Rerank | 0.395 | 0.483 | 0.714 | 0.129 | 0.258 | 0.491 |
| LLMLingua-2 | 0.000 | 0.000 | 0.003 | 0.105 | 0.216 | 0.453 |
| Evidence recall | Compression ratio | |||||
|---|---|---|---|---|---|---|
| Method | 32K | 64K | 128K | 32K | 64K | 128K |
| Head and tail | 0.973 | 0.986 | 1.000 | 0.241 | 0.427 | 0.653 |
| BM25 | 0.932 | 0.942 | 0.980 | 0.239 | 0.426 | 0.652 |
| Dense | 0.946 | 0.986 | 0.980 | 0.240 | 0.425 | 0.652 |
| Dense+Rerank | 0.973 | 0.986 | 0.980 | 0.240 | 0.417 | 0.609 |
| LLMLingua-2 | 0.122 | 0.188 | 0.220 | 0.226 | 0.389 | 0.585 |
| Evidence recall | Compression ratio | |||
|---|---|---|---|---|
| Method | 32K | 64K | 32K | 64K |
| Head and tail | 0.000 | 0.333 | 0.249 | 0.498 |
| BM25 | 0.264 | 0.467 | 0.247 | 0.496 |
| Dense | 0.641 | 0.908 | 0.247 | 0.496 |
| Dense+Rerank | 0.683 | 0.908 | 0.247 | 0.495 |
| LLMLingua-2 | 0.000 | 0.000 | 0.210 | 0.450 |
| Evidence recall | Compression ratio | |||
|---|---|---|---|---|
| Method | 32K | 64K | 32K | 64K |
| Head and tail | 0.301 | 0.645 | 0.305 | 0.609 |
| BM25 | 0.940 | 0.976 | 0.302 | 0.607 |
| Dense | 0.967 | 0.993 | 0.302 | 0.607 |
| Dense+Rerank | 0.979 | 0.991 | 0.302 | 0.587 |
| LLMLingua-2 | 0.000 | 0.000 | 0.286 | 0.600 |
Evidence Selection
Evidence selection chooses the documents, passages, claims, or observations that best support an answer or decision.
See all 66 papers