FERN: Cross-Benchmark Table Retrieval for Text-to-SQL with Frozen Retrievers and Learned Ranking
Abstract
Text-to-SQL systems over large pooled catalogs must first retrieve the tables a question needs. We compare against the published table recall of RASL, which uses Claude 3.5 models to extract keywords and select tables, on pooled BIRD, Spider, and Fiben catalogs. FERN (Feature-Engineered Ranking with No generative LLM) instead fuses three frozen retrievers into 100 candidates, describes each with up to 120 features, and reorders them with LightGBM rankers fitted once on BIRD's training questions. On Spider, where it was not fitted, its candidate pool alone reaches RASL's published recall at fifteen tables (98.0), and the ranker raises the share of questions with every gold table in the top five (complete@5) from 83.9% to 95.6%. At five tables, FERN's recall is within one point of RASL's on Spider (96.7 vs 97.0) and on BIRD with its evidence hints (96.6 vs 97.5), and 8.2 points below on BIRD without them (89.3). On 121 real BEAVER questions, without BEAVER fitting, its recall at fifteen (82.1) falls between two BEAVER-trained retrievers (80.8, 84.5), but its complete@15 is below both (49.6 vs 54.5 and 62.0). On Fiben, it falls far short at fifteen tables (54.2 vs 69.2, different question sets), although its recall at fifty is numerically close to RASL's question-only retrieval result (80.7 vs 79.4). On BIRD dev with evidence, non-retrieval features raise complete@5 by 7.7 points over retrieval signals alone. On BIRD dev, median latency is 208 ms per question on one A100, without hosted model calls.
Figures & tables
| Dataset | Databases | Tables | Columns | Test questions | Gold tables per question |
|---|---|---|---|---|---|
| BIRD dev | 80 | 597 | 4,337 | 1,534 | 1.94 |
| Spider dev | 166 | 876 | 4,503 | 1,034 | 1.51 |
| Fiben | 1 | 152 | 374 | 298 (RASL: 279) | 3.82 |
| BEAVER, three warehouses | 3 | 381 | 121 real; 1,000 synthetic | 3.93; 3.52 |
| RASL full | FERN-ev | FERN-q | FERN-schema | |
|---|---|---|---|---|
| Supervision | 200 calibration questions per dataset (none on Fiben) | 9,394 labeled BIRD train questions, fitted once | same | same |
| BIRD evidence | not stated; its examples include evidence-like clauses | yes | no | no |
| Cell values | not indexed | yes | yes | no |
| Column and value descriptions (BIRD) | yes | no | no | no |
| Generative model calls per question | 2 (keywords, table selection) | 0 | 0 | 0 |
| Embedding models | Cohere Embed v3 | Qwen3-Embedding-0.6B, BGE-large | same | same |
| Family | n | Content |
|---|---|---|
| score | 30 | each signal’s raw score, rank, z-score, gap to the best candidate and percentile |
| agree | 9 | minimum, maximum, mean and standard deviation of the six ranks; how many signals place the table in their top 5 and top 15; rank differences Qwen3 minus BM25, Qwen3 minus BGE, name card minus full card |
| db | 14 | for the pool score and for Qwen3: the database’s best score, mean score and probability share (softmax of the candidates’ z-scores, summed by database), the database’s rank by best score, the table’s rank within its database, its gap to the database’s best; plus the number of candidates from the database and its table count |
| lex | 8 | overlap of question tokens with table-name tokens and with all name tokens of the table, IDF-weighted overlap, overlap count, best fuzzy match, fuzzy matches above 0.8, natural column names found in the question, table name found in the question |
| cov | 3 | share of question tokens covered by the table’s tokens; tokens it adds beyond the top candidate and beyond the top three |
| ev | 3 | evidence contains the table name, how many column names, how many natural column names (zero without evidence) |
| Input | Source | Used by |
|---|---|---|
| Question text | the benchmark question | retrievers, features |
| Evidence text | BIRD’s evidence field | FERN-ev only: appended to the question for all retrievers and for every feature computed from question text (lex, cov, val literals, type and q flags, emb), plus the ev features and the evidence flag |
| Schema: names, types, keys | the catalog | retrievers (names only), features (names, types and keys) |
| Cell values, row counts | the catalog’s SQLite files, never the gold SQL | val and meta features, FERN-ev and FERN-q |
| Retriever scores | frozen models | score, agree and db features |
| Gold tables | parsed from gold SQL for BIRD, Spider and Fiben; supplied by BEAVER for BEAVER | fitting the rankers (BIRD train) and evaluation only |
| BIRD R@5 | BIRD R@15 | Spider R@5 | Spider R@15 | Fiben R@5 | Fiben R@15 | |
|---|---|---|---|---|---|---|
| BM25 (RASL’s Table 2) | 68.3 | 82.8 | 86.5 | 93.9 | 33.3 | 38.6 |
| DBCopilot | 85.8 | 94.6 | 91.6 | 97.6 | 41.1 | 56.9 |
| RASL retriever | 70.5 | 92.6 | 72.1 | 94.1 | 34.6 | 64.5 |
| RASL full | 97.5 | 97.8 | 97.0 | 98.0 | 69.1 | 69.2 |
| FERN pool alone | 86.8 | 96.2 | 89.6 | 98.0 | 24.8 | 39.3 |
| FERN | 96.6 [95.9, 97.4] | 98.9 [98.4, 99.2] | 96.7 [95.6, 97.6] | 99.1 [98.6, 99.6] | 36.7 [33.7, 39.5] | 54.2 [50.8, 57.4] |
| Dataset | Method | R@5 | R@10 | R@15 | C@5 | C@10 | C@15 |
|---|---|---|---|---|---|---|---|
| BIRD, evidence | BM25 rerank of pool | 50.1 | 66.0 | 73.6 | 31.4 | 47.0 | 56.1 |
| pool | 86.8 | 93.6 | 96.2 | 75.0 | 87.4 | 91.9 | |
| FERN-ev | 96.6 | 98.5 | 98.9 | 93.9 | 96.8 | 97.7 | |
| BIRD, question only | BM25 rerank of pool | 30.0 | 48.1 | 58.5 | 16.3 | 30.4 | 40.1 |
| pool | 81.5 | 90.4 | 94.2 | 67.4 | 81.6 | 88.3 | |
| FERN-q | 89.3 | 95.1 | 97.1 | 83.3 | 91.9 | 95.0 |
| k | BIRD, FERN-ev | BIRD, FERN-q | Spider, FERN-q | BIRD pool alone (R) |
|---|---|---|---|---|
| 1 | 54.7 / 21.6 | 45.8 / 18.1 | 64.6 / 44.8 | 47.6 |
| 2 | 84.4 / 69.7 | 70.7 / 51.6 | 86.2 / 77.4 | 68.9 |
| 3 | 92.5 / 85.5 | 81.4 / 69.8 | 93.2 / 89.7 | 78.1 |
| 4 | 95.4 / 91.3 | 86.3 / 78.3 | 95.6 / 93.9 | 84.0 |
| 5 | 96.6 / 93.9 | 89.3 / 83.3 | 96.7 / 95.6 | 86.8 |
| 6 | 97.4 / 95.4 | 91.0 / 85.7 | 96.9 / 95.9 | 89.1 |
| Questions | Method | Trained on BEAVER | R@5 | R@15 [95% CI] | C@5 | C@15 |
|---|---|---|---|---|---|---|
| 121 real | BM25 rerank of pool | no | 48.9 | 70.4 | 16.5 | 29.8 |
| pool | no | 55.3 | 78.8 | 20.7 | 39.7 | |
| FERN-schema | no | 57.0 | 82.1 [78.3, 85.5] | 22.3 | 49.6 | |
| NIFT-SQL | yes | 56.6 | 80.8 | 24.0 | 54.5 | |
| retriever fine-tuned on BEAVER (label-matched) | yes | 53.0 | 84.5 | 22.3 | 62.0 | |
| 1,000 synthetic | BM25 rerank of pool | no | 53.9 | 83.9 | 13.7 | 53.3 |
| Features | CV R@5 | CV C@5 | CV R@15 | CV C@15 | dev R@5 | dev C@5 | dev R@15 | dev C@15 |
|---|---|---|---|---|---|---|---|---|
| retrieval signals only (53) | 78.6 | 70.0 | 90.7 | 86.8 | 91.9 | 86.2 | 98.2 | 96.6 |
| other families only (67) | 86.4 | 77.1 | 94.6 | 89.9 | 95.4 | 91.9 | 98.5 | 97.1 |
| all (120) | 89.2 | 82.1 | 95.8 | 92.8 | 96.6 | 93.9 | 98.9 | 97.7 |
| all minus retrieval-only | +12.1 [11.4, 12.9] | +6.0 [5.5, 6.5] | +7.7 [6.3, 9.1] | +0.6 [0.3, 1.0] | ||||
| all minus other-only | +5.0 [4.5, 5.6] | +2.9 [2.5, 3.3] | +2.0 [1.0, 2.9] | +0.4 [0.1, 0.7] |
| Family removed | dev C@5 | dev R@15 |
|---|---|---|
| db (14) | +0.8 [0.1, 1.6] | +0.2 [0.1, 0.4] |
| graph (7) | +0.8 [0.1, 1.6] | -0.1 [-0.2, 0.1] |
| val (6) | +0.7 [0.0, 1.4] | +0.1 [-0.1, 0.4] |
| lex (8) | +0.3 [-0.4, 1.0] | +0.1 [-0.1, 0.3] |
| score, agree, cov, ev, type, meta, q, emb | between -0.3 and +0.1, all intervals include zero | between -0.1 and 0.0 |
| Stage | Median (ms) | 90th percentile (ms) |
|---|---|---|
| embed the question with Qwen3 and BGE (GPU) | 71.9 | 87.6 |
| BM25, two views (CPU) | 6.6 | 10.5 |
| dense scores and pool | 0.4 | 12.8 |
| features (CPU, Python) | 122.7 | 146.0 |
| ranker (CPU) | 6.4 | 10.2 |
| total | 208.3 | 246.9 |
| Signals fused | R@100 | C@100 |
|---|---|---|
| BM25 | 93.7 | 87.2 |
| Qwen3 | 98.8 | 97.5 |
| BGE | 98.4 | 96.8 |
| BM25 + Qwen3 | 98.9 | 97.7 |
| BM25 + BGE | 98.4 | 96.5 |
| Qwen3 + BGE (selected by the rule) | 99.3 | 98.4 |
| Ranker | C@5 | C@15 | Decision |
|---|---|---|---|
| all signals | 82.6 | 92.7 | |
| without BM25, full card | 82.1 | 92.9 | keep |
| without BM25, name card | 81.8 | 93.1 | keep |
| without Qwen3, full card | 81.9 | 92.5 | keep |
| without Qwen3, name card | 81.6 | 92.8 | keep |
| without Qwen3, best column | 82.4 | 92.8 | drop |
| Ranker | Labels | Features | R@15 | C@5 | C@15 |
|---|---|---|---|---|---|
| hybrid search (BM25 + Qwen3) | sqlglot | 94.2 | 69.4 | 88.5 | |
| 12 score features, stacked | regex | 12 | 97.9 | 87.0 | 96.1 |
| all features with the decision model, stacked | regex | 139 | 98.7 | 93.0 | 97.5 |
| retrieval signals only, lambdarank | sqlglot | 58 | 98.1 | 86.4 | 96.5 |
| other features only, lambdarank | sqlglot | 66 | 98.1 | 91.5 | 96.3 |
| all features without the decision model, stacked | sqlglot | 124 | 98.8 | 93.0 | 97.6 |
Table Retrieval
Table retrieval finds relevant tables and schema elements for question answering, analytics, and text-to-SQL.
See all 20 papers