cs.IRAug 17, 2026
SaveThe Commercial Tax: Rent-vs-Own Blind Spots in Multi-Hop Retrieval Benchmarks
Organizations: Toryx
Abstract
Enterprises connect language models to their own data through retrieval. The benchmarks that rank multi-hop retrieval systems leave out two facts a buyer needs before a published number can be used: whether the retrieval backbone may be deployed commercially, and what it costs to build. On licensing: the field's dense-retrieval anchor, NV-Embed-v2, is licensed cc-by-nc-4.0. Of the four leading MuSiQue systems we audit (HippoRAG-2, PropRAG, SAG, KET-RAG), three depend on it for their best numbers and none says so. On performance: we measure thirteen embedders from eight makers on one identical MuSiQue harness with bootstrap confidence intervals throughout. Until mid-2026 there was a real commercial tax: the best commercially-licensed embedder trailed the anchor by 2.31 Recall@5 points (95% CI [0.91, 3.71], p=0.001). NVIDIA's Nemotron-3-Embed-8B, released 2026-07-16, has closed it: +0.24 at Recall@5 (95% CI [-0.94, +1.43], p=0.69), -0.58 at Recall@10 (p=0.28). It matches the anchor, does not beat it, and is the only entrant that is commercially licensed, free to self-host, and indistinguishable from the anchor; every other entrant meeting the first two conditions sits 5.2 to 14.6 points below. The durable finding is the paid-versus-free divide: API embedders charge per token on every re-index, self-hosted ones charge nothing. On cost: three of five audited systems (adding Microsoft's GraphRAG) do not disclose indexing cost, and the only published GraphRAG dollar figures span 11x inside one third-party paper (USD 2.30 vs USD 24.94 to index a 5.64 MB corpus once); extrapolated to 1 TB that undisclosed choice separates roughly USD 428K from $4.6M. Our cost model keeps one-time embedding apart from recurring answering: at 1 TB, embedding sits 7.5x-900x below graph construction, and a year of answering at 10,000 queries/day sits 350x or more below it.
Figures & tables
Figure 1: Retrieval-augmented generation: offline indexing builds a vector index once from the document corpus; online, a query is embedded, matched against that same index for top- nearest neighbors, and the retrieved passages are handed to an LLM to compose an answer.
| Embedder | Maker | License | Recall@5 | 95% CI | Recall@10 | 95% CI |
|---|---|---|---|---|---|---|
| Nemotron-3-Embed-8B (self-host) | NVIDIA | Commercial | 69.79 | [68.13, 71.43] | 77.54 | [75.96, 79.09] |
| NV-Embed-v2 (research anchor) | NVIDIA | Non-commercial | 69.55 | [67.89, 71.20] | 78.12 | [76.58, 79.63] |
| Gemini embedding-001 | Commercial | 67.24 | [65.45, 69.01] | 76.35 | [74.71, 77.97] | |
| Nemotron-3-Embed-1B (NIM) | NVIDIA | Commercial | 64.32 | [62.63, 65.98] | 72.79 | [71.13, 74.43] |
| Llama-Nemotron-Embed-1B-v2 (NIM) | NVIDIA | Commercial | 63.73 | [62.03, 65.42] | 71.35 | [69.69, 73.01] |
| Cohere Embed v4 (Bedrock) | Cohere | Commercial | 60.21 | [58.45, 61.98] | 69.08 | [67.34, 70.81] |
Table 1: Recall@5/@10 across the full 13-embedder panel, one identical harness, every passage embedded as title\ntext . Confidence intervals are percentile bootstrap at throughout (the 1,000 questions are resampled with replacement times and the interval is the middle 95% of the resulting values), marginal rows and paired families alike; the resampling unit is the evaluation question. Printed values use binary-float half-even rounding, so a displayed may round either way. On Recall@5, only NV-Embed-v2 is statistically indistinguishable from the top entrant; every other embedder is significantly below it at 95%, and that verdict survives multiplicity correction. That verdict is Recall@5 only: the corrected family was computed at and no top-entrant-referenced Recall@10 family was run, so the @10 column carries marginal intervals and no multiplicity claim. It would not carry the same one: at the anchor sits above the top entrant (78.12 vs 77.54), which is the sign reversal the abstract reports. Pairwise inference must not be read from the overlap of these marginal intervals: the paired comparisons, both correction families, and the simultaneous intervals are in § 4.3 . BAAI is the Beijing Academy of Artificial Intelligence.
Figure 2: Recall@5 across the panel with 95% bootstrap confidence intervals, coded by license (hatching repeats the license coding so the figure survives greyscale printing). Nemotron-3-Embed-8B and NV-Embed-v2 overlap almost entirely, which is what the paired test reports ( ). These are marginal intervals and show magnitude and precision only; pairwise separation must not be read off their overlap. § 4.3 explains why, and the corrected paired comparisons in Table 1 are the inferential statement.
| System | Cost disclosed in own paper? | What the paper actually reports |
|---|---|---|
| HippoRAG-2 | Tokens, not dollars | 9.2M input / 3.0M output tokens to index the corpus (its Table 12); $1.59 at gpt-4o-mini batch rates is our conversion (Appendix E ), not a disclosed price |
| PropRAG | Yes | $4 to index the identical corpus (Appendix A.5) |
| SAG | No | Zero dollar figures despite qualitative cost discussion and claimed production scale; reports only a token-count-vs-recall curve not convertible to dollars |
| KET-RAG | Yes | 2.30 for Microsoft’s Graph-RAG on that corpus — the third-party figure used in Table 3 . Its introduction separately cites $21 for knowledge-graph (KG-RAG) indexing of a 3.2MB HotpotQA sample, as motivation, not as KET-RAG’s own cost |
| Microsoft GraphRAG (own paper) | No | Zero dollar figures; reports only wall-clock indexing time (281 minutes, one small dataset) |
| NVIDIA NeMo NIMs (platform) | License/infrastructure pricing only | $1–8/hr AI Enterprise license tiers plus GPU compute — structurally a different pricing model than per-corpus figures |
Table 2: Cost-disclosure audit across the five audited systems. The sixth row, NVIDIA NeMo NIMs, is the hosting platform, not an audited system, and is listed for context only; it is excluded from the five-system counts in the abstract and § 4.5 .
| System | Rate ($/MB) | 5GB | 100GB | 1TB |
|---|---|---|---|---|
| HippoRAG-2 (token-based reconstr.) | $0.28 | $1,440 | $28,900 | $296,000 |
| PropRAG | $0.71 | $3,630 | $72,600 | $744,000 |
| GraphRAG (KET-RAG, low-cost cfg.) | $0.41 | $2,090 | $41,800 | $428,000 |
| GraphRAG (KET-RAG, high-perf. cfg.) | $4.42 | $22,600 | $452,800 | $4,637,000 |
| KET-RAG (own system, low-cost) | $0.34 | $1,720 | $34,300 | $351,000 |
| GraphRAG (token-based reconstr.) | $3.46 | $17,700 | $354,000 | $3,624,000 |
Table 3: Standardized indexing cost at three illustrative corpus sizes (linear extrapolation; see caveat in text). The 1.59/5.64MB ==19.49–7.2M. That range brackets KET-RAG’s high-performance configuration while sitting far above its low-cost one: GraphRAG’s build cost is configuration-dominated, which is the finding.
Figure 3: Indexing cost at scale. GraphRAG’s cost spans 11x between KET-RAG’s low-cost and high-performance configurations (the ratio is scale-invariant under linear rates); the independent token-based reconstruction brackets the high-performance figure. A multi-million-dollar configuration choice at 1TB, about which GraphRAG’s own paper discloses nothing.
| Embedder | Rate | MuSiQue corpus | @10GB |
|---|---|---|---|
| OpenAI text-embedding-3-small | $0.02/1M | $0.03 | $46 |
| Voyage voyage-3.5 | $0.06/1M | $0.08 | $138 |
| Gemini embedding-001 (batch) | $0.075/1M | $0.10 | $173 |
| Cohere Embed v4 (Bedrock) | $0.12/1M | $0.15 | $277 |
| OpenAI text-embedding-3-large | $0.13/1M | $0.17 | $300 |
| Gemini embedding-001 (standard) | $0.15/1M | $0.19 | $346 |
Table 4: One-time embedding cost on the 11,656-passage MuSiQue corpus (5.64MB of raw text, 1.27M tokens, 1.6% of them evaluation queries), and extrapolated to a 10GB corpus. Token density here is empirical to this corpus ( 4.66 characters/token, queries included, with 5.64MB read as binary per the Table 3 convention). Appendix C instead applies a 4-characters/token approximation to legal text, so its 1TB embedding figures run about 10% above what size-scaling this table would give; the two are not meant to be scaled into one another.
| Embedder | Cost @k=5 (batch/std) | Cost @k=10 (batch/std) |
|---|---|---|
| Nemotron-3-Embed-8B | 0.116 | 0.225 |
| Gemini embedding-001 | 0.113 | 0.215 |
| Nemotron-3-Embed-1B | 0.110 | 0.209 |
| Llama-Nemotron-Embed-1B-v2 | 0.118 | 0.229 |
| OpenAI text-embedding-3-large | 0.116 | 0.217 |
| Cohere Embed v4 | 0.113 | 0.215 |
Table 5: Answering cost at k=5 and k=10, all 1,000 queries, gpt-4o-mini as the LLM, at batch and standard API rates. Twelve rows, not thirteen: NV-Embed-v2 is absent because answering cost depends on the retrieved passages’ token length, and the anchor’s answering run was not repeated after the index correction. Its cost would fall inside the range shown: the twelve span 0.060 at k=5, a spread of $0.007 per 1,000 queries, and the spread, not the cost, is the small number here. We do not report a figure we did not measure.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Embedder | Access | Notes |
|---|---|---|
| NV-Embed-v2 [ 15 ] | Self-hosted | Non-commercial research anchor, measured on this harness, not taken from the literature (§ 4.2 ) |
| Nemotron-3-Embed-8B-BF16 [ 16 ] | Self-hosted | OpenMDW-1.1, “ready for commercial use.” Released 2026-07-16; not served by NVIDIA’s hosted NIM endpoint |
| Nemotron-3-Embed-1B-BF16 [ 17 ] | NIM API (served id nemotron-3-embed-1b ) | Cheaper Nemotron tier |
| Llama-Nemotron-Embed-1B-v2 | NIM API | Cheaper Nemotron tier |
| nv-embedqa-e5-v5 [ 18 ] | NIM API | 335M params, fine-tuned E5-Large-Unsupervised, trained on NVIDIA’s own commercially-viable QA data |
| text-embedding-3-small / -large [ 20 ] | Direct API | — |
Table 6: Panel composition: access channel and per-model notes. Maker and license for each entrant are in Table 1 .
Figure 4: What an embedder does (a), and how cosine similarity turns embedded vectors into a ranked retrieval result (b).
| Matter | Files | Extracted text | Tokens |
|---|---|---|---|
| Gaming strategic acquisition | 3,391 | 225.4 MB | 56.3M |
| Restaurant PE buyout | 3,378 | 217.8 MB | 54.5M |
| Two matters | 6,769 | 443.2 MB | 110.8M |
Table 7: Extracted text volume of two single legal-diligence matters, measured from the public LAB corpus. Token counts use a 4-characters-per-token approximation (extracted-text sizes in this appendix are decimal MB, the convention its printed token counts imply; reading them as binary shifts every figure by 5% and the matched-scale factor’s 7.5x lower bound to 7.2x).
| Volume | Embedding | Indexing |
|---|---|---|
| One matter | 8 | 997 |
| 23 matters ( 5GB) | 194 | 22,900 |
| 100 matters/year | 845 | 99,700 |
| 4,650 matters (1TB) | 39.3K | 4.64M |
Table 8: The same corpus priced two ways, per re-index. Embedding ranges across the four API providers of Table 4 and is O(N)$ assumption: if entity extraction and community detection scale super-linearly with graph density, the true figures are higher, and disproportionately so in the bottom row.
| Embedder | title+text | text-only | |
|---|---|---|---|
| Gemini embedding-001 | 67.24 | 65.78 | |
| Llama-Nemotron-Embed-1B-v2 | 63.73 | 59.81 | |
| Nemotron-3-Embed-1B | 64.32 | 60.86 | |
| mxbai-embed-large-v1 | 55.71 | — | |
| Qwen3-VL-Embedding-8B | 59.88 | — | |
| BGE-M3 | 54.93 | — |
Table 9: Recall@5 on the matched title\ntext corpus and on a body-text-only corpus. is run-to-run variation on the matched corpus. All thirteen entrants reproduce within 0.6 points and were therefore indexed correctly: twelve against their own original run, the anchor against a dedicated replicate, for the reason given in the text; is signed re-measurement minus original, so text-embedding-3-large ’s means its original run read 60.02 and the re-measurement 59.48, still far nearer its title\ntext figure than its 58.69 text-only one, which is the comparison that establishes it was indexed correctly. NV-Embed-v2’s original index was the text-only one, so it had no matched-corpus predecessor; its comes instead from a dedicated replicate run (2026-08-08): the full pipeline re-executed end-to-end with the corpus re-embedded from raw text into a fresh cache, same configuration. The two runs agree on every one of the 1,000 per-question scores, so exactly: deterministic local inference, in line with the other locally-hosted entrants, whose deltas span to . The separating its two corpus formats is a format penalty and appears below, not in this column. Where a text-only column is blank the model was measured against its already-cached corpus and not re-embedded in both formats. For NV-Embed-v2 both columns report the web-search query instruction, which is also the best of the four swept instructions on the text-only index (67.09 vs 67.07/66.64/64.07), so the row differences the same configuration and the penalty is not inflated by comparing a swept maximum to an unswept point; mixing instructions across the two columns would make the row’s own subtraction meaningless.
| Embedder | R@10 R@5 lift | 95% CI |
|---|---|---|
| text-embedding-3-large | ||
| voyage-3.5 | ||
| text-embedding-3-small | ||
| Gemini embedding-001 | ||
| Cohere Embed v4 | ||
| Qwen3-VL-Embedding-8B |
Table 10: The paired intervals behind every lift and format-penalty claim in § 4.2 and this appendix: Recall@10 Recall@5 lift for all thirteen entrants (top block) and the corpus-format penalty for the ten measured in both formats (bottom block). Paired bootstrap over the same 1,000 questions, , seed 42; produced by scripts/panel_intervals.py and scripts/panel_simultaneous_ci.py from the released per-question vectors; the simultaneous block prints all twelve Dunnett-type intervals instead of asserting their verdict. † interval includes zero: no penalty is claimed for this entrant. Two apparent oddities are verified, not errors: NV-Embed-v2 and nv-embedqa-e5-v5 print identical lift rows because their mean lifts are exactly equal (8.575) and these intervals live on a coarse question-mean grid — the underlying per-question vectors differ on 311 questions, their standard errors differ (0.545 vs 0.541, panel_derived_stats.json ), and their format-penalty rows differ; the endpoints coincide at full stored precision and not merely at two-decimal display, the grid being coarse enough for two distinct vectors to land on identical percentiles; and two penalties differ by from subtracting Table 9 ’s rounded cells (cell rounding; the artifact carries unrounded values). These are secondary analyses, reported uncorrected (Table 1 caption).
More in this topic
Dense Retrieval
Dense retrieval uses learned vector representations to find relevant documents, passages, products, or tools by semantic similarity.
See all 153 papers