The Commercial Tax: Rent-vs-Own Blind Spots in Multi-Hop Retrieval Benchmarks

Luis M. Sanchez; Kosrow Dehnad

cs.IRAug 17, 2026

The Commercial Tax: Rent-vs-Own Blind Spots in Multi-Hop Retrieval Benchmarks

Authors: Luis M. Sanchez, Kosrow Dehnad

Organizations: Toryx

Abstract

Enterprises connect language models to their own data through retrieval. The benchmarks that rank multi-hop retrieval systems leave out two facts a buyer needs before a published number can be used: whether the retrieval backbone may be deployed commercially, and what it costs to build. On licensing: the field's dense-retrieval anchor, NV-Embed-v2, is licensed cc-by-nc-4.0. Of the four leading MuSiQue systems we audit (HippoRAG-2, PropRAG, SAG, KET-RAG), three depend on it for their best numbers and none says so. On performance: we measure thirteen embedders from eight makers on one identical MuSiQue harness with bootstrap confidence intervals throughout. Until mid-2026 there was a real commercial tax: the best commercially-licensed embedder trailed the anchor by 2.31 Recall@5 points (95% CI [0.91, 3.71], p=0.001). NVIDIA's Nemotron-3-Embed-8B, released 2026-07-16, has closed it: +0.24 at Recall@5 (95% CI [-0.94, +1.43], p=0.69), -0.58 at Recall@10 (p=0.28). It matches the anchor, does not beat it, and is the only entrant that is commercially licensed, free to self-host, and indistinguishable from the anchor; every other entrant meeting the first two conditions sits 5.2 to 14.6 points below. The durable finding is the paid-versus-free divide: API embedders charge per token on every re-index, self-hosted ones charge nothing. On cost: three of five audited systems (adding Microsoft's GraphRAG) do not disclose indexing cost, and the only published GraphRAG dollar figures span 11x inside one third-party paper (USD 2.30 vs USD 24.94 to index a 5.64 MB corpus once); extrapolated to 1 TB that undisclosed choice separates roughly USD 428K from $4.6M. Our cost model keeps one-time embedding apart from recurring answering: at 1 TB, embedding sits 7.5x-900x below graph construction, and a year of answering at 10,000 queries/day sits 350x or more below it.

Figures & tables

Figure 1: Retrieval-augmented generation: offline indexing builds a vector index once from the document corpus; online, a query is embedded, matched against that same index for top-

k

nearest neighbors, and the retrieved passages are handed to an LLM to compose an answer.

Embedder	Maker	License	Recall@5	95% CI	Recall@10	95% CI
Nemotron-3-Embed-8B (self-host)	NVIDIA	Commercial	69.79	[68.13, 71.43]	77.54	[75.96, 79.09]
NV-Embed-v2 (research anchor)	NVIDIA	Non-commercial	69.55	[67.89, 71.20]	78.12	[76.58, 79.63]
Gemini embedding-001	Google	Commercial	67.24	[65.45, 69.01]	76.35	[74.71, 77.97]
Nemotron-3-Embed-1B (NIM)	NVIDIA	Commercial	64.32	[62.63, 65.98]	72.79	[71.13, 74.43]
Llama-Nemotron-Embed-1B-v2 (NIM)	NVIDIA	Commercial	63.73	[62.03, 65.42]	71.35	[69.69, 73.01]
Cohere Embed v4 (Bedrock)	Cohere	Commercial	60.21	[58.45, 61.98]	69.08	[67.34, 70.81]

Table 1: Recall@5/@10 across the full 13-embedder panel, one identical harness, every passage embedded as title\ntext . Confidence intervals are percentile bootstrap at

B=10^{5}

throughout (the 1,000 questions are resampled with replacement

B=100{,}000

times and the interval is the middle 95% of the resulting values), marginal rows and paired families alike; the resampling unit is the evaluation question. Printed values use binary-float half-even rounding, so a displayed

x.xx5

may round either way. On Recall@5, only NV-Embed-v2 is statistically indistinguishable from the top entrant; every other embedder is significantly below it at 95%, and that verdict survives multiplicity correction. That verdict is Recall@5 only: the corrected family was computed at

k=5

and no top-entrant-referenced Recall@10 family was run, so the @10 column carries marginal intervals and no multiplicity claim. It would not carry the same one: at

k=10

the anchor sits above the top entrant (78.12 vs 77.54), which is the sign reversal the abstract reports. Pairwise inference must not be read from the overlap of these marginal intervals: the paired comparisons, both correction families, and the simultaneous intervals are in § 4.3 . BAAI is the Beijing Academy of Artificial Intelligence.

Figure 2: Recall@5 across the panel with 95% bootstrap confidence intervals, coded by license (hatching repeats the license coding so the figure survives greyscale printing). Nemotron-3-Embed-8B and NV-Embed-v2 overlap almost entirely, which is what the paired test reports (

p=0.69

). These are marginal intervals and show magnitude and precision only; pairwise separation must not be read off their overlap. § 4.3 explains why, and the corrected paired comparisons in Table 1 are the inferential statement.

System	Cost disclosed in own paper?	What the paper actually reports
HippoRAG-2	Tokens, not dollars	9.2M input / 3.0M output tokens to index the corpus (its Table 12); $1.59 at gpt-4o-mini batch rates is our conversion (Appendix E ), not a disclosed price
PropRAG	Yes	$\sim$ $4 to index the identical corpus (Appendix A.5)
SAG	No	Zero dollar figures despite qualitative cost discussion and claimed production scale; reports only a token-count-vs-recall curve not convertible to dollars
KET-RAG	Yes	$1.89 to index the MuSiQue corpus with its own system (its Table 2, low-cost versions). That table — captioned “Overall performance of RAG methods in low-cost versions” , row MS-Graph-RAG , column MuSiQue/USD — separately reports$ 2.30 for Microsoft’s Graph-RAG on that corpus — the third-party figure used in Table 3 . Its introduction separately cites $21 for knowledge-graph (KG-RAG) indexing of a 3.2MB HotpotQA sample, as motivation, not as KET-RAG’s own cost
Microsoft GraphRAG (own paper)	No	Zero dollar figures; reports only wall-clock indexing time (281 minutes, one small dataset)
NVIDIA NeMo NIMs (platform)	License/infrastructure pricing only	$1–8/hr AI Enterprise license tiers plus GPU compute — structurally a different pricing model than per-corpus figures

Table 2: Cost-disclosure audit across the five audited systems. The sixth row, NVIDIA NeMo NIMs, is the hosting platform, not an audited system, and is listed for context only; it is excluded from the five-system counts in the abstract and § 4.5 .

System	Rate ($/MB)	5GB	100GB	1TB
HippoRAG-2 (token-based reconstr.)	$0.28	$\sim$ $1,440	$\sim$ $28,900	$\sim$ $296,000
PropRAG	$0.71	$\sim$ $3,630	$\sim$ $72,600	$\sim$ $744,000
GraphRAG (KET-RAG, low-cost cfg.)	$0.41	$\sim$ $2,090	$\sim$ $41,800	$\sim$ $428,000
GraphRAG (KET-RAG, high-perf. cfg.)	$4.42	$\sim$ $22,600	$\sim$ $452,800	$\sim$ $4,637,000
KET-RAG (own system, low-cost)	$0.34	$\sim$ $1,720	$\sim$ $34,300	$\sim$ $351,000
GraphRAG (token-based reconstr.)	$3.46	$\sim$ $17,700	$\sim$ $354,000	$\sim$ $3,624,000

Table 3: Standardized indexing cost at three illustrative corpus sizes (linear extrapolation; see caveat in text). The

/MB column is displayed rounded but every figure is computed from the unrounded rate (e.g. HippoRAG-2 is

1.59/5.64MB

=

0.2819/MB, from its disclosed 9.2M/3.0M indexing tokens at gpt-4o-mini batch rates: a reconstruction like the GraphRAG token row, not a published price). Sizes are binary in this section’s cost tables: 1GB

1,024MB, 1TB

1,048,576MB (Appendix C states its own, decimal, convention). The GraphRAG token-based row uses the low end of the

19.49–

38.99 reconstruction of § 4.4 ; its high end would put 1TB at

7.2M. That range brackets KET-RAG’s high-performance configuration while sitting far above its low-cost one: GraphRAG’s build cost is configuration-dominated, which is the finding.

Figure 3: Indexing cost at scale. GraphRAG’s cost spans 11x between KET-RAG’s low-cost and high-performance configurations (the ratio is scale-invariant under linear rates); the independent token-based reconstruction brackets the high-performance figure. A multi-million-dollar configuration choice at 1TB, about which GraphRAG’s own paper discloses nothing.

Embedder	Rate	MuSiQue corpus	@10GB
OpenAI text-embedding-3-small	$0.02/1M	$0.03	$46
Voyage voyage-3.5	$0.06/1M	$0.08	$138
Gemini embedding-001 (batch)	$0.075/1M	$0.10	$173
Cohere Embed v4 (Bedrock)	$0.12/1M	$0.15	$277
OpenAI text-embedding-3-large	$0.13/1M	$0.17	$300
Gemini embedding-001 (standard)	$0.15/1M	$0.19	$346

Table 4: One-time embedding cost on the 11,656-passage MuSiQue corpus (5.64MB of raw text,

\sim

1.27M tokens,

\sim

1.6% of them evaluation queries), and extrapolated to a 10GB corpus. Token density here is empirical to this corpus (

\approx

4.66 characters/token, queries included, with 5.64MB read as binary per the Table 3 convention). Appendix C instead applies a 4-characters/token approximation to legal text, so its 1TB embedding figures run about 10% above what size-scaling this table would give; the two are not meant to be scaled into one another.

Embedder	Cost @k=5 (batch/std)	Cost @k=10 (batch/std)
Nemotron-3-Embed-8B	$0.058 /$ 0.116	$0.112 /$ 0.225
Gemini embedding-001	$0.057 /$ 0.113	$0.108 /$ 0.215
Nemotron-3-Embed-1B	$0.055 /$ 0.110	$0.105 /$ 0.209
Llama-Nemotron-Embed-1B-v2	$0.059 /$ 0.118	$0.114 /$ 0.229
OpenAI text-embedding-3-large	$0.058 /$ 0.116	$0.108 /$ 0.217
Cohere Embed v4	$0.056 /$ 0.113	$0.107 /$ 0.215

Table 5: Answering cost at k=5 and k=10, all 1,000 queries, gpt-4o-mini as the LLM, at batch and standard API rates. Twelve rows, not thirteen: NV-Embed-v2 is absent because answering cost depends on the retrieved passages’ token length, and the anchor’s answering run was not repeated after the index correction. Its cost would fall inside the range shown: the twelve span

0.053–

0.060 at k=5, a spread of $0.007 per 1,000 queries, and the spread, not the cost, is the small number here. We do not report a figure we did not measure.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Embedder	Access	Notes
NV-Embed-v2 [ 15 ]	Self-hosted	Non-commercial research anchor, measured on this harness, not taken from the literature (§ 4.2 )
Nemotron-3-Embed-8B-BF16 [ 16 ]	Self-hosted	OpenMDW-1.1, “ready for commercial use.” Released 2026-07-16; not served by NVIDIA’s hosted NIM endpoint
Nemotron-3-Embed-1B-BF16 [ 17 ]	NIM API (served id nemotron-3-embed-1b )	Cheaper Nemotron tier
Llama-Nemotron-Embed-1B-v2	NIM API	Cheaper Nemotron tier
nv-embedqa-e5-v5 [ 18 ]	NIM API	335M params, fine-tuned E5-Large-Unsupervised, trained on NVIDIA’s own commercially-viable QA data
text-embedding-3-small / -large [ 20 ]	Direct API	—

Appendix

Table 6: Panel composition: access channel and per-model notes. Maker and license for each entrant are in Table 1 .

Figure 4: What an embedder does (a), and how cosine similarity turns embedded vectors into a ranked retrieval result (b).

Matter	Files	Extracted text	$\sim$ Tokens
Gaming strategic acquisition	3,391	225.4 MB	56.3M
Restaurant PE buyout	3,378	217.8 MB	54.5M
Two matters	6,769	443.2 MB	110.8M

Appendix

Table 7: Extracted text volume of two single legal-diligence matters, measured from the public LAB corpus. Token counts use a 4-characters-per-token approximation (extracted-text sizes in this appendix are decimal MB, the convention its printed token counts imply; reading them as binary shifts every figure by

\approx

5% and the matched-scale factor’s 7.5x lower bound to

\approx

7.2x).

Volume	Embedding	Indexing
One matter	$1–$ 8	$64–$ 997
23 matters ( $\approx$ 5GB)	$26–$ 194	$1,460–$ 22,900
100 matters/year	$113–$ 845	$6,350–$ 99,700
4,650 matters (1TB)	$5.2K–$ 39.3K	$296K–$ 4.64M

Appendix

Table 8: The same corpus priced two ways, per re-index. Embedding ranges across the four API providers of Table 4 and is

0 for every self-hostable model; indexing ranges across the six disclosed or reconstructed rates of Table 3 . Like Table 3 these are linear extrapolations, and for graph-construction systems that is an optimistic

O(N)$ assumption: if entity extraction and community detection scale super-linearly with graph density, the true figures are higher, and disproportionately so in the bottom row.

Embedder	title+text	$\Delta$	text-only
Gemini embedding-001	67.24	$+0.00$	65.78
Llama-Nemotron-Embed-1B-v2	63.73	$+0.00$	59.81
Nemotron-3-Embed-1B	64.32	$-0.00$	60.86
mxbai-embed-large-v1	55.71	$+0.01$	—
Qwen3-VL-Embedding-8B	59.88	$-0.02$	—
BGE-M3	54.93	$+0.06$	—

Appendix

Table 9: Recall@5 on the matched title\ntext corpus and on a body-text-only corpus.

\Delta

is run-to-run variation on the matched corpus. All thirteen entrants reproduce within 0.6 points and were therefore indexed correctly: twelve against their own original run, the anchor against a dedicated replicate, for the reason given in the text;

\Delta

is signed re-measurement minus original, so text-embedding-3-large ’s

-0.54

means its original run read 60.02 and the re-measurement 59.48, still far nearer its title\ntext figure than its 58.69 text-only one, which is the comparison that establishes it was indexed correctly. NV-Embed-v2’s original index was the text-only one, so it had no matched-corpus predecessor; its

\Delta

comes instead from a dedicated replicate run (2026-08-08): the full pipeline re-executed end-to-end with the corpus re-embedded from raw text into a fresh cache, same configuration. The two runs agree on every one of the 1,000 per-question scores, so

\Delta=+0.00

exactly: deterministic local inference, in line with the other locally-hosted entrants, whose deltas span

-0.02

+0.10

. The

2.46

separating its two corpus formats is a format penalty and appears below, not in this column. Where a text-only column is blank the model was measured against its already-cached corpus and not re-embedded in both formats. For NV-Embed-v2 both columns report the web-search query instruction, which is also the best of the four swept instructions on the text-only index (67.09 vs 67.07/66.64/64.07), so the row differences the same configuration and the penalty is not inflated by comparing a swept maximum to an unswept point; mixing instructions across the two columns would make the row’s own subtraction meaningless.

Embedder	R@10 $-$ R@5 lift	95% CI
text-embedding-3-large	$+10.67$	$[+9.48,+11.87]$
voyage-3.5	$+9.60$	$[+8.44,+10.79]$
text-embedding-3-small	$+9.40$	$[+8.29,+10.53]$
Gemini embedding-001	$+9.11$	$[+8.03,+10.22]$
Cohere Embed v4	$+8.88$	$[+7.81,+9.97]$
Qwen3-VL-Embedding-8B	$+8.65$	$[+7.56,+9.78]$

Appendix

Table 10: The paired intervals behind every lift and format-penalty claim in § 4.2 and this appendix: Recall@10

-

Recall@5 lift for all thirteen entrants (top block) and the corpus-format penalty for the ten measured in both formats (bottom block). Paired bootstrap over the same 1,000 questions,

B=10^{5}

, seed 42; produced by scripts/panel_intervals.py and scripts/panel_simultaneous_ci.py from the released per-question vectors; the simultaneous block prints all twelve Dunnett-type intervals instead of asserting their verdict. † interval includes zero: no penalty is claimed for this entrant. Two apparent oddities are verified, not errors: NV-Embed-v2 and nv-embedqa-e5-v5 print identical lift rows because their mean lifts are exactly equal (8.575) and these intervals live on a coarse question-mean grid — the underlying per-question vectors differ on 311 questions, their standard errors differ (0.545 vs 0.541, panel_derived_stats.json ), and their format-penalty rows differ; the endpoints coincide at full stored precision and not merely at two-decimal display, the grid being coarse enough for two distinct vectors to land on identical percentiles; and two penalties differ by

\pm 0.01

from subtracting Table 9 ’s rounded cells (cell rounding; the artifact carries unrounded values). These are secondary analyses, reported uncorrected (Table 1 caption).

Dense Retrieval

Dense retrieval uses learned vector representations to find relevant documents, passages, products, or tools by semantic similarity.

See all 153 papers

Jul 26, 2026

Choosing a Text Embedding Model: A Practical Benchmarking and Decision Framework

Choosing the right text embedding model is one of the most consequential -- and most frequently under-examined -- decisions in building a retrieval or search system, yet the model that tops a leaderboard is rarely the best choice for a given deployment. This report develops a practical, evidence-based framework for embedding model selection, built on a benchmarking study that evaluates T3EM (Text 3 Embedding Model), a commercial API-based embedding model, against a broad set of open-source alternatives on English-language retrieval tasks, and situates these findings within the wider Massive Text Embedding Benchmark (MTEB) landscape spanning classification, clustering, semantic similarity, reranking, pair classification, bitext mining, and summarization. Beyond raw benchmark scores, the report traces the full path from embedding model to retrieved result -- how embeddings are produced, how they are indexed and searched at scale, and how document chunking strategy shapes retrieval quality -- so that model choice can be reasoned about as one decision within a complete retrieval pipeline rather than in isolation. The result is a consolidated set of practical recommendations for selecting an embedding model according to task, latency, cost, and deployment constraints.

Madhav S Baidya

Indian Institute of Technology (BHU) Varanasi

Model Selection Text Embedding Evaluation Text Embedding Models IR Evaluation

Ask PDF

Jul 13, 2026

Cost-Governed RAG: Unified Per-Tenant Cost Attribution Across Retrieval and Generation in Multi-Tenant LLM Systems

Enterprise Retrieval-Augmented Generation (RAG) deployments face a critical governance gap: while LLM generation cost is metered per token, the retrieval layer - vector memory, similarity compute, and embedding API calls - remains an unattributed shared cost, enabling invisible cross-subsidization among tenants. We present Cost-Governed RAG, an architecture that integrates a codebook-oblivious vector index (TurboVec) with a multi-tenant LLM governance gateway, creating a unified observability stack where embedding, retrieval, and generation costs are jointly attributable per tenant. The architecture exploits TurboVec's deterministic, closed-form memory formula to enable near-exact per-tenant retrieval cost calculation - a property unavailable in graph-based indexes with non-linear memory overhead. Deployed on Snowpark Container Services within a cloud data platform's governance boundary, the system achieves 99.96% end-to-end cost attribution accuracy across 100 simulated tenants (10M vectors, log-normal size distribution) with telemetry overhead below 0.04% of query latency. The architecture reduces retrieval infrastructure cost by 3.1-9.0x compared to managed vector database services under the pricing assumptions detailed in Section IV. We formalize a three-layer cost model and demonstrate that codebook-oblivious quantization enables deterministic per-tenant cost attribution while also removing the shared-codebook leakage surface present in trained quantizers - the latter observation being exploratory and subject to the limitations described in Section VII.

Navnit Shukla

AI Platform Engineering Snowflake Inc. Tustin, California, USA

Retrieval-Augmented Generation Cost-Aware Inference

Ask PDF

May 22, 2026

Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems

We benchmark Google Embeddings (GE2), a Vertex-AI-hosted bi-encoder with 2,048-token context and explicit task-type conditioning, against five open-source alternatives: BGE-M3, E5-large, Multilingual-E5-large (mE5-L), LaBSE, and Paraphrase-Multilingual-MPNet (mMPNet). Evaluation covers four BEIR subsets, a synthetic Italian RAG corpus, a chunking ablation considering 5 sizes of tokens with three strategies, and per-query latency on commodity CPU hardware. GE2 ranks first on every task, achieving BEIR avg.nDCG@10 = 0.638 and IT-RAG-Bench nDCG@10 = 0.282, but at 231.6 ms median latency, it is roughly 14x slower than the fastest local models. mE5-L reaches within 0.003 nDCG of GE2 on Italian at 31 ms, making it the preferred option when sub-100 ms SLAs matter. A more striking finding concerns LaBSE, which, despite widespread multilingual deployment scores 0.188 average nDCG@10 on BEIR, below every dedicated retrieval model including mMPNet. Chunking experiments show that all six models saturate at 32-token chunks on our corpus, with semantic chunking providing measurable gains only at 16 tokens.

Stefano Cirillo, Domenico Desiato, Giuseppe Polese +1

University of Salerno · Fisciano, Salerno, Italy · University of Bari +1

Retrieval-Augmented Generation Text Embedding Evaluation Multilingual IR Text Embedding Models

Ask PDF

Abstract

Dense Retrieval

Explore similar work

Choosing a Text Embedding Model: A Practical Benchmarking and Decision Framework

Cost-Governed RAG: Unified Per-Tenant Cost Attribution Across Retrieval and Generation in Multi-Tenant LLM Systems

Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems