Semantic query engines ask a large language model (LLM) to judge individual rows, and benchmarks compare these engines on the quality of a few executions at temperature 0. We measure how much another execution changes. We re-execute the text and tabular workloads of SemBench, LRO-Bench, and UDA-Bench through their own harnesses, log every one of 1.2 million model calls, and reanalyze SemBench's published executions. At temperature 0, between 0.07% and 10.2% of rows change their operator output over five executions, depending on the model, and up to 24.0% on one task. Plans amplify these changes exponentially in the number of rows an answer exposes, as a possible-worlds model predicts: on UDA-Bench, a few percent of extracted values change, but about two thirds of the answers do. In SemBench's published results, 21% of text results change quality between executions, including on engines whose plans are deterministic. Several per-query system comparisons, and 16 of 28 tests of LRO-Bench's implementation recommendations, are not settled by the executions the benchmarks run. Changing rows are rows on which models disagree with each other; identical requests within one execution disagree as often as across executions; and one execution's decision margin or reasoning length predicts which rows change. Voting reduces the variance but does not remove it, and a calibrated cascade keeps its guarantee while the rows it returns change. We derive reporting rules for benchmarks and design rules for engines.
Figures & tables
Figure 1. The count that Movie Q3 returns in each execution at temperature 0, per deployment. Marker labels give the number of executions that returned the count; grey deployments returned one count in every execution. Dot plot of the counts returned by repeated executions of one query on six model deployments, with the gold count marked.
Plan operator
Items deciding the answer
Within
Full output (map, set, join)
all n items
C
23/24
LIMIT k
prefix up to the k -th match
S
22/24
COUNT , ratio, counts
spread of the count
S
18/18
AVG
spread of the mean
S
6/6
Rank by group mean
groups near a tie
S
6/6
SQL over extractions
values the SQL reads
C
202/369
Table 1. Which items decide the answer under each plan operator, and how often the possible-worlds prediction is within 0.15 of measured reproduction: the closed form e−x (C) or the simulation through the plan (S). SemBench rows count query–deployment cells; the last row counts UDA-Bench query–deployment pairs.
Query
Output
Answer
Items
Movie Q1
bool
first 5 positive reviews
2,000
Movie Q2
bool
first 5 qualifying reviews
120
Movie Q3
bool
count of positive reviews
120
Movie Q4
bool
share of positive reviews
120
Movie Q8
label
reviews per sentiment
120
Movie Q9
score
score per review
256
Table 2. SemBench queries executed repeatedly. Items is the number of semantic-operator outputs analysed per execution; for the joins, on a 40-review sample, it excludes each review paired with itself.
Model
Type
Stack
Logprobs
S
L
U
gpt-oss-120B
reasoning
M
no
10
5
5
Llama-3.3-70B-Instruct
instruct
M
no
5
5
5
Llama-3.3-70B-Instruct
instruct
V
yes
5
5
–
Qwen2.5-72B-Instruct
instruct
V
yes
5
–
–
Mistral-Small-3.1-24B
instruct
M
no
5
–
–
Granite-3.3-8B-Instruct
instruct
V
yes
5
5
5
Table 3. Model deployments. M is a managed inference service, V a vLLM deployment, and C a commercial API. The last columns give executions per query on SemBench (S, including joins), LRO-Bench (L), and UDA-Bench (U); later executions of some deployments are used only where stated.
System
Cells
Moves
Range ≥0.1
Max range
Best − mean
LOTUS
49
24%
6%
0.41
0.010
Palimpzest
64
16%
5%
0.30
0.007
ThalamusDB
47
19%
9%
0.97
0.016
BigQuery
64
27%
8%
1.00
0.035
Text queries
101
21%
5%
0.44
0.007
All queries
224
21%
7%
1.00
0.017
Table 4. SemBench’s published repeats: five executions of each query with Gemini 2.5 Flash, at the scale factor SemBench uses for its repeat tables. A cell is one system on one query. Moves is the share of cells whose quality differs across the five runs. Range is the largest difference between two runs of a cell. Best − mean is how far the luckiest run sits above the published mean.
Figure 2. Left: share of SemBench’s published text cells whose quality range over five executions exceeds x . Right: share of pairs of systems on a query whose mean qualities differ and whose comparison needs more than n executions per system to be separated at 95%; dotted line at SemBench’s five. Left: step curves of the share of cells above each quality range, per system. Right: step curves of the share of system pairs needing more than n executions, for SemBench's systems and our deployments.
gpt-oss-120B
Llama-70B (M)
Llama-70B (V)
Qwen-72B
Mistral-24B
Granite-8B
Query
F
R
Δ Q
F
R
Δ Q
F
R
Δ Q
F
R
Δ Q
F
R
Δ Q
F
R
Δ Q
Movie Q1 filter, LIMIT 5
4.2
.53
.00
0.1
1
.00
0.1
1
.00
0.1
1
.00
0.1
1
.00
3.2
1
.00
Movie Q2 filter, LIMIT 5
4.4
.62
.20
0.0
1
.00
0.0
1
.00
0.0
1
.00
0.0
1
.00
2.5
1
.00
Movie Q3 filter, COUNT
2.8
.36
.29
0.0
1
.00
0.0
1
.00
0.0
1
.00
0.0
1
.00
2.5
.30
.14
Movie Q4 filter, ratio
3.1
.27
.21
0.0
1
.00
0.0
1
.00
0.0
1
.00
0.0
1
.00
5.0
.10
.21
Movie Q8 label, GROUP BY
3.3
.29
.08
0.0
1
.00
0.0
1
.00
0.0
1
.00
0.0
1
.00
1.7
.40
.04
Table 5. Repeated SemBench executions at temperature 0. F: item flip rate (%), the share of items whose output is not identical in five executions (for gpt-oss-120B, the expected rate over five of its ten executions). R: answer reproduction, the share of pairs of executions that return an identical answer (with five executions, ten pairs, so R moves in steps of 0.1; for gpt-oss-120B, 45 pairs of ten). Δ Q: range of SemBench’s quality metric across executions. Shaded: the answer changes but the quality does not.
Output
GPT-4o
Claude Sonnet 5
Open-weight
Boolean filters
0.00
2.22
0.00–3.5
Two-way label
0.83
0.00
0.00–3.3
1–5 scores
9.04
4.65
0.13–17.5
Pooled
7.49
4.13
0.11–15.1
Table 6. Item flip rate over five executions, in percent, of the commercial models on the SemBench filters and maps, by output type, next to the lowest and highest rate among the six open-weight deployments. Movie Q1, Medical Q1, Q4, and Q10 are excluded for all deployments: the content filter of GPT-4o rejected every attempt.
Figure 3. Item flip rate over five executions by the output type of the operator, pooled over SemBench queries (log scale; zero shown at 0.01%). Grouped bar chart of item flip rates per model for Boolean, label, score, and extraction outputs.
Figure 4. Left: item flip rate of gpt-oss-120B against the number of executions observed, by output type (exact expectation over subsets of the ten executions). Right: answer reproduction against the expected number of items on which two executions disagree, one point per SemBench query (joins included) and deployment. Filled points are answers that expose every item, for which the possible-worlds model predicts R≈e−x ; hollow points are answers under LIMIT , an aggregate, or a ranking. Left: line plot of flip rate rising with the number of executions for four output types. Right: scatter of reproduction against expected disagreements on a log axis, with an exponential decay curve.
gpt-oss-120B (M)
Llama-70B (V)
Llama-70B (M)
Granite-8B (V)
Operator
Impl.
Fail
Differ
Range
Fail
Differ
Range
Fail
Differ
Range
Fail
Differ
Range
Select
ALL
0
21/60
0.056
0
7/60
0.008
27
7/33
0.049
12
14/48
0.033
ONE
1
25/60
0.055
6
9/55
0.008
35
2/27
0.002
13
5/47
0.017
Match
ALL
0
30/60
0.145
3
9/59
0.021
41
11/26
0.089
17
9/43
0.050
ONE
9
33/55
0.074
20
19/40
0.013
58
2/3
0.077
12
29/49
0.023
SEMI
5
39/57
0.106
0
14/60
0.010
15
28/46
0.057
3
34/58
0.016
Table 7. LRO-Bench, five executions of every scored query in each implementation. Per deployment: queries with a failed execution, answered queries whose answer differs between executions, and the mean range of LRO-Bench’s quality metric over answered queries. ALL issues one call for the operand, ONE one call per element, SEMI a hybrid.
Q5
Q6
Q7
Deployment
Flip
Repro
Flip
Repro
Flip
Repro
F1 range
Mistral-24B (M)
0.1
1.00
0.0
1.00
0.0
1.00
0.000
Qwen-72B (V)
0.1
1.00
0.2
1.00
0.3
0.10
0.002
Llama-70B (V)
0.3
1.00
0.2
1.00
0.2
0.20
0.001
Llama-70B (M)
2.7
0.10
1.4
0.30
1.5
0.00
0.009
gpt-oss-120B (M)
13.5
0.00
15.1
0.00
15.2
0.00
0.019
Table 8. Exact semantic self-joins on 40 reviews (1,560 pairs of distinct reviews). Flip is the share of pairs whose verdict changes across executions (%); Repro is answer reproduction. Q5 and Q6 return the first ten same- and opposite-sentiment pairs, Q7 all opposite-sentiment pairs. Flip rates are over five executions; for gpt-oss-120B, the expected rate over five of its ten.
Answers differ
Deployment
Values
Sel.
Filt.
Agg.
Join
Mix.
All
F1
Range
Queries
8
60
10
13
32
123
gpt-oss-120B (M)
7.5
6
44
8
13
13
84
0.44
0.031
Llama-70B (M)
4.0
8
40
10
10
15
83
0.51
0.011
Granite-8B (V)
5.2
8
48
9
6
13
84
0.38
0.021
Table 9. UDA-Bench Player, five executions of 123 queries. Values is the share of extracted (document, attribute) values that change (%). Per query class, the number of queries whose result table differs between executions. F1 is UDA-Bench’s quality, mean over queries with a non-empty ground truth, with the mean range across executions.
Recommendation
gpt-oss
Llama (M)
Llama (V)
Granite
Filter, rows: ONE > ALL
∼
–
∼
✓
Filter, columns, tables: ALL > ONE
✓
✓
✓
✓
Match, rows: SEMI ≥ ALL
∼
–
∼
∼
Match, cells, columns: ALL > SEMI
✓
∼
∼
✓
Impute, columns: ONE > ALL
∼
–
∼
∼
Impute, cells, rows: ALL > ONE
✓
–
✓
∼
Table 10. LRO-Bench’s implementation recommendations on our executions. ✓ : the 95% interval of the mean quality difference over queries is above zero and every execution agrees; ∼ : the interval contains zero; –: fewer than three queries.
Figure 5. Share of queries whose operator outputs (Out.), final answer (Ans.), and benchmark quality (Qual.) are identical over five executions, per deployment. All levels count the same queries, those with a quality score: on UDA-Bench, 101 of 123; on LRO-Bench, those answered in at least two executions. LRO-Bench compares raw replies; UDA-Bench compares the extracted values each query reads. Slope chart with three panels, one per benchmark; each line is a deployment rising from outputs to answer to quality.
Figure 6. Left: share of later executions whose output differs from the first, by the first execution’s decision margin (log scale), with the fitted noise model (dotted), for deployments with log-probabilities and at least ten changing items. Right: item flip rate over five executions against the number of other deployments whose majority output differs from the plurality of the others; with five other deployments, at most three can dissent from a plurality (bins of at least 20 items; zero shown at 0.01%). Left: observed share of differing later outputs falling with the decision margin, with fitted curves. Right: flip rates rising with cross-model dissent.
Movie Q3
Movie Q8
Movie Q9
Pooled
Setting
Disagree
Moved
Disagree
Moved
Disagree
Moved
Δ disagree [95% CI]
Granite-8B: Concurrency 1
1.65 (10)
0.0
0.17 (10)
0.8
1.68 (10)
0.4
-0.23 [-0.66, +0.18]
Granite-8B: Concurrency 8
1.74 (10)
0.8
0.46 (10)
0.8
1.55 (10)
0.4
-0.20 [-0.65, +0.21]
Granite-8B: Concurrency 20 (ref.)
2.56 (10)
–
0.61 (10)
–
1.48 (10)
–
–
Granite-8B: Concurrency 64
1.85 (10)
0.0
0.17 (10)
0.8
1.52 (10)
0.0
-0.26 [-0.62, +0.07]
Granite-8B: Shuffled input order
2.65 (10)
0.8
0.46 (10)
0.8
1.40 (10)
0.4
-0.06 [-0.38, +0.26]
Table 11. Cause probes on Movie Q3, Q8, and Q9. Each setting changes one factor. Disagreement is the pairwise item disagreement within the setting (%); moved is the share of items whose majority output differs from the reference setting of the same model (%); repeats in parentheses.
Query
One
Vote 3
Vote 5
Cost of 3
Minority ≥3
Movie Q1
2.1
1.5
1.1
2.02
46%
Movie Q2
2.1
1.1
0.8
2.02
29%
Movie Q3
1.2
0.6
0.8
2.02
17%
Movie Q4
1.4
0.6
0.8
2.02
17%
Movie Q8
1.6
1.1
0.0
2.01
40%
Movie Q9
9.9
5.7
5.5
2.10
51%
Table 12. Repetition replayed on ten executions of gpt-oss-120B. Disagreement is the probability that two combined executions differ on an item (%). Vote of 3 costs 2+p calls per item when the third call is made only where the first two disagree. Ten executions form only two groups of five, so Vote 5 rests on a single pair and can exceed Vote 3. Minority ≥3 is the share of changing items whose less common output occurs in at least three of the ten executions, so that a vote of three often still returns it.
Violation
Query
Oracle
Dis.
Rows
Same
Other
Repro
Medical Q1
gpt-oss-120B
2.21
321
1.0
2.2
24
Medical Q1
Llama-70B (M)
0.18
350
1.1
1.1
84
Medical Q4
gpt-oss-120B
0.39
1065
0.1
0.2
94
Medical Q4
Llama-70B (M)
0.03
836
0.9
0.9
98
Table 13. Proxy cascades calibrated for 90% recall with 95% confidence on 400 oracle-labelled rows (300 samples), with Granite-8B as proxy. Dis.: pairwise disagreement of the oracle’s executions. Rows: mean rows returned. Violation: share of calibrations whose recall falls below 90% against the oracle execution that labelled the sample (same) or another one (other). Repro: for a fixed sample, the share of pairs of oracle executions whose calibrations return the same rows. All rates in %.
Semantic query engines evaluate natural-language filters with a large language model, and every evaluation is a costly oracle call. Optimizers that reduce these calls face a dilemma: cascades that trust a cheap proxy miss their precision and recall targets on many queries, methods with finite-sample guarantees save little, and all of them discard the oracle's labels when the query ends. We present SemViews, a certified semantic filter that orders tuples by signals that cost no oracle calls (the labels that earlier queries left behind as semantic views, a local embedding, and a helper model's score), refines that order with labels it must pay for anyway, and certifies its answer with fresh stratified oracle samples. The answer meets user precision and recall targets with probability at least 1−δ for every workload and every catalog, so a stale or misleading view can cost oracle calls but never accuracy. On four workloads replayed against 725,000 recorded oracle labels, SemViews meets both targets on 99.63% of queries and saves 28%-45% of the oracle calls, whereas the only prior method in our study with the same guarantee needs 1.32x-1.71x as many calls. When a question returns in different words, views cut its calls by a further 27%-36%, with both targets met on at least 99.8% of such queries; similarity caches and entailment-based reuse meet them on only 17%-47%.
LLMs deployed for natural-language querying of analytical databases suffer from two intertwined failures - incorrect answers and confident hallucinations - both rooted in the same cause: the model is forced to infer business semantics that the schema does not encode. We test whether supplying those semantics as context closes the gap. We benchmark three frontier LLMs (Claude Opus 4.7, Claude Sonnet 4.6, GPT-5.4) on 100 natural-language questions over the Cleaned Contoso Retail Dataset in ClickHouse, using a paired single-shot protocol. Each model is evaluated twice: once given only the warehouse schema, and once given the schema plus a 4 KB hand-authored markdown document describing the dataset's measures, conventions, and disambiguation rules. Adding the document improves accuracy by +17 to +23 percentage points across all three models. With it, the three models are statistically indistinguishable (67.7-68.7%); without it, they are also indistinguishable (45.5-50.5%). Every cross-cluster comparison is significant at p < 0.01. The presence of the semantic-layer document accounts for essentially all of the significant variance; model choice within tier does not. We interpret this as a structural result: explicit business semantics suppress the dominant class of text-to-SQL errors not by making the model more capable, but by changing what the model is being asked to do.
LLM analytics agents are evaluated on SQL syntax accuracy, but production failures look different: questions with two valid business definitions, questions the warehouse cannot answer, deprecated columns after a schema change, and queries that execute successfully while returning the wrong business number. No execution-match metric can score them. This paper introduces WarehouseReliabilityBench, 400 frozen tasks over two synthetic warehouses in which roughly half the correct responses are a clarification, an abstention or a refusal, with pinned denominators and a pre-registered paired bootstrap fixing each claim verb before the numbers existed. QueryProof, a 7B agent, uses rules derived from a semantic layer and physical catalog to determine its behaviour, and gates every answer on deterministic post-execution checks. On an 80-task synthetic test split evaluated once, QueryProof outperforms a direct-prompted 32B baseline by +0.237 [+0.112, +0.375] Business Truth Rate at 71.0% lower cost per correct answer; against a cost-matched few-shot baseline the accuracy gain holds but the cost difference does not resolve. This compares systems rather than model sizes: the 32B baseline receives none of the scaffolding. False success falls from 0.754 to 0.351 of returned answers, and no wrong number was returned on an answerable task (0 of 24), though 13 answers went to questions requiring clarification or abstention. Removing the routing layer changes little (0.562 against 0.537), so the result does not depend on escalation. Routing tuned on validation over-abstains on test, and the fitted confidence model loses to the heuristic it replaced. Resampling template families rather than tasks widens both accuracy intervals to include zero, so the effect's direction is better supported than its magnitude. The gain tracks the deterministic layer, though no component ablation was run.