100x Cost & Latency Reduction: Performance Analysis of AI Query Approximation using Lightweight Proxy Models

Yeounoh Chung; Rushabh Desai; Jian He; Yu Xiao; Thibaud Hottelier; Yves-Laurent Kom Samo; Pushkar Khadilkar; Xianshun Chen

cs.DBMar 16, 2026

100x Cost & Latency Reduction: Performance Analysis of AI Query Approximation using Lightweight Proxy Models

Authors: Yeounoh Chung, Rushabh Desai, Jian He, Yu Xiao, Thibaud Hottelier, Yves-Laurent Kom Samo, Pushkar Khadilkar, Xianshun Chen,

Organizations: Google Cloud Sunnyvale, USA

Abstract

Several data warehouse and database providers have recently introduced extensions to SQL called AI Queries, enabling users to specify functions and conditions in SQL that are evaluated by LLMs, thereby broadening significantly the kinds of queries one can express over the combination of structured and unstructured data. LLMs offer remarkable semantic reasoning capabilities, making them an essential tool for complex and nuanced queries that blend structured and unstructured data. While extremely powerful, these AI queries can become prohibitively costly when invoked thousands of times. This paper provides an extensive evaluation of a recent AI query approximation approach that enables low cost analytics and database applications to benefit from AI queries. The approach delivers >100x cost and latency reduction for the semantic filter operator and also important gains for semantic ranking. The cost and performance gains come from utilizing cheap and accurate proxy models over embedding vectors. We show that despite the massive gains in latency and cost, these proxy models preserve accuracy and occasionally improve accuracy across various benchmark datasets, including the extended Amazon reviews benchmark that has 10M rows. We present an OLAP-friendly architecture within Google BigQuery for this approach for purely online (ad hoc) queries, and a low-latency HTAP database-friendly architecture in AlloyDB that could further improve the latency by moving the proxy model training offline. We present techniques that accelerate the proxy model training.

Figures & tables

Approach	Latency Improvement	Cost Savings
Online Proxy Model	329x	728x
Offline Proxy Model	991x	792x

Table 1. Latency and cost gains of lightweight proxy models on the query in Example 1.1 using a 10-million row table.

Metric / Data Size		Offline Proxy	Online Proxy	LLM
Accuracy	1115	0.99	0.98	0.96
Latency $\uparrow$	1115	634x	5.4x	1x
Latency $\uparrow$	100K	632x	294x	1x

Table 2. Proxy models (Embedding + Logistic Regression) VS. LLM-based baseline performance and normalized latency improvement against the LLM baseline for AI.IF using Kaggle SPAM email classification dataset.

Figure 1. AI query execution plan construction with proxy model approximation process. We parse the AI query and extract semantic operators (

O_{i}

) along with the semantic queries/prompts (

Q_{i}

), unstructured data columns (

C_{i}

). We apply proxy model approximation for each operator. Proxy models can be trained online with best known or default configurations, or prepared offline for known query patterns depending on the analytics database architectures (OLAP or HTAP). Adaptive proxy model selection between the proxy and LLM, based on automatic online model evaluation results, allows us to choose a cost-efficient and accurate execution strategy.

Dataset	Task	Domain	Relevancy	Table (rows)	Imbalance Ratio 1	Example Prompt
California Housing ( Pace and Barry, 1997 )	Text	Housing	Binary	20K	6.71	Location in Latitude & Longitude is near the ocean.
Amazon Reviews 10k ( Zhang et al., 2015 )	Text	eCommerce	Binary	10K	4.69	Reviews describe at least one issue with the context of the problem.
BBC News ( Greene and Cunningham, 2006 )	Topic	News	5 categories	2.2K	1.32	Classify new article into following categories (business / entertainment / politics / sport / tech)
IMDB Movie Reviews ( Maas et al., 2011 )	Sentiment	Movies	Binary	99K	1.10	Movie review is positive.
Amazon Polarity ( Zhang et al., 2015 )	Sentiment	eCommerce	Binary	400K	1.00	Review is positive.
Mental Health ( Ji et al., 2022 )	Sentiment	Health	Binary	51.6K	3.41	Mental health is anxiety.

Table 3. Classification benchmark datasets for AI.IF. For the evaluation of the semantic filter ( AI.IF ), datasets with multiple categories/labels ( Relevancy > Binary ) are used to pose a one-vs-rest semantic query (e.g., matching one specific category against all others).

Dataset	Task	Domain	Relevancy	Queries	Documents	Relevant Docs/Query
TREC-COVID ( Roberts et al., 2021 )	Bio-medical IR	Bio-medical	3-level	50	171K	493.5
TREC-DL-2022 ( Craswell et al., 2022 )	Ranking/IR	Misc.	4-level	500	369K	189.3
FIQA-2018 ( Maia et al., 2018 )	Bio-medical IR	Bio-medical	Binary	648	57K	2.6
SCIDOCS ( Cohan et al., 2020 )	Argument retrieval	Misc.	Binary	1000	25K	4.9
SciFact ( Wadden et al., 2020 )	Fact checking	Scientific	Binary	300	5K	1.1
HellaSwag 1 ( Zellers et al., 2019 )	Commonsense reasoning	Daily life	Binary	200	800	1.0

Table 4. Information Retrieval (IR) benchmark datasets for AI.RANK. These datasets are designed to test the performance of relevant document retrieval and are typically characterized by heavy class imbalance, where only a handful of relevant documents exist per query within corpora that can contain hundreds of thousands or even millions of documents.

Dataset	Labels	Per-label Prompt (Binary)	Macro F1 Proxy	Macro F1 LLM	Relative Acc. 1
California Housing	$1$	Location in Latitude & Longitude belongs to Southern California.	0.953	0.953	1.000
California Housing	$1$	Location in Latitude & Longitude is near the ocean.	0.344	0.354	0.971
Amazon Reviews 10k	$2$	Review is {sentiment label}	$0.860$	$0.739$	1.163
BBC News	$5$	News Content describes news in {news topic label} sector	0.830	0.823	1.008
IMDB Movie Reviews	$2$	Movie review is {sentiment label}	0.940	0.950	0.989
Amazon Polarity	$2$	Review is {sentiment label}	0.965	0.959	1.005

Table 5. Macro F1 and relative accuracy of the proxy vs. LLM baseline for semantic filtering. For multi-label datasets, results are averaged over one-vs-rest binary classification tasks.

	On-The-Fly		Pre-Computed
Table Size	Cost $\uparrow$	Latency $\uparrow$	Cost $\uparrow$	Latency $\uparrow$
10,000	2x	1.3x	9x	1.6x
100,000	2.5x	3x	81x	6x
1,000,000	2.5x	5x	422x	30x
10,000,000	2.5x	5x	728x	320x

Table 6. Cost and latency improvement of online proxy model approximation VS. the LLM baseline, with (On-the-Fly) and without (Pre-Computed) embeddings, tested using Amazon Polarity.

Table Size	Cost $\uparrow$	Latency $\uparrow$
10,000	9x	111x
100,000	81x
1,000,000	422x
10,000,000	728x

Table 7. Cost and latency improvement of offline proxy model approximation VS. the LLM baseline, tested using Amazon Polarity. Offline training includes the same processing (vCPU), labeling and embedding costs as in Table 6 per the same train sample size.

Figure 2. Relative wall-clock time of each step of the proxy model optimization with pre-computed embeddings. Only the sampling and prediction steps scale with the input relation size. The fraction of time spent doing prediction does not increase between 1m and 10m rows because BigQuery is able to use more parallelism as the amount of data increases.

Figure 3. Proxy model performance (nDCG@10 on online training sample) for rubric-based ranking on TREC-DL-2022. The online Proxy performs better after labeling >120 samples; Adaptive Proxy relies on LLM until the Proxy becomes as good as the baseline.

	Re-Ranker	LLM	Proxy
FIQA-2018	0.432 ( $\pm$ 0.363)	0.0703 ( $\pm$ 0.107)	0.0304 ( $\pm$ 0.027)
SciFact	0.746 ( $\pm$ 0.368)	0.508 ( $\pm$ 0.416)	0.0098 ( $\pm$ 0.009)
SCIDOCS	0.251 ( $\pm$ 0.267)	0.107 ( $\pm$ 0.151)	0.0185 ( $\pm$ 0.0469)
TREC-DL-2022	0.419 ( $\pm$ 0.245)	0.537 ( $\pm$ 0.164)	0.446 ( $\pm$ 0.241)
TREC-COVID	0.738 ( $\pm$ 0.232)	0.551 ( $\pm$ 0.167)	0.535 ( $\pm$ 0.215)
HellaSwag	0.617 ( $\pm$ 0.252)	0.247 ( $\pm$ 0.306)	0.134 ( $\pm$ 0.251)

Table 8. Semantic ranking performance (nDCG@10) on the IR benchmark datasets from Table 4 . We report the average nDCG@10 over all queries (

\pm

1-std). TREC-DL-2022 ranks items by 4-level scoring rubric, for which LLM and its Proxy perform better than Re-Ranker.

	Re-Ranker	LLM	Proxy
Cost	$0.025\times$	$2.5\times$	$1\times$
Latency	$0.11\times$	$1.1\times$	$1\times$

Table 9. Semantic ranking cost and latency comparison (1x for Proxy) for ranking 500 candidate documents. Proxy requires labeling 200 documents with LLM for training.

Panel of Figure 4 — Figure 4. Impact of sampling strategies on training data imbalance ratios, measured across various datasets of varying degrees of dataset characteristics: imbalance ratio ( $\rho$ ), for classification benchmark datasets (a)(b); Relevant docuemtns per query ( $\gamma$ ) / the number of corpus documents, for IR benchmark datasets (c)(d).

Random	Top-K	AL (Stratified)
$1\times$	$43761\times$	$51639\times$

Table 10. Latency overhead multipliers (1

\times

for Random) for sampling 1000 sample instances from the Toxic Conversations dataset with 52K rows.

Panel of Figure 5 — Figure 5. Impact of different imbalanced label training techniques, as described in Section 4.2 .

Balanced	Downsampled	Bootstrapped	Oversampled
$1.30\times$	$1.41\times$	$2.46\times$	$3.07\times$

Table 11. Latency overhead multipliers (1

\times

for imbalanced training) across different imbalanced training techniques, with simulated imbalanced ratio of 10.0

Figure 6. Impact of embedding model and dimensionality on proxy model classification performance. Note that Gecko ( text-embedding-005 ) and Gemma ( embeddinggemma-300m ( Google, 2025 ) ) support embedding dimension size up to 768, and Gemini ( gemini-embedding-001 ) supports up to 3072.

Model	$\mathbf{D_{\text{max}}}$	Latency (sec)	Cost 1 ($)	Size (MB)
Gemma	768	1x	1x	10.35
Gecko	768	1.81x	18.1x	10.35
Gemini	3072	4.18x	27.6x	41.41
1 Cost is computed based on Gemini embedding model pricing ( Google Cloud, 2025c ) for Gecko and Gemini, and the cost of running a high CPU VM (8 vCPU, 16 GB RAM) for Gemma.

Table 12. Embedding generation cost comparison for Tweet Sentiment test split (3534 rows)–this should increases linearly with the number of rows, since we are processing a fixed number of rows (20) per generation request.

Figure 7. Embedding distinctiveness illustrated by PCA visualization (X-axis is PC1, Y-axis is PC2) and separability scores across various benchmark datasets with Gemma and Gecko embedding models, dimension $D=768$

Metric	LR	SVM	RF	XGB
F1 Score (default)	0.867	0.861	0.841	0.858
F1 Score (tuned)	0.867	0.861	0.836	0.848
Training Latency	1x	49.84x	1.42x	3.18x

Table 13. Normalized training latency to the LR training latency (1x) and tuned model performance on Tweet Sentiment dataset.

Table 14. Proxy and LLM performance (F1-Score) comparison across data slices of the California Housing dataset. This comparison highlights the difference between training on a global sample versus training specifically on sliced data.

Dataset	LLM (P / R)	Approx (P / R) by Sample Size
		1,000	4,000	8,000
BBC News	0.94 / 0.94	0.95 / 0.95	—	—
DBpedia	0.98 / 0.98	0.83 / 0.65	0.94 / 0.95	0.96 / 0.96
BBC News has 5 categories, DBpedia ( The Devastator, 2022 ) has 14 categories.

Table 15. AI.CLASSIFY performance comparison: LLM vs. Proxy Approximation (Precision / Recall)

Cost-Aware Inference

Cost-aware inference allocates models, computation, and serving resources to meet quality goals at lower latency, energy use, and monetary cost.

See all 280 papers

Larch: Learned Query Optimization for Semantic Predicates
Jun 6, 2026Fuheng Zhao, Pawel Liskowski, Zihan Li +5Cost-Aware Inference
Snowflake Inc. USA · Snowflake Inc. Poland
Large Databases Need Small, Open-Weight Language Models
Jun 30, 2026Parker Glenn, Alfy SamuelSmall Language Models Open-Weight Language Models
Capital One
Business Truth, not SQL Accuracy: A Rule-Gated 7B Analytics Agent Outperforms a Direct-Prompted 32B Baseline
Aug 10, 2026Morris LeeLLM Answer Verification LLM Agent Evaluation
Independent Researcher

Jun 6, 2026J/K move · Enter open · S save

Larch: Learned Query Optimization for Semantic Predicates

Fuheng Zhao, Pawel Liskowski, Zihan Li, Benjamin Han+4

Snowflake Inc. USA · Snowflake Inc. Poland

Cost-Aware Inference

Open paper AskPDF arXiv

With the advent of Large Language Models (LLMs), many database systems introduced semantic operators that enabled analytical queries over unstructured data (e.g. text, images, videos). Semantic operators typically incur high inference costs and latencies making semantic (AI) SQL queries challenging to apply on large scale datasets. At the same time, their semantic nature leads database engines to treat them as black boxes, making AISQL queries difficult to optimize. In this paper, we introduce Larch, a framework for optimizing the execution of semantic filters in AI SQL queries. Larch was inspired by two key observations: i) the high latency of semantic operators leaves significant room for computationally-heavy runtime optimization techniques, ii) unstructured data are typically accompanied by semantic information in the form of embeddings allowing for efficient semantic comparisons between AI_FILTER prompts and data values. Based on these two key observations, we present two Larch variants: Larch-A2C and Larch-Sel. Larch-A2C encodes arbitrary semantic filters expression tree using an embedding-augmented Gated Graph Neural Network and formulates the filter evaluation order as a Markov decision process. In contrast, Larch-Sel leverages a supervised learning model to predict filter selectivities, subsequently applying dynamic programming to find a near-optimal evaluation order for each input row. Evaluated across diverse real-world datasets and comprehensive synthetic workloads, both Larch variants always outperform existing semantic filter optimization techniques in terms of token usage. Our results demonstrate that Larch is robust across diverse workloads, reducing total token cost overhead by 3x-19x compared to Palimpzest and Quest.

Abstract

Cost-Aware Inference

Explore similar work

Larch: Learned Query Optimization for Semantic Predicates

Large Databases Need Small, Open-Weight Language Models

Business Truth, not SQL Accuracy: A Rule-Gated 7B Analytics Agent Outperforms a Direct-Prompted 32B Baseline