AptMQL-Bench: From Text-to-SQL to Text-to-MQL via Access-Pattern Schema Design and Data-Preserving Migration

Hy Nguyen; Nabi Rezvani; Robin Vujanic

cs.CLOct 2, 2026

AptMQL-Bench: From Text-to-SQL to Text-to-MQL via Access-Pattern Schema Design and Data-Preserving Migration

Authors: Hy Nguyen, Nabi Rezvani, Robin Vujanic

Organizations: The University of Sydney · MongoDB Research

Abstract

Document databases such as MongoDB are core infrastructure for modern applications, and natural-language interfaces to them---text-to-MQL---would let non-experts query complex, semi-structured data without mastering the query language. Progress on this task depends on high-quality benchmarks, which are most practically obtained by converting an existing text-to-SQL benchmark to the document setting. Unfortunately, existing efforts rely on heuristics for mechanical conversion: the document schema mirrors the relational foreign-key graph, and each query mirrors its source SQL. As a result in our experiments, these approaches fail to migrate 6 of 21 BIRD databases outright, silently drop up to 25.9% of rows on others, and yield schemas whose ground-truth queries run over an order of magnitude slower as the data scales. We instead propose a conversion pipeline, driven by coding agents with human-in-the-loop verification, that designs each document schema from expected access patterns and rewrites queries to be MongoDB-native. Applying it to BIRD, we build an access-pattern-based text-to-MQL benchmark (AptMQL-Bench). It includes 21 document-oriented databases, 3,186 natural-language requests, and their associated MQL queries---whose databases are migrated from SQLite without data loss and scale efficiently. The strongest model, Claude Opus 4.5, achieves only 57.38% accuracy without external knowledge evidence and 70.34% with it. This indicates that realistic text-to-MQL generation remains challenging.

Figures & tables

Figure 1: Conversion pipeline from Text-to-SQL (left) to Text-to-MQL (right).

	Databases				NL Requests				MQLs
Benchmark	# DB	Design by	# Coll./DB	# R.Docs/DB	# NL	Length*	Diversity	Annot.	# MQL	Length*	Depth*	Annot.
NL-to-MongoSH	8	H	2.8	45k	766	89	0.52	L	766	2.81	4.01	L
DocSpider	159	P	5.1	10k	4,663	70	0.53	L+H	4,663	3.3	3.79	H
TEND-v2 †	154	P	2.2	1k	17k			L+H	17k			L+H
SM3-Text-to-Query	1	H	5	1.8k	8k	82	0.19	P+H	8k	4.54	3.18	P+H
TEND-v3	11	H	2.9	24k	1210	395	0.40	L+H	1210	7.28	8.14	L+H

Table 1: Statistics of AptMQL - Bench and related text-to-MQL benchmarks. The "Design by" and "Annot." columns indicate how each benchmark’s databases and its NL/MQL annotations were produced—by a P rogram using template synthesis, an L LM, a coding A gent, or a H uman (combinations joined by “+”). #Coll./DB and #R.Docs/DB are the average numbers of collections and root documents per database. For NL requests, Length is the average character count and Diversity the ratio of distinct to total 3-grams ( Li et al., 2016 ) under GPT-2 tokenisation ( Radford et al., 2019 ) ; for MQLs, Length is the average number of aggregation stages and Depth the average nesting depth. † TEND-v2 does not release NL requests and MQLs.

Models	Exec-Rate		Strict-EX		Soft-EX
	w/o knowledge	w/ knowledge	w/o knowledge	w/ knowledge	w/o knowledge	w/ knowledge
Llama-3.1-70B	80.57	82.80 (+2.23)	15.44	24.48 (+9.04)	24.92	37.07 (+12.15)
GPT-OSS-20B	81.36	85.88 (+4.52)	22.25	36.94 (+14.69)	32.49	51.00 (+18.51)
Qwen-3.5-35B	91.56	93.72 (+2.16)	20.37	31.83 (+11.46)	39.17	56.06 (+16.89)
Gemma-4-31B	99.00	99.09 (+0.09)	38.20	52.39 (+14.19)	46.74	61.96 (+15.22)
GPT-OSS-120B	96.80	97.83 (+1.03)	29.76	44.19 (+14.43)	44.51	62.46 (+17.95)

Table 2: Performance of open-source and proprietary LLMs.

Figure 3: Execution time of ground-truth MQLs versus database size (log–log), averaged across databases.

BIRD training databases
Database	#Tables	#Records	One-to-One	FK-Embedding	Access-Pattern
beer_factory	7	14.0k	✓	$\times$	✓
cs_semester	5	197	✓	✓	✓
food_inspection_2	5	701k	✓	$\times$	✓
hockey	22	96.1k	✓	9.60%	✓
mondial_geo	34	21.1k	✓	1.01%	✓

Table 3: Row-level data loss under three conversions: One-to-One ( DocSpider ), FK-Embedding ( TEND-v2 ), and Access-Pattern ( ours ). ✓ marks a lossless migration,

\times

marks a database that could not be migrated at all, and a number presents the percentage of source rows lost. The bottom three rows are databases used in the execution-time study in Figure 3 and are derived from the Spider benchmark Yu et al. (2018) ; the rest are the BIRD training and development databases of AptMQL - Bench.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Count	Typical Cause
Syntax Error	4	Fail to execute the MQLs due to syntax errors.
Acceptable Prediction	37	Semantically reasonable prediction.
		Gold uses arbitrary tie-breaks, benchmark-specific conventions, or SQL-style row preservation. Returns the answer at an incompatible row or document level.
Misunderstanding Knowledge Evidence	12	Fails to understand or apply the provided MongoDB evidence.
		Often ignores explicit collection paths, field mappings, value definitions, or aggregation instructions.
Misunderstanding Database Content	8	Misinterprets field meanings, categorical values, or stored data representations.

Appendix

Table 4: Error categories over 100 manually analysed Opus predictions with external knowledge.

Database	Conc.	Coll.	Tbl.	X-Coll.	Total
california_schools	15	3	5	2	25
card_games	13	4	22	3	42
codebase_community	16	5	9	7	37
debit_card_specializing	14	3	3	4	24
european_football_2	11	3	10	5	29
financial	30	3	4	5	42

Appendix

Table 5: Estimated reviewer effort (minutes per database) for schema design and validation over the eleven development databases: Conc. = conceptual design, Coll. = collection validation, Tbl. = table (source-anchored) validation, X-Coll. = cross-collection validation.

SQLite table	Disposition	Target collection(s)	Ret.
customers	Standalone	customers	✓
rootbeerbrand	Standalone	brands	✓
rootbeerreview	Standalone	reviews , brands	✓
transaction	Standalone	transactions	✓
location	Standalone	locations	✓
geolocation	Embedded	locations	✓

Appendix

Table 6: Conceptual-level data-loss check for beer_factory : every SQLite table maps to a destination in the proposed design ( standalone , embedded , or referenced ), so all source data remains retrievable. Column-level checking happens during migration validation (Section 3.2 ).

Source (SQLite)	Target (MongoDB)	Type	Transformation
rootbeerbrand.BrandID	brands._id	String	INTEGER $\rightarrow$ String
rootbeerbrand.BrandName	brands.brandName	String	Direct mapping
brands.rating		Document	Computed pattern (from rootbeerreview )
rootbeerreview.StarRating	brands.rating.average	Number	AVG(StarRating) grouped by BrandID ; null if no reviews
rootbeerreview.*	brands.rating.reviewCount	Number	COUNT(*) grouped by BrandID ; 0 if no reviews
brands.featuredReviews		Array	Subset pattern: top 3 by StarRating , then ReviewDate

Appendix

Table 7: SQLite-to-MongoDB logical design for the brands collection, covering the fields shown in Figure 2 .

Property of the 634 correct Soft-EX / incorrect Strict-EX cases	Value
Prediction has strictly more fields than gold	634 (100%)
Extra fields per case (median / mean / max)	1 / 10.1 / 1076
Gold result projects a single field	531 (83.8%)
Prediction returns document-shaped output (has an id field)	325 (51.3%)
Differ by row count	0 (0%)
Differ by a disagreeing value	0 (0%)

Appendix

Table 8: Field-level analysis of the 634 cases where Soft-EX and Strict-EX disagree (Claude-Opus-4-5, with knowledge). Every disagreement is the prediction returning the correct answer plus extra fields.

Request	Ground-truth result	Prediction result (extra fields)
Where is the Malaysian Grand Prix held? Give the location coordinates.	[{lat: 2.76083, lng: 101.738}]	[{location: "Kuala Lumpur", lat: 2.76083, lng: 101.738}] one extra context field
For all drivers who finished race No. 872, who is the youngest?	[{forename: "Sergio", surname: "Pérez"}]	[{_id: "21521", driverId: "815", forename: "Sergio", surname: "Pérez", dob: 1990-01-26}] whole driver document
For the only business in “Yuma” city, how many “medium” reviews did it get?	[{count: 4}]	[{_id: "5969", medium_review_count: 4}] aggregate value plus id

Appendix

Table 9: Representative disagreement cases (Claude-Opus-4-5, with knowledge). Each prediction is scored incorrect by Strict-EX and correct by Soft-EX; in all three the prediction returns the requested answer together with additional document fields.

Panel of Figure 7 — Figure 7: Average query execution time (log scale) versus scale factor for Access-Pattern, FK-Embedding, and 1-1 Conversion across the Spider and BIRD benchmarks.

Benchmark Construction

Benchmark construction develops tasks, datasets, environments, and protocols that make AI capabilities measurable and comparisons reliable.

See all 67 papers

MDB-Link: Hierarchical Schema Linking for Multi-Database Text-to-SQL
Aug 10, 2026Beiyu Xu, Zhenyu Wu, Jiaoyan Chen +1Schema Linking Text-to-SQL
University of Manchester Faculty of Science and Engineering Department of Computer Science
BIRD-History: A Benchmark for History-Driven Text-to-SQL with Fine-Grained Knowledge Annotations
Aug 29, 2026Yunfan Zhou, Qiming Shi, Yizhou Yang +2Text-to-SQL Text-to-SQL Evaluation
State Key Lab of CAD&CG, Zhejiang University · School of Software Technology, Zhejiang University
Human-Level Text-to-SQL via Reinforcement Learning on Verified Data, Without Pipeline Engineering
Mar 20, 2026Yuxuan Zhu, Tengjun Jin, Yoojin Choi +1Text-to-SQL Reinforcement Learning with Verifiable Rewards
University of Illinois · Bridgewater AIA Labs

Aug 10, 2026J/K move · Enter open · S save

MDB-Link: Hierarchical Schema Linking for Multi-Database Text-to-SQL

Beiyu Xu, Zhenyu Wu, Jiaoyan Chen, Riza theresa Batista-navarro

University of Manchester Faculty of Science and Engineering Department of Computer Science

Schema Linking Text-to-SQL Database Systems

Open paper AskPDF arXiv

Traditional Text-to-SQL research and benchmarks assume a known target database, overlooking settings in which a query must be routed within a large, heterogeneous database collection. We therefore study schema linking in a multi-database setting, where the system must first locate the target database and then construct a compact, SQL-relevant schema for generation. We propose MDB-Link, a hierarchical schema-linking framework that retrieves question-relevant columns from a global index, aggregates retrieval evidence to shortlist databases, and uses a budget-aware large language model (LLM) for database reranking, table selection, and column grounding. With Qwen2.5-14B, MDB-Link outperforms LinkAlign on MMQA, Spider2-Snow, and BIRD-dev in database localization and column selection while producing schema subsets close in size to the gold schemas. Exact match improves from 16.88 to 51.41 on MMQA, 2.50 to 9.17 on Spider2-Snow, and 12.52 to 38.01 on BIRD-dev. MDB-Link also runs faster than LinkAlign and AutoLink, demonstrating the effectiveness of hierarchical schema reduction for downstream SQL generation.

Abstract

Benchmark Construction

Explore similar work

MDB-Link: Hierarchical Schema Linking for Multi-Database Text-to-SQL

BIRD-History: A Benchmark for History-Driven Text-to-SQL with Fine-Grained Knowledge Annotations

Human-Level Text-to-SQL via Reinforcement Learning on Verified Data, Without Pipeline Engineering