Build2SPARQL: A Large-Scale Text-to-SPARQL Benchmark Dataset for Building Knowledge Graph Querying

Wooyoung Jung

cs.AISep 22, 2026

Build2SPARQL: A Large-Scale Text-to-SPARQL Benchmark Dataset for Building Knowledge Graph Querying

Authors: Wooyoung Jung

Organizations: Assistant Professor, Dept. of Civil and Architectural Engineering and Mechanics, Univ. of Arizona, Tucson, AZ 85719.

Abstract

Building automation systems are increasingly represented as semantic knowledge graphs (KGs) using ontologies such as Brick and ASHRAE 223P, creating a machine-readable substrate for artificial-intelligence applications. One promising application is translating natural-language questions into SPARQL (text-to-SPARQL), which would let building operators query these graphs through language agents, but progress is limited by the scarcity of large natural-language/SPARQL benchmarks. This paper presents Build2SPARQL, a large-scale benchmark for building KGs generated by a KG-grounded pipeline: SPARQL queries are produced and validated entirely by graph-traversal code, while large language models generate only the natural-language questions, keeping query correctness independent of model behavior. The pipeline mines six query-pattern families -- linear chains, branching, UNION, aggregation, OPTIONAL, and attribute-filtered -- and phrases each query across five vocabulary registers. Applied to 201 building KGs (180 Brick, 21 ASHRAE 223P), it yields 6,136 executable SPARQL queries and 30,680 questions. A two-rater human validation of 300 questions found 98.8% semantic fidelity, 98.8% naturalness, and 84.0% operational plausibility. A retrieval-augmented evaluation across three open-weight language models raised exact-match accuracy from 0.2-20% (zero-shot) to 56-65% (three-shot retrieved).

Figures & tables

	Small scale	Large scale
Broad patterns	BuildingQA [ Mulayim et al. 2025a ]	–
Narrow patterns	DA-KGQA [ Mulayim et al. 2025b ]	BuildingGPT [ Li and Wang 2026 ] , BuildingGPT2 [ Li et al. 2026 ] , OntoSage [ Devmane et al. 2026 ]

Table 1: NL/SPARQL dataset landscape for building KGs.

Figure 1: Build2SPARQL five-stage pipeline.

Q	Register label	Key instruction
1	Technical	Name the equipment types and relationships in the query directly. Use ontology class names and property identifiers where natural.
2	Systems-engineering	Use building domain terminology. Paraphrase ontology class names into engineering language but keep them recognizable as specific equipment types.
3	Operational	Describe what information is needed without naming specific ontology classes or data model terms.
4	Instance-grounded	When named entity examples are listed, use one verbatim in the question (e.g. “Is AHU-3 reporting…” rather than “Is the AHU reporting…”, where AHU denotes an Air Handling Unit). Describe a practical task or check, not a data query.
5	Plain English	Write a plain English question using no ontology terms, no SPARQL vocabulary, no class names, and no equipment identifiers. Paraphrase the information need in everyday language accessible to any building occupant.

Table 2: Register-specific prompt directives for NL question generation.

Pipeline stage	Model	Temperature	Max output tokens
Semantic filtering	Gemma 4 4B ( gemma4:e4b )	0.3	512
Query refinement	Gemma 4 4B ( gemma4:e4b )	0.3	800
NL question generation	Gemma 4 31B ( gemma4:31b )	0.7	150

Table 3: LLM generation configuration. All stages use open-weight Gemma 4 models served locally via Ollama with no fine-tuning.

Metric	Count
Buildings	201
Real	81
Mortar testbed (Brick)	47
Brick reference models	13
ASHRAE 223P example models	21
Synthetic (BuildGraph, Brick)	120

Table 4: Build2SPARQL corpus overview.

Pattern type	Queries	Share (%)
Linear chain	5,009	81.6
Branching (Y-shape)	815	13.3
Aggregation (COUNT)	120	2.0
Attribute-filtered	100	1.6
UNION disjunction	51	0.8
Optional (OPTIONAL)	41	0.7

Table 5: Query pattern-type distribution.

Q	Register	Question
1	Technical	Which brick:Start_Stop_Command instances are linked via brick:isPointOf to brick:Supply_Fan entities?
2	Systems-engineering	Which supply fans are associated with which start/stop command points?
3	Operational	Which supply fans have corresponding start and stop controls?
4	Instance-grounded	Which start/stop command is mapped to SupplyFan_AHU01 ?
5	Plain English	Which control command is used to turn the supply fan on and off?

Table 6: Five register levels for a representative linear-chain query.

Pattern type	$n$	Lex. sim.	Dist. open.	Q4 inst. (%)
Linear chain	5,009	0.170	0.714	81
Branching (Y-shape)	815	0.156	0.910	82
Aggregation (COUNT)	120	0.176	0.753	52
UNION disjunction	51	0.243	0.600	22
Optional (OPTIONAL)	41	0.176	0.883	17
Attribute-filtered	100	0.198	0.677	56

Table 7: Quality metrics by pattern type.

Model	Condition	Arity F1 (%)	Exec (%)	Exact Match F1 (%)	Row F1	Ent. Set F1
Gemma 4 4B	Zero-shot	68.1	91.0	3.8	0.039	0.390
Gemma 4 4B	3-shot (random)	74.1	93.2	25.2	0.257	0.754
Gemma 4 4B	3-shot (retrieved)	87.4	89.4	60.9	0.615	0.958
Llama 3.1 8B	Zero-shot	66.7	90.6	0.2	0.016	0.214
Llama 3.1 8B	3-shot (random)	71.5	88.1	22.0	0.229	0.797
Llama 3.1 8B	3-shot (retrieved)	91.7	88.1	56.0	0.575	0.907

Table 8: NL-to-SPARQL baseline results on the held-out test set. All models achieve 100% syntactic validity.

Model	Condition	Q1	Q2	Q3	Q4	Q5
Gemma 4 4B	Zero-shot	15.8	3.1	0.8	0.0	0.0
Gemma 4 4B	3-shot (random)	32.7	26.4	26.7	9.0	27.6
Gemma 4 4B	3-shot (retrieved)	81.2	74.4	66.7	17.0	60.0
Llama 3.1 8B	Zero-shot	0.0	0.0	0.0	1.0	0.0
Llama 3.1 8B	3-shot (random)	27.7	22.5	21.7	14.0	27.6
Llama 3.1 8B	3-shot (retrieved)	78.2	63.6	57.5	21.0	57.1

Table 9: Exact Match F1 (%) by register level, 3-shot retrieved.

Figure 2: Exact Match F1 (%) by register, 3-shot retrieved. Q4 underperforms Q5 due to instance-specificity mismatch.

Model	Lin.	Brnch.	Union	Aggr.	Opt.	Filt.
Gemma 4 4B	63.5	40.0	9.1	33.3	50.0	20.0
Llama 3.1 8B	58.5	40.0	18.2	26.7	33.3	20.0
Qwen 2.5 32B	67.6	40.0	18.2	26.7	66.7	60.0

Table 10: Exact Match F1 (%) by pattern type, 3-shot retrieved.

	Exact Match F1		Row F1
Model	Realistic	Oracle	Realistic	Oracle
Gemma 4 4B ( $n=120$ )	0.592	0.558	0.598	0.566
Llama 3.1 8B ( $n=45$ )	0.667	0.511	0.682	0.519

Table 11: Retrieval-realism ablation: a matched-modality oracle (retrieval keyed on the pipeline’s query-result summary at both index and query) versus the realistic question-keyed retrieval reported above, on a paired subset of eight real Mortar buildings.

Scale and coverage
Metric	Build2SPARQL	BuildingQA
Buildings	201	4
Queries	6,136	27
NL questions	30,680	188
Pattern types	6	3
Mean BGP triples	3.9	5.7

Table 12: Build2SPARQL vs. BuildingQA.

Resource	Construction	Bldg.	Ontology	Queries	NL q.	Br	Un	Op	Ag	Fi
BuildingQA a	expert	4	Brick, 223P	27	188	Y	Y	n.r.	N	n.r.
DA-KGQA	method eval	1	Brick	19	19	N	N	N	N	N
OntoSage b	synthesis+FT	3	Brick	$\sim$ 120,000 pairs		n.r.	n.r.	n.r.	P	n.r.
BuildingGPT	6 templates	45	Brick	18,555	18,555	N	N	N	P	N
BuildingGPT2 c	templates+FT	45	Brick	18,555	18,555	N	N	N	P	N
Build2SPARQL	KG-mined	201	Brick, 223P	6,136	30,680	Y	Y	Y	Y	Y

Table 13: Structural comparison of building-domain NL/SPARQL resources. Operator columns: Br branching, Un UNION , Op OPTIONAL , Ag grouped aggregation, Fi unit/quantity filter; FT (in Construction) denotes fine-tuning. Marks: Y present, N absent, P partial, n.r. not reported. P denotes flat COUNT of a single class (BuildingGPT/BuildingGPT2) or aggregation handled outside SPARQL (OntoSage).

Pool	Test building identifiers	$n$
BuildGraph hospital	BG_hospital_{2004, 2013, pre1980} _0000--0004	15
Mortar	bldg6, bldg10, bldg15, bldg16, bldg17, bldg22, bldg23, bldg25	8
Brick / 223P reference	apartment, IBAL, NIST-HPL, pnnl-bdg2-1, soda_brick	5
Total test		28
Retrieval pool (all other)		173

Table 14: Reference evaluation split. Test buildings listed by pool; retrieval pool is all remaining 173 buildings.

Criterion	Question the rater answers	Scale
C1 Semantic fidelity	Does the question ask for what the paired SPARQL query actually returns—same entities, relationship, and conditions?	Y/B/N
C2 Naturalness	Does it read as fluent, grammatical English a practitioner might say? (For the Technical register, verbatim ontology tokens are expected and not penalized.)	Y/B/N
C3 Operational plausibility	Could a building operator, energy manager, or commissioning engineer realistically ask this in practice?	Y/N

Table 15: Human-validation criteria and rating scales.

By register
Group	C1 Sem. fidelity	C2 Naturalness	C3 Oper. plaus.
Q1 Technical	100.0	99.2	80.0
Q2 Systems-engineering	100.0	100.0	80.0
Q3 Operational	99.2	98.3	80.0
Q4 Instance-grounded	97.5	99.2	98.3
Q5 Plain English	97.5	97.5	81.7

Table 16: Human-validation pass rates (%; Y

{}=1

, B

{}=0.5

, N

{}=0

) by register and by SPARQL pattern type, after adjudication.

Benchmark Construction

Benchmark construction develops tasks, datasets, environments, and protocols that make AI capabilities measurable and comparisons reliable.

See all 68 papers

Natural Language Questions as an Interface for Knowledge Graphs: QRAKEN Graph Distillation and Semantic Self-Healing
Oct 6, 2026Remo Grillo, Lukas Klic, Giovanni ColavizzaLLM Grounding Knowledge Graphs
University of Bologna, Via Zamboni 33, 40126 Bologna, Italy · I Tatti — The Harvard University Center for Italian Renaissance Studies, Via di Vincigliata 26, 50135 Florence, Italy · Department of Communication, University of Copenhagen, Karen Blixens Plads 8, 2300 Copenhagen S, Denmark
Towards Researcher Agents for Knowledge-Graph Question Answering
Aug 7, 2026Tommaso Soru, Abdulsobur OyewaleLLM Agent Self-Improvement Knowledge Graph Question Answering
1Liber AI Research, London, United Kingdom
Natural Language Knowledge Graph Query Execution: Leveraging Controlled Semantics in the LLM Context Window
Sep 13, 2026Blake G. FitchKnowledge Graph Question Answering Natural Language to SPARQL
Max Planck Institute for Biological Cybernetics, Tübingen, Germany

Oct 6, 2026J/K move · Enter open · S save

Natural Language Questions as an Interface for Knowledge Graphs: QRAKEN Graph Distillation and Semantic Self-Healing

Remo Grillo, Lukas Klic, Giovanni Colavizza

University of Bologna, Via Zamboni 33, 40126 Bologna, Italy · I Tatti — The Harvard University Center for Italian Renaissance Studies, Via di Vincigliata 26, 50135 Florence, Italy · Department of Communication, University of Copenhagen, Karen Blixens Plads 8, 2300 Copenhagen S, Denmark

LLM Grounding Knowledge Graphs Natural Language to SPARQL

Open paper AskPDF arXiv

Natural-language access to RDF knowledge graphs is a core Semantic Web ambition. Large language models (LLMs) have advanced Text-to-SPARQL, yet on unfamiliar graphs they often generate valid queries that misrepresent the populated data model. QRAKEN is a training-free, ontology-agnostic neurosymbolic pipeline grounding generation in empirical graph evidence rather than schema expectations. An offline distiller produces TTQL, a compact description of populated multi-hop patterns, conditional frequencies and path-conditioned literal examples, plus a class-property co-occurrence matrix. Online, TTQL guides the LLM, while deterministic syntax, vocabulary and data-model checks provide diagnostics for iterative refinement. On CK25 (First International Text2SPARQL Challenge), under matched-condition recomputation on a QLever snapshot, QRAKEN achieves strict F1 of 0.643

\pm

0.026 with GPT-4.1 mini and 0.652

\pm

0.012 with GPT-5.4: relative gains of 30% and 32% over the strongest recomputed participant, outperforming systems using the same base model family. Ablations identify TTQL patterns as the dominant driver (+0.31 strict F1 over a shape-only baseline); the refinement loop provides a cheap safety net, rejecting triple patterns unsupported by the co-occurrence matrix. Compared with auto-derived SHACL, TTQL yields 64% higher strict F1, supporting the value of empirical patterns beyond schema exposure. With two local 35B 4-bit open-weight models at zero marginal cost, the same pipeline matches the strongest recomputed participant, and TTQL advantages over shape-only and SHACL baselines persist. Results on a single, relatively small benchmark provide an initial empirical signal; monolithic TTQL injection on very open cross-domain graphs remains the main limitation.

Abstract

Benchmark Construction

Explore similar work

Natural Language Questions as an Interface for Knowledge Graphs: QRAKEN Graph Distillation and Semantic Self-Healing

Towards Researcher Agents for Knowledge-Graph Question Answering

Natural Language Knowledge Graph Query Execution: Leveraging Controlled Semantics in the LLM Context Window