IFC-MCQ: Automatically Generating Multiple-Choice Benchmarks for Evaluating LLM Agents on IFC Data Retrieval

Rabindra Lamsal; Sisi Zlatanova; Xuesong Shen

cs.CLOct 21, 2026

IFC-MCQ: Automatically Generating Multiple-Choice Benchmarks for Evaluating LLM Agents on IFC Data Retrieval

Authors: Rabindra Lamsal, Sisi Zlatanova, Xuesong Shen

Organizations: GRID Lab, School of Built Environment, The University of New South Wales, Sydney, Australia · School of Civil and Environmental Engineering, The University of New South Wales, Sydney, Australia

Abstract

Industry Foundation Classes (IFC), the standard for building information exchange in architecture, engineering, and construction (AEC), is increasingly queried through large language models (LLMs) using natural language. Evaluating these systems at scale is difficult because free-text answers require costly human or LLM-based review. We propose IFC-MCQ, a method that automatically generates multiple-choice benchmarks from IFC files for exact option-set grading. The taxonomy covers 15 categories and 36 subtypes across four groups, with answer keys derived from the file and distractors constructed from file-derived values or predefined rules. We applied IFC-MCQ to IFC models of varying sizes and modelling characteristics to assess generation coverage and evaluated two agentic configurations on selected models using tool- and code-based retrieval. The evaluation exposed category-specific failure modes and showed that performance can vary with the source IFC model and retrieval interface. IFC-MCQ enables scalable benchmarking of LLM-based IFC retrieval in AEC workflows.

Figures & tables

Work	What it does	Evaluation set; grading	Evaluation size
Systems
Yin et al. [44]	NL to BIMQL via GNN and ontology	Study-specific; automated query matching	Fixed query types
Ibba et al. [21]	NL to SPARQL over an IFC knowledge graph	Study-specific; manual	28 questions
Iranmanesh et al. [22]	NL to Cypher over Neo4j	Study-specific; manual	60 questions
Avgoren [5]	NL to Cypher over Neo4j with reduced IFC relationships	Study-specific; manual	135 questions
Pacheco and Berkmiller [33]	NL to Cypher over Neo4j	Qualitative examples; no formal grading	Not reported

Table 1: Natural language (NL) querying systems for IFC and related benchmarks.

Group	Cat.	Name
(1) Direct retrieval	CLS	Class identification
	ATT	Attribute retrieval
	PRP	Property retrieval
	BLN	Boolean property verification
	EXS	Existence / absence
(2) Relational	TYP	Type object resolution

Table 2: IFC-MCQ question taxonomy. Groups organize categories by interaction type. Some categories further divide into subtypes (refer to Table 3 ).

Cat.	Subtype	Type †	Probe (input $\to$ output)
Direct retrieval
CLS	class by GUID	S	entity GUID $\to$ IFC class.
ATT	attribute value	S	entity, attribute $\to$ stored value.
PRP	property value	S	entity, pset, property $\to$ stored value.
BLN	boolean check	B	entity, claimed Boolean $\to$ whether it matches.
EXS	property existence	B	entity, class-common property $\to$ whether populated.

Table 3: Categories and their subtypes considered in this work. † S (single-choice), M (multi-choice), B (boolean.)

Figure 1: An overview of the IFC-MCQ pipeline.

Panel of Figure 2 — Figure 2: Rendered views of the ten IFC models used in this study. The views are illustrative and are not shown at a common scale. All models are the property of their respective authors; none were created in this work.

IFC Model	Storeys	Elements	Spaces	Walls	Slabs	Windows	Doors
FZK Haus	2	131	7	13	4	11	5
Duplex	4	268	21	57	21	24	14
NiedriHaus	4	146	19	45	8	16	16
IAI Demo	4	189	21	38	6	30	22
Office	3	986	99	487	2	69	102
Institute	5	989	82	119	26	206	77

Table 4: Summary of IFC building models and their contained elements. The reported statistics are based on counts of IfcBuildingStorey , IfcElement , IfcSpace , IfcWall , IfcSlab , IfcWindow , and IfcDoor entities.

Figure 3: Overview of the agentic evaluation. The tool agent retrieves information through a fixed set of 17 read-only tools, while the code agent writes and executes IfcOpenShell code against the IFC model.

Figure 4: System prompt template used by the tool agent.

Figure 5: System prompt template used by the code agent.

Proprietary frontier models
LLM	Size (params)
qwen3.7-max [ 36 ]	n/a †
gpt-5.6-luna [ 31 ]	n/a †
compact
gpt-5.4-mini [ 32 ]	n/a †
gpt-5.4-nano [ 32 ]	n/a †

Table 5: LLMs evaluated in this work. † Parameter count has not been publicly disclosed.

Cat.	Type	Question	Options
CNT	S	Count the IfcSlab instances (subclasses included) in FZK-Haus.ifc .	(A) 15 (B) 4 ∗ (C) 1 (D) 88 (E) 7
CLS	S	What is the IFC class of the entity (GlobalId 2OrWItJ6zAwBNp0OUxK$CR ) in Duplex.ifc ?	(A) IfcRoof (B) IfcSlab (C) IfcFooting (D) IfcWall (E) IfcBeam ∗
SPA	S	Which storey does IfcBeam (GlobalId 3sG$iUZAX499eUK87UfZk_ , Name “Sparren-17”) belong to?	(A) IfcSite “Gelaende”, (B) IfcBuildingStorey “Dachgeschoss” ∗ , (C) IfcBuilding “FZK-Haus”, (D) IfcBuildingStorey “Erdgeschoss”
ATT	S	What is the value of the OverallWidth attribute of IfcDoor (GlobalId 2qiPPF3FrF8OIqfrKiSUqm , Name “Innentuer-3”)?	(A) 1.01 (B) 2.01 (C) 0.885 ∗ (D) 2 (E) 1
QTY	S	Read the quantity “Length” from quantity set “BaseQuantities” of IfcWallStandardCase (GlobalId 3Ttjr$59XEWfWN1WUHjelZ , Name “Wand-Ext-OG-4”). What is its value in metres?	(A) 3.8 m (B) 3 m (C) 4.08 m (D) 10 m (E) 12 m ∗
PRP	S	For IfcBeam (GlobalId 31Quw2HAvFqv$9_JgCuemp , Name “Sparren-6”), read “Slope” from property set “Pset_BeamCommon”. What is its value?	(A) 0 (B) 34.5 (C) 42 (D) 25.5 (E) 30 ∗

Table 6: Representative items from FZK Haus and Duplex. Correct answers are marked with an asterisk.

IFC Model	Generated	Verified	Skipped during generation	10 to <40 items	<10 items
FZK Haus	596 (6.72s)	all (3.16s)	PRP (3)	AGG (36)	-
Duplex	581 (4s)	all (1.42s)	PRP (181), EXS (68)	QTY (21)	-
NiedriHaus	495 (3.87s)	all (1.88s)	PRP (63), EXS (68)	PRP (19), TYP (32), AGG (13), CMP (30)	EXS (1)
IAI Demo	571 (3.81s)	all (2.42s)	EXS (12)	AGG (19), CMP (32)	-
Office	597 (9.02s)	all (2.83s)	PRP (98), EXS (28)	CMP (37)	-
Institute	589 (4.08s)	all (2.17s)	-	AGG (33), CMP (36)	-

Table 7: Results from the IFC-MCQ pipeline across different IFC models. “Generated” denotes candidate items produced by the category generators, with the overall generation time shown in brackets. “Verified” denotes candidates retained after separate verification, with the overall verification time shown in brackets. “Skipped” denotes candidate-generation attempts for which a valid option-set could not be constructed. Categories with fewer than 40 items did not reach the generation cap; categories with fewer than 10 verified items fall below the reporting threshold.

Audit criterion	Passed
Grammatically well formed and placeholders correctly instantiated	89/89 (100%)
Human-derived answer matched the benchmark answer key	89/89 (100%)
Answer uniquely represented among the options	89/89 (100%)

Table 8: Results of the stratified human audit of generated benchmark items.

Figure 6: Performance of 17 LLMs using the tool agent on all 1,177 items generated from FZK Haus and Duplex, reported by question group, question type, and source IFC model.

Figure 7: Tool agent accuracy by IFC-MCQ category for 17 LLMs on the 1,177-item Stage 1 benchmark.

Figure 8: Tool agent accuracy by IFC-MCQ subtype for 17 LLMs on the 1,177-item Stage 1 benchmark.

LLM	FZK Haus+Duplex	Institute+Office+Trapelo+TNW
gpt-5.6-luna	100.0 (300/300)	97.6 (576/590)
deepseek-v4-flash	99.7 (299/300)	93.9 (554/590)
gpt-oss-120b	99.3 (298/300)	97.1 (573/590)
qwen3.6-27b	100.0 (300/300)	98.0 (578/590)
gemma-3-27b-it	87.0 (261/300)	82.7 (488/590)

Table 9: Tool agent overall performance with selected LLMs.

LLM	FZK+Duplex	Institute+Office+Trapelo+TNW
gpt-5.6-luna	98.3 (295/300)	97.6 (576/590)
deepseek-v4-flash	97.7 (293/300)	97.5 (575/590)
gpt-oss-120b	92.7 (278/300)	92.5 (546/590)
qwen3.6-27b	95.7 (287/300)	95.8 (565/590)
gemma-3-27b-it	69.7 (209/300)	62.7 (370/590)

Table 10: Code agent overall performance with selected LLMs.

LLM	Tool agent, all six models	Code agent, all six models	Code $-$ tool
gpt-5.6-luna	98.4 (876/890)	97.9 (871/890)	$-0.6$ pp
deepseek-v4-flash	95.8 (853/890)	97.5 (868/890)	$+1.7$ pp
gpt-oss-120b	97.9 (871/890)	92.6 (824/890)	$-5.3$ pp
qwen3.6-27b	98.7 (878/890)	95.7 (852/890)	$-2.9$ pp
gemma-3-27b-it	84.2 (749/890)	65.1 (579/890)	$-19.1$ pp

Table 11: Performance comparison between tool and code agents.

	Five LLMs		Other four LLMs, excluding gemma-3-27b-it
Category	Tool agent	Code agent	Tool agent	Code agent
CLS	100.0 (300/300)	100.0 (300/300)	100.0 (240/240)	100.0 (240/240)
ATT	100.0 (300/300)	99.7 (299/300)	100.0 (240/240)	100.0 (240/240)
PRP	95.7 (287/300)	98.3 (295/300)	100.0 (240/240)	99.2 (238/240)
BLN	98.3 (295/300)	95.0 (285/300)	100.0 (240/240)	98.3 (236/240)
EXS	87.7 (263/300)	92.0 (276/300)	97.5 (234/240)	90.8 (218/240)

Table 12: Tool agent and code agent accuracy by IFC-MCQ category across six IFC models. Each non-BND category contains 300 answers, corresponding to 5 LLMs × 6 IFC models × 10 items. BND contains 250 answers because TNW has no BND items. After excluding gemma-3-27b-it , the corresponding totals are 240 for non-BND categories and 200 for BND.

LLM	Overall	Single	Multi	Boolean
high-performing proprietary and open-weight LLMs in Stage 2 and 3
gpt-5.6-luna	49.7 (442/890)	49.7	30.1	61.5
qwen3.6-27b	49.7 (442/890)	49.6	24.7	65.8
low-performing LLM in Stages 2 and 3
gemma-3-27b-it	38.3 (341/890)	38.1	8.2	58.1
low-performing LLM in Stage 1

Table 13: Accuracy without access to the source IFC models on the 890 items used in Stages 2 and 3.

	Mean calls		One call (%)		Truncated
IFC model	Tool	Code	Tool	Code	Tool	Code
FZK Haus	1.30	1.89	82.7	63.7	0	1
Duplex	1.17	1.77	87.9	63.7	0	9
Institute	1.34	2.09	80.3	60.9	39	15
Office	1.22	1.92	84.0	62.0	26	7
Trapelo	1.36	1.91	78.0	61.3	39	12

Table 14: Calls per item across the six IFC models for the five LLMs in Stages 2 and 3. One call denotes the percentage of items answered after a single call. Truncated denotes the number of answers for which an observation was truncated at 6,000 characters.

	Tool		Code
LLM	Input	Output	Input	Output
gpt-5.6-luna	2,533	98	2,398	188
qwen3.6-27b	2,667	1107	5,000	1,476
gpt-oss-120b	3,716	347	4,247	948
deepseek-v4-flash	5,369	1,904	9,480	3,276
gemma-3-27b-it	4,213	200	5,302	552

Table 15: Mean input and output tokens per item across the six IFC models for the five LLMs in Stages 2 and 3. Output includes reasoning tokens, if applicable.

Synthetic Benchmark Generation

Synthetic benchmark generation creates controlled, verifiable evaluation tasks and datasets to test models across diverse settings.

See all 289 papers

IfcLLM: Natural Language Querying of IFC Models through Complementary Relational and Graph Representations
May 13, 2026Rabindra Lamsal, Sisi Zlatanova, Haowen Xu +2Building Information Modeling LLM Agents
GRID Lab, School of Built Environment, The University of New South Wales, Sydney, Australia · School of Civil and Environmental Engineering, The University of New South Wales, Sydney, Australia
BIM-Edit: Benchmarking Large Language Models for IFC-Based Building Information Modeling
Jun 18, 2026Bharathi Kannan Nithyanantham, Clemens Kujat, Tobias Sesterhenn +5LLM Evaluation Building Information Modeling
University of Rostock · Clausthal University of Technology
Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
May 21, 2026Ryo Kanazawa, Koyo Hidaka, Teppei Miyamoto +9LLM Evaluation Benchmark Construction
ONESTRUCTION Inc. Tottori, Japan · AWS GenAI Innovation Center Tokyo, Japan

May 13, 2026J/K move · Enter open · S save

IfcLLM: Natural Language Querying of IFC Models through Complementary Relational and Graph Representations

Rabindra Lamsal, Sisi Zlatanova, Haowen Xu, Yafei Sun+1

GRID Lab, School of Built Environment, The University of New South Wales, Sydney, Australia · School of Civil and Environmental Engineering, The University of New South Wales, Sydney, Australia

Building Information Modeling LLM Agents

Open paper AskPDF arXiv

The Industry Foundation Classes (IFC) standard is central to building data exchange across the lifecycle, from design and construction to facility management and Digital Twin integration. In operational settings, stakeholders increasingly require access to building information without specialist knowledge of IFC's complex, deeply nested schema, motivating natural language interfaces. Existing Large Language Model (LLM)-based querying approaches typically rely on a single data representation, which is not equally suited to attribute retrieval and spatial reasoning. We present IfcLLM, a framework that combines complementary relational and graph representations, routing each query type to the more suitable backend. An LLM agent integrates both through iterative retry-and-refine reasoning, recovering from failures without user input. Evaluated on 30 query scenarios across three IFC models, the top-performing configuration achieved 96% accuracy. Results with open-weight LLMs further demonstrate the potential for local deployment in data-sensitive AEC settings.

Abstract

Synthetic Benchmark Generation

Explore similar work

IfcLLM: Natural Language Querying of IFC Models through Complementary Relational and Graph Representations

BIM-Edit: Benchmarking Large Language Models for IFC-Based Building Information Modeling

Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements