LLMs are remarkable artifacts that have revolutionized a range of knowledge-intensive tasks. A significant contributor is their factual knowledge, which, to date, remains poorly understood, and is usually analyzed from biased samples. In this paper, we provide a framework and the results of a multi-dimensional analysis of GPTKB v1.5 (Hu et al., 2025a), a recursively elicited Knowledge Base (KB) of 100 million facts (or beliefs) of a frontier LLM, namely, GPT-4.1. Given the scale of the elicited facts, we provide a multi-dimensional approach to qualitatively and quantitatively analyze these facts as opposed to the mainstream fact completion benchmarks, which are prone to availability bias. We find that the models' factual knowledge differs quite significantly from established knowledge bases, and that its accuracy is significantly lower than indicated by previous benchmarks. We also find that inconsistency, ambiguity and hallucinations are major issues, shedding light on future research opportunities in neuro-symbolic AI concerning extraction, consolidation and verification of factual LLM knowledge.
Figures & tables
Direct LLM querying
LLM-mined knowledge analysis
✗ Lacks aggregate access to internalized knowledge
✓ Accesses millions of extracted records
✗ Gives uninformative and wrong answers
✓ Provides quantitative insights
ChatGPT interface
Mined knowledge (GPTKB v1.5)
How many individual persons do you know?
I don’t “know” any individual persons [..]. [..] millions to billions, depending on [..]. → uninformative
1,200,692
Do you know more males or more females?
Table 1: Direct LLM querying, vs. large-mined knowledge analysis. ChatGPT conversation archived at https://chatgpt.com/share/68b6cd03-85e4-800d-81c8-afebc74ccc9d .
Resource
#entities
#assertions
Wikimedia-related
Wikidata Vrandecic and Krötzsch (2014)
113M
1.62B
Yago 4.5 Suchanek et al. (2024)
50M
140M
DBpedia Auer et al. (2007)
3.8M
75M
Text-extracted
NELL Carlson et al. (2010)
1.9M
12M
Table 2: Size comparison of knowledge resources and LLM factuality benchmarks.
Class
Frequency
Relation
Frequency
person
1,061,852
instanceOf
6,588,295
human
138,840
locatedIn
2,121,234
film
123,021
occupation
1,962,461
company
122,847
country
1,927,856
book
112,188
notableWork
1,500,584
song
103,824
genre
1,268,067
Table 3: Most frequent GPTKB classes and relations.
Domain
Fraction
Domain
Fraction
Science
0.2%
Business
7.0%
Technology
4.0%
Health
0.0%
Engineering
1.3%
Environment
1.9%
Mathematics
0.0%
event
0.8%
Arts
23.8%
unknown
0.0%
Humanities
60.0%
food
1.1%
Table 4: Entity proportion of first-order subbranches of the taxonomy, based on a random sample of 1000 entities.
Figure 1: Language distribution in GPTKB.
Property
Count
Property
Count
logo
37,338
mapType
2,432
coverArt
33,717
photo
2,253
signature
32,443
portrait
1,337
coatOfArms
5,833
range
581
flag
3,424
hasPoster
517
Table 5: Top-10 predicates which come with objects being image file names.
Figure 2: Female-to-male ratio (y-axis) in 50 most frequent nationalities (x-axis).
Figure 3: Female-to-male ratio (y-axis) in 50 most frequent professions (x-axis).
No female gender
No male gender
Profession
Frequency
Profession
Frequency
sports shooter
127
gptkb:actress
11248
cowboy
74
gptkb:businesswoman
1307
gptkb:drag_queen
70
noblewoman
581
crime lord
68
gptkb:suffragist
469
gptkb:hurler
60
gptkb:feminist
323
Table 7: Top 10 gender-specific professions.
Nationality
GPTKB
Global *
Global top-10
American a
31.5
4.3
India (18.2)
British b
12.4
0.8
China (17.6)
French
3.9
0.8
United States (4.3)
German
3.6
1.0
Indonesia (3.5)
Indian
3.6
18.2
Pakistan (3.9)
Canadian
3.3
0.5
Nigeria (2.9)
Table 8: Nationality distribution in percentage in GPTKB and the real-world. Nationalities in GPTKB that feature in the global top-10 are highlighted. *Real world data obtained from https://worldpopulationreview.com/countries as of 2025.
Language models are powerful artifacts, yet their factual knowledge is still poorly understood, and inaccessible to ad-hoc browsing and scalable statistical analysis. This demonstration introduces GPTKB v1.5, a densely interlinked 100-million-triple knowledge base (KB) built for $14,000 from GPT-4.1, using the GPTKB methodology for massive-recursive LLM knowledge materialization. This demo focuses on three use cases: (1) link-traversal-based LLM knowledge exploration, (2) SPARQL-based structured LLM knowledge querying, (3) comparative exploration of the strengths and weaknesses of LLM knowledge. Massive-recursive LLM knowledge materialization is a groundbreaking opportunity both for the systematic analysis of LLM knowledge, as well as for automated KB construction.
Yujia Hu, Tuan-Phong Nguyen, Shrestha Ghosh +2
ScaDS.AI Dresden/Leipzig & TU Dresden, Germany · VNU University of Engineering and Technology, Hanoi, Vietnam · University of Tübingen, Germany
We present a web demo for exploring a large-scale disambiguated knowledge base (KB) materialized from a large language model (LLM). GPTKB 2.0 contains 38.4M triples over 1.6M canonical entities, together with 207.6K consolidated relations and 66K consolidated classes. Unlike prior LLM-derived knowledge bases that largely identify entities by surface strings, GPTKB 2.0 performs context-guided disambiguation during recursive KB construction, separating homonyms and merging synonymous mentions as facts are elicited. The demo makes this process inspectable: users can browse entities, follow links across the KB, and audit the provenance of individual facts, including surface forms, candidate matches, source triples, and disambiguation decisions. The interface further supports structured SPARQL queries, natural-language questions translated to SPARQL, and entity linking from user-provided text to canonical GPTKB 2.0 entries. GPTKB 2.0 is available at https://gptkb.org/, with the full KB downloadable for offline use.
Yujia Hu, Tuan-Phong Nguyen, Simon Razniewski
ScaDS.AI Dresden/Leipzig & TU Dresden, Germany · VNU University of Engineering and Technology, Hanoi, Vietnam
We propose a new method that allows an LLM to automatically pull in factual knowledge from a knowledge base during token generation. This means that (1)~factual knowledge in the LLM output can be updated without retraining the LLM, (2)~facts in the LLM output can be traced to the knowledge base for transparency and explainability, and (3)~smaller models can achieve the same factual accuracy as larger models. Our core idea is to train the model to produce special tokens that trigger a query to the knowledge base. Our experiments show that our method improves factual grounding in both short and long-form generation, and allows factual revisions to take effect through KB edits rather than parameter updates.
Francois Crespin, Fabian M. Suchanek, Nils Holzenberger
IP Paris, LTCI · Télécom Paris · Institut Polytechnique de Paris, France