Frugal Knowledge Graph Construction with Local LLMs: A Zero-Shot Pipeline, Self-Consistency and Wisdom of Artificial Crowds

Pierre Jourlin

cs.AIApr 13, 2026

Frugal Knowledge Graph Construction with Local LLMs: A Zero-Shot Pipeline, Self-Consistency and Wisdom of Artificial Crowds

Authors: Pierre Jourlin

Organizations: LIA · Avignon Université, Laboratoire d’Informatique d’Avignon (LIA)

Abstract

This paper presents an empirical study of a multi-model zero-shot pipeline for knowledge graph construction and exploitation, executed entirely through local inference on consumer-grade hardware. We propose a reproducible evaluation framework integrating two external benchmarks (DocRED, HotpotQA), WebQuestionsSP-style synthetic data, and the RAGAS evaluation framework in an automated pipeline. On 500 document-level relations, our system achieves an F1 of 0.70 $\pm$ 0.041 in zero-shot, compared to 0.80 for supervised DREEAM. Text-to-query achieves an accuracy of 0.80 $\pm$ 0.06 on 200 samples. Multi-hop reasoning achieves an Exact Match (EM) of 0.46 $\pm$ 0.04 on 500 HotpotQA questions, with a RAGAS faithfulness of 0.96 $\pm$ 0.04 on 50 samples. Beyond the pipeline, we study diversity mechanisms for difficult multi-hop reasoning. On 181 questions unsolvable at zero temperature, self-consistency (k=5, T =0.7) recovers up to 23% EM with a single Mixture-of-Experts (MoE) model, but the cross-model oracle (3 architectures x 5 samples) reaches 46.4%. We highlight an agreement paradox: strong consensus among samples signals collective hallucination rather than a reliable answer, echoing the work of Moussa{ï}d et al. on the wisdom of crowds. Extending to the full pipeline (500 questions), self-consistency (k=3) raises EM from 0.46 to 0.48 $\pm$ 0.04. A confidence-routing cascade mechanism (Phi-4 $\rightarrow$ GPT-OSS, k=5) achieves an EM of 0.55 $\pm$ 0.04, the best result obtained, with 45.4% of questions rerouted. Finally, we show that V3 prompt engineering applied to other models does not reproduce the gains observed with Gemma-4, confirming the specific prompt/model interaction. The entire system runs in $\sim$ 5 h on a single RTX 3090, without any training, for an estimated carbon footprint of 0.09 kg CO2 eq.

Figures & tables

Source / Framework	Task	N	Metrics
Re-DocRED [ 3 ]	Relation extraction	500	P, R, F1
Synthetic ⋆	Text-to-Query	200	Accuracy, Valid Cypher
HotpotQA [ 5 ]	Multi-hop reasoning	500	EM, Token-F1
Synthetic ⋆ + RAGAS [ 7 ]	Anti-hallucination	50	Faithfulness, Relevance, Ctx. Precision
⋆ Questions generated by the LLM from manual seeds (20 for Text-to-Query, 6 for RAG). RAG metrics are computed by the RAGAS framework. See § Evaluation Protocol.

Table 1: Tasks, data, and evaluation metrics.

Task	Model	Parameters	Quant.
Relation extraction	Gemma-4-27B-A4B-it	27B (4B active)	Q4_K_M
Text-to-Query	Qwen3-Deep	8B	Q4_K_M
Multi-hop reasoning	Phi-4	14B	Q4_K_M
Conversational RAG	Mistral-Small	24B	Q4_K_M

Table 2: Models used per task.

Metric	Value	95% CI	Target
Precision	0.7379	[0.70; 0.77]	—
Recall	0.6700	[0.63; 0.71]	—
F1-Score	0.7023	[0.67; 0.74]	0.85

Table 3: Relation extraction results (500 DocRED samples). All models are zero-shot. The 95% CIs are computed by bootstrap.

Model	P	R	F1	Unparsed samples (%)
Mistral-Small	0.666	0.666	0.666	0.0
GPT-OSS (20B)	0.823	0.550	0.659	33.2
Phi-4-Reasoning+	0.656	0.656	0.656	0.0
Phi-4	0.618	0.618	0.618	0.0
Qwen-3 (14B)	0.917	0.176	0.295	80.8
Gemma-4 (26B, pipeline)	1.000	0.020	0.039	98.0

Table 4: Raw extraction benchmark on 500 Re-DocRED samples. JSON format, minimal prompt, zero-shot.

System	Paradigm	F1 (%)	Hardware
DREEAM [ 9 ]	Supervised (fine-tuned)	80.2	Multi-GPU †
ATLOP [ 8 ]	Supervised (fine-tuned)	77.8	Multi-GPU †
GPT-3 few-shot [ 10 ]	Few-shot (cloud API)	$\sim$ 72 ∗	OpenAI API
GPT-3 zero-shot [ 10 ]	Zero-shot (cloud API)	$\sim$ 30	OpenAI API
ChatGPT zero-shot [ 11 ]	Zero-shot (cloud API)	$\sim$ 25	OpenAI API
SYNSYNTH (Gemma-4 Q4)	Zero-shot (local)	70.2	1 $\times$ RTX 3090

Table 5: Comparison with supervised and zero-shot baselines on DocRED.

Metric	Value	95% CI	Target
Accuracy	0.795	[0.74; 0.85]	0.90
Valid Cypher rate	1.00	—	—

Table 6: Text-to-Query results (200 samples).

Metric	Value	95% CI	Target
Exact Match (EM)	0.458	[0.42; 0.50]	0.70
Token-F1	0.520	[0.49; 0.55]	—
Mean chain length	3.24	—	3

Table 7: Multi-hop reasoning results (500 HotpotQA samples). Standard metrics: EM and token-F1.

Metric	Value	95% CI
Mean faithfulness	0.9572	[0.91; 1.00]
Mean relevance	0.8220	[0.76; 0.88]
Context precision	0.9480	[0.90; 0.99]

Table 8: RAGAS evaluation results (50 samples).

$n$	EM V4	EM V1	F1 V4	chain	$\Delta$ V4 $-$ V1
10	0.444	0.440	0.432	3.73	$+$ 0.004
50	0.344	0.330	0.313	4.31	$+$ 0.014
200	0.380	0.356	0.341	3.87	$+$ 0.024
500	0.378	0.308	0.359	3.74	$+$ 0.070
1000	0.406	0.246	0.405	3.46	$+$ 0.160
3000	0.406	0.190	0.387	3.80	$+$ 0.216

Table 9: QLoRA learning curve on HotpotQA (500 test questions). The V4 format (structured reasoning chains) is compared to the V1 format (degenerate chains).

Model	Size	EM	F1	chain	JSON%
GPT-OSS-20B	20B MoE	0.177	0.201	2.25	12.2
Phi-4-Reasoning+	14B	0.166	0.156	10.9	23.8
Phi-4	14B	0.133	0.165	3.39	0.0
Magistral-24B	24B	0.110	0.142	4.31	0.6
Gemma-4-26B	26B MoE	0.083	0.066	0.86	77.3
Qwen3-14B	14B	0.072	0.088	2.13	4.4

Table 10: Benchmark of 8 zero-shot LLMs on the 181 difficult questions.

Phase	N	Duration
Relation extraction	500	3 h 55 min
Text-to-Query	200	$\sim$ 3 min
Multi-hop reasoning	500	22 min 47 s
Conversational RAG	50	$\sim$ 10 min
Total pipeline	—	$\sim$ 4 h 30

Table 11: Execution times of the full pipeline.

Approach	Hardware	Training	Inference	F1
SYNSYNTH (Gemma-4 Q4)	1 $\times$ RTX 3090	None	$\sim$ 4 h	0.70
GPT-3 zero-shot [ 10 ]	Cloud API	None	—	$\sim$ 0.30
GPT-3 few-shot [ 10 ]	Cloud API	None	—	$\sim$ 0.72
DREEAM [ 9 ]	Multi-GPU †	$\sim$ 24–48 h	$<$ 1 min	0.80
ATLOP [ 8 ]	Multi-GPU †	$\sim$ 12–24 h	$<$ 1 min	0.78
† Exact resources not reported in the original papers.

Table 12: Frugality comparison between SYNSYNTH and baselines.

Version	Extr. F1	Query	Multi. EM	RAG F.	Duration
V1 (baseline)	0.263	0.80	0.462	1.00	24 h 10
V2 (+ constrained decoding)	0.260	0.80	0.462	1.00	2 h 53
V3 (+ synonyms + prompt)	0.702	0.80	0.458	0.94	4 h 23
V4 (+ QLoRA multi-hop)	0.702	0.795	0.406 ‡	0.96	$\sim$ 5 h
V5a (+ SC $k{=}3$ )	0.702	0.795	0.482	0.96	$\sim$ 6 h
V5b (+ Cascade)	0.702	0.795	0.552	0.96	$\sim$ 8 h

Table 13: Score evolution by pipeline version. Extr. F1: relation extraction F1 (DocRED); Query: Text-to-Query accuracy (synthetic data); Multi. EM: multi-hop reasoning exact match (HotpotQA); RAG F.: RAGAS faithfulness.

Scenario	EM	95% CI	#Mod.	VRAM	Rel. cost
V4 QLoRA only ( $n{=}3000$ )	0.406	—	1	4 GB	$\times$ 1
V4 + 8 LLM vote	0.446	—	9	19 GB	$\times$ 130
Phi-4 zero-shot (ref.)	0.462	$[0.42\,;\,0.50]$	1	9 GB	$\times$ 2.5
V5a SC ( $k{=}3$ )	0.482	$[0.44\,;\,0.52]$	1	9 GB	$\times$ 7.5
V4 + 8-model Oracle (theor.)	0.520	—	9	19 GB	$\times$ 130
Random cascade (45%)	0.524	$\pm\,0.008$	2	14 GB	$\times$ 12

Table 14: Definitive Pareto frontier: cost vs. performance on 500 HotpotQA questions.

Model	Type	EM ${}_{\text{voted}}$	Oracle k	F1 ${}_{\text{voted}}$	$\Delta$ vs. $T{\approx}$ 0
GPT-OSS 20B	MoE	0.232	0.332	0.240	$+$ 23.2 pts
Phi-4-Reasoning+	Reasoner	0.171	0.298	0.138	$+$ 17.1 pts
Phi-4	Standard	0.144	0.210	0.172	$+$ 14.4 pts
Cross-model oracle	3 $\times k{=}5$	0.309	0.464	—	$+$ 46.4 pts

Table 15: Self-consistency (

k{=}5

T{=}0.7

) on the 181 unsolvable questions (3 models).

Inter-response agreement	$N$	EM ${}_{\text{voted}}$	Oracle k
High ( $\geq 0.8$ )	84 (46%)	0.214	0.250
Medium $[0.4;\ 0.8[$	74 (41%)	0.311	0.459
Low ( $<0.4$ )	23 (13%)	0.043	0.217

Table 16: Agreement paradox: EM by consensus level (GPT-OSS 20B).

Metric	Value	95% CI
Voted EM	0.482	$[0.440\,;\,0.522]$
Voted F1	0.479	$[0.437\,;\,0.518]$
Oracle ( $k{=}3$ )	0.568	$[0.522\,;\,0.608]$
Mean agreement	0.738	—
Mean agreement is the average proportion, across all questions, of the $k$ responses that match the majority response (e.g., 4 identical responses out of 5 $\to$ agreement $=0.8$ ).

Table 17: Self-consistency (

k{=}3

) on 500 HotpotQA questions (full pipeline).

Metric	Value	95% CI
Voted EM	0.552	$[0.508\,;\,0.594]$
Voted F1	0.534	$[0.493\,;\,0.576]$
Oracle ( $k{=}5$ )	0.658	$[0.616\,;\,0.698]$
Mean agreement	0.837	—
Primary accepted	54.6%	—
Rerouted	33.6%	—

Table 18: Cascade Phi-4

\to

GPT-OSS (

k{=}5

) on 500 HotpotQA questions.

Configuration	$\theta_{h}$	EM	Rerouted	% sec.
Agreement cascade	0.4	0.514	59	11.8%
Agreement cascade	0.6	0.544	149	29.8%
Agreement cascade	0.8	0.552	227	45.4%
Phi-4 only ( $k{=}5$ )	—	0.496	0	0%
Random cascade	—	0.524 $\pm$ 0.008	${\sim}$ 227	45%
Two-model oracle	—	0.588	—	—

Table 19: Sensitivity to threshold

\theta_{h}

and routing baselines (500 questions).

Model	P	R	F1 V3	Raw F1	$\Delta$ V3
Gemma-4 26B (pipeline)	0.738	0.670	0.702	0.039	+0.663
Mistral-Small 24B	0.660	0.660	0.660	0.666	$-$ 0.006
Phi-4-Reasoning+ 14B	0.660	0.660	0.660	0.656	$+$ 0.004
GPT-OSS 20B	0.818	0.538	0.649	0.660	$-$ 0.011

Table 20: V3 cross-model extraction (500 DocRED samples).

Model	Mean EM	$\sigma_{\text{EM}}$	Mean Oracle	$\sigma_{\text{Or.}}$
GPT-OSS 20B	0.217	0.032	0.335	0.024
Phi-4-Reasoning+	0.167	0.011	0.309	0.008
Phi-4	0.130	0.008	0.223	0.008

Table 21: Inter-run variance (

R{=}5

) of self-consistency on 181 difficult questions.

Metric	Mean	$\sigma$	Values
Voted EM	0.550	0.003	0.552 / 0.546 / 0.552
Voted F1	0.533	0.002	0.534 / 0.531 / 0.535
Oracle ( $k{=}5$ )	0.653	0.005	0.658 / 0.650 / 0.650
Mean agreement	0.830	0.006	0.837 / 0.829 / 0.825
% rerouted	45.5	0.1	45.4 / 45.6 / 45.6

Table 22: Inter-run variance (

R{=}3

) of the V5b cascade on 500 HotpotQA questions. The Values column details the individual results of each run (run 0 / run 1 / run 2 ).

Category	N	%	Description
Knowledge gap	64	51.6	Obscure facts absent from pre-training
Numerical reasoning	32	25.8	Dates, populations, exact quantities
Format mismatch	18	14.5	Partial F1 but EM ${=}$ 0 (aliases, granularity)
Entity confusion	10	8.1	Correct semantic type, wrong instance

Table 23: Taxonomy of the 124 unsolvable multi-hop reasoning questions.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Parameter	Extraction	Query	Multi-hop	RAG
Temperature	0.3	0.3	0.3	0.3
Top-p	0.9	0.9	0.9	0.9
num_ctx	8192	8192	8192	8192
num_predict	1024	2048	4096	2048
Output format	JSON	JSON	JSON	Text + JSON
Timeout (s)	600	600	600	600

Appendix

Table 24: Ollama inference hyperparameters for each task.

KG Construction

Knowledge graph construction turns text, tables, images, and databases into structured entities and relationships that machines can query and reason over.

See all 195 papers

Jul 13, 2026

RAGU: A Multi-Step GraphRAG Engine with a Compact Domain-Adapted LLM

Graph retrieval-augmented generation (GraphRAG) enhances large language models with structured knowledge, yet existing systems construct knowledge graphs in a single extraction pass, producing noisy entities and brittle retrieval. RAGU, an open-source modular GraphRAG engine, addresses this by separating extraction from consolidation: entities and relations pass through two-stage typed extraction, DBSCAN-backed deduplication, LLM summarization, and Leiden community detection. A key insight motivates a compact extractor: the skills an in-pipeline LLM needs - comprehension, extraction, reasoning over context - are language skills that grow only weakly with model size, unlike factual world knowledge. Accordingly, we train Meno-Lite-0.1, a 7B model optimized for language skills, which outperforms Qwen2.5-32B on knowledge-graph construction (+12.5% relative harmonic mean) and matches it on English GraphRAG tasks. On GraphRAG-Bench (Medical), RAGU retrieves the most complete context at every factoid level (evidence recall up to 0.84 vs.

\leq

0.76) and overtakes HippoRAG2 on synthesis tasks; on multi-hop factoid QA, the apparent HippoRAG2 advantage is shown to be largely an answer-format artifact. RAGU is installable via

\texttt{pip install graph_ragu}

, runs on a single GPU, and is released under MIT. The source code is publicly available at https://github.com/RaguTeam/RAGU, and the Meno-Lite-0.1 model can be obtained from https://huggingface.co/bond005/meno-lite-0.1.

Mikhail Komarov, Ivan Bondarenko, Stanislav Shtuka +5

1ITMO University · 2Novosibirsk State University · 3Far Eastern Federal University +1

RAG Evaluation KG Construction LLM Reasoning with Graphs Graph Retrieval-Augmented Generation Small Language Models Knowledge Representation+2

Ask PDF

Jan 28, 2026

UltRAG: a Universal Simple Scalable Recipe for Knowledge Graph RAG

Large language models (LLMs) frequently generate confident yet factually incorrect content when used for language generation (a phenomenon often known as hallucination). Retrieval augmented generation (RAG) tries to reduce factual errors by identifying information in a knowledge corpus and putting it in the context window of the model. While this approach is well-established for document-structured data, it is non-trivial to adapt it for Knowledge Graphs (KGs), especially for queries that require multi-node/multi-hop reasoning on graphs. We introduce UltRAG, a training-free KG-RAG recipe that combines LLM query generation, a fully inductive neural query executor, and LLM arbitration. This off-the-shelf composition achieves state-of-the-art results on Knowledge Graph Question Answering (KGQA) tasks without retraining the LLM or executor, while enabling language models to interface with Wikidata-scale graphs (116M entities, 1.6B relations) at comparable or lower costs. Our ablation studies indicate that these gains come from the full system design rather than from any single component.

Dobrik Georgiev, Kheeran K. Naidu, Alberto Cattaneo +3

Graphcore Research

Multi-Hop QA LLM Reasoning with Graphs Graph Retrieval-Augmented Generation Knowledge Graph Question Answering Knowledge Graphs

Ask PDF

Apr 8, 2026

ROZA Graphs: Self-Improving Near-Deterministic RAG through Evidence-Centric Feedback

Language model agents reason from scratch on every query, discarding their chain of thought after each run. The result is lower accuracy and high run-to-run variance. We introduce reasoning graphs, which persist the per-evidence chain of thought as structured edges. Unlike prior memory that retrieves distilled strategies by query similarity, reasoning graphs enable evidence-centric feedback: for every candidate item, the system traverses all incoming evaluation edges across prior runs to surface how that specific item has been judged before. We further introduce retrieval graphs, which feed a planner that prunes consistently-rejected candidates over successive runs. Together they form a ROZA graph: a self-improving feedback loop in which accuracy gains scale with gold-passage reuse (reasoning graph) and efficiency gains scale with candidate-pool overlap (retrieval graph). The base model remains frozen; all gains come from context engineering via graph traversal. We evaluate on MuSiQue and HotpotQA, plus a high-reuse deployment subset. Four findings stand out. (1) Dose-response: accuracy improves monotonically with evidence-profile coverage, reaching +10.6pp over Vanilla RAG at 50%+ coverage on the same questions (47% error reduction,

p<0.0001

; per-question Spearman

ρ=+0.144

p<10^{-6}

n=1{,}100

). (2) Multi-hop scaling: 4-hop accuracy improves by +11.0pp (

p=0.0001

). (3) Cross-cluster prediction: the cluster-level gain is predicted by gold-passage reuse density (

r=0.604

p=0.001

n=26

clusters). (4) High-reuse Pareto dominance: highest or tied-for-highest accuracy alongside 46% lower cost and 46% lower latency. Per-passage decision consistency across repeated runs (

N=73

paired probes,

K=10

runs each, two model families, three temperatures) rises by +8 to +13pp on a fixed 20-passage context and by +12 to +21pp when the retrieval graph also prunes (all

p<0.005

Matthew Penaroza

Retrieval-Augmented Generation Graph Retrieval-Augmented Generation LLM Reasoning with Graphs Memory-Augmented Language Model Agents Reasoning Consistency in Language Models

Ask PDF

Abstract

KG Construction

Explore similar work

RAGU: A Multi-Step GraphRAG Engine with a Compact Domain-Adapted LLM

UltRAG: a Universal Simple Scalable Recipe for Knowledge Graph RAG

ROZA Graphs: Self-Improving Near-Deterministic RAG through Evidence-Centric Feedback