Do LLMs Understand Context? A Knowledge Graph-Based Evaluation Framework

Subavarshana Arumugam; Mamta Nallaretnam; Kithuni Wickramasinghe; Chamath Gunapala; Pragatheeswaran Vipulanandan; Kamal Premaratne; Uthayasanker Thayasivam

cs.AISep 24, 2026

Do LLMs Understand Context? A Knowledge Graph-Based Evaluation Framework

Authors: Subavarshana Arumugam, Mamta Nallaretnam, Kithuni Wickramasinghe, Chamath Gunapala, Pragatheeswaran Vipulanandan, Kamal Premaratne, Uthayasanker Thayasivam

Organizations: Department of Computer Science & Engineering, University of Moratuwa, Sri Lanka · Department of Electrical and Computer Engineering, University of Miami, USA

Abstract

While large language models (LLMs) have achieved remarkable linguistic capabilities, a profound question lingers at their core: do these models truly comprehend context or simply excel at pattern matching on an unprecedented scale? Contextual understanding in LLMs refers to the ability to correctly extract relevant information from a given context, integrate it into a coherent internal representation, and reason over it to produce factually consistent and contextually grounded responses. However, traditional methods such as BiLingual Evaluation Understudy (BLEU) and perplexity simply measure surface-level performance. This reveals a critical gap in question answering (QA), where responses must be contextually grounded rather than simply being memorized associations. To fill this void, we propose a novel knowledge graph (KG) based evaluation framework for LLM contextual understanding in QA. Central to this is Semantic Structural Similarity for KGs (S3KG), a hybrid similarity measure combining structural and semantic signals into a single score. In addition, a diagnostic analysis framework is developed to identify and categorize reasoning errors at the triplet level, enabling fine-grained analysis of model failures. Together, across nine benchmarks, S3KG achieves F1 gains of up to $+7.6$ points over the strongest baseline and AUROC up to $0.973$ .

Figures & tables

Figure 1: Methodology pipeline for LLM comparison and evaluation.

Similarity scores (✓ = correct prediction; $\times$ = incorrect prediction)
	Positive Example	Negative Example
Text $s_{1}$	His father returned as a finished violinist of the Russian School to Bombay.	Renzo Furlan won 6–3, 6–4 against Thomas Johansson in the finals.
Text $s_{2}$	His father returned to Bombay as a finished violinist of the Russian school.	Thomas Johansson won 6–3, 6–4 against Renzo Furlan in the finals.
Triplet ( $s_{1}$ )	(father, returned_to, Bombay)	(Renzo Furlan, won_against, Thomas Johansson)
Triplet ( $s_{2}$ )	(father, returned_to, Bombay)	(Thomas Johansson, won_against, Renzo Furlan)
True Label	1 (Similar)	0 (Not Similar)

Table 1: Similarity scores on two PAWS-Wiki pairs (threshold

=0.5

;✓ = correct,

\times

= incorrect).The positive pair differs only in word order; the negative pair swaps the subject and object of the winning relation. All seven baselines assign near-identical high scores to both pairs, failing on the negative example. For the positive pair , S3KG achieves the optimal score of

1.00

, whereas surface-form methods such as BLEU (

0.58

) and ROUGE-L (

0.85

) underperform by penalising inconsequential word-order variation.For the negative pair , S3KG scores

0.48

—the only sub-threshold result—correctly predicting Not Similar . The score is not zero because the sentences share substantial content; only the relational direction differs. The KG component isolates this reversal via the directed triplet, reducing the score from the near-

1.0

surface baseline to just below the decision threshold, while

\alpha=0.5

balances surface and structural similarity.

Dataset	S3KG (Ours)	ROUGE-1	ROUGE-2	ROUGE-L	BLEU	BERTScore	MiniLM	sent-T5-base
MRPC	0.692	0.745	0.720	0.729	0.687	0.758	0.723	0.765
PAWS-Wiki	0.766	0.678	0.715	0.735	0.716	0.691	0.687	0.674
Semantic-KG Combined	0.834	0.732	0.707	0.722	0.715	0.757	0.770	0.774
Wiki Swap	0.872	1.000	0.860	0.729	0.868	0.747	0.821	0.767
Semantic-KG Codex 400	0.932	0.835	0.822	0.792	0.806	0.823	0.875	0.872
Semantic-KG FindKG	0.767	0.745	0.717	0.719	0.711	0.739	0.802	0.848

Table 2: F1 Score and ROC-AUC of S3KG and seven baselines across all nine benchmark datasets. The best-performing method per dataset is bolded . S3KG is shown using the best

\alpha

variant per dataset, selected by maximum F1 over

\alpha

swept over

\alpha\in\{0.0,0.1,\ldots,1.0\}

(full sweep in Appendix E ). S3KG attains the highest score on 6 of 9 datasets and the highest meaningful score on the Wikipedia Entity-Swap anti-circularity control, with gains of up to

+7.6

F1 over the strongest baseline on KG-rich paragraph datasets.

Dataset	Best $\alpha$	F1	AUROC
MRPC	0.3	0.692	0.673
PAWS-Wiki	0.5	0.766	0.795
STS12	0.1	0.786	0.834
SK-Codex 400	0.5	0.932	0.973
SK-Combined	0.5	0.834	0.829
SK-FindKG	0.0	0.767	0.796

Table 3: Best S3KG variant per dataset, selected by maximum F1 via grid search over

\alpha\in\{0.0,0.1,\ldots,1.0\}

. KG-rich paragraph datasets gain up to

+7.6

F1 over the strongest baseline; sparse or noisy KGs yield reduced margins.

Dataset	Model	GoldSim	CtxSim	CUS
	Gemma-7B	0.6889	0.7031	0.6774
MesaQA	Llama-2-7B	0.6570	0.7236	0.6752
	Mistral-7B	0.6491	0.7356	0.6780
	Falcon-7B	0.6036	0.6458	0.6063
	Gemma-7B	0.5235	0.6401	0.5587
PubMedQA	Llama-2-7B	0.5220	0.6567	0.5651

Table 4: LLM Evaluation Results (mean over

N{=}400

samples,,

\alpha{=}0.5

). GoldSim measures factual alignment with the reference answer; CtxSim measures faithfulness to the supporting context; CUS (Equation 3 ) is their harmonic mean, penalising imbalanced profiles. Mistral-7B achieves the best CUS on both datasets, reflecting consistently balanced factual and contextual understanding, while Falcon-7B underperforms across all metrics and both domains.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Model	$T=0.0$	$T=0.3$	$T=0.7$	$T=1.0$
	Llama-2-7B	0.6570	0.6588	0.6580	0.6406
MesaQA	Gemma-7B	0.6889	0.7060	0.7015	0.6931
	Mistral-7B	0.6491	0.6567	0.6523	0.6444
	Falcon-7B	0.6036	0.6085	0.5558	0.4662
	Llama-2-7B	0.5220	0.5185	0.5143	0.5087
PubMedQA	Gemma-7B	0.5235	0.5149	0.5153	0.5204

Appendix

Table 5: Mean GoldSim per model across temperatures. CUS remains stable across temperature settings, with

T=0.0

serving as a reliable default for controlled evaluation.

Dataset	Model	$T=0.0$	$T=0.3$	$T=0.7$	$T=1.0$
	Llama-2-7B	0.7236	0.7212	0.7248	0.7026
MesaQA	Gemma-7B	0.7031	0.7217	0.7045	0.7013
	Mistral-7B	0.7356	0.7585	0.7287	0.7223
	Falcon-7B	0.6458	0.6571	0.6122	0.5195
	Llama-2-7B	0.6567	0.6563	0.6594	0.6468
PubMedQA	Gemma-7B	0.6401	0.6540	0.6434	0.6319

Appendix

Table 6: Mean ContextSim per model across temperatures. CtxSim is more sensitive to temperature than GoldSim, yet remains stable for most models, with

T=0.3

yielding peak contextual faithfulness across both datasets before declining at higher temperatures.

Metric	KEA Baseline	Ours
Micro Precision	0.853	0.847
Micro Recall	0.738	0.949
Macro Precision	0.953	0.803
Macro Recall	0.641	0.946
Micro F1	0.836	0.895
Macro F1	0.622	0.782

Appendix

Table 7: Comparison of TAU alignment performance between our method and the KEA baseline, evaluated on manually annotated MesaQA and PubMed subsets.

	MRPC		PAWS-Wiki		STS12
Method	F1	AUROC	F1	AUROC	F1	AUROC
S3KG (Ours)	0.692	0.673	0.766	0.795	0.786	0.834
ROUGE-1	0.745	0.784	0.678	0.490	0.725	0.754
ROUGE-2	0.720	0.721	0.715	0.721	0.681	0.656
ROUGE-L	0.729	0.760	0.735	0.807	0.703	0.710
BLEU	0.687	0.677	0.716	0.747	0.671	0.644

Appendix

Table 8: Performance on Short-Text Datasets. S3KG leads on PAWS-Wiki (F1 0.766, AUROC 0.795) but trails sentence-T5-base on MRPC and STS12, where dense semantic representations have a natural advantage over structural signals.

Method	F1	AUROC
S3KG (Ours)	0.872	0.890
ROUGE-2	0.860	0.772
ROUGE-L	0.729	0.311
BLEU	0.868	0.745
BERTScore	0.747	0.645
MiniLM	0.821	0.811

Appendix

Table 9: Results on Wikipedia Entity-Swap (

N=400

). S3KG attains the highest F1 and AUROC across all retained baselines; ROUGE-1 is excluded as its near-perfect score is a token-overlap artifact rather than a meaningful signal.

	C400		Comb.		Find		GloBI		Oreg.
Method	F1	AUROC	F1	AUROC	F1	AUROC	F1	AUROC	F1	AUROC
S3KG (Ours)	0.932	0.973	0.834	0.829	0.767	0.796	0.892	0.935	0.812	0.892
ROUGE-1	0.835	0.917	0.732	0.728	0.745	0.745	0.784	0.833	0.745	0.782
ROUGE-2	0.822	0.894	0.707	0.711	0.717	0.706	0.776	0.833	0.752	0.791
ROUGE-L	0.792	0.855	0.722	0.717	0.719	0.721	0.763	0.800	0.792	0.835
BLEU	0.806	0.884	0.715	0.708	0.711	0.710	0.775	0.819	0.745	0.794

Appendix

Table 10: Performance on KG-Perturbed Paragraph Datasets. C400: SK-Codex 400; Comb.: SK-Combined; Find: SK-FindKG; Oreg.: SK-Oregano. S3KG leads on four of five datasets; sentence-T5-base outperforms on SK-FindKG, the only dataset where pure dense embeddings dominate (

\alpha=0.0

is optimal).

	MRPC		PAWS-Wiki		STS12		SK-Codex 400		SK-Combined
$\alpha$	F1	AUROC	F1	AUROC	F1	AUROC	F1	AUROC	F1	AUROC
0.0	0.676	0.654	0.694	0.739	0.783	0.828	0.871	0.969	0.776	0.801
0.1	0.681	0.664	0.745	0.781	0.786	0.834	0.922	0.973	0.791	0.825
0.2	0.683	0.669	0.760	0.790	0.780	0.834	0.927	0.973	0.819	0.829
0.3	0.692	0.673	0.764	0.793	0.780	0.831	0.927	0.973	0.828	0.830
0.4	0.680	0.674	0.764	0.795	0.780	0.827	0.929	0.973	0.833	0.829

Appendix

Table 11: Full S3KG

\alpha

sweep across short-text and combined datasets (F1 / AUROC). The best

\alpha

per dataset (by maximum F1) is bolded . Balanced datasets such as PAWS-Wiki and SK-Combined peak at

\alpha\in[0.4,0.6]

, while STS12 favours the KG-heavy end (

\alpha=0.1

	SK-FindKG		SK-GloBI		SK-Oregano		Wiki Swap
$\alpha$	F1	AUROC	F1	AUROC	F1	AUROC	F1	AUROC
0.0	0.767	0.796	0.811	0.898	0.777	0.884	0.821	0.892
0.1	0.760	0.798	0.883	0.934	0.803	0.892	0.872	0.890
0.2	0.762	0.793	0.888	0.937	0.811	0.892	0.869	0.890
0.3	0.748	0.788	0.889	0.936	0.812	0.892	0.868	0.890
0.4	0.744	0.783	0.891	0.936	0.812	0.892	0.865	0.890

Appendix

Table 12: Full S3KG

\alpha

sweep across KG-perturbed and entity-swap datasets (F1 / AUROC). The best

\alpha

per dataset (by maximum F1) is bolded . Structure-dominated datasets (SK-FindKG, Wiki Swap) favour low

\alpha\in\{0.0,0.1\}

, confirming the WL kernel’s advantage on relational perturbations.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Evaluation of Contextual Understanding in Large Language Models
Sep 8, 2026Subavarshana Arumugam, Mamta Nallaretnam, Kithuni Wickramasinghe +4LLM Evaluation LLM Grounding
Department of Computer Science and Engineering, University of Moratuwa, Moratuwa, Sri Lanka · Department of Electrical and Computer Engineering, University of Miami, Coral Gables, Florida, USA.
ContextGuard: Structured Self-Auditing for Context Learning in Language Models
May 26, 2026Hongbo Jin, Chi Wang, Haoran Tang +5LLM Auditing Instruction Following
Peking University · SCUT · Tsinghua University
Improving Answer Extraction in Context-based Question Answering Systems Using LLMs
Jun 4, 2026Hafez Abdelghaffar, Ahmed Alansary, Ali HamdiLLM Fine-Tuning Question Answering
Faculty of Computer Science MSA University Giza, Egypt

Sep 8, 2026J/K move · Enter open · S save

Evaluation of Contextual Understanding in Large Language Models

Subavarshana Arumugam, Mamta Nallaretnam, Kithuni Wickramasinghe, Chamath Gunapala+3

Department of Computer Science and Engineering, University of Moratuwa, Moratuwa, Sri Lanka · Department of Electrical and Computer Engineering, University of Miami, Coral Gables, Florida, USA.

LLM Evaluation LLM Grounding LLM Reasoning with Graphs Knowledge Graphs

Open paper AskPDF arXiv

Large Language Models (LLMs) demonstrate impressive performance across diverse NLP tasks, yet their ability to exhibit genuine contextual understanding remains uncertain. Traditional evaluation metrics such as perplexity, BiLingual Evaluation Understudy (BLEU), or surface-level accuracy fail to reveal how well LLMs extract, integrate, and reason over contextual information--a gap particularly critical in question answering, where models must align responses with contextually grounded knowledge rather than memorized associations. We propose a novel knowledge graph-based evaluation framework introducing Semantic Structural Similarity for KGs (S3KG), a hybrid similarity measure integrating structural and semantic similarity into a continuous evaluation score, alongside a diagnostic framework for categorizing reasoning errors. To validate this pipeline, we evaluate S3KG against established metrics on a curated question-answer (QA) benchmark, demonstrating its effectiveness in measuring correctness, faithfulness, and interpretability in LLM-generated responses.

Abstract

LLM Evaluation

Explore similar work

Evaluation of Contextual Understanding in Large Language Models

ContextGuard: Structured Self-Auditing for Context Learning in Language Models

Improving Answer Extraction in Context-based Question Answering Systems Using LLMs