Scoped Semantic Invariance as a Foundation for LLM Reliability

Rosario Uceda-Sosa; Maria Chang; Karthikeyan Natesan Ramamurthy; Moninder Singh

cs.AIOct 23, 2026

Scoped Semantic Invariance as a Foundation for LLM Reliability

Authors: Rosario Uceda-Sosa, Maria Chang, Karthikeyan Natesan Ramamurthy, Moninder Singh

Organizations: IBM T. J. Watson Research Center, New York, USA

Abstract

LLMs often answer semantically equivalent queries inconsistently, even when the underlying facts are known. We formalize consistency as semantic invariance over equivalence classes of queries and show that invariance under arbitrary meaning-preserving transformations is undecidable in general. This motivates scoped semantic invariance, where model behavior is evaluated relative to bounded semantic domains. We operationalize this framework using knowledge graphs (KGs), which act as explicit semantic control structures defining local semantic constraints and grounding contexts. Across the MITRE D3FEND ontology and eight Wikidata- derived domains, we show that inconsistencies are structured rather than random, concentrating in localized hotspot subgraphs organized around recurring TREE, STAR, and CYCLE patterns that remain stable across models and domains. We further show that selective graph grounding improves correctness and cross-model consistency, while indiscriminate grounding introduces regressions. A layered grounding strategy achieves the strongest monotonicity profile, preserving correct knowledge while reducing inconsistency. Finally, we show that hotspot structures discovered from simple consistency probes transfer to more realistic compositional reasoning tasks.

Figures & tables

Figure 1: Pipeline overview. CHECK establishes baseline consistency and accuracy. GAUGE identifies hotspot subgraphs and structural failure patterns. GROUND applies selective semantic grounding strategies.

Figure 2: Examples of hotspot subgraph patterns. Red edges denote inconsistent relations accepted by the model; green edges denote corrective structural grounding derived from the KG.

Approach	Bal Acc (%)	Bal Acc $\Delta$ from BL	Consistency (%)	Consistency $\Delta$ from BL
BL	86.469	0.000	95.350	0.000
NRAG	96.871	10.402	97.994	2.644
VO	96.659	10.190	97.888	2.538
PAT	96.871	10.402	97.994	2.644
LAY	96.935	10.466	98.008	2.657

Table 1: Balanced accuracy and consistency results pooled across clusters from Baseline (BL), Naive Rag (NRAG), Verdict-Only (VO), Patterns (PAT) and Layered (LAY).

Domain	Patterns	% STAR	% TREE	% CYCLE	Hot Edges	Total Edges	Density
academic disciplines	20	70.00	25.00	5.00	49	411	11.92
dishes	43	51.16	34.88	13.95	128	1038	12.33
finance	35	65.71	25.71	8.57	104	592	17.57
home appliances	29	41.38	20.69	37.93	71	491	14.46
medical specialties	34	61.76	20.59	17.65	117	706	16.57
music genres	44	50.00	47.73	2.27	200	1921	10.41

Table 2: Hotspot subgraph patterns obtained from the baseline (BL) across domains.

Model	Patt’s	% STAR	% TREE	% CYCLE	Hot Edges	# Edges	Density
DeepSeek-V3.2	1574	53.30	43.70	3.00	2723	12774	21.30
Mistral-Small-3.2-24B	1751	52.90	44.00	3.00	3330	12774	26.10
Qwen3.5	1507	50.80	45.40	3.80	2402	12774	18.80
gpt-oss-120b	1278	53.10	43.70	3.20	2208	12774	17.30
granite-4.0-h-small	1571	52.30	44.40	3.20	2770	12774	21.70
llama-3-3-70b-instruct	1108	50.70	44.30	5.00	1706	12774	13.40

Table 3: Hotspot subgraph patterns after running the baseline across models.

Held-out Model	# Edges	Base	Prec.	Recall	Acc.	Lift
Qwen3.5-397B-A17B-FP8	12774	0.239	0.733	0.838	0.888	3.06
DeepSeek-V3.2	12774	0.285	0.860	0.768	0.899	3.02
granite-4.0-h-small	12774	0.302	0.838	0.704	0.869	2.77
llama-3-3-70b-instruct	12774	0.194	0.566	0.808	0.843	2.92
Mistral-Small-3.2-24B	12774	0.392	0.955	0.581	0.825	2.43
gpt-oss-120b	12774	0.208	0.630	0.836	0.863	3.02

Table 4: Leave-one-model-out prediction of edge hotness. For each held-out model, an edge is predicted hot if it is hot for

\geq 1

of the other five models; Base is the held-out model’s true hot rate.

Approach	Avg Tokens	Avg Triples	Fixed	%Fixed	Broken	%Broken	F/B Ratio	KRR (%)
NRAG	808	50	11030	12.1	1520	1.7	7.3 $\times$	97.05
VO	30	1	10993	12.2	1620	1.8	6.8 $\times$	97.79
PAT	182	3.5	11209	12.3	1518	1.6	7.4 $\times$	97.95
LAY	182	3.5	11238	12.4	1422 ∗	1.6 ∗	7.9 $\times$	98.06

Table 5: Cluster-level ground per approach. ∗ LAY introduces no new breaks; reported breaks persist from earlier grounding stages.

Pattern	Baseline	VO	LAY	VO lift	LAY lift	LAY – VO lift
CYCLE	0.55	0.94	0.97	0.39	0.42	0.03
STAR	0.51	0.85	0.87	0.34	0.36	0.02
TREE	0.55	0.93	0.93	0.38	0.38	0.00

Table 6: Baseline accuracy and lift comparisons across structural pattern types.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Cluster type	Clusters	Description
edge-yes	2,045	Direct subconcept edge in the ontology. Expected: yes .
edge-inv	2,045	Inverse of a direct edge; tests asymmetry of the subconcept relation. Expected: no .
edge-no	1,227	Verified non-edge between co-occurring concepts. Expected: no .
hierarchy	720	Transitive (multi-hop) subconcept relationship; requires reasoning over intermediate nodes. Expected: yes .
property-hierarchy	1,943	Relationship via a non-taxonomic property (e.g., connected_to ); requires relational semantics beyond pure taxonomy. Expected: yes .
Total	7,980

Appendix

Table 7: Dataset composition for the SI-cyber domain.

Subgraph	Nodes	Edges	Hot Frac.	Consistency
HS_0007	25	50	1.000	0.757
HS_0003	39	40	1.000	0.707
HS_0006	27	26	1.000	0.741
HS_0004	22	23	1.000	0.688
HS_0010	13	13	1.000	0.719

Appendix

Table 8: Top-5 hotspot subgraphs in SI-cyber , ranked by edge count. Mean balanced accuracy is not available at the subgraph level since per-edge balanced accuracy requires per-model data (see Table 6 ).

Target concept	# Edges	Property	Failure axis
Defensive Technique	30	subConceptOf	Axis 1 — Taxonomic
Executable File	26	connected to	Axis 2 — Relational
Software	23	subConceptOf	Axis 1 — Taxonomic
Firmware	18	connected to	Axis 2 — Relational
Operating System	15	connected to	Axis 2 — Relational
Hardware Device	15	connected to	Axis 2 — Relational

Appendix

Table 9: Top-10 target nodes in hotspot edges for SI-CYBER, grouped by failure axis. The two axes together account for 78% of all hotspot edges.

Model	# Params	Max tokens	Answer trigger	Chat template
gpt-oss-120b	120B	500	(none)	harmony
Llama-3.3-70B	70B	20	Answer:	default
Qwen3.5-397B	397B	20	(none)	default*
DeepSeek-V3.2	685B	20	Answer (Yes or No):	deepseek
Granite-4.0-h-small	32B	20	Answer:	default
Mistral-Small-24B	24B	20	Answer:	mistral

Appendix

Table 10: Models, model parameters, and model-specific prompt settings (max output tokens, answer trigger, and chat template). *For Qwen3.5-397B, the default chat template was used with enable_thinking set to false .

Approach	Avg Tokens	Avg Triples	Fixed	%Fixed	Broken	%Broken	F/B Ratio	KRR (%)
NRAG	808	50.00	61768	12.80	8367	1.80	7.26 $\times$	98.20
VO	30	1.00	69276	14.40	6973	1.50	9.93 $\times$	98.40
PAT	182	3.50	70443	14.50	6981	1.40	10.09 $\times$	98.50
LAY	182	3.50	70502	14.70	6149 ∗	1.30 ∗	11.47 $\times$	98.60

Appendix

Table 11: Query-level grounding per approach. ∗ LAY introduces no new breaks; reported breaks persist from earlier grounding stages.

Dataset	# Question Clusters	# Questions	# Input Tokens
SI-Cyber	7,980	45,232	519,187
Wikidata Domains	7,244	35,560	367,353
Curated complex queries	n/a	25	1,540
Total	15,224	80,817	888,080

Appendix

Table 12: Number of question clusters, questions (i.e. inference calls), and input tokens for each of the three datasets used. These provide an infrastructure- and model-agnostic picture of the inference requirements of our experiments.

Figure 5: Balanced accuracy and consistency improvements from baseline (BL) for Verdict-Only (VO), Pattern (PAT) and Layered (LAY) averaged across domains.

Figure 6: Balanced accuracy and consistency improvements from baseline (BL) for Verdict-Only (VO), Pattern (PAT) and Layered (LAY) per model.

Edge	Agg. acc	Agg. cons	n_hot	Hot?
church music $\rightarrow$ liturgical music (exp.: yes)	0.58	0.75	5/6	Yes
liturgical music $\rightarrow$ church music (exp.: no)	0.08	0.92	6/6	Yes

Appendix

Table 13: Aggregate accuracy, consistency, hot fraction of Cycle between Liturgical music and Church music

Edge	Agg. acc	Agg. cons	n_hot	Hot?
pediatric_neurosurgeon $\rightarrow$ childhood disease	0.67	0.67	5/6	Yes
pediatric_neurosurgeon $\rightarrow$ neurology	0.85	0.85	3/6	Yes
pediatric_neurosurgeon $\rightarrow$ pediatric surgery	0.90	0.90	2/6	Yes
pediatric_neurosurgeon $\rightarrow$ pediatrics	0.92	0.92	2/6	Yes

Appendix

Table 14: Aggregate accuracy, consistency, hot fraction of STAR with hub neurosurgeon

Edge	Agg. acc	Agg. cons	n_hot	Hot?
GNU Project $\rightarrow$ operating system	0.75	0.75	6/6	Yes
GNU Project $\rightarrow$ Unix-like OS	0.75	0.75	6/6	Yes
GNU Project $\rightarrow$ GNU/Linux	0.71	0.71	5/6	Yes
Linux $\rightarrow$ operating system	0.83	0.83	4/6	Yes
GNU/Linux $\rightarrow$ operating system	0.88	0.88	3/6	Yes
Linux $\rightarrow$ freely redistributable software	0.85	0.85	2/6	Yes

Appendix

Table 15: Aggregate accuracy, consistency, hot fraction of TREE from Operating System

Metric	BL	VO	PAT	LAY
Raw Accuracy	0.690	0.920	0.950	0.960
Balanced Accuracy	0.515	0.900	0.903	0.919
Cross-model Consistency	0.868	0.948	0.974	–

Appendix

Table 16: Ensemble (majority vote across five models).

Type	BL	VO	PAT	LAY
cycle_directionality	0.750	1.000	0.938	1.000
generality_with_evidence	0.464	0.964	0.964	0.964
graph_explanation_hub	0.000	0.750	0.750	0.750
transitive_justification	0.846	0.885	0.962	0.962

Appendix

Table 17: Balanced accuracy by question type for the ensemble.

Reasoning Consistency in Language Models

Reasoning consistency in language models measures whether conclusions and reasoning remain reliable across equivalent prompts, samples, steps, and contexts.

See all 122 papers

Semantic Invariance in Agentic AI
Mar 13, 2026I. de Zarzà, J. de Curtò, Jordi Cabot +2AI Agent Reliability Language Model Robustness
Human Centered AI, Data & Software, LUXEMBOURG Institute of Science and Technology, L-4362 Esch-sur-Alzette, Luxembourg · Department of Computer Applications in Science & Engineering, BARCELONA Supercomputing Center, 08034 Barcelona, Spain · Escuela T´ecnica Superior de Ingenier´ıa (ICAI), Universidad Pontificia Comillas, 28015 Madrid, Spain +2
Can We Trust LLM's Logic? Quantifying Uncertainty, Coherence, and Robustness via a Graph-Based Framework
Jul 9, 2026Riccardo Revalor, Jalees Rehman, Debjit PalCoT Faithfulness LLM Reasoning with Graphs
¶Department of Electrical and Computer Engineering · ‡Department of Biochemistry and Molecular Genetics
LLM Self-Explanations Fail Semantic Invariance
Mar 1, 2026Stefan SzeiderFaithfulness of Language Model Explanations Language Model Self-Assessment
Algorithms and Complexity Group, TU Wien, Vienna, Austria

Mar 13, 2026J/K move · Enter open · S save

Semantic Invariance in Agentic AI

I. de Zarzà, J. de Curtò, Jordi Cabot, Pietro Manzoni+1

Human Centered AI, Data & Software, LUXEMBOURG Institute of Science and Technology, L-4362 Esch-sur-Alzette, Luxembourg · Department of Computer Applications in Science & Engineering, BARCELONA Supercomputing Center, 08034 Barcelona, Spain · Escuela T´ecnica Superior de Ingenier´ıa (ICAI), Universidad Pontificia Comillas, 28015 Madrid, Spain+2

AI Agent Reliability Language Model Robustness Metamorphic Testing Scientific Reasoning in Language Models

Open paper AskPDF arXiv

Large Language Models (LLMs) increasingly serve as autonomous reasoning agents in decision support, scientific problem-solving, and multi-agent coordination systems. However, deploying LLM agents in consequential applications requires assurance that their reasoning remains stable under semantically equivalent input variations, a property we term semantic invariance. Standard benchmark evaluations, which assess accuracy on fixed, canonical problem formulations, fail to capture this critical reliability dimension. To address this shortcoming, in this paper we present a metamorphic testing framework for systematically assessing the robustness of LLM reasoning agents, applying eight semantic-preserving transformations (identity, paraphrase, fact reordering, expansion, contraction, academic context, business context, and contrastive formulation) across seven foundation models spanning four distinct architectural families: Hermes (70B, 405B), Qwen3 (30B-A3B, 235B-A22B), DeepSeek-R1, and gpt-oss (20B, 120B). Our evaluation encompasses 19 multi-step reasoning problems across eight scientific domains. The results reveal that model scale does not predict robustness: the smaller Qwen3-30B-A3B achieves the highest stability (79.6% invariant responses, semantic similarity 0.91), while larger models exhibit greater fragility.

Abstract

Reasoning Consistency in Language Models

Explore similar work

Semantic Invariance in Agentic AI

Can We Trust LLM's Logic? Quantifying Uncertainty, Coherence, and Robustness via a Graph-Based Framework

LLM Self-Explanations Fail Semantic Invariance