CHI: A Composite Hallucination Index Unifying Entity, Relation, and Quantity Dimensions for Summarization Evaluation

Praveenkumar Katwe; Rakesh Chandra Balabantaray; Kali Prasad Vittala

cs.CLSep 27, 2026

CHI: A Composite Hallucination Index Unifying Entity, Relation, and Quantity Dimensions for Summarization Evaluation

Authors: Praveenkumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad Vittala

Organizations: International Institute of Information Technology Bhubaneswar, India · Salesforce India Pvt Ltd, Bengaluru, India

Abstract

Faithfulness evaluation of abstractive summaries remains an open challenge, with existing metrics addressing only isolated hallucination types: factual entity errors, relational inconsistencies, or numerical fabrications, without capturing their co-occurrence or interaction. We introduce CHI (Composite Hallucination Index), the first unified hallucination metric that decomposes faithfulness errors into three orthogonal dimensions: entity hallucination (EHI), relation hallucination (RHI*), and quantity hallucination (QHI). Each dimension employs a shared softmax-normalized architecture over Venn diagram-derived factors representing extractiveness, positive hallucination, over-focus, negative hallucination, and lost focus. The novel QHI component introduces tolerance-aware numerical matching with exact, epsilon, derived, and temporal comparison modes. We fuse the three dimensions via harmonic mean to produce a single composite score that penalizes weakness in any dimension. We validate CHI on 800 source articles spanning four domains (news, medical, legal, financial) with summaries from five generation systems. Empirical results demonstrate that: (i) the three dimensions are statistically orthogonal (mean rho = 0.148), confirming they capture distinct error types; (ii) CHI achieves the highest system-level correlation with human judgments (rho = 0.66, p = 0.006) on SummEval, outperforming ROUGE (rho = 0.53), EHI (rho = 0.58), and all individual components; and (iii) ablation studies confirm that all three dimensions contribute unique variance, with the full composite outperforming any individual component while providing decomposable error diagnostics unavailable from single-score baselines. CHI provides practitioners with a decomposable, interpretable, and efficient faithfulness metric suitable for both offline evaluation and online monitoring of summarization systems.

Figures & tables

Input Texts Source (

I

) Reference (

R

) Generated (

G

)

\Big\downarrow

Entity Extraction spaCy NER

I_{E},R_{E},G_{E}

Relation Extraction SVO dep. parsing + Sentence-BERT matching

I_{R},R_{R},G_{R}

Quantity Extraction Regex + spaCy NER Tolerance-aware matching

I_{Q},R_{Q},G_{Q}

\Big\downarrow

5 Factors EF, PH, OF, NH, LF 6 Factors EF, PH, OF, NH, LH, LF 5 Factors QEF, QPH, QOF, QNH, QLF

\Big\downarrow

Softmax normalization

EHI

\in[0,1]

Entity faithfulness RHI ∗

\in[0,1]

Relation faithfulness QHI

\in[0,1]

Quantity faithfulness

Figure 1: CHI computation pipeline. Three parallel extraction branches process the same input texts, each producing Venn-diagram factors from the respective linguistic unit (entities, SVO relations, quantities). Softmax normalization yields three comparable sub-indices on

[0,1]

, fused via harmonic mean into the composite CHI score. The pipeline is fully deterministic and requires no API calls or GPU (CPU-only with optional GPU acceleration for Sentence-BERT in RHI ∗ ).

4 Domains

\times

200 articles = 800 source documents

\downarrow

5 Models : BART, T5, Extractive, Corrupted, Reference

\downarrow

4000 summaries

CHI metrics Baselines EHI, RHI ∗ , QHI, CHI ROUGE, BERTScore, AlignScore, SummaC

\downarrow

SummEval Human Judgments 100 art.

\times

16 sys., 3 annotators, 4 dims (1–5)

Figure 2: Experimental pipeline: data

\rightarrow

generation

\rightarrow

metric computation

\rightarrow

correlation with human judgments at system and summary levels.

Domain	Source	N	Avg. Words	Avg. Quant.
News	XSUM	200	410	3.2
Medical	PubMed	200	1,796	8.7
Legal	CNN/DM (filtered)	200	821	5.1
Financial	CNN/DM (filtered)	200	899	11.4

Table 1: Dataset statistics. “Avg. Quant.” denotes mean number of extractable quantities per source document. CNN/DM articles are filtered for domain relevance using keyword and section classifiers.

Correlation	Spearman $\rho$	Kendall $\tau$
RHI ∗ vs. Linear RHI	0.968	0.878
RHI ∗ vs. Human (system-level)	0.506	0.333
Linear RHI vs. Human (system-level)	0.510	0.340

Table 2: Validation of RHI ∗ against the published linear RHI [ 6 ] . High inter-variant correlation confirms architectural consistency.

Metric	System-Level		Summary-Level
	$\rho$	$\tau$	$\rho$	$\tau$
QHI	0.39	0.27	0.080	0.064
ROUGE-L (quant. subset)	0.32	0.21	0.095	0.072
BERTScore (quant. subset)	—	—	0.067	0.048
AlignScore (quant. subset)	—	—	$-$ 0.12	$-$ 0.09

Table 3: QHI standalone performance on quantity-specific human judgments. “quant. subset” restricts evaluation to sentences containing numerical content.

	EHI	RHI ∗	QHI
EHI	1.000	0.195	$-$ 0.149
RHI ∗	0.195	1.000	$-$ 0.100
QHI	$-$ 0.149	$-$ 0.100	1.000
Mean pairwise $\|\rho\|$ : 0.148 (all $<0.3$ )

Table 4: Pairwise Spearman correlations between CHI dimensions. Low correlations (

\bar{\rho}<0.3

) confirm orthogonality; each dimension captures distinct hallucination phenomena.

	Consist.	Coher.	Relev.	Avg.	Rank
CHI	0.25	0.54	0.70	0.66	1
EHI	0.15	0.49	0.62	0.58	2
RHI ∗	0.51	0.38	0.50	0.56	3
ROUGE-2	0.13	0.38	0.62	0.55	4
ROUGE-1	0.16	0.36	0.61	0.53	5
MiniCheck	0.35	0.32	0.38	0.41	6

Figure 3: System-level Spearman

\rho

heatmap: each metric (row) vs. each human evaluation dimension (column) on SummEval (16 systems). Darker green = stronger positive correlation. CHI achieves the highest Average correlation (0.66) and dominates on Relevance (0.70). RHI ∗ leads on Consistency (0.51), the faithfulness-specific dimension.

Metric	System-Level		Summary-Level		Cost/doc	Interpretable	Decomposable
Metric	$\rho$	$\tau$	$\rho$	$\tau$	Cost/doc	Interpretable	Decomposable
ROUGE-L	0.39	0.27	0.139	0.109	$0.00	✓	—
BERTScore	—	—	0.097	0.076	$0.00	—	—
AlignScore	—	—	0.140	0.105	$0.00	—	—
SummaC	—	—	0.134	0.098	$0.00	—	—
MiniCheck	0.41	0.30	0.380	0.289	$0.00	—	—

Table 5: System-level and summary-level Spearman

\rho

with human judgments on SummEval (100 articles, 16 systems). System-level: correlation of system averages with average human score across 4 dimensions. Summary-level: correlation with human consistency across all 1600 summaries. CHI achieves the highest system-level correlation (

\rho=0.66

p=0.006

) while providing decomposable diagnostics unavailable from single-score metrics.

Configuration	System		Summary
	$\rho$	$\tau$	$\rho$	$\tau$
EHI only	0.58	0.47	0.058	0.046
RHI ∗ only	0.56	0.45	—	—
QHI only	0.39	0.27	0.080	0.064
EHI + RHI ∗	0.62	0.48	—	—
EHI + QHI	0.60	0.46	0.084	0.066

Table 6: Ablation study. Each dimension contributes unique variance; the full 3-way composite significantly outperforms all subsets (

p<0.05

, Williams’ test).

Human Dimension	CHI	Best Baseline	$\Delta$	Key Dim.
Consistency	0.25	RHI ∗ (0.51)	$-$ 0.26	RHI ∗
Coherence	0.54 ∗	EHI (0.49)	+0.05	EHI
Relevance	0.70 ∗∗	ROUGE-2 (0.62)	+0.08	All
Average (all 4)	0.66 ∗∗	EHI (0.58)	+0.08	All

Table 7: System-level Spearman

\rho

(16 systems, SummEval) against each human evaluation dimension. CHI achieves the highest correlation with Relevance (

\rho=0.70

p=0.002

) and the overall Average (

\rho=0.66

p=0.006

{}^{*}p<0.05

{}^{**}p<0.01

Metric	Time/doc (s)	Cost/doc ($)	GPU Required
ROUGE-L	0.01	0.000	No
BERTScore	0.50	0.000	Yes (recommended)
AlignScore	0.80	0.000	Yes (recommended)
MiniCheck	7.6	0.000	No †
CHI (ours)	4.3	0.000	No †
GPT-4-judge	3.0	0.150	No (API)

Table 8: Computational cost comparison. † Both MiniCheck and CHI run on CPU only; GPU accelerates inference but is not required. CHI and MiniCheck are both free to run (no API cost), comparable in speed ( 4–8s/doc on CPU), and require no external services. GPT-4-judge requires API access at $0.15/document.

Domain	% Entity	% Relation	% Quantity
News	38.2	42.5	19.3
Medical	28.4	51.8	19.8
Legal	33.6	48.9	17.5
Financial	25.1	35.7	39.2

Table 9: Distribution of hallucination errors by type across domains. Financial texts show disproportionate quantity errors; medical texts show high relational error rates.

Metric	Good Summary	Bad Summary	Gap	Correct Rank?
EHI (plain)	0.272	0.298	$-$ 0.026	NO
EHI (UMLS)	0.470	0.345	+0.125	YES
CHI (plain)	0.299	0.339	$-$ 0.040	NO
CHI (UMLS-enhanced)	0.353	0.357	$-$ 0.004	$\sim$

Table 10: Impact of UMLS normalization on a medical example where the “good” summary uses synonyms (heart attack, aspirin, Glucophage) and the “bad” summary fabricates conditions. Without UMLS, plain EHI incorrectly ranks the bad summary higher. UMLS normalization restores correct ranking by resolving synonyms to canonical concepts (NH:

1.0\rightarrow 0.22

; EF:

0.0\rightarrow 0.72

Explore similar work

A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarization
Aug 8, 2026Praveen Kumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad Vittala +1Citation Hallucination Detection Summarization
Dept. of CSE IIIT Bhubaneswar Bhubaneswar, Odisha, India · Informatica Bengaluru, Karnataka, India · TEKsystems Global Services Bengaluru, Karnataka, India
Preserving What Matters: Semantic Scaffolds Beyond Saturation in Summarization Evaluation
Sep 18, 2026Nikhil Reddy Pottanigari, Ramin Fahimi, Noah Bolger +2Summarization Scaffolds
ServiceNow Canada
Abstractiveness Metrics for Evaluating Text Summarization: A Refined Formulation with Empirical Validation
Jul 12, 2026Praveenkumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad VittalaSummarization Evaluation Metrics
Department of Computer Science and Engineering, International Institute of Information Technology, Bhubaneswar, India · Salesforce India Pvt Ltd, Bengaluru, India

Aug 8, 2026, cs.CLJ/K move · Enter open · S save

A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarization

Praveen Kumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad Vittala, Naman Kabadi

Dept. of CSE IIIT Bhubaneswar Bhubaneswar, Odisha, India · Informatica Bengaluru, Karnataka, India · TEKsystems Global Services Bengaluru, Karnataka, India

Citation Hallucination Detection Summarization Relation Extraction Entailment Extraction Decomposition

Open paper AskPDF arXiv

Abstractive text summarization systems frequently generate fluent yet unfaithful summaries by fabricating or distorting relationships between entities and events. Such relation-level hallucinations undermine the reliability of generated summaries, particularly in high-stakes domains. In this work, we present a refined and grounded framework for evaluating relation hallucination in abstractive summarization. We present the empirical Relation Hallucination Index (RHI) by introducing a dependency-aware relation extraction algorithm that incorporates lemmatization-based normalization, named entity grounded subject resolution, passive agent recovery, negation-aware verb modeling, reporting verb filtering, nominal relation fallback, clausal propagation, and systematic deduplication. These enhancements improve the structural fidelity of extracted relation triples and reduce spurious matches during evaluation. In addition, we introduce a normalized formulation of RHI to ensure scale-invariant comparison between datasets and models. The revised metric decomposes hallucination into interpretable components, aggregates relation hallucination metric into a normalized relation faithfulness score. Extensive evaluation across multiple state-of-the-art summarization models demonstrates that the grounded extraction process yields more stable and discriminative hallucination measurements. The proposed framework advances automated relation-level faithfulness evaluation and supports coherence-aware, hallucination-sensitive model analysis.

Abstract

Explore similar work

A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarization

Preserving What Matters: Semantic Scaffolds Beyond Saturation in Summarization Evaluation

Abstractiveness Metrics for Evaluating Text Summarization: A Refined Formulation with Empirical Validation