CHI: A Composite Hallucination Index Unifying Entity, Relation, and Quantity Dimensions for Summarization Evaluation

Praveenkumar Katwe; Rakesh Chandra Balabantaray; Kali Prasad Vittala

cs.CLSep 27, 2026

CHI: A Composite Hallucination Index Unifying Entity, Relation, and Quantity Dimensions for Summarization Evaluation

Authors: Praveenkumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad Vittala

Organizations: International Institute of Information Technology Bhubaneswar, India · Salesforce India Pvt Ltd, Bengaluru, India

Abstract

Faithfulness evaluation of abstractive summaries remains an open challenge, with existing metrics addressing only isolated hallucination types: factual entity errors, relational inconsistencies, or numerical fabrications, without capturing their co-occurrence or interaction. We introduce CHI (Composite Hallucination Index), the first unified hallucination metric that decomposes faithfulness errors into three orthogonal dimensions: entity hallucination (EHI), relation hallucination (RHI*), and quantity hallucination (QHI). Each dimension employs a shared softmax-normalized architecture over Venn diagram-derived factors representing extractiveness, positive hallucination, over-focus, negative hallucination, and lost focus. The novel QHI component introduces tolerance-aware numerical matching with exact, epsilon, derived, and temporal comparison modes. We fuse the three dimensions via harmonic mean to produce a single composite score that penalizes weakness in any dimension. We validate CHI on 800 source articles spanning four domains (news, medical, legal, financial) with summaries from five generation systems. Empirical results demonstrate that: (i) the three dimensions are statistically orthogonal (mean rho = 0.148), confirming they capture distinct error types; (ii) CHI achieves the highest system-level correlation with human judgments (rho = 0.66, p = 0.006) on SummEval, outperforming ROUGE (rho = 0.53), EHI (rho = 0.58), and all individual components; and (iii) ablation studies confirm that all three dimensions contribute unique variance, with the full composite outperforming any individual component while providing decomposable error diagnostics unavailable from single-score baselines. CHI provides practitioners with a decomposable, interpretable, and efficient faithfulness metric suitable for both offline evaluation and online monitoring of summarization systems.

Figures & tables

Input Texts Source (

I

) Reference (

R

) Generated (

G

)

\Big\downarrow

Entity Extraction spaCy NER

I_{E},R_{E},G_{E}

Relation Extraction SVO dep. parsing + Sentence-BERT matching

I_{R},R_{R},G_{R}

Quantity Extraction Regex + spaCy NER Tolerance-aware matching

I_{Q},R_{Q},G_{Q}

\Big\downarrow

5 Factors EF, PH, OF, NH, LF 6 Factors EF, PH, OF, NH, LH, LF 5 Factors QEF, QPH, QOF, QNH, QLF

\Big\downarrow

Softmax normalization

EHI

\in[0,1]

Entity faithfulness RHI ∗

\in[0,1]

Relation faithfulness QHI

\in[0,1]

Quantity faithfulness

Figure 1: CHI computation pipeline. Three parallel extraction branches process the same input texts, each producing Venn-diagram factors from the respective linguistic unit (entities, SVO relations, quantities). Softmax normalization yields three comparable sub-indices on

[0,1]

, fused via harmonic mean into the composite CHI score. The pipeline is fully deterministic and requires no API calls or GPU (CPU-only with optional GPU acceleration for Sentence-BERT in RHI ∗ ).

4 Domains

\times

200 articles = 800 source documents

\downarrow

5 Models : BART, T5, Extractive, Corrupted, Reference

\downarrow

4000 summaries

CHI metrics Baselines EHI, RHI ∗ , QHI, CHI ROUGE, BERTScore, AlignScore, SummaC

\downarrow

SummEval Human Judgments 100 art.

\times

16 sys., 3 annotators, 4 dims (1–5)

Figure 2: Experimental pipeline: data

\rightarrow

generation

\rightarrow

metric computation

\rightarrow

correlation with human judgments at system and summary levels.

Domain	Source	N	Avg. Words	Avg. Quant.
News	XSUM	200	410	3.2
Medical	PubMed	200	1,796	8.7
Legal	CNN/DM (filtered)	200	821	5.1
Financial	CNN/DM (filtered)	200	899	11.4

Table 1: Dataset statistics. “Avg. Quant.” denotes mean number of extractable quantities per source document. CNN/DM articles are filtered for domain relevance using keyword and section classifiers.

Correlation	Spearman $\rho$	Kendall $\tau$
RHI ∗ vs. Linear RHI	0.968	0.878
RHI ∗ vs. Human (system-level)	0.506	0.333
Linear RHI vs. Human (system-level)	0.510	0.340

Table 2: Validation of RHI ∗ against the published linear RHI [ 6 ] . High inter-variant correlation confirms architectural consistency.

Metric	System-Level		Summary-Level
	$\rho$	$\tau$	$\rho$	$\tau$
QHI	0.39	0.27	0.080	0.064
ROUGE-L (quant. subset)	0.32	0.21	0.095	0.072
BERTScore (quant. subset)	—	—	0.067	0.048
AlignScore (quant. subset)	—	—	$-$ 0.12	$-$ 0.09

Table 3: QHI standalone performance on quantity-specific human judgments. “quant. subset” restricts evaluation to sentences containing numerical content.

	EHI	RHI ∗	QHI
EHI	1.000	0.195	$-$ 0.149
RHI ∗	0.195	1.000	$-$ 0.100
QHI	$-$ 0.149	$-$ 0.100	1.000
Mean pairwise $\|\rho\|$ : 0.148 (all $<0.3$ )

Table 4: Pairwise Spearman correlations between CHI dimensions. Low correlations (

\bar{\rho}<0.3

) confirm orthogonality; each dimension captures distinct hallucination phenomena.

	Consist.	Coher.	Relev.	Avg.	Rank
CHI	0.25	0.54	0.70	0.66	1
EHI	0.15	0.49	0.62	0.58	2
RHI ∗	0.51	0.38	0.50	0.56	3
ROUGE-2	0.13	0.38	0.62	0.55	4
ROUGE-1	0.16	0.36	0.61	0.53	5
MiniCheck	0.35	0.32	0.38	0.41	6

Figure 3: System-level Spearman

\rho

heatmap: each metric (row) vs. each human evaluation dimension (column) on SummEval (16 systems). Darker green = stronger positive correlation. CHI achieves the highest Average correlation (0.66) and dominates on Relevance (0.70). RHI ∗ leads on Consistency (0.51), the faithfulness-specific dimension.

Metric	System-Level		Summary-Level		Cost/doc	Interpretable	Decomposable
Metric	$\rho$	$\tau$	$\rho$	$\tau$	Cost/doc	Interpretable	Decomposable
ROUGE-L	0.39	0.27	0.139	0.109	$0.00	✓	—
BERTScore	—	—	0.097	0.076	$0.00	—	—
AlignScore	—	—	0.140	0.105	$0.00	—	—
SummaC	—	—	0.134	0.098	$0.00	—	—
MiniCheck	0.41	0.30	0.380	0.289	$0.00	—	—

Table 5: System-level and summary-level Spearman

\rho

with human judgments on SummEval (100 articles, 16 systems). System-level: correlation of system averages with average human score across 4 dimensions. Summary-level: correlation with human consistency across all 1600 summaries. CHI achieves the highest system-level correlation (

\rho=0.66

p=0.006

) while providing decomposable diagnostics unavailable from single-score metrics.

Configuration	System		Summary
	$\rho$	$\tau$	$\rho$	$\tau$
EHI only	0.58	0.47	0.058	0.046
RHI ∗ only	0.56	0.45	—	—
QHI only	0.39	0.27	0.080	0.064
EHI + RHI ∗	0.62	0.48	—	—
EHI + QHI	0.60	0.46	0.084	0.066

Table 6: Ablation study. Each dimension contributes unique variance; the full 3-way composite significantly outperforms all subsets (

p<0.05

, Williams’ test).

Human Dimension	CHI	Best Baseline	$\Delta$	Key Dim.
Consistency	0.25	RHI ∗ (0.51)	$-$ 0.26	RHI ∗
Coherence	0.54 ∗	EHI (0.49)	+0.05	EHI
Relevance	0.70 ∗∗	ROUGE-2 (0.62)	+0.08	All
Average (all 4)	0.66 ∗∗	EHI (0.58)	+0.08	All

Table 7: System-level Spearman

\rho

(16 systems, SummEval) against each human evaluation dimension. CHI achieves the highest correlation with Relevance (

\rho=0.70

p=0.002

) and the overall Average (

\rho=0.66

p=0.006

{}^{*}p<0.05

{}^{**}p<0.01

Metric	Time/doc (s)	Cost/doc ($)	GPU Required
ROUGE-L	0.01	0.000	No
BERTScore	0.50	0.000	Yes (recommended)
AlignScore	0.80	0.000	Yes (recommended)
MiniCheck	7.6	0.000	No †
CHI (ours)	4.3	0.000	No †
GPT-4-judge	3.0	0.150	No (API)

Table 8: Computational cost comparison. † Both MiniCheck and CHI run on CPU only; GPU accelerates inference but is not required. CHI and MiniCheck are both free to run (no API cost), comparable in speed ( 4–8s/doc on CPU), and require no external services. GPT-4-judge requires API access at $0.15/document.

Domain	% Entity	% Relation	% Quantity
News	38.2	42.5	19.3
Medical	28.4	51.8	19.8
Legal	33.6	48.9	17.5
Financial	25.1	35.7	39.2

Table 9: Distribution of hallucination errors by type across domains. Financial texts show disproportionate quantity errors; medical texts show high relational error rates.

Metric	Good Summary	Bad Summary	Gap	Correct Rank?
EHI (plain)	0.272	0.298	$-$ 0.026	NO
EHI (UMLS)	0.470	0.345	+0.125	YES
CHI (plain)	0.299	0.339	$-$ 0.040	NO
CHI (UMLS-enhanced)	0.353	0.357	$-$ 0.004	$\sim$

Table 10: Impact of UMLS normalization on a medical example where the “good” summary uses synonyms (heart attack, aspirin, Glucophage) and the “bad” summary fabricates conditions. Without UMLS, plain EHI incorrectly ranks the bad summary higher. UMLS normalization restores correct ranking by resolving synonyms to canonical concepts (NH:

1.0\rightarrow 0.22

; EF:

0.0\rightarrow 0.72

Explore similar work

Aug 8, 2026cs.CL

A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarization

Abstractive text summarization systems frequently generate fluent yet unfaithful summaries by fabricating or distorting relationships between entities and events. Such relation-level hallucinations undermine the reliability of generated summaries, particularly in high-stakes domains. In this work, we present a refined and grounded framework for evaluating relation hallucination in abstractive summarization. We present the empirical Relation Hallucination Index (RHI) by introducing a dependency-aware relation extraction algorithm that incorporates lemmatization-based normalization, named entity grounded subject resolution, passive agent recovery, negation-aware verb modeling, reporting verb filtering, nominal relation fallback, clausal propagation, and systematic deduplication. These enhancements improve the structural fidelity of extracted relation triples and reduce spurious matches during evaluation. In addition, we introduce a normalized formulation of RHI to ensure scale-invariant comparison between datasets and models. The revised metric decomposes hallucination into interpretable components, aggregates relation hallucination metric into a normalized relation faithfulness score. Extensive evaluation across multiple state-of-the-art summarization models demonstrates that the grounded extraction process yields more stable and discriminative hallucination measurements. The proposed framework advances automated relation-level faithfulness evaluation and supports coherence-aware, hallucination-sensitive model analysis.

Praveen Kumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad Vittala +1

Dept. of CSE IIIT Bhubaneswar Bhubaneswar, Odisha, India · Informatica Bengaluru, Karnataka, India · TEKsystems Global Services Bengaluru, Karnataka, India

Citation Hallucination Detection Summarization Relation Extraction Entailment Extraction Decomposition

Ask PDF

Sep 18, 2026cs.CL

Preserving What Matters: Semantic Scaffolds Beyond Saturation in Summarization Evaluation

Summarization ships in countless production systems, making model selection a routine decision that depends on measuring summary quality. Existing metrics struggle to support this: ROUGE captures only surface overlap, while LLM-as-judge scores saturate to near-identical values that fail to rank models effectively. We observe this saturation across three public datasets, two proprietary datasets, and multilingual settings. Motivated by this, we introduce Semantic Scaffold, an evaluation framework that extracts a hierarchical representation of facts, questions, and entity attributes from a source text, labeling each as a main point or supporting detail, and reusing this structure as a fixed reference for scoring summaries. From this representation, we derive three diagnostic metrics: Fact Preservation Score (FPS), Question Preservation Score (QPS), and Entity Preservation Score (EPS), designed to reward the preservation of essential information while penalizing detail overload, and position them as interpretable diagnostics that remain informative where holistic axes collapse. Finally, we analyze four recurring failure modes of ROUGE and LLM-as-judge scores, demonstrating that scaffold-based evaluation remains informative where conventional metrics collapse.

Nikhil Reddy Pottanigari, Ramin Fahimi, Noah Bolger +2

ServiceNow Canada

Summarization Scaffolds Preservation Saturation

Ask PDF

Jul 12, 2026cs.CL

Abstractiveness Metrics for Evaluating Text Summarization: A Refined Formulation with Empirical Validation

Quantifying abstractiveness in generated summaries is essential for evaluating summarization models beyond surface-level metrics like ROUGE. We introduce Reference Abstraction (RA), Summary Abstraction (SA), and Abstraction Ratio (AR) -- a set of principled heuristic metrics that measure how much a summary diverges from extractive copying of the source text. The formulation uses the harmonic mean of document lengths modulated by a cubic non-overlap factor, yielding dimensionally consistent, bounded output with non-linear sensitivity to the extractive-abstractive boundary. Evaluation on 100 XSUM documents across four summarization models (BART-large-cnn, Pegasus-xsum, DistilBart, MT5-small) demonstrates that the metrics successfully discriminate between extractive models (SA ~ 0.12-0.26) and abstractive models (SA ~ 0.96-1.77), and that the Abstraction Ratio identifies summaries requiring manual evaluation for potential hallucination. Code and results are available at https://github.com/katweNLP/AbstractionStudy.

Praveenkumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad Vittala

Department of Computer Science and Engineering, International Institute of Information Technology, Bhubaneswar, India · Salesforce India Pvt Ltd, Bengaluru, India

Summarization Evaluation Metrics Ablation Extraction Empirical Study

Ask PDF