Inter-dimension Dependence for Multi-Dimensional Evaluation of Open-Ended Text

Haoyuan Li; Snigdha Chaturvedi

cs.CLAug 24, 2026

Inter-dimension Dependence for Multi-Dimensional Evaluation of Open-Ended Text

Authors: Haoyuan Li, Snigdha Chaturvedi

Organizations: University of North Carolina at Chapel Hill

Abstract

LLM-as-a-judge methods are widely used for evaluating the quality of generated open-ended text. Such evaluations are generally multi-dimensional, since the error patterns in texts can be different for different dimensions. Therefore, reliable LLM judges should evaluate each target dimension independently. To quantify the extent to which LLM judges depend on non-target dimensions when evaluating a target dimension, i.e., inter-dimension dependence, we propose CorrGap. To measure this, CorrGap uses the difference in correlations between LLM-predicted scores and ground truth scores across different groups of texts. Using CorrGap, we show that inter-dimension dependence is pervasive across LLM judges in open-ended text evaluation tasks. To mitigate inter-dimension dependence, we propose DimCheck, a method that iteratively removes unrelated evidence from COTs generated by LLM judges in a step-wise way. We show that DimCheck mitigates inter-dimension dependence and outperforms strong baselines across three LLMs and four tasks. We also show that smaller trained LLMs can approximate larger LLMs in DimCheck, with much lower inference costs.

Figures & tables

Target Dimension : Relevance

Non-target Dimension : Consistency, Coherence, Fluency

The summary includes some important information such as Ze Maria’s representation of Brazil and his role as the third coach at Ceahlaul. However, it is missing key details about the timeline of events (e.g., being reinstated and then fired again) and the context of the club’s relegation threat. Additionally, the summary is somewhat disjointed and lacks a clear narrative flow. It also omits the reason for his second firing and the replacement by Vanya Radinovic. These omissions reduce the relevance.

Table 1: An example COT generated by Llama3.3-70B-Instruct as a judge when evaluating a summary on relevance (the target dimension). The text in red is related to a non-target dimension, coherence, but not relevance.

Figure 1: Example workflow of DimCheck on a COT. The evidence which is unrelated to the target dimension and removed by Dimcheck in the current step is in red , evidence removed in all previous steps are with strikethrough .

	Task	# Dim.	#Inp.	#Out.
SummEval	news summarization	4	100	16
Topical-Chat	knowledge-grounded conversation	4	60	6
Hanna	narrative generaion	6	96	11
OpinSumm.	review summarization	4	10	14

Table 2: Datasets Overview. We show the task, number of evaluation dimensions (# Dim.), number of inputs (#Inp.) and number of outputs per input (#Out.).

	Llama3.3-70b		Gemma3-27b		Qwen3-32b
	Fact.	Analyze.	Fact.	Analyze	Fact.	Analyze.
CorrGap	5.9	20.5	5.6	12.0	5.0	22.4
w/o control	6.2	24.6	8.4	20.7	11.5	24.9

Table 3: Evaluation of CorrGap using FactScore (Fact.) and Analyze-Rate (Analyze.) on the SummEval dataset. The correlation differences that are statistically significant (

p<0.05

) are bolded . CorrGap correctly identify FactScore as having low inter-dimension dependence.

$\gamma$	SummEval		Topical Chat		Hanna		OpinSumm.		Average
	CG	w/o c.	CG	w/o c.	CG	w/o c.	CG	w/o c.	CG	w/o c.
100	0.0	9.1	0.0	0.0	0.0	0.0	0.5	0.9	0.1	2.5
67	0.0	0.1	0.0	0.1	0.0	0.0	5.0	6.0	1.3	1.6
33	0.8	2.3	3.5	6.1	0.0	0.1	14.6	18.9	4.7	6.8
0	23.8	12.9	23.9	20.0	23.2	21.3	24.3	24.8	23.8	19.8

Table 4: P-values of CorrGap (CG) and its variants without controlling the ground truth scores (w/o c.) on synthetic scores with different inter-dimension dependence (

\gamma

). The best-performing metric is bolded . CorrGap accurately distinguishes between synthetic scores with low or high inter-dimension dependence.

	SummEval	Topical Chat	Hanna	OpinSummEval
Low Variance	50.43	43.84	8.33	83.36
High Variance	50.58	45.24	8.50	81.02

Table 5: Inter-annotator agreements for ground-truth scores in low- and high-variance groups. Similar inter-annotator agreements between two groups show that ground-truth scores are less likely to be affected by inter-dimension dependence.

	SummEval[4]		Topical Chat[4]		Hanna[6]		OpinSumm.[4]		Avg.
	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	overall $\uparrow$
	G-Eval ( Liu et al., 2023 )
Llama3.1-8b	32.7	25.6(4)	41.7	9.2(2)	25.6	9.6(2)	29.6	12.4(0)	32.4	14.2	59.1
Llama3.3-70b	25.9	15.4(4)	34.1	11.6(2)	28.3	11.4(4)	33.9	23.6(3)	30.5	15.5	57.5
Gemma3-12b	39.1	22.4(4)	45.4	17.5(3)	18.0	7.1(2)	34.5	24.3(2)	34.3	17.8	58.2
Gemma3-27b	44.2	24.8(4)	44.1	12.9(3)	27.4	9.9(3)	37.2	25.6(2)	38.2	18.3	60.0

Table 6: Correlation (

\tau

), inter-dimension dependence (

CG

), and number of dimensions whose inter-dimension dependence is statistically significant (in brackets) for different evaluation frameworks and LLMs. Inter-dimension dependence is pervasive.

	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	overall $\uparrow$
	SummEval[4]		Topical Chat[4]		Hanna[6]		OpinSumm.[4]		Avg.
	Llama3.3-70b
Analyze-Rate	46.1	30.0(4)	53.3	13.1(3)	34.6	11.3(3)	32.0	18.9(3)	41.5	18.3	61.6
Analyze-Rate+Other	45.1	30.5(4)	52.0	13.8(3)	35.9	12.6(3)	34.9	17.4(2)	42.0	18.6	61.7
Analyze-Rate+Joint	41.4	28.6(4)	53.5	13.9(2)	37.2	13.7(6)	33.4	19.5(2)	41.4	18.9	61.2
Self-consistency	47.3	28.3 (4)	52.7	14.3 (3)	35.5	13.2 (4)	34.6	19.3 (2)	42.5	18.8	61.9

Table 7: Correlation (

\tau

), inter-dimension dependence (

CG_{\tau}

), and number of dimensions whose inter-dimension dependence is statistically significant (in brackets) for methods on reducing inter-dimension dependence. The best-performing method is bolded . DimCheck shows the best overall performance.

	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	overall
	Llama3.3-70b		Gemma3-27b		Qwen3-32b		Avg.
DimCheck	42.0	16.4	40.3	14.2	40.4	13.9	40.9	14.8	52.7
w/1 step	42.0	16.3	40.4	15.6	40.3	15.6	40.9	15.8	52.2
w/o other	42.0	16.9	40.4	15.9	40.8	15.8	41.1	16.2	52.1
w/o think	41.7	16.6	40.1	15.5	40.8	15.1	40.9	15.7	52.3

Table 8: Average performance of DimCheck and its ablated variants across datasets.

	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	overall
	Llama3.3-70b		Gemma3-27b		Qwen3-32b		Avg.
Analyze-Rate	45.5	30.9	42.2	28.6	41.1	25.5	42.9	28.3	57.3
+DimCheck	45.1	28.5	41.4	24.3	40.7	22.9	42.4	25.2	58.6
w/1 step	43.8	28.9	41.2	26.4	41.2	24.8	42.1	26.7	57.7
w/o think	43.0	30.6	42.9	29.3	41.7	25.5	42.5	28.4	57.0
w/trained	45.0	29.8	41.9	24.3	41.2	24.9	42.7	26.3	58.2

Table 9: Performance of trained smaller LLMs to edit COTs (DimCheck w/trained). DimCheck w/trained shows close overall performance to DimCheck.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

	SummEval	Topi. Chat	Hanna	OpinSumm.	Avg.
Llama3.3-70b
Analyze-Rate	29.9	47.1	29.0	25.4	32.8
Analyze-Rate+Other	28.6	45.3	29.6	29.1	33.2
Analyze-Rate+Joint	25.9	46.8	30.4	26.7	32.5
Self-consistency	31.3	46.0	28.9	26.9	33.3
Self-reflection	29.5	45.8	26.1	23.3	31.2

Appendix

Table 10: Correlation between predicted and ground-truth scores on high-variance groups for different methods. The best average correlation across datasets is bolded . DimCheck shows the highest average correlation among all methods.

You will be given a reasoning process generated by an LLM judge that is instructed to evaluate a {output_type} on a target dimension. You will also be given evaluation instructions on the target dimension and a non-target dimension.

Your task is to first check the reasoning process to find out whether it contains evidence that is closely related to the non-target dimension but not related to the target dimension. For this step, you should not consider any dimensions other than the given target dimension and non-target dimension.

If the reasoning process contains such evidence, you should remove all such evidence from the reasoning process. You should only make necessary modifications without changing other parts of the reasoning process. You should not add content that is not already presented in the original reasoning process. You should then return the modified reasoning process without extra comments. If all contents are removed, return ’EMPTY’.

If you do not find such evidence, return ’NO CHANGE’.

Reasoning Process:

{reasoning_process}

Evaluation instruction for the target dimension:

Appendix

Table 11: Prompt used by DimCheck

	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	overall $\uparrow$
	SummEval[4]		Topical Chat[4]		Hanna[6]		OpinSumm.[4]		Avg.
	Llama3.3-70b
Analyze-Rate+DimCheck	45.9	27.9(4)	54.1	12.5(3)	35	11.1(3)	33.2	14.1(1)	42	16.4	62.8
w/rand	45.2	27.3(4)	54.7	11.5(2)	35.2	10.9(3)	33.6	14.9(1)	42.2	16.1	63.0
	Gemma3-27b
Analyze-Rate+DimCheck	42.9	22.3(4)	51.3	9.5(2)	29.2	10.1(3)	38	15.0(1)	40.3	14.2	63.1

Appendix

Table 12: Correlation (

\tau

), inter-dimension dependence (

CG_{\tau}

), and number of dimensions whose inter-dimension dependence is statistically significant (in brackets) for DimCheck and its variant using random order to remove unrelated dimension. DimCheck’s performance is stable with different prompts.

Figure 2: Example COTs and scores edited by DimCheck. The evidence removed by DimCheck is in red .

	SummEval		Topical-Chat		Hanna		OpinSumm		Average
	COT	Score	COT	Score	COT	Score	COT	Score	COT	Score
Llama3.3-70b	46.3	8.8	53.4	3.7	60.7	3.3	63.9	15.9	56.1	7.9
Gemma3-27b	84.1	14.1	66.0	3.6	89.6	5.7	96.3	16.3	84.0	9.9
Qwen3-32b	54.4	11.4	65.9	2.6	78.3	6.9	59.8	12.1	64.6	8.2

Appendix

Table 13: Proportion of COTs and scores edited by DimCheck.

	SummEval		Topical Chat		Hanna		OpinSumm.		Average
	low	high	low	high	low	high	low	high	low	high
	G-Eval
Llama3.1-8b	9.5	23.1	19.8	10.0	8.8	7.9	9.9	6.2	12.0	11.8
Llama3.3-70b	12.8	11.4	12.1	9.8	10.5	9.9	22.9	13.9	14.6	11.2
Gemma3-12b	13.8	21.8	14.4	18.2	5.1	10.6	21.5	11.9	13.7	15.6
Gemma3-27b	10.6	25.0	15.7	8.0	7.3	11.4	19.3	11.5	13.2	14.0

Appendix

Table 14: Inter-dimension dependence on low-quality and high-quality groups. Inter-dimension dependence can vary across texts of different quality levels.

Figure 3: Interface for Human Evaluation

	SummEval	Topical-Chat	Hanna	OpinSummEval	Average
w/ control	0.00	0.00	0.00	0.31	0.08
w/o control	1.51	0.11	0.09	1.00	0.68

Appendix

Table 15: KL-divergence between the empirical distributions of low-variance group and high-variance group for CorrGap. Without controling, There can be big difference between the empirical distributions.

Figure 4: Distribution of ground truth scores on different dimensions on the SummEval dataset. Ground truth scores across dimensions have very different distributions

You will be given one summary written for a source document.

Your task is to rate the summary on one metric.

Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.Evaluation Criteria:

Relevance (1-5) - selection of important content from the source. The summary should receive high score if it include only important information from the source document. Please penalize summaries which contained redundancies and excess information.

Evaluation Steps:

1. Read the summary and the source document carefully.

2. Compare the summary to the source documents and identify the main points of the source documents.

Appendix

Table 16: Prompt for Analyze-Rate+Other on the SummEval dataset for the relevance dimension

You will be given one summary written for a source document.

Your task is to rate the summary on four metrics.

Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.

You should evaluate each metric independently based on its corresponding criteria and evlauation steps.

Evaluation Criteria 1:

Evaluation Steps 1:

Appendix

Table 17: Prompt for Analyze-Rate+Joint on the SummEval dataset

You will be given a reasoning process generated by an LLM judge that is instructed to evaluate a summary on a target dimension. You will also be given evaluation instructions on the target dimension and three non-target dimensions.

Your task is to first check the reasoning process to find out whether it contains evidence that is closely related to any of the three non-target dimensions but not related to the target dimension. For this step, you should not consider any dimensions other than the given three non-target dimensions.

Reasoning Process:

{reasoning_process}

Evaluation instruction for the target dimension:

You will be given one summary written for a source document.

Appendix

Table 18: Prompt for SFT on the SummEval dataset

	Relevance	Coherence	Empathy	Surprise	Engagement	Complexity
	Relevance	Consistency	Coherence	Fluency
SummEval	22.4	20.5	31.8	34.2
	Engaging	Natural	Coherence	Groundness
Topical Chat	1.5	14.4	13.9	12.0
Hanna	5.2	5.4	5.6	12.0	17.9	15.1
	Aspect Relevance	Self Coherence	Sentiment Consistency	Readability

Appendix

Table 19: Average CorrGap when using Analyze-rate with Llama-3.3-70B-Instruct, Gemma-3-27B-it, and Qwen3-32B for each evaluation dimension. Different dimensions show very different CorrGap values.

	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	overall $\uparrow$
	SummEval[4]		Topical Chat[4]		Hanna[6]		OpinSumm.[4]		Avg.
	Llama3.3-70b
Analyze-Rate	46.1	30.0(4)	53.3	13.1(3)	34.6	11.3(3)	32.0	18.9(3)	41.5	18.3	61.6
Analyze-Rate+Other	45.1	30.5(4)	52.0	13.8(3)	35.9	12.6(3)	34.9	17.4(2)	42.0	18.6	61.7
Analyze-Rate+Joint	41.4	28.6(4)	53.5	13.9(2)	37.2	13.7(6)	33.4	19.5(2)	41.4	18.9	61.2
Analyze-Rate+DimCheck	45.9	27.9 (4)	54.1	12.5 (3)	35.0	11.1 (3)	33.2	14.1 (1)	42.0	16.4	62.8

Appendix

Table 20: Correlation (

\tau

), inter-dimension dependence (

CG_{\tau}

), and number of dimensions whose inter-dimension dependence is statistically significant (in brackets) for DimCheck and its variant using different prompts. The best-performing method is bolded . DimCheck’s performance is stable with different prompts.

Your task is to first check the reasoning process to identify evidence that is closely related to the non-target dimension but not related to the target dimension. For this step, you should not consider any dimensions other than the given target dimension and non-target dimension.

If you do not find such evidence, return ’NO CHANGE’.

Reasoning Process:

{reasoning_process}

Evaluation instruction for the target dimension:

Appendix

Table 21: A variant prompt for DimCheck

	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	$\tau\uparrow$	CG $\downarrow$	overall $\uparrow$
	SummEval		Topical Chat		Hanna		OpinSumm.		Avg.
	Llama3.3-70b
DimCheck	45.9	27.9 (4)	54.1	12.5 (3)	35.0	11.1 (3)	33.2	14.1 (1)	42.0	16.4	62.8
DimCheck w/1 step	45.2	28.5(4)	53.8	11.9(3)	34.9	10.2(3)	34.1	14.7(1)	42.0	16.3	62.8
DimCheck w/o other	46.0	29.3(4)	53.4	12.3(3)	35.0	10.7(3)	33.4	15.6(1)	42.0	16.9	62.5
DimCheck w/o think	44.2	28.1(4)	52.8	12.9(2)	34.9	10.0(3)	35.1	15.5(1)	41.7	16.6	62.5

Appendix

Table 22: Correlation (

\tau

), inter-dimension dependence (

CG_{\tau}

), and number of dimensions whose inter-dimension dependence is statistically significant (in brackets) for DimCheck and its ablated variants. The best-performing method is bolded . DimCheck outperforms all its ablated variants.

You will be given one summary written for a source document.

You will also be given your previous evaluations.

Your task is to decide whether your previous evaluation is affected by other dimensions and refine your score if it is affected by other dimensions.

Evaluation Criteria:

Evaluation Steps:

1. Read the summary and the source document carefully.

Appendix

Table 23: Prompt for Self-Reflection on the SummEval dataset for the relevance dimensions.

Language Model Generation Evaluation

Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.

See all 396 papers

Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluation
Aug 31, 2026Anshul Bagaria, Sowmya S Sundaram, Gokul S Krishnan +1LLM-as-a-Judge Language Model Generation Evaluation
Centre for Responsible AI (CeRAI), Wadhwani School of Data Science and AI (WSAI), IIT Madras, Chennai, India
*-PLUIE: Personalisable metric with Llm Used for Improved Evaluation
Feb 17, 2026Quentin Lemesle, Léane Jourdan, Daisy Munson +4Language Model Generation Evaluation LLM Evaluation
Univ Rennes, CNRS, IRISA, EXPRESSION, 22300 Lannion, France · Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes, France · Univ Rennes, CNRS, IRISA, SOTERN, 22300 Lannion, France +1
When Gradients Collide: Failure Modes of Multi-Objective Prompt Optimization for LLM Judges
May 25, 2026Parth Darshan, Abhishek DivekarLLM-as-a-Judge Large Language Model-Guided Optimization
♢IIT Jodhpur · ♠Amazon

Aug 31, 2026J/K move · Enter open · S save

Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluation

Anshul Bagaria, Sowmya S Sundaram, Gokul S Krishnan, Balaraman Ravindran

Centre for Responsible AI (CeRAI), Wadhwani School of Data Science and AI (WSAI), IIT Madras, Chennai, India

LLM-as-a-Judge Language Model Generation Evaluation Rubric-Based Evaluation Bias in LLM-as-a-Judge

Open paper AskPDF arXiv

LLM-as-a-Judge pipelines are increasingly used to evaluate AI-generated text, based on the assumption that judgments arise from reasoning over candidate responses with respect to a rubric. We show that this assumption warrants further scrutiny. Classifiers trained only on rubric text, without access to any evaluated response, achieve nontrivial predictive performance on judge outputs. This suggests that rubric formulations encode recoverable evaluative signals, allowing scores to be partially anticipated independently of model outputs. Finally, counterfactual perturbations reveal that judges often fail to reliably update their decisions when either the candidate response or the rubric criterion is reversed. Our findings raise concerns about the reliability of rubric-based LLM evaluation and highlight the need for further methodological study of automated evaluation via LLMs.

Abstract

Language Model Generation Evaluation

Explore similar work

Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluation

*-PLUIE: Personalisable metric with Llm Used for Improved Evaluation

When Gradients Collide: Failure Modes of Multi-Objective Prompt Optimization for LLM Judges