COMPREHEND: Ultra-Fine-Grained Benchmark Understanding by Entailment Hierarchy
Organizations: University of Massachusetts, Amherst, USA · University of Arizona, Tucson, USA
Abstract
Understanding what kind of prompts users typically ask and how well an LLM performs on these prompt categories is crucial for LLM practitioners. The existing methods satisfy the demand by hierarchically clustering the user prompts and synthesizing the cluster names using an LLM to build a taxonomy. However, the cluster names from these methods are usually too general for humans to understand the detailed prompt distributions and identify the exact strengths and weaknesses of LLMs. In this study, we propose COMPREHEND, which automatically constructs an ultra-fine-grained entailment hierarchy for the prompts. The entailment hierarchy replaces the cluster names in the taxonomy with sentences to provide more precise cluster descriptions for more fine-grained categorization. Compared with EvalTree, a state-of-the-art taxonomy construction method, COMPREHEND improves an entailment/faithfulness metric from 22.6 to 41.7 and an informative metric from 16.3 to 39.1 in WildBench. Our experiments also show that the weakness report generated based on our entailment hierarchy is much more informative than EvalTree in terms of predicting the response scores in the benchmark. All the code is released in https://github.com/JackNapier20/COMPREHEND_public.
Figures & tables
| Methods | Clustering | Leaf Main Task | Parent Generation |
|---|---|---|---|
| EvalTree | Recursive Kmeans | Capability | Capability |
| EvalTree + Our Prompts | Recursive Kmeans | Main Task | Common Prompt |
| EvalTree + Our Prompts + Org. | Recursive Kmeans | Original | Common Prompt |
| Comprehend - Main - Eval | Match | Original | Common Prompt |
| Comprehend - Main | Match + Assignment | Original | Common Prompt + Regen. |
| Comprehend | Match + Main + Assignment | Original | Common Prompt + Regen. |
| Data | Methods | Entail avg | Entail min | Info avg | Info min | Specificity | Len (words) |
|---|---|---|---|---|---|---|---|
| WB256 | EvalTree | 36.3 ±1.7 | 22.6 ±1.6 | 22.9 ±1.0 | 16.3 ±0.9 | 13.0 ±2.2 | 5.7 ±0.1 |
| EvalTree + Our Prompts | 47.1 ±1.6 | 32.9 ±1.8 | 36.3 ±1.4 | 26.3 ±1.4 | 25.4 ±2.1 | 15.8 ±0.3 | |
| EvalTree + Our Prompts + Org. | 50.4 ±1.7 | 35.5 ±1.9 | 41.0 ±1.6 | 31.6 ±1.6 | 23.8 ±2.2 | 21.4 ±0.9 | |
| Comprehend - Main - Eval | 53.5 ±1.7 | 38.8 ±1.9 | 44.2 ±1.5 | 34.9 ±1.6 | 26.4 ±2.4 | 21.3 ±1.0 | |
| Comprehend - Main | 55.4 ±1.7 | 40.7 ±2.0 | 45.6 ±1.6 | 36.0 ±1.7 | 29.3 ±2.3 | 21.1 ±0.9 | |
| Comprehend | 55.9 ±1.7 | 41.7 ±1.9 | 49.1 ±1.5 | 39.1 ±1.7 | 32.5 ±2.3 | 21.6 ±1.0 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Data | Generation model | Method | Entail avg | Entail min | Info avg | Info min | Specificity | Len (words) |
|---|---|---|---|---|---|---|---|---|
| WB256 | Claude Sonnet 4.6 | EvalTree | 36.3 ±1.7 | 22.6 ±1.6 | 22.9 ±1.0 | 16.3 ±0.9 | 13.0 ±2.2 | 5.69 ±0.13 |
| EvalTree (MiniLM) | 39.7 ±1.9 | 24.8 ±1.9 | 21.2 ±1.0 | 14.0 ±0.8 | 17.7 ±2.2 | 5.46 ±0.12 | ||
| Comprehend | 55.9 ±1.7 | 41.7 ±1.9 | 49.1 ±1.5 | 39.1 ±1.7 | 32.5 ±2.3 | 21.64 ±1.02 | ||
| GLM 5.2 | EvalTree | 38.3 ±1.9 | 23.8 ±1.9 | 16.1 ±0.8 | 11.2 ±0.6 | 13.8 ±2.3 | 2.96 ±0.04 | |
| EvalTree (MiniLM) | 36.9 ±2.0 | 22.5 ±1.9 | 14.5 ±0.7 | 10.5 ±0.5 | 7.7 ±2.6 | 2.96 ±0.04 | ||
| Comprehend | 60.5 ±1.8 | 47.9 ±2.0 | 40.7 ±1.5 | 32.2 ±1.5 | 33.5 ±2.6 | 15.33 ±1.19 |
| Method | Parents | Entail avg | Entail min | Info avg | Info min | Specificity | Len (words) |
|---|---|---|---|---|---|---|---|
| EvalTree ( ) | 63 | 31.2 ±3.5 | 19.9 ±3.2 | 20.6 ±2.0 | 13.2 ±1.4 | 25.5 ±3.7 | 6.5 ±0.3 |
| EvalTree ( ) | 30 | 32.1 ±5.5 | 20.2 ±5.0 | 19.9 ±2.7 | 14.0 ±2.1 | 27.4 ±5.2 | 7.1 ±0.4 |
| EvalTree ( ) | 20 | 28.8 ±5.9 | 17.6 ±5.2 | 18.1 ±2.2 | 12.7 ±1.9 | 24.3 ±5.7 | 7.8 ±0.5 |
| EvalTree ( ) | 13 | 24.5 ±8.9 | 19.9 ±8.8 | 13.7 ±1.4 | 10.9 ±1.7 | 22.0 ±9.4 | 7.8 ±0.7 |
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,539 papers