From Plausible Hierarchies to Useful Taxonomies: Evaluating Agentic Harnesses on Customer Feedback

Prabhath Chellingi; Raviraja G; Viraj Bagal

cs.AIOct 7, 2026

From Plausible Hierarchies to Useful Taxonomies: Evaluating Agentic Harnesses on Customer Feedback

Authors: Prabhath Chellingi, Raviraja G, Viraj Bagal

Organizations: Enterpret

Abstract

Taxonomies are the symbolic representations through which AI systems organize evidence, aggregate patterns, and answer questions over large document collections. Over customer feedback, the category tree decides how every record is counted and routed, which problems get seen, and which team owns them. Agentic harnesses now make it easy to generate a plausible-looking hierarchy, and such trees are checked today with generic, individually scoped checks: each name fits its description, sits under the right parent, and stays distinct from its siblings. We ask a more operational question: when is a generated hierarchy actually useful as a production taxonomy? We build six taxonomies over two proprietary feedback corpora (1,940 and 5,000 records): for each corpus, a production reference and two repeated runs of the same harness under identical inputs. All six pass every generic naming and structure check, and a deeper product-coverage check even prefers the generated trees. Yet in every generated tree at least 97.7% of leaf names merely restate an ancestor's name (13.9% and 2.9% in the references), and in one, three of every four records fall under multiple top-level categories. We introduce two families of whole-tree metrics: structural discriminators test whether a tree's shape was learned from the data or imposed by its generator; team partitionability tests whether branches split feedback into groups teams can own. Trees the generic checks rate as equally correct differ by 27 percentage points in cross-branch leakage, and only one of the two beats a random split. Surface plausibility is an insufficient measure of taxonomy quality: evaluation must measure the whole tree as well as each node.

Figures & tables

	Small sample			Large sample
Metric 1 1 1 $\uparrow$ higher is better; every metric in this table is higher-is-better.	Live	C-1	C-2	Live	D-1	D-2
Name-desc. match $\uparrow$	0.98	0.97	0.98	0.98	1.00	1.00
Desc. completeness $\uparrow$	1.00	1.00	1.00	1.00	1.00	1.00
Hierarchy correctness $\uparrow$	0.94	1.00	0.92	0.93	0.94	0.99
Sibling distinctness $\uparrow$	0.94	0.88	0.94	0.95	0.91	0.99
Product-term coverage $\uparrow$	0.72	0.74	0.86	0.80	0.91	0.90

Table 1: The generic checks (top four rows) and the deep-judged product-term coverage check (bottom row), six artifacts across two corpora. Every artifact passes every check, with name-description match at 0.97–1.00 everywhere; the checks read as “all six are professional-grade.” On product-term coverage, every generated artifact outscores its reference (bold), on both corpora.

Panel of Figure 1 — Figure 1: One leaf path each from Live and C-1, both real, verbatim node names; column headers give each artifact’s measured ancestor-containment rate. The phrase in quotation marks (set in red) is copied character-for-character from C-1’s L3 node into every descendant below it: Theme and Subtheme are each almost entirely copied text with a single new word appended. Name-description match passes both chains at 0.97–0.98 (Table 1 ), because it only ever compares a node to its own description, never to its ancestors.

Panel of Figure 2 — Figure 2: Fan-out shape (child counts real; node identities schematic). C-2’s L2 $\to$ L3 level pair collapses to a pure wrapper chain: all 158 parents have exactly one child . Live’s fan-out spans 1–31, the shape long-tailed feedback volume produces. Hierarchy correctness scores every individual edge in both trees as correct (Table 1 ) because it never looks at how many siblings a node has.

	Small sample			Large sample
Metric 2 2 2 $\uparrow$ higher is better; $\downarrow$ lower is better; $\star$ no single best direction – max fan-out reflects the shape of the underlying data, not quality directly (§ 5 ).	Live	C-1	C-2	Live	D-1	D-2
Containment $\downarrow$	13.9%	97.7%	100%	2.9%	97.7%	100%
Information gain $\uparrow$	0.76	0.33	0.34	0.81	0.74	0.58
Type-token ratio $\uparrow$	0.25	0.10	0.06	0.14	0.07	0.05
Cap-suspect pairs $\downarrow$	0	1	1	1	0	1
Cross-branch reuse $\uparrow$	48	0	0	73	0	0

Table 2: Structural discriminators at the leaf level, six artifacts across two corpora. Every generated artifact collapses on containment, information gain, diversity, and reuse; max fan-out (

\star

) is distributional context, not a quality score. The large-sample reference trips the cap-suspect flag once, at a genuinely narrow level pair (30 parents, none above 3 children) – the threshold false positive anticipated in Limitations – while D-2’s flagged pair is degenerate (451 parents, none above 2). Bold marks the headline finding. The generic checks flagged none of this.

Panel of Figure 3 — Figure 3: Branch leakage and co-occurrence modularity ( $Q$ , Newman modularity of the branch partition: positive means branches hold together tighter than a random split, negative means worse). Both records and both quotes are real, taken verbatim from the corpus and its predictions; percentages give each pattern’s share of that artifact’s records. Left: a C-2 record about a specific model’s behavior predicted onto both a model-routing branch and an agent-behavior branch, the majority pattern for C-2 (76.4% of records leak), consistent with $Q<0$ . Right: a Live record staying inside one branch, the majority pattern there (80.2% of records touch exactly one branch), consistent with $Q>0$ . Hierarchy correctness (Table 1 ) scores C-1 and C-2 within 0.08 of each other; this figure’s metrics separate them by 27 points of leakage and a sign flip , because they are the only ones that route records through the tree at all.

	Small sample			Large sample
Metric 3 3 3 $\uparrow$ higher is better; $\downarrow$ lower is better; $\star$ no single best direction – see the dividability definition.	Live	C-1	C-2	Live	D-1	D-2
L1 leakage $\downarrow$	19.8%	49.8%	76.4%	19.3%	34.7%	39.3%
L1 record purity $\uparrow$	0.92	0.78	0.61	0.93	0.83	0.81
L1 modularity $\uparrow$	0.34	0.12	$-$ 0.12	0.43	0.08	0.16
L1 dividability $\star$	0.32	0.87	0.71	0.70	0.59	0.62
L2 leakage $\downarrow$	22.0%	68.1%	79.4%	22.6%	38.1%	43.7%

Table 3: Team partitionability, six artifacts, each scored over its own classification of its corpus’s records. On both corpora the references leak about a fifth of records across top-level branches; every generated artifact leaks between a third and three-quarters . Bold marks the C-1 vs. C-2 contrast discussed in the text. Dividability (

\star

) is workload-balance context, not a quality score.

Explore similar work

TaxCE : A Framework for Automated Taxonomy Construction and Evaluation at Scale
Aug 31, 2026Sandeep Sricharan Mukku, Albert Aristotle Nanda, Rohit PyatiTaxonomy Topic Modeling
Amazon
Fantastic Adaptive Taxonomies and How to Use Them
Jul 17, 2026Mert Cemri, Andrei Cojocaru, Melissa Pan +9Taxonomy Traces
University of California, Berkeley · Apple · Bespoke Labs
ATLAS: Agentic Taxonomy of Large-Scale Software Ecosystems
Jun 19, 2026Junyi Lu, Mengyao Lyu, Jiahui Wu +6Taxonomy Open-Source
Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Nanyang Technological University, Singapore +2

Aug 31, 2026, cs.CLJ/K move · Enter open · S save

TaxCE : A Framework for Automated Taxonomy Construction and Evaluation at Scale

Sandeep Sricharan Mukku, Albert Aristotle Nanda, Rohit Pyati

Amazon

Taxonomy Topic Modeling Multi-Dimensional Evaluation Feedback

Open paper AskPDF arXiv

Organizing unstructured feedback text into hierarchical taxonomy is a fundamental challenge in NLP, particularly in domains where feedback arrives at massive scale in varied forms such as reviews, transcripts, and surveys. Existing approaches either produce shallow hierarchies, neglect long-tail topics, or lack rigorous evaluation frameworks. We present TaxCE, a fully automated framework that constructs multi-level hierarchical taxonomies from raw text through progressive condensation of corpus content into actionable segments, deduplicated semantic units, and granular topics with definitions, which are then organized bottom-up into a hierarchy with corpus-groundedness. We also introduce three corpus-grounded evaluation metrics, Exclusivity, Exhaustivity, and Granularity (EEG), and integrate them into a metrics-in-the-loop iterative refinement mechanism that diagnoses deficiencies and applies targeted corrections until convergence. Extensive experiments demonstrate that TaxCE consistently outperforms existing baselines spanning classical topic models, neural methods, and LLM-based approaches, with average improvements of 11.8, 20.5, and 15.7 percentage points in exclusivity, exhaustivity, and granularity respectively over the strongest baseline. Human evaluation further confirms superior taxonomy quality, actionability, and navigability.

Abstract

Explore similar work

TaxCE : A Framework for Automated Taxonomy Construction and Evaluation at Scale

Fantastic Adaptive Taxonomies and How to Use Them

ATLAS: Agentic Taxonomy of Large-Scale Software Ecosystems