From Retrieval to Customer Context: Evaluating Frontier-Model Systems for Voice-of-Customer Analysis

Raviraja G; Viraj Bagal; Prabhath Chellingi

cs.LGOct 7, 2026

From Retrieval to Customer Context: Evaluating Frontier-Model Systems for Voice-of-Customer Analysis

Authors: Raviraja G, Viraj Bagal, Prabhath Chellingi

Organizations: Enterpret

Abstract

Organizations increasingly use frontier language models to analyze customer feedback, but answer quality also depends on how that feedback is organized and made available. We define a \emph{customer context graph} as a unified model of customer and business context. Typed relationships connect customer objects (feedback, conversations, users, and accounts), operational objects (tickets, support agents, opportunities, and competitors), and analytical or action objects (taxonomy concepts, evidence, insights, work items, and outcomes). This lets an agent investigate not only what customers say, but why, who is affected, what action followed, who owns it, and whether it was resolved. For this experiment, the graph is populated from public Cursor feedback; the same architecture can support any type of feedback source. We compare Agentic RAG, a Deep Research Agent, and a Customer Context Graph-backed Agent on the same 9,432 public Cursor feedback records using 30 realistic product, incident, comparison, and metadata questions. Without exhaustive ground truth, we jointly score responses on answer quality (coverage and organization), analytical depth (specificity and decomposition), and evidence quality (citation support and traceability), using a comparative rubric calibrated on 28 of the 30 questions. We sample cited records against their claims and weight the three dimensions equally. Under this aligned rubric, the Customer Context Graph-backed Agent scores 0.961 overall, versus 0.710 for the Deep Research Agent and 0.651 for Agentic RAG, and leads the Deep Research Agent on 27 of 30 paired questions (sign-test p < 10^(-5); strictly best on 26 of 30). Its largest advantage is analytical depth (0.967 versus 0.642), reflecting more specific, hierarchically developed findings with quantified themes and traceable evidence...

Figures & tables

Panel of Figure 1 — Figure 1: End-to-end benchmark architecture. A neutral parsing layer supplies the same feedback text and source metadata to all candidates. Each system keeps its native preparation and question-execution architecture, but all outputs are converted to a common answer package and scored by the same paired evaluator.

Panel of Figure 2 — Figure 2: Component architecture of the three evaluated candidates. Agentic RAG uses one analyst over retrieval and exact-analysis tools. The Deep Research Agent adds decomposition, specialist roles, an evidence ledger, verification, and revision over that retrieval layer. The context graph system first connects feedback, summaries, taxonomy predictions, metadata, and available user information; its agent then computes aggregates and drills down to source records when producing an answer.

Held constant	Frozen 9,432-record corpus + SHA-256; identical query text and as-of date; base analyst model, both judges, retrieval reranker; answer contract; retry policy; alias-shuffled evaluator presentation.
Allowed to differ (the treatment)	Whether the system reconstructs product concepts per query or reads pre-computed summaries, taxonomy predictions, metadata, user information, and graph relationships stored at ingestion; the orchestration budget the analyst model is dispatched into.
Can therefore claim	The complete-system effect of persistent customer context on this corpus, question set, and calibrated rubric; not the isolated causal effect of any component in isolation.

Table 1: Experimental controls for the three-system comparison.

Statistic	Value	Coverage
Normalized feedback records	9,432	100%
Collection window	2026-04-16–2026-07-15 UTC
Public feedback sources	3	100%
Records with rating	171	1.81%
Records with normalized version	5	0.05%
Records with public country	118	1.25%

Table 2: Frozen Cursor corpus used by all three systems.

Depth	Required reasoning	$n$
1	Exact lookup, metadata coverage, capability checks	11
2	Focused semantic classification or evidence synthesis	10
3	Comparisons across source, time, or cohort	3
4	Multi-stage VoC, strategy, incident, and workflow decisions	6
	Total	30

Table 3: Benchmark depth. Counts are fixed by the query manifest.

Panel of Figure 3 — Figure 3: The simple protocol compares anonymized answers and audits every resolvable citation group separately.

System	Overall (95% CI)	Answer quality	Analytical depth	Evidence quality	Queries
Agentic RAG	0.651 [0.613, 0.692]	0.625	0.642	0.686	30/30
Deep Research Agent	0.710 [0.668, 0.752]	0.683	0.642	0.806	30/30
Context Graph Agent	0.961 [0.936, 0.983]	0.958	0.967	0.958	30/30

Table 4: Macro-averaged automatic evaluation across 30 paired queries. Overall combines the three dimensions equally. Per-dimension scores for answer quality and analytical depth are drawn from a discrete comparative-tier scale on

[0.5,1]

for completed answers with 0.25 spacing (see Section 5 ); evidence quality is a harmonic mean of sampled citation precision and comparative evidence-strength score in

[0,1]

. Because the scale is discrete and the primary comparison is paired, means are descriptive; the primary comparison statistic is paired

W/T/L

( Table 5 ). Best values in bold; bracketed overall values are bootstrap 95% CIs over 30 queries (

B=5{,}000

, seed 42).

Comparison	Metric	Wins	Ties	Losses	$\overline{\Delta}$ (95% CI)	Sign-test $p$
Context graph $-$ Deep	Overall	27	1	2	+0.251 [+0.193, +0.305]	$1.6\!\times\!10^{-6}$
	Answer quality	25	1	4	+0.275	$1.0\!\times\!10^{-4}$
	Analytical depth	28	0	2	+0.325	$8.7\!\times\!10^{-7}$
	Evidence quality	27	2	1	+0.153	$2.2\!\times\!10^{-7}$
Context graph $-$ Agentic	Overall	29	0	1	+0.310 [+0.263, +0.354]	$5.8\!\times\!10^{-8}$
	Answer quality	28	1	1	+0.333	$1.1\!\times\!10^{-7}$

Table 5: Paired query-level comparisons with a 0.01 tie tolerance.

W/T/L

counts are from the perspective of the first system;

\overline{\Delta}

is its mean score minus the second system’s score, with a bootstrap 95% confidence interval over 30 queries (

B=5{,}000

resamples, seed 42);

p

reports a two-sided sign test on non-tied paired outcomes. Because the completed-answer score scale is discrete with 0.25 spacing (see Table 4 and Section 5 ), the paired

W/T/L

result is the primary comparison statistic and

\overline{\Delta}

is descriptive.

System	Cited	Member-audited	Traceable aggregate	Unresolved	Records sampled
Agentic RAG	712	708	0	4	708
Deep Research Agent	594	594	0	0	594
Context Graph Agent	913	127	786	0	127

Table 6: Citation-audit coverage. “Member-audited” citations have at least one frozen record judged against the nearby claim. Traceable aggregates retain drill-down metadata but were not member-sampled.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

System	Abridged evaluated response	Overall
Agentic RAG	Runs a model-coded exclusive classification over 8,890 of 9,432 records (94.25% coverage; 2,728 assigned to “other/no clear pain point”) and ranks seven pain-point categories: billing, pricing, usage limits, credits, overages, and refunds (2,195); model choice, Auto-mode opacity, and context/routing (1,461); agent/code quality and unsafe edits (875); trust, privacy, and ownership concerns (748); product instability, regressions, and IDE performance (629); mobile/iOS parity gaps (155); and support/account escalation (99). Each category is followed by cited examples and an analyst interpretation.	0.556
Deep Research Agent	Runs an exhaustive semantic classification over all 9,432 records into ten exclusive primary pain-point categories and reports both counts and corpus share: model quality and Auto-mode control (892; 9.5%); token burn and unclear usage attribution (628; 6.7%); usage caps and quota lockout (370; 3.9%); billing surprises and commercial trust (353; 3.7%); long-session context loss (232; 2.5%); update regressions (207; 2.2%); UI churn (156; 1.7%); agent out-of-scope changes (119; 1.3%); support and refund friction (116; 1.2%); and crashes or severe local instability (79; 0.8%). It also flags 2,133 records classified as positive value or mitigation as a counter-signal.	0.776
Context Graph Agent	Ranks recurring themes over 5,035 customer conversations (Reddit 4,868 at 96.7%; App Store 114; Trustpilot 53) organized into seven clusters: quota and usage-limit friction (180), throughput rate-limit disruption (162), degraded behavior when rate limits are hit (95), Auto routing mismatch (92), Composer/model quality regressions (86), poor AI output quality (82), and a service-availability/agent-safety cluster whose top themes are pre-execution safety controls (77) and service availability issues (77). Each main count links to its evidence set; leading themes decompose into cited subtheme drivers and are illustrated with verbatim customer quotes.	1.000

Appendix

Table 7: Abridged outputs for q001 : “What are the biggest pain points Cursor users are reporting?” Text preserves the principal grouping, quantification, and evidence pattern of each evaluated answer; it is shortened for space. Counts and denominators are taken verbatim from the canonical answer packages listed in the response appendix.

System	Abridged evaluated response	Overall
Agentic RAG	Compares 30-day calendar windows (preceding 2026-05-17 to 2026-06-15 with 557 records; recent 2026-06-16 to 2026-07-15 with 8,652 records) and finds that a keyword reliability proxy falls in share from 17.2% to 9.8% even as absolute counts rise. The only rising sub-signal is a regression/update-language proxy that grows from 1 to 37 records, and 113 App Store reviews appear only in the recent window (no preceding baseline). It labels the broad “reliability worsening” claim as low confidence.	0.556
Deep Research Agent	Uses closely matched rolling 14-day windows (3,720 versus 3,717 records). Its combined reliability proxy rises from 301 to 345 (+14.6%), while native-tool, harness, MCP, and file-operation language rises from 79 to 132 (+67.1%). It localizes a GLM 5.2 tool-call/markup cluster to 2026-07-08 and 2026-07-09 (21 of 25 recent GLM matches), distinguishes increasing, flat, and decreasing signals across category proxies, and assigns medium confidence.	0.930
Context Graph Agent	Reports a concrete recent-period reliability signal across 9,408 conversations (May 31 to Jul 15) with 86 reliability-adjacent themes and five failure clusters (rate-limit and quota reliability, agent execution stalls, core product reliability, billing/metering errors, client-side crashes and freezes), each with citation-linked recent and preceding counts. It declines to claim an increase because the preceding period (Apr 16 to May 30) contains only 24 Trustpilot-only conversations while the recent period is 98.4% Reddit; the only apples-to-apples slice is Trustpilot’s $24\rightarrow 29$ move.	0.786

Appendix

Table 8: Abridged outputs for q013 : “Are there signs of a recently increasing reliability issue?” This counterexample shows a query on which bespoke query-time reconstruction outperforms the stored-context response. Counts and windows are taken verbatim from the canonical answer packages listed in the response appendix.

System	Abridged evaluated response	Overall
Agentic RAG	States the denominator (63 of 113 App Store reviews; 54 five-star and 9 four-star; 2026-06-29 to 2026-07-13) and organizes praise into five specific, separately developed themes—mobility (“I can work anywhere now”), cloud-agent control as the primary use case, concrete productivity gains from fewer idle agents, focused mobile UX, and continuity with desktop workflows. Each theme carries several directly quoted reviews with record citations and closes with an explicit analyst interpretation distinguishing what is being praised (a control plane for agents) from what is not (mobile code editing).	0.887
Deep Research Agent	Establishes the same denominator with an auditable exact query and rating-coverage checks (63 of 113; 55.8%). Reports five praise themes but scores them as retrieved-evidence counts over a 39-record retrieval set (mobile agent work 18/39; interface polish 8/39; agent visibility 6/39; pre-existing Cursor affinity 7/39; mixed praise-plus-request 10/39), and states plainly that no exhaustive classification was scoped to the 63 high-rated reviews, so these are not prevalence estimates. The disclosure is exemplary but the resulting themes stay closer to retrieval bookkeeping than to decision-ready analysis.	0.608
Context Graph Agent	Reports the same 63-conversation denominator with a drill-down citation, and is the only system to flag that App Store coverage begins only in late June 2026 (an ingestion gap rather than an absence of reviews). It returns five citation-linked praise themes with conversation counts—mobile agent companion (11), general product delight (8), usability and design (5), value of off-device agent execution (5), speed (2)—plus explicit mixed-intent and non-additive-count caveats. It earns perfect evidence quality (1.000) but tier-two answer quality and analytical depth (0.750 each): its second-largest cluster is the catch-all “general product delight — undifferentiated love for Cursor overall,” precisely the kind of umbrella label the rubric’s one-problem test penalizes, and its leading specific theme covers only 11 of 63 reviews.	0.833

Appendix

Table 9: Abridged outputs for q022 : “What are App Store users praising in high-rated reviews?” This is the benchmark’s clearest unforced context-graph loss: unlike q013 , where the context graph agent loses by correctly refusing an unsupported trend claim, here it is simply outperformed. All three systems agree on the same exact denominator (63 of 113 App Store reviews rated 4–5), so the gap is attributable to decomposition quality rather than quantification. Counts are verbatim from the canonical answer packages.

Depth	$n$	Agentic	Deep Res.	Context graph
1	11	0.682	0.692	0.955
2	10	0.672	0.679	0.967
3	3	0.555	0.834	0.892
4	6	0.606	0.733	0.996

Appendix

Table 10: Mean overall score by benchmark depth level. Depth levels are frozen in the query manifest ( Table 3 ); slices are descriptive.

Analysis family	$n$	Agentic RAG	Deep Research	Context graph
Executive VoC	5	0.554	0.800	0.974
Product area	5	0.616	0.764	0.951
Reliability and incidents	5	0.632	0.733	0.918
Pricing and packaging	5	0.700	0.633	0.996
Source and metadata	5	0.767	0.637	0.928
Competition and retention	4	0.635	0.693	1.000

Appendix

Table 11: Mean overall score by analysis family. These descriptive slices are small, especially the single cross-functional query, and are not independent hypothesis tests.

Dimension	What the evaluator checks
Answer quality	Explicit question coverage, organization, presentation, specificity, useful quantification, and suitability for the intended reader.
Analytical depth	Diversity and granularity of findings; coherent themes, drivers, subthemes, contexts, counterevidence, implications, and synthesis. Broad umbrella themes are disfavored when they merge distinct problems.
Evidence quality	Relevance of deterministically sampled cited records plus comparative evidence strength: claim-level support, denominators, count semantics, aggregate drill-down links, and representative record citations.
Overall	Unweighted mean of the three dimensions for each query, followed by macro-averaging across the 30 questions.

Appendix

Table 12: High-level evaluation protocol.

Role	Model	Provider	Sampling	Budget
Agentic RAG
Analyst (planning, tool use, synthesis)	gpt-5.5	OpenAI	Provider default	50 turns
Retrieval reranker	rerank-2.5	Voyage	Provider default	—
Deep Research Agent
Triage router	gpt-5.5	OpenAI	Provider default	50 turns
Planner / replanner	gpt-5.5	OpenAI	Provider default	50 turns

Appendix

Table 13: Model identifiers, sampling settings, and budgets for every role in the evaluated run. gpt-5.5 denotes the OpenAI gpt-5.5 snapshot in effect on the frozen evaluation date; where a provider does not expose a pinned snapshot, the identifier resolves to the provider’s default alias at run time and the resolved snapshot is recorded in the run manifest. “Provider default” means neither the temperature, top_p , nor reasoning_effort parameters were overridden.

Component	Required content
Run	Candidate, model, prompt/config hashes, corpus hash, timestamps, status
Scope	Query ID and requested output context
Answer	Final rendered text
Citation ledger	Citation ID, nearby claim, cited record or aggregate members
Usage	Model calls, tokens, tools, cost fields, failures

Appendix

Table 14: Required answer-package components for the shared evaluation.

Family	$n$	Family	$n$
Executive VoC	5	Product area	5
Reliability and incidents	5	Pricing and packaging	5
Source and metadata	5	Competition and retention	4
Cross-functional action	1

Appendix

Table 15: Workflow families in the 30-query benchmark.

Explore similar work

May 19, 2026cs.CL

ContextRAG: Extraction-Free Hierarchical Graph Construction for Retrieval-Augmented Generation

Graph-structured retrieval-augmented generation (RAG) systems can improve answer quality on multi-hop questions, but many current systems rely on large language models (LLMs) to extract entities, relations, and summaries during indexing. These calls add token and wall-clock costs that grow with corpus size. We present ContextRAG, a graph RAG system whose graph topology is constructed without LLM-based entity or relation extraction. ContextRAG derives a fuzzy concept graph over chunk embeddings using residual-quantization k-means and Formal Concept Analysis with Lukasiewicz residuated logic. Bridge-like and meet-derived context nodes are induced by soft fuzzy join and meet operations, rather than by LLM-written graph edges. On a 130-task UltraDomain subset, ContextRAG builds its index with 30 LLM calls and 22,073 tokens. In contrast, a local HiRAG reproduction stress test required 870 indexing calls and 3.54M tokens on a 20-task subset before failing during graph construction; linear extrapolation to 130 tasks implies over 23M indexing tokens. ContextRAG obtains 33.6% F1 overall and 36.8% F1 on multi-hop tasks. An activation analysis shows that queries retrieving at least one lattice-derived node in the top five achieve +3.9 percentage points F1 over queries that do not; this association is diagnostic rather than causal.

Roman Prosvirnin, Sergei Kuznetsov, Seungmin Jin

HSE University

Graph-Based Retrieval Augmented Generation

Ask PDF

Jun 24, 2026cs.CL

Is GraphRAG Needed? From Basic RAG to Graph-/Agentic Solutions with Context Optimization

As advanced RAG variants like GraphRAG and Agentic RAG emerge, one leading question is when and how to use them. Here, we introduce a framework for different RAG scenarios evaluation and comparison on semi-structured knowledge bases, including regular RAG, GraphRAG, Modular RAG and Agentic RAG. We provide implementation for 9 standardized RAG scenarios, and conduct experiments for a comprehensive comparison. These scenarios are designed for real use cases regarding data and domain restrictions, spanning from simple document-based retrieval to advanced features such as hybrid text-graph retrieval, integration with computed or pre-defined domain knowledge graphs, agentic multi-step planning, and agent-graph integration. Besides, we present a novel context engineering method for GraphRAG and Agentic RAG, addressing the context/memory overflow issues, efficiently managing text and graph retrievals with new representations and agentic loop design, leading to 19%-53% reduction on token usage. Moreover, further analysis identifies a retrieval-generation gap where expanded retrieval does not proportionally improve generation quality, suggesting retrieval-oriented metrics overstate advanced retrieval benefits. This work provides data-driven insights on when and how to use them for building production-ready intelligent RAG systems.

Long Chen, Ryan Razkenari, Yuxuan Zhou +5

1Generative AI Innovation Center, Amazon Web Services (AWS) · 2Cisco Systems, Inc.

Graph-Based Retrieval Augmented Generation Graph-Llm Integrations Knowledge Graphs Reasoning Gaps Prompt Optimization

Ask PDF

Sep 17, 2026cs.AI

RAFT: A Stateful Retrieval-Augmented Framework for Troubleshooting Agents

Effective troubleshooting agents in enterprise customer support depend on retrieving actionable guidance from similar historical cases, yet existing retrieval-augmented generation (RAG) systems treat support cases as static documents and overlook their multi-stage, stateful nature. We introduce RAFT (Retrieval-Augmented Framework for Troubleshooting Agents), a stateful RAG framework that abstracts each closed historical case into a directed chain of timeline entries and retrieves at the entry level, surfacing cases whose intermediate states match the active case and returning the parent-case trajectory anchored at the matched state; an optional case-level graph links cases through a configurable similarity representation. We evaluate this retrieval layer directly, which, unlike evaluating a full agent system, requires no production deployment. Because public multi-stage troubleshooting data is extremely rare, we pair a synthetic benchmark built from Microsoft Learn Windows Server documentation with real Apache Jira issues carrying human-created duplicate labels. RAFT improves Case Hit over vanilla RAG and GraphRAG baselines at every stage of case progress, with statistically significant gains over the strongest baseline; the Jira results provide directional evidence that the advantage transfers to real case histories. We release our benchmark, implementation, and the Apache Jira evaluation set.

Mingxuan Zhang, Xiaowen Wang, Anupma Sharan +4

Microsoft, Redmond, WA, USA

Hievi-Rag Clinical Notes Graph Representations

Ask PDF