cs.HCMar 6, 2026
SaveLexara: A User-Centered Toolkit for Evaluating Large Language Models for Conversational Visual Analytics
Organizations: Tableau Research, Salesforce, Palo Alto, CA, USA
Abstract
Large Language Models (LLMs) are transforming Conversational Visual Analytics (CVA) by enabling data analysis through natural language. However, evaluating LLMs for CVA remains a challenge: requiring programming expertise, overlooking real-world complexity, and lacking interpretable metrics for multi-format (visualizations and text) outputs. Through interviews with 22 CVA developers and 16 end-users, we identified use cases, evaluation criteria and workflows. We present Lexara, a user-centered evaluation toolkit for CVA that operationalizes these insights into: (i) test cases spanning real-world scenarios; (ii) interpretable metrics covering visualization quality (data fidelity, semantic alignment, functional correctness, design clarity) and language quality (factual grounding, analytical reasoning, conversational coherence) using rule-based and LLM-as-a-Judge methods; and (iii) an interactive toolkit enabling experimental setup and multi-format and multi-level exploration of results without programming expertise. We conducted a two-week diary study with six CVA developers, drawn from our initial cohort of 22. Their feedback demonstrated Lexara's effectiveness for guiding appropriate model and prompt selection.
Figures & tables
Figure 1. Illustration of how practitioners evaluate the multi-format CVA response. The example shows (left) a user utterance and corresponding model outputs, and (right) the evaluation criteria identified in our formative studies: visualization response quality assessed by looking at both the rendered visualization and grammar specification (data fidelity, chart type, functionality, design), natural language response quality (factual grounding, analytical thinking and conversation quality). This figure provides a conceptual overview and does not reflect the actual UI of any CVA system. This figure shows how a user’s request, “Scatter of displacement vs MPG colored by origin,” is evaluated. At the top left, the raw data table is partially shown, with fields such as acceleration, cylinders, displacement, horsepower, and origin. To its right are partial JSON snippets specifying encodings: displacement on the x-axis and MPG on the y-axis, with “mark: point.” Further to the right, scatterplots illustrate different evaluation dimensions. The first scatterplot shows only functionality, plotting displacement versus MPG. The next highlights chart type as a scatterplot, followed by a version adding design with color encoding for car origin (Europe, USA, Japan). The rightmost scatterplot shows the final visualization with all elements together. Below the scatterplots, four blocks of natural language evaluation criteria are presented: “Factual Grounding” describes the negative relationship between engine displacement and MPG. “Insightfulness” highlights how different origins cluster by efficiency and performance. “Assumptions Disclosure” notes assumptions about consistent measurement across countries. “Conversation Quality” combines these insights into a coherent explanation. Together, the figure demonstrates how evaluators assess visualization outputs and language responses across multiple dimensions.
| Test | User Utterance | Expected Response | Model Response | Metrics |
|---|---|---|---|---|
| 1 Turn 1 | <Quantity> on y-axis and <Region> on x-axis | A vertical bar chart with Region on the x-axis and the sum of Quantity on the y-axis (range 0–140). The Central region has the highest total ( 137), with other regions lower. | A bar chart comparing Quantity (y-axis) across Regions (x-axis), allowing a quick scan of how quantities differ by region. | Data Fidelity = 100% ; Field Similarity = 100% ; Chart Type Similarity = 100% ; Axis Accuracy = 100% ; Filter Accuracy = 100% ; Sort Accuracy = 100% ; Visual Encoding Accuracy = 100% . Overall Viz = 100% . NL: Factual Grounding 70% ; Assumptions Disclosure = 40% ; Insightfulness = 40% ; Coherence 90% . Overall NL 65% . |
| 1 Turn 2 | Sort by <Quantity> | A vertical bar chart of Regions, sorted in descending order by the sum of Quantity, so the highest-quantity region (Central, 137) appears first. | A bar chart ordered by Quantity values for each Region; the description notes sorting by quantity but does not specify whether the order is ascending or descending. | Data Fidelity = 100% ; Field Similarity = 100% ; Chart Type Similarity = 100% ; Axis Accuracy = 50% ; Filter Accuracy = 100% ; Sort Accuracy = 0% ; Visual Encoding Accuracy = 100% . Overall Viz 65% . NL: Factual Grounding 60% ; Assumptions Disclosure = 40% ; Insightfulness = 20% ; Coherence 80% . Overall NL 50% . |
| 2 | show me top accounts by attendees | A horizontal bar chart of the Top 10 Account Names, ranked by total (sum) Item Quantity and sorted from highest to lowest, making it easy to spot the highest-volume accounts. | A horizontal bar chart listing the Top 10 Account Names ranked by sum(Item Quantity), with bars sorted descending to highlight the largest contributors. (Accounts with null Item Quantity don’t contribute to the totals.) | Data Fidelity = 100% ; Field Similarity 100% ; Chart Type Similarity = 100% ; Axis Accuracy = 100% ; Filter Accuracy = 100% ; Sort Accuracy = 100% ; Visual Encoding Accuracy 95% . Overall Viz 98% . NL: Factual Grounding = 100% ; Assumptions Disclosure = 100% ; Insightfulness = 20% ; Coherence 95% . Overall NL 85% . |
| 3 | count orders by categories | A bar chart that counts orders per Category—the x-axis lists categories and the y-axis shows the count of Order IDs. | A bar chart breaking down Order IDs by Category, with each category represented by a different color to distinguish groups. | Data Fidelity = 100% ; Field Similarity = 100% ; Chart Type Similarity 50% ; Axis Accuracy = 100% ; Filter Accuracy = 100% ; Sort Accuracy = 100% ; Visual Encoding Accuracy 85% . Overall Viz 90% . NL: Factual Grounding 60% ; Assumptions Disclosure = 20% ; Insightfulness = 20% ; Coherence 90% . Overall NL 50% . |
| 4 | revenue versus earnings | A scatterplot with Sales on the x-axis and Profit on the y-axis, one point per record. Plotting raw values reveals how profit changes with sales. | A scatterplot comparing total Sales (x) to total Profit (y) so it collapses to a single point representing Sum of Sales vs. Sum of Profit. | Data Fidelity = 0% ; Axis Accuracy = 95% ; All other viz metrics = 100% . Overall Viz = 85% ; NL: Factual Grounding 25% ; Assumptions Disclosure = 0% ; Insightfulness = 20% ; Coherence 90% . Overall NL 30% . |
Table 1. Worked examples showing how Lexara metrics assess visualization and natural language quality responses interpretively. This table shows four worked examples of user utterances, expected responses, model responses, and detailed metrics. Each row corresponds to a test case. In the first row, the utterance “Quantity on y-axis and Region on x-axis” is shown with expected and model bar charts of Quantity by Region. The expected response sorts regions by quantity, while the model output also shows a bar chart. Metrics such as Data Fidelity (100%), Field Similarity (100%), and Assumptions Disclosure (40%) are listed. The second row shows “Sort by Quantity” with bar charts ordered by quantity; however, the model does not specify ascending or descending. Metrics show perfect axis accuracy but 0% for Sort Accuracy. The third row shows “Show me top accounts by attendees” with expected and model horizontal bar charts listing top accounts, with some mismatches in ordering. The fourth row shows “Revenue versus earnings,” where the expected response is a scatterplot of Sales versus Profit per record, while the model produces an aggregated scatterplot collapsing all records into one point. Metrics reflect an aggregation mismatch with Data Fidelity at 0Each row includes natural language descriptions of expected and actual charts, alongside metrics for visualization and language quality.
Figure 2. Lexara ’s interactive CVA evaluation interface supports two core workflows: (1) an Evaluation Setup Panel where practitioners upload datasources, define test cases, specify prompts, models, expected outputs, and configure CVA-specific metrics; and (2) an Interactive Results Table that streams model outputs—visualizations, structured specs, and natural language—side-by-side. The table enables multi-granular inspection, with expandable metric categories, on-hover explanations, and tools to trace divergences between expected and actual outputs.
Figure 3. For each user request, the system aligns expected and actual outputs across three formats: visualizations, natural language explanations, and JSON specifications. By surfacing detailed differences (e.g., encodings, aggregations, chart types), the interface enables practitioners pinpoint divergences, understand model behavior, and diagnose strengths or failure modes for various analytic tasks.
Figure 4. The overview panel (top left) highlights recommended model–prompt pairs and aggregated metrics. The label view (top right) breaks down results by chart type, ambiguity, and context-handling. The utterance-level view (bottom) contrasts expected vs. actual responses with detailed metric explanations.
| PID | # of Evaluation Experiments | # of Test Cases Executed | # of Models | # of System Prompts |
|---|---|---|---|---|
| P1 | 5 | 62 | 4 | 3 |
| P2 | 10 | 103 | 10 | 2 |
| P3 | 3 | 16 | 5 | 1 |
| P4 | 8 | 45 | 5 | 1 |
| P5 | 2 | 32 | 4 | 2 |
| P6 | 10 | 85 | 10 | 2 |
Table 2. Summary of evaluation experiments conducted by participants ( – ) during the two-week diary study. Each participant ran multiple experiments across varying test cases, models, and prompts. These figures provide a quantitative overview of how Lexara was appropriated in practice, complementing our qualitative insights.
Figure 5. A browser plugin recorded participants’ interactions with a popular CVA tool, capturing their utterances, model responses, in-the-loop evaluations via Likert-type scales, and corrected expected outputs. Screenshot series of a browser plugin recording participants' interactions with a conversational visual analytics tool. On the left, the tool's main interface shows data fields, filters, and a chart canvas in Tableau Agent. On the right, the plugin panel logs the participant’s utterances, model responses, Likert-type scales, and fields for corrections. The first screenshot shows a simple text table with minimal charting; the second shows a partially completed bar chart; the third shows a fully rendered stacked bar chart with colored bars by region. The side panel consistently shows user utterances, system responses, and rating sliders.
| Datasource | Domain | Details | Source |
|---|---|---|---|
| Superstore | Business & Finance | Contains information about products, sales, and profits that can be used to identify key areas of improvement within this fictitious company. | Tableau |
| The 2014 Inc. 5000 | Business & Finance | Inc. Magazine’s annual list of the 5000 fastest growing private companies in the U.S., compiled by measuring each company’s percentage revenue growth over a four-year period. | Inc. Magazine |
| Global Sport Finances | Business & Finance | The top-paying pro sports teams and the top paid athletes. | ESPN |
| American University Data (IPEDS) | Education | Primary source for data on colleges, universities, and technical/vocational postsecondary institutions in the U.S. | National Center for Education Statistics |
| edX/HarvardX (AY 2012–2013) | Education | De-identified data from the first year (Fall 2012, Spring 2013, Summer 2013) of MITx and HarvardX courses on the edX platform. | Harvard Dataverse |
| Life Expectancy WHO | Healthcare | Historical and current life expectancy by country, often paired with other health indicators. | World Health Organization |
Table 3. Overview of datasources across business & finance, education, and healthcare domains, each linked to its original source.
Figure 6. In the formative study, participants reviewed three anonymized model outputs presented in a static table, alongside traditional NLG metrics including F1, Precision, Recall for each response.
Figure 7. Lexara metrics are both reliable and aligned with human judgments: (a–b) most metrics show and above ; (c–d) models with higher Lexara visualization and natural language scores are also preferred by humans. Figure comprises four panels validating Lexara's evaluation metrics against human judgment. Panel (a) shows a bar chart of inter-rater reliability measured by Cohen's kappa across 13 metrics. Most metrics show kappa values between 0.45 and 0.80, indicating moderate to high agreement. The highest reliability (approximately 0.78-0.80) is observed for Data Fidelity, Field Similarity, and Chart Type Similarity. Factual Grounding also shows high reliability around 0.80. Interactivity shows the lowest reliability at approximately 0.45. The median kappa across all metrics is 0.65. Panel (b) displays a bar chart of metric-human alignment using Spearman's rho correlation. Values range from approximately 0.57 to 0.82. Factual Grounding achieves the strongest alignment at 0.82. Data Fidelity, Field Similarity, and Chart Type Similarity show correlations between 0.68 and 0.79. Natural language metrics including Insightfulness, Coherence, and Follow-up Relevance show moderate correlations between 0.57 and 0.71. Panel (c) presents a scatter plot comparing mean Lexara visualization scores (x-axis, 0-100) against mean human preference scores (y-axis, 0-100) for ten LLM models. Models are labeled and color-coded, including claude-3.7-sonnet and Deepseek (r1) at the upper right (high on both metrics), o4-mini, GPT-5-mini, and GPT-5-nano in the middle range, and GPT-5 at the lower left. The positive correlation (Spearman ρ = 0.79, p < 0.01) indicates models scoring higher on Lexara visualization metrics are also preferred by human evaluators. Panel (d) shows a similar scatter plot for natural language and conversation scores. The x-axis represents mean Lexara natural language scores (0-100) and y-axis shows human preference (0-100). o4-mini and claude-opus-4 appear in the upper right quadrant, claude-3.7-sonnet and Deepseek (r1) in the middle-upper range, and GPT-5, GPT-5-mini, and GPT-5-nano cluster in the lower left. The correlation is ρ = 0.74 (p < 0.05), demonstrating that Lexara's natural language quality metrics align with practitioner preferences. Overall, the figure demonstrates that Lexara's automated metrics are both internally reliable (consistent between human raters) and externally valid (aligned with human quality judgments), supporting their use for systematic CVA evaluation.
More in this topic
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,408 papers