cs.HCMar 6, 2026

Lexara: A User-Centered Toolkit for Evaluating Large Language Models for Conversational Visual Analytics

Authors: Srishti Palani, Vidya Setlur

Organizations: Tableau Research, Salesforce, Palo Alto, CA, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large Language Models (LLMs) are transforming Conversational Visual Analytics (CVA) by enabling data analysis through natural language. However, evaluating LLMs for CVA remains a challenge: requiring programming expertise, overlooking real-world complexity, and lacking interpretable metrics for multi-format (visualizations and text) outputs. Through interviews with 22 CVA developers and 16 end-users, we identified use cases, evaluation criteria and workflows. We present Lexara, a user-centered evaluation toolkit for CVA that operationalizes these insights into: (i) test cases spanning real-world scenarios; (ii) interpretable metrics covering visualization quality (data fidelity, semantic alignment, functional correctness, design clarity) and language quality (factual grounding, analytical reasoning, conversational coherence) using rule-based and LLM-as-a-Judge methods; and (iii) an interactive toolkit enabling experimental setup and multi-format and multi-level exploration of results without programming expertise. We conducted a two-week diary study with six CVA developers, drawn from our initial cohort of 22. Their feedback demonstrated Lexara's effectiveness for guiding appropriate model and prompt selection.

Figures & tables

Figure 1. Illustration of how practitioners evaluate the multi-format CVA response. The example shows (left) a user utterance and corresponding model outputs, and (right) the evaluation criteria identified in our formative studies: visualization response quality assessed by looking at both the rendered visualization and grammar specification (data fidelity, chart type, functionality, design), natural language response quality (factual grounding, analytical thinking and conversation quality). This figure provides a conceptual overview and does not reflect the actual UI of any CVA system. This figure shows how a user’s request, “Scatter of displacement vs MPG colored by origin,” is evaluated. At the top left, the raw data table is partially shown, with fields such as acceleration, cylinders, displacement, horsepower, and origin. To its right are partial JSON snippets specifying encodings: displacement on the x-axis and MPG on the y-axis, with “mark: point.” Further to the right, scatterplots illustrate different evaluation dimensions. The first scatterplot shows only functionality, plotting displacement versus MPG. The next highlights chart type as a scatterplot, followed by a version adding design with color encoding for car origin (Europe, USA, Japan). The rightmost scatterplot shows the final visualization with all elements together. Below the scatterplots, four blocks of natural language evaluation criteria are presented: “Factual Grounding” describes the negative relationship between engine displacement and MPG. “Insightfulness” highlights how different origins cluster by efficiency and performance. “Assumptions Disclosure” notes assumptions about consistent measurement across countries. “Conversation Quality” combines these insights into a coherent explanation. Together, the figure demonstrates how evaluators assess visualization outputs and language responses across multiple dimensions.
TestUser UtteranceExpected ResponseModel ResponseMetrics
1 Turn 1<Quantity> on y-axis and <Region> on x-axisA vertical bar chart with Region on the x-axis and the sum of Quantity on the y-axis (range 0–140). The Central region has the highest total ( 137), with other regions lower.A bar chart comparing Quantity (y-axis) across Regions (x-axis), allowing a quick scan of how quantities differ by region.Data Fidelity = 100% ; Field Similarity = 100% ; Chart Type Similarity = 100% ; Axis Accuracy = 100% ; Filter Accuracy = 100% ; Sort Accuracy = 100% ; Visual Encoding Accuracy = 100% . Overall Viz = 100% . NL: Factual Grounding 70% ; Assumptions Disclosure = 40% ; Insightfulness = 40% ; Coherence 90% . Overall NL 65% .
1 Turn 2Sort by <Quantity>A vertical bar chart of Regions, sorted in descending order by the sum of Quantity, so the highest-quantity region (Central, 137) appears first.A bar chart ordered by Quantity values for each Region; the description notes sorting by quantity but does not specify whether the order is ascending or descending.Data Fidelity = 100% ; Field Similarity = 100% ; Chart Type Similarity = 100% ; Axis Accuracy = 50% ; Filter Accuracy = 100% ; Sort Accuracy = 0% ; Visual Encoding Accuracy = 100% . Overall Viz 65% . NL: Factual Grounding 60% ; Assumptions Disclosure = 40% ; Insightfulness = 20% ; Coherence 80% . Overall NL 50% .
2show me top accounts by attendeesA horizontal bar chart of the Top 10 Account Names, ranked by total (sum) Item Quantity and sorted from highest to lowest, making it easy to spot the highest-volume accounts.A horizontal bar chart listing the Top 10 Account Names ranked by sum(Item Quantity), with bars sorted descending to highlight the largest contributors. (Accounts with null Item Quantity don’t contribute to the totals.)Data Fidelity = 100% ; Field Similarity 100% ; Chart Type Similarity = 100% ; Axis Accuracy = 100% ; Filter Accuracy = 100% ; Sort Accuracy = 100% ; Visual Encoding Accuracy 95% . Overall Viz 98% . NL: Factual Grounding = 100% ; Assumptions Disclosure = 100% ; Insightfulness = 20% ; Coherence 95% . Overall NL 85% .
3count orders by categoriesA bar chart that counts orders per Category—the x-axis lists categories and the y-axis shows the count of Order IDs.A bar chart breaking down Order IDs by Category, with each category represented by a different color to distinguish groups.Data Fidelity = 100% ; Field Similarity = 100% ; Chart Type Similarity 50% ; Axis Accuracy = 100% ; Filter Accuracy = 100% ; Sort Accuracy = 100% ; Visual Encoding Accuracy 85% . Overall Viz 90% . NL: Factual Grounding 60% ; Assumptions Disclosure = 20% ; Insightfulness = 20% ; Coherence 90% . Overall NL 50% .
4revenue versus earningsA scatterplot with Sales on the x-axis and Profit on the y-axis, one point per record. Plotting raw values reveals how profit changes with sales.A scatterplot comparing total Sales (x) to total Profit (y) so it collapses to a single point representing Sum of Sales vs. Sum of Profit.Data Fidelity = 0% ; Axis Accuracy = 95% ; All other viz metrics = 100% . Overall Viz = 85% ; NL: Factual Grounding 25% ; Assumptions Disclosure = 0% ; Insightfulness = 20% ; Coherence 90% . Overall NL 30% .
Table 1. Worked examples showing how Lexara metrics assess visualization and natural language quality responses interpretively. This table shows four worked examples of user utterances, expected responses, model responses, and detailed metrics. Each row corresponds to a test case. In the first row, the utterance “Quantity on y-axis and Region on x-axis” is shown with expected and model bar charts of Quantity by Region. The expected response sorts regions by quantity, while the model output also shows a bar chart. Metrics such as Data Fidelity (100%), Field Similarity (100%), and Assumptions Disclosure (40%) are listed. The second row shows “Sort by Quantity” with bar charts ordered by quantity; however, the model does not specify ascending or descending. Metrics show perfect axis accuracy but 0% for Sort Accuracy. The third row shows “Show me top accounts by attendees” with expected and model horizontal bar charts listing top accounts, with some mismatches in ordering. The fourth row shows “Revenue versus earnings,” where the expected response is a scatterplot of Sales versus Profit per record, while the model produces an aggregated scatterplot collapsing all records into one point. Metrics reflect an aggregation mismatch with Data Fidelity at 0Each row includes natural language descriptions of expected and actual charts, alongside metrics for visualization and language quality.
Figure 2. Lexara ’s interactive CVA evaluation interface supports two core workflows: (1) an Evaluation Setup Panel where practitioners upload datasources, define test cases, specify prompts, models, expected outputs, and configure CVA-specific metrics; and (2) an Interactive Results Table that streams model outputs—visualizations, structured specs, and natural language—side-by-side. The table enables multi-granular inspection, with expandable metric categories, on-hover explanations, and tools to trace divergences between expected and actual outputs.
Figure 3. For each user request, the system aligns expected and actual outputs across three formats: visualizations, natural language explanations, and JSON specifications. By surfacing detailed differences (e.g., encodings, aggregations, chart types), the interface enables practitioners pinpoint divergences, understand model behavior, and diagnose strengths or failure modes for various analytic tasks.
Figure 4. The overview panel (top left) highlights recommended model–prompt pairs and aggregated metrics. The label view (top right) breaks down results by chart type, ambiguity, and context-handling. The utterance-level view (bottom) contrasts expected vs. actual responses with detailed metric explanations.
PID# of Evaluation Experiments# of Test Cases Executed# of Models# of System Prompts
P156243
P210103102
P331651
P484551
P523242
P61085102
Table 2. Summary of evaluation experiments conducted by participants ( P1P1 – P6P6 ) during the two-week diary study. Each participant ran multiple experiments across varying test cases, models, and prompts. These figures provide a quantitative overview of how Lexara was appropriated in practice, complementing our qualitative insights.
Figure 5. A browser plugin recorded participants’ interactions with a popular CVA tool, capturing their utterances, model responses, in-the-loop evaluations via Likert-type scales, and corrected expected outputs. Screenshot series of a browser plugin recording participants' interactions with a conversational visual analytics tool. On the left, the tool's main interface shows data fields, filters, and a chart canvas in Tableau Agent. On the right, the plugin panel logs the participant’s utterances, model responses, Likert-type scales, and fields for corrections. The first screenshot shows a simple text table with minimal charting; the second shows a partially completed bar chart; the third shows a fully rendered stacked bar chart with colored bars by region. The side panel consistently shows user utterances, system responses, and rating sliders.
DatasourceDomainDetailsSource
SuperstoreBusiness & FinanceContains information about products, sales, and profits that can be used to identify key areas of improvement within this fictitious company.Tableau
The 2014 Inc. 5000Business & FinanceInc. Magazine’s annual list of the 5000 fastest growing private companies in the U.S., compiled by measuring each company’s percentage revenue growth over a four-year period.Inc. Magazine
Global Sport FinancesBusiness & FinanceThe top-paying pro sports teams and the top paid athletes.ESPN
American University Data (IPEDS)EducationPrimary source for data on colleges, universities, and technical/vocational postsecondary institutions in the U.S.National Center for Education Statistics
edX/HarvardX (AY 2012–2013)EducationDe-identified data from the first year (Fall 2012, Spring 2013, Summer 2013) of MITx and HarvardX courses on the edX platform.Harvard Dataverse
Life Expectancy WHOHealthcareHistorical and current life expectancy by country, often paired with other health indicators.World Health Organization
Table 3. Overview of datasources across business & finance, education, and healthcare domains, each linked to its original source.
Figure 6. In the formative study, participants reviewed three anonymized model outputs presented in a static table, alongside traditional NLG metrics including F1, Precision, Recall for each response.
Figure 7. Lexara metrics are both reliable and aligned with human judgments: (a–b) most metrics show κ\kappa and ρ\rho above 0.60.6 ; (c–d) models with higher Lexara visualization and natural language scores are also preferred by humans. Figure comprises four panels validating Lexara's evaluation metrics against human judgment. Panel (a) shows a bar chart of inter-rater reliability measured by Cohen's kappa across 13 metrics. Most metrics show kappa values between 0.45 and 0.80, indicating moderate to high agreement. The highest reliability (approximately 0.78-0.80) is observed for Data Fidelity, Field Similarity, and Chart Type Similarity. Factual Grounding also shows high reliability around 0.80. Interactivity shows the lowest reliability at approximately 0.45. The median kappa across all metrics is 0.65. Panel (b) displays a bar chart of metric-human alignment using Spearman's rho correlation. Values range from approximately 0.57 to 0.82. Factual Grounding achieves the strongest alignment at 0.82. Data Fidelity, Field Similarity, and Chart Type Similarity show correlations between 0.68 and 0.79. Natural language metrics including Insightfulness, Coherence, and Follow-up Relevance show moderate correlations between 0.57 and 0.71. Panel (c) presents a scatter plot comparing mean Lexara visualization scores (x-axis, 0-100) against mean human preference scores (y-axis, 0-100) for ten LLM models. Models are labeled and color-coded, including claude-3.7-sonnet and Deepseek (r1) at the upper right (high on both metrics), o4-mini, GPT-5-mini, and GPT-5-nano in the middle range, and GPT-5 at the lower left. The positive correlation (Spearman ρ = 0.79, p < 0.01) indicates models scoring higher on Lexara visualization metrics are also preferred by human evaluators. Panel (d) shows a similar scatter plot for natural language and conversation scores. The x-axis represents mean Lexara natural language scores (0-100) and y-axis shows human preference (0-100). o4-mini and claude-opus-4 appear in the upper right quadrant, claude-3.7-sonnet and Deepseek (r1) in the middle-upper range, and GPT-5, GPT-5-mini, and GPT-5-nano cluster in the lower left. The correlation is ρ = 0.74 (p < 0.05), demonstrating that Lexara's natural language quality metrics align with practitioner preferences. Overall, the figure demonstrates that Lexara's automated metrics are both internally reliable (consistent between human raters) and externally valid (aligned with human quality judgments), supporting their use for systematic CVA evaluation.
More in this topic

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Explore similar work

CardsList
Sep 15, 2026

Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents

Conversational visual analytics (CVA) agents powered by large language models generate visualizations and natural-language explanations from open-ended queries. Evaluating these multimodal outputs is challenging: curated reference benchmarks are costly to author, cannot comprehensively capture the space of valid responses, and are unavailable in production. Building on the Lexara evaluation framework, we introduce Lexara-RF, a reference-free set of metrics that scores CVA outputs using only the prompt, data, and model response. We reformulate evaluation as verification: 13 metrics operationalize visualization design theory and Gricean cooperative principles as computable consistency, intent-alignment, and design validity checks. On a human-rated corpus of CVA test-cases, Lexara-RF achieves alignment comparable to reference-based formulations, outperforms surface-similarity NLG baselines, and localizes structurally grounded failures with high accuracy.

Srishti Palani, Vidya Setlur

Tableau Research, Palo Alto, CA, USA

Agent EvaluationVisual AnalyticsReference-Free EvaluationMultimodal Agent Evaluation
AskPDF
Feb 23, 2026

Do Large Language Models Understand Data Visualization Rules?

Data visualization rules-derived from decades of research in design and perception-ensure trustworthy chart communication. While prior work has shown that large language models (LLMs) can generate charts or flag misleading figures, it remains unclear whether they can reason about and enforce visualization rules directly. Constraint-based systems such as Draco encode these rules as logical constraints for precise automated checks, but maintaining symbolic encodings requires expert effort, motivating the use of LLMs as flexible rule validators. In this paper, we present the first systematic evaluation of LLMs against visualization rules using hard-verification ground truth derived from Answer Set Programming (ASP). We translated a subset of Draco's constraints into natural-language statements and generated a controlled dataset of 2,000 Vega-Lite specifications annotated with explicit rule violations. LLMs were evaluated on both accuracy in detecting violations and prompt adherence, which measures whether outputs follow the required structured format. Results show that frontier models achieve high adherence (Gemma 3 4B / 27B: 100%, GPT-oss 20B: 98%) and reliably detect common violations (F1 up to 0.82),yet performance drops for subtler perceptual rules (F1 < 0.15 for some categories) and for outputs generated from technical ASP formulations.Translating constraints into natural language improved performance by up to 150% for smaller models. These findings demonstrate the potential of LLMs as flexible, language-driven validators while highlighting their current limitations compared to symbolic solvers.

Martin Sinnona, Valentin Bonas, Emmanuel Iarussi +1

Universidad Torcuato Di Tella, Buenos Aires, Argentina · Universidad de Buenos Aires, Buenos Aires, Argentina · Consejo Nacional de Investigaciones Científicas y Técnicas, Buenos Aires, Argentina

Data VisualizationLLM Evaluation
AskPDF
Apr 1, 2026

True (VIS) Lies: Analyzing How Generative AI Recognizes Intentionality, Rhetoric, and Misleadingness in Visualization Lies

This study investigates the ability of multimodal Large Language Models (LLMs) to identify and interpret misleading visualizations, and recognize these observations along with their underlying causes and potential intentionality. Our analysis leverages concepts from visualization rhetoric and a newly developed taxonomy of authorial intents as explanatory lenses. We formulated three research questions and addressed them experimentally using a dataset of 2,336 COVID-19-related tweets, half of which contain misleading visualizations, and supplemented it with real-world examples of perceptual, cognitive, and conceptual errors drawn from VisLies, the IEEE VIS community event dedicated to showcasing deceptive and misleading visualizations. To ensure broad coverage of the current LLM landscape, we evaluated 16 state-of-the-art models. Among them, 15 are open-weight models, spanning a wide range of model sizes, architectural families, and reasoning capabilities. The selection comprises small models, namely Nemotron-Nano-V2-VL (12B parameters), Mistral-Small-3.2 (24B), DeepSeek-VL2 (27B), Gemma3 (27B), and GTA1 (32B); medium-sized models, namely Qianfan-VL (70B), Molmo (72B), GLM-4.5V (108B), LLaVA-NeXT (110B), and Pixtral-Large (124B); and large models, namely Qwen3-VL (235B), InternVL3.5 (241B), Step3 (321B), Llama-4-Maverick (400B), and Kimi-K2.5 (1000B). In addition, we employed OpenAI GPT-5.4, a frontier proprietary model. To establish a human perspective on these tasks, we also conducted a user study with visualization experts to assess how people perceive rhetorical techniques and the authorial intentions behind the same misleading visualizations. This allows comparison between model and expert behavior, revealing similarities and differences that provide insights into where LLMs align with human judgment and where they diverge.

Graziano Blasilli, Marco Angelini

Sapienza University of Rome, Rome, Italy. · Link Campus University, Rome, Italy.

Data VisualizationVLM EvaluationIntent Classification
AskPDF