cs.CLSep 6, 2026
SaveInsightChain: Optimized Chain-of-Insight Analytics for LLM-driven Data Visualization
Organizations: Huawei Technologies Co., Ltd.
Abstract
Large language models (LLMs) are increasingly used for automated data visualization, yet existing approaches often frame visualization generation as a single-step mapping from user query to figure or code, overlooking the iterative analytical reasoning process of expert analysts. We present InsightChain, a four-stage visualization prompting pipeline (Explore--Focus--Test--Present) that emulates expert analytical workflows, together with VG-COPRO, a vision-guided automatic prompt optimization (APO) method adapted to jointly optimize such multi-stage, executable pipelines. To address the evaluation gap for complex data visualization, we introduce the Insight Progression Metric (IPM), a rubric combining four text-based dimensions with a vision-based dimension. We assess IPM through a 100-chain human pilot and an expanded 300-chain agent-based evaluation spanning all ten domains. Experiments on public datasets show that InsightChain consistently outperforms competing prompting baselines. Existing APO methods fail to yield consistent gains on this multi-stage task, whereas VG-COPRO improves performance in both in-domain and cross-domain settings.
Figures & tables
Figure 1: System overview. InsightChain (left) consists of (i) a four-stage Explore Focus Test Present pipeline, ingesting tabular datasets and emitting an output artefact ( ) at each stage that conditions the next; and (ii) VG-COPRO, the prompt optimizer that wraps an OPRO-style proposer LM in COPRO’s per-predictor outer loop, with three modifications selecting candidate prompts on a 20-example validation pool by composite IPM score. IPM (right): a five-dimensional rubric with weights scored by an LLM-as-judge, producing a composite score that serves both as the evaluation metric and as the VG-COPRO compile signal.
Figure 2: IPM discriminability on the 100-chain pilot. Left : aggregate 5-dim IPM by quality band. Right : per-dimension ANOVA statistic (all ).
| In-domain (n=35) | Cross-domain (n=135) | ||||||||
| Task model | Method | IPM | IPM | ||||||
| Gemini 3 Flash | InsightChain (pre-APO) | 7.24 | – | – | – | 7.38 | – | – | – |
| Single-shot 4-prompt | 6.05 | 6.38 | |||||||
| LIDA | 5.79 | 5.93 | |||||||
| MIPROv2 | 7.20 | – | 7.15 | – | |||||
| COPRO | 7.08 | – | 7.05 | – | |||||
Table 1: Main-result table. at the same task model; is a one-sided paired -test; is the paired Cohen’s effect size. Negative means the pre-APO wins. Within each task model, rows are grouped as: (a) InsightChain before APO optimization, (b) non-chain baselines, and (c) InsightChain with APO.
Figure 3: Qualitative comparison of visualizations on the malaria_inc dataset. (a) Non-chain baseline: Produces a misleading chart where the title ( ) contradicts the annotated value ( ), triggering faithfulness-gate violation (VQ capped at 3). (b) InsightChain pipeline (pre-APO): Correct analytical pattern but poor layout and visual clutter. (c) InsightChain with VG-COPRO: A dual-trend plot with clear annotations and insights.
| Chain | Single | LIDA | |||
| (a) Other Evaluation Metrics ( ) | |||||
| CLIP img–cap. | |||||
| C2T BLEU | – | – | |||
| LIDA VLM | |||||
| IPM (ours) | 7.36 | ||||
| (b) Judge-free Code Validation ( ) | |||||
Table 2: Results on cross-domain test set (means reported for InsightChain, Single-shot, and LIDA). and measure the difference and effect size between InsightChain and Single-shot. (a) Three common visualization metrics fail to separate the methods, while IPM distinctly separates them ( ). (b) Two code metrics favor InsightChain ( ).
| In-dom ( ) | Cross-dom ( ) | |||||
| Variant | IPM | IPM | ||||
| Hybrid | ||||||
| + Mod 1 | ||||||
| + Mod 2 | ||||||
| + Mod 3 | ||||||
| VG-COPRO | ||||||
Table 3: Ablation of the three modifications for VG-COPRO (§ 3.4 ). Hybrid baseline is based on the results of OPRO on chain pipeline. Each modification produces a significant improvement on both splits (all for rows other than the Hybrid baseline). Mod 2 yields the largest mean ; the full combination yields the largest paired Cohen’s , indicating lowest dispersion.
| Split | n | Chain | Single-shot | ||
| In-domain | 35 | ||||
| Cross-domain | 135 |
Table 4: T2I transferability: mean IPM of InsightChain (T2I Present) vs. single-shot T2I baseline on the full test set. refers to the number of sampled datasets. All by one-sided paired -test.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Dimension | MAE | h IPM | ||
| Exploration Breadth | ||||
| Focus Relevance | ||||
| Hypothesis Rigor | ||||
| Insight Communication | ||||
| Visual Quality | ||||
| Composite |
Table 5: Human–IPM agreement on blind-annotated pilot chains. , = Pearson / Spearman correlations between human and IPM scores on the 1–10 scale (all ); MAE = mean absolute error; h IPM = human-mean minus IPM-mean (the scale-offset bias). Composite row is human overall-quality vs. the IPM 5-dim weighted composite. The vision-judged Visual Quality is essentially calibrated to the human ( ); the four text dimensions show a uniform -to- strictness offset that does not affect rank-based agreement.
| (a) Per-dimension self-consistency ( runs) | |||
| Dimension | n | ||
| Exploration Breadth | 100 | ||
| Focus Relevance | 100 | ||
| Hypothesis Rigor | 100 | ||
| Insight Communication | 100 | ||
| Visual Quality | 0 99 | ||
Table 6: IPM judge reliability. (a) Per-dimension self-consistency on the 100-chain pilot: all five dimensions are independently rescored three times at temperature ; = number of evaluated chains with complete three-run scores, = Krippendorff’s interval alpha across three judge runs, and = mean pairwise Pearson . (b) Cross-family validation against the in-family Gemini 3 Flash judge: “Gem” is the in-family mean on the matched subsample, “XJ” is the cross-family judge’s mean, and reports the mean score difference. Both cross-family judges return means at or above Gemini, providing no evidence of in-family score inflation on this matched subsample.
Figure 4: Inter-dimension Pearson correlations on 508 complete chains from the original three-condition evaluation: 170 datasets evaluated with single-shot, uncompiled-chain, and LIDA baselines, excluding two failed LIDA evaluations. Mean off-diagonal ; max (Foc–Hyp). Cronbach’s on the full 5-tuple is ; the leave-one-out values are Expl , Foc , Hyp , IC , VQ — Visual Quality is the only dimension whose removal raises , consistent with its capturing variance not covered by the four text-judged dimensions. Expl = Exploration Breadth, Foc = Focus Relevance, Hyp = Hypothesis Rigor, IC = Insight Communication, VQ = Visual Quality.
| Source | Pairs | Construction |
| Seed pair | Same dataset and prompt; different seed | |
| Regeneration | Same dataset and quality regime | |
| Metric-close | Same regime, different datasets, |
Table 7: Composition of the close-pair evaluation set.
| Metric | Multi- stage | Vision- judged | Faithful- ness gate | Insight- aware | Spec- free |
| nvBench-style spec match ( Luo et al., 2021 ) | ✗ | ✗ | ✗ | ✗ | ✗ |
| Chart-to-Text BLEU ( Kantharaj et al., 2022 ) | ✗ | ✗ | ✗ | ✗ | ✓ |
| CLIP ( Radford et al., 2021 ) | ✗ | ✓ | ✗ | ✗ | ✓ |
| LIDA partial VLM eval ( Dibia, 2023 ) | ✗ | partial | ✗ | partial | ✓ |
| InsightBench rubric ( Sahu et al., 2025 ) | text only | ✗ | ✗ | ✓ | ✓ |
| IPM (ours) | ✓ | ✓ | ✓ | ✓ | ✓ |
Table 8: Visualization quality metrics by design property. Multi-stage : does the metric score intermediate chain stages rather than only the final figure? Vision-judged : does it consume the rendered image? Faithfulness gate : is there an explicit cap for misleading figures? Insight-aware : does it reward task-specific finding extraction? Spec-free : does it avoid requiring a gold chart specification?
| Method | Search algorithm | What is searched |
| MIPROv2 | Bayesian / TPE | instr. + demos |
| COPRO | Coordinate ascent | instr. only |
| OPRO | LLM-as-optimizer | instr. only |
| EvoPrompt | Evolutionary | instr. only |
| Plan-and-Solve | none (template) | — |
Table 9: The five APO paradigms compiled and evaluated under identical splits, metric, and budget. Plan-and-Solve has no learnable parameters and serves as a structured-template control.
Figure 5: Checkpoint-reconstructed optimization dynamics over three compile iterations. Each panel plots the validation-pool mean of the incumbent selected under each method’s own objective. Curves are comparable over iterations within a method, but absolute heights should not be compared across metric variants: OPRO, EvoPrompt, and Mod 1 use four text dimensions, whereas Mod 3 uses the full five-dimensional IPM. VG-COPRO and the isolated LODO variant are omitted because their worst-fold selection objective is not directly comparable to the plotted mean-score selection trajectories; methods without reconstructable validation-mean histories are also omitted.
Figure 6: Complete healthcare analysis chain. The example shows the pipeline revising an exploratory impression after formal testing rather than forcing a positive finding.
Figure 7: Complete transportation analysis chain. This held-out-domain example progresses from broad enforcement patterns to a quantified agency-level difference with uncertainty and effect size.
Figure 8: Complete ecommerce analysis chain. This held-out-domain example preserves a nuanced negative decision: the directional effect is significant, but its magnitude does not meet the threshold specified before the Test stage and is dominated by outliers.
Figure 9: Annotator handbook (1/8): annotation unit, required outputs, and scoring anchors for Exploration Breadth .
Figure 10: Annotator handbook (2/8): scoring anchors for Focus Relevance and Hypothesis Rigor .
Figure 11: Annotator handbook (3/8): scoring anchors for text-only Insight Communication .
Figure 12: Annotator handbook (4/8): scoring anchors and calibration examples for rendered Visual Quality , including the faithfulness and display gates.
Figure 13: Annotator handbook (5/8): holistic quality and the three independent final-insight judgments.
Figure 14: Annotator handbook (6/8): row-level annotation workflow and time budget.
Figure 15: Annotator handbook (7/8): calibration and blinding rules.
Figure 16: Annotator handbook (8/8): derived agreement measures and handling of malformed, missing, or ambiguous cases.
| Benchmark | # Datasets | Domains | Multi- step | Cross- domain | Insight Metric | Vision Judge |
| nvBench ( Luo et al., 2021 ) | 750 | Narrow | ✗ | ✗ | ✗ a | ✗ |
| Prompt4Vis ( Li et al., 2025a ) | subset | Narrow | ✗ | ✗ | ✗ | ✗ |
| Chart-to-Text ( Kantharaj et al., 2022 ) | 44k | – | ✗ | ✗ | ✗ b | ✗ |
| LIDA exemplars ( Dibia, 2023 ) | ad-hoc | Few | ✗ | ✗ | ✗ | Partial |
| InsightBench ( Sahu et al., 2025 ) | 100 | Mixed | Text-only | ✗ | Text | ✗ |
| InsightEval ( Zhu et al., 2026 ) | – | Mixed | Text-only | ✗ | Text | ✗ |
Table 10: InsightChain vs. prior LLM-visualization and insight-quality benchmarks. a Relies on chart-spec matching. b Relies on text-based BLEU scores. ∗ Includes 5 held-out domains. To our knowledge, InsightChain is the first benchmark to combine a held-out cross-domain partition with a multi-step chain evaluation and a vision-aware judge.
| Source | # | % |
| TidyTuesday (community-curated) | 55 | 19.3 |
| OpenML (keyword search per domain) | 53 | 18.6 |
| Other public CSV (Kaggle / project pages) | 130 | 45.6 |
| World Bank, FRED & other macro feeds | 11 | 3.9 |
| UCI ML Repository | 8 | 2.8 |
| Yfinance / equity feeds | 4 | 1.4 |
Table 11: Source composition of the InsightChain experiment datasets. Synthetic templates ( ) are confined to demographics and energy where real public coverage was thinnest after quality filtering.
Figure 17: Distribution of datasets across the 10 target domains.
| Split | n | Source |
| Train | 95 | 5 train dom. 19 |
| Val | 20 | 5 train dom. 4 |
| In-domain test | 35 | 5 train dom. 7 |
| Cross-domain test | 135 | 5 hold-out dom. (all) |
Table 12: Split sizes. refers to the number of sampled datasets. The Train split is used for compilation only, and Val for APO candidate selection. The cross-domain test set is the central testbed for the transfer claim of § 5.3 .
Figure 18: Per-domain mean 5-dim IPM under the Gemini 3 Flash task model. InsightChain wins every in-domain training domain and every cross-domain hold-out.