InsightChain: Optimized Chain-of-Insight Analytics for LLM-driven Data Visualization

Hanya Sun; Chen Zhang; Sheng Liang; Yongyue Zhang; Yong Liu

cs.CLSep 6, 2026

InsightChain: Optimized Chain-of-Insight Analytics for LLM-driven Data Visualization

Authors: Hanya Sun, Chen Zhang, Sheng Liang, Yongyue Zhang, Yong Liu

Organizations: Huawei Technologies Co., Ltd.

Abstract

Large language models (LLMs) are increasingly used for automated data visualization, yet existing approaches often frame visualization generation as a single-step mapping from user query to figure or code, overlooking the iterative analytical reasoning process of expert analysts. We present InsightChain, a four-stage visualization prompting pipeline (Explore--Focus--Test--Present) that emulates expert analytical workflows, together with VG-COPRO, a vision-guided automatic prompt optimization (APO) method adapted to jointly optimize such multi-stage, executable pipelines. To address the evaluation gap for complex data visualization, we introduce the Insight Progression Metric (IPM), a rubric combining four text-based dimensions with a vision-based dimension. We assess IPM through a 100-chain human pilot and an expanded 300-chain agent-based evaluation spanning all ten domains. Experiments on public datasets show that InsightChain consistently outperforms competing prompting baselines. Existing APO methods fail to yield consistent gains on this multi-stage task, whereas VG-COPRO improves performance in both in-domain and cross-domain settings.

Figures & tables

Figure 1: System overview. InsightChain (left) consists of (i) a four-stage Explore $\to$ Focus $\to$ Test $\to$ Present pipeline, ingesting tabular datasets and emitting an output artefact ( $o_{t}$ ) at each stage that conditions the next; and (ii) VG-COPRO, the prompt optimizer that wraps an OPRO-style proposer LM in COPRO’s per-predictor outer loop, with three modifications selecting candidate prompts on a 20-example validation pool by composite IPM score. IPM (right): a five-dimensional rubric with weights $(0.20,0.25,0.20,0.25,0.10)$ scored by an LLM-as-judge, producing a composite score that serves both as the evaluation metric and as the VG-COPRO compile signal.

Figure 2: IPM discriminability on the 100-chain pilot. Left : aggregate 5-dim IPM by quality band. Right : per-dimension ANOVA

F

statistic (all

p<10^{-4}

		In-domain (n=35)				Cross-domain (n=135)
Task model	Method	IPM	$\Delta$	$p_{1}$	$d$	IPM	$\Delta$	$p_{1}$	$d$
Gemini 3 Flash	InsightChain (pre-APO)	7.24	–	–	–	7.38	–	–	–
	Single-shot 4-prompt	6.05	$-1.19$	$1.7{\times}10^{-7}$	$1.07$	6.38	$-1.00$	$1.6{\times}10^{-22}$	$1.01$
	LIDA	5.79	$-1.46$	$5.6{\times}10^{-10}$	$1.40$	5.93	$-1.45$	$1.0{\times}10^{-23}$	$1.05$
	MIPROv2	7.20	$-0.05$	$0.63$	–	7.15	$-0.24$	$1.00$	–
	COPRO	7.08	$-0.16$	$0.82$	–	7.05	$-0.33$	$1.00$	–

Table 1: Main-result table.

\Delta=\text{method}-\text{pre-APO}

at the same task model;

p_{1}

is a one-sided paired

t

-test;

d

is the paired Cohen’s effect size. Negative

\Delta

means the pre-APO wins. Within each task model, rows are grouped as: (a) InsightChain before APO optimization, (b) non-chain baselines, and (c) InsightChain with APO.

Panel of Figure 3 — Figure 3: Qualitative comparison of visualizations on the malaria_inc dataset. (a) Non-chain baseline: Produces a misleading chart where the title ( $37\%$ ) contradicts the annotated value ( $54.9\%$ ), triggering faithfulness-gate violation (VQ capped at 3). (b) InsightChain pipeline (pre-APO): Correct analytical pattern but poor layout and visual clutter. (c) InsightChain with VG-COPRO: A dual-trend plot with clear annotations and insights.

	Chain	Single	LIDA	$\Delta$	$d$
(a) Other Evaluation Metrics ( $n=135$ )
CLIP img–cap.	$0.305$	$0.300$	$0.300$	$+0.00$	$0.05$
C2T BLEU	$\sim\!0$	$\sim\!0$	$\sim\!0$	–	–
LIDA VLM	$9.38$	$9.47$	$9.38$	$-0.10$	$-0.19$
IPM (ours)	7.36	$6.32$	$5.97$	$\mathbf{+1.04}$	$\mathbf{1.07}$
(b) Judge-free Code Validation ( $n=133$ )

Table 2: Results on cross-domain test set (means reported for InsightChain, Single-shot, and LIDA).

\Delta

and

d

measure the difference and effect size between InsightChain and Single-shot. (a) Three common visualization metrics fail to separate the methods, while IPM distinctly separates them (

d>1

). (b) Two code metrics favor InsightChain (

p<10^{-5}

	In-dom ( $n{=}35$ )			Cross-dom ( $n{=}135$ )
Variant	IPM	$\Delta$	$d$	IPM	$\Delta$	$d$
Hybrid	$7.55$	$+0.49^{\dagger}$	$0.28$	$7.27$	$+0.24^{\dagger}$	$0.19$
+ Mod 1	$8.54$	$+1.30$	$1.06$	$8.65$	$+1.26$	$0.84$
+ Mod 2	$8.82$	$\mathbf{+1.57}$	$1.47$	$8.74$	$\mathbf{+1.36}$	$1.06$
+ Mod 3	$8.50$	$+1.26$	$0.93$	$8.61$	$+1.23$	$0.95$
VG-COPRO	$\mathbf{8.74}$	$+1.49$	$\mathbf{1.62}$	$\mathbf{8.63}$	$+1.24$	$\mathbf{1.15}$

Table 3: Ablation of the three modifications for VG-COPRO (§ 3.4 ). Hybrid baseline is based on the results of OPRO on chain pipeline. Each modification produces a significant improvement on both splits (all

p_{1}<10^{-5}

for rows other than the Hybrid baseline). Mod 2 yields the largest mean

\Delta

; the full combination yields the largest paired Cohen’s

d

, indicating lowest dispersion.

Split	n	Chain	Single-shot	$\Delta$	$d$
In-domain	35	$8.04$	$5.45$	$+2.59$	$2.17$
Cross-domain	135	$7.60$	$5.81$	$+1.79$	$1.50$

Table 4: T2I transferability: mean IPM of InsightChain (T2I Present) vs. single-shot T2I baseline on the full test set.

n

refers to the number of sampled datasets. All

p_{1}<10^{-14}

by one-sided paired

t

-test.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Dimension	$r$	$\rho$	MAE	h $-$ IPM
Exploration Breadth	$0.78$	$0.76$	$2.53$	$+2.18$
Focus Relevance	$0.75$	$0.75$	$1.54$	$+1.17$
Hypothesis Rigor	$0.53$	$0.51$	$2.63$	$+2.24$
Insight Communication	$0.65$	$0.67$	$2.53$	$+2.22$
Visual Quality	$0.68$	$\mathbf{0.77}$	$1.79$	$\mathbf{-0.04}$
Composite	$\mathbf{0.87}$	$\mathbf{0.88}$	$1.88$	$+1.86$

Appendix

Table 5: Human–IPM agreement on

n=97

blind-annotated pilot chains.

r

\rho

= Pearson / Spearman correlations between human and IPM scores on the 1–10 scale (all

p<10^{-7}

); MAE = mean absolute error; h

-

IPM = human-mean minus IPM-mean (the scale-offset bias). Composite row is human overall-quality vs. the IPM 5-dim weighted composite. The vision-judged Visual Quality is essentially calibrated to the human (

\Delta=-0.04

); the four text dimensions show a uniform

+1.2

-to-

+2.2

strictness offset that does not affect rank-based agreement.

(a) Per-dimension self-consistency ( $3$ runs)
Dimension	n	$\alpha$	$\bar{r}$
Exploration Breadth	100	$0.959$	$0.959$
Focus Relevance	100	$0.894$	$0.894$
Hypothesis Rigor	100	$0.918$	$0.918$
Insight Communication	100	$0.934$	$0.935$
Visual Quality	0 99	$0.926$	$0.928$

Appendix

Table 6: IPM judge reliability. (a) Per-dimension self-consistency on the 100-chain pilot: all five dimensions are independently rescored three times at temperature

0.5

;

n

= number of evaluated chains with complete three-run scores,

\alpha

= Krippendorff’s interval alpha across three judge runs, and

\bar{r}

= mean pairwise Pearson

r

. (b) Cross-family validation against the in-family Gemini 3 Flash judge: “Gem” is the in-family mean on the matched subsample, “XJ” is the cross-family judge’s mean, and

\Delta=\text{XJ}-\text{Gem}

reports the mean score difference. Both cross-family judges return means at or above Gemini, providing no evidence of in-family score inflation on this matched subsample.

Figure 4: Inter-dimension Pearson correlations on 508 complete chains from the original three-condition evaluation: 170 datasets evaluated with single-shot, uncompiled-chain, and LIDA baselines, excluding two failed LIDA evaluations. Mean off-diagonal $\bar{r}=0.19$ ; max $r=0.50$ (Foc–Hyp). Cronbach’s $\alpha$ on the full 5-tuple is $0.39$ ; the leave-one-out values are $-$ Expl $0.37$ , $-$ Foc $0.29$ , $-$ Hyp $0.22$ , $-$ IC $0.37$ , $-$ VQ $\mathbf{0.44}$ — Visual Quality is the only dimension whose removal raises $\alpha$ , consistent with its capturing variance not covered by the four text-judged dimensions. Expl = Exploration Breadth, Foc = Focus Relevance, Hyp = Hypothesis Rigor, IC = Insight Communication, VQ = Visual Quality.

Source	Pairs	Construction
Seed pair	$20$	Same dataset and prompt; different seed
Regeneration	$40$	Same dataset and quality regime
Metric-close	$20$	Same regime, different datasets, $\|\Delta\mathrm{IPM}\|<0.5$

Appendix

Table 7: Composition of the close-pair evaluation set.

Metric	Multi- stage	Vision- judged	Faithful- ness gate	Insight- aware	Spec- free
nvBench-style spec match ( Luo et al., 2021 )	✗	✗	✗	✗	✗
Chart-to-Text BLEU ( Kantharaj et al., 2022 )	✗	✗	✗	✗	✓
CLIP ( Radford et al., 2021 )	✗	✓	✗	✗	✓
LIDA partial VLM eval ( Dibia, 2023 )	✗	partial	✗	partial	✓
InsightBench rubric ( Sahu et al., 2025 )	text only	✗	✗	✓	✓
IPM (ours)	✓	✓	✓	✓	✓

Appendix

Table 8: Visualization quality metrics by design property. Multi-stage : does the metric score intermediate chain stages rather than only the final figure? Vision-judged : does it consume the rendered image? Faithfulness gate : is there an explicit cap for misleading figures? Insight-aware : does it reward task-specific finding extraction? Spec-free : does it avoid requiring a gold chart specification?

Method	Search algorithm	What is searched
MIPROv2	Bayesian / TPE	instr. + demos
COPRO	Coordinate ascent	instr. only
OPRO	LLM-as-optimizer	instr. only
EvoPrompt	Evolutionary	instr. only
Plan-and-Solve	none (template)	—

Appendix

Table 9: The five APO paradigms compiled and evaluated under identical splits, metric, and budget. Plan-and-Solve has no learnable parameters and serves as a structured-template control.

Figure 5: Checkpoint-reconstructed optimization dynamics over three compile iterations. Each panel plots the validation-pool mean of the incumbent selected under each method’s own objective. Curves are comparable over iterations within a method, but absolute heights should not be compared across metric variants: OPRO, EvoPrompt, and Mod 1 use four text dimensions, whereas Mod 3 uses the full five-dimensional IPM. VG-COPRO and the isolated LODO variant are omitted because their worst-fold selection objective is not directly comparable to the plotted mean-score selection trajectories; methods without reconstructable validation-mean histories are also omitted.

Figure 6: Complete healthcare analysis chain. The example shows the pipeline revising an exploratory impression after formal testing rather than forcing a positive finding.

Figure 7: Complete transportation analysis chain. This held-out-domain example progresses from broad enforcement patterns to a quantified agency-level difference with uncertainty and effect size.

Figure 8: Complete ecommerce analysis chain. This held-out-domain example preserves a nuanced negative decision: the directional effect is significant, but its magnitude does not meet the $200\%$ threshold specified before the Test stage and is dominated by outliers.

Figure 9: Annotator handbook (1/8): annotation unit, required outputs, and scoring anchors for Exploration Breadth .

Figure 10: Annotator handbook (2/8): scoring anchors for Focus Relevance and Hypothesis Rigor .

Figure 11: Annotator handbook (3/8): scoring anchors for text-only Insight Communication .

Figure 12: Annotator handbook (4/8): scoring anchors and calibration examples for rendered Visual Quality , including the faithfulness and display gates.

Figure 13: Annotator handbook (5/8): holistic quality and the three independent final-insight judgments.

Figure 14: Annotator handbook (6/8): row-level annotation workflow and time budget.

Figure 15: Annotator handbook (7/8): calibration and blinding rules.

Figure 16: Annotator handbook (8/8): derived agreement measures and handling of malformed, missing, or ambiguous cases.

Benchmark	# Datasets	Domains	Multi- step	Cross- domain	Insight Metric	Vision Judge
nvBench ( Luo et al., 2021 )	750	Narrow	✗	✗	✗ a	✗
Prompt4Vis ( Li et al., 2025a )	subset	Narrow	✗	✗	✗	✗
Chart-to-Text ( Kantharaj et al., 2022 )	44k	–	✗	✗	✗ b	✗
LIDA exemplars ( Dibia, 2023 )	ad-hoc	Few	✗	✗	✗	Partial
InsightBench ( Sahu et al., 2025 )	100	Mixed	Text-only	✗	Text	✗
InsightEval ( Zhu et al., 2026 )	–	Mixed	Text-only	✗	Text	✗

Appendix

Table 10: InsightChain vs. prior LLM-visualization and insight-quality benchmarks. a Relies on chart-spec matching. b Relies on text-based BLEU scores. ∗ Includes 5 held-out domains. To our knowledge, InsightChain is the first benchmark to combine a held-out cross-domain partition with a multi-step chain evaluation and a vision-aware judge.

Source	#	%
TidyTuesday (community-curated)	55	19.3
OpenML (keyword search per domain)	53	18.6
Other public CSV (Kaggle / project pages)	130	45.6
World Bank, FRED & other macro feeds	11	3.9
UCI ML Repository	8	2.8
Yfinance / equity feeds	4	1.4

Appendix

Table 11: Source composition of the InsightChain experiment datasets. Synthetic templates (

n=23

) are confined to demographics and energy where real public coverage was thinnest after quality filtering.

Figure 17: Distribution of datasets across the 10 target domains.

Split	n	Source
Train	95	5 train dom. $\times$ 19
Val	20	5 train dom. $\times$ 4
In-domain test	35	5 train dom. $\times$ 7
Cross-domain test	135	5 hold-out dom. (all)

Appendix

Table 12: Split sizes.

n

refers to the number of sampled datasets. The Train split is used for compilation only, and Val for APO candidate selection. The cross-domain test set is the central testbed for the transfer claim of § 5.3 .

Figure 18: Per-domain mean 5-dim IPM under the Gemini 3 Flash task model. InsightChain wins every in-domain training domain and every cross-domain hold-out.

Explore similar work

Apr 22, 2026cs.CL

Beyond Pixels: Introspective and Interactive Grounding for Visualization Agents

Vision-Language Models (VLMs) frequently misread values, hallucinate details, and confuse overlapping elements in charts. Current approaches rely solely on pixel interpretation, creating a Pixel-Only Bottleneck: agents treat interactive charts as static images, losing access to the structured specification that encodes exact values. We introduce Introspective and Interactive Visual Grounding (IVG), a framework that combines (1) spec-grounded introspection, which queries the underlying specification for deterministic evidence, with (2) view-grounded interaction, which manipulates the view to resolve visual ambiguity. To enable evaluation without VLM bias, we present iPlotBench, a benchmark of 500 interactive Plotly figures with 6,706 binary questions and ground-truth specifications. Experiments show that introspection improves data reconstruction fidelity, while the combination with interaction achieves the highest QA accuracy (0.81), with +6.7 % gains on overlapping geometries. We further demonstrate IVG in deployed agents that explore data autonomously and collaborate with human users in real time.

Yiyang Lu, Woong Shin, Ahmad Maroof Karimi +3

William & Mary · Oak Ridge National Laboratory

Scientific Visualization Data Visualization Chart QA Vision-Language Grounding Tool-Using Vision-Language Agents Visually Grounded Reasoning

Ask PDF

Sep 1, 2025cs.CL

Vis-CoT: A Human-in-the-Loop Framework for Interactive Visualization and Intervention in LLM Chain-of-Thought Reasoning

Large language models (LLMs) show strong reasoning via chain-of-thought (CoT) prompting, but the process is opaque, which makes verification, debugging, and control difficult in high-stakes settings. We present Vis-CoT, a human-in-the-loop framework that converts linear CoT text into an interactive reasoning graph. Users can visualize the logical flow, identify flawed steps, and intervene by pruning incorrect paths and grafting new, user-defined premises. This shifts interaction from passive observation to active collaboration, steering models toward more accurate and trustworthy conclusions. Across GSM8K and StrategyQA, Vis-CoT improves final-answer accuracy by up to 24 percentage points over non-interactive baselines. A user study also shows large gains in perceived usability and trust. Vis-CoT points to a practical path for more reliable, understandable, and collaborative reasoning by combining LLMs with targeted human oversight.

Kaviraj Pather, Elena Hadjigeorgiou, Arben Krasniqi +4

Department of Computer Science & Engineering University of Mauritius R´eduit 80837, Mauritius · Department of Electrical & Computer Engineering Cyprus University of Technology 3036 Limassol, Cyprus · Faculty of Electrical & Computer Engineering University of Prishtina ”Hasan Prishtina” 10000 Prishtina, Kosovo +4

LLM Interpretability Human-in-the-Loop AI Visual CoT LLM Reasoning

Ask PDF

Nov 21, 2025cs.CV

VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning

Chain-of-Thought (CoT) prompting has proven remarkably effective for eliciting complex reasoning in large language models (LLMs). Yet, its potential in multimodal large language models (MLLMs) remains largely untapped, hindered by the absence of large-scale datasets that capture the rich, spatially grounded reasoning intrinsic to visual understanding. Existing visual-CoT resources are typically small, domain-specific, or lack the human-like stepwise structure necessary for compositional visual reasoning. In this paper, we introduce VisReason, a large-scale dataset designed to advance visual Chain-of-Thought reasoning. VisReason comprises 489K annotated examples spanning four diverse domains, each featuring multi-round, human-like rationales that guide MLLMs through interpretable visual reasoning steps. Building upon this, we curate VisReason-Pro, a 165K subset produced with a stronger expert-level GPT annotator, enriched with detailed reasoning traces and 3D spatial grounding via depth-informed annotations. Fine-tuning the state-of-the-art Qwen2.5-VL model on VisReason and VisReason-Pro yields substantial improvements in step-by-step visual reasoning accuracy, interpretability, and cross-benchmark generalization. These results demonstrate that VisReason equips MLLMs with more systematic and generalizable reasoning capabilities. We envision VisReason as a cornerstone for cultivating human-like visual reasoning, paving the way toward the next generation of multimodal intelligence.

Lingxiao Li, Yifan Wang, Xinyan Gao +3

Boston University, Boston, MA, USA · Stony Brook University, Stony Brook, NY, USA · MMLab, The Chinese University of Hong Kong, Hong Kong SAR, China

Visual Spatial Reasoning Visual Reasoning Multimodal Large Language Models Multimodal CoT Reasoning Visually Grounded Reasoning Visual CoT

Ask PDF