LYNX: Analytical Program Synthesis via Grounding and Closed-Loop Refinement
Authors: Ruichen Xia, Xiao Han, Dingyu Yang, Luwei Yang, Sai Wu, Dongxiang Zhang, Binhui Yao, Gang Chen
Organizations: The State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China · Institute of Computing Innovation, Zhejiang University, Hangzhou, China · Zhejiang Geely Holding Group Co., Ltd., Hangzhou, China
LLM-powered data analytics aims to translate natural language instructions into executable analytical programs over underlying data. However, existing approaches, whether treating LLMs as intent-to-code translators or as standalone analytical engines, often fail to reliably align domain semantics with complex schemas and heterogeneous metadata, resulting in brittle execution and semantically inconsistent analytical outputs in real-world workloads. We present LYNX, a multi-agent framework that formulates LLM-driven analytics as a schema- and knowledge-constrained analysis problem rather than a single-pass generation task. LYNX organizes analytical synthesis into a closed-loop workflow of planning, generation, verification, and refinement through three role-specialized agents that collaboratively enforce schema grounding, semantic consistency, and execution correctness. To address domain ambiguity and schema misalignment, LYNX integrates Semantic-Structural Distillation and Stage-Aware Allocation (SSD-SAA). A delta-guided iterative revision continuously diagnoses and repairs semantic and execution-level failures, substantially improving synthesis robustness. We evaluate LYNX on two NL-to-SQL benchmarks, BIRD-SQL Mini-Dev and the more challenging Spider 2.0-Lite. Experimental results demonstrate strong performance and efficiency, achieving 10.66% to 59.54% execution accuracy across lightweight and commercial models, while reducing token consumption by 70.53% compared with the prior most efficient method. Our framework displays high generalization over different backbones, from open-source, lightweight models to the latest commercial models. Additional case studies on quantitative finance analytics and complex multi-relational SQL workloads further demonstrate the framework's robustness, scalability, and practical applicability in real-world analytical scenarios.
Figures & tables
Fig. 1 : The Framework of the Multi-agent System
Fig. 2 : Decoupled Domain Knowledge Example
Fig. 3 : Risks Contents Fallback Mechanism
Fig. 4 : Rubber Duck Debugging Strategy
Model
Method
Easy
Medium
Hard
Total EX
Avg. Tokens
GPT-4o
DIN-SQL
5.79
0.43
0.00
1.46
—
CHESS
9.92
3.00
1.24
3.84
—
DAIL-SQL
13.20
5.58
1.24
5.68
—
LYNX (ours)
44.36
37.25
22.14
34.05
13.8K
DeepSeek-R1
RSL-SQL
—
—
—
30.53
50.0K
LinkAlign
—
—
—
33.09
—
TABLE I : Comparison of performance results on Spider 2.0-Lite
Backbone Model
EX %
Avg. Times
Avg. Tokens
Qwen3-14B
10.66
6.50
48.4K
GPT-4o
34.05
5.16
13.8K
DeepSeek-V3
35.45
4.25
13.4K
DeepSeek-R1
51.13
4.80
11.2K
Claude-Sonnet-4.6
57.58
4.29
7.1K
Gemini-3.1-Pro
59.54
3.70
6.4K
TABLE II: LYNX performance on different backbones
Metric
DeepSeek-V3
Qwen3-14B
Executability
88.42%
49.36%
Avg. Attempts
4.25
6.50
Few-Shot ( ≤ 3 attempts)
68%
43%
Executable Rate
TABLE III : Comparison of Executability Performances
Fig. 5 : Attempt Distribution of Models
Methods (Models)
EX (%)
VES
Soft-F1
TA + codellama-34B [ 37 ]
27.57
-
-
GPT-3.5-turbo [ 49 ]
38.00
37.33
41.84
llama3-70B-instruct
40.80
39.02
44.38
Claude-2 [ 50 ]
42.70
-
-
TA + llama3-70B-instruct [ 48 ]
42.80
41.37
46.66
TA + DeepSeek [ 48 ]
43.74
-
-
TABLE IV : Performance Comparison on BIRD Mini-Dev
Model
Metric
Simple
Moderate
Hard
Total
Qwen3-14B
EX
57.43
39.20
33.33
43.40
VES
54.11
36.10
31.44
40.48
Soft-F1
62.95
45.22
42.55
49.92
DeepSeek-V3
EX
63.51
47.20
34.31
49.40
VES
58.69
43.02
32.14
45.44
Soft-F1
68.22
54.91
42.50
56.31
TABLE V : Detailed Results across Difficulty Levels
Fig. 6 : Scores on Different SQL Dialects
agent
function
EX % ( Δ )
-
Full model (Ours)
35.45
Generator
w/o SSD-SAA
≈0 ( ↓≈ 100% )
Critic
w/o Delta-guided memory
27.36 ( ↓ 22.8% )
Reviser
w/o Iterative revision
22.43 ( ↓ 36.7% )
w/o Logical verification
30.23 ( ↓ 14.7% )
TABLE VI: Ablation Results on Spider 2.0-lite
Agent
Tokens
Runtime
Prompt
Completion
Generator
3520±627
496±102
14.94±4.08s
Critic
3462±1128
934±228
6.82±1.40s
Reviser
2438±413
473±113
10.16±3.92s
TABLE VII: Token Consumption and Runtime Analysis
Fig. 7 : Comparison of History Length after Attempts
Fig. 8 : Details for Stage-aware Reasoning Efficacy
TABLE VIII : Revision Loop Effects across Models
Metric
Method
Difficulty
Simple
Moderate
Hard
Total
EX
w/o Semantics
62.16
44.00
36.27
47.80
w/ Semantics
63.51
47.20
34.31
49.40
Δ
+1.35
+3.20
-1.96
+1.60
VES
w/o Semantics
56.74
40.17
34.23
43.86
w/ Semantics
58.69
43.02
32.14
45.44
TABLE IX : Semantic Impact to Subsets on BIRD
Fig. 9 : Definitions for External Knowledge
Fig. 10 : The Pipeline of the Generated Python Script
Figure 20
Fig. 13 : Case Multi-step Process
Stage
general-purpose LLM (GPT-5.2)
Our Framework (w/ revision)
Schema
Hallucinate table name
use correct FQN
Challenge
Standard casting fails on dirty data
Detected error and applied SAFE_CAST
Logic
Implicit type mismatch in comparisons
Explicitly compare INT64 vs STRING
Outcome
Execution Error "404 Not Found" ✗
Successfully executed Correct Answer ✓
TABLE X : Comparative Analysis of Case "bq282"
Fig. 14 : An Example of Challenges
Fig. 15 : Stage-aware Reasoning
Date
Open
High
Low
Close
…
Industry
2009-01-05
55.09
56.84
54.76
56.73
…
Real Estate
2009-01-06
56.73
59.03
56.18
59.03
…
Real Estate
⋮
⋮
⋮
⋮
⋮
⋱
⋮
2025-08-15
285.33
294.87
283.35
291.29
…
Real Estate
TABLE XI : Sample of Stock Trading Data (000006.SZ)
Large language model (LLM)-based program synthesis turns natural-language goals and examples into executable programs, often using tests or formal constraints.
LLMs deployed for natural-language querying of analytical databases suffer from two intertwined failures - incorrect answers and confident hallucinations - both rooted in the same cause: the model is forced to infer business semantics that the schema does not encode. We test whether supplying those semantics as context closes the gap. We benchmark three frontier LLMs (Claude Opus 4.7, Claude Sonnet 4.6, GPT-5.4) on 100 natural-language questions over the Cleaned Contoso Retail Dataset in ClickHouse, using a paired single-shot protocol. Each model is evaluated twice: once given only the warehouse schema, and once given the schema plus a 4 KB hand-authored markdown document describing the dataset's measures, conventions, and disambiguation rules. Adding the document improves accuracy by +17 to +23 percentage points across all three models. With it, the three models are statistically indistinguishable (67.7-68.7%); without it, they are also indistinguishable (45.5-50.5%). Every cross-cluster comparison is significant at p < 0.01. The presence of the semantic-layer document accounts for essentially all of the significant variance; model choice within tier does not. We interpret this as a structural result: explicit business semantics suppress the dominant class of text-to-SQL errors not by making the model more capable, but by changing what the model is being asked to do.
Large language models (LLMs) are increasingly used to generate queries, invoke tools, and construct analytical workflows. Although recent advances have substantially improved workflow generation and execution, the semantic information required to operationalize analytical concepts often lies beyond what is explicitly represented in database schemas and data values. We present a cross-domain formative study of operationalization failures in agent-generated analytical workflows. Across 236 analytical intents spanning finance, human resources, and public safety domains, we identify 153 recurring failures despite successful workflow generation and execution. Our analysis reveals five recurring classes of failures: comparative grounding, process reasoning, quantitative reasoning, role confusion, and policy grounding. These findings suggest a semantic gap between user-level analytical concepts and the information available to workflow-generation systems. More broadly, they raise questions about the admissibility of analytical operations and suggest that future agentic data systems may require richer semantic representations to bridge the gap between analytical intent and executable computation.
Large language model (LLM) agents are increasingly tasked with complex real-world analysis (e.g., in financial forecasting, scientific discovery), yet their reasoning suffers from stochastic instability and lacks a verifiable, compositional structure. To address this, we introduce Analytica, a novel agent architecture built on the principle of Soft Propositional Reasoning (SPR). SPR reframes complex analysis as a structured process of estimating the soft truth values of different outcome propositions, allowing us to formally model and minimize the estimation error in terms of its bias and variance. Analytica operationalizes this through a parallel, divide-and-conquer framework that systematically reduces both sources of error. To reduce bias, problems are first decomposed into a tree of subpropositions, and tool-equipped LLM grounder agents are employed, including a novel Jupyter Notebook agent for data-driven analysis, that help to validate and score facts. To reduce variance, Analytica recursively synthesizes these grounded leaves using robust linear models that average out stochastic noise with superior efficiency, scalability, and enable interactive "what-if" scenario analysis. Our theoretical and empirical results on economic, financial, and political forecasting tasks show that Analytica improves 15.84% accuracy on average over diverse base models, achieving 71.06% accuracy with the lowest variance of 6.02% when working with a Deep Research grounder. Our Jupyter Notebook grounder shows strong cost-effectiveness that achieves a close 70.11% accuracy with 90.35% less cost and 52.85% less time. Analytica also exhibits highly noise-resilient and stable performance growth as the analysis depth increases, with a near-linear time complexity, as well as good adaptivity to open-weight LLMs and scientific domains.