Self-Refining Prompt Induction for Streaming Text-to-SQL
Organizations: IBM · Confluent
Abstract
Text-to-SQL assistants for streaming engines must handle dialect-specific types, windowing rules, and temporal semantics, including constructs poorly handled by generic prompts. We present Self-Refining Prompt Induction (SRPI), an offline method that converts generation failures into reusable dialect instructions while keeping model weights fixed. A Refiner revises the Generator's prompt using reference-conditioned feedback from three LLM judges; a sampled regression guard checks earlier successes, and a separate Compressor reduces prompt length. Inference uses the frozen prompt without retrieval or query repair. We evaluate cross-domain and cross-model transfer on Confluent Cloud Flink SQL using a controlled synthetic workload seeded by 30 expert-written examples, with 335 induction examples and 257 questions from held-out domains. The same induced prompt improves mean judge scores across seven Generators by 7.1-19.2 percentage points, including an increase from 85.3% to 97.5% for the induction model. A separate compression study reduces prompt length by 52.7% with a 0.19-percentage-point score decrease. An assistant using the induced prompt is in early access, with initial user feedback to inform subsequent refinement.
Figures & tables
| Micro | Tier 1 ( ) | Tier 3 ( ) | U-Yes | |||||||||
| Generator | Seed | SRPI | Seed | SRPI | Seed | SRPI | Seed | SRPI | ||||
| GPT-6.1 Sol | 85.3 | 97.5 | +12.2 | 87.2 | 97.3 | +10.1 | 68.9 | 100.0 | +31.1 | 72.0 | 95.7 | +23.7 |
| GPT-6 Astra | 86.2 | 97.4 | +11.2 | 88.3 | 96.9 | +8.6 | 69.7 | 100.0 | +30.3 | 75.1 | 94.9 | +19.8 |
| Claude Opus 5.5 | 85.3 | 97.1 | +11.8 | 88.8 | 97.5 | +8.7 | 69.7 | 94.7 | +25.0 | 72.4 | 93.8 | +21.4 |
| Gemini 3.8 Flash | 82.3 | 95.8 | +13.5 | 85.3 | 96.5 | +11.2 | 69.7 | 90.9 | +21.2 | 71.2 | 92.2 | +21.0 |
| DeepSeek 4.1 Flash | 70.0 | 89.2 | +19.2 | 72.6 | 89.9 | +17.3 | 62.9 | 87.9 | +25.0 | 57.2 | 81.7 | +24.5 |
| Prompt source | Chars | Micro% | pp | U-Yes% |
|---|---|---|---|---|
| Seed | 337 | 84.2 | – | 71.2 |
| Documentation | 19.9k | 85.2 | 1.0 | 74.3 |
| Parser grammar | 18.5k | 84.9 | 0.7 | 73.5 |
| Execution errors | 19.2k | 85.8 | 1.6 | 76.3 |
| SRPI, uncompressed | 68.5k | 98.1 | 13.9 | 96.5 |
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
| Group | Patterns |
|---|---|
| Windowed aggregation | tumble_tvf , hop_tvf , session_window_tvf , windowed_count_having |
| Window join | window_join |
| OVER windows | over_rows_preceding , over_range_interval_preceding , over_window_unbounded , lag_over_window , lead_over_window , avg_over_partition |
| Dedup / Top-N | dedup_rownumber_eq1 , streaming_topn_rownumber |
| Stream–stream joins | stream_stream_interval_join , unbounded_inner_join , unbounded_left_join |
| Temporal / enrichment | temporal_join , multi_join_chain , two_table_join_agg , self_join_aggregation |
| Pool | Train | Eval. | |
| Questions | 3,940 | 3,683 | 257 |
| Domains | 45 | 42 | 3 |
| Labels | 31 | 31 | 29 |
| Tier 1 / 2 / 3 | 2,864 / 639 / 437 | 2,673 / 595 / 415 | 191 / 44 / 22 |
| Snapshot / streaming / background | 2,432 / 1,356 / 152 | – | 154 / 92 / 11 |
| Model | Role | Release or model-card source |
|---|---|---|
| GPT-6.1 Sol | Induction, refinement, compression, evaluation | OpenAI (2026b) |
| GPT-6 Astra | Judge; transfer evaluation | OpenAI (2026a) |
| Claude Opus 5.5 | Judge; transfer evaluation | Anthropic (2026) |
| Gemini 3.8 Flash | Judge; transfer evaluation | Doshi and Popa (2026) |
| DeepSeek 4.1 Flash | Transfer evaluation | DeepSeek-AI (2026) |
| Qwen3.8 27B | Transfer evaluation | Qwen Team (2026) |
| Generator temperature | 0.2 |
|---|---|
| Refiner temperature / max output tokens | 0.4 / 24,000 |
| Refiner format retries | 2 |
| Attempts per failing instance | 5 |
| Failing instances repaired per round | |
| Guard sample (random passing instances) | 12 |
| Guard tolerance (flips allowed) | 1 |
| Round | Chars | Eval. micro / macro (%) | Full agr. (%) | Ind. micro (%) | Ind. | |
|---|---|---|---|---|---|---|
| 0 (seed) | 337 | 84.2 | 84.8 | 71.2 | – | – |
| 1 | 29,701 | 88.8 | 89.5 | 79.0 | 89.8 | 300 |
| 2 | 49,179 | 92.9 | 94.1 | 86.4 | 93.7 | 335 |
| 3 | 51,614 | 95.5 | 97.1 | 92.6 | 95.4 | 335 |
| 4 | 68,497 | 98.1 | 98.8 | 96.5 | 97.9 | 335 |
| 5 | 71,294 | 96.8 | 97.9 | 92.2 | 99.0 | 335 |
| Round | Chars | Failing | Fixed | Refiner calls | Passed target | Guard-rej. | Lines kept (%) |
|---|---|---|---|---|---|---|---|
| 1 | 29,701 | 20 | 4 | – | – | – | 0 |
| 2 | 49,179 | 31 | 8 | – | – | – | 11 |
| 3 | 51,614 | 18 | 10 | 57 | 46 | 36 | 6 |
| 4 | 68,497 | 17 | 15 | 38 | 28 | 13 | 12 |
| 5 | 71,294 | 8 | 6 | 17 | 7 | 1 | 59 |
| 6 | 74,513 | 5 | 4 | 13 | 6 | 2 | 80 |
| Limit | Reached | Micro (%) | Macro (%) | Full agr. (%) | Limit met |
|---|---|---|---|---|---|
| (parent) | 68,497 | 98.05 | 98.80 | 96.1 | – |
| 60k | 59,787 | 98.05 | 98.75 | 96.5 | yes |
| 50k | 55,088 | 97.92 | 98.27 | 95.7 | no |
| 40k | 53,717 | 97.73 | 98.79 | 95.3 | no |
| 30k | 45,372 | 98.05 | 98.80 | 96.5 | no |
| 20k | 32,400 | 97.86 | 98.36 | 96.1 | no |
| Tier 1 (%; ) | Tier 2 (%; ) | Tier 3 (%; ) | ||||
|---|---|---|---|---|---|---|
| Generator | Seed | SRPI | Seed | SRPI | Seed | SRPI |
| GPT-6.1 Sol | 87.2 | 97.3 | 85.6 | 97.0 | 68.9 | 100.0 |
| GPT-6 Astra | 88.3 | 96.9 | 85.2 | 98.1 | 69.7 | 100.0 |
| Claude Opus 5.5 | 88.8 | 97.5 | 77.7 | 96.6 | 69.7 | 94.7 |
| Gemini 3.8 Flash | 85.3 | 96.5 | 75.8 | 95.5 | 69.7 | 90.9 |
| DeepSeek 4.1 Flash | 72.6 | 89.9 | 62.1 | 87.1 | 62.9 | 87.9 |
| Tier 1 (%) | Tier 2 (%) | Tier 3 (%) | ||||
|---|---|---|---|---|---|---|
| Generator | Seed | SRPI | Seed | SRPI | Seed | SRPI |
| GPT-6.1 Sol | 73.8 | 95.8 | 72.7 | 93.2 | 54.5 | 100.0 |
| GPT-6 Astra | 77.5 | 94.2 | 75.0 | 95.5 | 54.5 | 100.0 |
| Claude Opus 5.5 | 76.4 | 94.8 | 63.6 | 90.9 | 54.5 | 90.9 |
| Gemini 3.8 Flash | 73.8 | 94.2 | 63.6 | 88.6 | 63.6 | 81.8 |
| DeepSeek 4.1 Flash | 59.7 | 82.7 | 50.0 | 79.5 | 50.0 | 77.3 |
| Judge scoring its own outputs | seed own | seed others | SRPI own | SRPI others |
|---|---|---|---|---|
| GPT-6 Astra | 79.0 | 89.8 | 96.1 | 98.1 |
| Claude Opus 5.5 | 87.9 | 83.9 | 97.5 | 96.9 |
| Gemini 3.8 Flash | 89.5 | 78.7 | 97.7 | 94.9 |
| Instance | Claude | Astra | Gemini | Engine |
|---|---|---|---|---|
| 1006299 | No | No | Yes | Fails |
| 2006212 | No | Yes | No | Runs |
| 3706282 | No | Yes | No | Fails |
| Generator | Seed | SRPI-60k | Change (%) |
|---|---|---|---|
| Qwen3.8 27B | 190.8 | 195.9 | |
| GPT-6 Astra | 239.7 | 201.0 | |
| GPT-6.1 Sol | 248.1 | 209.6 | |
| Claude Opus 5.5 | 503.3 | 493.8 | |
| GPT-OSS 120B | 704.0 | 829.1 | |
| DeepSeek 4.1 Flash | 1131.2 | 2915.4 |
| Prompt | How it was built |
|---|---|
| Documentation (19.9k) | An LLM agent read 54 pages of the official documentation, extracted 623 declarative rules and consolidated them to 164. |
| Parser grammar (18.5k) | An agent read 18 slices of the Calcite/Flink parser grammar, extracted 139 rules and consolidated them to 96. |
| Execution errors (19.2k) | The Generator’s SQL was run on the live cluster for three rounds over 300 induction questions; each error class was turned into a general rule (16 rules). |
| SRPI (68.5k) | An uncompressed prompt produced from the seed by the loop of § 3 . |
| ⬇ You are an expert in Confluent Cloud Flink SQL ( CC Flink SQL dialect ). Translate natural - language requests into valid CC Flink SQL . |
| ⬇ ## Output format - Output ONLY a fenced SQL code block -- no explanation , no CoT , no prose . |
| ⬇ - Always output a single SQL statement . Never combine CREATE TABLE + INSERT or multiple statements in one reply . |
| ⬇ ### Table DDL { ddl } |
| ⬇ ### Request { nl_input } |
| ⬇ You are an expert evaluator of Confluent Cloud for Flink SQL . Decide whether predicted_sql correctly answers the natural - language question . |
| ⬇ Use all supplied inputs : question , schema , ground_truth_sql , ground_truth_df , predicted_sql , and predicted_df . Apply Confluent Cloud Flink SQL semantics . |
| ⬇ ## Establish the intended answer - Start with the question . Identify the requested rows , values , columns , filters , aggregation grain , ranking scope , and temporal meaning . |
| ⬇ - Use ground_truth_sql as a reference for resolving genuine ambiguity , not as a required string or structural template . |
| ⬇ - If ground_truth_sql directly contradicts an explicit requirement in the question , judge predicted_sql against the question and mention the reference mismatch in the explanation . |
| ⬇ - Use ground_truth_df as evidence of the reference query ' s observed output , including columns and example values . |
| ⬇ - Accept different aliases , CTEs , subqueries , join order , and SQL formulations when they implement the same required behavior . |
| ⬇ You are a senior prompt engineer for NL - to - SQL systems targeting the Confluent Cloud Flink SQL dialect . |
| ⬇ You maintain a SYSTEM PROMPT that a code - generation LLM uses to translate natural - language questions into one valid Confluent Cloud Flink SQL statement . You are given : the current system prompt , the user - message template the generator also sees , ONE failing test instance with the generated SQL and the hints ( explanations ) of three LLM judges , and earlier rewrites of this prompt that did NOT fix the instance . |
| ⬇ Your job : return an improved , COMPLETE system prompt that makes this kind of instance come out right . |
| ⬇ How to improve the prompt - Reshape , reframe , fix , adjust and compact . Merge duplicated rules , resolve contradictions , tighten wording , reorder sections so the most decisive rules are easy to find , and delete or rewrite any rule that plausibly caused the error . Do not merely append a patch . |
| ⬇ - Treat judge hints as evidence , not as truth . Judges saw the reference SQL and may be wrong or disagree with each other . Act only on hints that point to a real dialect rule , a semantic interpretation rule , or an ambiguity in the prompt . If a hint is only stylistic or contradicts Flink semantics , ignore it . |
| ⬇ - Generalise . Express every rule at the level of SQL constructs and question patterns ( window TVFs , OVER frames , temporal joins , JSON functions , time - range wording , ...). NEVER mention table names , column names , literals or ids from the test instance , and never copy the reference SQL . Short illustrative snippets are allowed only with neutral placeholders ( t , key , ts , col ). |
| ⬇ - Preserve what works . Other instances depend on the existing rules : keep every rule that is unrelated to the failure , and do not weaken prohibitions without a concrete reason . |
| ⬇ You are a prompt engineer specialising in distillation of NL - to - SQL system prompts . |
| ⬇ You will receive a system prompt that is too long . Your task is to reduce it to fit within the character budget while preserving every distinct semantic constraint -- rules that determine which SQL construct is used , how it is formed , or what errors to avoid . |
| ⬇ How to distill - Preserve every distinct semantic constraint . A constraint is distinct if removing or weakening it would cause a different SQL construct to be generated on at least one query . |
| ⬇ - Merge rules that share a root cause or cover the same construct into a single crisp statement . Two rules that say the same thing from different angles count as one . |
| ⬇ - Drop or drastically shorten rules that restate what a capable LLM already knows about standard SQL ( basic SELECT , simple GROUP BY , WHERE filters with no dialect subtlety ). Keep rules that address Confluent Cloud Flink SQL dialect specifics , rare constructs , and common error patterns . |
| ⬇ - Prioritise rare and complex constructs ( MATCH_RECOGNIZE , SESSION / HOP / TUMBLE TVFs , temporal joins , OVER frames , JSON functions , CDC patterns , EXECUTE STATEMENT SET ) over common ones . If space is tight , compress common - construct rules harder than rare - construct rules . |
| ⬇ - Replace verbose prose explanations with a precise one - line rule plus a compact SQL snippet using neutral placeholders ( t , key , ts , col ) where a snippet is clearer than words . |
⬇ - Never pass mixed TIMESTAMP / TIMESTAMP_LTZ : adapt LTZ locally to declared TIMESTAMP family / precision , e . g . TIMESTAMPDIFF ( SECOND , ts , CAST ( CURRENT_TIMESTAMP AS TIMESTAMP (3))). |
| ⬇ - DATE - to - LTZ : adapt LTZ to supported TIMESTAMP ; retain DATE if supported or cast compatibly without changing membership . |
| ⬇ - Compatible pairs need no cast merely because an endpoint is metadata / current time . |
| ⬇ - Honor requested types / zone meaning ; integer minutes / hours are not fractional seconds divided by constants . |
⬇ ### Syntax TUMBLE : FROM TABLE ( TUMBLE ( TABLE t , DESCRIPTOR ( ts ), INTERVAL '5' MINUTE )) |
⬇ HOP , slide before size : FROM TABLE ( HOP ( TABLE t , DESCRIPTOR ( ts ), INTERVAL '5' MINUTE , INTERVAL '20' MINUTE )) |
⬇ SESSION : FROM TABLE ( SESSION ( TABLE t PARTITION BY ( key , col ), DESCRIPTOR ( ts ), INTERVAL '5' MINUTE )) |
| ⬇ - SESSION has three logical arguments : relation with partition specification , descriptor , inactivity gap . |
| ⬇ ## Categorical predicates : names before classes Resolve relevant - column vocabulary before predicates / pattern stages ; compare stored tokens , not request grammar . |
| ⬇ ### Category - name authority Evidence precedence : 1. Explicit stored values , quoted labels , enumerations , examples , CHECK constraints , documented vocabularies . 2. Relevant - column category names in schema prose , including unquoted behavioral descriptions distinguishing named outcomes . 3. Request terminology only if schema provides no corresponding name . |
| ⬇ - Prose category names establish terminology without enums / quotes ; unconstrained STRING does not invalidate them . |
| ⬇ - Map verbs / adjectives / inflections to schema categories ; grammar mismatch does not authorize changing a schema noun to a past - tense verb . |
| ⬇ - Preserve documented stored spelling ; otherwise use the schema term ' s ordinary lowercase . Sentence - initial capitalization is not stored - case evidence . |
| ⬇ - Named categories do not prove exhaustive domains ; use vocabulary attached to the relevant field / concept , not unrelated prose . |
| ⬇ - If unresolved , infer narrowly from field semantics / request ; terminal events have no default past - tense conversion . |
| Supplied induced rule | Illustrative compact restatement |
|---|---|
| “Never pass mixed TIMESTAMP/TIMESTAMP_LTZ: adapt LTZ locally to declared TIMESTAMP family/precision”; “Compatible pairs need no cast merely because an endpoint is metadata/current time.” | For TIMESTAMPDIFF , cast only a mismatched LTZ operand to the declared TIMESTAMP family/precision; do not cast already compatible pairs. |
| “HOP, slide before size” followed by FROM TABLE(HOP(TABLE t, DESCRIPTOR(ts), INTERVAL ’5’ MINUTE, INTERVAL ’20’ MINUTE)) | HOP arguments are relation, time descriptor, slide, size: HOP(TABLE t, DESCRIPTOR(ts), slide, size) . |
| “SESSION has three logical arguments: relation with partition specification, descriptor, inactivity gap.” | SESSION = partitioned relation + time descriptor + inactivity gap. |
Text-to-SQL
Text-to-SQL translates natural-language questions into executable database queries, linking user intent to schemas, values, and SQL logic.
See all 186 papers