VIGIL: Verifier-Informed Gated Improvement Loop for Spreadsheet Question Answering

Kang Li; Lu He; Sandarsita Guntupalli

cs.AIOct 3, 2026

VIGIL: Verifier-Informed Gated Improvement Loop for Spreadsheet Question Answering

Authors: Kang Li, Lu He, Sandarsita Guntupalli

Organizations: Atlassian · Carnegie Mellon University

Abstract

Enterprise agents should improve from delayed feedback without allowing every correction to rewrite system behavior. We study continual harness learning for corpus-level spreadsheet question answering. Building on FiCo (Find-then-Compute), a static retrieval-and-execution backbone, we introduce VIGIL (Verifier-Informed Gated Improvement Loop). Within a question, VIGIL verifies and repairs diversely prompted Structured Query Language (SQL) candidates. Across episodes, delayed labels update only a contract calibrator and query/result-column selector. The base model, prompts, retriever, and recorded candidate pool remain fixed in the continual-learning protocol. With the gold workbook and expected type supplied, the ungated and dual-gated full-replay variants reach 82.4% and 82.0% forward accuracy over 79 documents, from a 75.8% static baseline. The dual gate has the larger retrospective gain (2.7 versus 2.3 points), and its 6.3-point forward gain has a 95% t-interval of 5.6-6.9. In a separate stricter split that excludes 16 documents and 380 questions from fitting and online promotion, the accuracy-only gate raises mean held-out accuracy across ten final harnesses from 80.3% to 86.7%. Calibrator-only adaptation gains 5.9 points, close to the combined 6.4-point gain. Yet three of 26 gate-approved updates reduce held-out accuracy relative to their incumbents, so replay-buffer non-regression does not imply held-out non-regression. MiMoTable and external-task case studies test within-task verification and reuse the same promote-or-retain discipline. Overall, the results support bounded, auditable harness adaptation while revealing where finite replay gates fail to generalize beyond their promotion buffers.

Figures & tables

Component	Within-question repair	Across-episode learning
Workbook retriever and loader	Frozen or oracle-supplied	Frozen or oracle-supplied
Base model and generation prompts	Frozen	Frozen
Candidate SQL	Repairable	Frozen replay pool
Answer-contract calibrator	Frozen	Learnable
Query/result-column selector	Frozen	Learnable
Parser, execution, and hard checks	Non-relaxable	Non-relaxable

Table 1: Mutable scope of the reported adaptation experiments. Learning does not regenerate candidates or update the base model, prompts, retriever, or loader.

	Track	Systems / intervention	Oracle / controlled input	$n$	Valid claim
RQ1	End to end	Prefix-RAG compared with FiCo	None	1,810 / 508	Shared-judge end-to-end difference
RQ2	Controlled ablation	Single-component removal	Gold workbook (compute rows)	1,810	Paired effect in fixed runner
RQ3	Verifier / bridge	Repair ladder and workbook/type $2\times 2$	Varied by arm	1,810	Repair and oracle effects
RQ3	Official MiMoTable	Direct compared with semantic gate	Benchmark-supplied workbook	641	Official-prompt judged accuracy
RQ4a	Full replay	Static, ungated, and dual-gated	Workbook + type, delayed labels	$1{,}810\times 10$	Forward and retrospective gain
RQ4b	Held-out audit	Static and five adaptive variants	Workbook + type, delayed labels	$1{,}430$ per ordering + 380 audit	Held-out transfer and gate behavior

Table 2: Evaluation ledger. Changing runner, execution engine, oracle, or evaluator defines a new track. Gold-workbook treatments are not end-to-end competitors. RQ4a and RQ4b use different partitions. RQ4b learns from 1,430 questions per ordering and evaluates on 380 fixed audit questions.

System	DataBench	MiMoTable retrieval
Prefix-RAG baseline	12.8%	22.2%
FiCo	76.2%	79.7%
$\Delta$	$+63.4$	$+57.5$

Table 3: End-to-end accuracy for corpus-selection tracks.

\Delta

is measured in percentage points.

Adaptation budget	Accepted	Correct	Overall accuracy	$\Delta$
Two initial candidates, no repair	1,516	1,335	73.8%	—
$+$ first repair round	1,574	1,379	76.2%	$+2.4$
$+$ second repair round	1,576	1,381	76.3%	$+2.5$

Table 4: Paired Vigil repair ladder on 1,810 DataBench questions with a gold workbook and expected-type metadata. Abstentions count as incorrect.

Official-workbook system	Mean judged accuracy
Claude 3.5 Sonnet, reported English text-only	79.0%
Direct GPT-5.4	85.66%
Vigil semantic agreement	85.70%

Table 5: Official MiMoTable English text-only results (

n=641

). The benchmark supplies the workbook, and GPT-4o judgments are averaged over three complete repetitions. The 0.05-point difference quoted in the text is computed from unrounded means.

Treatment	Forward accuracy	$\Delta$	Coverage	Selective accuracy	Retrospective gain	Harmful deployed
Static harness	75.8%	—	87.1%	87.0%	—	—
Ungated update	82.4%	+6.7	94.2%	87.5%	$+2.3$	$8/40$
Dual replay gate	82.0%	$+6.3$	93.8%	87.5%	+2.7	0/16

Table 6: Full replay over ten five-episode orderings. Forward metrics use pre-update performance on episodes 2–5. Harmful/deployed counts include only proposals with a successor episode. Bold marks column bests, and deltas use unrounded values.

Figure 2: Forward transfer during full replay. The vertical line marks the point after which feedback from episode 1 is available. Pre-update accuracy is averaged over ten document orderings, and bars show 95% $t$ -intervals. Each episode contains a different document set, so variation in the static line reflects episode difficulty rather than model change.

Final harness	Accuracy	$\Delta$	Coverage	Selective accuracy
Static	80.3%	—	90.5%	88.7%
Calibrator only	86.2%	$+5.9$	96.7%	89.1%
Selector only	80.5%	$+0.2$	90.5%	88.9%
Combined, ungated	86.6%	$+6.3$	96.7%	89.5%
Combined, accuracy-only gate	86.7%	+6.4	96.8%	89.6%
Combined, dual gate	86.5%	$+6.3$	96.6%	89.6%

Table 7: Final performance on the fixed 380-question, 16-document audit excluded from fitting and online promotion. Values average ten ordering-specific final harnesses. Bold marks metric bests; selective accuracy is tied and deltas use unrounded values.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Outcome	Count	Share
Correct	103	54%
Retrieval within document	59	31%
Retrieval across documents	26	14%
Extraction error	2	1%
Generation error	0	0%

Appendix

Table 8: Web-page control results. In this audited corpus, retrieval errors dominate after extraction.

Representation	Coverage@5	End-to-End
350-token chunks	16%	5%
Row packing	29%	10%
Precomputed summaries	39%	15%
Multi-granularity [ Gulati et al., 2026b ]	6%	6%
Whole-table execution	—	78%

Appendix

Table 9: Diagnostic results under a fixed top-5 retrieval budget. Coverage@5 is whether retrieved text contains the evidence needed to answer. Whole-table execution bypasses answer-from-chunks.

Workbook supplied	Gold type	Predicted type
Gold workbook	76.3%	70.1%
FiCo -selected workbook	66.3%	60.8%

Appendix

Table 10: Deterministic QA accuracy in the workbook/type oracle-removal experiment. The fully controlled-to-bridge gap is 15.5 points (document-cluster bootstrap 95% confidence interval 12.7–18.5).

Observed signal	Events	Potential remediation
Count intent but no COUNT	293	Intent representation / generation prompt
Parser error	236	Structured decoding / repair examples
Type or result-shape mismatch	193	Contract calibration / result-column selection
Binder error	169	Schema normalization / grounding
Read-only or single-query violation	122	Constrained decoding

Appendix

Table 11: Frequent verifier signals across candidate trajectories. Signals are non-exclusive event counts, not question-level error rates. Only contract calibration and query/result-column selection are learned in the continual experiment.

Diagnostic	DataBench	MiMoTable
Document recall@10	95.1%	99.6%
Gold document selected for execution	81.3%	90.9%
SQL execution without exception	98.5%	97.2%
Final answer correct	76.2%	79.7%

Appendix

Table 12: Deployed FiCo stage cascade. High retrieval and execution rates do not imply final-answer correctness. All rows belong to the end-to-end track.

Stage	Intervention	Full	Ablated	$\Delta$
Find	Remove summaries (final)	21.3%	18.7%	$-2.6$
Find	Remove schema reranking (final)	21.3%	20.4%	$-0.9$
Find	Remove empty-result fallback (final)	21.3%	21.3%	$0.0$
Compute	Remove representative/sample values (denotation)	25.9%	18.3%	$-7.6$
Compute	Remove representative/sample values (final)	27.8%	20.8%	$-7.0$

Appendix

Table 13: Paired DataBench ablations in a fixed controlled runner. Compute rows use the gold workbook. Only within-row deltas are interpreted.

Answer type	Questions	Correct	Accuracy
Boolean	387	328	84.8%
Number	412	322	78.2%
Category	333	240	72.1%
List[category]	329	210	63.8%
List[number]	349	281	80.5%

Appendix

Table 14: Gold-workbook Vigil accuracy by expected answer type. The expected type is controlled metadata in this track.

Terminal candidate stage	Questions	Correct	Accuracy
Initial candidate 1	1,456	1,157	79.5%
Initial candidate 2	258	178	69.0%
Repair 1	89	44	49.4%
Repair 2	7	2	28.6%

Appendix

Table 15: Verifier treatment by terminal candidate stage, including accepted and rejected terminal candidates. Repair rows are conditioned on both initial candidates failing and are therefore harder. Accuracies are descriptive, not causal effects.

Question	Baseline	FiCo
“Among the top five countries by reported coronavirus cases, which has the highest case-fatality ratio?”	“I don’t know. The context only lists the United States.”	Brazil (2.189%) from a grouped ratio query.
“Among non-drinkers, is gambling mostly Yes or No?”	“Based on context, I cannot determine.”	Mostly No: 89 No versus 33 Yes.
“Do PASS or FAIL movies have a higher profitable share?”	“I don’t know. The context does not contain financial data.”	PASS: 64.2% versus FAIL: 61.8%.

Appendix

Table 16: Representative cases in which full-table execution answers an aggregation that the prefix context cannot support.

System	Selection rule	Mean (%)	$\Delta$ vs. direct (points)
Direct GPT-5.4	Official prompt, medium reasoning	85.66	—
One verifier	Workbook-aware replace/retain	85.58	$-0.08$
Two-verifier consensus	Second verifier authorizes replacement	85.59	$-0.07$
Three-candidate deliberation	Randomized direct/calculation/audit arbitration	85.51	$-0.15$
Calculation candidate	Calculation-specialized prompt	84.96	$-0.70$
Audit candidate	Audit-specialized prompt	84.70	$-0.96$

Appendix

Table 17: Complete gold-blind MiMoTable sweep. Every mean averages three complete judgments by the official GPT-4o prompt. The 0.05-point semantic-agreement difference is descriptive and not statistically resolved. Choosing this variant after the released-test sweep is test-adaptive.

CL-Bench domain	Highest released mean	Adapter mean	$\Delta$
Blind spectrum monitoring	46.198	55.605	$+9.407$
Sales prediction	10.073	11.202	$+1.129$
Exploitable poker	343.020	346.400	$+3.380$

Appendix

Table 18: Mean total reward over five official-schedule runs per domain. The comparator is the highest five-run mean in the released artifacts at the pinned commit. These descriptive comparisons do not extend to other metrics.

Table QA

Table question answering studies how AI systems retrieve, interpret, and reason over structured data to answer natural-language questions.

See all 66 papers

Harness Learning Enables Generalizable Test-Time Adaptation
Sep 28, 2026Alvin Zhang, Xuecheng Liu, Zixuan Wang +6Test-Time Adaptation Meta-Learning
Spreadsheet-RL: Advancing Large Language Model Agents on Realistic Spreadsheet Tasks via Reinforcement Learning
May 21, 2026Banghao Chi, Yining Xie, Mingyuan Wu +9Tool-Using Agents LLM Agent Training
University of Illinois Urbana-Champaign · Meta
Beyond Rows to Reasoning: Agentic Retrieval for Multimodal Spreadsheet Understanding and Editing
Mar 6, 2026Anmol Gulati, Sahil Sen, Waqar Sarguroh +1Agentic RAG Spreadsheet Automation
Commercial Technology and Innovation Office, PricewaterhouseCoopers, U.S.

Sep 28, 2026J/K move · Enter open · S save

Harness Learning Enables Generalizable Test-Time Adaptation

Alvin Zhang, Xuecheng Liu, Zixuan Wang, Fahim Tajwar+5

Test-Time Adaptation Meta-Learning Agent Harness Optimization RL for LLM Agents

Open paper AskPDF arXiv

A language-model agent is jointly defined by its model and its harness, the executable program that organizes model calls, tool use, and information flow. Because different tasks call for different ways of organizing these operations, the harness needs to be adapted using feedback from the task at hand. We introduce harness learning, which trains a proposer model to revise a solver's harness using execution feedback. We formulate this process as meta-learning over executable programs, with harness revisions playing the role of weight updates in gradient-based adaptation. We train the proposer with reinforcement learning, using the task performance of revised harnesses as the reward. At test time, the proposer uses feedback from successive executions on a new task to refine the harness, without performing any parameter-space update. Experiments on reasoning and multi-hop question answering show that harness learning improves revision quality and that the ability to adapt at test time transfers to unseen tasks. Policies trained on individual revisions can continue improving harnesses over multiple rounds, while the benefits of training on revision sequences vary across settings. These findings suggest a path towards continually learning agents that turn accumulated experience into generalizable improvements.

Abstract

Table QA

Explore similar work

Harness Learning Enables Generalizable Test-Time Adaptation

Spreadsheet-RL: Advancing Large Language Model Agents on Realistic Spreadsheet Tasks via Reinforcement Learning

Beyond Rows to Reasoning: Agentic Retrieval for Multimodal Spreadsheet Understanding and Editing