GRACE: Parallel Best-of-N Multi-Agent Analysis of Closed-Form Questions over Dataframes with Guardrail Selection and Adaptive Retry
Organizations: Engineering Ingegneria Informatica S.p.A., Piazzale dell’Agricoltura 24, 00144 Roma, Italy
Abstract
LLM agents answer closed-form questions over a tabular dataframe by writing and executing Python, almost always as a single rollout: one interpretation, one script, one result, accepted without comparison. The few pipelines that produce several candidates reconcile them by answer-agreement voting, which needs a single comparable scalar and always returns one, even when every candidate is wrong. We present GRACE (Generative, Ranked, Adaptive Code Ensemble), a production-oriented pipeline that instead runs parallel best-of-N selection specialised to dataframe code. N agents run concurrently under a strict, machine-checkable output contract, diversified by persona hints, per-retry rotation, and a two-regime temperature schedule. Rather than vote, GRACE judges the executed artefacts: a pairwise tournament pre-filters the pool and a domain-specialised guardrail either selects the best candidate or, when none is defensible, rejects the whole pool and abstains with a diagnosis (an option voting structurally lacks), which in turn drives an adaptive retry with per-agent feedback. On InfiAgent-DABench (DAEval, M = 246), using Qwen3.6-35B-A3B as the backbone, GRACE raises answer match from 39% for a single-rollout baseline to 71% (+32 pp, 1.8x), recovering 53% of previously wrong answers. On the same DAEval task, the smaller dense Qwen3.5-9B lifts answer match from 38% to 63% (+25 pp, 1.7x). A controlled 3 x 3 study that sweeps the number of parallel agents against the retry budget finds, on the 35B backbone, the two levers largely substitutable: once either one is generous, the other adds little, identifying a cheap operating point for deployment. The same directional pattern replicates on the independent DataBench benchmark, again on the Qwen3.6-35B-A3B backbone, where answer match rises by 15 percentage points (from 70% to 85%) across 400 sampled questions.
Figures & tables
| Benchmark | Representative questions |
| DAEval | • Calculate the mean, median, and population standard deviation of passenger fares separately for each passenger class, and interpret the results. • Encode categorical variables and train and evaluate a regression model to predict passenger survival. |
| DataBench | • How many billionaires are there in the Technology category? • List the three countries with the most billionaires. |
| correct candidate in pool? | sel. correct | sel. wrong | rejected | total |
| yes | 176 (86.3%) | 17 (8.3%) | 11 (5.4%) | 204 |
| no | – | 53 (89.8%) | 6 (10.2%) | 59 |
| all | 176 (66.9%) | 70 | 17 | 263 |
Table QA
Table question answering studies how AI systems retrieve, interpret, and reason over structured data to answer natural-language questions.
See all 69 papers