Agent-Guided Configuration Tuning for SMT Solvers
Organizations: Tsinghua University, China · Shanghai Jiao Tong University, China · University of Toronto, Canada
Abstract
SMT solvers expose many configuration choices that determine how well their existing algorithms perform on a workload. We present, to our knowledge, the first workflow that uses general-purpose LLM agents to tune SMT solver configurations. Agents read solver documentation, source code, and persistent experiment records to propose option changes, and a deterministic harness measures each proposal, validates the returned models, and selects changes separately for each competition Division. Over roughly nine weeks, our tuning campaign recorded 4,545 proposals for cvc5 across 13 Divisions and 36 logics. We evaluate the tuned configurations on the 2026 and 2025 SMT-COMP ModelValidation benchmarks under a 24-second cutoff. On problems not used for training, they validate more problems than official cvc5 in ten of the 13 Divisions in 2026 and in eight in 2025, with geometric-mean gains of 7.8% and 6.8% across Divisions. On the complete official selections, which include training problems, the tuned floating-point counts exceed those of every individual competition participant in both years. Further studies show how configuration changes make existing algorithms effective by changing access to strategies, scheduling, and option compatibility, and how benchmark families trade off against one another. The agents' experiment traces contain hypothesis-based proposals in every tuning group, and 64.3% of the extracted hypotheses draw on prior measurements. These results show that agent-guided tuning can specialize a mature SMT solver and that accumulated experimental evidence informs sustained search.
Figures & tables
| Phase | Duration | Workload and transition | Records | Proposals |
|---|---|---|---|---|
| I | About 2 weeks | 10,104 problems; five groups; host A; answer-based search | 517 | 511 |
| II | About 3 weeks | Same problems and host; inherited models validated; per-Division selection | 997 | 997 |
| III | About 3.5 weeks | 15,252 problems; six groups; phase-II configurations remeasured on hosts B/C | 3,045 | 3,037 |
| Total | 13 Divisions, 36 logics | 4,559 | 4,545 |
| Quantity | Recorded value |
|---|---|
| Campaign coverage | 4,559 records; 4,545 proposals; 13 Divisions and 36 logics |
| Training problems | Phases I–II: 10,104; phase III: 15,252 |
| Timed measurements | 4,407 execution intervals: 4,396 proposal, 11 baseline |
| Recorded measurement cost | 24,219.5 slot hours (about 24,220); four logical CPUs per slot |
| Phase-III allocation | 22 slots on two hosts; six groups; six slots for NIA |
| Per-problem limits | 24 s wall time and 96 s CPU time |
| 2026 test | 2025 test | |||||||
| Division | Official | Tuned | Official | Tuned | ||||
| Nonlinear integer arithmetic | 919 | 481 | 640 | +159 | 4,937 | 2,758 | 3,313 | +555 |
| ADT + bit-vectors | 931 | 824 | 879 | +55 | 4,486 | 3,989 | 4,308 | +319 |
| ADT + linear arithmetic | 517 | 501 | 505 | +4 | 535 | 535 | 535 | +0 |
| Bit-vectors | 1,230 | 1,140 | 1,142 | +2 | 7,359 | 6,892 | 6,808 | -84 |
| Datatypes | 537 | 310 | 314 | +4 | 1,715 | 1,128 | 1,137 | +9 |
| 2026 complete selection | 2025 complete selection | |||||||||
| Division | Official | Tuned | Change | Official | Tuned | Change | ||||
| Nonlinear integer arithmetic | 1,901 | 761 | 1,322 | +561 | +73.7% | 7,516 | 2,940 | 5,034 | +2,094 | +71.2% |
| ADT + bit-vectors | 1,513 | 1,201 | 1,365 | +164 | +13.7% | 5,249 | 4,220 | 4,874 | +654 | +15.5% |
| ADT + linear arithmetic | 798 | 736 | 758 | +22 | +3.0% | 772 | 730 | 749 | +19 | +2.6% |
| Bit-vectors | 1,909 | 1,674 | 1,712 | +38 | +2.3% | 8,211 | 7,271 | 7,301 | +30 | +0.4% |
| Datatypes | 871 | 461 | 497 | +36 | +7.8% | 1,943 | 1,221 | 1,267 | +46 | +3.8% |
| Check | 2026 | 2025 |
|---|---|---|
| Problems | 19,094 | 59,750 |
| Wrong answers or wrong models found | 0 | 0 |
| Missing partial-function values | 348 | 570 |
| Model parsing errors | 12 | 67 |
| Total unverified | 360 | 637 |
| Same validation error in official cvc5 | 342 | 626 |
| Case | Configuration / mechanism | sat within 24 s | Additional observation |
|---|---|---|---|
| NIA | Reference portfolio: internal limit 1,200 s | 1 / 20 | 0 / 20 started a second strategy. |
| Reference + experimental internal 24 s limit | 8 / 20 | 20 / 20 started multiple strategies. | |
| Reference + five portfolio workers | 12 / 20 | 20 / 20 started multiple strategies. | |
| Tuned configuration | 20 / 20 | Answers came from five strategies. | |
| LIA | Reference portfolio: model-output option conflict | 5 / 20 | Fallback occurred on all 20. |
| Reference + experimental internal 24 s limit | 5 / 20 | Time limit does not fix the conflict. |
| Phase | Records | Text present | With predictions | Hypotheses | Prospective |
|---|---|---|---|---|---|
| I | 517 | 502 | 95 | 101 | 63 |
| II | 997 | 952 | 104 | 127 | 102 |
| III | 3,045 | 3,035 | 553 | 736 | 736 |
| Total | 4,559 | 4,489 | 752 | 964 | 901 |
| Hypotheses | Supported | Contradicted | Mixed | Inconclusive | Not evaluated |
| 964 | 96 | 113 | 4 | 258 | 493 |
| Quantity | Value | Interpretation |
|---|---|---|
| Prior measurements | 620 (64.3%) | Most common evidence category. |
| Source or documentation inspection | 203 (21.1%) | Solver behavior or option definitions motivate proposals. |
| Theoretical prior / other | 134 / 7 | One evidence label per hypothesis. |
| Agreement on prediction presence | 2,791 / 2,992 (93.3%) | Both readers flag 402 records. |
| Direction-list agreement when both flag | 208 / 402 (51.7%) | Sorted direction lists match exactly, counting repeated directions. |
| Observation | Recorded evidence | Workflow boundary |
|---|---|---|
| NIA repeat-measurement diagnosis | In a session midway through phase III, the agent reported ten runs of configurations equivalent to the incumbent, with mean 3,386 and standard deviation about 5.4; the incumbent record of 3,396 was their maximum. | Acceptance still compared candidates with the incumbent record. Recognizing noise did not itself change that rule. |
| NIA family tradeoff | The same session described removing nl-icp as solving SAT14 while losing ITS. Recorded training comparisons show the tradeoff. | The measured net score governed acceptance; no family-preservation rule was evaluated. |
| Floating-point continued search | The accounting snapshot from the day before tuning ended records 1,361 rounds and about 229 measurement hours after the last acceptance. | The workflow had no automatic stopping rule. This observation does not establish an optimal stopping threshold. |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Category | Description | Tunable options | Changed options |
|---|---|---|---|
| Random seeds | Seeds controlling randomized choices. | 2 | 1 |
| Portfolio | Use of the predefined portfolio and number of concurrent portfolio workers. | 2 | 2 |
| SAT back ends | Choice of SAT and bit-vector solvers; back-end simplification. | 4 | 3 |
| SAT search | Restarts, randomized search, and conflict or lemma processing. | 11 | 5 |
| Decision heuristics | Decision modes and justification heuristics. | 4 | 3 |
| Preprocessing | Assertion simplification, rewriting, and other preprocessing options. | 20 | 13 |
Satisfiability Modulo Theories
Satisfiability modulo theories (SMT) checks whether logical formulas are satisfiable under theories such as arithmetic, strings, and bit-vectors.
See all 23 papers