Counterexample-Guided Refinement

Follow this topic Ask about this topic

Counterexample-guided refinement improves a model, specification, program, or evaluator by proposing a candidate, checking it with a verifier or test environment, then using concrete failures to revise it. Formal methods use CEGAR and CEGIS loops with abstraction, SAT/SMT solving, symbolic execution, contracts, and invariant synthesis; newer systems apply similar feedback loops to LLM-generated code, specifications, planning heuristics, and compiled knowledge. Benchmarks include Frama-C and VerifyThis programs, planning domains, regex induction, and code-generation suites, where counterexamples can improve sample efficiency and expose errors missed by static checks. A key challenge is getting feedback that is valid and informative: bounded search, weak or noisy oracles, and repeated revisions can limit correctness, generalization, and convergence.

Latest papers

14 papers2 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Secure Agentic Coding through Counterexample-Grounded Feedback
Oct 19, 2026Tue Le, Anh Nguyen Hoang, Gabriel Sasseville +3Secure Code Generation Agentic Code Generation
Quantum AI & Cyber Security Institute (QACI), FPT Corporation · Mila - Quebec Artificial Intelligence Institute · Université de Montréal +1
Beyond the Verdict: Agentic Code Judging with Verifiable Feedback
Oct 13, 2026Zheye Deng, Sha Li, Changlong Yu +4LLM-as-a-Judge Code Generation Evaluation
HKUST · Amazon
Metrics That Write Themselves: Evolving an Evaluator from Its Own Blind Spots
Aug 19, 2026Xing Zhang, Yanwei Cui, Guanghui Wang +2Counterexample-Guided Refinement Code Generation Evaluation
AWS Generative AI Innovation Center
Self-evolving network verifiers
Aug 11, 2026Ioannis Protogeros, Tibor Schneider, Laurent VanbeverCounterexample-Guided Refinement Formal Verification
ETH Zürich Zürich, Switzerland
Modal CEGAR-tableaux with RECAR and resolution-based SAT-shortcuts
Jun 30, 2026Rajeev Goré, Cormac KikkertAutomated Theorem Proving Counterexample-Guided Refinement
Faculty of Information Technology, Monash University, Australia · Cormac Kikkert Research
Unassigned Agents in Compilation-based Multi-agent Path Finding
Jun 14, 2026Pavel SurynekCounterexample-Guided Refinement Multi-Agent Path Finding
Faculty of Information Technology, Czech Technical University in Prague Thákurova 9, 163 00, Prague, Czechia
Counterexample Guided Learning in the Large using Reasoning Agents
Jun 9, 2026Hongyi Liu, Frederic Sala, Thomas Reps +1Counterexample-Guided Refinement LLM-Based Program Synthesis
Department of Computer Sciences, University of Wisconsin-Madison
2-ASP(Q) programs with weak constraints: Complexity and efficient implementation
May 26, 2026Andrea Cuteri, Giuseppe Mazzotta, Francesco RiccaLogic Programming Counterexample-Guided Refinement
University of Calabria, Rende, Italy
ConVer: Using Contracts and Loop Invariant Synthesis for Scalable Formal Software Verification
May 26, 2026Muhammad A. A. Pirzada, Weiqi Wang, Yiannis Charalambous +2Counterexample-Guided Refinement Formal Verification
The University of Manchester · Manchester, UK
Property-Guided LLM Program Synthesis for Planning
May 15, 2026André G. Pereira, Augusto B. Corrêa, Jendrik SeippSymbolic Planning Counterexample-Guided Refinement
Federal University of Rio Grande do Sul · Brazil · University of Oxford +3
WiCER: Wiki-memory Compile, Evaluate, Refine Iterative Knowledge Compilation for LLM Wiki Systems
May 8, 2026Juan M. HuertaKV Caching Counterexample-Guided Refinement
Zinnia Tech Solutions 600 Steamboat Road Greenwich, CT 06830, USA
The Counterexample Game: Iterated Conceptual Analysis and Repair in Language Models
May 5, 2026Daniel Drucker, Kyle MahowaldLLM Self-Correction LLM Evaluation
Department of Philosophy The University of Texas at Austin · Department of Linguistics The University of Texas at Austin
SpecPylot: Python Specification Generation using Large Language Models
Apr 17, 2026Ragib Shahariar Ayon, Shibbir AhmedCounterexample-Guided Refinement Formal Verification
Texas State University San Marcos, TX, USA
Let the Abyss Stare Back Adaptive Falsification for Autonomous Scientific Discovery
Mar 30, 2026Peiran Li, Fangzhou Lin, Shuo Xing +5Autonomous Scientific Discovery Counterexample-Guided Refinement
Texas A&M University · VecTrue AI · Worcester Polytechnic Institute +3

Oct 19, 2026J/K move · Enter open · S save

Secure Agentic Coding through Counterexample-Grounded Feedback

Tue Le, Anh Nguyen Hoang, Gabriel Sasseville, Doseok Jang+2

Quantum AI & Cyber Security Institute (QACI), FPT Corporation · Mila - Quebec Artificial Intelligence Institute · Université de Montréal+1

Secure Code Generation Agentic Code Generation Reinforcement Learning with Verifiable Rewards Counterexample-Guided Refinement Execution-Guided Code Generation

Open paper AskPDF arXiv

As coding models and agent harnesses become more capable, reliable security verification is becoming a central bottleneck. Improving secure code generation requires a signal whose reported progress remains aligned with actual program behavior under optimization; otherwise, an optimizer can improve the signal's verdict without removing the underlying vulnerability. Existing model-based approaches require models to either infer program behavior from source code or specify correct outcomes for adversarial tests, placing substantial reasoning demands on the security signal. We introduce \falcon{}, a counterexample-grounded verification framework that separates attack exploration from behavioral verification and connects them through execution traces of the program under test. In test-time repair, \falcon{} yields substantially stronger held-out functional-security gains than alternative security signals across secure code generation benchmarks, while preserving or improving functionality. As an RL reward, \falcon{} also outperforms static-analysis and learned rewards on held-out functional-security, even when all policies successfully optimize their respective training rewards. Combining \falcon{} with either proxy reward further improves held-out performance, suggesting that execution-grounded feedback can mitigate reward overoptimization while preserving useful guidance from existing security signals.