Code Generation Evaluation
Overview
Code generation evaluation studies how well AI systems produce programs that work and satisfy the requirements developers intended. Common approaches include executable unit tests and pass@k benchmarks, repository-level issue resolution, and tests of generated applications through browsers or other interactive environments. Researchers also assess qualities beyond passing tests, such as security, maintainability, performance, instruction adherence, and fidelity to domain-specific behavior. A central challenge is that test success can overstate real-world reliability, while stochastic outputs, benchmark contamination, and imperfect automated judges make comparisons difficult.
Latest papers
228 papers13 papers in the last four weeks, down 55% on the four weeks before. 0.1% of all new papers.