Paper ID: 2410.08911 • Published Oct 11, 2024
Test-driven Software Experimentation with LASSO: an LLM Benchmarking Example
Marcus Kessel
TL;DR
Get AI-generated summaries with premium
Get AI-generated summaries with premium
Empirical software engineering faces a critical gap: the lack of standardized
tools for rapid development and execution of Test-Driven Software Experiments
(TDSEs) - that is, experiments that involve the execution of software subjects
and the observation and analysis of their "de facto" run-time behavior. In this
paper we present a general-purpose analysis platform called LASSO that provides
a minimal set of domain-specific languages and data structures to conduct
TDSEs. By empowering users with an executable scripting language to design and
execute TDSEs, LASSO enables efficient evaluation of run-time semantics and
execution characteristics in addition to statically determined properties. We
present an example TDSE that demonstrates the practical benefits of LASSO's
scripting capabilities for assessing the reliability of LLMs for code
generation by means of a self-contained, reusable and extensible study script.
The LASSO platform is freely available at:
this https URL, and a demo video is available on
YouTube: this https URL