Conv-to-Bench: Evaluating Language Models Via User-Assistant Dialogues In Code Tasks

Victor M. dos Santos; Andre C. Castro; Samuel L. de S. Toledo; Bruno M. L. Calura; Lisandra C. de M. Menezes; Raul C. R. Mata; Telma W. de L. Soares; Bryan L. M. de Oliveira

cs.CLApr 7, 2026

Conv-to-Bench: Evaluating Language Models Via User-Assistant Dialogues In Code Tasks

Authors: Victor M. dos Santos, Andre C. Castro, Samuel L. de S. Toledo, Bruno M. L. Calura, Lisandra C. de M. Menezes, Raul C. R. Mata, Telma W. de L. Soares, Bryan L. M. de Oliveira

Organizations: Institute of Mathematics and Computer Science, University of São Paulo, Brazil · HUG Labs, Brazil · Institute of Informatics, Federal University of Goiás, Brazil · Advanced Knowledge Center for Immersive Technologies (AKCIT), Brazil

Abstract

The rapid advancement of Large Language Models (LLMs) has outpaced the scalability of traditional evaluation benchmarks, which remain heavily dependent on labor-intensive expert curation. We address this bottleneck with Conv-to-Bench, a multi-stage framework that automatically transforms authentic multi-turn user-assistant dialogues into structured, verifiable requirement checklists. By leveraging the "instructional evolution" found in real-world conversational logs, our approach deconstructs fragmented user intent into consolidated instructions and binary evaluation criteria. Applied to the programming domain, Conv-to-Bench produces evaluation sets that demonstrate near-perfect alignment with human-authored standards like BigCodeBench, achieving Spearman correlations of up to $ρ$ = 1.000 with significantly lower computational overhead. Validation of the LLM-as-a-judge framework further confirms its reliability, with the primary evaluator achieving substantial agreement with human-verified ground truth ( $κ$ = 0.705). Our comprehensive ablation studies reveal that while multi-turn interactions capture the iterative evolution of user intent, instruction-centric extraction provides a more robust foundation. Ultimately, Conv-to-Bench provides a scalable, cost-effective paradigm for maintaining high-fidelity evaluation standards as user-centric AI applications continue to diversify.

Figures & tables

Figure 1: Overview of the framework. The diagram illustrates the proposed multi-stage process for transforming raw, multi-turn conversational data into structured (instruction, requirement checklist) evaluation pairs. In the user messages, text highlighted in blue corresponds to extracted instructions, while text highlighted in red represents user feedback.

Evaluator	Negative	Positive	Macro	Cohen’s $\kappa$
	F1-score
Gemini-2.5-Flash	0.773	0.932	0.852	0.705
GPT-5-Mini	0.740	0.907	0.824	0.649
Deepseek-Coder-33B-Instruct	0.574	0.887	0.731	0.464

Table 1: Classification performance for the evaluator models shows that while results are not perfect, the Macro F1-score and Cohen’s

\kappa

are satisfactory for proprietary models in automated code evaluation tasks within the Conv-to-Bench code benchmark. Higher values indicate better classification performance.

		BigCodeBench (Instruct)				BigCodeBench (Complete)
		Spearman		Kendall		Spearman		Kendall
Evaluator	Subset	$\rho$	$p$	$\tau$	$p$	$\rho$	$p$	$\tau$	$p$
Gemini-2.5-Flash	Full	1.000	$<$ 0.001	1.000	0.003	0.943	0.005	0.867	0.017
	Instructions-Only	1.000	$<$ 0.001	1.000	0.003	0.943	0.005	0.867	0.017
GPT-5-Mini	Full	0.886	0.019	0.733	0.056	0.943	0.005	0.867	0.017
	Instructions-Only	0.943	0.005	0.867	0.017	1.000	$<$ 0.001	1.000	0.003

Table 2: Conv-to-Bench code benchmark scores show near-perfect correlation with the BigCodeBench (Full Set), with Instructions-Only scores standing out for their exceptionally high alignment. Spearman (

\rho

) and Kendall (

\tau

) correlations, reporting coefficients and

p

-values. The highest correlations (

\rho

) per metric are bolded. A higher

\rho

(closer to 1.0) indicates a stronger correlation, while a low

p

-value (e.g.,

<0.05

) indicates statistical significance.

		BigCodeBench (Instruct)				BigCodeBench (Complete)
		Spearman		Kendall		Spearman		Kendall
Evaluator	Subset	$\rho$	$p$	$\tau$	$p$	$\rho$	$p$	$\tau$	$p$
Gemini-2.5-Flash	Full	0.952	$<$ 0.001	0.857	0.002	0.994	$<$ 0.001	0.982	0.001
	Instructions-Only	0.952	$<$ 0.001	0.857	0.002	0.994	$<$ 0.001	0.982	0.001
GPT-5-Mini	Full	0.976	$<$ 0.001	0.929	$<$ 0.001	0.946	$<$ 0.001	0.837	0.004
	Instructions-Only	0.952	$<$ 0.001	0.857	0.002	0.970	$<$ 0.001	0.909	0.002

Table 3: Conv-to-Bench code benchmark demonstrates robust consistency on the BigCodeBench (Hard Set), showing a more balanced performance across subsets than observed in the BigCodeBench (Full Set). Spearman (

\rho

) and Kendall (

\tau

) correlations, reporting coefficients and

p

-values. The highest correlations (

\rho

) per metric are bolded. A higher

\rho

(closer to 1.0) indicates a stronger correlation, while a low

p

-value (e.g.,

<0.05

) indicates statistical significance.

		BigCodeBench (Instruct)				BigCodeBench (Complete)
		Spearman		Kendall		Spearman		Kendall
Evaluator	Benchmark	$\rho$	$p$	$\tau$	$p$	$\rho$	$p$	$\tau$	$p$
Gemini-2.5-Flash	Conv-to-Bench (Ours)	1.000	$<$ 0.001	1.000	0.003	0.943	0.005	0.867	0.017
	Arena-Hard-Auto (Code)	0.943	0.005	0.867	0.017	0.886	0.019	0.733	0.056
GPT-5-Mini	Conv-to-Bench (Ours)	0.886	0.019	0.733	0.056	0.943	0.005	0.867	0.017
	Arena-Hard-Auto (Code)	0.829	0.042	0.733	0.056	0.771	0.072	0.600	0.136

Table 4: Conv-to-Bench (Instructions-Only) demonstrates superior alignment with the BigCodeBench (Full Set) compared to the programming-specific subset of the Arena-Hard-Auto benchmark. For our method, the Instructions-Only configuration is utilized due to its stronger performance in previous evaluations. A higher

\rho

(closer to 1.0) indicates a stronger correlation, while a low

p

-value (e.g.,

<0.05

) indicates statistical significance.

		BigCodeBench (Instruct)				BigCodeBench (Complete)
		Spearman		Kendall		Spearman		Kendall
Evaluator	Benchmark	$\rho$	$p$	$\tau$	$p$	$\rho$	$p$	$\tau$	$p$
Gemini-2.5-Flash	Conv-to-Bench (Ours)	0.952	$<$ 0.001	0.857	0.002	0.994	$<$ 0.001	0.982	0.001
	Arena-Hard-Auto (Code)	0.905	0.002	0.786	0.006	0.970	$<$ 0.001	0.909	0.002
GPT-5-Mini	Conv-to-Bench (Ours)	0.905	0.002	0.786	0.006	0.946	$<$ 0.001	0.837	0.004
	Arena-Hard-Auto (Code)	0.952	$<$ 0.001	0.857	0.002	0.922	0.001	0.837	0.004

Table 5: Conv-to-Bench (Instructions-Only) maintain competitive rank alignment with the BigCodeBench (Hard Set), outperforming the programming-specific subset of the Arena-Hard-Auto benchmark. This analysis employs the Instructions-Only method for Conv-to-Bench following its better results in earlier tests. A higher

\rho

(closer to 1.0) indicates a stronger correlation, while a low

p

-value (e.g.,

<0.05

) indicates statistical significance.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Verbosity Bias Analysis. This visualization assesses the relationship between response length (tokens) and normalized instruction scores across the LLM judges. While gray dots denote individual task evaluations, the red line tracks the moving average trend. The low Pearson correlation coefficients ( $r$ ) and non-significant $p$ -values indicate a lack of verbosity bias, confirming that the judges’ scoring remains independent of response length.

	Hard Subset (%)			Full Subset (%)
Model	Complete	Instruct	Average	Complete	Instruct	Average
GPT-4.1-Nano-2025-04-14	31.8	28.4	30.1	–	–	–
Gemini-2.0-Flash-001	33.8	23.6	28.7	–	–	–
Qwen2.5-Coder-7B-Instruct	20.3	20.3	20.3	48.8	40.4	44.6
CodeQwen1.5-7B-Chat	15.5	18.9	17.2	43.6	39.6	41.6
DeepSeek-coder-6.7B-Instruct	15.5	10.1	12.8	43.8	35.5	39.6

Appendix

Table 6: BigCodeBench performance metrics across model variants. Detailed performance scores for the “Hard” and “Full” subsets. Results represent the percentage of tasks successfully completed in both ’Complete’ and ’Instruct’ settings. Missing values (–) indicate data not available in the official benchmark leaderboard.

	Mean Tokens ( $\mu$ )
Evaluator	Prompt	Completion
Gemini-2.5-Flash	1,168,581.38	193,097.38
GPT-5-Mini	1,227,954.25	674,392.25

Appendix

Table 7: Token usage per full benchmark execution. Total prompt and completion token counts required to evaluate a single model across the 387-item dataset. Lower token counts represent higher computational and cost efficiency for the evaluator framework.

Category	Instruction Example / Failure Reason
Anonymization	“Write a NAME_2 page using NAME_1.js for the front-end and FastAPI for the backend.”
	Failure: The de-identification process, performed by the original dataset providers, compromised technical semantics, rendering the instruction unintelligible and impossible to fulfill.
Ambiguity	“Write me a python code for the best virtual assistant possible.”
	Failure: The subjectivity of the phrase “best possible” and the absence of specific functional requirements preclude a deterministic implementation.

Appendix

Table 8: Samples of instructions discarded during quality filtering.

Figure 3: Custom interface for gold-standard dataset validation. Experts evaluated a balanced sample of outputs from eight models to mitigate architectural bias. The interface displays the model response alongside the specific requirement, where annotators provide a binary ground-truth label $y\in\{0,1\}$ based on criterion fulfillment.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,355 papers

CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Models
May 28, 2026Vedant PadwalLanguage Model Generation Evaluation Code Generation
Independent
Beyond Ideal Instruction: A Comprehensive Framework for Evaluating LLMs in Realistic Interactions
Jun 2, 2026Xuan Yang, Hao Xu, Tingfeng Hui +4LLM Evaluation Tool-Use Evaluation
Department of Data Science, City University of Hong Kong · 3Li Auto Inc. · 2Hong Kong Institute of AI for Science, City University of Hong Kong +1
From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs
Apr 15, 2026Itay Itzhak, Eliya Habba, Gabriel Stanovsky +1LLM Evaluation Human Preference Evaluation
Technion – Israel Institute of Technology · The Hebrew University of Jerusalem

May 28, 2026J/K move · Enter open · S save

CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Models

Vedant Padwal

Independent

Language Model Generation Evaluation Code Generation Code Generation Evaluation Code Generation Benchmarks LLM Code Generation Multilingual Code Generation

Open paper AskPDF arXiv

This paper introduces Code Bench, a benchmark capable of evaluating Large Language Models (LLMs) concise code generation abilities in 60 programming languages. Based on code golf, a recreational programming competition focused on minimal character or byte solutions, the benchmark provides a distinctive measure of LLMs ability to produce efficient, concise code. Unlike existing benchmarks limited by fixed problem sets and language coverage, CodeGolf Bench leverages the code.golf platform to provide new problems and live human performance baselines. Evaluation of nine LLMs on Python and C++ tasks demonstrates that reasoning models significantly outperform non-reasoning models, achieving best average percentile of 70.97%. This performance gap is particularly pronounced in C++, highlighting reasoning's importance for languages with strict syntax requirements. Non-reasoning models struggle more with efficiency optimization across both languages, with best percentiles significantly lower than reasoning counterparts. CodeGolf Bench offers a dynamic framework for evaluating LLM code generation capabilities against evolving human performance on code golf.

Abstract

LLM Evaluation

Explore similar work

CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Models

Beyond Ideal Instruction: A Comprehensive Framework for Evaluating LLMs in Realistic Interactions

From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs