Conv-to-Bench: Evaluating Language Models Via User-Assistant Dialogues In Code Tasks
Authors: Victor M. dos Santos, Andre C. Castro, Samuel L. de S. Toledo, Bruno M. L. Calura, Lisandra C. de M. Menezes, Raul C. R. Mata, Telma W. de L. Soares, Bryan L. M. de Oliveira
Organizations: Institute of Mathematics and Computer Science, University of São Paulo, Brazil · HUG Labs, Brazil · Institute of Informatics, Federal University of Goiás, Brazil · Advanced Knowledge Center for Immersive Technologies (AKCIT), Brazil
The rapid advancement of Large Language Models (LLMs) has outpaced the scalability of traditional evaluation benchmarks, which remain heavily dependent on labor-intensive expert curation. We address this bottleneck with Conv-to-Bench, a multi-stage framework that automatically transforms authentic multi-turn user-assistant dialogues into structured, verifiable requirement checklists. By leveraging the "instructional evolution" found in real-world conversational logs, our approach deconstructs fragmented user intent into consolidated instructions and binary evaluation criteria. Applied to the programming domain, Conv-to-Bench produces evaluation sets that demonstrate near-perfect alignment with human-authored standards like BigCodeBench, achieving Spearman correlations of up to ρ = 1.000 with significantly lower computational overhead. Validation of the LLM-as-a-judge framework further confirms its reliability, with the primary evaluator achieving substantial agreement with human-verified ground truth (κ = 0.705). Our comprehensive ablation studies reveal that while multi-turn interactions capture the iterative evolution of user intent, instruction-centric extraction provides a more robust foundation. Ultimately, Conv-to-Bench provides a scalable, cost-effective paradigm for maintaining high-fidelity evaluation standards as user-centric AI applications continue to diversify.
Figures & tables
Figure 1: Overview of the framework. The diagram illustrates the proposed multi-stage process for transforming raw, multi-turn conversational data into structured (instruction, requirement checklist) evaluation pairs. In the user messages, text highlighted in blue corresponds to extracted instructions, while text highlighted in red represents user feedback.
F1-score
Evaluator
Negative
Positive
Macro
Cohen’s κ
Gemini-2.5-Flash
0.773
0.932
0.852
0.705
GPT-5-Mini
0.740
0.907
0.824
0.649
Deepseek-Coder-33B-Instruct
0.574
0.887
0.731
0.464
Table 1: Classification performance for the evaluator models shows that while results are not perfect, the Macro F1-score and Cohen’s κ are satisfactory for proprietary models in automated code evaluation tasks within the Conv-to-Bench code benchmark. Higher values indicate better classification performance.
BigCodeBench (Instruct)
BigCodeBench (Complete)
Spearman
Kendall
Spearman
Kendall
Evaluator
Subset
ρ
p
τ
p
ρ
p
τ
p
Gemini-2.5-Flash
Full
1.000
< 0.001
1.000
0.003
0.943
0.005
0.867
0.017
Instructions-Only
1.000
< 0.001
1.000
0.003
0.943
0.005
0.867
0.017
GPT-5-Mini
Full
0.886
0.019
0.733
0.056
0.943
0.005
0.867
0.017
Instructions-Only
0.943
0.005
0.867
0.017
1.000
< 0.001
1.000
0.003
Table 2: Conv-to-Bench code benchmark scores show near-perfect correlation with the BigCodeBench (Full Set), with Instructions-Only scores standing out for their exceptionally high alignment. Spearman ( ρ ) and Kendall ( τ ) correlations, reporting coefficients and p -values. The highest correlations ( ρ ) per metric are bolded. A higher ρ (closer to 1.0) indicates a stronger correlation, while a low p -value (e.g., <0.05 ) indicates statistical significance.
BigCodeBench (Instruct)
BigCodeBench (Complete)
Spearman
Kendall
Spearman
Kendall
Evaluator
Subset
ρ
p
τ
p
ρ
p
τ
p
Gemini-2.5-Flash
Full
0.952
< 0.001
0.857
0.002
0.994
< 0.001
0.982
0.001
Instructions-Only
0.952
< 0.001
0.857
0.002
0.994
< 0.001
0.982
0.001
GPT-5-Mini
Full
0.976
< 0.001
0.929
< 0.001
0.946
< 0.001
0.837
0.004
Instructions-Only
0.952
< 0.001
0.857
0.002
0.970
< 0.001
0.909
0.002
Table 3: Conv-to-Bench code benchmark demonstrates robust consistency on the BigCodeBench (Hard Set), showing a more balanced performance across subsets than observed in the BigCodeBench (Full Set). Spearman ( ρ ) and Kendall ( τ ) correlations, reporting coefficients and p -values. The highest correlations ( ρ ) per metric are bolded. A higher ρ (closer to 1.0) indicates a stronger correlation, while a low p -value (e.g., <0.05 ) indicates statistical significance.
BigCodeBench (Instruct)
BigCodeBench (Complete)
Spearman
Kendall
Spearman
Kendall
Evaluator
Benchmark
ρ
p
τ
p
ρ
p
τ
p
Gemini-2.5-Flash
Conv-to-Bench (Ours)
1.000
< 0.001
1.000
0.003
0.943
0.005
0.867
0.017
Arena-Hard-Auto (Code)
0.943
0.005
0.867
0.017
0.886
0.019
0.733
0.056
GPT-5-Mini
Conv-to-Bench (Ours)
0.886
0.019
0.733
0.056
0.943
0.005
0.867
0.017
Arena-Hard-Auto (Code)
0.829
0.042
0.733
0.056
0.771
0.072
0.600
0.136
Table 4: Conv-to-Bench (Instructions-Only) demonstrates superior alignment with the BigCodeBench (Full Set) compared to the programming-specific subset of the Arena-Hard-Auto benchmark. For our method, the Instructions-Only configuration is utilized due to its stronger performance in previous evaluations. A higher ρ (closer to 1.0) indicates a stronger correlation, while a low p -value (e.g., <0.05 ) indicates statistical significance.
BigCodeBench (Instruct)
BigCodeBench (Complete)
Spearman
Kendall
Spearman
Kendall
Evaluator
Benchmark
ρ
p
τ
p
ρ
p
τ
p
Gemini-2.5-Flash
Conv-to-Bench (Ours)
0.952
< 0.001
0.857
0.002
0.994
< 0.001
0.982
0.001
Arena-Hard-Auto (Code)
0.905
0.002
0.786
0.006
0.970
< 0.001
0.909
0.002
GPT-5-Mini
Conv-to-Bench (Ours)
0.905
0.002
0.786
0.006
0.946
< 0.001
0.837
0.004
Arena-Hard-Auto (Code)
0.952
< 0.001
0.857
0.002
0.922
0.001
0.837
0.004
Table 5: Conv-to-Bench (Instructions-Only) maintain competitive rank alignment with the BigCodeBench (Hard Set), outperforming the programming-specific subset of the Arena-Hard-Auto benchmark. This analysis employs the Instructions-Only method for Conv-to-Bench following its better results in earlier tests. A higher ρ (closer to 1.0) indicates a stronger correlation, while a low p -value (e.g., <0.05 ) indicates statistical significance.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Verbosity Bias Analysis. This visualization assesses the relationship between response length (tokens) and normalized instruction scores across the LLM judges. While gray dots denote individual task evaluations, the red line tracks the moving average trend. The low Pearson correlation coefficients ( r ) and non-significant p -values indicate a lack of verbosity bias, confirming that the judges’ scoring remains independent of response length.
Hard Subset (%)
Full Subset (%)
Model
Complete
Instruct
Average
Complete
Instruct
Average
GPT-4.1-Nano-2025-04-14
31.8
28.4
30.1
–
–
–
Gemini-2.0-Flash-001
33.8
23.6
28.7
–
–
–
Qwen2.5-Coder-7B-Instruct
20.3
20.3
20.3
48.8
40.4
44.6
CodeQwen1.5-7B-Chat
15.5
18.9
17.2
43.6
39.6
41.6
DeepSeek-coder-6.7B-Instruct
15.5
10.1
12.8
43.8
35.5
39.6
Appendix
Table 6: BigCodeBench performance metrics across model variants. Detailed performance scores for the “Hard” and “Full” subsets. Results represent the percentage of tasks successfully completed in both ’Complete’ and ’Instruct’ settings. Missing values (–) indicate data not available in the official benchmark leaderboard.
Mean Tokens ( μ )
Evaluator
Prompt
Completion
Gemini-2.5-Flash
1,168,581.38
193,097.38
GPT-5-Mini
1,227,954.25
674,392.25
Appendix
Table 7: Token usage per full benchmark execution. Total prompt and completion token counts required to evaluate a single model across the 387-item dataset. Lower token counts represent higher computational and cost efficiency for the evaluator framework.
Category
Instruction Example / Failure Reason
Anonymization
“Write a NAME_2 page using NAME_1.js for the front-end and FastAPI for the backend.”
Failure: The de-identification process, performed by the original dataset providers, compromised technical semantics, rendering the instruction unintelligible and impossible to fulfill.
Ambiguity
“Write me a python code for the best virtual assistant possible.”
Failure: The subjectivity of the phrase “best possible” and the absence of specific functional requirements preclude a deterministic implementation.
Appendix
Table 8: Samples of instructions discarded during quality filtering.
Figure 3: Custom interface for gold-standard dataset validation. Experts evaluated a balanced sample of outputs from eight models to mitigate architectural bias. The interface displays the model response alongside the specific requirement, where annotators provide a binary ground-truth label y∈{0,1} based on criterion fulfillment.
This paper introduces Code Bench, a benchmark capable of evaluating Large Language Models (LLMs) concise code generation abilities in 60 programming languages. Based on code golf, a recreational programming competition focused on minimal character or byte solutions, the benchmark provides a distinctive measure of LLMs ability to produce efficient, concise code. Unlike existing benchmarks limited by fixed problem sets and language coverage, CodeGolf Bench leverages the code.golf platform to provide new problems and live human performance baselines. Evaluation of nine LLMs on Python and C++ tasks demonstrates that reasoning models significantly outperform non-reasoning models, achieving best average percentile of 70.97%. This performance gap is particularly pronounced in C++, highlighting reasoning's importance for languages with strict syntax requirements. Non-reasoning models struggle more with efficiency optimization across both languages, with best percentiles significantly lower than reasoning counterparts. CodeGolf Bench offers a dynamic framework for evaluating LLM code generation capabilities against evolving human performance on code golf.
Despite great advances in tool-use capabilities of large language models (LLMs), existing evaluation benchmarks struggle to fully align with real-world scenarios. Such benchmarks mostly rely on simulated idealized user assumptions and lacks experience-oriented evaluation. These limitations fail to account for the ambiguity, uncooperative behaviors, and shifting intentions characteristic of real-world users. To fill this gap, we propose RUT-Bench, a dedicated benchmark designed to assess LLMs under diverse Real-world User Tool calling scenarios. RUT-Bench supports high-fidelity simulations covering both ideal rational patterns and heterogeneous non-ideal behaviors across single-turn and multi-turn dialogues. We conduct comprehensive evaluations on 19 widely adopted open-source and proprietary LLMs using our benchmark. Experimental results reveal that no tested LLMs achieve an overall success rate above 40%, and nearly all of them experience noticeable performance drops when facing more complicated non-ideal user inputs. Our code and data is available at https://github.com/Miaow-Lab/RUT-Bench.
Xuan Yang, Hao Xu, Tingfeng Hui +4
Department of Data Science, City University of Hong Kong · 3Li Auto Inc. · 2Hong Kong Institute of AI for Science, City University of Hong Kong +1
Evaluating LLMs is challenging, as benchmark scores often fail to capture models' real-world usefulness. Instead, users often rely on ``vibe-testing'': informal experience-based evaluation, such as comparing models on coding tasks related to their own workflow. While prevalent, vibe-testing is often too ad hoc and unstructured to analyze or reproduce at scale. In this work, we study how vibe-testing works in practice and then formalize it to support systematic analysis. We first analyze two empirical resources: (1) a survey of user evaluation practices, and (2) a collection of in-the-wild model comparison reports from blogs and social media. Based on these resources, we formalize vibe-testing as a two-part process: users personalize both what they test and how they judge responses. We then introduce a proof-of-concept evaluation pipeline that follows this formulation by generating personalized prompts and comparing model outputs using user-aware subjective criteria. In experiments on coding benchmarks, we find that combining personalized prompts and user-aware evaluation can change which model is preferred, reflecting the role of vibe-testing in practice. These findings suggest that formalized vibe-testing can serve as a useful approach for bridging benchmark scores and real-world experience.
Itay Itzhak, Eliya Habba, Gabriel Stanovsky +1
Technion – Israel Institute of Technology · The Hebrew University of Jerusalem