This paper describes an approach and use case to estimate the correctness of LLM-generated COBOL summaries without access to human-authored references. The core intuition is that inter-model agreement on semantics is informative about truth, and that an LLM-as-judge can provide higher-fidelity pairwise assessments than traditional string- or embedding-based metrics. We operationalise this intuition with (i) controlled summary generation from diverse models; (ii) a LLM evaluator that compares pairs of summaries and emits structured judgments; and (iii) a calibrated aggregation rule that combines supportive evidence while penalising only corroborated contradictions.
Figures & tables
Figure 1. Overview of the ensemble-based confidence estimation pipeline. A COBOL code chunk and its dependencies are summarised by three models (Infosys, LLaMA 3.3–70B, Grok–3). Pairwise comparisons with an LLM-as-judge yield structured scores, which are then aggregated into a final confidence estimate.
Figure 2. Perplexity-to-confidence normalization (Equation ( 2 )). Summaries with PPL≤PPLmin saturate at maximum score 10 ; score decreases linearly to 0 at PPLmax and is clamped beyond. The plot shows: constant score of 10 for PPL≤1.2 ; linear decrease from 10 to 0 over interval [1.2,8.5] ; constant score of 0 for PPL≥8.5 .
Figure 3. Coverage assessment via G-Eval. The (chunk, summary) pair is scored k times ( τ>0 ) on a 1–5 rubric; scores are averaged and rescaled to [2,10] (Equation ( 4 )), then applied as multiplicative penalty to base confidence (Equation ( 5 )).
Confidence Range
Code Chunks
Acceptable
Review/Reject
High (Conf ≥7.0 )
17 (57%)
16 (94%)
1 (6%)
Medium ( 4.0≤ Conf <7.0 )
10 (33%)
6 (60%)
4 (40%)
Low (Conf <4.0 )
3 (10%)
0 (0%)
3 (100%)
Overall
30 (100%)
22 (73%)
8 (27%)
Table 1. Evaluation results on AWS CardDemo COBOL code chunks ( n=100 ). Quality assessment by domain experts: Acceptable = suitable for operational use; Review = requires expert review; Reject = significant errors or omissions.
As Large Language Models (LLMs) have become capable of generating long and descriptive code summaries, accurate and reliable evaluation of factual consistency has become a critical challenge. However, previous evaluation methods are primarily designed for short summaries of isolated code snippets. Consequently, they struggle to provide fine-grained evaluation of multi-sentence functionalities and fail to accurately assess dependency context commonly found in real-world code summaries. To address this, we propose ReFEree, a reference-free and fine-grained method for evaluating factual consistency in real-world code summaries. We define factual inconsistency criteria specific to code summaries and evaluate them at the segment level using these criteria along with dependency information. These segment-level results are then aggregated into a fine-grained score. We construct a code summarization benchmark with human-annotated factual consistency labels. The evaluation results demonstrate that ReFEree achieves the highest correlation with human judgment among 13 baselines, improving 15-18% over the previous state-of-the-art. Our code and data are available at https://github.com/bsy99615/ReFEree.git.
Suyoung Bae, CheolWon Na, Jaehoon Lee +3
College of Computing and Informatics, Sungkyunkwan University, South Korea
The progress of large language models (LLMs) has fueled claims that model-generated summaries rival or even surpass human-written references, raising questions about whether summarization remains an open research problem. We re-examine this narrative through a multi-track evaluation covering diverse datasets and state-of-the-art LLMs, combining controlled human assessment, bias-mitigated LLM-as-Judge protocols, factuality verification against external knowledge, and corpus-level linguistic analysis. Our findings reveal a more nuanced landscape in which human references continue to demonstrate advantages in informativeness and faithfulness, whereas LLM outputs are preferred mainly for surface-level coherence and fluency. Factuality verification indicates that human references remain more reliable, particularly for claims involving reasoning or synthesis, and linguistic analysis uncovers a pattern of stylistic homogeneity across different models. These observations suggest that current LLMs have raised the floor of summarization quality, but the ceiling of their performance remains below human capabilities.
Dongqi Liu, Chenxi Whitehouse, Zheng Zhao +3
Saarland University, Max Planck Institute for Informatics · Tencent YouTu Research · University of Cambridge +2
Large language models (LLMs) have become integral to modern software development, enabling automated code generation at scale. However, validating the correctness of LLM-generated code remains a critical and largely unsolved challenge. Existing approaches either rely on dynamic consensus across multiple code candidates - making them costly and difficult to scale - or on static reasoning that is susceptible to dynamic bugs and order bias. In this paper, we propose TRAILS~ (Targeted Reasoning Agreement via Inputs and Specifications), an approach that grounds LLM reasoning with concrete (input, output) pairs. TRAILS~ first generates diverse test inputs via category partitioning based on the specification, then executes them against the candidate code and prompts LLMs to assess whether the resulting input-output pairs conform to the specification - without ever reasoning over the code itself. Scores are aggregated across inputs, to determines whether the program is likely correct. We evaluate TRAILS~ on two datasets, LiveCodeBench and CoCoClaNeL, across three LLMs (Qwen3Coder-30B, Devstral-Small-24B, and Olmo3.1-Instruct), comparing against HoarePrompt and a Zero-Shot Chain-of-Thought baseline. TRAILS~ improves Matthew Correlation Coefficient by up to 39% relative to Zero-Shot COT and consistently outperforms HoarePrompt. Beyond accuracy, TRAILS~ demonstrates greater stability across seeded runs, reducing sensitivity to LLM non-determinism, and assigns correct labels to a larger set of unique code samples than competing approaches.