As Large Language Models (LLMs) have become capable of generating long and descriptive code summaries, accurate and reliable evaluation of factual consistency has become a critical challenge. However, previous evaluation methods are primarily designed for short summaries of isolated code snippets. Consequently, they struggle to provide fine-grained evaluation of multi-sentence functionalities and fail to accurately assess dependency context commonly found in real-world code summaries. To address this, we propose ReFEree, a reference-free and fine-grained method for evaluating factual consistency in real-world code summaries. We define factual inconsistency criteria specific to code summaries and evaluate them at the segment level using these criteria along with dependency information. These segment-level results are then aggregated into a fine-grained score. We construct a code summarization benchmark with human-annotated factual consistency labels. The evaluation results demonstrate that ReFEree achieves the highest correlation with human judgment among 13 baselines, improving 15-18% over the previous state-of-the-art. Our code and data are available at https://github.com/bsy99615/ReFEree.git.
Figures & tables
Figure 1: When evaluating factual consistency in real-world code summaries, existing methods ( bottom left ) fail to capture that the explanation of the ‘key_prefix’ in the summary exhibits a type error. In contrast, our method ( bottom right ) uses segment-level evaluation with related ‘key_prefix’ information, better aligning with human judgment.
Figure 2: Overview of ReFEree framework. Our evaluation process first performs code-related information searching to consider dependency relations in the input code. The summary is then segmented at the sentence level, and an LLM evaluates each segment according to the four factual inconsistency criteria to obtain segment-level scores. These scores are subsequently aggregated to compute the final consistency score.
Figure 3: Correlation between human judgment and the scores from each method. Each plot shows the relationship between scores obtained from each method (x-axis) and human labels (y-axis) , with red for Python and blue for Java. The gray dashed line represents perfect correlation. Since human labels range from 1 to 5, there are no values at y = 0.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Stability of ReFEree across summary lengths
Figure 5: Human evaluation interfaces: The upper image shows the summary-level human annotation interface, while the lower image shows the segment-level interface.
Metric
Explanation
Code
def convert_to_idn(url): parts = list(urllib.parse.urlsplit(url)) try: parts[1].encode(‘ascii’) except UnicodeEncodeError: host = parts[1].rsplit(‘:’, 1) newhost = [] port = ‘’ if len(host) == 2: port = host.pop() for h in host[0].split(‘.’): newhost.append(h.encode(‘idna’).decode(‘utf-8’)) parts[1] = ‘.’.join(newhost) if port: parts[1] += ‘:’ + port return urllib.parse.urlunsplit(parts) else: return url
Generated Summary
The ‘convert_to_idn‘ function is designed to convert the hostname portion of a URL into its Internationalized Domain Name (IDN) ASCII-compatible encoding using the Punycode standard. It first splits the URL into its components using the ‘urlparse‘ function, then attempts to encode the hostname in ASCII. If the hostname contains Unicode characters, it separates the hostname from the port if present, encodes each label of the hostname into IDNA format, and finally recombines the parts, preserving the port number. The function returns a list of URL components with the converted hostname. This utility is particularly useful in web crawlers and multilingual domain management systems where URLs need to be normalized for consistent processing and storage.
Human Label
0.6
ROUGE-L
0.1831
BERTScore
0.8149
SIDE
0.7039
Appendix
Table 20: Additional examples of ReFEree and existing methods: Human labels for the code and generated summary are normalized to values between 0 and 1 to allow comparison with other metrics. Factual inconsistencies between the project-level code and the generated summary are highlighted in red . C1–C4 represent the individual factual inconsistency criteria described in Table 9 .
The ‘system‘ function serves as a lightweight wrapper around the ‘subprocess‘ module to execute shell commands. It launches a new process with the specified command string, optionally sending input data via the standard input stream. Internally, it uses the ‘execute_command‘ function, which manages process creation and communication asynchronously, allowing for non-blocking interaction with the subprocess. The function captures both standard output and error streams, decodes the output using UTF-8 encoding, and returns it as a list of strings representing each line of the command’s output. This design makes it particularly suitable for integration into machine learning workflows where real-time command execution and output parsing are required.
Human Label
0.4
ROUGE-L
0.1898
BERTScore
0.8297
SIDE
0.9999
Appendix
Table 21: Additional examples of ReFEree and existing methods: Human labels for the code and generated summary are normalized to values between 0 and 1 to allow comparison with other metrics. Factual inconsistencies between the project-level code and the generated summary are highlighted in red . C1–C4 represent the individual factual inconsistency criteria described in Table 9 .
Reinforcement learning with evaluation metrics as rewards is widely used to enhance specific capabilities of language models. However, for tasks such as factually consistent summarisation, existing metrics remain underdeveloped, limiting their effectiveness as signals for shaping model behaviour.While individual factuality metrics are unreliable, their combination can more effectively capture diverse factual errors. We leverage this insight to introduce an automated training pipeline that improves factual consistency in summaries by aggregating scores from different weak metrics. Our approach avoids the need for complex reward shaping by mapping scores to preferences and filtering out cases with high disagreement between metrics. For each source document, we generate lexically similar summary pairs by varying decoding strategies, enabling the model to learn from factual differences caused by subtle lexical differences. This approach constructs a high-quality preference dataset using only source documents.Experiments demonstrate consistent factuality gains across models, ranging from early encoder-decoder architectures to modern large language models, with smaller models reaching comparable factuality to larger ones.
Yuxuan Ye, Raul Santos-Rodriguez, Edwin Simpson
Intelligent System Laboratory University of Bristol United Kingdom
Evaluating factual consistency is essential for reliable text summarization, particularly in high-stakes domains such as healthcare and news. However, most existing evaluation metrics overlook Bangla, a widely spoken yet under-resourced language, and often depend on reference summaries. We introduce BanglaSummEval, a reference-free, question-answering-based framework for evaluating factual consistency in Bangla summarization. The proposed method assesses both factual accuracy and content coverage through automatically generated questions and answers derived from the source document and the summary. A single multilingual instruction-tuned language model handles question generation, question answering, candidate answer extraction, and question importance weighting. This unified design reduces system complexity and computational cost. To capture semantic consistency beyond surface-level overlap, we use BERTScore-Recall for answer comparison. We validate BanglaSummEval on 300 human-written summaries from educational and medical domains, demonstrating strong correlation with expert human judgments (Pearson's r=0.694, Spearman's ρ=0.763). By providing interpretable, step-wise diagnostics alongside reliable evaluation scores, BanglaSummEval offers a practical and transparent solution for factual consistency evaluation in low-resource language settings.
Ahmed Rafid, Rumman Adib, Fariya Ahmed +2
Department of Computer Science and Engineering, Islamic University of Technology, Bangladesh · York University, Canada
Improving the quality of model-generated summaries, especially factuality, the accuracy of a summary with respect to its source content, remains a challenge. While reranking could select the optimal output from multiple generated candidates, it is limited to only using the source as guidance, resulting in unreliable summaries. To address this limitation, we propose ConSUM that reranks candidate summaries by considering two factors: consistency to the source document and consensus among the other candidates. Consensus is established using Minimum Bayes Risk (MBR) decoding over the set of generated summaries, while ensuring consistency by employing factuality-aware metrics that compare the summary against the source. Rigorous testing demonstrates that our system is competitive with existing methods, with human evaluations further confirming that its generated summaries are preferred over those from other systems. Our code is available at https://github.com/naist-nlp/ConSUM .
Riza Setiawan Soetedjo, Yusuke Sakai, Hidetaka Kamigaito +3
Nara Institute of Science and Technology (NAIST) · Chungnam National University · Institute of Science Tokyo