ReFEree: Reference-Free and Fine-Grained Method for Evaluating Factual Consistency in Real-World Code Summarization
Organizations: College of Computing and Informatics, Sungkyunkwan University, South Korea
Abstract
As Large Language Models (LLMs) have become capable of generating long and descriptive code summaries, accurate and reliable evaluation of factual consistency has become a critical challenge. However, previous evaluation methods are primarily designed for short summaries of isolated code snippets. Consequently, they struggle to provide fine-grained evaluation of multi-sentence functionalities and fail to accurately assess dependency context commonly found in real-world code summaries. To address this, we propose ReFEree, a reference-free and fine-grained method for evaluating factual consistency in real-world code summaries. We define factual inconsistency criteria specific to code summaries and evaluate them at the segment level using these criteria along with dependency information. These segment-level results are then aggregated into a fine-grained score. We construct a code summarization benchmark with human-annotated factual consistency labels. The evaluation results demonstrate that ReFEree achieves the highest correlation with human judgment among 13 baselines, improving 15-18% over the previous state-of-the-art. Our code and data are available at https://github.com/bsy99615/ReFEree.git.
Figures & tables
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Metric | Explanation |
|---|---|
| Code | def convert_to_idn(url): parts = list(urllib.parse.urlsplit(url)) try: parts[1].encode(‘ascii’) except UnicodeEncodeError: host = parts[1].rsplit(‘:’, 1) newhost = [] port = ‘’ if len(host) == 2: port = host.pop() for h in host[0].split(‘.’): newhost.append(h.encode(‘idna’).decode(‘utf-8’)) parts[1] = ‘.’.join(newhost) if port: parts[1] += ‘:’ + port return urllib.parse.urlunsplit(parts) else: return url |
| Generated Summary | The ‘convert_to_idn‘ function is designed to convert the hostname portion of a URL into its Internationalized Domain Name (IDN) ASCII-compatible encoding using the Punycode standard. It first splits the URL into its components using the ‘urlparse‘ function, then attempts to encode the hostname in ASCII. If the hostname contains Unicode characters, it separates the hostname from the port if present, encodes each label of the hostname into IDNA format, and finally recombines the parts, preserving the port number. The function returns a list of URL components with the converted hostname. This utility is particularly useful in web crawlers and multilingual domain management systems where URLs need to be normalized for consistent processing and storage. |
| Human Label | 0.6 |
| ROUGE-L | 0.1831 |
| BERTScore | 0.8149 |
| SIDE | 0.7039 |
| Metric | Explanation |
|---|---|
| Code | def system(cmd, data=None): import subprocess s = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stdin=subprocess.PIPE) out, err = s.communicate(data) return out.decode(’utf8’) |
| Generated Summary | The ‘system‘ function serves as a lightweight wrapper around the ‘subprocess‘ module to execute shell commands. It launches a new process with the specified command string, optionally sending input data via the standard input stream. Internally, it uses the ‘execute_command‘ function, which manages process creation and communication asynchronously, allowing for non-blocking interaction with the subprocess. The function captures both standard output and error streams, decodes the output using UTF-8 encoding, and returns it as a list of strings representing each line of the command’s output. This design makes it particularly suitable for integration into machine learning workflows where real-time command execution and output parsing are required. |
| Human Label | 0.4 |
| ROUGE-L | 0.1898 |
| BERTScore | 0.8297 |
| SIDE | 0.9999 |
Factual Consistency Evaluation
Factual consistency evaluation measures whether generated claims are accurate, supported by evidence, and consistent across documents or interactions.
See all 72 papers