cs.AINov 6, 2026

Ensemble Methodology for Reference-Free Confidence Estimation in LLM-based Code Summarization

Authors: Marco Colombo, Balaji A Jayaram, Sowmyakumari Selvaraj, Sumit Tokkar, Animesh Kumar, Aravinda Prasad Sistla, Ravishankar Shetty, Rigel Gjomemo

Organizations: University of Illinois Urbana-Champaign · Infosys · University of Illinois Chicago

Abstract

This paper describes an approach and use case to estimate the correctness of LLM-generated COBOL summaries without access to human-authored references. The core intuition is that inter-model agreement on semantics is informative about truth, and that an LLM-as-judge can provide higher-fidelity pairwise assessments than traditional string- or embedding-based metrics. We operationalise this intuition with (i) controlled summary generation from diverse models; (ii) a LLM evaluator that compares pairs of summaries and emits structured judgments; and (iii) a calibrated aggregation rule that combines supportive evidence while penalising only corroborated contradictions.

Figures & tables

Explore similar work

CardsList
  1. ReFEree: Reference-Free and Fine-Grained Method for Evaluating Factual Consistency in Real-World Code Summarization

    Apr 12, 2026Suyoung Bae, CheolWon Na, Jaehoon Lee +3Factual Consistency EvaluationSummarization Evaluation

  2. Summarization is Not Dead Yet

    Jun 6, 2026Dongqi Liu, Chenxi Whitehouse, Zheng Zhao +3LLM EvaluationText Summarization

  3. Inferring Code Correctness from Specification

    May 28, 2026Tambon Florian, Papadakis MikeAutomated Software TestingCode Generation Evaluation