This paper describes an approach and use case to estimate the correctness of LLM-generated COBOL summaries without access to human-authored references. The core intuition is that inter-model agreement on semantics is informative about truth, and that an LLM-as-judge can provide higher-fidelity pairwise assessments than traditional string- or embedding-based metrics. We operationalise this intuition with (i) controlled summary generation from diverse models; (ii) a LLM evaluator that compares pairs of summaries and emits structured judgments; and (iii) a calibrated aggregation rule that combines supportive evidence while penalising only corroborated contradictions.
Figures & tables
Figure 1. Overview of the ensemble-based confidence estimation pipeline. A COBOL code chunk and its dependencies are summarised by three models (Infosys, LLaMA 3.3–70B, Grok–3). Pairwise comparisons with an LLM-as-judge yield structured scores, which are then aggregated into a final confidence estimate.
Figure 2. Perplexity-to-confidence normalization (Equation ( 2 )). Summaries with PPL≤PPLmin saturate at maximum score 10 ; score decreases linearly to 0 at PPLmax and is clamped beyond. The plot shows: constant score of 10 for PPL≤1.2 ; linear decrease from 10 to 0 over interval [1.2,8.5] ; constant score of 0 for PPL≥8.5 .
Figure 3. Coverage assessment via G-Eval. The (chunk, summary) pair is scored k times ( τ>0 ) on a 1–5 rubric; scores are averaged and rescaled to [2,10] (Equation ( 4 )), then applied as multiplicative penalty to base confidence (Equation ( 5 )).
Confidence Range
Code Chunks
Acceptable
Review/Reject
High (Conf ≥7.0 )
17 (57%)
16 (94%)
1 (6%)
Medium ( 4.0≤ Conf <7.0 )
10 (33%)
6 (60%)
4 (40%)
Low (Conf <4.0 )
3 (10%)
0 (0%)
3 (100%)
Overall
30 (100%)
22 (73%)
8 (27%)
Table 1. Evaluation results on AWS CardDemo COBOL code chunks ( n=100 ). Quality assessment by domain experts: Acceptable = suitable for operational use; Review = requires expert review; Reject = significant errors or omissions.