Comprehensive Evaluation of Large Language Model Responses: A Multi-Factor Scoring System

Yiming Gai; Junde Lu; Xuefei Huang

cs.CLJul 8, 2026

Comprehensive Evaluation of Large Language Model Responses: A Multi-Factor Scoring System

Authors: Yiming Gai, Junde Lu, Xuefei Huang

Organizations: School of Computer Science and Engineering, Beihang University, Beijing, China · Data Science and Intelligent Computing Laboratory, Hangzhou International Innovation Institute, Beihang University, Hangzhou, Zhejiang 311115, P.R.China

Abstract

The remarkable performance of large language models (LLMs) in linguistic tasks underscores an urgent need for comprehensive evaluation of their response quality. Prevailing methods, often confined to singular dimensions, fall short of capturing the full spectrum of model capabilities. This study introduces a multifactor scoring paradigm, integrating accuracy, conciseness, factual consistency, readability, and coherence, complemented by a graphical user interface (GUI) for visualizing outcomes. Evaluations on the TruthfulQA dataset unveil mainstream LLMs' strengths in reasoning tasks (peaking at a composite score of 0.6104) alongside pervasive limitations in navigating complex facts and ambiguities. Transcending the narrow lens of traditional metrics, this framework offers a transparent, adaptable avenue to illuminate model potential and deficiencies. Though presently focused on English tasks, its horizons beckon toward multilingual domains. This work carves a novel path for knowledge engineering and model refinement.

Explore similar work

A Consensus-Based Framework for Relative Preference Evaluation of Large Language Models
Jul 19, 2026Mohtashim KhanLarge Language Model Evaluation Preference Alignment
Evaluation of Contextual Understanding in Large Language Models
Sep 8, 2026Subavarshana Arumugam, Mamta Nallaretnam, Kithuni Wickramasinghe +4Contextual Semantic Similarity Metrics
(Towards) Scalable Reliable Automated Evaluation with Large Language Models
Jul 30, 2026Bertil Braun, Martin ForellLarge Language Model Evaluation Competency Questions

Jul 19, 2026 · cs.CLJ/K move · Enter open · S save

A Consensus-Based Framework for Relative Preference Evaluation of Large Language Models

Mohtashim Khan

Traditional benchmarks for LLMs primarily rely on static datasets and objective scoring metrics, which often fail to capture differences in response quality when multiple answers are acceptable. In such settings, correctness alone is insufficient to distinguish between responses that vary in clarity, completeness, and usefulness. This paper introduces a consensus-based evaluation framework that measures relative preference among model-generated responses rather than absolute correctness. Instead of evaluating outputs against a fixed ground truth, we assess how a panel of diverse LLMs ranks anonymized candidate responses to the same prompt. This approach treats aggregate inter-model agreement as a proxy for perceived response quality under blind conditions. We conduct a controlled study using five state-of-the-art LLMs across multiple domains, including programming, general knowledge, safety, logical reasoning, and mathematics. Each model generates responses and independently ranks peer outputs through a structured voting process. Scores are aggregated into a Relative Intelligence Index (RII), representing how frequently a model's responses are preferred by other models. Our findings reveal consistent preference patterns across domains, with certain models more frequently ranked highly by their peers. However, we emphasize that these results reflect inter-model preference alignment rather than objective correctness or human judgment. This framework provides a scalable, model-driven method for comparative evaluation, offering an alternative perspective on response quality in scenarios where multiple valid answers exist. While not directly aligned with human evaluation, prior work suggests that aggregated model preferences can partially correlate with human judgments, motivating this as a proxy signal.

Large Language Model Evaluation Preference Alignment Responses Consensus

Open paper Ask about it PDF

Abstract

Explore similar work

A Consensus-Based Framework for Relative Preference Evaluation of Large Language Models

Evaluation of Contextual Understanding in Large Language Models

(Towards) Scalable Reliable Automated Evaluation with Large Language Models