Decentralized large language model (LLM) inference networks can pool heterogeneous compute to scale serving, but they require lightweight and incentive-compatible mechanisms to assess output quality. Prior work introduced cost-aware Proof of Quality (PoQ) and adaptive robust PoQ to allocate rewards under evaluator heterogeneity and adversarial behavior. In this paper, we focus on the quality signal itself and propose a multi-dimensional quality scoring framework that decomposes output quality into modular dimensions, including model and cost priors, structure quality, semantic quality, query-output alignment, and agreement/uncertainty. Using logged outputs from QA and summarization tasks, we systematically audit dimension reliability and show that seemingly reasonable dimensions can be task-dependent and even negatively correlated with reference quality without calibration. While the default composite underperforms a strong single semantic evaluator, ablations reveal that removing unreliable dimensions and re-normalizing weights yields a calibrated composite that matches or exceeds the best single- evaluator and consensus baselines. Finally, we integrate the composite score as a drop-in quality signal in PoQ and demonstrate complementary benefits with robust aggregation and adaptive trust weighting under adversarial evaluator attacks.
Figures & tables
Figure 1: Overview of the proposed multi-dimensional quality scoring framework and its integration into Proof of Quality (PoQ) for decentralized LLM inference. Candidate outputs are scored by multiple dimension modules and combined into a composite quality signal that can be used for consensus and rewards.
Symbol / Term
Meaning
q
User query / prompt
i∈I
Inference node index
yi
Candidate output produced by inference node i
e∈E
Evaluator / metric index
se(yi,q)
Score from evaluator e for output yi given query q
s^(yi,q)
Aggregated score used by consensus / rewards
Table 1: Notation and key components in decentralized inference with PoQ.
Property
Description
Alignment
Correlates with human/ground-truth preference on target tasks; avoids reward hacking.
Robustness
Stable under evaluator noise and resilient to adversarial evaluators or manipulations.
Interpretability
Dimensions explain why an output is scored highly/poorly, aiding audits and debugging.
Cost awareness
Supports heterogeneous evaluation cost/latency and enables efficient sampling or tiered evaluation.
Task adaptability
Allows task-specific weighting/calibration to mitigate metric mismatch across domains.
Composability
Integrates as a drop-in module with PoQ aggregation, trust weighting, and reward mechanisms.
Table 2: Desired properties of quality signals for decentralized inference and PoQ integration.
Figure 2: Modular architecture of multi-dimensional quality scoring. Each dimension module produces a normalized score; the composite score s^(q,y) is then used as a PoQ-compatible quality signal for aggregation and incentives.
Dimension
Instantiation (examples)
Default weight
Model prior
Preference-derived model ranking (Elo/TrueSkill style) [ 9 , 5 ]
0.15
Cost-efficiency prior
Quality-per-cost tendency under a serving budget (offline benchmark)
Table 3: Dimension definitions and typical instantiations used in our framework. Weights are default values used as a starting point; later sections perform ablations and task-wise calibration.
Table 4: Experimental setup overview. Fill in concrete counts/models to match your run logs.
Module
Typical computation
Cost tier
Priors
Lookup / offline benchmark
Low
Structure quality
Heuristics over text
Low
Semantic quality
Embedding similarity [ 19 , 8 ]
Medium
Query-output alignment
NLI/consistency evaluator [ 12 ]
Medium–High
Agreement/uncertainty
Cross-evaluator dispersion
Depends on evaluator set
Table 5: Example cost/latency tiers for dimensions and evaluators. The purpose is to support cost-aware evaluation and PoQ sampling [ 22 ] . Replace “Low/Med/High” with measured values if available.
Figure 3: Unified correlation summary across individual evaluators, consensus methods, and the default composite score.
Method / Dimension
Pearson ↑
Spearman ↑
Composite (default weights)
0.513
0.682
Best single evaluator (sts_paraphrase)
0.754
0.806
Best consensus baseline (median over evaluators)
0.749
0.801
Semantic quality (dimension)
0.733
0.776
Structure quality (dimension)
0.466
0.601
Query-output alignment (dimension)
-0.437
-0.238
Table 6: Overall correlations with the reference signal (GT) on 2000 samples.
Figure 4: Correlation heatmap (GT) across evaluators, consensus baselines, composite, and dimensions.
Figure 5: Per-dimension correlation with GT. Semantic quality is strongly aligned overall, while alignment and agreement dimensions can be negatively correlated without calibration.
Signal (vs. GT)
QA
Summarization
Semantic quality
0.863 / 0.796
0.433 / 0.357
Query-output alignment
-0.571 / -0.493
0.102 / 0.058
Agreement / uncertainty
-0.553 / -0.401
0.188 / 0.239
Composite (default)
0.742 / 0.720
0.443 / 0.408
Composite (calibrated)
0.893 / 0.795
0.438 / 0.408
Table 7: Task-wise correlations (Pearson / Spearman) with GT. “Composite (calibrated)” removes the alignment and agreement dimensions and re-normalizes the remaining weights.
Figure 6: Dimension heatmap visualization for diagnosing dimension behavior across settings (e.g., models/tasks).
Figure 7: Radar chart view of multi-dimensional scores for qualitative comparison across outputs or model groups.
Figure 8: Composite score versus the best single evaluator baseline. Despite using multiple signals, the default composite can underperform the strongest semantic evaluator without calibration.
Figure 9: Composite and single-evaluator correlations against GT. This view highlights when composites help and when they degrade alignment relative to individual evaluators.
Figure 10: Weight ablation and dimension removal results. Removing unreliable dimensions can substantially improve alignment.
Variant
Pearson ↑
Spearman ↑
Default
0.513
0.682
Equal weights
0.447
0.643
No priors
0.341
0.543
Prior-heavy
0.502
0.613
Semantic only
0.733
0.776
No structure
0.457
0.619
Table 8: Ablation variants and correlations with GT. “Calibrated (remove alignment+agreement)” is computed by removing those dimensions and re-normalizing remaining weights.
Figure 11: PoQ reward outcomes under baseline quality signals. The reward landscape reflects the alignment properties of the quality signal used for consensus and incentives.
Figure 12: Adaptive trust weighting behavior in PoQ. Trust-weight updates help suppress unreliable evaluators and complement multi-dimensional scoring.
Component
Settings
Quality signal
Single evaluator; consensus baseline; composite (default / calibrated)
Table 9: PoQ integration study configurations. We evaluate the composite score under a grid of attack types, attack ratios, and defense mechanisms.
Figure 13: Defense summary in PoQ integration experiments. Robust aggregation and adaptive trust-weighting aim to reduce malicious evaluator influence.
Figure 14: Adversarial defense comparison across attacker ratios and defense mechanisms. Calibrated composite scoring and robust PoQ defenses can be complementary.
Decentralized LLM inference networks need lightweight, reference-free quality evaluation for Proof of Quality (PoQ). We present PoQ-Judge, a framework that trains dedicated judge models to score query-output pairs without ground-truth references. We study three architectures across the quality-cost tradeoff: a TextCNN judge, a MiniLM cross-encoder, and a DeBERTa judge. Using two-stage training on UltraFeedback plus GPT-labeled in-domain data, the best model reaches 0.747 Pearson correlation with the ground-truth proxy on a held-out test set, outperforming reference-based evaluators from prior work. As a reference-free component in composite scoring, it achieves 0.645 Pearson correlation, matching the best single reference-based evaluator while removing the need for reference answers. We also show that online calibration identifies semantic quality as the dominant dimension and that cascade evaluation reduces cost by 72.7 percent with only modest quality loss. Results are much stronger on QA than summarization, pointing to proxy quality as the main remaining limitation.
Large language model optimization is an active research area, spanning quantization of model weights, early-exit methods for skipping layers, and speculative decoding. Each track uses its own quality measures, typically an idiosyncratic benchmark score. Few approach the measurement precision required by other scientific disciplines. We propose a rigorous methodology for measuring output quality, suitable for cross-system and cross-technique comparison. We score outputs with an LLM as a judge, but calibrate the judge formally: we compare its scores on two ordinary runs of a model given the same prompts, verifying that it shows no systematic preference between statistically equivalent outputs and measuring its per-sample noise. Each design also includes a 'null' condition, provably identical in distribution to the unmodified model, whose measured difference must be zero. With this one instrument we measure several acceleration techniques on the same prompts, so their quality costs can be compared. Perceived quality proves highly dependent on the domain of discourse. A 4-bit model was indistinguishable from its 16-bit original down to our design's +/-0.3-point resolution, in English prose and Chinese alike. At 3-bit precision the same prompts lost 0.5 points in English prose, 0.9 in Chinese, and 1.1 on multi-step math; early exit that cost 0.7 points on prose cost 2.5 on math, cutting correctly solved problems from 19 of 27 to 6. The pattern held for models from Alibaba and from Meta, but not its magnitude: the same quantizer cost Meta's model 1.8 points where it cost Alibaba's 0.7. A model's certainty about a token predicts how likely it is to differ from the full model's choice, but not how much that difference affects judged quality, so acceptance rules relying on certainty cannot distinguish errors that matter from errors that don't.
Jerry Kaplan
Adjunct Lecturer, Computer Science and Masters in International Policy, Stanford University
Evaluating the quality and relevance of textual outputs from Large Language Models (LLMs) remains challenging and resource-intensive. Existing automated metrics often fail to capture the complexity and variability inherent in LLM-generated outputs. Moreover, these metrics typically rely on explicit reference standards, limiting their use mostly to domains with objective benchmarks. This work introduces a novel evaluation framework designed to approximate expert-level assessments of LLM-generated content. The proposed method employs pairwise comparisons of outputs by multiple LLMs, reducing biases from individual models. An Elo rating system is used to generate stable and interpretable rankings. Adjustable agreement thresholds, from full unanimity to majority voting, allow flexible control over evaluation confidence and coverage. The method's effectiveness is demonstrated through evaluating competency profiles extracted from scientific abstracts. Preliminary results show that automatically derived rankings correlate well with expert judgments, significantly reducing the need for extensive human intervention. By offering a scalable, consistent, and domain-agnostic evaluation layer, the framework supports more efficient and reliable quality assessments of LLM outputs across diverse applications.