A Multi-Dimensional Quality Scoring Framework for Decentralized LLM Inference with Proof of Quality

Arther Tian; Alex Ding; Frank Chen; Simon Wu; Aaron Chan

cs.LGMar 4, 2026

A Multi-Dimensional Quality Scoring Framework for Decentralized LLM Inference with Proof of Quality

Authors: Arther Tian, Alex Ding, Frank Chen, Simon Wu, Aaron Chan

Organizations: DGrid AI

Abstract

Decentralized large language model (LLM) inference networks can pool heterogeneous compute to scale serving, but they require lightweight and incentive-compatible mechanisms to assess output quality. Prior work introduced cost-aware Proof of Quality (PoQ) and adaptive robust PoQ to allocate rewards under evaluator heterogeneity and adversarial behavior. In this paper, we focus on the quality signal itself and propose a multi-dimensional quality scoring framework that decomposes output quality into modular dimensions, including model and cost priors, structure quality, semantic quality, query-output alignment, and agreement/uncertainty. Using logged outputs from QA and summarization tasks, we systematically audit dimension reliability and show that seemingly reasonable dimensions can be task-dependent and even negatively correlated with reference quality without calibration. While the default composite underperforms a strong single semantic evaluator, ablations reveal that removing unreliable dimensions and re-normalizing weights yields a calibrated composite that matches or exceeds the best single- evaluator and consensus baselines. Finally, we integrate the composite score as a drop-in quality signal in PoQ and demonstrate complementary benefits with robust aggregation and adaptive trust weighting under adversarial evaluator attacks.

Figures & tables

Figure 1: Overview of the proposed multi-dimensional quality scoring framework and its integration into Proof of Quality (PoQ) for decentralized LLM inference. Candidate outputs are scored by multiple dimension modules and combined into a composite quality signal that can be used for consensus and rewards.

Symbol / Term	Meaning
$q$	User query / prompt
$i\in\mathcal{I}$	Inference node index
$y_{i}$	Candidate output produced by inference node $i$
$e\in\mathcal{E}$	Evaluator / metric index
$s_{e}(y_{i},q)$	Score from evaluator $e$ for output $y_{i}$ given query $q$
$\hat{s}(y_{i},q)$	Aggregated score used by consensus / rewards

Table 1: Notation and key components in decentralized inference with PoQ.

Property	Description
Alignment	Correlates with human/ground-truth preference on target tasks; avoids reward hacking.
Robustness	Stable under evaluator noise and resilient to adversarial evaluators or manipulations.
Interpretability	Dimensions explain why an output is scored highly/poorly, aiding audits and debugging.
Cost awareness	Supports heterogeneous evaluation cost/latency and enables efficient sampling or tiered evaluation.
Task adaptability	Allows task-specific weighting/calibration to mitigate metric mismatch across domains.
Composability	Integrates as a drop-in module with PoQ aggregation, trust weighting, and reward mechanisms.

Table 2: Desired properties of quality signals for decentralized inference and PoQ integration.

Panel of Figure 2 — Figure 2: Modular architecture of multi-dimensional quality scoring. Each dimension module produces a normalized score; the composite score $\hat{s}(q,y)$ is then used as a PoQ-compatible quality signal for aggregation and incentives.

Dimension	Instantiation (examples)	Default weight
Model prior	Preference-derived model ranking (Elo/TrueSkill style) [ 9 , 5 ]	0.15
Cost-efficiency prior	Quality-per-cost tendency under a serving budget (offline benchmark)	0.10
Structure quality	Length/format checks, repetition/degeneration signals	0.20
Semantic quality	Sentence embeddings / learned semantic metrics [ 19 , 8 , 25 ]	0.25
Query-output alignment	NLI/consistency style evaluators [ 12 , 10 ]	0.15
Agreement / uncertainty	Cross-evaluator dispersion / uncertainty proxy [ 24 , 7 ]	0.15

Table 3: Dimension definitions and typical instantiations used in our framework. Weights are default values used as a starting point; later sections perform ablations and task-wise calibration.

Component	Configuration
Tasks	QA; Summarization
Inference models	Multiple heterogeneous LLMs (quality/cost diversity)
Evaluators	Semantic metrics; learned metrics; NLI/consistency evaluators
Dimensions ( $K$ )	Priors; structure; semantics; alignment; agreement/uncertainty
Composite form	Weighted sum with normalization and clipping (Eq. )
PoQ simulation	Cost-aware sampling; robust aggregation; trust weighting [ 22 , 23 ]

Table 4: Experimental setup overview. Fill in concrete counts/models to match your run logs.

Module	Typical computation	Cost tier
Priors	Lookup / offline benchmark	Low
Structure quality	Heuristics over text	Low
Semantic quality	Embedding similarity [ 19 , 8 ]	Medium
Query-output alignment	NLI/consistency evaluator [ 12 ]	Medium–High
Agreement/uncertainty	Cross-evaluator dispersion	Depends on evaluator set

Table 5: Example cost/latency tiers for dimensions and evaluators. The purpose is to support cost-aware evaluation and PoQ sampling [ 22 ] . Replace “Low/Med/High” with measured values if available.

Figure 3: Unified correlation summary across individual evaluators, consensus methods, and the default composite score.

Method / Dimension	Pearson $\uparrow$	Spearman $\uparrow$
Composite (default weights)	0.513	0.682
Best single evaluator (sts_paraphrase)	0.754	0.806
Best consensus baseline (median over evaluators)	0.749	0.801
Semantic quality (dimension)	0.733	0.776
Structure quality (dimension)	0.466	0.601
Query-output alignment (dimension)	-0.437	-0.238

Table 6: Overall correlations with the reference signal (GT) on 2000 samples.

Figure 4: Correlation heatmap (GT) across evaluators, consensus baselines, composite, and dimensions.

Figure 5: Per-dimension correlation with GT. Semantic quality is strongly aligned overall, while alignment and agreement dimensions can be negatively correlated without calibration.

Signal (vs. GT)	QA	Summarization
Semantic quality	0.863 / 0.796	0.433 / 0.357
Query-output alignment	-0.571 / -0.493	0.102 / 0.058
Agreement / uncertainty	-0.553 / -0.401	0.188 / 0.239
Composite (default)	0.742 / 0.720	0.443 / 0.408
Composite (calibrated)	0.893 / 0.795	0.438 / 0.408

Table 7: Task-wise correlations (Pearson / Spearman) with GT. “Composite (calibrated)” removes the alignment and agreement dimensions and re-normalizes the remaining weights.

Figure 6: Dimension heatmap visualization for diagnosing dimension behavior across settings (e.g., models/tasks).

Figure 7: Radar chart view of multi-dimensional scores for qualitative comparison across outputs or model groups.

Figure 8: Composite score versus the best single evaluator baseline. Despite using multiple signals, the default composite can underperform the strongest semantic evaluator without calibration.

Figure 9: Composite and single-evaluator correlations against GT. This view highlights when composites help and when they degrade alignment relative to individual evaluators.

Figure 10: Weight ablation and dimension removal results. Removing unreliable dimensions can substantially improve alignment.

Variant	Pearson $\uparrow$	Spearman $\uparrow$
Default	0.513	0.682
Equal weights	0.447	0.643
No priors	0.341	0.543
Prior-heavy	0.502	0.613
Semantic only	0.733	0.776
No structure	0.457	0.619

Table 8: Ablation variants and correlations with GT. “Calibrated (remove alignment+agreement)” is computed by removing those dimensions and re-normalizing remaining weights.

Figure 11: PoQ reward outcomes under baseline quality signals. The reward landscape reflects the alignment properties of the quality signal used for consensus and incentives.

Figure 12: Adaptive trust weighting behavior in PoQ. Trust-weight updates help suppress unreliable evaluators and complement multi-dimensional scoring.

Component	Settings
Quality signal	Single evaluator; consensus baseline; composite (default / calibrated)
Aggregation defenses	Mean; Median; Trimmed mean; Adaptive trust weighting [ 23 ]
Attack types	Multiple malicious evaluator strategies (see Fig. 13 )
Attack ratios	Multiple attacker fractions (swept)
Evaluation budget	Cost-aware sampling / fixed budget variants [ 22 ]
Simulation	Monte Carlo rounds per configuration

Table 9: PoQ integration study configurations. We evaluate the composite score under a grid of attack types, attack ratios, and defense mechanisms.

Figure 13: Defense summary in PoQ integration experiments. Robust aggregation and adaptive trust-weighting aim to reduce malicious evaluator influence.

Figure 14: Adversarial defense comparison across attacker ratios and defense mechanisms. Calibrated composite scoring and robust PoQ defenses can be complementary.

Language Model Generation Evaluation

Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.

See all 395 papers

PoQ-Judge: A Multi-Architecture Evaluation Framework for Cost-Aware Proof-of-Quality in Decentralized LLM Inference
Apr 20, 2026Arther Tian, Alex Ding, Frank Chen +2LLM Evaluation LLM-as-a-Judge
DGrid AI
A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Quality
Sep 16, 2026Jerry KaplanLLM Quantization LLM Evaluation
Adjunct Lecturer, Computer Science and Masters in International Policy, Stanford University
(Towards) Scalable Reliable Automated Evaluation with Large Language Models
Jul 30, 2026Bertil Braun, Martin ForellLLM Evaluation Pairwise Comparison
KIT

Apr 20, 2026J/K move · Enter open · S save

PoQ-Judge: A Multi-Architecture Evaluation Framework for Cost-Aware Proof-of-Quality in Decentralized LLM Inference

Arther Tian, Alex Ding, Frank Chen, Simon Wu+1

DGrid AI

LLM Evaluation LLM-as-a-Judge LLM Inference Cost-Aware Inference

Open paper AskPDF arXiv

Decentralized LLM inference networks need lightweight, reference-free quality evaluation for Proof of Quality (PoQ). We present PoQ-Judge, a framework that trains dedicated judge models to score query-output pairs without ground-truth references. We study three architectures across the quality-cost tradeoff: a TextCNN judge, a MiniLM cross-encoder, and a DeBERTa judge. Using two-stage training on UltraFeedback plus GPT-labeled in-domain data, the best model reaches 0.747 Pearson correlation with the ground-truth proxy on a held-out test set, outperforming reference-based evaluators from prior work. As a reference-free component in composite scoring, it achieves 0.645 Pearson correlation, matching the best single reference-based evaluator while removing the need for reference answers. We also show that online calibration identifies semantic quality as the dominant dimension and that cascade evaluation reduces cost by 72.7 percent with only modest quality loss. Results are much stronger on QA than summarization, pointing to proxy quality as the main remaining limitation.

Abstract

Language Model Generation Evaluation

Explore similar work

PoQ-Judge: A Multi-Architecture Evaluation Framework for Cost-Aware Proof-of-Quality in Decentralized LLM Inference

A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Quality

(Towards) Scalable Reliable Automated Evaluation with Large Language Models