UniRank: A Multi-Agent Calibration Pipeline for Estimating University Rankings from Anonymized Bibliometric Signals

Pedram Riyazimehr; Seyyed Ehsan Mahmoudi

cs.SIFeb 21, 2026

UniRank: A Multi-Agent Calibration Pipeline for Estimating University Rankings from Anonymized Bibliometric Signals

Authors: Pedram Riyazimehr, Seyyed Ehsan Mahmoudi

Organizations: NotionWave

Abstract

We present UniRank, a multi-agent LLM pipeline that estimates university positions across global ranking systems using only publicly available bibliometric data from OpenAlex and Semantic Scholar. The system employs a three-stage architecture: (a) zero-shot estimation from anonymized institutional metrics, (b) per-system tool-augmented calibration against real ranked universities, and (c) final synthesis. Critically, institutions are anonymized -- names, countries, DOIs, paper titles, and collaboration countries are all redacted -- and their actual ranks are hidden from the calibration tools during evaluation, preventing LLM memorization from confounding results. On the Times Higher Education (THE) World University Rankings ( $n=352$ ), the system achieves MAE = 251.5 rank positions, Median AE = 131.5, PNMAE = 12.03%, Spearman $ρ= 0.769$ , Kendall $τ= 0.591$ , hit rate @50 = 20.7%, hit rate @100 = 39.8%, and a Memorization Index of exactly zero (no exact-match zero-width predictions among all 352 universities). The systematic positive-signed error (+190.1 positions, indicating the system consistently predicts worse ranks than actual) and monotonic performance degradation from elite tier (MAE = 60.5, hit@100 = 90.5%) to tail tier (MAE = 328.2, hit@100 = 20.8%) provide strong evidence that the pipeline performs genuine analytical reasoning rather than recalling memorized rankings. A live demo is available at https://unirank.scinito.ai .

Figures & tables

Figure 1: UniRank system architecture. Data from OpenAlex and Semantic Scholar is aggregated, normalized, and anonymized before entering the three-stage LLM pipeline. During evaluation, the target university is hidden from the ranking store (dashed line) to prevent data leakage.

Metric	Source	Description
worksCount	OA	Total publications
citedByCount	OA	Total citations
hIndex	OA	Hirsch index
i10Index	OA	Papers with 10+ citations
2yr mean citedness	OA	FWCI proxy
citationsPerWork	OA	Average citations/paper

Table 1: Bibliometric feature set (16 indicators).

Figure 2: Three-stage pipeline: Stage 1 produces coarse zero-shot estimates from anonymized metrics. Stage 2 refines per-system with tool-augmented calibration (parallel). Stage 3 synthesizes the final report.

Figure 3: Anonymization before/after comparison. All identifying information (name, country, DOIs, paper titles, collaboration countries) is redacted; only numeric metrics are preserved.

Metric	Value
MAE	251.5
Median AE	131.5
RMSE	411.4
PNMAE	12.03%
Signed Error	$+190.1$
Spearman’s $\rho$	0.769

Table 2: Aggregate evaluation metrics (THE,

n{=}352

Tier	N	MAE	Med.	@50	@100	SE
Elite	21	60.5	33.0	57.1	90.5	$+$ 60.2
Strong	80	211.6	93.0	30.0	53.8	$+$ 205.8
Mid	99	247.8	122.5	18.2	38.4	$+$ 214.0
Lower	99	287.0	192.0	11.1	29.3	$+$ 198.5
Tail	53	328.2	309.0	15.1	20.8	$+$ 157.2

Table 3: Per-tier performance breakdown (THE).

Figure 4: Error distribution by evaluation tier (THE). Box plots show median (line), IQR (box), and outliers. Performance degrades monotonically from elite to tail.

Figure 5: Hit rate comparison across tiers at three thresholds (@25, @50, @100). Elite tier achieves 90.5% hit@100.

	Predicted
Actual	Elite	Strong	Mid	Lower	Tail
Elite	4	15	2	0	0
Strong	0	36	32	6	6
Mid	0	6	49	26	18
Lower	0	1	13	40	45
Tail	0	0	2	5	46

Table 4: Tier confusion matrix (actual

\times

predicted).

ID	Mode	Description
F1	Reputation	Rank boosted by reputation/teaching signals unobservable from bibliometrics
F2	Teaching	Rank driven by teaching metrics (student–staff ratio, doctoral ratio)
F3	Scale	Small-but-excellent or large-but-average confusion
F4	Regional	Systematic regional over/under-estimation
F5	Anchor Drift	Calibration anchors on unrepresentative comparators
F6	Data Gap	OpenAlex provides fragmented or incorrect institutional data

Table 5: Failure mode definitions (left) and prevalence in top-20 worst predictions (right).

Test	Stat.	$p$	Interpretation
Wilcoxon	$W{=}27{,}921$	0.343	Not significant
McNemar	$\chi^{2}{=}1.786$	0.181	Not significant
Kruskal–Wallis	$H{=}42.09$	$10^{-8}$	Tiers differ ∗∗∗

Table 6: Statistical test results (initial vs. calibrated).

Metric	Initial	Calibrated	$\Delta$
MAE	256.8	251.5	$-2.1\%$
Median AE	149.5	130.8	$-12.5\%$
Spearman $\rho$	0.770	0.769	$-0.001$
Hit @50	17.9%	20.7%	$+2.8$ pp
Hit @100	36.1%	39.8%	$+3.7$ pp
Improved	191/352 (54.3%)

Table 7: Calibration impact summary.

Figure 7: Initial vs. calibrated absolute error. Points below the diagonal indicate calibration helped. Calibration improved 54.3% of predictions overall.

Figure 8: Data completeness radar. Green = what UniRank can measure; gray = what THE weights. The gap between series—particularly on Teaching (15%), Reputation (18%), and Industry (4%)—explains the systematic positive signed error.

Missing Data	Impact	Systems
Reputation surveys	30% QS, 18% THE	QS, THE
Employer surveys	15% QS	QS
Student–staff ratios	5% QS, 15% THE	QS, THE
Intl. student/faculty	10% QS	QS
Nobel/Fields winners	30% ARWU	ARWU
Industry income	4% THE	THE

Table 8: Missing data dimensions and ranking impact.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Detail
Model	GPT-5.2 (via @ai-sdk/openai , Helicone gateway)
Reasoning effort	medium
Temperature	Default (model-determined)
Max calibration steps	12 per system
Structured output	Zod schemas via Vercel AI SDK
Eval concurrency	3 parallel evaluations

Appendix

Table 17

Metric	Tier 1 (Top 10)	Tier 2 (Top 100)	Tier 3 (Top 300)	Tier 4 (Top 500)
h-index	2135 $\pm$ 291	1196 $\pm$ 205	787 $\pm$ 145	560 $\pm$ 102
2yr citedness	4.60 $\pm$ 1.10	3.40 $\pm$ 0.70	2.95 $\pm$ 0.60	2.85 $\pm$ 0.77
Research excellence (%)	19.5 $\pm$ 3.5	20.5 $\pm$ 2.1	20.0 $\pm$ 2.8	19.5 $\pm$ 3.5
Intl. collab (%)	40.5 $\pm$ 9.2	40.5 $\pm$ 12.0	39.0 $\pm$ 12.7	38.0 $\pm$ 12.7

Appendix

Table 9: Tier benchmarks (mean

\pm

SD) for key metrics.

Figure 9: Tier benchmarks for key bibliometric indicators. Error bars show $\pm 1$ SD.

#	University	Actual	Pred.	AE	Tier
1	Northwestern University	30	30	0	strong
2	Tianjin University	208	208	1	mid
3	Univ. of Kwazulu-Natal	566	565	1	lower
4	Hong Kong Baptist Univ.	214	215	1	mid
5	Vrije Univ. Brussel	234	235	1	mid
6	Univ. of Alberta	119	120	1	strong

Appendix

Table 10: Full evaluation results (THE,

n{=}352

), sorted by AE ascending.

Figure 10: Error heatmap (tier $\times$ region). Cell values show mean AE. Darker cells indicate higher error. Cells with $N<5$ are marked with an asterisk.

Higher Education

See all 15 papers

Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference
Jun 3, 2026Abhishek DivekarLLM-as-a-Judge IR Evaluation
Amazon
The Model Knows Your Project, Not You: Measuring Recognition in LLMs with NameRank
Jul 14, 2026Bojie Li, Noah ShiLLM Evaluation Factual Knowledge in Language Models
Pine AI · University of Washington
Ranked by Position: Order Sensitivity as an Exploitable Attack Surface in LLM Listwise Recommenders
Jul 26, 2026Ge Zhang, Jingru Cheng, Huiyuan ChenLearning to Rank Adversarial Attacks
Stanford University · Independent Researcher

Jun 3, 2026J/K move · Enter open · S save

Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference

Abhishek Divekar

Amazon

LLM-as-a-Judge IR Evaluation Prediction-Powered Inference

Open paper AskPDF arXiv

With PRECISE, we extended Prediction-Powered Inference to produce bias-corrected estimates of ranking evaluation metrics by combining a small human-labeled set with a large LLM-judged set. PPI is provably unbiased regardless of the LLM judge's error profile. We make it applicable to hierarchical metrics like Precision@K, where annotations are per-document but the metric is per-query, by reducing the output-space computation from O(2^|C|) to O(2^K). On the ESCI benchmark, augmenting 30 human annotations with Claude 3 Sonnet judgments reduces the standard error of Precision@4 estimates from 4.45 to 3.50 (a 21% relative reduction). In a production system, our framework correctly identified the best of three system variants from 100 human labels and 2 hours of domain-expert annotation; A/B testing confirmed this ranking with +407 bps in daily sales.

Abstract

Higher Education

Explore similar work

Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference

The Model Knows Your Project, Not You: Measuring Recognition in LLMs with NameRank

Ranked by Position: Order Sensitivity as an Exploitable Attack Surface in LLM Listwise Recommenders