Crystal: Characterizing Relative Impact of Scholarly Publications

Hannah Collison; Benjamin Van Durme; Daniel Khashabi

cs.DLMar 25, 2026

Crystal: Characterizing Relative Impact of Scholarly Publications

Authors: Hannah Collison, Benjamin Van Durme, Daniel Khashabi

Organizations: Johns Hopkins University

Abstract

Assessing a cited paper's impact is typically done by analyzing its citation context in isolation within the citing paper. While this focuses on the most directly relevant text, it prevents relative comparisons across all the works a paper cites. We propose Crystal, which instead jointly ranks all cited papers within a citing paper using large language models (LLMs). To mitigate LLMs' positional bias, we rank each list three times in a randomized order and aggregate the impact labels through majority voting. This joint approach leverages the full citation context, rather than evaluating citations independently, to more reliably distinguish impactful references. Crystal outperforms a prior state-of-the-art impact classifier by +9.5% accuracy and +8.3% F1 on a dataset of human-annotated citations. Crystal further gains efficiency through fewer LLM calls and outperforms prior baselines using an open-weight model, enabling scalable, cost-effective citation impact analysis. In a case study of ACL Test-of-Time award-winning papers, we find that Crystal's impact characterizations align closely with long-term scientific recognition. We release Crystal-Bank, a 46.8k-paper dataset with rankings and impact labels, along with code.

Figures & tables

Figure 1: A citing paper $p$ (left) references papers $\{A,B,C,D\}$ (right), with each arrow denoting a citation edge $(p\rightarrow\tilde{p})$ for $\tilde{p}\in\{A,B,C,D\}$ . Crystal considers all cited papers jointly to assess the relative impact $\tilde{p}$ had on $p$ , unlike prior work that considers only $p$ and $\tilde{p}$ in isolation, evaluating impact independently .

Figure 2: Overview of Crystal, which characterizes the relative impact of a target paper $p^{*}$ on each of its citing papers, e.g. $q_{1}$ . 1 Identify all citing papers of $p^{*}$ ( $q_{1}$ , $q_{2}$ , $q_{3}$ ). 2 For each citing paper, rank its references by impact (e.g., $p_{1}$ , $p_{2}$ , $p^{*}$ for $q_{1}$ ). 3 The result is an impact-based ranking of $q_{1}$ ’s references. See § 4 for details.

Model	Method	Accuracy( $\uparrow$ )	Bal. Acc.( $\uparrow$ )	MCC( $\uparrow$ )	P( $\uparrow$ )	R( $\uparrow$ )	F1( $\uparrow$ )	F1 $-$ ( $\uparrow$ )
Baselines	Random	$49.9{\scriptstyle\pm 1.2}$	50.2	0.00	33.1	50.9	40.1	57.0
	All Imp-Rev	$32.9{\scriptstyle\pm 1.1}$	50.0	0.00	32.9	100.0	49.5	0.0
	All Non-Imp	$67.1{\scriptstyle\pm 1.1}$	50.0	0.00	0.0	0.0	0.0	80.3
	Valenzuela	$70.6{\scriptstyle\pm 1.1}$	67.1	0.35	58.0	53.2	55.5	79.5
Gpt-5.1	UKP	$66.7{\scriptstyle\pm 1.1}$	65.9	0.30	49.6	63.5	55.7	73.4
	Crystal	$78.6{\scriptstyle\pm 1}$	75.8	0.54	72.2	63.7	67.7	85.5

Table 1: Comparison of our approach Crystal against prior state-of-the-art method UKP Arnaout et al. (2025) and four baselines: random label selection, always predicting “impact-revealing,” always predicting “other,” and the method of Valenzuela et al. (2015) . Reported accuracies (in percentages) include standard error (

\pm

SE). Bal. Acc. = balanced accuracy; MCC = Matthews Correlation Coefficient; F1

-

= F1 on the negative (non-impact-revealing) class. Green cells show positive

\Delta:=

Crystal

-

UKP; bold green cells show Crystal’s gains. Our approach outperforms the prior state-of-the-art method and all other baselines across all models.

Dataset	#Papers	Actual Citing	S2 API	API + Scrape
ACL 1996	577	29,308	4,216	18,451
ACL 2000	1,184	40,681	6,639	27,997

Table 2: Coverage statistics for ACL 1996 and 2000 papers and their citing papers. “#Papers” denotes the number of papers published at ACL in each year. “Actual Citing” is the total number of papers citing these publications. “S2 API” is the subset of citing papers with non-empty reference data returned by the Semantic Scholar API. “API + Scrape” includes additional citing papers recovered via website scraping, which substantially improves coverage of reference information.

Figure 3: Impact distribution of ACL 1996 (left) and ACL 2000 (right) publications, ranked by the number of high -impact citations as predicted by Crystal, showing the top 10 in each year with the top-ranked paper at the top of each plot. Each bar shows the breakdown of high , medium , and low impact citations per paper, with the high-impact count annotated. Papers awarded the ACL Test-of-Time (ToT) Award are highlighted with a yellow border: Berger et al. (1996) and Gildea and Jurafsky (2002) . In both years, ToT-winning papers rank among the top in high –impact citations, suggesting that Crystal’s impact characterization aligns with long-term community recognition of foundational contributions.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

System	Predicted	Ground-Truth
Arnaout et al.	Impact-revealing	Impact-revealing
	Other	Other
Valenzuela et al.	Meaningful	Impact-revealing
	Non-meaningful	Other
Crystal (ours)	High	Impact-revealing
	Medium	Other

Appendix

Table 3: Evaluation-time label mappings to ground-truth categories from Arnaout et al. (2025) , applied only at evaluation time.

Panel of Figure 4 — Figure 4: Main Crystal prompt used in all primary experiments. The model jointly ranks all references $R$ of a given paper $P$ by impact, assigning each to one of three categories ( High , Medium , Low ). Bracketed fields ( [main_paper_title] , etc.) are filled at inference time.

Figure 6: Confusion matrices per model for assigning impact labels through majority vote. Rows represent actual labels; columns represent predicted categories. To compare both approaches (collapsing High to impact-revealing and Medium/Low to other, as described in 5 ), we find that our method produces more discriminative boundaries, substantially reducing false positives across all three models. Notably, Gpt-5.1 achieves this while maintaining comparable recall (0.65 vs. 0.64), demonstrating that our method improves precision without sacrificing sensitivity.

Model	Method	Accuracy( $\uparrow$ )	Bal. Acc.( $\uparrow$ )	MCC( $\uparrow$ )	P( $\uparrow$ )	R( $\uparrow$ )	F1( $\uparrow$ )	F1 $-$ ( $\uparrow$ )
Qwen3	UKP	$60.8{\scriptstyle\pm 1.2}$	64.9	0.28	44.5	76.7	56.3	64.5
	Crystal (2 labels)	$69.3{\scriptstyle\pm 1.1}$	68.9	0.37	57.1	60.0	58.5	78.8
	$\Delta$ 2L $-$ UKP	$+8.5$	$+4.0$	$+0.09$	$+12.6$	$-16.7$	$+2.2$	$+14.3$
	Crystal (3 labels) †	$75.1{\scriptstyle\pm 1.0}$	71.1	0.46	70.0	53.3	60.5	83.9
	$\Delta$ 3L $-$ 2L †	$+5.8$	$+2.2$	$+0.09$	$+12.9$	$-6.7$	$+2.0$	$+5.1$

Appendix

Table 4: Results of Ablation 1, evaluated with Qwen3-30B, ablating Crystal’s three-label formulation (high, medium, low) down to two labels (impact-revealing vs. other) following Arnaout et al. (2025) .

\Delta_{\text{2L

UKP}}

shows the gain of Crystal (2 labels) over UKP.

\Delta_{\text{3L

2L}}

shows the further gain of three labels over two. Green cells indicate positive differences. † Full Crystal (3 labels) results shown for reference only. The finer granularity of three labels gives the model a more calibrated decision space: the model more reliably identifies truly high-impact citations.

Model	Method	Accuracy( $\uparrow$ )	Bal. Acc.( $\uparrow$ )	MCC( $\uparrow$ )	P( $\uparrow$ )	R( $\uparrow$ )	F1( $\uparrow$ )	F1 $-$ ( $\uparrow$ )
Qwen3	Independent	$63.0{\scriptstyle\pm 1.1}$	70.3	0.40	46.8	91.7	61.9	63.9
	Crystal(Joint)	$75.1{\scriptstyle\pm 1.0}$	71.1	0.46	70.0	53.3	60.5	83.9
	$\Delta$	$+12.1$	$+0.8$	$+0.06$	$+23.2$	$-38.4$	$-1.4$	$+20.0$

Appendix

Table 5: Shows the results of Ablation 2, which compares considering citation contexts independently against jointly ranking of all citations within a paper, evaluated with Qwen3-30B.

\Delta:=\text{Crystal}-\text{Independent}

. Green cells indicate gains from joint ranking. The precision gains show that considering contexts jointly improves the identification of truly high-impact citations and corrects the overprediction bias observed in the independent setting.

Panel of Figure 7 — Figure 7: Prompt used in Ablation 1, which collapses Crystal’s three-label schema to two labels (impact-revealing vs. other) following Arnaout et al. (2025) . Bracketed fields ( [main_paper_title] , etc.) are filled at inference time.

Figure 8: Confusion matrix (row-wise probabilities) for Ablation 1 (two-label variant). Compared to Figure 6 (Qwen3-30b), the three-label setting introduces a Medium bin that captures borderline cases the binary model misclassifies as impact-revealing , reducing false positives (precision: $57.1\%\rightarrow 70\%$ , F1: $+2\%$ ). Thus, we conclude that three impact labels yield more true impact-revealing classifications and more calibrated judgements.

Figure 9: Confusion matrix (row-wise probabilities) for Ablation 2 (independent ranking). The independent variant shows a strong bias toward predicting High . In contrast, joint ranking (as shown in Figure 6 ) improves class discrimination and calibration, reducing overprediction and yielding more balanced, reliable predictions.

Panel of Figure 10 — Figure 10: Prompt used in Ablation 2, which scores each citation context independently (i.e., in isolation, without access to the full reference list). The model is queried three times per citation and the majority label is taken as the final prediction. Bracketed fields ( [citing_title] , etc.) are filled at inference time.

Model	Method	Accuracy( $\uparrow$ )	Bal. Acc.( $\uparrow$ )	MCC( $\uparrow$ )	P( $\uparrow$ )	R( $\uparrow$ )	F1( $\uparrow$ )	F1 $-$ ( $\uparrow$ )
Gpt-5.1	Run 1	$76.8{\scriptstyle\pm 1.0}$	75.3	0.53	71.7	62.7	66.9	85.2
	Run 2	$77.1{\scriptstyle\pm 1.0}$	75.3	0.52	71.1	63.2	66.9	85.1
	Run 3	$76.9{\scriptstyle\pm 1.0}$	75.7	0.53	71.1	64.2	67.5	85.2
	Crystal	$78.6{\scriptstyle\pm 1.0}$	75.8	0.54	72.2	63.7	67.7	85.5
	$\Delta$	$+1.5$	$+0.1$	$+0.01$	$+0.5$	$-0.5$	$+0.2$	$+0.3$
o4-mini	Run 1	$53.5{\scriptstyle\pm 1.2}$	72.4	0.50	76.0	53.0	62.5	85.5

Appendix

Table 6: Shows the results of Ablation 3, which studies the effect of majority voting across three randomized orderings vs. any single run.

\Delta:=\text{Crystal}-\text{best single run}

, where the best is taken column-wise across the three runs. Green cells show gains from majority voting. Bold indicates the highest F1 overall. Majority voting consistently matches or exceeds even the strongest individual run , demonstrating robustness to positional bias.

Figure 11: Mean number of missing references in the aggregated ranks per model as a function of the number of references in the citing paper. All three models rank nearly all references when the reference list is less than 40. However, they omit more references as the list grows longer. o4-mini exhibits the steepest degradation, while Gpt-5.1 and Qwen3-30B are more robust.

Figure 12: Distribution of citation context length (in characters) for cases unanimously classified by Qwen3-30B , Gpt-5.1 , and o4-mini under Crystal. Left: unanimous false positives (FP, n=45) versus unanimous true negatives (TN, n=972), with medians of 954 and 238 characters respectively. Right: unanimous false negatives (FN, n=166) versus unanimous true positives (TP, n=245), with medians of 280 and 1,110 characters respectively. Dashed lines mark the medians for each group.

Panel of Figure 13 — Figure 13: Examples of contexts that were misclassified as impact-revealing by Qwen3-30B , Gpt-5.1 , and o4-mini under Crystal. In each case the cited work is described in detail and attributed to its authors, although the citing paper does not build on it.

Panel of Figure 14 — Figure 14: Examples of contexts that were misclassified as not impactful by Qwen3-30B , Gpt-5.1 , and o4-mini under Crystal. The citing paper builds on or is inspired by the cited work, but the mention is brief or the cited paper is grouped with other references in the same citation.

Citation Impact Classifier	Model	Acc.( $\uparrow$ )	P( $\uparrow$ )	R( $\uparrow$ )	F1( $\uparrow$ )
Ours (Ord. Reg.)	Gpt-5.1	$78.9{\scriptstyle\pm 1}$	73.6	62.7	67.7
	o4-mini	$64.3{\scriptstyle\pm 1.1}$	76.4	54.5	62.2
	Qwen3 30B	$75.9{\scriptstyle\pm 1}$	75.6	48.6	59.1

Appendix

Table 7: Results using an ordinal regression model to predict impact labels. Reported accuracies include the standard error (

\pm

SE).

Figure 15: Confusion matrices per model for assigning impact labels with an ordinal regression model. Rows represent actual labels; columns represent predicted categories. Gpt-5.1 achieves the best performance in identifying impactful citations and reducing false positive. However, we see a slight decrease in performance with Qwen3-30b.

Dataset	High	Medium	Low
ACL 1996	3,934 (19%)	6,190 (30%)	10,696 (51%)
ACL 2000	5,987 (17%)	12,184 (34%)	17,901 (50%)

Appendix

Table 8: Distribution of impact labels assigned by Crystal across all citations of ACL 1996 and ACL 2000 publications. “High”, “Medium”, and “Low” report the number (and percentage) of citations assigned to each impact category.

Figure 17: Yearly citation impact distribution for the top four ACL 1996 papers ranked by high-impact citations under Crystal. Each subplot stacks high -, medium -, and low -impact citations by year of the citing paper. Test-of-Time award winners ( Berger et al. (1996) , top-left; Carletta (1996) , bottom-right) are marked with a $\star$ and highlighted with a yellow border. Berger et al. (1996) accumulates the largest volume of high-impact citations of the four papers, sustained across more than two decades. Carletta (1996) exhibits a stable but smaller stream of high-impact citations, with most of its citation volume concentrated in the low-impact category.

Figure 18: Yearly citation impact distribution for the top four ACL 2000 papers ranked by high-impact citations under Crystal (as shown in Figure 3 ). Each subplot stacks high -, medium -, and low -impact citations by year of the citing paper. The Test-of-Time award winner ( Gildea and Jurafsky (2002) , bottom-right) is marked with a

\star

and highlighted with a yellow border. Its high-impact citation profile remains stable through 2025, while several non-awarded top-ranked papers peak earlier and decline more steeply.

Figure 19: The main layout of the custom annotation interface used in the pilot study, showing the overall task setup and instructions for ranking references by impact.

Figure 20: The full list of references from an annotator’s paper displayed within the interface, allowing users to rank their references by dragging and reordering them according to the impact criteria.

Figure 21: A feature of the interface that reveals the citation context when the "Show Citation Contexts" button is clicked, illustrating how each reference was cited in the paper.

Figure 22: The final ranked list of references generated within the interface, showing the outcome of the annotation task with color-coded impact categories. Annotators submit their final ranked list of references.

Figure 23: Shows the Spearman correlation between each annotator’s ranking and the ranking generated by Gpt -5.1 using the same prompt from our experiments. The $n$ value above each bar represents the total number of references ranked in that paper. All correlation values exceeded $0.7$ , indicating strong agreement.

Bibliometric Analysis

Bibliometric analysis uses publication, citation, collaboration, and text data to map and measure the structure and evolution of research.

See all 29 papers

SciImpact: A Multi-Dimensional, Multi-Field Benchmark for Scientific Impact Prediction
Apr 18, 2026Hangxiao Zhu, Yuyu Zhang, Ping Nie +1LLM Evaluation
Texas A&M University · Verdent AI · University of Waterloo
LLM4Impact: Integrating Heterogeneous Information for Scientific Impact Prediction
Oct 7, 2026Yong Cao, Markus Flicke, Haoyu He +2
University of Tübingen, Tübingen AI Center · KE:SAI
LLM-Metrics: Measuring Research Impact Through Large Language Model Memory
May 21, 2026Si Shen, Wenhua Zhao, Danhao Zhu
School of Economics and Management Department of Computer Science and Engineering Nanjing University of Science and Technology Nanjing, 210094, China · College of Information Management Nanjing Agricultural University Nanjing, 210095, China · Department of Criminal Science and Technology Jiangsu Police Institute Nanjing, 210031, China

Apr 18, 2026J/K move · Enter open · S save

SciImpact: A Multi-Dimensional, Multi-Field Benchmark for Scientific Impact Prediction

Hangxiao Zhu, Yuyu Zhang, Ping Nie, Yu Zhang

Texas A&M University · Verdent AI · University of Waterloo

LLM Evaluation

Open paper AskPDF arXiv

The rapid growth of scientific literature calls for automated methods to assess and predict research impact. Prior work has largely focused on citation-based metrics, leaving limited evaluation of models' capability to reason about other impact dimensions. To this end, we introduce SciImpact, a large-scale, multi-dimensional benchmark for scientific impact prediction spanning 19 fields. SciImpact captures various forms of scientific influence, ranging from citation counts to award recognition, media attention, patent reference, and artifact adoption, by integrating heterogeneous data sources and targeted web crawling. It comprises 215,928 contrastive paper pairs reflecting meaningful impact differences in both short-term (e.g., Best Paper Award) and long-term settings (e.g., Nobel Prize). We evaluate 11 widely used large language models (LLMs) on SciImpact. Results show that off-the-shelf models exhibit substantial variability across dimensions and fields, while multi-task supervised fine-tuning consistently enables smaller LLMs (e.g., 4B) to markedly outperform much larger models (e.g., 30B) and surpass powerful closed-source LLMs (e.g., o4-mini). These results establish SciImpact as a challenging benchmark and demonstrate its value for multi-dimensional, multi-field scientific impact prediction. Our project homepage is https://flypig23.github.io/sciimpact-homepage/

Abstract

Bibliometric Analysis

Explore similar work

SciImpact: A Multi-Dimensional, Multi-Field Benchmark for Scientific Impact Prediction

LLM4Impact: Integrating Heterogeneous Information for Scientific Impact Prediction

LLM-Metrics: Measuring Research Impact Through Large Language Model Memory