More Computational Resources Do Not Ensure Higher Scholarly Impact: Evidence from Leading NLP Conference Papers

Shuai Chen; Tong Bao; Jitong Peng; Chengzhi Zhang

cs.CLAug 22, 2026

More Computational Resources Do Not Ensure Higher Scholarly Impact: Evidence from Leading NLP Conference Papers

Authors: Shuai Chen, Tong Bao, Jitong Peng, Chengzhi Zhang

Organizations: Department of Information Management, Nanjing University of Science and Technology, China

Abstract

Computational resources are increasingly central to NLP research, but how closely reported GPU capability aligns with scholarly impact remains unclear. We analyze 13,921 ACL, EMNLP, and NAACL main-conference papers published between 2020 and 2025, using GPU resources as our operational measure of computational resources. From full texts, we extract GPU models and counts, standardize each paper's largest reported configuration into a comparable hardware-capability measure, and link these data to citation, award, topic, and institutional metadata. GPU reporting became more common but remained incomplete, while reported capability increased mainly through newer hardware generations and medium-scale multi-GPU configurations. Resource concentration substantially exceeded impact concentration: the annual top 20% of GPU-quantifiable papers accounted for 83.9%-89.9% of reported GPU capability, but only 27%-32% of citations and 20%-33% of paper awards. In adjusted models, a tenfold increase in aggregate reported GPU capability was associated with a 3.52-percentage-point increase in within-NLP topic-year citation percentile, but increased model R^2 by only 0.0042. GPU count showed more consistent positive associations with citation and award outcomes than newer hardware generation. Overall, reported GPU resources are associated with scholarly impact but provide little standalone explanation of research influence.

Figures & tables

Figure 1: Estimated training compute for selected milestone language models has increased substantially over time, alongside advances in accelerator hardware.

Figure 2: Overview of our computational resource measurement and analysis framework. We collect ACL, EMNLP, and NAACL papers from the ACL Anthology and enrich them with OpenAlex metadata, extract reported GPU models and quantities with human validation and LLM-based extraction, normalize raw hardware mentions to canonical GPU attributes, and use the resulting corpus to analyze reporting trends, reported compute scale, concentration across topics, organizations, and regions, and scholarly impact.

Group	#Papers	Definition	Example
Model-reported	6,900 (49.6%)	At least one reported GPU model can be normalized to hardware specifications	RTX 4090
Strict	5,360 (38.5%)	Both a standardizable GPU model and an explicit device quantity are observed	8×A100 GPU

Table 1: Dataset statistics used in this study.

Figure 3: Reporting completeness of GPU resources in NLP papers.

Figure 4: Evolution of reported GPU scale, generation, and capacity in NLP research. The left panel shows GPU count over time; the middle panel shows the share of GPU generations; the right panel shows normalized reported GPU capacity distributions and annual 95th percentile (P95) values.

Figure 5: Reported GPU capacity by institutional access regime. Panel a shows median reported peak GPU configuration capacity across institutional access regimes. Points denote medians, horizontal intervals show the interquartile range, and sample sizes are reported on the right. Panel b shows share of papers in the yearly top-20% reported GPU capacity tail. The dashed vertical line marks the 20% reference level.

Figure 6: Reported GPU hardware capacity across major NLP topics: median, interquartile range (IQR), and the 90th percentile (P90). The full results for all 29 topic categories are reported in .

Figure 7: Concentration of reported GPU capability and scholarly impact. Panel (a) shows annual shares of total reported GPU capability, citations, and awards accounted for by papers in the top 20% of reported GPU capability. Panel (b) shows overlap between high-capability and high-impact papers. High-capability papers are defined as the top 20% by reported GPU capability, and high-impact papers as the top 10% by citations within publication-year-by-venue groups.

GPU-capability cutoff	Citation-impact cutoff
GPU-capability cutoff	Top 5%	Top 10%	Top 20%
Top 10%	$2.10\times$	$1.60\times$	$1.49\times$
Top 20%	$2.14\times$	$\mathbf{1.59\times}$	$1.53\times$
Top 30%	$1.93\times$	$1.55\times$	$1.49\times$

Table 2: Robustness to alternative thresholds. Each entry reports the ratio of high-impact rates between papers above and below the GPU-capability cutoff. Values above one indicate higher citation impact among high-capability papers.

Outcome	Aggregate GPU capability	GPU count	Ampere or newer	$\Delta R^{2}$ aggregate / joint
NLP topic-year percentile (primary)	$+3.52^{**}$ pp	$+4.57^{***}$ pp	$+4.16^{**}$ pp	$0.0042/0.0093$
OpenAlex percentile (secondary)	$+1.26$ pp	$+1.96$ pp	$+2.02$ pp	$0.0009/0.0033$
$\log(1+\mathrm{citations})$ , OLS	$+18.8^{***}\%$	$+24.7^{***}\%$	$+15.0^{*}\%$	$0.0052/0.0086$
Citation count, PPML	$+61.2^{***}\%$	$+69.3^{***}\%$	$+31.0^{*}\%$	–
Top-10% cited, LPM	$+3.84^{**}$ pp	$+4.41^{**}$ pp	$+2.17$ pp	$0.0043/0.0051$
Awarded, LPM	$+0.86$ pp	$+1.33^{*}$ pp	$-0.25$ pp	$0.0011/0.0018$

Table 3: Adjusted associations between reported GPU resources and scholarly impact.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Type of dataset	Role in analysis
Paper corpus	Defines the 13,921 ACL/EMNLP/NAACL main conference papers.
Parsed paper text	Source for GPU hardware mentions and evidence spans.
OpenAlex metadata	Provides citation counts, author/team size, and citation percentile measures.
Organization variables	Capture industry participation, industry-academia collaboration, cross-sector collaboration, international collaboration, and organization counts.
Topic labels	Provide a 29-topic taxonomy based on title and abstract classification.
Award labels	Identify best paper and related award indicators for 2020–2025 main conference papers.

Appendix

Table 4: Dataset components.

Case type	Example pattern	Annotation decision
Complete GPU reporting	trained on 8 NVIDIA A100 80 GB GPUs	Annotate GPU model, memory, and quantity; eligible for paper-level capacity estimation.
Model-only reporting	experiments ran on A100 GPUs	Annotate GPU model; quantity is null , and lower-bound analyses assign one GPU.
Ambiguous hardware reporting	experiments were run on GPU servers	Not standardizable to a specific GPU model; excluded from capacity estimation.
Citation or prior-work hardware	Brown et al. used 1024 TPU v3 chips	Prior-work hardware; not annotated as this paper’s hardware.
Model/API/software names	we use GPT-4o, Claude, Gemini, and vLLM	Model/API/software names are not hardware; not annotated.

Appendix

Table 5: Boundary cases in computational resource annotation.

Model	Matching criterion	Precision	Recall	F1
deepseek-v3.2	Name only	$0.918\pm 0.0028$	$\mathbf{0.948\pm 0.0013}$	$\mathbf{0.933\pm 0.0019}$
deepseek-v3.2	Exact match	$0.865\pm 0.0021$	$\mathbf{0.894\pm 0.0024}$	$0.879\pm 0.0020$
gpt-4o-mini	Name only	$0.907\pm 0.0099$	$0.842\pm 0.0066$	$0.874\pm 0.0069$
gpt-4o-mini	Exact match	$0.864\pm 0.0130$	$0.802\pm 0.0115$	$0.832\pm 0.0115$
gpt-5.4-mini	Name only	$\mathbf{0.927\pm 0.0092}$	$0.933\pm 0.0096$	$0.930\pm 0.0092$
gpt-5.4-mini	Exact match	$0.842\pm 0.0126$	$0.848\pm 0.0133$	$0.845\pm 0.0128$

Appendix

Table 6: LLM extraction performance on the human-validated set. Scores are reported as mean

\pm

standard deviation. Bold values indicate the best score within each matching criterion.

Figure 8: Prompt used to extract computational resources.

Figure 9: Prompt used to classify NLP topics.

Reporting outcome	$N$	Event rate	LPM $R^{2}$	$\Delta$ LPM $R^{2}$	Logit AUC	$\Delta$ AUC	McFadden pseudo- $R^{2}$	$\Delta$ pseudo- $R^{2}$
Reports at least one standardizable GPU model	13,745	0.5011	0.0804	0.0009	0.6572	0.0014	0.0602	0.0007
Reports standardizable GPU model and quantity	13,745	0.3875	0.0910	0.0013	0.6768	0.0015	0.0735	0.0011

Appendix

Table 7: Incremental explanatory power of organizational variables for GPU reporting sample membership.

Predictor	Coef. (pp)	95% CI	$p$ -value
Outcome: Reports at least one standardizable GPU model
Industry participation	-0.935	[-4.523, 2.654]	0.610
Industry–academia collaboration	1.938	[-2.287, 6.162]	0.369
Cross-sector collaboration	2.329	[-0.298, 4.957]	0.082
International collaboration	-1.595	[-3.627, 0.436]	0.124
log(1 + number of organizations)	-2.320	[-5.672, 1.031]	0.175

Appendix

Table 8: Full linear probability model coefficients for GPU reporting sample membership.

Audit outcome	Papers	Share
Consumption-related signal visible	92	38.3%
No consumption-related signal visible	148	61.7%
Total	240	100.0%

Appendix

Table 9: Manual audit of consumption-related reporting among 240 papers reporting GPU usage.

Figure 10: GPU hardware reporting completeness across venues.

Figure 11: Reported GPU count distributions over time.

Figure 12: Transition of leading reported GPU models in NLP papers.

Figure 13: Growth in reported GPU memory capacity in NLP papers.

Figure 14: Distribution of reported peak GPU configuration capacity in NLP papers.

Figure 15: Country-level concentration in high reported GPU compute.

Figure 16: Reported GPU hardware capacity across NLP research topics. Panel (a) reports the median, interquartile range, and P90 of paper-level peak reported GPU configuration capacity. Panel (b) reports the share of papers at or above the year-specific P90 capacity threshold. Because ties at the P90 threshold are retained, the overall share can exceed 10%; the dashed line indicates the empirical overall share of 19.5%.

Predictor	M1	M2	M3	M4	M5	M6
Industry	$0.267^{***}$					$0.474^{***}$
	$(0.016)$					$(0.042)$
Industry-academia		$0.195^{***}$				$-0.299^{***}$
		$(0.017)$				$(0.047)$
Cross-sector			$0.160^{***}$			$0.057^{*}$
			$(0.015)$			$(0.023)$

Appendix

Table 10: Associations between institutional characteristics and reported GPU capacity.

Citation cutoff	Coef. (pp)	95% CI	$p$	$\Delta R^{2}$
Top 5%	+4.04	[1.81, 6.26]	$<0.001$	0.0085
Top 10%	+3.84	[1.14, 6.53]	0.005	0.0043
Top 20%	+7.20	[3.80, 10.59]	$<0.001$	0.0086

Appendix

Table 11: Sensitivity of the continuous reported GPU-capability model to alternative high-impact thresholds. Coefficients are percentage-point differences associated with a tenfold increase in reported GPU capability. All models include the same fixed effects and controls as the baseline specification.

Outcome	Estimator	$N$	$\beta$	SE	95% CI	$p$	Implied effect	$\Delta R^{2}$
NLP topic–year percentile (primary)	OLS	2,194	0.0352	0.0115	[0.0127, 0.0577]	0.002	+3.52 pp	0.0042
OpenAlex field-normalized percentile	OLS	2,194	0.0126	0.0088	[ $-0.0046$ , 0.0298]	0.151	+1.26 pp	0.0009
$\log(1+\mathrm{citations})$	OLS	2,194	0.1719	0.0491	[0.0757, 0.2681]	$<0.001$	+18.8%	0.0052
Citation count	PPML	2,194	0.4775	0.0852	[0.3105, 0.6445]	$<0.001$	+61.2%	–
Top-10% cited	LPM	2,194	0.0384	0.0138	[0.0114, 0.0654]	0.005	+3.84 pp	0.0043
Awarded	LPM	5,357	0.0086	0.0045	[ $-0.0002$ , 0.0174]	0.056	+0.86 pp	0.0011

Appendix

Table 12: Full adjusted models relating aggregate reported GPU capability to scholarly impact. The implied effects correspond to a tenfold increase in aggregate reported GPU capability. For OLS log-citation and PPML models, percentage effects are calculated as

100\times(e^{\beta}-1)

. For percentile and linear-probability outcomes, effects are expressed in percentage points (pp). Incremental

R^{2}

is calculated against a controls-only model estimated on the same sample; it is not reported for PPML.

Outcome	Term	$\beta$	SE	95% CI	$p$	Implied effect	Joint $\Delta R^{2}$
NLP topic–year percentile	GPU count	0.0457	0.0133	[0.0196, 0.0718]	$<0.001$	+4.57 pp	0.0093
	Newer generation	0.0416	0.0135	[0.0151, 0.0681]	0.002	+4.16 pp
OpenAlex field-normalized percentile	GPU count	0.0196	0.0101	[ $-0.0002$ , 0.0394]	0.052	+1.96 pp	0.0033
	Newer generation	0.0202	0.0105	[ $-0.0004$ , 0.0408]	0.055	+2.02 pp
$\log(1+\mathrm{citations})$	GPU count	0.2206	0.0571	[0.1087, 0.3325]	$<0.001$	+24.7%	0.0086
	Newer generation	0.1396	0.0552	[0.0314, 0.2478]	0.011	+15.0%

Appendix

Table 13: Full joint models separating reported GPU count from hardware generation. GPU count and the Ampere-or-newer/equivalent indicator are entered simultaneously. GPU-count effects correspond to a tenfold increase in the reported count; newer-generation effects compare Ampere-or-newer/equivalent hardware with earlier generations. Citation models use

N=2{,}194

and the award model uses

N=5{,}357

. The joint incremental

R^{2}

is the increase from adding both hardware terms to the controls-only model estimated on the same sample.

Term	$\beta$ (SE)	Profile 95% CI	Penalized LRT $p$	Holm $p$	OR [95% CI]
GPU count (tenfold increase)	0.5380 (0.1883)	[0.1730, 0.8907]	0.0043	0.0085	1.713 [1.189, 2.437]
Ampere-or-newer/equivalent	$-0.1115$ (0.2734)	[ $-0.6322$ , 0.4362]	0.6833	0.6833	0.894 [0.531, 1.547]

Appendix

Table 14: Firth penalized logistic regression of paper-award status on reported GPU count and hardware generation. The model uses the same 5,357 papers and 111 award-positive cases as the joint linear probability model. GPU count and the Ampere-or-newer/equivalent indicator are entered simultaneously with the baseline fixed effects and controls. The joint null that both hardware coefficients equal zero is rejected,

\chi^{2}(2)=8.3178

p=0.0156

Outcome	Specification	$\beta$ (SE)	95% CI	$p$	$R^{2}$ controls/full	$\Delta R^{2}$
NLP topic–year percentile	Common-sample baseline	0.0313 (0.0118)	[0.0081, 0.0545]	0.008	0.0895/0.0928	0.0032
	+ pre-publication controls	0.0274 (0.0121)	[0.0037, 0.0512]	0.024	0.1330/0.1354	0.0023
	+ public artifact	0.0265 (0.0121)	[0.0028, 0.0502]	0.028	0.1362/0.1384	0.0022
OpenAlex field-normalized percentile	Common-sample baseline	0.0116 (0.0091)	[ $-0.0062$ , 0.0295]	0.202	0.0909/0.0916	0.0008
	+ pre-publication controls	0.0105 (0.0094)	[ $-0.0080$ , 0.0290]	0.264	0.1172/0.1178	0.0006
	+ public artifact	0.0102 (0.0095)	[ $-0.0084$ , 0.0287]	0.282	0.1179/0.1185	0.0006

Appendix

Table 15: Robustness of aggregate reported GPU-capability estimates to expanded observable controls. All models use the same complete-case sample of 2,077 papers. The coefficient represents the association with a tenfold increase in aggregate reported GPU capability. Incremental

R^{2}

compares each full model with a model containing the same controls but excluding reported GPU capability.

Characteristic	Main Conference	Findings	Main + Findings
Total papers	13,921	9,917	23,838
Papers reporting standardized GPU model	6,900	5,824	12,724
Reporting rate (%)	49.6	58.7	53.4
Papers reporting GPU model + count	5,360	4,186	9,546
Strict reporting rate (%)	38.5	42.2	40.0
Citation-analysis sample	2,194	1,620	3,814

Appendix

Table 16: Sample coverage and reported GPU resources in Main Conference and Findings papers.

Outcome	Main Conference $\beta$ (SE) [ $\Delta R^{2}$ ]	Findings $\beta$ (SE) [ $\Delta R^{2}$ ]	Pooled $\beta$ (SE) [ $\Delta R^{2}$ ]	Findings $-$ Main difference ( $p$ )
NLP topic–year citation percentile	0.035 (0.011) [0.0042]	0.046 (0.014) [0.0071]	0.039 (0.009) [0.0051]	$+0.011$ (0.551)
OpenAlex field-normalized percentile	0.013 (0.009) [0.0009]	0.040 (0.011) [0.0073]	0.024 (0.007) [0.0029]	$+0.027$ (0.051)
$\log(1+\mathrm{citations})$	0.172 (0.049) [0.0052]	0.207 (0.054) [0.0083]	0.185 (0.036) [0.0061]	$+0.035$ (0.630)
Citation count (PPML)	0.477 (0.085) [—]	0.390 (0.090) [—]	0.456 (0.066) [—]	$-0.088$ (0.481)
Top-10% cited (LPM)	0.038 (0.014) [0.0043]	0.059 (0.018) [0.0093]	0.047 (0.011) [0.0062]	$+0.020$ (0.372)
$N$	2,194	1,620	3,814	3,814

Appendix

Table 17: Replication of citation-impact associations and incremental explanatory power across publication tracks.

Track	High-impact rate among high-capability papers	High-impact rate among other papers	Risk ratio	High-capability share among high-impact papers	High-capability papers not high-impact
Main Conference	14.5%	9.1%	1.59	28.6%	85.5%
Findings	15.8%	9.0%	1.75	30.6%	84.2%
Main + Findings	14.7%	9.2%	1.60	28.6%	85.3%

Appendix

Table 18: Overlap between high reported GPU capability and high citation impact across publication tracks.

Crystal: Characterizing Relative Impact of Scholarly Publications
Mar 25, 2026Hannah Collison, Benjamin Van Durme, Daniel KhashabiBibliometric Analysis Learning to Rank
Johns Hopkins University
The Future of NLP may not be at NLP Conferences: Scholarly Migration Patterns in Natural Language Processing
Jul 2, 2026David JurgensScientific Communication Bibliometric Analysis
School of Information, University of Michigan
Papers Without Code: Availability of GitHub Repositories Linked in *CL Publications
Sep 28, 2026Selina Meyer, Michael RothScientific Reproducibility
University of Technology Nuremberg

Mar 25, 2026J/K move · Enter open · S save

Crystal: Characterizing Relative Impact of Scholarly Publications

Hannah Collison, Benjamin Van Durme, Daniel Khashabi

Johns Hopkins University

Bibliometric Analysis Learning to Rank LLM Reranking

Open paper AskPDF arXiv

Assessing a cited paper's impact is typically done by analyzing its citation context in isolation within the citing paper. While this focuses on the most directly relevant text, it prevents relative comparisons across all the works a paper cites. We propose Crystal, which instead jointly ranks all cited papers within a citing paper using large language models (LLMs). To mitigate LLMs' positional bias, we rank each list three times in a randomized order and aggregate the impact labels through majority voting. This joint approach leverages the full citation context, rather than evaluating citations independently, to more reliably distinguish impactful references. Crystal outperforms a prior state-of-the-art impact classifier by +9.5% accuracy and +8.3% F1 on a dataset of human-annotated citations. Crystal further gains efficiency through fewer LLM calls and outperforms prior baselines using an open-weight model, enabling scalable, cost-effective citation impact analysis. In a case study of ACL Test-of-Time award-winning papers, we find that Crystal's impact characterizations align closely with long-term scientific recognition. We release Crystal-Bank, a 46.8k-paper dataset with rankings and impact labels, along with code.

Abstract

Explore similar work

Crystal: Characterizing Relative Impact of Scholarly Publications

The Future of NLP may not be at NLP Conferences: Scholarly Migration Patterns in Natural Language Processing

Papers Without Code: Availability of GitHub Repositories Linked in *CL Publications