Structural Diversity Drives Disruptive Scientific Innovation

Yichun Peng; Saike He; Peijie Zhang; Kang Zhao; Yi Yang; Ning Zhang; Qingpeng Zhang; Daniel Dajun Zeng

cs.SIApr 2, 2026

Structural Diversity Drives Disruptive Scientific Innovation

Authors: Yichun Peng, Saike He, Peijie Zhang, Kang Zhao, Yi Yang, Ning Zhang, Qingpeng Zhang, Daniel Dajun Zeng,

Organizations: State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · University of Chinese Academy of Sciences, Beijing 101408, China · Department of Business Analytics, Tippie College of Business, The University of Iowa, Iowa City, IA 52242, United States of America · The University of Hong Kong, Institute of Data Science & Department of Pharmacology and Pharmacy ·

Abstract

Scientific innovation increasingly depends on collaboration, yet the organizational structure that fosters breakthrough ideas remains poorly understood. Existing metrics - such as team size or compositional diversity - capture readily observable characteristics but not the deeper architecture of collaboration. We introduce Structural Diversity (SD): the extent to which a team bridges multiple distinct knowledge communities within its prior collaboration network. Using a century-scale dataset of 260 million scientific publications (1900-2025) and combining causal inference with a quasi-natural experiment based on a U.S. National Science Foundation policy change in 2012, we show that SD is a powerful and robust predictor of disruptive innovation, outperforming traditional team novelty indicators such as team freshness and edge density. Moreover, SD positively interacts with team size and is able to mitigate the well-known "curse of scale" by transforming scale from a liability into a resource for creative synthesis. We find that one mechanism underlying this effect is Disciplinary Integration (DI): teams with higher SD can more effectively combine heterogeneous knowledge into novel configurations. Our findings position SD as both a new theoretical construct and an actionable design principle for organizing scientific collaboration. By linking the architecture of team assembly to the dynamics of creative discovery, our work offers a structural explanation for how collective intelligence can be systematically engineered to foster disruptive innovation.

Figures & tables

Figure 1: SD as a robust predictor of disruptive scientific innovation. (A-B) Illustration of how SD captures the potential of distinct knowledge communities for innovation. (C) Each point represents one discipline (19 in total). The dashed line shows a statistically positive correlation between the mean SD (standardized) and the mean CD Index (field-normalized) across all areas of science. Bubble size is proportional to the number of papers in each discipline. (D) Relationship between the number of CCs within a team’s historical co-authorship network and the CD Index, stratified by a paper’s team size from 2 to 7 authors (N = $256,600,000$ papers). Each subplot shows that, holding team size constant, more CCs are associated with higher disruptive impact. Error bars represent $95\%$ confidence intervals. (E-G) Mean CD Index as a function of SD, team freshness, and edge density. Points show the mean values for equal-sized bins of papers. Each colored regression line represents the simple linear fit of these binned averages ( $R^{2}$ refers to the goodness of fit for the binned data). Histogram and kernel density curve at the top of each subplot show the distribution of the respective (standardized) metric.

Dependent Variable: CD Index (field-normalized)
	Model 1	Model 2	Model 3
SD (standardized)	$0.026$ ( $p<0.001$ )	$0.026$ ( $p<0.001$ )	$0.025$ ( $p<0.001$ )
Content Controls
Title Word Count		$-0.002$ ( $p<0.001$ )	$-0.002$ ( $p<0.001$ )
Title Readability Score		$0.001$ ( $p<0.055$ )	$0.001$ ( $p=0.070$ )
Title Promotional Words (%)		$0.016$ ( $p<0.001$ )	$0.012$ ( $p<0.001$ )

Table 1: OLS regression of Structural Diversity (SD) on the field-normalized CD Index.

Figure 2: Disciplinary heterogeneity of SD and the mitigation of “Curse of Scale”. (A) Heterogeneous effects of SD on disruptive innovation across 19 disciplines. Each horizontal bar shows the regression coefficient ( $p<0.001$ for all 19 disciplines) between SD (standardized) and CD Index (field-normalized) by fitting the same regression model in Table 1 to all papers in each discipline. The effect of SD is strongest in natural sciences (e.g., Medicine, Geology, and Physics) and weakest in social sciences such as Business and Economics. (B) Positive interaction effect between SD (standardized) and Team Size (see regression coefficients in Table 2 ). The margins plot shows the predicted CD Index (field-normalized) as a function of SD (standardized) for different team sizes. The slope becomes steeper as team size increases (from purple to light green), indicating that SD can effectively mitigate the “curse of scale” by amplifying the disruptive potential of large teams. Error bars represent $95\%$ confidence intervals.

Dependent variable: CD Index (field-normalized)
Main Predictors
SD (Standardized)	$0.016$ ( $p<0.001$ )
Team Size (log)	$-0.031$ ( $p<0.001$ )
Interaction
SD $\times$ Team Size (log)	$0.017$ ( $p<0.001$ )
Content Controls

Table 2: OLS regression reveals a statistically positive interaction effect between Structural Diversity (SD) and Team Size on CD Index.

Figure 3: Robustness test using the Propensity Score Matching framework further supports the positive association between SD and disruptive innovation. The analysis is based on $690,972$ Computer Science papers. (A) Kernal density estimation of the CD Index for treated teams (Top quartile SD, orange) and their matched control teams (Bottom quartile SD, blue). The distribution of CD Index for high-SD teams is statistically right-shifted than that for low-SD teams (mean CD: $0.06$ vs. $-0.06$ , $p<0.001$ ), indicating a large increase in team disruptiveness. (B) Robustness checks using a symmetric interval matching approach based on ten deciles of SD. The “D1” pair represents the comparison between the most structurally diverse teams (Top $10\%$ ) and the least diverse teams (Bottom $10\%$ ), whereas the “D5” pair compares two sets of matched teams with SD near the median. The dots represent point estimates of the Average Treatment Effect on the Treated (ATT) of SD on CD Index. The positive effect of SD is substantial at the two extremes but converges to zero for teams with the average level of structural diversity. Error bars indicate $95\%$ bootstrapped confidence intervals.

Figure 4: A quasi-natural experiment based on the NSF policy change in 2012 supports the correlation between SD and disruptive innovation. (A) The percentage of teams by the number of Connected Components (CC) before (2010-2011, $N=18,267$ ) and after (2012-2013, $N=23,191$ ) the NSF policy implementation. All papers ( $N=41,458$ ) were NSF-funded in the AMiner dataset. The inset highlights a sharp increase in teams with high CC counts (right tail) in the post-policy period (Mann-Whitney U test, $p<0.001$ ). (B) Kernel density estimation of SD shows a distributional shift towards higher structural diversity after the policy change (Mann-Whitney U test, $p<0.001$ ). (C) Violin plots comparing the CD Index before and after the policy implementation. The median (or mean) CD Index increased statistically in the post-policy period ( $p<0.001$ in a two-sided t-test), suggesting that the policy-induced structural diversification translated into more creative research outputs.

Figure 5: Disciplinary Integration (DI) mediates a sizable fraction of the effect of SD on disruptive innovation. (A) Path diagram linking team SD to their CD Index via disciplinary integration. SD significantly predicts DI ( $\beta=0.153$ , $p<0.001$ ), which in turn predicts CD Index ( $\beta=0.028$ , $p<0.001$ ). About $18\%$ of the effect of SD on CD (total effect size: $0.024$ , $p<0.001$ ) can be explained by DI (indirect effect size: $0.004$ , $p<0.001$ ). (B) The same path diagram for Team Freshness (TF). Unlike SD, TF shows no predictive power on DI. The indirect path to innovation via DI is also insignificant, demonstrating that the unique power of SD lies in team members’ capacity to synthesize distinct knowledge domains. The mediation analysis is conducted based on N = $260,400,000$ papers indexed in the OpenAlex database. All regression models in this analysis further control for all confounding variables except Team Freshness listed in Table 1 .

Scientific Discovery

Scientific discovery uses computational methods to find hypotheses, laws, mechanisms, and materials from data, literature, and experiments.

See all 86 papers

Sparking Scientific Creativity via LLM-Driven Interdisciplinary Inspiration
Mar 12, 2026Priyanka Kargupta, Shuhaib Mehri, Dilek Hakkani-Tur +1Research Idea Generation Large Language Model-Guided Scientific Discovery
Siebel School of Computing and Data Science, University of Illinois at Urbana-Champaign, Urbana, IL, United States
Beyond Single-Dimension Novelty: How Combinations of Theory, Method, and Results-based Novelty Shape Scientific Impact
Apr 14, 2026Yi Zhao, Yang Chenggang, Yuzhuo Wang +3
School of Management, Anhui University, Hefei 230039, China · Department of Information Management, Nanjing University of Science and Technology, Nanjing, 210094, China · School of Information Management, Central China Normal University, Wuhan 430079, China
Diversity Collapse in Multi-Agent LLM Systems: Structural Coupling and Collective Failure in Open-Ended Idea Generation
Apr 20, 2026Nuo Chen, Yicheng Tong, Yuzhe Yang +5Multi-Agent LLM Systems Multi-Agent Collaboration
National University of Singapore · The Chinese University of Hong Kong, Shenzhen

Mar 12, 2026J/K move · Enter open · S save

Sparking Scientific Creativity via LLM-Driven Interdisciplinary Inspiration

Priyanka Kargupta, Shuhaib Mehri, Dilek Hakkani-Tur, Jiawei Han

Siebel School of Computing and Data Science, University of Illinois at Urbana-Champaign, Urbana, IL, United States

Research Idea Generation Large Language Model-Guided Scientific Discovery Computational Creativity

Open paper AskPDF arXiv

Despite interdisciplinary research leading to larger and longer-term impact, most work remains confined to single-domain academic silos. Recent AI-based approaches to scientific discovery show promise for interdisciplinary research, but many prioritize rapidly designing experiments and solutions, bypassing the exploratory, collaborative reasoning processes that drive creative interdisciplinary breakthroughs. As a result, prior efforts largely prioritize automating scientific discovery rather than augmenting the reasoning processes that underlie scientific disruption. We present Idea-Catalyst, a novel framework that systematically identifies interdisciplinary insights to support creative reasoning in both humans and large language models. Starting from an abstract research goal, Idea-Catalyst is designed to assist the brainstorming stage, explicitly avoiding premature anchoring on specific solutions. The framework embodies key metacognitive features of interdisciplinary reasoning: (a) defining and assessing research goals, (b) awareness of a domain's opportunities and unresolved challenges, and (c) strategic exploration of interdisciplinary ideas based on impact potential. Concretely, Idea-Catalyst decomposes an abstract goal (e.g., improving human-AI collaboration) into core target-domain research questions that guide the analysis of progress and open challenges within that domain. These challenges are reformulated as domain-agnostic conceptual problems, enabling retrieval from external disciplines (e.g., Psychology, Sociology) that address analogous issues. By synthesizing and recontextualizing insights from these domains back into the target domain, Idea-Catalyst ranks source domains by their interdisciplinary potential. Empirically, this targeted integration improves average novelty by 21% and insightfulness by 16%, while remaining grounded in the original research problem.

Abstract

Scientific Discovery

Explore similar work

Sparking Scientific Creativity via LLM-Driven Interdisciplinary Inspiration

Beyond Single-Dimension Novelty: How Combinations of Theory, Method, and Results-based Novelty Shape Scientific Impact

Diversity Collapse in Multi-Agent LLM Systems: Structural Coupling and Collective Failure in Open-Ended Idea Generation