The Weight of Teamwork: Does Communication Structure Help LLM Scientists Search for Scientific Laws?

Jianing Shi; Zaibin Zhang; Shijun Lei; Zhenfei Yin

cs.MAOct 16, 2026

The Weight of Teamwork: Does Communication Structure Help LLM Scientists Search for Scientific Laws?

Authors: Jianing Shi, Zaibin Zhang, Shijun Lei, Zhenfei Yin

Organizations: The University of Chicago (uchicago.edu) · Dalian University of Technology (dlut.edu.cn) · Northwestern Polytechnical University (nwpu.edu.cn) · University of Oxford (ox.ac.uk)

Abstract

Team performance is often reported as one delivered score, even though that score is produced in two stages: a system generates candidate answers and a selection rule returns one of them. We use symbolic regression as an observable scientific-search setting and compare fixed-budget LLM teams that differ in peer visibility and update timing, evaluating both their frozen candidate archives and the expressions selected from those archives. Distributing the call budget across more independent trajectories produced the largest measured gain, improving mean held-out fitting score Q by 0.70 at the widest tested allocation. Under a common archive-wide selector, the implemented Ring policy reduced mean Q by 0.33 relative to Independent, while Star and Complete remained unresolved. Holding the generated archives fixed, changing the delivery rule changed the measured architecture gaps, including the sign of the Complete point estimate. These results support reporting candidate generation and final selection as separate endpoints. They do not identify a communication mechanism or a universally preferred topology; rather, they show that an architecture ranking is not identified by communication structure alone when the rule that returns the final answer is left unspecified.

Figures & tables

Figure 1: Information available to each agent. Five panels show Solo, Independent, Ring, Star, and Complete. Arrows indicate whose responses a recipient can read. In Star, solid arrows run from hub to leaves and dashed arrows from leaves to hub. Ring uses the latest predecessor response in rotating serial order; Star updates leaves before the hub; Complete uses a previous-round snapshot. Solo and Independent have self-history without peer visibility. All panels represent 50 scheduled calls per problem. The policies vary visibility and update timing together, so graph connectivity alone does not specify the treatment.

Contrast	Mean $\Delta Q$ [95% CI]	Median	W/L/T	$r_{rb}$	Holm $p$
Ring - Independent	-0.33 [-0.49, -0.18]	-0.01	21/75/4	-0.61	$5.75\times 10^{-7}$
Star - Independent	+0.03 [-0.08, +0.13]	$-5.64\times 10^{-7}$	43/50/7	-0.01	0.91
Complete - Independent	+0.10 [-0.08, +0.34]	$+1.73\times 10^{-5}$	53/42/5	+0.16	0.36

Table 1: Communication comparisons and allocation reference. Mean held-out

Q

differences and 95% paired bootstrap intervals are primary. Medians, wins/losses/ties, signed-rank rank-biserial correlations (

r_{rb}

), and Holm-adjusted Wilcoxon values are complementary summaries. Panel A contains the three exploratory communication contrasts at

N=5

. Panel B varies

N

without peer communication, retaining 50 calls per problem. Positive allocation differences favor the larger

N

. Appendix F gives the full stratified results.

Figure 2: Policy gaps across delivery rules. Points show Qwen mean held-out

Q

differences from Independent for Solo, Ring, Star, and Complete under each rule; error bars are 95% paired bootstrap intervals. Lines connect discrete rules and do not represent a continuous intervention. Both policies use the same rule on fixed archives. Valid pairs range from 96 to 100; Appendix D lists the cohorts. The vertical range is [-2, 2]. Outward arrows mark clipped

d_{5}

upper limits: Solo 4.13, Ring 4.21, and Complete 3.12. The zero line is labeled “same as Independent.” Appendix D classifies sign changes using simple-effect uncertainty together with corrected interactions.

Rule	Mean $\Delta Q$ [95% CI]	Median	Interaction Holm $p$
$d_{1}$ : archive minimum training error	+0.10 [-0.08, +0.34]	$+1.73\times 10^{-5}$	Reference
$d_{2}$ : final-round minimum training error	-0.75 [-1.56, -0.25]	$-6.10\times 10^{-3}$	$1.05\times 10^{-7}$
$d_{3}$ : final-round plurality	-0.78 [-1.59, -0.28]	$-9.58\times 10^{-3}$	$6.14\times 10^{-8}$
$d_{4}$ : archive-wide plurality	-0.59 [-1.40, -0.10]	$-1.38\times 10^{-3}$	0.07

Table 2: Complete minus Independent under the focal delivery rules. All four rows use 100 paired Qwen problems. Means and intervals describe simple effects; the last column tests each rule’s change from

d_{1}

using the prespecified family of 20 signed-rank interactions. The focal emphasis on

d_{2}

–

d_{4}

is exploratory. The common-rule mean interval includes zero. Appendix D gives all architectures, rules, and sign-change classifications, including the supported Ring case under the weakest rule, the last-message stress test

d_{5}

Figure 3: Fixed-budget independent exploration. (A) Mean held-out

Q

N=1,2,5,10

, with 95% problem-bootstrap intervals and 50 calls per problem. The dashed line marks

Q=0

(NMSE approximately 1). (B) Counts with NMSE

<1

and an exploratory descriptive count below the strict lower-tail threshold

Q_{10}(N=5)=-0.76

. Selection uses archive-wide training fit. The positive paired gains do not imply that the absolute mean at

N=10

exceeds zero with the same confidence.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Condition	Agents or trajectories	Rounds per trajectory	Scheduled calls per problem
Matched Solo	1	50	50
Independent, N=2	2	25	50
Independent, N=5	5	10	50
Independent, N=10	10	5	50
Ring	5	10	50
Star	5	10	50

Appendix

Table 6

Setting	Qwen	Terra
Model identifier	qwen2.5:7b-instruct	gpt-5.6-terra
Service	Local Ollama, OpenAI-compatible endpoint	Official OpenAI endpoint
Temperature	0.70	1.00
Output limit	max_tokens=512	max_completion_tokens=2048
Reasoning setting	No reasoning_effort field	reasoning_effort=none
Displayed training examples	5	5

Appendix

Table 7

Policy	Visible peer responses	Update timing
Solo	None	One trajectory receives its own previous responses.
Independent	None	Each trajectory receives its own previous responses.
Ring	The latest response from the predecessor	Agents run in a rotating serial order. A prompt uses the latest predecessor response available at that call.
Star	Leaves receive the latest hub response; the hub receives the current leaf responses	Each round has a leaf phase followed by a hub phase.
Complete	Previous-round responses from all four peers	Each round uses a fixed previous-round snapshot, preventing access to same-round peer updates.

Appendix

Table 8

Rule	Pool and selection objective	Tie handling
d1: archive minimum training error	All valid messages across agents and rounds; minimum finite training NMSE	Lexicographically smallest message_id
d2: final-round minimum training error	Valid messages in the maximum round_idx ; minimum finite training NMSE	Earliest message in archived message order
d3: final-round plurality	Valid messages in the maximum round_idx , grouped by frozen canonical skeleton; largest group	Choose the minimum-training-NMSE representative within each tied group, using archive order for equal errors; choose among tied-group representatives by the same criteria
d4: global plurality	All valid messages grouped by frozen canonical skeleton; largest group	Same procedure as d3
d5: last message	The final archived message, if valid	No tie
d6: complexity penalty	All valid messages parseable by SymPy ( Meurer et al., 2017 ) ; minimum training NMSE plus normalized complexity penalty	Earliest message in archived message order

Appendix

Table 9

Analysis family	Comparisons in the family	Correction
Qwen allocation	1 to 2, 2 to 5, 5 to 10, 1 to 5, and 1 to 10	Holm across 5 tests
Pooled Qwen communication	Ring, Star, and Complete minus Independent under d1	Holm across 3 exploratory tests
Within-stratum allocation	Independent minus matched Solo	One-test family; Holm value equals raw value
Within-stratum communication	Ring, Star, and Complete minus Independent	Holm across 3 tests within each stratum/layer
Within-stratum policy comparisons	The three pairwise comparisons among Ring, Star, and Complete	Holm across 3 tests within each stratum/layer
Delivery simple effects	Each architecture minus Independent under each delivery rule	Exported simple-effect $p$ values are raw

Appendix

Table 10

Figure C1: Frozen common-rule held-out ID score distributions across the 100 Qwen tasks. Q=0 corresponds approximately to NMSE=1.

Figure C2: Restricted OOD sensitivity on the selected Synth tasks only. These tasks have a pregenerated OOD array; the Transform tasks do not. Q=0 corresponds approximately to NMSE=1.

Architecture	Rule	n	W/L/T	Mean delta Q [95% CI]	Median	Raw p
Complete	d1	100	53/42/5	0.10 [-0.08, 0.34]	$1.73\!\times\!10^{-5}$	0.18
Complete	d2	100	29/65/6	-0.75 [-1.56, -0.25]	$-6.10\times 10^{-3}$	$7.27\!\times\!10^{-6}$
Complete	d3	100	28/65/7	-0.78 [-1.59, -0.28]	$-9.58\times 10^{-3}$	$1.86\!\times\!10^{-6}$
Complete	d4	100	32/60/8	-0.59 [-1.40, -0.10]	$-1.38\times 10^{-3}$	$3.63\times 10^{-3}$
Complete	d5	98	54/39/5	1.22 [0.11, 3.12]	$3.93\!\times\!10^{-4}$	0.02
Complete	d6	99	53/39/7	0.10 [-0.07, 0.35]	$2.56\!\times\!10^{-4}$	0.10

Appendix

Table D1: Qwen delivery simple effects.

Architecture	Rule	n	W/L/T	Mean delta Q [95% CI]	Raw p
Complete	d1	41	4/13/24	-0.36 [-1.43, 0.73]	0.08
Complete	d2	41	3/19/19	-0.90 [-2.00, 0.10]	$2.40\times 10^{-3}$
Complete	d3	41	4/15/22	-0.74 [-1.88, 0.30]	0.03
Complete	d4	41	6/12/23	-0.92 [-2.05, -0.02]	0.05
Complete	d5	41	11/11/19	$1.08\times 10^{-3}$ [-0.20, 0.20]	0.91
Complete	d6	40	4/13/23	-0.37 [-1.49, 0.75]	0.08

Appendix

Table D2: Terra delivery simple effects.

Architecture	Rule	n	W/L/T	Mean interaction [95% CI]	Median interaction	Raw p	Holm p
Complete	d2	100	20/69/11	-0.84 [-1.85, -0.26]	-0.01	$6.99\!\times\!10^{-9}$	$1.05\!\times\!10^{-7}$
Ring	d2	100	32/30/38	-0.06 [-0.16, 0.02]	0	0.64	1
Star	d2	99	14/67/18	-0.51 [-0.84, -0.27]	-0.02	$8.15\!\times\!10^{-10}$	$1.39\!\times\!10^{-8}$
Solo	d2	100	31/26/43	-0.08 [-0.20, $6.37\times 10^{-3}$ ]	0	0.45	1
Complete	d3	100	21/72/7	-0.87 [-1.88, -0.28]	-0.03	$3.84\!\times\!10^{-9}$	$6.14\!\times\!10^{-8}$
Ring	d3	100	38/29/33	-0.05 [-0.15, 0.03]	0	0.80	1

Appendix

Table D3: Qwen d1-referenced delivery interactions.

Architecture	Rule	n	W/L/T	Mean interaction [95% CI]	Raw p	Holm p
Complete	d2	41	6/15/20	-0.54 [-1.51, 0.35]	0.05	0.89
Ring	d2	41	8/7/26	0.29 [-0.53, 1.08]	0.31	1
Star	d2	41	6/18/17	-0.77 [-1.91, 0.22]	0.04	0.75
Solo	d2	41	7/9/25	-0.32 [-1.40, 0.66]	0.68	1
Complete	d3	41	8/13/20	-0.38 [-1.40, 0.55]	0.31	1
Ring	d3	41	10/9/22	0.15 [-0.80, 1.06]	0.57	1

Appendix

Table D4: Terra d1-referenced delivery interactions.

Candidate sign change	Source classification	Scope
Complete d2	Point-estimate sign change	The d1 advantage is not established.
Complete d3	Point-estimate sign change	The d1 advantage is not established.
Complete d4	Not supported after correction	The original interaction already fails Holm correction.
Star d5	Not supported after correction	The interaction does not establish a change.
Ring d5	Supported sign-change classification	Confined to the last-message stress test, the weakest rule in this batch.

Appendix

Table 17

Figure D1: Full Qwen architecture by delivery-rule matrix on frozen candidate archives. Cells summarize held-out ID Q; zero corresponds approximately to NMSE=1. Availability depends on the selector.

Figure D2: Within-layer delivery rankings for Qwen and the Terra Transform subset. Rankings are descriptive; effect magnitudes are not compared across model layers.

N	Mean Q [95% CI]	Median	Q1	Q3	NMSE < 1	NMSE < 0.10	NMSE < 0.01
1	-0.53 [-0.71, -0.36]	-0.12	-0.84	-0.01	11	0	0
2	-0.30 [-0.45, -0.15]	-0.06	-0.45	$-5.62\times 10^{-3}$	12	3	0
5	-0.10 [-0.19, $5.49\times 10^{-3}$ ]	-0.03	-0.18	$-3.60\!\times\!10^{-5}$	19	2	1
10	0.17 [-0.08, 0.49]	$-1.70\times 10^{-3}$	-0.12	0.05	38	4	3

Appendix

Table E1: Absolute Qwen endpoints.

Figure E1: Median held-out ID Q under fixed-budget allocation. This descriptive curve complements the mean and confidence intervals in Figure 3 ; Q=0 corresponds approximately to NMSE=1.

Contrast	n	Mean delta Q [95% CI]	Median	W/L/T	r_rb	Raw p	Holm p
Independent - Matched Solo	50	0.35 [0.16, 0.56]	0.07	34/12/4	0.60	$2.31\!\times\!10^{-4}$	$2.31\!\times\!10^{-4}$
Ring - Independent	50	-0.30 [-0.48, -0.14]	-0.02	10/36/4	-0.64	$7.84\!\times\!10^{-5}$	$2.35\!\times\!10^{-4}$
Star - Independent	50	0.11 [-0.03, 0.27]	0	24/19/7	0.19	0.29	0.29
Complete - Independent	50	0.23 [-0.08, 0.70]	$1.73\!\times\!10^{-5}$	28/17/5	0.28	0.11	0.21
Ring - Star	50	-0.41 [-0.62, -0.24]	-0.02	9/36/5	-0.75	$2.29\!\times\!10^{-6}$	$4.57\!\times\!10^{-6}$
Ring - Complete	50	-0.53 [-1.00, -0.22]	-0.19	7/39/4	-0.79	$4.36\!\times\!10^{-7}$	$1.31\!\times\!10^{-6}$

Appendix

Table 22

Contrast	n	Mean delta Q [95% CI]	Median	W/L/T	r_rb	Raw p	Holm p
Independent - Matched Solo	50	0.52 [0.26, 0.84]	0.04	44/6/0	0.85	$5.80\!\times\!10^{-9}$	$5.80\!\times\!10^{-9}$
Ring - Independent	50	-0.35 [-0.63, -0.12]	-0.01	11/39/0	-0.58	$2.21\!\times\!10^{-4}$	$6.63\!\times\!10^{-4}$
Star - Independent	50	-0.06 [-0.21, 0.07]	$-2.03\times 10^{-3}$	19/31/0	-0.21	0.20	0.40
Complete - Independent	50	-0.04 [-0.18, 0.09]	$6.44\!\times\!10^{-5}$	25/25/0	0.03	0.84	0.84
Ring - Star	50	-0.30 [-0.55, -0.08]	-0.01	14/36/0	-0.50	$1.78\times 10^{-3}$	$3.56\times 10^{-3}$
Ring - Complete	50	-0.32 [-0.61, -0.07]	-0.02	12/38/0	-0.57	$3.24\!\times\!10^{-4}$	$9.72\!\times\!10^{-4}$

Appendix

Table 23

Figure F1: Absolute median Qwen held-out ID scores by task family under the common selector. Q=0 corresponds approximately to NMSE=1. The displayed family-specific ordering is descriptive and does not establish an interaction.

Policy	Problems meeting the concentration threshold
Solo	81
Ring	72
Star	56
Complete	46
Independent	0

Appendix

Table 25

Figure G1: Exploratory archived best-so-far trajectories versus cumulative calls. Candidate selection and the evaluation split follow the plotted analysis. These archive diagnostics do not identify causal peer influence.

Figure G2: Exploratory depth-return curves under the fixed-budget allocations. A flat aggregate curve does not imply that every individual trajectory is unchanged. Q=0 corresponds approximately to NMSE=1.

Policy	Original excess overlap	After token stripping
Ring	0.15	0.15
Star	0.11	0.12
Complete	0.02	0.03

Appendix

Table 28

Figure H1: Exploratory function-space dispersion over checkpoints. Between-problem summaries preserve the analysis-specific valid-unit denominators.

Figure H2: Exploratory symbolic dispersion using operator/variable multiset Jaccard, not tree edit distance.

Figure H3: Exploratory normalized symbolic-to-function dispersion ratio. Solo’s extreme ratios result from positive function-dispersion denominators near numerical zero; the existing analysis excludes zero/nonfinite denominators as undefined but retains finite near-zero cases without clipping, so these magnitudes are numerically ill-conditioned, not evidence of a large effect.

Figure H4: Illustration for I.44.4_4_0 (Problem 34): Complete and Ring in function and symbolic spaces. PCA and MDS are fitted on this problem only; axes have no across-problem meaning. The reference is marked separately. This is a single-task illustration, not a statistical conclusion.

Contrast	Mean delta Q [95% CI]	W/L/T	Raw p	Holm p	Family
Independent - Solo	+1.04 [0.06, 2.07]	18/3/20	$1.87\times 10^{-3}$	$1.87\times 10^{-3}$	Allocation, 1 test
Ring - Independent	-1.18 [-2.04, -0.48]	2/19/20	$5.43\!\times\!10^{-4}$	$1.63\times 10^{-3}$	Communication, 3 tests
Star - Independent	-0.57 [-1.56, +0.40]	3/14/24	0.01	0.03	Communication, 3 tests
Complete - Independent	-0.36 [-1.43, +0.73]	4/13/24	0.08	0.08	Communication, 3 tests
Ring - Star	-0.61 [-1.66, +0.30]	6/13/22	0.06	0.13	Policy, 3 tests
Ring - Complete	-0.82 [-1.93, +0.15]	4/13/24	0.04	0.11	Policy, 3 tests

Appendix

Table 33

Figure I1: Descriptive Terra held-out ID quantiles on the common 41-task Transform cohort. Q=0 corresponds approximately to NMSE=1. This restricted layer does not establish robustness on Synth.

Transform contrast	Historical raw $p$	Historical Holm $p$	Full-vector raw $p$	Full-vector Holm $p$
Independent - matched Solo	$2.31\!\times\!10^{-4}$	$2.31\!\times\!10^{-4}$	$3.76\!\times\!10^{-4}$	$3.76\!\times\!10^{-4}$
Ring - Independent	$7.84\!\times\!10^{-5}$	$2.35\!\times\!10^{-4}$	$1.53\!\times\!10^{-4}$	$4.60\!\times\!10^{-4}$
Star - Independent	0.29	0.29	0.28	0.28
Complete - Independent	0.11	0.21	0.11	0.21
Ring - Star	$2.29\!\times\!10^{-6}$	$4.57\!\times\!10^{-6}$	$1.04\!\times\!10^{-5}$	$2.09\!\times\!10^{-5}$
Ring - Complete	$4.36\!\times\!10^{-7}$	$1.31\!\times\!10^{-6}$	$3.17\!\times\!10^{-6}$	$9.50\!\times\!10^{-6}$

Appendix

Table 35

Item	Execution record and treatment in the analysis
API constraints	On August 26, GPT-5-family probes rejected max_tokens and temperature 0.70. The runner switched these requests to max_completion_tokens ; the Terra layer used temperature 1.00 and reasoning_effort=none . Model, reasoning setting, and temperature are treated as a separate layer.
Matched Solo	The July Solo condition had 1 x 10=10 calls per problem, versus 5 x 10=50 in the five-agent conditions. pilot_solo_matched , launched August 26 and completed August 27, used 1 x 50=50 and superseded the July Solo run for fixed-budget inference. Historical outputs remain preserved.
N=2 continuation	The first request for manifest index 130, BPG5, timed out after 69 completed problems and 3,450 persisted calls. The exact-parameter resume skipped those 69 completed problems, restarted index 130 from its first call, and completed all 100 problems. The timed-out request produced no usable model output.
Terra Synth	Terra Solo Synth completed 50 tasks, but the Independent Synth run stopped on its first request with HTTP 429 credit_balance_exhausted : 0/50 completed problems, zero persisted successful calls, and USD 0.00. No result is imputed.
Post-hoc diversity analysis	A prespecified stopping rule on the pooled paired correlation was triggered on September 5 and the downstream analyses were not run. The stratified test was then executed as a recorded deviation, with the reason and the fixed assignment rule stated in Appendix G.3. No new model calls were made; the frozen archives and their aggregate SHA-256 were unchanged.
Removed self-reproduction statistic	The multi-agent lag-5 result was removed from inference on September 4. For length-10 sequences, the approved circular-shift null is invariant at lag 5 and produces a degenerate point-mass null; $p=1$ is not informative. Multi-agent lags 1–4 remain available, and Solo lags 1–10 are descriptive.

Appendix

Table 36

Multi-LLM Collaboration

Multi-LLM collaboration studies how large language models share work, exchange information, and combine outputs to improve capability, reliability, or cost.

See all 85 papers

Can LLMs Discover Scientific Laws in Real and Parallel Worlds?
Sep 1, 2026Yiming Huang, Ziche Liu, Junxia Cui +11Scientific Discovery Scientific Hypothesis Generation
University of California San Diego · University of California, Irvine · Cushing Academy +1
ScienceArena: Benchmarking LLMs on Latest Scientific Olympiad Competitions
Aug 31, 2026Guangxiang Zhao, Qilong Shi, Xusen Xiao +13LLM Evaluation Scientific Reasoning
1Qiyuan Tech · 2Tsinghua University · 3The University of Hong Kong +1
The Little Scientist: LLM Agent-Driven Discovery via the Scientific Method
Aug 16, 2026Travis SmithAI Agents for Scientific Discovery Automated Algorithm Discovery
Independent Researcher, S˜ao Miguel, Azores, Portugal

Sep 1, 2026J/K move · Enter open · S save

Can LLMs Discover Scientific Laws in Real and Parallel Worlds?

Yiming Huang, Ziche Liu, Junxia Cui, Zhuohang Wu+10

University of California San Diego · University of California, Irvine · Cushing Academy+1

Scientific Discovery Scientific Hypothesis Generation Large Language Model-Guided Scientific Discovery

Open paper AskPDF arXiv

Scientific law discovery has long been central to scientific progress, proceeding through iterative cycles of generating hypotheses, testing them against empirical evidence, and refining them under scientific constraints. As large language models (LLMs) become increasingly involved in scientific research, whether they can discover scientific laws and how to evaluate this ability remain open questions. A central evaluation challenge is to move beyond familiar published equations while keeping discovery tasks grounded in scientific data and constraints. We introduce SciLaws-Bench, a curated collection of scientific task packages grounded in the source literature, each linking a scientific problem, supporting data, published reference equations, and scientific-validity rubrics. Through agent-assisted curation and human verification, we assemble 118 problems spanning six disciplines, drawing on 381 papers, 291 candidate laws, and roughly 8M data points. Each problem supports two complementary evaluation settings. SciLaws-Real uses fixed scientific data to evaluate proposed laws for held-out predictive fit and scientific validity. SciLaws-Parallel evaluates recovery of a newly synthesized structural variant of a published equation through active queries to a simulator calibrated to the source data. Our evaluation reveals three limitations: good predictive fit need not imply scientific validity, recovering a published formula does not establish recovery of its new structural terms, and candidate selection remains a bottleneck in scientific law discovery. Project page: https://yiyihum.github.io/SciLaws-Bench

Abstract

Multi-LLM Collaboration

Explore similar work

Can LLMs Discover Scientific Laws in Real and Parallel Worlds?

ScienceArena: Benchmarking LLMs on Latest Scientific Olympiad Competitions

The Little Scientist: LLM Agent-Driven Discovery via the Scientific Method