Organizations: The University of Chicago (uchicago.edu) · Dalian University of Technology (dlut.edu.cn) · Northwestern Polytechnical University (nwpu.edu.cn) · University of Oxford (ox.ac.uk)
Team performance is often reported as one delivered score, even though that score is produced in two stages: a system generates candidate answers and a selection rule returns one of them. We use symbolic regression as an observable scientific-search setting and compare fixed-budget LLM teams that differ in peer visibility and update timing, evaluating both their frozen candidate archives and the expressions selected from those archives. Distributing the call budget across more independent trajectories produced the largest measured gain, improving mean held-out fitting score Q by 0.70 at the widest tested allocation. Under a common archive-wide selector, the implemented Ring policy reduced mean Q by 0.33 relative to Independent, while Star and Complete remained unresolved. Holding the generated archives fixed, changing the delivery rule changed the measured architecture gaps, including the sign of the Complete point estimate. These results support reporting candidate generation and final selection as separate endpoints. They do not identify a communication mechanism or a universally preferred topology; rather, they show that an architecture ranking is not identified by communication structure alone when the rule that returns the final answer is left unspecified.
Figures & tables
Figure 1: Information available to each agent. Five panels show Solo, Independent, Ring, Star, and Complete. Arrows indicate whose responses a recipient can read. In Star, solid arrows run from hub to leaves and dashed arrows from leaves to hub. Ring uses the latest predecessor response in rotating serial order; Star updates leaves before the hub; Complete uses a previous-round snapshot. Solo and Independent have self-history without peer visibility. All panels represent 50 scheduled calls per problem. The policies vary visibility and update timing together, so graph connectivity alone does not specify the treatment.
Contrast
Mean ΔQ [95% CI]
Median
W/L/T
rrb
Holm p
Ring - Independent
-0.33 [-0.49, -0.18]
-0.01
21/75/4
-0.61
5.75×10−7
Star - Independent
+0.03 [-0.08, +0.13]
−5.64×10−7
43/50/7
-0.01
0.91
Complete - Independent
+0.10 [-0.08, +0.34]
+1.73×10−5
53/42/5
+0.16
0.36
Table 1: Communication comparisons and allocation reference. Mean held-out Q differences and 95% paired bootstrap intervals are primary. Medians, wins/losses/ties, signed-rank rank-biserial correlations ( rrb ), and Holm-adjusted Wilcoxon values are complementary summaries. Panel A contains the three exploratory communication contrasts at N=5 . Panel B varies N without peer communication, retaining 50 calls per problem. Positive allocation differences favor the larger N . Appendix F gives the full stratified results.
Figure 2: Policy gaps across delivery rules. Points show Qwen mean held-out Q differences from Independent for Solo, Ring, Star, and Complete under each rule; error bars are 95% paired bootstrap intervals. Lines connect discrete rules and do not represent a continuous intervention. Both policies use the same rule on fixed archives. Valid pairs range from 96 to 100; Appendix D lists the cohorts. The vertical range is [-2, 2]. Outward arrows mark clipped d5 upper limits: Solo 4.13, Ring 4.21, and Complete 3.12. The zero line is labeled “same as Independent.” Appendix D classifies sign changes using simple-effect uncertainty together with corrected interactions.
Rule
Mean ΔQ [95% CI]
Median
Interaction Holm p
d1 : archive minimum training error
+0.10 [-0.08, +0.34]
+1.73×10−5
Reference
d2 : final-round minimum training error
-0.75 [-1.56, -0.25]
−6.10×10−3
1.05×10−7
d3 : final-round plurality
-0.78 [-1.59, -0.28]
−9.58×10−3
6.14×10−8
d4 : archive-wide plurality
-0.59 [-1.40, -0.10]
−1.38×10−3
0.07
Table 2: Complete minus Independent under the focal delivery rules. All four rows use 100 paired Qwen problems. Means and intervals describe simple effects; the last column tests each rule’s change from d1 using the prespecified family of 20 signed-rank interactions. The focal emphasis on d2 – d4 is exploratory. The common-rule mean interval includes zero. Appendix D gives all architectures, rules, and sign-change classifications, including the supported Ring case under the weakest rule, the last-message stress test d5 .
Figure 3: Fixed-budget independent exploration. (A) Mean held-out Q at N=1,2,5,10 , with 95% problem-bootstrap intervals and 50 calls per problem. The dashed line marks Q=0 (NMSE approximately 1). (B) Counts with NMSE <1 and an exploratory descriptive count below the strict lower-tail threshold Q10(N=5)=−0.76 . Selection uses archive-wide training fit. The positive paired gains do not imply that the absolute mean at N=10 exceeds zero with the same confidence.
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
Condition
Agents or trajectories
Rounds per trajectory
Scheduled calls per problem
Matched Solo
1
50
50
Independent, N=2
2
25
50
Independent, N=5
5
10
50
Independent, N=10
10
5
50
Ring
5
10
50
Star
5
10
50
Appendix
Table 6
Setting
Qwen
Terra
Model identifier
qwen2.5:7b-instruct
gpt-5.6-terra
Service
Local Ollama, OpenAI-compatible endpoint
Official OpenAI endpoint
Temperature
0.70
1.00
Output limit
max_tokens=512
max_completion_tokens=2048
Reasoning setting
No reasoning_effort field
reasoning_effort=none
Displayed training examples
5
5
Appendix
Table 7
Policy
Visible peer responses
Update timing
Solo
None
One trajectory receives its own previous responses.
Independent
None
Each trajectory receives its own previous responses.
Ring
The latest response from the predecessor
Agents run in a rotating serial order. A prompt uses the latest predecessor response available at that call.
Star
Leaves receive the latest hub response; the hub receives the current leaf responses
Each round has a leaf phase followed by a hub phase.
Complete
Previous-round responses from all four peers
Each round uses a fixed previous-round snapshot, preventing access to same-round peer updates.
Appendix
Table 8
Rule
Pool and selection objective
Tie handling
d1: archive minimum training error
All valid messages across agents and rounds; minimum finite training NMSE
Lexicographically smallest message_id
d2: final-round minimum training error
Valid messages in the maximum round_idx ; minimum finite training NMSE
Earliest message in archived message order
d3: final-round plurality
Valid messages in the maximum round_idx , grouped by frozen canonical skeleton; largest group
Choose the minimum-training-NMSE representative within each tied group, using archive order for equal errors; choose among tied-group representatives by the same criteria
d4: global plurality
All valid messages grouped by frozen canonical skeleton; largest group
Same procedure as d3
d5: last message
The final archived message, if valid
No tie
d6: complexity penalty
All valid messages parseable by SymPy ( Meurer et al., 2017 ) ; minimum training NMSE plus normalized complexity penalty
Earliest message in archived message order
Appendix
Table 9
Analysis family
Comparisons in the family
Correction
Qwen allocation
1 to 2, 2 to 5, 5 to 10, 1 to 5, and 1 to 10
Holm across 5 tests
Pooled Qwen communication
Ring, Star, and Complete minus Independent under d1
Holm across 3 exploratory tests
Within-stratum allocation
Independent minus matched Solo
One-test family; Holm value equals raw value
Within-stratum communication
Ring, Star, and Complete minus Independent
Holm across 3 tests within each stratum/layer
Within-stratum policy comparisons
The three pairwise comparisons among Ring, Star, and Complete
Holm across 3 tests within each stratum/layer
Delivery simple effects
Each architecture minus Independent under each delivery rule
Exported simple-effect p values are raw
Appendix
Table 10
Figure C1: Frozen common-rule held-out ID score distributions across the 100 Qwen tasks. Q=0 corresponds approximately to NMSE=1.
Figure C2: Restricted OOD sensitivity on the selected Synth tasks only. These tasks have a pregenerated OOD array; the Transform tasks do not. Q=0 corresponds approximately to NMSE=1.
Table D4: Terra d1-referenced delivery interactions.
Candidate sign change
Source classification
Scope
Complete d2
Point-estimate sign change
The d1 advantage is not established.
Complete d3
Point-estimate sign change
The d1 advantage is not established.
Complete d4
Not supported after correction
The original interaction already fails Holm correction.
Star d5
Not supported after correction
The interaction does not establish a change.
Ring d5
Supported sign-change classification
Confined to the last-message stress test, the weakest rule in this batch.
Appendix
Table 17
Figure D1: Full Qwen architecture by delivery-rule matrix on frozen candidate archives. Cells summarize held-out ID Q; zero corresponds approximately to NMSE=1. Availability depends on the selector.
Figure D2: Within-layer delivery rankings for Qwen and the Terra Transform subset. Rankings are descriptive; effect magnitudes are not compared across model layers.
N
Mean Q [95% CI]
Median
Q1
Q3
NMSE < 1
NMSE < 0.10
NMSE < 0.01
1
-0.53 [-0.71, -0.36]
-0.12
-0.84
-0.01
11
0
0
2
-0.30 [-0.45, -0.15]
-0.06
-0.45
−5.62×10−3
12
3
0
5
-0.10 [-0.19, 5.49×10−3 ]
-0.03
-0.18
−3.60×10−5
19
2
1
10
0.17 [-0.08, 0.49]
−1.70×10−3
-0.12
0.05
38
4
3
Appendix
Table E1: Absolute Qwen endpoints.
Figure E1: Median held-out ID Q under fixed-budget allocation. This descriptive curve complements the mean and confidence intervals in Figure 3 ; Q=0 corresponds approximately to NMSE=1.
Contrast
n
Mean delta Q [95% CI]
Median
W/L/T
r_rb
Raw p
Holm p
Independent - Matched Solo
50
0.35 [0.16, 0.56]
0.07
34/12/4
0.60
2.31×10−4
2.31×10−4
Ring - Independent
50
-0.30 [-0.48, -0.14]
-0.02
10/36/4
-0.64
7.84×10−5
2.35×10−4
Star - Independent
50
0.11 [-0.03, 0.27]
0
24/19/7
0.19
0.29
0.29
Complete - Independent
50
0.23 [-0.08, 0.70]
1.73×10−5
28/17/5
0.28
0.11
0.21
Ring - Star
50
-0.41 [-0.62, -0.24]
-0.02
9/36/5
-0.75
2.29×10−6
4.57×10−6
Ring - Complete
50
-0.53 [-1.00, -0.22]
-0.19
7/39/4
-0.79
4.36×10−7
1.31×10−6
Appendix
Table 22
Contrast
n
Mean delta Q [95% CI]
Median
W/L/T
r_rb
Raw p
Holm p
Independent - Matched Solo
50
0.52 [0.26, 0.84]
0.04
44/6/0
0.85
5.80×10−9
5.80×10−9
Ring - Independent
50
-0.35 [-0.63, -0.12]
-0.01
11/39/0
-0.58
2.21×10−4
6.63×10−4
Star - Independent
50
-0.06 [-0.21, 0.07]
−2.03×10−3
19/31/0
-0.21
0.20
0.40
Complete - Independent
50
-0.04 [-0.18, 0.09]
6.44×10−5
25/25/0
0.03
0.84
0.84
Ring - Star
50
-0.30 [-0.55, -0.08]
-0.01
14/36/0
-0.50
1.78×10−3
3.56×10−3
Ring - Complete
50
-0.32 [-0.61, -0.07]
-0.02
12/38/0
-0.57
3.24×10−4
9.72×10−4
Appendix
Table 23
Figure F1: Absolute median Qwen held-out ID scores by task family under the common selector. Q=0 corresponds approximately to NMSE=1. The displayed family-specific ordering is descriptive and does not establish an interaction.
Policy
Problems meeting the concentration threshold
Solo
81
Ring
72
Star
56
Complete
46
Independent
0
Appendix
Table 25
Figure G1: Exploratory archived best-so-far trajectories versus cumulative calls. Candidate selection and the evaluation split follow the plotted analysis. These archive diagnostics do not identify causal peer influence.
Figure G2: Exploratory depth-return curves under the fixed-budget allocations. A flat aggregate curve does not imply that every individual trajectory is unchanged. Q=0 corresponds approximately to NMSE=1.
Policy
Original excess overlap
After token stripping
Ring
0.15
0.15
Star
0.11
0.12
Complete
0.02
0.03
Appendix
Table 28
Figure H1: Exploratory function-space dispersion over checkpoints. Between-problem summaries preserve the analysis-specific valid-unit denominators.
Figure H2: Exploratory symbolic dispersion using operator/variable multiset Jaccard, not tree edit distance.
Figure H3: Exploratory normalized symbolic-to-function dispersion ratio. Solo’s extreme ratios result from positive function-dispersion denominators near numerical zero; the existing analysis excludes zero/nonfinite denominators as undefined but retains finite near-zero cases without clipping, so these magnitudes are numerically ill-conditioned, not evidence of a large effect.
Figure H4: Illustration for I.44.4_4_0 (Problem 34): Complete and Ring in function and symbolic spaces. PCA and MDS are fitted on this problem only; axes have no across-problem meaning. The reference is marked separately. This is a single-task illustration, not a statistical conclusion.
Contrast
Mean delta Q [95% CI]
Median
W/L/T
Raw p
Holm p
Family
Independent - Solo
+1.04 [0.06, 2.07]
0
18/3/20
1.87×10−3
1.87×10−3
Allocation, 1 test
Ring - Independent
-1.18 [-2.04, -0.48]
0
2/19/20
5.43×10−4
1.63×10−3
Communication, 3 tests
Star - Independent
-0.57 [-1.56, +0.40]
0
3/14/24
0.01
0.03
Communication, 3 tests
Complete - Independent
-0.36 [-1.43, +0.73]
0
4/13/24
0.08
0.08
Communication, 3 tests
Ring - Star
-0.61 [-1.66, +0.30]
0
6/13/22
0.06
0.13
Policy, 3 tests
Ring - Complete
-0.82 [-1.93, +0.15]
0
4/13/24
0.04
0.11
Policy, 3 tests
Appendix
Table 33
Figure I1: Descriptive Terra held-out ID quantiles on the common 41-task Transform cohort. Q=0 corresponds approximately to NMSE=1. This restricted layer does not establish robustness on Synth.
Transform contrast
Historical raw p
Historical Holm p
Full-vector raw p
Full-vector Holm p
Independent - matched Solo
2.31×10−4
2.31×10−4
3.76×10−4
3.76×10−4
Ring - Independent
7.84×10−5
2.35×10−4
1.53×10−4
4.60×10−4
Star - Independent
0.29
0.29
0.28
0.28
Complete - Independent
0.11
0.21
0.11
0.21
Ring - Star
2.29×10−6
4.57×10−6
1.04×10−5
2.09×10−5
Ring - Complete
4.36×10−7
1.31×10−6
3.17×10−6
9.50×10−6
Appendix
Table 35
Item
Execution record and treatment in the analysis
API constraints
On August 26, GPT-5-family probes rejected max_tokens and temperature 0.70. The runner switched these requests to max_completion_tokens ; the Terra layer used temperature 1.00 and reasoning_effort=none . Model, reasoning setting, and temperature are treated as a separate layer.
Matched Solo
The July Solo condition had 1 x 10=10 calls per problem, versus 5 x 10=50 in the five-agent conditions. pilot_solo_matched , launched August 26 and completed August 27, used 1 x 50=50 and superseded the July Solo run for fixed-budget inference. Historical outputs remain preserved.
N=2 continuation
The first request for manifest index 130, BPG5, timed out after 69 completed problems and 3,450 persisted calls. The exact-parameter resume skipped those 69 completed problems, restarted index 130 from its first call, and completed all 100 problems. The timed-out request produced no usable model output.
Terra Synth
Terra Solo Synth completed 50 tasks, but the Independent Synth run stopped on its first request with HTTP 429 credit_balance_exhausted : 0/50 completed problems, zero persisted successful calls, and USD 0.00. No result is imputed.
Post-hoc diversity analysis
A prespecified stopping rule on the pooled paired correlation was triggered on September 5 and the downstream analyses were not run. The stratified test was then executed as a recorded deviation, with the reason and the fixed assignment rule stated in Appendix G.3. No new model calls were made; the frozen archives and their aggregate SHA-256 were unchanged.
Removed self-reproduction statistic
The multi-agent lag-5 result was removed from inference on September 4. For length-10 sequences, the approved circular-shift null is invariant at lag 5 and produces a degenerate point-mass null; p=1 is not informative. Multi-agent lags 1–4 remain available, and Solo lags 1–10 are descriptive.
Multi-LLM collaboration studies how large language models share work, exchange information, and combine outputs to improve capability, reliability, or cost.
Scientific law discovery has long been central to scientific progress, proceeding through iterative cycles of generating hypotheses, testing them against empirical evidence, and refining them under scientific constraints. As large language models (LLMs) become increasingly involved in scientific research, whether they can discover scientific laws and how to evaluate this ability remain open questions. A central evaluation challenge is to move beyond familiar published equations while keeping discovery tasks grounded in scientific data and constraints. We introduce SciLaws-Bench, a curated collection of scientific task packages grounded in the source literature, each linking a scientific problem, supporting data, published reference equations, and scientific-validity rubrics. Through agent-assisted curation and human verification, we assemble 118 problems spanning six disciplines, drawing on 381 papers, 291 candidate laws, and roughly 8M data points. Each problem supports two complementary evaluation settings. SciLaws-Real uses fixed scientific data to evaluate proposed laws for held-out predictive fit and scientific validity. SciLaws-Parallel evaluates recovery of a newly synthesized structural variant of a published equation through active queries to a simulator calibrated to the source data. Our evaluation reveals three limitations: good predictive fit need not imply scientific validity, recovering a published formula does not establish recovery of its new structural terms, and candidate selection remains a bottleneck in scientific law discovery. Project page: https://yiyihum.github.io/SciLaws-Bench
Yiming Huang, Ziche Liu, Junxia Cui +11
University of California San Diego · University of California, Irvine · Cushing Academy +1
Benchmark saturation and data contamination increasingly obscure genuine scientific reasoning in frontier LLMs. We introduce ScienceArena, an olympiad-style benchmark from thirteen public science competitions in physics, chemistry, and biology, including IPhO and IChO 2025--2026, IBO 2023, USAPhO 2026, and USNCO 2025. Its open-ended, multi-step problems use process-credit rubrics, making faithful scoring difficult. We build ScienceArena through an expert-audited digitization pipeline that converts official exams, figures, solutions, and rubrics into structured items verified by olympiad medalists. To scale evaluation beyond costly human grading, we calibrate LLM-as-judge against medalist ground truth on archived answers from five models across IPhO and IChO; two strong judges stay within one point of expert total scores. Medalist notes show that failures often stem from visual grounding, structure fidelity, and global problem control rather than missing terminology. Evaluating fourteen recent LLMs with interleaved solving, we find that top models obtain medal-equivalent rubric scores on several public international exams, while chemistry and long-horizon consistency remain key bottlenecks. We provide an interactive demo.
Guangxiang Zhao, Qilong Shi, Xusen Xiao +13
1Qiyuan Tech · 2Tsinghua University · 3The University of Hong Kong +1
What happens when you teach an LLM-based agent the scientific method? Motivation: Scientific discovery emerges from cycles of hypothesis, implementation, empirical testing, and feedback. Can this process be automated? We approach automated algorithm design through the lens of the scientific method, where an LLM-based agent goes through each step of the process in an ordered, iterative fashion. Results: We present The Little Scientist, a framework in which a "Scientist agent" works inside an evaluation environment that benchmarks its code and returns structured per-instance diagnostics. When the Scientist plateaus at a local optimum, a "Kuhn agent" injects a paradigm-shifting conjecture paired with a cross-disciplinary inspiration, forcing exploration of a different region of the LLM's latent space. We demonstrate the framework on two problems that require fundamentally different modes of discovery. For protein fitness prediction, the Scientist discovered Delta V, an ensemble calibration strategy that ranks first on the ProteinGym DMS Substitutions Zero-Shot leaderboard across all five official evaluation metrics, exceeding the #2 model (VenusREM) by +0.033 mean Spearman correlation across 217 DMS assays. For DNA motif discovery, the Scientist wrote an algorithm from scratch--DALE (Dual-seed Algorithm for Latent Enumeration)--that outperforms STREME (the default in the MEME Suite) across 132 ENCODE transcription factors (mean AUROC 0.842 vs. 0.803, Wilcoxon p < 10^{-6}) while running 11x faster. This demonstrates that the framework can produce genuinely novel algorithms, not just optimize existing components. Together, these results show that an LLM agent stepping through the scientific method can discover both new algorithms and new ensemble strategies that outperform prior solutions. The entire research program consumed 704M tokens on a single virtual machine with no GPUs
Travis Smith
Independent Researcher, S˜ao Miguel, Azores, Portugal