Assessing Language Models for Scientific Applications: Inertial Confinement Fusion as a Use Case
Authors: P. B. Radha, D. D. Meyerhofer, M. Bhattarai, B. Cheng, L. Diaz, C. L. Rousculp, A. G. Seaton, G. Gupta, P. Gurumurthy, V. Gutta, A. Jhunjhunwala, A. Vem, P. Cook, G. Rowe
Organizations: Los Alamos National Laboratory, Los Alamos, NM · NVIDIA Corporation, Santa Clara, CA
Advancing language models for specialized scientific applications requires evaluations of scientific fluency, physical reasoning, and quantitative analyses. Using inertial confinement fusion (ICF) as a test case, we have developed an expert-curated benchmark of 250 questions spanning qualitative reasoning, numerical calculation, symbolic derivation, and equation formulation. Responses have been evaluated using deterministic numerical and symbolic checks together with language-model judges. We compared a variety of out-of-the-box closed-source and open-source large language models with Prospero, a model specialized through combined domain-adaptive pretraining and supervised fine-tuning for ICF. We also conducted initial retrieval-assisted evaluations. Prospero improved its aggregate score by 10.7 percentage points over its base model, with gains across all question types. Frontier models achieved the highest aggregate scores, but critical physics errors persisted, particularly in numerical calculations. We note that domain-specific performance can improve through specialization, but benchmark gains alone do not establish scientific reliability. Retrieval improved performance for a smaller model but yielded no statistically significant improvement for a frontier model. Two scientific stress tests further illustrate useful capabilities and the need for expert assessment of physical assumptions and model validity. Based on these findings, we recommend evaluating existing domain capabilities and retrieval benefits before committing to substantial additional specialization.
Figures & tables
Figure 2: Categories of questions developed for the golden-set - the set of 250 question and answer pairs generated by ICF experts. The numbers at the end of the bars indicate the number of questions of each type in the golden-set.
hydroefficiency, shocks, instabilities, compression, energy scaling
Table 1: Examples of topics within each area of the questions in the golden-set. Note that some topics can be listed in multiple categories.
Question Type
Percent in set
Qualitative/Reasoning
29.2
Numerical calculations
25.6
Symbolic derivation/Reasoning
24.0
Equation formulation
20.8
Multiple-choice
0.4
Table 2: Percentage of questions of each type within the golden-set of 250 questions
Figure 3: Workflow used to assess LLM outputs. Blocks 4 and 5 together form an effective single scoring stage: Block 4 lists the physics-aware equivalence criteria (numerical tolerance, unit consistency, algebraic equivalence) that the judges in Block 5 apply when comparing each answer with the answers in the golden-set. API: Application Programming Interface; CAS: Computer Algebra System; 10,000 bootstrap samples: statistical uncertainty estimated from 10,000 resamples of the fixed scores.
Question Type
Rubric Weighting
Numeric
55% value; 15% units and dimensions; 20% method; 10% assumptions and uncertainty.
Weighted mean of explicit subpart scores; every subpart is required for completeness.
Table 3: Scoring rubric across each question type
Figure 4: Pairwise agreement between judges on scores (blue) and on the critical-error rates (orange).
Figure 5: Stability of judge ranking. Consensus among the three judges is used for ranking. The top models are unanimously assessed as the best performing models. Models following those change places but never more than two spots. Only the top eight models of the 22 assessed are shown. Model references in text.
Figure 6: Process followed for Domain adaptive pretraining (DAPT) and supervised fine-tuning (SFT)
Figure 7: Example of an intermediate stage during the training process for the standardized benchmark GPQA. With increasing ICF training or continuous pretraining (CPT) performance drops (blue line) but can be recovered with supervised fine-tuning (SFT). The drop in performance is correlated with the increasing percentage of questions that are unanswered (orange line). Accuracy is recovered when the percentage of unanswered questions drops to zero.
Figure 8: Overall performance of the top 4 models, Prospero and its base (Llama-3.3-Nemotron-49B-Instruct-128K) used in this work on the 250-question ICF benchmark. Bars show the three-judge rubric score; horizontal intervals are 95% confidence intervals from a 10,000-replicate question bootstrap. Each question contributes the mean of three attempts, and missing attempts count as zero.
Figure 9: Heat map showing scores of the top 4 models on the golden-set by ICF subarea/category compared with Prospero, and its base model (Llama-3.3-Nemotron-49B-Instruct-128K).
Figure 10: Heat map showing scores of the top 4 models on the golden-set by question type compared with Prospero and its base model (Llama-3.3-Nemotron-49B-Instruct-128K). The single multiple-choice question is not included.
Figure 11: Prospero performance on golden-set questions versus the base model.
Figure 12: Model performance for (a) the smaller Gemma 4 model; and (b) the larger GPT-5.4 model as a function of the literature available to the model. (c) Gemma 4 improves because its critical or physics errors drop with retrieval whereas GPT-5.4 performance does not improve significantly with additional retrieval provided by URSA or ODR.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 13: Rayleigh-Taylor mode under three explicit assumptions. Purely growing hyperbolic cosine function (solid blue); amplitude from the piecewise method using the terminal bubble velocity v(t0.1) given by Eq. 8 (dashed purple); amplitude from the piecewise method using the terminal bubble velocity vNL given by Eq. 5 (dashed-dot green).
Model
Observations
Claude Sonnet 5
Uses cosh solution for both times and flags nonlinear behavior but does not calculate the appropriate nonlinear value
Inkling Small
Explains both choices for the initial mode, flags nonlinear behavior but does not calculate the appropriate nonlinear value
GPT-5.4
Uses exponential solution for both times; ignores nonlinear behavior
Nemotron Ultra
Uses cosh solution for both times; ignores nonlinear behavior
Appendix
Table 4: Observations about the solutions for the top four performing models. The reference solution uses the cosh form, corresponding to an initially stationary interface. None of the models correctly calculates the nonlinear amplitude using either model, although some flag nonlinearity.
Figure 14: Calculated p- 11 B reactivity (line) compared to values obtained by inspection of Fig. 5 in Ref. [ 53 ] versus ion temperature
Figure 15: Analytic solution for the ratio of the fusion to radiation energy as a function of ion temperature. This is for the analytically derived optimum ratio of boron number fraction. At higher ratios of electron-to-ion temperature no solution is found.
Figure 16: Ignition threshold for a hot spot p- 11 B ignition problem. Sensitivity to uncertainties in fusion reactivity, radiation loss, and alpha particle range is also shown. Some lines are missing as they were studied but the model found no plausible physical result.
Scenario
T i (keV)
T e (keV)
x B
P f /P Br
ρ R (g/cm 2 )
fα
Equal temperature 48.36% recycled
286.9
286.9
0.088
1.22
82.3
0.92
Te/Ti=0.33 No recycling
320.2
106.7
0.088
1.11
58.8
0.97
Te/Ti=0.30 No recycling
306.7
92.0
0.092
1.16
36.6
0.97
Te/Ti=0.10 No recycling
243.7
24.37
0.14
1.74
4.84
0.96
Appendix
Table 5: Conditional power balance solutions identified for the p– 11B ignition model under different temperature and recycling scenarios.
Inertial confinement fusion (ICF) is a leading pathway toward clean energy, but each shot at the National Ignition Facility costs on the order of one million dollars, making accurate AI surrogates a high-value target. We study exogenous-driven ICF waveform prediction, where a 512-step neutron-rate diagnostic must be inferred directly from a laser pulse and target design parameters, with no historical response observed. The regime stresses standard time-series predictors with temporal sparsity (picosecond peak in a nanosecond window), input-output scale mismatch (under 300 real shots), and peak sensitivity (picosecond timing). We propose ICF-DLM, to our knowledge the first LM-based ICF predictor, combining (i) a physics-typed decomposition into yield YDT, peak timing tpeak, and local waveform wlocal; (ii) bidirectional denoising that defers commitment to peak location; and (iii) a physics-driven PPO reward re-injecting metric structure across numeric tokens. On ICFBench (50K simulations + 232 experimental shots), ICF-DLM cuts peak-timing error from 11.6 to 9.2 steps over a matched autoregressive LLaMA-3-8B and outperforms classical sequence models and LLM-based time-series predictors. Beyond ICF, the recipe shows potential to address science domains with low data and sparse events.
Domain-specialized language models are widely used for scientific question answering, but stronger general-purpose systems raise a sharper question: when does domain-specific fine-tuning remain valuable for open-ended scientific reasoning? We study this in astronomy with a curated QA benchmark from publicly available 2017--2026 Olympiad-style materials. The free-response subset contains 300 questions, including 204 text-only and 96 image-linked examples. We compare open-weight and API-served general-purpose, multimodal, and astronomy-specialized models using judge-based correctness and complementary reference metrics. Strong general-purpose models establish the highest correctness baseline in this testbed, while analyses of metric agreement, judge sensitivity, benchmark composition, and modality reveal variation not captured by a single leaderboard. These results motivate treating domain specialization as a task- and deployment-dependent property and highlight the role of domain-specific evaluation in determining which models, capabilities, and evaluation criteria are appropriate for scientific workflows.
Vanessa Lama, Sanjay Das, Emily Herron +5
Oak Ridge National Laboratory, Oak Ridge, Tennessee, USA · The Ohio State University, Columbus, Ohio, USA · Max-Planck-Institut für Astronomie, Königstuhl 17, D-69117 Heidelberg, Germany +1
As language models accelerate scientific research by automating hypothesis generation and implementation, a new bottleneck emerges: evaluating and filtering hundreds of AI-generated ideas without exhaustive experimentation. We ask whether LMs can learn to forecast the empirical success of research ideas before any experiments are run. We study comparative empirical forecasting: given a benchmark-specific research goal and two candidate ideas, predict which will achieve better benchmark performance. We construct a dataset of 11,488 idea pairs grounded in objective outcomes from PapersWithCode. While off-the-shelf 8B-parameter models struggle (30% acc.), SFT dramatically boosts performance to 77.1%, outperforming GPT-5 (61.1%). By framing evaluation as a reasoning task via Reinforcement Learning with Verifiable Rewards (RLVR), we train models to discover latent reasoning paths, achieving 71.35% acc. with interpretable justifications. Through additional ablations and out-of-distribution tests, we show robustness to surface-level heuristics and transfer to both a cross-domain time-split test set and an independently constructed test set. Our results demonstrate that compute-efficient small language models can serve as effective, objective verifiers, offering a scalable path for autonomous scientific discovery.
Srujan P Mule, Aniketh Garikaparthi, Manasi Patwardhan