Organizations: School of Highway, Chang’an University, Xi’an 710064, China · School of Civil Engineering, Southeast University, Nanjing 211189, China · Institute for Risk and Reliability, Leibniz University Hannover, Hannover, 30167, Germany · Department of Civil and Environmental Engineering, University of Liverpool, Liverpool, L69 3GH, United Kingdom · International Joint Research Center for Resilient Infrastructure & International Joint Research Center for Engineering Reliability and Stochastic Mechanics, Tongji University, Shanghai, 200092, China
Large language models (LLMs) have achieved strong performance across diverse tasks, from question answering to code generation. Recent studies have found that LLMs can perform numerical prediction on tabular datasets via in-context learning. However, it remains unclear under what conditions and how LLMs use variable names and in-context examples for numerical prediction. Here we show that semantic grounding, the match between variable names and the values they describe, allows five LLMs without task-specific training to match or outperform seven supervised machine learning (ML) models across 20 datasets from nine scientific domains, particularly when examples are scarce. Reasoning traces indicate that variable names allow LLMs to draw on domain-specific knowledge acquired during pretraining. Using controlled tasks based on physical formulas, we further show that LLMs integrate domain-specific knowledge with in-context examples in a way consistent with Bayesian updating. Specifically, when the relationship between the inputs and the target cannot be inferred from the examples alone, LLMs rely on domain-specific knowledge as the prior. Conversely, as contradicting examples accumulate, LLM predictions shift progressively towards these examples, which act as the likelihood. Our work reveals how LLMs use domain-specific knowledge in numerical prediction and positions them as data-efficient predictors for data-scarce scientific domains.
Figures & tables
Figure 1: Overview of the study design. The top panel shows how each tabular dataset pairs variable names, which carry semantic information, with numerical values, which carry numerical information. The 20 datasets used in this study span nine scientific domains, and six of them were released after the training cutoffs of all five large language models (LLMs). On the left, seven supervised machine learning (ML) models are given the labelled examples. On the right, the five LLMs receive the same examples in a prompt and return a single numerical prediction without any parameter updates. Four questions are addressed. (i) What improves the numerical predictions of LLMs? Prompts without names ( Anonymised ) are compared with prompts that include them ( Correct variable names ). (ii) When do variable names help? To test whether names must match the values they describe (semantic grounding), the values are rescaled ( Rescaled values ), the names are reassigned across features ( Shuffled names ) or replaced with names from an unrelated domain ( Cross-domain names ). (iii) Why do variable names help? Reasoning excerpts from DeepSeek-V3.2 on the Shear Force dataset are shown, with domain-specific terms in red and generic terms in blue. (iv) How is domain-specific knowledge integrated with the in-context examples? In the controlled relation tasks, the two inputs share the same value in every example, so only their names reveal the roles of the two inputs. Some of the examples are then replaced by ones that contradict the names. The prior weight measures how strongly each LLM follows the names rather than the examples (logos from top to bottom are Gemini 3 Flash, Claude Opus 4.5, GPT-5.2, Qwen3.5-397B and DeepSeek-V3.2). Supplementary Methods gives the detailed prompts.
Figure 2: Numerical prediction performance of large language models (LLMs) and supervised machine learning (ML) models across scientific domains. Five LLMs are compared with the three best-performing of seven supervised ML models (TabPFN, tuned XGBoost and tuned random forest) on 20 datasets over five random seeds, and n is the number of examples per model. Summaries are medians over seeds and then across datasets, and group summaries pool all models in a group within each dataset. a , Coefficient of determination ( R2 ) for each dataset at n=100 , grouped by domain. Points show individual models and ranges span the models in each group. Shading marks post-cutoff datasets. b , Weighted absolute percentage error (WAPE) and R2 from n=0 to n=100 . Bars and lines show group summaries and points show individual models. Whiskers show nominal 95% confidence intervals for the median, and arrows mark values beyond the plotting range. Supervised ML models are not shown at n=0 because they require training examples. c , Per-dataset comparison. The advantage of the LLMs at small n holds on nearly every dataset and narrows as examples are added, with the two groups close at n=100 . WAPE is on the left and R2 on the right. Points plot the LLM group summary against the supervised ML group summary for each dataset from n=5 to n=100 , and lines connect the same dataset across numbers of examples. Numbers give the count of datasets in the shaded region, where the LLMs perform better, and stars mark post-cutoff datasets.
Figure 3: Effect of variable names on the numerical predictions of large language models (LLMs) across numbers of examples. Five LLMs are evaluated on 20 datasets under the Anonymised and Correct variable names conditions over five random seeds, and n is the number of examples per model. Condition summaries are medians over seeds and LLMs within each dataset and then across datasets. Improvement is the increase in the coefficient of determination ( R2 ) from Anonymised to Correct variable names . a , Left, R2 of the two conditions across n . Thick lines show condition summaries and thin dashed lines show individual LLMs (squares for Correct variable names and circles for Anonymised ). Grey lines show three supervised machine learning (ML) models, which are not shown at n=0 . TabPFN, the strongest at most n , is shown in dark grey with triangles, and tuned XGBoost (diamonds) and tuned random forest (inverted triangles) are shown in lighter grey dashed lines. Bands show nominal 95% confidence intervals (CIs) for the condition summaries and TabPFN. Right, improvement across n . Bars show the improvement, labelled above each bar, with 95% CIs from 2,000 paired bootstrap resamples of datasets, and points show individual LLMs. In both plots, the vertical axis is broken to show values at n=0 , with a logarithmic segment below −0.45 (left) or in the upper part (right). b , Improvement at n=5 for each LLM and dataset, computed from medians over seeds. Columns are datasets in the order of Fig. 2 a. The rightmost column shows each LLM’s overall improvement, which equals its point at n=5 in the right plot of a . Negative values are outlined in red and labelled, and stars mark post-cutoff datasets.
Figure 4: Sources of the improvement from variable names in large language models (LLMs). Five LLMs are evaluated on 20 datasets at n=5 examples per model over five random seeds. Improvement is the increase in the coefficient of determination ( R2 ) over the corresponding Anonymised condition, and summaries are medians computed as in Fig. 3 . Coloured points and ribbons denote individual LLMs. a , Bars show the improvement for each LLM from the target name, the feature names or all variable names. Error bars show 95% confidence intervals (CIs) from 2,000 paired bootstrap resamples of datasets. b , Median R2 without and with feature names in the original prompt format and in a standardised format, in which the features are given as codes and both conditions include the target name and units (Methods). Bars show condition summaries with 95% CIs across datasets, points show individual LLMs and brackets give the improvement. c , Improvement under Correct variable names , Rescaled values , Shuffled names and Cross-domain names . Bars show the improvement, and error bars show 95% CIs as in a . d , R2 under prompt guidance. Bars show condition summaries, with values given on the right, and points show individual LLMs. CIs are not shown. e , Summary of the improvement contrasts in a and c . Numerical magnitude denotes the difference in improvement between Correct variable names and Rescaled values in c ; variable identity denotes that between Rescaled values and Shuffled names ; and target identity denotes the gain from adding the target name to the feature names in a . Ribbon width indicates the size of each contrast, and dashed ribbons indicate negative contrasts.
Figure 5: Effect of variable names on the reasoning traces of large language models (LLMs). a , t-distributed stochastic neighbour embedding (t-SNE) maps of LLM reasoning traces at n=5 examples per model on 20 datasets over five random seeds, embedded after removing all words that appear in the prompt (Methods). Coloured points show traces under Correct variable names , by domain, and grey points show traces under Anonymised . The large map shows Qwen3.5-397B and DeepSeek-V3.2, which return their full reasoning (150 traces per condition and dataset), and the list gives the words most over-represented in each domain and under Anonymised . The small maps show each LLM separately (60 traces per condition and dataset), using reasoning summaries for Claude Opus 4.5, GPT-5.2 and Gemini 3 Flash, which do not return full reasoning. b , Assignment of a trace to a domain, illustrated with a sentence from a DeepSeek-V3.2 trace on the Shear Force dataset. Words that appear in the prompt (struck out) are removed, and the trace is embedded with a sentence encoder. A classifier trained on traces with all variable names from other seeds then gives the probability of each domain, and p is the probability of the trace’s own domain. Bar heights are illustrative. c , Reasoning and prediction accuracy of DeepSeek-V3.2 and Qwen3.5-397B as variable names are added step by step, from no names and the target name only (0% of feature names) to all feature names (100%), on 20 datasets at n=5 with one seed (Methods). First plot, distribution of p pooled over both LLMs, shown as stacked density curves. Second plot, pˉ , the mean of p over the traces of each dataset, for each LLM, with the dashed line marking chance (1/9) and values labelled at no names and 100%. Third plot, number of domain terms per trace (Methods). Fourth plot, change in the coefficient of determination ( R2 ) relative to no names, paired within each dataset before taking the median across datasets. In the second to fourth plots, points are medians across datasets with 95% confidence intervals from 2,000 bootstrap resamples of datasets, and the dotted line marks where the target name is added.
Figure 6
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
Supplementary Figure 1: Performance on the post-cutoff datasets at n=100 . a , Release dates of the six datasets relative to the latest cutoff or release date of the five LLMs, 16 February 2026 ( Supplementary Table 19 ). Blue marks the experimental datasets, and gold marks the urban-mobility datasets. b , R2 for each dataset and method. c , WAPE on a logarithmic scale. Bars are medians over five seeds, and medians across datasets are given in Supplementary Table 1 . RF denotes random forest.
Method
All six datasets
Three experimental datasets
R2 at n=100
Claude Opus 4.5
0.902±0.006
0.930±0.012
GPT-5.2
0.833±0.058
0.863±0.047
Gemini 3 Flash
0.877±0.027
0.922±0.023
Qwen3.5-397B
0.770±0.081
0.762±0.049
DeepSeek-V3.2
0.761±0.034
0.766±0.053
Appendix
Supplementary Table 1 ∣ Method-level performance on the post-cutoff datasets at n=100 .
Supplementary Figure 2: R2 of each dataset across numbers of examples. Curves show the five LLMs and the three best-performing supervised ML models, and points are medians over five seeds. Vertical scales differ between panels.
Supplementary Figure 3: WAPE of each dataset across numbers of examples. As Supplementary Fig. 2 , with logarithmic vertical scales.
a , Method-level performance
Method
n=0
n=5
n=10
n=25
n=50
n=100
Claude Opus 4.5
−0.286±0.098
0.415±0.059
0.638±0.044
0.794±0.024
0.875±0.019
0.907±0.009
GPT-5.2
−1.841±0.641
0.429±0.098
0.580±0.058
0.769±0.024
0.828±0.031
0.871±0.020
Gemini 3 Flash
−0.630±0.132
0.417±0.065
0.609±0.068
0.806±0.027
0.863±0.020
0.917±0.012
Qwen3.5-397B
−0.272±0.060
0.189±0.072
0.497±0.079
0.728±0.017
0.809±0.043
0.805±0.017
DeepSeek-V3.2
−1.085±0.094
0.177±0.156
0.436±0.065
0.684±0.082
0.754±0.018
0.767±0.021
Appendix
Supplementary Table 2 ∣R2 of every method across numbers of examples.
Method
n=0
n=5
n=10
n=25
n=50
n=100
Claude Opus 4.5
62.1±4.5
28.9±3.4
20.4±1.7
14.3±1.6
13.4±2.6
10.1±1.6
GPT-5.2
78.7±2.4
31.0±4.3
22.5±4.1
17.8±1.6
13.8±1.2
12.8±0.9
Gemini 3 Flash
59.1±3.1
29.0±2.7
21.6±4.3
14.2±1.6
12.2±1.9
10.5±1.3
Qwen3.5-397B
64.2±4.1
32.8±1.8
28.3±3.4
19.2±2.2
16.2±2.0
13.1±1.8
DeepSeek-V3.2
79.2±2.8
34.7±1.8
28.8±3.7
18.2±2.4
16.0±2.2
15.2±1.1
Median of five LLMs
66.0±3.0
31.7±4.1
22.9±4.3
17.6±1.6
14.1±1.6
12.6±0.8
Appendix
Supplementary Table 3 ∣ WAPE (%) of every method across numbers of examples.
Supplementary Figure 4: Differences between LLMs and supervised ML models for each dataset and number of examples. Each cell is the median over five seeds of the R2 difference, LLMs minus supervised ML models, with each group represented by its median across models within a seed. Values are shown for cells in which the supervised ML models performed better by more than 0.01. Datasets are grouped by domain, and stars mark the six post-cutoff datasets.
n=5
n=100
Dataset
R2
WAPE (pp)
R2
WAPE (pp)
Buccolingual Diameter
+0.501
+10.3
−0.005
−0.3
Carbon Intensity
+0.938
+20.5
+0.095
+4.6
Corrosion Degree
+0.207
+7.2
−0.025
+1.7
Daily Trips (Citi Bike)
+0.255
+13.5
−0.026
−0.6
Daily Trips (Divvy)
+0.000
−0.3
−0.020
+0.0
Appendix
Supplementary Table 4 ∣ Differences between LLMs and supervised ML models for each dataset at n=5 and n=100 .
Number of examples
CARTE HL
Signed-rank 95% CI
Dataset-bootstrap 95% CI
P
Datasets improved
Five LLMs combined HL
n=5
+0.008
−0.005 to +0.063
−0.003 to +0.062
0.379
11/20
+0.299
n=10
+0.012
−0.006 to +0.064
+0.001 to +0.054
0.145
14/20
+0.152
n=25
+0.018
−0.028 to +0.073
−0.026 to +0.072
0.398
15/20
+0.092
n=50
+0.025
−0.009 to +0.065
−0.014 to +0.074
0.149
11/20
+0.049
n=100
+0.004
−0.013 to +0.021
−0.012 to +0.019
0.613
9/20
+0.030
Appendix
Supplementary Table 5 ∣ Effect of variable names on CARTE and the LLMs across numbers of examples.
Supplementary Figure 5: Effect of variable names on each dataset at n=5 . Each row shows Correct variable names minus Anonymised for one dataset. Small points show the 25 LLM–seed differences, blue dots and bars show the Hodges–Lehmann estimate and its 95% confidence interval, and the black line indicates the estimate over all 500 cells. Rows are ordered by the R2 estimate. a , R2 , with the axis truncated at 2.2 and an arrow marking an interval that extends to +5.3 . b , WAPE in percentage points, with positive values denoting improvement, on a symmetric-logarithmic axis that is linear within ±10 .
Supplementary Figure 6: Effects of feature names, the target name and rescaled values at n=5 . a , b , Feature and target names given or withheld. c , d , Numerical values rescaled by random factors of 5 to 31.6 or their reciprocals, with or without the names (Methods). a , c , Median R2 of each condition across the five LLMs and for each LLM. b , d , Paired Hodges–Lehmann effects and their interaction, with signed-rank (black) and dataset-bootstrap (pale) 95% confidence intervals and estimates for each LLM (dots). Numerical results are given in Supplementary Table 6 for a,b and in Supplementary Table 22 for c,d.
Condition
Five LLMs combined
Claude Opus 4.5
GPT-5.2
Gemini 3 Flash
Qwen3.5-397B
DeepSeek-V3.2
Anonymised (neither)
0.003±0.104
0.000±0.120
−0.077±0.160
0.089±0.177
−0.057±0.070
−0.103±0.139
Feature names only
0.337±0.113
0.406±0.089
0.325±0.117
0.407±0.139
0.117±0.109
0.022±0.125
Target name only
0.066±0.064
0.194±0.057
0.098±0.099
0.157±0.174
−0.014±0.056
−0.135±0.070
Correct variable names (both)
0.346±0.071
0.415±0.059
0.429±0.098
0.417±0.065
0.189±0.072
0.177±0.156
Contrast
HL
95% CI (signed rank)
95% CI (dataset bootstrap)
P
Feature names, target present
0.1960
0.1665 to 0.2259
0.1298 to 0.2729
1.5×10−44
Appendix
Supplementary Table 6 ∣ Feature-name and target-name contributions at n=5 .
Dataset
Anonymised
Feature names
Difference
Paired
Buccolingual Diameter §
+0.151
+0.457
+0.306
+0.344
Carbon Intensity §
−0.098
+0.425
+0.523
+0.435
Corrosion Degree
+0.776
+0.801
+0.025
+0.028
Daily Trips (Citi Bike) §
−0.191
+0.167
+0.357
+0.274
Daily Trips (Divvy) §
−0.099
−0.088
+0.011
+0.014
Drug–Polymer Solubility
−0.127
−0.147
−0.020
+0.002
Appendix
Supplementary Table 7 ∣ Improvement from feature names for each dataset in the standardised prompt format at n=5 .
Feature
Original
Feature names
Text-value
Dataset
Anonymised
names
values
contribution
contribution
Buccolingual Diameter
+0.151
+0.457
+0.816
+0.344
+0.330
Carbon Intensity
−0.098
+0.425
+0.406
+0.435
+0.051
Daily Trips (Citi Bike)
−0.191
+0.167
+0.357
+0.274
+0.144
Daily Trips (Divvy)
−0.099
−0.088
−0.065
+0.014
+0.214
Elastic Modulus (FDM)
−0.195
−0.161
+0.320
+0.063
+0.492
Appendix
Supplementary Table 8 ∣ Improvement from feature names and from text values in the nine datasets with text values at n=5 .
Supplementary Figure 7: Effects of perturbing variable names at n=5 . a , b , Median R2 and WAPE of each condition across the five LLMs and for each LLM. c , d , Loss of performance from Correct variable names to Shuffled names , from Shuffled names to Cross-domain names and from Cross-domain names to Anonymised (paired Hodges–Lehmann estimates, positive values denote a loss), with 95% confidence intervals and estimates for each LLM (dots). Numerical values are given in Supplementary Table 22 .
Supplementary Figure 8: Prompt conditions and the improvement from variable names across numbers of examples. a , b , Median R2 and WAPE of the four prompt conditions for each LLM ( Supplementary Table 9 and Supplementary Table 10 ). c , d , Improvement from variable names ( Correct variable names minus Anonymised ) for each LLM (coloured lines) and for the five LLMs combined (black line), as paired Hodges–Lehmann estimates with 95% confidence intervals (band).
Prompt condition / model
n=0
n=5
n=10
n=25
n=50
n=100
Anonymised
Median of five LLMs
−13.422±1.599
0.003±0.104
0.311±0.191
0.582±0.038
0.762±0.025
0.825±0.023
Claude Opus 4.5
−21.893±6.613
0.000±0.120
0.435±0.121
0.707±0.039
0.847±0.022
0.881±0.024
GPT-5.2
−2.7×103±1.9×105
−0.077±0.160
0.425±0.150
0.628±0.035
0.759±0.023
0.835±0.019
Gemini 3 Flash
−13.730±1.340
0.089±0.177
0.471±0.150
0.726±0.052
0.827±0.018
0.879±0.013
Qwen3.5-397B
−12.045±1.434
−0.057±0.070
0.241±0.205
0.525±0.098
0.666±0.063
0.836±0.037
Appendix
Supplementary Table 9 ∣R2 under the four prompt conditions.
Prompt condition / model
n=0
n=5
n=10
n=25
n=50
n=100
Anonymised
Median of five LLMs
126.5±5.5
41.6±2.8
28.0±5.4
21.8±1.8
17.8±1.9
15.8±1.7
Claude Opus 4.5
130.0±13.2
38.2±3.4
24.0±4.8
18.5±2.1
13.6±2.7
12.4±1.5
GPT-5.2
1,190.3±4,394.1
44.8±4.0
28.4±4.1
21.9±1.7
16.9±1.3
15.3±1.4
Gemini 3 Flash
125.2±6.6
38.4±3.4
26.5±2.8
19.2±3.1
13.7±1.4
11.9±1.0
Qwen3.5-397B
106.4±8.7
49.1±2.9
29.3±5.3
25.2±2.2
21.0±3.0
16.7±2.8
Appendix
Supplementary Table 10 ∣ WAPE (%) under the four prompt conditions.
Anonymised
Correct variable names
LLM
Without reasoning
With reasoning
Change
Datasets improved
Without reasoning
With reasoning
Change
Datasets improved
Output tokens
Cost ratio
Five LLMs combined
0.003
0.021
−0.012
46/100
0.346
0.309
−0.005
49/100
1,374
7.2
Claude Opus 4.5
0.000
0.076
+0.025
12/20
0.415
0.422
+0.005
12/20
956
6.2
GPT-5.2
−0.077
0.101
+0.075
12/20
0.429
0.306
−0.096
7/20
463
11.0
Gemini 3 Flash
0.089
0.030
+0.046
14/20
0.417
0.400
−0.017
8/20
1,167
11.2
Qwen3.5-397B
−0.057
−0.148
−0.017
7/20
0.189
0.130
−0.008
9/20
2,010
11.4
Appendix
Supplementary Table 11 ∣ Extended reasoning at n=5 , by LLM.
Supplementary Table 12 ∣ Words most characteristic of the reasoning traces with and without variable names.
DeepSeek-V3.2
Qwen3.5-397B
Feature names
Probability
Terms
Domains
Own domain, %
Distance
Probability
Terms
Domains
Own domain, %
Distance
No names
0.39
1.0
3.9
35
0.50
0.42
0.6
4.3
47
0.60
0% (target only)
0.56
4.0
3.4
55
0.39
0.67
3.5
2.9
85
0.50
25%
0.63
8.8
3.1
69
0.36
0.79
8.9
2.3
88
0.39
50%
0.72
8.7
2.7
90
0.30
0.78
9.0
2.2
85
0.36
75%
0.80
8.9
2.1
91
0.24
0.84
12.5
2.0
92
0.35
Appendix
Supplementary Table 13 ∣ Reasoning traces of DeepSeek-V3.2 and Qwen3.5-397B as feature names are added at n=5 .
Task
Field
Inputs ( x1 then x2 )
(p,q)
[a,b]
Units ( x , y )
Reaction rate
Chemistry and nuclear physics
concentration of NO concentration of O2
(2,1)
[−6.5,−5.5]
mmol L -1 mmol L -1 s -1
Beam deflection
Structural and solid mechanics
cross-section width cross-section depth in the bending direction
(−1,−3)
[7.5,8.5]
mm mm
Conductor resistance
Electrical engineering and communications
axial length of the conductor radius of the circular cross-section
(1,−2)
[−1,1]
mm Ω
Steady conduction
Heat transfer, optics and atmosphere
side length of the square heat-transfer face conduction thickness
(2,−1)
[−2,0]
mm W
Laminar pipe flow
Fluid mechanics
radius of the circular channel length of the channel
(4,−1)
[−6.5,−5.5]
μ m nL s -1
Bending stress
Structural and solid mechanics
cross-section width cross-section depth in the bending direction
(−1,−2)
[6.5,7.5]
mm MPa
Appendix
Supplementary Table 14 ∣ Controlled relation tasks, by task.
Task
Field
Inputs ( x1 then x2 )
Relation
Constants
Units ( x , y )
Linear
Soil shear strength
Geotechnics
effective cohesion of the soil effective normal stress acting on the shear surface
x1+x2tanϕ′
ϕ′=20∘
kPa kPa
Cantilever moment
Structural and solid mechanics
load at the free end load at a quarter of the span from the support
Lx1+4Lx2
L=4 m
kN kN m
Bearing capacity
Geotechnics
effective cohesion of the soil overburden pressure at the footing base
Ncx1+Nqx2
ϕ′=20∘ ; Nc=14.83 , Nq=6.40
kPa kPa
Exponential
Fall with linear drag
Dynamics
time elapsed since release drag time constant of the sphere
vt[x1−x2(1−e−x1/x2)]
vt=2 m s -1
s mm
Appendix
Supplementary Table 14 ∣ Controlled relation tasks, by task (continued).
Task
Original
Anonymous
Swapped
Separation
Positive
Power laws
Reaction rate
+1.00
+0.00
−1.00
1.92 [ 1.75 , 2.00 ]
99/100
Beam deflection
+0.95
+0.00
−0.82
1.51 [ 1.36 , 1.63 ]
99/100
Conductor resistance
+0.98
+0.00
−1.00
1.88 [ 1.80 , 1.95 ]
100/100
Steady conduction
+1.00
+0.00
−0.99
1.71 [ 1.67 , 1.90 ]
98/100
Laminar pipe flow
+1.00
+0.00
−1.00
1.83 [ 1.72 , 1.93 ]
100/100
Appendix
Supplementary Table 14 ∣ Controlled relation tasks, by task (continued).
Deviation from
Deviation from
Closer to
LLM
Separation
Positive
assigned (%)
exchanged (%)
assigned (%)
Claude Opus 4.5
1.86 [1.76, 1.91]
495/500
9.2
100.0
87.4
GPT-5.2
1.47 [1.37, 1.57]
476/500
18.7
100.0
79.5
Gemini 3 Flash
2.00 [2.00, 2.00]
497/500
0.7
100.0
91.1
Qwen3.5-397B
1.00 [0.92, 1.06]
459/500
30.9
93.6
74.1
DeepSeek-V3.2
0.46 [0.40, 0.55]
375/500
56.2
93.1
62.9
Appendix
Supplementary Table 15 ∣ Controlled relation tasks, by LLM.
Contradicting examples, k
Task
Condition
0
1
2
3
4
5
Prior weight
Power law
Reaction rate
Original
+1.00
+0.50
+0.65
+0.26
+0.02
−0.43
0.46 [ 0.34 , 0.60 ]
Anonymous
+0.00
−0.08
−0.19
−0.45
−0.60
−0.70
Beam deflection
Original
+0.91
+0.47
+0.18
−0.22
−0.60
−0.70
0.29 [ 0.24 , 0.36 ]
Anonymous
+0.00
−0.20
−0.36
−0.63
−0.80
−0.93
Appendix
Supplementary Table 16 ∣ Contradicting examples, by task.
Contradicting examples, k
LLM
Condition
0
1
2
3
4
5
Prior weight
Five LLMs combined
Original
+0.89
+0.56
+0.37
+0.17
+0.00
−0.24
0.43 [ 0.38 , 0.51 ]
Anonymous
+0.00
+0.00
−0.11
−0.41
−0.68
−0.86
Claude Opus 4.5
Original
+1.00
+0.88
+0.93
+0.62
+0.42
+0.00
0.75 [ 0.64 , 0.85 ]
Anonymous
+0.00
−0.15
−0.31
−0.55
−0.81
−0.94
GPT-5.2
Original
+0.84
+0.52
+0.28
+0.13
−0.08
−0.65
0.41 [ 0.32 , 0.50 ]
Appendix
Supplementary Table 17 ∣ Contradicting examples, by LLM.
Supplementary Figure 9: Relation scores with 20 in-context examples and the Bayesian observer. Points show the median relation score over 12 tasks and 10 instances when k of 20 examples contradict the knowledge elicited by the names, under Original assignment (filled) and Anonymous (open). Translucent curves show the Bayesian observer, in which the knowledge elicited by the names forms the prior and both conditions share the likelihood ratio per example. The observer was fitted to each task, and the curves show the median across tasks.
Supplementary Figure 10: Prior and likelihood ratio of each LLM. a , Ratio ρ of the log likelihood ratio per example with names to that without names. ρ=1 means that, in the observer, the knowledge elicited by the names changes only the prior, and the shaded band is the equivalence margin of 1.5-fold. b , Median prior probability of the Original relation, the relation implied by the names, against the median likelihood ratio per contradicting example under Anonymous , across 12 tasks. Both come from the observer with a shared ρ . Dotted lines mark an uninformative prior (0.5) and uninformative examples (1). Error bars show 95% confidence intervals.
Supplementary Figure 11: Out-of-sample prediction of the Bayesian observer. a , Observed relation scores under Original assignment (points, mean of 12 task medians) and the observer’s prediction (line) when the prior and the likelihood ratio were fitted to both conditions at k≤5 only (shaded). Numbers give the mean deviation of the observed from the predicted scores at k=8 –20 with its 95% confidence interval, in red where it excludes zero. b , Mean absolute deviation of the observed from the predicted relation scores at k=8 –20 under Original assignment , for each task and LLM. Red outlines mark deviations larger than 0.5.
Dataset
Domain
Samples
Features
Target variable
Split
Splitting Strength
Structural engineering
550
9
Splitting tensile strength (MPa)
Random
Shear Force
Structural engineering
156
9
Shear force (kN)
Random
Max Displacement
Structural engineering
230
12
Maximum impact displacement (mm)
Random
MIDR (MDOF)
Structural engineering
320
15
Maximum interstorey drift ratio
Random
Torsional Strength
Structural engineering
150
7
Torsional strength (kN m)
Random
FRP–Steel Bond
Structural engineering
113
12
Ultimate bearing capacity (kN)
Random
Appendix
Supplementary Table 18 ∣ The 20 real-world tabular datasets.
Scientific reasoning rarely stops at what is directly observable; it often requires uncovering hidden structure from data. From estimating reaction constants in chemistry to inferring demand elasticities in economics, this latent structure recovery is what distinguishes scientific reasoning from curve fitting. Large language models (LLMs) can often recall and apply relevant scientific formulas, but we show that this ability is surprisingly easy to suppress. We show that adding in-context examples makes models rely less on pretrained domain knowledge, even when those examples are generated by the very same formula. Rather than reinforcing knowledge-driven derivation, examples shift computation toward empirical pattern fitting. We document this knowledge displacement on 60 latent structure recovery tasks across five scientific domains, 6,000 trials, and four models. This displacement is consistent across domains, but its accuracy consequences depend on how the displaced strategy compares to the one that replaces it: the same shift can lower accuracy, leave it unchanged, or appear to improve it. In all cases, however, the model shifts away from knowledge-driven reasoning. For practitioners deploying LLMs on scientific tasks, the message is cautionary: in-context examples may displace, rather than reinforce, the knowledge they are intended to support.
Chaemin Jang, Woojin Park, Hyeok Yun +2
Korea Advanced Institute of Science and Technology · Shanghai Jiao Tong University
Large Language Models (LLMs) have recently been successfully applied to regression tasks -- such as time series forecasting and tabular prediction -- by leveraging their in-context learning abilities. However, their autoregressive decoding process may be ill-suited to continuous-valued outputs, where obtaining predictive distributions over numerical targets requires repeated sampling, leading to high computational cost and inference time. In this work, we investigate whether distributional properties of LLM predictions can be recovered without explicit autoregressive generation. To this end, we study a set of regression probes trained to predict statistical functionals (e.g., mean, median, quantiles) of the LLM's numerical output distribution directly from its internal representations. Our results suggest that LLM embeddings carry informative signals about summary statistics of their predictive distributions, including the numerical uncertainty. This investigation opens up new questions about how LLMs internally encode uncertainty in numerical tasks, and about the feasibility of lightweight alternatives to sampling-based approaches for uncertainty-aware numerical predictions.
Julianna Piskorz, Katarzyna Kobalczyk, Mihaela van der Schaar
Department of Applied Mathematics and Theoretical Physics, University of Cambridge
Large language models (LLMs) are increasingly applied to complex problems, from automated code debugging to science itself. Such problems require generating plausible hypotheses from sparse, noisy, and ambiguous data and updating them as more information arrives. Yet how closely LLMs approach optimal inductive reasoning remains unclear. We study this question in the classic number game, in which a learner infers a concept from a few ambiguous positive integers: {2,16,4,8}, for example, is explained both by powers of 2'' and by numbers up to 20.'' We probe the concepts LLMs hypothesize from the observed examples and use their behavioral patterns to understand how they generate and update hypotheses. We measure the LLM posterior over hypotheses in three tasks (prediction, evaluation, and generation) and compare LLM behavior with a Bayesian model and with humans. LLMs favor narrower hypotheses than humans and show a robust evaluation--generation gap: in evaluation they select hypotheses more consistent with the observed examples, whereas in generation they propose narrower hypotheses that are more often rules. They also update poorly: as examples accumulate, their accuracy declines while their confidence grows. Thinking reduces, and can even reverse, the gap; it also broadens the hypotheses LLMs favor, raising accuracy while lowering confidence. Whereas prior studies have shown that LLMs are good hypothesis proposers, our results and further comparisons with the Bayesian model show that LLM inductive reasoning lacks coherence across the three tasks, suggesting that LLMs do not maintain a stable representation shared across tasks that make the same demand. Our results highlight a limitation of LLMs in inductive inference, particularly in settings such as science, where hypotheses must go beyond the data.
Hua-Dong Xiong
School of Psychological and Brain Sciences, Georgia Tech