Neural networks are typically programmed through optimization, rather than by explicitly specifying their computations. Recent work on weight-space programming suggests that large language models (LLMs) can instead reason about and manipulate neural-network weights directly. In this work, we show that LLMs can directly program statistical computations in tabular foundation models (TFMs), whose parameters enable inference across different datasets. We investigate three complementary settings using GPT-6 Astra, without gradient-based optimization. First, Astra directly programs nanoTabPFN from zero-initialized weights, achieving a mean ROC AUC of 0.849 across 20 TabArena datasets, compared to 0.876 for a pretrained baseline. Second, Astra diagnoses and repairs a signed-product regression failure in pretrained TabPFN-3.5, restoring near-perfect predictions on held-out synthetic datasets while preserving performance on TabArena-Lite, with only a 0.08% average relative increase in error. Third, it writes KV-cache-editing programs that steer TabPFN-3.5's predictions on unseen synthetic datasets, demonstrating transformations such as reversing linear trends and converting sine waves into approximate square waves, without modifying model weights. Together, our findings suggest that LLM-driven programming could complement gradient-based optimization across TFMs' development and usage, while offering a new way to investigate and intervene in their internal computations.
Figures & tables
Figure 1: (A) Construct: Astra writes nanoTabPFN weights from scratch, achieving competitive performance relative to a pretrained baseline; (B) Repair: Astra programs a corrective computation into pretrained TabPFN-3.5 weights, resolving a signed-product regression failure; (C) Steer: Astra translates a natural-language instruction into a reusable KV-cache editing program that changes a fitted model’s predictions while keeping its weights frozen.
Figure 2: Signed-product intervention. (a) Schematic of the diagnosis and the edit, for a query with target y>0 . The pretrained model places predictive mass at both −y and y , so its mean (dashed) is near zero, while its early layers already encode the number of negative inputs. The edit (orange) carries this count to the regression head, where its parity selects the sign. (b) Median R2 over 60 hidden datasets per regime ( k inputs, N context rows, 500 query rows), before and after the edit.
Figure 3: Programming the KV-cache of TabPFN-3.5. Left: during development, Astra reads the KV-caches of training datasets and writes an edit program; in the test phase, the program edits the KV-cache of an unseen dataset, and the TFM predicts the query rows from the edited cache. Right: the best, median, and worst of test datasets for flip-the-slope (top) and sine-to-square-wave (bottom).
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
OpenML task
Rows
Features
Classes
anneal
363614
898
38
5
Bank_Customer_Churn
363619
10,000
10
2
blood-transfusion-service-center
363621
748
4
2
churn
363623
5,000
19
2
coil2000_insurance_policies
363624
9,822
85
2
credit-g
363626
1,000
20
2
Appendix
Table 1: The 20 evaluation datasets (TabArena-v0.1). Rows cover both parts of the split; features are the raw columns before preprocessing, from OpenML’s metadata.
WSP-NanoTabPFN
Turns
7
Python runs
5
Blocks returned for lack of explanation
1
Wall-clock time (min)
20
Nonzero weights
15,513 (0.42%)
ROC AUC
0.849
Appendix
Table 2: The session that wrote WSP-NanoTabPFN. Scores are means over the 20 datasets of Table 1 .
ROC AUC ↑
Log loss ↓
Dataset
WSP
SGD
WSP
SGD
anneal
0.987
0.989
0.208
0.070
Bank_Customer_Churn
0.849
0.854
0.399
0.348
blood-transfusion-service-center
0.694
0.717
0.501
0.501
churn
0.854
0.910
0.324
0.220
coil2000_insurance_policies
0.703
0.731
0.212
0.210
Appendix
Table 3: Per-dataset ROC AUC and log loss on TabArena’s first split of WSP-NanoTabPFN (WSP) and the SGD Baseline (SGD). WSP-NanoTabPFN has the higher AUC on 3 of the 20 datasets (diabetes, Is-this-a-good-customer, splice) and the lower log loss on 1 (diabetes).
Agent sessions
1
CLI turns (including final notes)
2
Wall-clock time / allowed budget
62 min / 5 h
Recorded commands / scripts
143 / 35
Feedback requests / final submissions
3 / 1
Changed tensors / weight entries
125 / approximately 105 million
Appendix
Table 4: The signed-product intervention session. Candidate counts refer to requests for evaluator feedback, not every local diagnostic or edit.
Figure 4: Predictive distributions of pretrained TabPFN-3.5 for four signed-product queries, from two freshly generated datasets. Each of the eight ensemble members (grey) places separate modes near y and −y , and so does their average (blue), which the estimator uses; its mean (dashed) therefore lies near zero.
k
N
Pretrained median R2
Edited median R2
Share with R2≥0.8
nMSE ratio
5
256
−0.018
0.995
96.7%
0.022
5
512
0.017
0.998
98.3%
0.007
6
256
−0.024
0.991
95.0%
0.036
6
512
−0.024
0.997
98.3%
0.009
Appendix
Table 5: Signed-product prediction on 60 hidden datasets per regime, each with k inputs, N context rows, and 500 query rows. The last columns report the edited model’s share of datasets with R2≥0.8 and the ratio of summed edited to pretrained nMSE. These are intervention results, not oracle diagnostics.
Task type
Tasks
Geometric-mean error ratio
Binary classification
30
1.0009
Multiclass classification
8
1.0020
Regression
13
0.9997
All tasks
51
1.0008
Appendix
Table 6: Preservation on TabArena-Lite. Ratios below one indicate lower error after the intervention. Per-task errors are averaged over three seeds before aggregation.
Figure 5: Programming the KV cache of TabPFN-3.5 on all twenty test datasets. Each panel shows one unseen dataset, ordered from best to worst (left to right, then top to bottom) as in Figure 3 : grey dots are labelled context rows, blue curves are the original TFM predictions, dashed curves are the targets, and orange curves are predictions from the edited KV cache. The top block shows flip the slope ; the bottom block shows sine to square wave .
Large language model (LLM)-based program synthesis turns natural-language goals and examples into executable programs, often using tests or formal constraints.