MoMaTune: Model, Match, Then Tune Language Models for Numerical Records

Weimin Lyu; Jingfen Zhang; Lingdao Sha; Peng sun; Xiaowei Chen; Vincent Gao; Rui Song

cs.AIOct 11, 2026

MoMaTune: Model, Match, Then Tune Language Models for Numerical Records

Authors: Weimin Lyu, Jingfen Zhang, Lingdao Sha, Peng sun, Xiaowei Chen, Vincent Gao, Rui Song

Organizations: Amazon

Abstract

Despite remarkable progress in language understanding and generation, decoder language models continue to exhibit striking failures on numerical data. Numerical examples are typically serialized as text by concatenating feature names with their values (e.g., fat:3.4), while conventional task adaptation applies loss only to the downstream prediction target. This leaves two structures weakly supervised: within-value sensitivity, where digit position determines numerical magnitude, because token cross-entropy contains no explicit numerical metric; and across-field structure, where measurements in the same record are statistically dependent, because target-only tuning provides no direct loss on the measurements or their dependencies. We introduce MoMaTune (Model, Match, then Tune), a two-stage post-training strategy organized around these supervision gaps. Stage 1 applies Model, ordinary next-token prediction densely over label-free serialized records, and optionally adds Match, a same-record consistency objective that aligns representations of different partial views of the same example. Stage 2 applies Tune, supervised fine-tuning for downstream target prediction. We evaluate MoMaTune on 169,285 records from eight public datasets, with 339 of 340 input fields numerical. At 1.5B parameters, Model raises the eight-dataset masked-value imputation aggregate from 0.106 to 0.401, while the three-dataset correct-context gain rises from 0.058 to 0.167. On diagnostics, Model also increases sensitivity to numerically consequential digit changes, while Match improves robustness to informative partial views. Adding Tune yields higher target-supervised embedding aggregates for each tested Stage-1 parent, and the two-stage recipes obtain higher aggregate scores than Tune alone. The same pattern is observed at 4B, 9.6B, and 27B parameters when model embeddings are used for downstream prediction.

Figures & tables

Figure 1: Token loss is numerically agnostic, but record-level adaptation improves magnitude sensitivity than target-only SFT. (a) With one digit per token, incrementing the first versus fourth fractional digit of 0.4271 produces a

1000\times

difference in numerical effect. With groups of up to three digits, incrementing [427] versus [132] by one in 0.427132 also produces a

1000\times

difference. Each intervention changes exactly one token. (b) We measure within-value sensitivity as the ratio between how far the pooled record embedding moves after a first-fractional-digit edit and after a fourth-fractional-digit edit. On the MiniBooNE diagnostic, applying the same objective densely through field-permuted Model increases this ratio from 2.08

\times

to 17.82

\times

at 1.5B. The

1000\times

difference in numerical effect is a reference, not a required embedding-distance ratio. Model denotes next-token prediction on records, and field-permuted means reordering the attribute-value fields as data augmentation.

\fnum@figure : MoMaTune centers on Model $\rightarrow$ Tune , with Match as an optional partial-view robustness component. Stage 1 always applies Model to label-free numerical records. Applications that require robustness to partial records can additionally apply Match . The resulting Stage-1 checkpoint initializes Stage 2, where Tune presents a fixed-order record and target question and applies loss only to the label token.

Frozen decoder LMs (mean-pooled hidden states)
Model / recipe	Classification: ROC–AUC $\uparrow$		Regression: $R^{2}\uparrow$						Emb-8 $\uparrow$
Model / recipe	MiniBooNE	HIGGS	Supercond.	YearPred	Naval	CPU act.	Protein	USDA	Emb-8 $\uparrow$
Qwen3-4B	.960	.642	.783	.160	.893	.926	.332	.940	.705
Qwen3-8B	.955	.636	.797	.172	.917	.931	.330	.946	.711
Qwen3.5-9B	.954	.644	.807	.149	.897	.928	.349	.939	.708
Qwen3-14B	.959	.660	.809	.156	.929	.947	.342	.951	.719

Table 1: Our MoMaTune improves linear-probe target prediction across eight nearly pure-numerical datasets. Classification tasks report ROC–AUC, the regression tasks report

R^{2}

. Emb-8 is unweighted mean of all eight dataset. The two field-permuted variants obtain the highest reported Emb-8 (0.753), with and without Match . All trained recipes in this table use a Qwen2.5-1.5B backbone; Tab. 4 reports the corresponding scale comparison from 1.5B to 27B. Additional classification metrics appear in Appx. Tab. 14 , and regression metrics in Tabs. 15 , 16 , and 17 .

Zero-shot decoder LMs
Model / recipe	Within-10% completion accuracy $\uparrow$								Gen-8 $\uparrow$
Model / recipe	MiniBooNE	HIGGS	Supercond.	YearPred	Naval	CPU act.	Protein	USDA	Gen-8 $\uparrow$
Qwen3-4B	.006	.130	.292	.024	.144	.194	.000	.260	.131
Qwen3-8B	.004	.142	.282	.024	.152	.168	.000	.258	.129
Qwen3.5-9B	.006	.172	.280	.034	.122	.174	.000	.286	.134
Qwen3-14B	.002	.158	.294	.030	.140	.172	.000	.290	.136

Table 2: Stage-1 training improves numerical completion. We test the masked-value completion by asking the LM to generate the missing value. Gen-8 is the unweighted mean across eight datasets. Model provides the largest improvement in numerical completion. Adding Match does not improve numerical completion, instead it benefits to robustness to partial views (Fig. 4 (b)) and end-to-end target prediction (Tab. 1 ).

Model / recipe	Macro within-10% accuracy			Per-dataset $\Delta_{\mathrm{ctx}}\uparrow$
Model / recipe	Correct $\uparrow$	Mismatched	$\Delta_{\mathrm{ctx}}\uparrow$	USDA	Supercond.	MiniBooNE
Frozen base	.142	.084	+.058	+.064	+.115	-.004
Model	.338	.171	+.167	+.170	+.284	+.047
Field-permuted Model	.463	.193	+.271	+.244	+.472	+.096

Table 3: Model improves the utilization of cross-field context. For each masked field, the correct and mismatched prompts have identical field names, order, target, and scorer; only the values in mismatched prompts are replaced by values from another record.

\Delta_{\mathrm{ctx}}

is correct minus mismatched within-10% accuracy. The first three columns are macro averages over three dataaset; the remaining columns show the corresponding per-dataset context gains

\Delta_{\mathrm{ctx}}

\fnum@figure : Model and Tune improve their intended capabilities, while Match stabilizes informative partial views. (a) Across three-dataset, fixed-order Model improves both the primary target-prediction metric and numerical completion relative to the frozen checkpoint at every scale. (b) For permuted-prefix views, Match reduces embedding displacement by

15.3\times

and raises half-view AUC by 5.4%. (c) The upper panel shows that Tune improves Emb-8 for every tested Stage-1 parent at 1.5B; the lower panel shows a positive three-dataset gain through 27B for the field-permuted Model + Match parent.

\fnum@figure : Field-order permutation improves completion beyond an equal-update fixed-order control. (a) Mean within-10% masked-value imputation over three-dataset from 1.5B to 27B, comparing standard fixed-order Model , extended fixed-order training, and field-permuted Model ; the dotted line marks the 0.455 frontier-model reference used in this diagnostic. (b) At 27B, field-permuted Model improves every dataset over the equal-update fixed-order control, increasing the three-dataset mean by 0.134; the gray band marks the

\pm 0.008

same-configuration imputation spread.

Scale	Frozen checkpoint $\rightarrow$ Model	Field-permuted Model + Match $\rightarrow$ MoMaTune	Tune only $\rightarrow$ MoMaTune
1.5B	.878 $\rightarrow$ .897 (+.019)	.904 $\rightarrow$ .914 (+.010)	.891 $\rightarrow$ .914 (+.023)
4B	.894 $\rightarrow$ .915 (+.021)	.922 $\rightarrow$ .930 (+.008)	.910 $\rightarrow$ .930 (+.020)
9.6B	.900 $\rightarrow$ .930 (+.030)	.934 $\rightarrow$ .935 (+.001)	.926 $\rightarrow$ .935 (+.009)
27B	.903 $\rightarrow$ .933 (+.030)	.936 $\rightarrow$ .942 (+.006)	.927 $\rightarrow$ .942 (+.015)

Table 4: Target-prediction comparisons across the tested model sizes. Each cell reports before

\rightarrow

after and the absolute difference in parentheses for the mean of MiniBooNE ROC–AUC and Superconductivity and USDA

R^{2}

. All three comparisons show positive point differences at the four tested model sizes.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Tokenization	Model	$\bar{d}_{1}/\bar{d}_{4}$ $\uparrow$	Spearman $\rho$ $\uparrow$
One digit per token	Qwen2.5-1.5B-Instruct	2.14	0.276
One digit per token	Qwen3-14B	2.04	0.196
One digit per token	Mistral-7B-v0.3	1.65	0.191
One digit per token	Gemma-4-12B	1.12	0.131
Groups of up to three	phi-4	1.94	0.353
Groups of up to three	OLMo-2-7B	1.81	0.159

Appendix

Table 5: Digit-blindness in frozen pretrained models, under both tokenization conventions. Each model is evaluated with its own tokenizer, so the accepted record set is derived per model and the sets are not identical across rows; the Qwen2.5-1.5B row is therefore an independent re-derivation of the frozen row of Tab. 19 (

2.14

here versus

2.08

there). Larger ratios indicate greater leading-digit sensitivity under this proxy. The reference row contrasts a first-digit edit ( 0.4271

\to

0.5271 , a

23.4\%

change in value) with a fourth-digit edit ( 0.4271

\to

0.4272 ,

0.0234\%

), a ratio of

1000\times

; it is a reference contrast, not an expected embedding ratio. MiniBooNE, 200 accepted records per model, frozen checkpoints only.

Dataset	Task	Numeric fields	Text fields	Records	Target
MiniBooNE	cls.	50	0	30,000	signal
HIGGS	cls.	28	0	30,000	class
Superconductivity	reg.	79	0	21,263	critical temperature
USDA FoodData	reg.	48	1	7,896	energy (kcal)
CPU activity	reg.	21	0	8,192	user CPU %
Naval propulsion	reg.	14	0	11,934	compressor decay

Appendix

Table 6: Public evaluation corpus. Seven of eight datasets are numerical-only; across schemas, 339/340 input fields are numerical. “Records” includes train, validation, and test splits.

Dataset	Numeric	Text	Train	Val.	Test	Gen. $n$	Target range / classes
MiniBooNE	50	0	24,000	3,000	3,000	500	binary
HIGGS	28	0	24,000	3,000	3,000	500	binary
Superconductivity	79	0	17,010	2,126	2,127	500	$[0.00021,185]$
USDA FoodData	48	1	6,316	789	791	500	$[0,902]$
CPU activity	21	0	6,553	819	820	500	$[0,99]$
Naval propulsion	14	0	9,547	1,193	1,194	500	$[0.95,1.0]$

Appendix

Table 7: Complete dataset statistics and input modality. “Gen.

n

” is the number of test records used for masked-value imputation and field lookup for each checkpoint.

Dataset	Prediction target	Task type	Embedding metrics	Generation metrics
MiniBooNE	signal	binary classification	Acc., ROC-AUC $\uparrow$	impute $\uparrow$ ; lookup $\uparrow$ ; PPL $\downarrow$
HIGGS	class	binary classification	Acc., ROC-AUC $\uparrow$	target Acc. $\uparrow$ ; impute $\uparrow$ ; lookup $\uparrow$ ; PPL $\downarrow$
Superconductivity	critical temperature	regression	$R^{2},r\uparrow$ ; MAE $\downarrow$	impute $\uparrow$ ; lookup $\uparrow$ ; PPL $\downarrow$
USDA FoodData	energy (kcal)	regression	$R^{2},r\uparrow$ ; MAE $\downarrow$	impute $\uparrow$ ; lookup $\uparrow$ ; PPL $\downarrow$
CPU activity	user CPU %	regression	$R^{2},r\uparrow$ ; MAE $\downarrow$	target log error $\downarrow$ ; impute $\uparrow$ ; lookup $\uparrow$ ; PPL $\downarrow$
Naval propulsion	compressor decay	regression	$R^{2},r\uparrow$ ; MAE $\downarrow$	target log error $\downarrow$ ; impute $\uparrow$ ; lookup $\uparrow$ ; PPL $\downarrow$

Appendix

Table 8: Dataset-to-metric map. Every dataset uses masked-value imputation, field lookup, and perplexity; target emission is reported only where the complete training-matched prompt fits the 496-token budget. Embedding metrics depend on whether the target is categorical or continuous.

Checkpoint	Scale	Decoder architecture	Vocabulary
Qwen2.5-1.5B-Instruct	1.5B	Dense Qwen2 decoder	151,936
Qwen3-4B-Instruct-2507	4B	Dense Qwen3 decoder	151,936
Qwen3.5-9B	9.6B	32-layer hybrid full/linear-attention decoder	248,320
Qwen3.8-27B	27B	64-layer hybrid full/linear-attention decoder	248,320

Appendix

Table 9: Model checkpoints used in our experiments. Vocabulary denotes tokenizer vocabulary size.

Recipe	Stage-1 order / views	Stage-1 objective	Stage 2	Budget	Purpose
Frozen decoder	fixed-order / 1	none	–	–	frozen initialization
Model	fixed-order / 1	Model	–	2 epochs	primary baseline
Match only	fixed-order / 1	Match only (disjoint views)	–	2 epochs	consistency-only control
Model + Match	fixed-order / 1	Model + Match (disjoint views)	–	2 epochs	Model + Match recipe
Tune only	fixed-order / 1	none	fixed-order Tune	Tune only	no-Stage-1 control
Extended-compute Model	fixed-order / 1	Model	–	20 epochs	10 $\times$ compute

Appendix

Table 10: The principal arms. “Views” is the number of deterministic row serializations per epoch. All Model recipes use weight 0.5 for that loss. Every serialization is right-truncated at 512 tokens; a “permuted-prefix view” denotes independent field-order permutation followed by truncation. Model denotes next-token record adaptation; Match denotes optional same-record consistency; and Tune denotes supervised fine-tuning (SFT).

Recipe	MiniBooNE	HIGGS	Supercond.	YearPred	Naval	CPU act.	Protein	USDA	Mean 8
Field-permuted Model $\rightarrow$ Tune	.963	.697	.817	.224	.960	.970	.424	.969	.753
Field-permuted Model + Match $\rightarrow$ Tune	.968	.691	.806	.219	.972	.965	.434	.967	.753

Appendix

Table 11: Matched complete-record comparison for the final Match intervention. Both recipes use field-permuted Model followed by the same Tune interface; the second additionally uses Match . The near-identical mean motivates evaluating Match through its partial-view robustness role rather than complete-record aggregate accuracy. Higher is better.

Recipe	Emb-8 $\uparrow$	Imputation $\uparrow$
Frozen decoder	.688	.106
Model	.730	.401
Match only	.614	.033
Model + Match	.733	.401
Field-permuted Model	.739	.448
Field-permuted Model + Match	.740	.445

Appendix

Table 12: Aggregate controls used in the ablation discussion. Emb-8 is the eight-dataset frozen-probe aggregate; imputation is the corresponding eight-dataset within-10% accuracy. The final two rows use field-permuted records during Tune to isolate the supervised interface. Higher is better.

$\lambda_{1}$	$\lambda_{2}$	Protein $R^{2}\uparrow$	USDA $R^{2}\uparrow$
0.5	0.00	.396	.942
0.5	0.25	.411	.940
0.5	0.50	.415	.939
0.5	1.00	.420	.937
0.5	2.00	.416	.938
0.5	4.00	.414	.938

Appendix

Table 13: The reported consistency weight is at the optimum on the dataset where Match has an effect, and irrelevant on one where it does not. Primary embedding metric (

R^{2}

) after Model + Match at 1.5B with permuted-prefix views, varying only the loss coefficients of Eq. equation 7 . The reported setting is in bold. The last two rows are controls:

\lambda_{1}{=}0

removes the record term, and

\lambda_{1}{=}1.0,\lambda_{2}{=}2.0

preserves the ratio at doubled scale. The same-configuration measurement spread on this metric is

0.005

, so differences below that are not resolved.

floors / ceilings
Model / recipe	Resources	MiniBooNE		HIGGS		Mean $\uparrow$
Model / recipe	Resources	ACC $\uparrow$	ROC–AUC $\uparrow$	ACC $\uparrow$	ROC–AUC $\uparrow$	Mean $\uparrow$
F2: char-TF-IDF probe floor	50k sparse features	0.833	0.890	0.554	0.552	.707
C1: XGBoost raw-feature ceiling	raw features	0.940	0.983	0.706	0.788	.854
zero-shot decoder LLMs (18 models)
Qwen2.5-1.5B	1.5B	0.898	0.956	0.617	0.655	.781

Appendix

Table 14: Detailed classification-probe results. ACC and ROC–AUC are reported for MiniBooNE and HIGGS; higher is better. Mean is the arithmetic mean of the four displayed metrics. Model denotes next-token record adaptation; Match denotes optional same-record consistency; and Tune denotes supervised fine-tuning (SFT).

floors / ceilings
Model / recipe	Resources	Supercond.	USDA	CPU	Naval	Protein	YearPred	Mean $\uparrow$
F2: char-TF-IDF probe floor	50k sparse features	0.865	0.860	0.281	0.904	0.201	0.005	.519
C1: XGBoost raw-feature ceiling	raw features	0.916	0.997	0.985	0.996	0.623	0.277	.799
zero-shot decoder LLMs (18 models)
Qwen2.5-1.5B	1.5B	0.769	0.910	0.891	0.860	0.311	0.152	.649
Qwen3-1.7B	1.7B	0.778	0.920	0.886	0.885	0.316	0.155	.657

Appendix

Table 15: Detailed regression-probe

R^{2}

by dataset; higher is better. Mean is the arithmetic mean over the six displayed datasets. Model denotes next-token record adaptation; Match denotes optional same-record consistency; and Tune denotes supervised fine-tuning (SFT).

floors / ceilings
Model / recipe	Resources	Supercond.	USDA	CPU	Naval $\times 10^{4}$	Protein	YearPred	Mean $\downarrow$
F2: char-TF-IDF probe floor	50k sparse features	8.60	44.30	8.83	26.03	4.46	8.13	16.725
C1: XGBoost raw-feature ceiling	raw features	5.59	4.75	1.48	6.57	2.70	6.62	4.618
zero-shot decoder LLMs (18 models)
Qwen2.5-1.5B	1.5B	12.10	35.93	4.29	41.12	4.11	7.33	17.480
Qwen3-1.7B	1.7B	11.84	34.51	4.45	37.67	4.10	7.32	16.648

Appendix

Table 16: Detailed regression-probe MAE by dataset; lower is better. Naval MAE is multiplied by

10^{4}

; other columns retain their target units. Mean is the arithmetic mean of these displayed values and is a descriptive aggregate because units differ across datasets. Model denotes next-token record adaptation; Match denotes optional same-record consistency; and Tune denotes supervised fine-tuning (SFT).

floors / ceilings
Model / recipe	Resources	Supercond.	USDA	CPU	Naval	Protein	YearPred	Mean $\uparrow$
F2: char-TF-IDF probe floor	50k sparse features	0.930	0.928	0.538	0.951	0.459	0.081	.648
C1: XGBoost raw-feature ceiling	raw features	0.957	0.999	0.993	0.998	0.790	0.527	.877
zero-shot decoder LLMs (18 models)
Qwen2.5-1.5B	1.5B	0.878	0.954	0.945	0.928	0.558	0.391	.776
Qwen3-1.7B	1.7B	0.883	0.959	0.942	0.941	0.564	0.395	.781

Appendix

Table 17: Detailed regression-probe Pearson

r

Frozen decoder LMs (mean-pooled hidden states)
Model / recipe	Classification: ROC–AUC $\uparrow$	Regression: $R^{2}\uparrow$	Emb-2 $\uparrow$
Model / recipe	Census-KDD (95.3% text)	NYC-Taxi (3.0% text)	Emb-2 $\uparrow$
Qwen3-8B	.940	.444	.692
Gemma-4-12B	.937	.490	.713
MoMaTune framework (Qwen2.5-1.5B backbone)
Zero-shot frozen decoder	.941	.433	.687

Appendix

Table 18: The complete MoMaTune recipe block re-run on records that are not numerical, in the format of Tab. 1 . The percentage after each dataset is the share of non-numeric characters in a serialized record, measured over field values. Emb-2 is the unweighted mean of the two columns. All trained recipes use a Qwen2.5-1.5B backbone, one seed, and the identical configuration reported in the main results; bold marks the best value in each column. On the text-heavy dataset every recipe stays at or above the frozen baseline and the best recipe exceeds it, but the recipe ordering is compressed because a frozen decoder already reaches

0.941

there. On the dataset with headroom the ordering of Tab. 1 reappears in full.

\fnum@figure : Digit-position sensitivity across selected 1.5B arms. Model produces the largest increase over the frozen checkpoint: both Model and field-permuted Model become more sensitive to a first-digit edit. Adding Match contracts the proxy for both Model and field-permuted Model . Every edited unit is complete in the encoded input and every variant changes one tokenizer token. MiniBooNE, 200 records; 95% paired record-bootstrap intervals are reported in Tab. 19 . Model denotes next-token record adaptation; Match denotes optional same-record consistency; and Tune denotes supervised fine-tuning (SFT).

Recipe	$\bar{d}_{1}/\bar{d}_{4}$ $\uparrow$	95% CI	Spearman $\rho$ $\uparrow$
Frozen decoder	2.078	[1.135, 3.761]	0.261
Model	13.832	[9.363, 19.260]	0.342
Match only	1.008	[0.871, 1.162]	-0.011
Model + Match	6.134	[4.472, 8.106]	0.333
Field-permuted Model	17.822	[9.559, 28.420]	0.394
Field-permuted Model + Match	3.038	[1.266, 8.038]	0.376

Appendix

Table 19: Complete MiniBooNE digit-position diagnostic. The input first-versus-fourth value-change ratio is approximately 1000

\times

; it is a reference contrast, not an expected embedding ratio. Larger ratios indicate greater leading-digit sensitivity under this proxy. Intervals use 10,000 paired percentile-bootstrap resamples over records and are conditional on the evaluated checkpoints.

\fnum@figure : Same-record consistency improves incomplete-record stability and prediction when trained with permuted-prefix views. The matched field-permuted Model to field-permuted Model + Match comparison uses identical evaluation views and differs only by Match . Panel (a) reports incomplete-view displacement; lower is better. Panel (b) reports full-view and mean incomplete-view ROC–AUC from a logistic probe fitted on full-view embeddings; higher is better. Match reduces displacement by

15.305\times

(95% CI [

15.257\times

15.354\times

]) and raises incomplete-view AUC by 0.0392 ([0.0233, 0.0555]); the full-view AUC difference is

+0.00145

([

-0.00453

, 0.00763]). MiniBooNE, 3,000 records.

Recipe	Same-record margin $\uparrow$	Displacement $\downarrow$	Displacement 95% CI	Full AUC $\uparrow$	Half AUC $\uparrow$	Half AUC 95% CI	Retention $\uparrow$
Frozen decoder	-0.000243	0.002419	[0.002405, 0.002432]	0.9357	0.6937	[0.6727, 0.7148]	0.7414
Model	-0.002811	0.091574	[0.090951, 0.092195]	0.9547	0.5841	[0.5583, 0.6094]	0.6118
Match only	0.000000	0.00000306	[0.00000301, 0.00000310]	0.8903	0.7187	[0.6999, 0.7372]	0.8072
Model + Match	0.000036	0.012001	[0.011934, 0.012068]	0.9515	0.5331	[0.5096, 0.5569]	0.5602
Field-permuted Model	-0.001106	0.004678	[0.004660, 0.004697]	0.9511	0.7192	[0.7034, 0.7352]	0.7562
Field-permuted Model + Match	-0.000027	0.000306	[0.000305, 0.000307]	0.9526	0.7584	[0.7420, 0.7754]	0.7962

Appendix

Table 20: Corrected MiniBooNE field-dropout diagnostic. Lower displacement and higher margins indicate representation stability; full- and mean half-view AUC prevent collapsed representations from appearing robust. Each half is a strict subset of complete units in its fixed full view. Identical masks and pairings are used across recipes; the diagnostic contains 3,000 MiniBooNE records. The displayed 95% intervals use the paired record-bootstrap protocol above.

Arm	MB	HIGGS	Superc.	USDA	CPU	Naval	Protein	Year	Mean
Frozen	.956	.655	.769	.910	.891	.860	.311	.152	.688
Model	.968	.675	.808	.914	.954	.920	.390	.213	.730
Model + Match	.968	.671	.813	.915	.953	.923	.408	.210	.733
Match only	.941	.607	.670	.869	.873	.618	.242	.093	.614
Model , 10 $\times$ compute	.971	.691	.820	.920	.930	.936	.436	.245	.744
Field-permuted Model	.967	.664	.808	.942	.969	.948	.396	.215	.739

Appendix

Table 21: Selected 1.5B embedding ablations by dataset. Model denotes next-token record adaptation; Match denotes optional same-record consistency; and Tune denotes supervised fine-tuning (SFT).

\fnum@figure : Fixed-order Model improves target prediction and record completion at every tested scale. Panel (a) reports the unweighted mean of MiniBooNE ROC–AUC, Superconductivity

R^{2}

, and USDA

R^{2}

; panel (b) reports mean within-10% masked-value imputation on the same datasets. Gray dashed circles denote frozen checkpoints, blue circles denote fixed-order Model , and orange diamonds add Match . Relative to the frozen checkpoint, Model improves both metrics from 1.5B to 27B; adding Match produces only small and inconsistent changes. Lines connect separately trained checkpoint families whose architectures and tokenizers differ, so this is a one-seed robustness sweep rather than a fitted scaling law. Model denotes next-token record adaptation; Match denotes optional same-record consistency; and Tune denotes supervised fine-tuning (SFT).

Training recipe	Scale	Embedding axis
Training recipe	Scale	MiniBooNE AUC $\uparrow$	Supercond. $R^{2}\uparrow$	USDA $R^{2}\uparrow$	Mean $\uparrow$
Frozen decoder	1.5B	.956	.769	.910	.878
	4B	.960	.783	.940	.894
	9.6B	.954	.807	.939	.900
	27B	.958	.804	.946	.903
Model	1.5B	.968	.808	.914	.897

Appendix

Table 22: Detailed model-size results for selected training recipes on the aligned three primary datasets. MiniBooNE reports ROC–AUC, while Superconductivity and USDA report

R^{2}

; higher is better throughout, and means are unweighted over the same three datasets. All intended cells are complete. Because checkpoint architecture and vocabulary also change, this is a robustness sweep rather than a fitted scaling law. The compact main-paper counterpart is Tab. 4 . Model denotes next-token record adaptation; Match denotes optional same-record consistency; and Tune denotes supervised fine-tuning (SFT).

Model	Post-training state	MB	HIGGS	Superc.	USDA	CPU	Naval	Protein	Year	Mean
1.5B	Frozen	.956	.655	.769	.910	.891	.860	.311	.152	.688
1.5B	Model	.968	.675	.808	.914	.954	.920	.390	.213	.730
4B	Frozen	.960	.642	.783	.940	.926	.893	.332	.160	.705
4B	Model	.970	.678	.830	.944	.961	.951	.433	.227	.749
9.6B	Frozen	.954	.644	.807	.939	.928	.897	.349	.149	.708
9.6B	Model	.974	.739	.857	.958	.970	.986	.506	.240	.779

Appendix

Table 23: Full eight-dataset embedding results available through 9.6B: AUC for MiniBooNE/HIGGS,

R^{2}

otherwise. Model denotes next-token record adaptation; Match denotes optional same-record consistency; and Tune denotes supervised fine-tuning (SFT).

Model	Post-training state	MB	HIGGS	Superc.	USDA	CPU	Naval	Protein	Year	Mean
Median floor	–	.242	.232	.276	.134	.242	.238	.282	.048	.212
1.5B	Frozen	.004	.126	.250	.196	.158	.086	.000	.028	.106
1.5B	Model	.214	.288	.494	.288	.366	.914	.576	.070	.401
4B	Frozen	.006	.130	.292	.260	.194	.144	.000	.024	.131
4B	Model	.226	.296	.572	.342	.388	.914	.618	.076	.429
9.6B	Frozen	.006	.172	.280	.286	.174	.122	.000	.034	.134

Appendix

Table 24: Full eight-dataset masked-value-imputation results available through 9.6B. Model denotes next-token record adaptation; Match denotes optional same-record consistency; and Tune denotes supervised fine-tuning (SFT).

Training recipe	Scale	MiniBooNE $\uparrow$	Supercond. $\uparrow$	USDA $\uparrow$	Mean $\uparrow$
Frozen decoder	1.5B	.004	.250	.196	.150
	4B	.006	.292	.260	.186
	9.6B	.006	.280	.286	.191
	27B	.008	.350	.288	.215
Model	1.5B	.214	.494	.288	.332
	4B	.226	.572	.342	.380

Appendix

Table 25: Masked-value imputation across model sizes for selected training recipes. Values are within-10% accuracy on the aligned three primary datasets; higher is better, and the mean is unweighted. All intended cells are complete.

Recipe	Model view	Match / Tune	Embedding $\uparrow$	Imputation $\uparrow$
Frozen decoder	–	–	.903	.215
Model	default view	–	.933	.476
Model + Match	default view	Match	.934	.465
Tune only	–	Tune only	.927	.065
Extended-compute Model	default view, 10 $\times$	–	.920	.479
Field-permuted Model	permuted-prefix views	–	.936	.613

Appendix

Table 26: Complete 27B grid on the aligned three-primary-dataset scope. The embedding column is the mean primary downstream metric; imputation is within-10% accuracy. Higher is better.

Recipe	HIGGS Acc. $\uparrow$	CPU $\downarrow$	Naval $\downarrow$	Protein $\downarrow$
Task-specific floor	0.518	0.086	0.013	0.755
Frozen decoder	0.091	n/p (100%)	n/p (89%)	n/p (100%)
Model	0.014	n/p (100%)	n/p (100%)	n/p (100%)
Tune only	0.720	0.013	0.002	0.310
Model $\rightarrow$ Tune	0.714	0.022	0.002	0.301
Model + Match $\rightarrow$ Tune	0.722	0.012	0.002	0.294

Appendix

Table 27: Corrected 1.5B target-emission results on the four prompt-clean datasets. The prompt matches Tune training and uses a 496-token budget. Bold marks the best Tune recipe per dataset. “n/p” denotes no parseable numerical output, with the percentage of 500 evaluated records in parentheses. Model denotes next-token record adaptation; Match denotes optional same-record consistency; and Tune denotes supervised fine-tuning (SFT).

LLM Post-Training

Large language model post-training adapts pretrained models into more useful, reliable, and specialized assistants through supervised and preference-based learning.

See all 246 papers

Jun 16, 2026

TuneAhead: Predicting Fine-tuning Performance Before Full Training Begins

Fine-tuning large language models (LLMs) is compute-intensive and error-prone: model performance depends sensitively on data quality and hyperparameter choices, and naïve runs can even degrade model performance. This raises a practical question:can we predict fine-tuning performance before committing to a full training run? We present TUNEAHEAD, a lightweight framework for pre-hoc prediction of fine-tuning performance. TUNEAHEAD encodes each candidate run as a meta-feature vector that combines static dataset descriptors with dynamic probe features from a short standardized probe. A predictor maps these features to performance estimates, while SHAP-based attributions provide interpretable diagnostics that reveal which specific features drive the prediction. Across 1,300+ fine-tuning runs on Qwen2.5-7B-Instruct, TUNEAHEAD consistently outperforms strong baselines such as Early-Stop Extrapolation and ProxyLM. On a held-out test set of 370 runs, TUNEAHEAD achieves an RMSE of 1.47 percentage points and places 95.1% of predictions within +3/-3 percentage points of the true score. These accurate continuous predictions support practical go/no-go screening policies that can reduce unnecessary full fine-tuning while retaining most promising runs.

Yuxiang Luo, Haonan Long, Chen Wang +6

The Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Ltd.

Fine-Tuning LLM Fine-Tuning Language Model Performance Prediction Fine-Tuning Language Models

Ask PDF

Aug 13, 2026

Numeracy in Large Language Models: Fundamental Limitations and Paths to Improvement

Large language models (LLMs) achieve strong results on mathematical reasoning benchmarks yet remain unreliable on elementary numerical tasks, including magnitude comparison, large-integer arithmetic, fractions, and scientific notation. This survey examines basic numerical understanding as a capability distinct from high-level mathematical reasoning. We propose the Numerical Grounding Framework (NGF), which decomposes numeracy into Representational Grounding (RG), mapping numeral forms to value, magnitude, and equivalent representations, and Procedural Grounding (PG), executing arithmetic operations in accordance with their mathematical definitions. Using NGF, we organize recent diagnostic benchmarks, failure modes, structural explanations, and mitigation strategies. We review evidence concerning tokenization, positional encoding, embedding geometry, and pretraining-data distribution. We also apply NGF in a coordinated evaluation of three frontier model families across Number Cookbook, NumericBench, and GSM-Symbolic, comparing atomic, contextual, and reasoning-assisted numeracy. Architectural interventions such as digit-aware tokenization and Abacus Embeddings can improve models trained from scratch but are generally unavailable to users of pretrained systems, for whom supervised fine-tuning, reasoning scaffolds, and external tools are more practical. We conclude with deployment recommendations and research directions for more reliable numerical behavior in foundation models.

Aoxin Ni

Department of Computer Science University of Chinese Academy of Sciences

Numerical Reasoning in Language Models LLM Grounding

Ask PDF

Sep 1, 2026

Post-Training Science for Supervised Fine-Tuning

Every supervised fine-tuning run forces the same chain of decisions, such as learning rate, batch size, LoRA or full fine-tuning, how many epochs, which optimiser, and what data to feed the model. Each of these is typically rediscovered from scratch for every new model and dataset. Here we measure them under one instrument: a sweep that varies one lever at a time, and spans dense and mixture-of-experts models in two families (Qwen3 and Llama), on four real-world customer SFT datasets, for both LoRA and full fine-tuning. These datasets give a controlled testbed: each task carries an evaluation built with the customer, and its training data is produced by iterative supervised fine-tuning that refines model outputs until they pass that evaluation, so the supervised target is internally consistent and the task judge we report against is the criterion the data was built to satisfy. We ask how the optimal learning rate and batch size move with model scale, family, and data, and whether one selection rule transfers across them; what LoRA trades against full fine-tuning, and how its rank and alpha set what the adapter can learn; whether validation loss (or other metrics, such as loss landscape flatness) faithfully ranks downstream quality; whether post-training gains scale with model size and data volume, on a model ladder extended through mixtures-of-experts to 235B parameters; how many epochs to train before general instruction-following erodes; and whether a geometry-aware optimiser improves on AdamW. Each recommendation is paired with a measure of its uncertainty.

Charles O'Neill, Mudith Jayasekara, Harry Partridge

1Baseten

Supervised Fine-Tuning Hyperparameter Optimization Parameter-Efficient Fine-Tuning

Ask PDF

Abstract

LLM Post-Training

Explore similar work

TuneAhead: Predicting Fine-tuning Performance Before Full Training Begins

Numeracy in Large Language Models: Fundamental Limitations and Paths to Improvement

Post-Training Science for Supervised Fine-Tuning