Beyond Correctness: Evaluating Semantic Knowledge in Cross-Table Transfer

Seokyong Sheem; Hochang Lee; Suyeong Lee; Daekyum Kim

cs.LGSep 28, 2026

Beyond Correctness: Evaluating Semantic Knowledge in Cross-Table Transfer

Authors: Seokyong Sheem, Hochang Lee, Suyeong Lee, Daekyum Kim

Organizations: Korea University

Abstract

Semantic knowledge is increasingly used to bridge heterogeneous schemas in tabular learning, but how much does that knowledge actually improve prediction? Studies in tabular learning commonly answer this question through semantic ablations that modify or suppress the supplied semantic knowledge. We show that these ablations can lead to misleading conclusions about predictive benefit: poor performance under altered semantics may be taken as evidence that the intended knowledge is beneficial. Across real and controlled experiments, altering semantic content can produce large performance differences even when the model gains little predictive benefit from having that semantic knowledge in the first place. To separate these effects, we distinguish two quantities: content sensitivity and predictive utility. Content sensitivity measures the change in performance when semantic content is altered, whereas predictive utility measures the benefit of the intended semantic knowledge relative to a suitable reference without that knowledge. This distinction motivates an evaluation framework in which the control is chosen according to the question being asked: altered controls assess sensitivity to semantic content, whereas claims that semantic knowledge improves prediction require a suitable reference. Even then, predictive utility is not fixed; it varies across suitable references and decreases when the reference can more easily recover the tested knowledge from other inputs or labeled examples. In a bounded audit of 25 semantic-ablation comparisons across nine studies, only one of 18 explicit predictive-utility claims is paired with a control that clearly isolates the tested semantic contribution. Together, these findings motivate a simple evaluation principle: semantic-ablation controls should be chosen and interpreted according to the question they are intended to answer.

Figures & tables

Figure 1: Overview of the evaluation framework. (a) Semantic knowledge relates heterogeneous source and target schemas. (b) The intended condition uses the specified content, the altered condition changes it, and the reference removes the tested component while preserving the remaining setup. (c) Intended–altered measures content sensitivity , whereas intended–reference measures predictive utility ; the two comparisons support different claims.

Knowledge	Intended	Altered	Reference
Measurement	Specified coding or numerical conversion	Altered coding or numerical conversion	Raw source values
Correspondence	Specified source–target pairing	Permuted source–target pairing	No shared pairing; separate source and target columns
Relation	Specified relation and direction	Reversed direction or random relation	No added value or constraint

Table 1: Condition construction for measurement, correspondence, and relation knowledge.

Figure 2: Suitable and unsuitable references for the three roles of semantic knowledge. Suitable references remove only the tested component while preserving the remaining predictive information and setup. Unsuitable controls either remove additional predictive information or alter, rather than remove, the tested component.

Setting	Data and task	Primary model	Metric
TabLLM reanalysis	Nine original-study datasets; released-code reanalysis with the name-masked reference	TabLLM (released)	AUROC
Clinical	NH–KN and pooled NH/KN–HRS; binary questionnaire and continuous examination endpoints	XGB	AUROC (binary); negative SD-normalized RMSE (continuous)
Hydrological	CAMELS-US to CAMELS-GB v2; 120 basins (CAMELS-120)	XGB	Negative log-RMSE
Controlled benchmark	Eight-variable NHANES panel with observed values and missingness retained; constructed schemas and synthetic targets	XGB	Negative nMSE

Table 2: Summary of the experimental settings.

TabLLM feature-name ablation
Setting	Altered condition	Content sensitivity	Predictive utility	Reference $-$ Altered
9-dataset mean	List Permuted Names	$+.112$	$+.088$	$+.024$
Credit-g	List Permuted Names	$+.066$	$-.003$	$+.069$
Bank	List Permuted Names	$-.032$	$+.089$	$-.121$
Real cross-table transfer: relation knowledge
NH–KN	Reversed	$+.183$	$-.002$	$+.185$

Table 3: Decomposition of content sensitivity across the evaluated settings.

Figure 3: Reference sensitivity of TabLLM predictive utility. Points show the 24 evaluated identifier assignments, diamonds mark the primary name-masked reference, and horizontal lines show the point-estimate range; orange rows indicate ranges that cross zero. Counts at right show the number of positive estimates.

Figure 4: Predictive utility decreases as labeled target support increases. (a) Controlled measurement-knowledge utility for pairwise XGB at the largest tested measurement mismatch. (b) Mean TabLLM feature-name utility across nine datasets relative to the name-masked reference. (c) Correspondence-knowledge utility across a post hoc six-endpoint subset defined by whether the target itself is a matched variable, with the full 13-endpoint trajectory shown alongside it for comparison. Intervals are 95% CIs; metrics differ across panels and magnitudes are not directly comparable.

Claim type	Supported	Unsupported	Indeterminate
Content sensitivity ( $n=4$ )	3	1	0
Predictive utility ( $n=18$ )	1	11	6

Table 4: Claim–control alignment in the published-ablation audit.

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Target setup	Model	Model settings	Runs	Inference unit
NHANES– KNHANES	Source rows plus $n_{\mathrm{target}}=256$ labeled target rows; disjoint target query	XGB	300 trees, depth 6, learning rate .05; source and target total weights balanced	10 seeds (50–59) per endpoint	Endpoint
NHANES/ KNHANES–HRS	All eligible source rows (endpoint-specific); 1/5/10% labeled target rows	XGB	300 trees, depth 5, learning rate .05; target weight 10	10 seeds (40–49) per labeled-target fraction	Endpoint
CAMELS-US– CAMELS-GB v2	1/3/5 labeled target basins; fixed disjoint query basins	XGB	300 trees, depth 6, learning rate .05; log-target regression; target weight 10	20 seeds (20–39) per labeled-target size	Basin
Controlled	2,048 source rows; $n_{\mathrm{target}}\in\{32,512\}$ ; 2,000 query rows	XGB, HistGB, TabPFN, MLP	Primary conditions use unit weights; XGB/HistGB: 300 depth-6 iterations; TabPFN 7.0.0 defaults; MLP: one 32-unit ReLU layer	20 realizations $\times$ 3 fixed families	Realization within family

Appendix

Table A1: Common experimental settings across the real-data and controlled experiments.

Analysis	Timing / status	Interpretation
Measurement endpoint extension	Frozen extension	The broader ten-endpoint result is descriptive.
Examination width-matched reference	Post-result construction	Enables a new predictive-utility comparison; it does not retroactively make the original ablation a suitable reference comparison.
Examination target-support ladder	Direction calibrated earlier; six-endpoint panel selected after the 13-endpoint expansion	Six-endpoint trajectory is treated as replication; the full 13-endpoint trajectory is exploratory.
Relation capacity analysis	Frozen extension	Evaluates learner-capacity dependence of relation utility.
Fixed-width noisy-proxy diagnostic	Post-result diagnostic; protocol fixed before execution	Tests reconstructibility while preserving input width and variable identity; the association within each noise level is post-result and descriptive.
Real-data reconstructibility probes	Frozen extension	The reconstruction-manipulation criterion was not met in the evaluated real-data settings.

Appendix

Table A2: Provenance information for analyses whose timing affects interpretation.

Endpoint	NHANES item	KNHANES item
Diabetes history	DIQ010	ALL__de1_dg
Hypertension history	BPQ020	ALL__di1_dg
Current smoking status	SMQ040	ALL__bs3_1
Kidney disease history	KIQ022	ALL__dn1_dg
Stroke history	MCQ160F	ALL__di3_dg
Arthritis history	MCQ160A	ALL__dm1_dg

Appendix

Table A3: Survey-label endpoints used in the NHANES–KNHANES measurement experiments. The first three rows form the focused panel; all ten rows form the extended panel.

Comparison	Removal	Preservation	Interpretation
Measurement altered control	$\times$	✓	Altered condition
Permuted correspondence	$\times$	✓	Content sensitivity only
Survey ablation without added columns	✓	$\times$	Descriptive ablation
Separate-domain reference	✓	✓	Suitable reference
Reversed / random relation control	$\times$	✓	Altered conditions
Relation unconstrained reference	✓	✓	Suitable reference

Appendix

Table A4: Application of the reference criteria across the measurement, correspondence, and relation experiments. All listed comparisons satisfy the Setup criterion.

Relation	Dir.	Supporting evidence	Scope of the evidence
Age $\rightarrow$ SBP	$+$	( Franklin et al., 1997 ; Mitchell et al., 2010 )	Population trends do not imply the same association under every conditioning set.
SBP $\leftrightarrow$ DBP	$+$	( Zhu et al., 2022 ; Franklin et al., 1997 )	The population association is positive, although DBP changes differently with age.
Fasting glucose $\leftrightarrow$ HbA1c	$+$	( Nathan et al., 2008 )	The cited relationship uses average rather than fasting glucose.
Total cholesterol $\leftrightarrow$ triglycerides	$+$	( Friedewald et al., 1972 ; Wolska et al., 2026 )	Their joint use in LDL estimation is not direct evidence of a general population correlation.
Weight $\rightarrow$ waist circumference	$+$	( Flegal et al., 2009 ; Ross et al., 2020 )	Much of the cited evidence concerns BMI rather than weight alone.
BUN $\leftrightarrow$ creatinine	$+$	( Hosten, 1990 ; Morgan et al., 1977 )	Both reflect renal function, but their association is not uniformly strong across populations.

Appendix

Table A5: External evidence for the relations used in the real-data experiments. The cited evidence supports the stated direction to different degrees and does not imply that each relation is a universal constraint.

Dataset	Content sensitivity	Predictive utility	Reference–altered
Bank	-.032 [-.044,-.019]	+.089 [+.078,+.101]	-.121 [-.136,-.106]
Blood	+.038 [-.020,+.095]	+.114 [+.032,+.197]	-.076 [-.156,+.003]
California	+.084 [+.070,+.097]	+.053 [+.039,+.066]	+.031 [+.018,+.042]
Car	+.409 [+.370,+.448]	+.300 [+.268,+.333]	+.108 [+.068,+.149]
Credit-g	+.066 [+.005,+.129]	-.003 [-.047,+.039]	+.069 [+.015,+.124]
Diabetes	+.087 [+.053,+.122]	+.164 [+.102,+.224]	-.077 [-.138,-.016]

Appendix

Table A6: Released-code TabLLM reproduction using the anonymous-name reference. Values are AUROC differences with 95% paired intervals.

Panel	Content sensitivity	Predictive utility	Reference–altered	Reference AUROC
Focused panel (3 endpoints)	+.292 [+.145, +.431]	+.384 [+.202, +.553]	-.093 [-.228, +.012]	.450
Extended panel (10 endpoints)	+.249 [+.182, +.309]	+.197 [+.110, +.294]	—	.556 [.499, .608]

Appendix

Table A7: Measurement-knowledge results with

n_{\mathrm{target}}=256

. Positive differences favor the intended condition. The ten-endpoint content-sensitivity result is descriptive because its frozen finite-pair coverage gate failed.

Quantity	Comparison	Mean [95% CI]
Content sensitivity	Intended – altered correspondence	+.070 [+.029, +.121]
Predictive utility	Intended – width-matched reference	+.047 [+.017, +.084]
Reference–altered	Width-matched reference – altered correspondence	+.022 [+.002, +.043]

Appendix

Table A8: Examination correspondence results with

n_{\mathrm{target}}=256

. The width-matched reference preserves the input width and underlying values while removing the source–target correspondence.

Figure A1: AUROC loss as intended correspondences are replaced by plausible alternative matches. Thin lines show individual endpoints, the thick line shows their mean, and the dashed line shows the fully permuted correspondence.

Setting	Reversed sensitivity	Random sensitivity	Predictive utility
NHANES–KNHANES	+.183 [+.081, +.298]	-.001 [-.006, +.003]	-.002
NHANES/KNHANES–HRS	+.266 [+.175, +.367]	+.041 [+.016, +.077]	+.002
CAMELS-120	+.144 [+.127, +.163]	+.002 [+.000, +.003]	.000 [-.001, +.001]

Appendix

Table A9: Relation-knowledge results under reversed and random altered conditions. Content sensitivity compares the intended relations with the indicated altered condition, while predictive utility compares the intended relations with the unconstrained reference. Positive values favor the intended condition.

Dataset	Primary	Minimum	Maximum	Span
Bank	+.089	+.022	+.166	.144
Blood	+.114	+.093	+.246	.153
California	+.053	-.011	+.105	.116
Car	+.300	+.270	+.345	.075
Credit-g	-.003	-.040	+.045	.086
Diabetes	+.164	+.088	+.217	.130

Appendix

Table A10: Predictive utility across the 24 evaluated anonymous-name references. Primary is the anonymous reference used in Section B.1 . Minimum and maximum are descriptive point estimates across the evaluated references.

Figure A2: Reference variation and evaluation uncertainty across the 24 evaluated TabLLM references. (a) Observed between-reference variation, its estimate after accounting for evaluation uncertainty, and the typical uncertainty of an individual reference estimate. (b) Simultaneous 95% intervals for the references with the lowest and highest estimated utility in each dataset; gray points show the remaining reference estimates.

Figure A3: Predictive utility as measurement mismatch increases. Curves show the mean over 20 realizations for the additive, pairwise, and sparse outcome families. The upper row uses

n_{\mathrm{target}}=32

, and the lower row uses

n_{\mathrm{target}}=512

Model	$n_{\mathrm{target}}$	Spearman $\rho$	Full-mismatch utility
XGB	32	.78–.90	+.259–+.260
HistGB	32	.79–.89	+.257–+.264
TabPFN	32	.75–.85	+.176–+.249
XGB	512	.55–.77	+.028–+.037
HistGB	512	.63–.76	+.030–+.035
TabPFN	512	.57–.80	+.022–+.035

Appendix

Table A11: Summary of the measurement-mismatch experiment across the three outcome families. Ranges report the minimum and maximum family-level values.

Figure A4: Performance as the constituent inputs of the supplied relation are progressively removed. The intended condition changes comparatively little, whereas the reference deteriorates, increasing predictive utility. Results are shown for the pairwise and sparse outcome families at two labeled-target sample sizes.

$n_{\mathrm{adapt}}$	Predictive utility	95% CI
0	+.088	[+.034,+.153]
4	+.073	[+.012,+.144]
32	+.035	[+.017,+.054]
512	+.007	[+.003,+.011]
Decrease, $0\rightarrow 512$	+.081	[+.025,+.148]

Appendix

Table A12: TabLLM feature-name predictive utility as labeled adaptation data increase. Intervals are 95% dataset-bootstrap intervals.

Figure A5: Dataset-level TabLLM feature-name utility across labeled adaptation levels. Although mean utility decreases with additional labeled data, individual datasets show heterogeneous and often nonmonotonic trajectories.

Endpoints	$n_{\mathrm{target}}=64$	$n_{\mathrm{target}}=256$	$n_{\mathrm{target}}=1024$	Decrease, $64\rightarrow 1024$
Six endpoints	+.069 [+.037,+.102]	+.047 [+.018,+.084]	+.018 [+.008,+.028]	+.052 [+.028,+.077]
All 13 endpoints	+.039 [+.007,+.069]	+.014 [-.013,+.042]	+.008 [-.001,+.017]	+.031 [+.003,+.056]

Appendix

Table A13: Predictive utility of examination correspondence knowledge as labeled target data increase. The six-endpoint panel contains targets that are themselves matched variables, while the 13-endpoint row reports the broader endpoint set.

Setting	Original utility	Rank-aligned utility	Utility increase per reversal
$n_{\mathrm{target}}=32$ , XGB	+.259–+.260	+.169–+.275	+.088 [+.046,+.129]
$n_{\mathrm{target}}=32$ , HistGB	+.257–+.264	+.167–+.278	+.088 [+.046,+.129]
$n_{\mathrm{target}}=512$ , XGB	+.028–+.037	+.008–+.019	+.004 [+.001,+.008]

Appendix

Table A14: Summary of the controlled measurement robustness checks. Rank-aligned utility removes order-preserving differences between the source and target representations while retaining polarity reversals.

Family	$n_{\mathrm{target}}$	Depth 6	Depth 2
Additive	32	+.0244	+.0017
Additive	512	+.0212	+.0012
Pairwise	32	+.0096	+.0003
Pairwise	512	+.0105	+.0003
Sparse	32	+.0068	-.0014
Sparse	512	+.0067	-.0015

Appendix

Table A15: Effect of XGBoost depth on relation predictive utility. The baseline uses depth 6; the reduced-capacity model uses depth 2.

Model / support	Component	Result
TransTab / $n_{\mathrm{target}}=32$	Shared cross-domain identity	Utility $+.114$ – $+.129$ ; 74–80% of the intended–altered gap is associated with altered-condition degradation
TransTab / $n_{\mathrm{target}}=512$	Shared cross-domain identity	Utility $+.007$ – $+.009$ ; 97–100% of the intended–altered gap is associated with altered-condition degradation
CARTE / $n_{\mathrm{target}}=32$	Measurement / correspondence / relation	Positive utility in 0/3, 0/3, and 0/3 outcome families, respectively
CARTE / $n_{\mathrm{target}}=512$	Measurement / correspondence / relation	Positive utility in 2/3, 2/3, and 0/3 outcome families, respectively

Appendix

Table A16: Additional model coverage. For CARTE, counts report the number of outcome families with a positive predictive-utility interval out of three evaluated families.

Setting	Units	Content sensitivity	Utility / ablation
Survey correspondence / XGB	3 endpoints	all positive; +.022–+.072	-.010–+.076
Survey correspondence / HistGB	3 endpoints	all positive; +.025–+.089	-.024–+.073
NHANES–KNHANES relation	6 endpoints	all positive; +.010–+.445	-.013–+.007
NHANES/KNHANES–HRS relation	7 endpoints	all positive; +.068–+.531	-.014–+.019
CAMELS relation	12 basins	all positive; +.032–+.463	-.004–+.008

Appendix

Table A17: Unit-level variation in the main real-data comparisons. Ranges summarize descriptive endpoint- or basin-level effects.

Figure A6: Exploratory measurement–correspondence interaction analysis. (a) In the controlled benchmark, interaction decreases as an increasing fraction of the source labels is changed. (b) The initial real-data probes do not show the same decrease.

Study	Included	Basis for decision
LIFT	Yes	Explicit feature-name semantic manipulations
TransTab	No	Uses column and categorical descriptions, but no reported ablation alters or removes this semantic knowledge
TabLLM	Yes	Explicit name/value semantic manipulations
PLATO	Yes	Explicit knowledge-graph ablations
CARTE	Yes	Explicit semantic-representation ablations
FeatLLM	Yes	Explicit feature-description ablation

Appendix

Table A18: Study screening for the published semantic-ablation audit.

Coding item	Agreement	Cohen’s $\kappa$
Content-sensitivity claim	25/25 (100%)	1.000
Predictive-utility claim	24/25 (96%)	.907
Content-sensitivity support	24/25 (96%)	.815
Removal	25/25 (100%)	1.000
Preservation	15/25 (60%)	.265
Setup	22/25 (88%)	.784

Appendix

Table A19: Agreement between the two human coders before resolving disagreements.

		Content sensitivity		Predictive utility
Study	ID	Claim	Support	Claim	Support
LIFT	C01-A	Y	S	Y	U
LIFT	C01-B	Y	S	Y	U
LIFT	C01-C	N	–	Y	U
LIFT	C01-D	N	–	Y	I
TabLLM	C03-A	Y	S	N	–

Appendix

Table A20: Comparison-level coding for the 25 published semantic-ablation comparisons. Claim is coded as Y (explicit claim), N (no corresponding claim), or ? (unclear). Control support is coded as S (Supported), U (Unsupported), or I (Indeterminate); a dash indicates that the corresponding claim was not made.

Study	ID	Claim	Claim basis	Source
LIFT	C01-A	Y	The gain over shuffled names is attributed to correct feature–value association in prompt format I.	Sec. 4.1 / Table 9, p. 8
LIFT	C01-B	Y	The same interpretation attributes the gain in prompt format II to correct feature–value association.	Sec. 4.1 / Table 9, p. 8
LIFT	C01-C	Y	Correctly incorporating feature names is stated to improve performance except on CMC; this interpretation also covers the no-name comparison.	Sec. 4.1 / Table 9, p. 8
LIFT	C01-D	Y	The same qualified feature-name benefit is stated for the second no-name comparison.	Sec. 4.1 / Table 9, p. 8
PLATO	C04-A	Y	Broader domain knowledge is interpreted as improving prediction beyond a feature-only knowledge graph.	Sec. 4.1 / Table 3, pp. 7, 9
PLATO	C04-B	Y	Feature information in the knowledge graph is interpreted as improving prediction relative to the no-KG configuration.	Sec. 4.1 / Table 3, pp. 7, 9

Appendix

Table A21: Source evidence used to code predictive-utility claims in the published-ablation audit. The table includes comparisons with explicit or unclear predictive-utility claims. Claim basis summarizes the interpretation given in the original study.

Component	Version
Python (principal analyses)	3.10.19
NumPy	2.2.6
pandas	2.3.3
scikit-learn	1.7.2
XGBoost	3.1.3
TabPFN	7.0.0

Appendix

Table A22: Software versions used across the principal analyses and model-specific extensions. Model-specific environments are listed separately where they differ from the principal analysis environment.

Artifact directory	Contents
provenance/	complete record of when each analysis was designed and the associated decisions
tabllm/	complete frozen reference library, reference-construction records, reference-sensitivity results, minimum and maximum utility for each dataset, results for all references with 512 labeled adaptation examples, and utility changes with labeled examples for each reference
robustness/measurement/	absolute AUROC results for the focused measurement analysis
robustness/correspondence/	correspondence results and record of the clean rerun
robustness/relation/	evidence supporting the retained relations, results after removing inputs used to compute relation values, and checks of how well relation values can be recovered
robustness/interaction/	complete interaction results, endpoint robustness checks, label corruption checks, and source-data integrity checks

Appendix

Table A23: Map of reproducibility artifacts available in the public repository.

Explore similar work

Towards Universal Tabular Embeddings: A Benchmark Across Data Tasks
Apr 23, 2026Liane Vogel, Kavitha Srinivas, Niharika D'Souza +3Tabular Learning Text Embeddings
Technical University of Darmstadt · IBM Research, USA
STRABLE: Benchmarking Tabular Machine Learning with Strings
May 12, 2026Gioia Blayer, Myung Jun Kim, Félix Lefebvre +8Tabular Learning Mle-Bench Lite
SODA Team, INRIA Saclay, Palaiseau, France · Prior Labs · University of Freiburg +3
TEmBed-T: A Multi-Dimensional Benchmark for Table-Level Embeddings
Jul 27, 2026Ayeen Poostforoushan, Liane Vogel, Carsten BinnigText Embeddings Multi-Turn Benchmark
Independent Researcher · Technical University of Darmstadt · Technical University of Darmstadt & DFKI & hessian.AI

Apr 23, 2026, cs.LGJ/K move · Enter open · S save

Towards Universal Tabular Embeddings: A Benchmark Across Data Tasks

Liane Vogel, Kavitha Srinivas, Niharika D'Souza, Sola Shirai+2

Technical University of Darmstadt · IBM Research, USA

Tabular Learning Text Embeddings

Open paper AskPDF arXiv

Tabular foundation models aim to learn universal representations of tabular data that transfer across tasks and domains, enabling applications such as table retrieval, semantic search and table-based prediction. Despite the growing number of such models, it remains unclear which approach works best in practice, as existing methods are often evaluated under task-specific settings that make direct comparison difficult. To address this, we introduce TEmBed, the Tabular Embedding Test Bed, a unified benchmark for systematically evaluating tabular embeddings across four representation levels: cell, row, column, and table. Evaluating a diverse set of tabular representation learning models, we show that which model to use depends on the task and representation level. Our results offer practical guidance for selecting tabular embeddings in real-world applications and lay the groundwork for developing more general-purpose tabular representation models.

Abstract

Explore similar work

Towards Universal Tabular Embeddings: A Benchmark Across Data Tasks

STRABLE: Benchmarking Tabular Machine Learning with Strings

TEmBed-T: A Multi-Dimensional Benchmark for Table-Level Embeddings