Compile the Table: Query-Calibrated Operator Compression for Tabular In-Context Learning

Xu Zhao; Jiaming Zhao; Bin Zhao; Yong Yang

cs.LGOct 8, 2026

Compile the Table: Query-Calibrated Operator Compression for Tabular In-Context Learning

Authors: Xu Zhao, Jiaming Zhao, Bin Zhao, Yong Yang

Abstract

Tabular in-context learning (ICL) has emerged as a training-free and accurate paradigm for tabular prediction, but current approaches to compressing its in-context examples face an accuracy-throughput tradeoff: fixed subsets can sacrifice accuracy, while query-specific retrieval limits cache reuse and batching across queries, reducing throughput. We propose QCOC (Query-Calibrated Operator Compression), which exploits the exchangeability and repeated use of in-context examples by compiling their full KV cache once into compact memory shared across subsequent queries. Instead of retaining raw examples, QCOC clusters their states into joint-KV prototypes, preserves per-cluster multiplicities and the original example count, and calibrates prototype values against attention query vectors produced by the in-context examples through an anchored closed-form solution. Prototype compression drives the speedup, while value fitting helps preserve accuracy. On 64 held-out OpenML-CC18 datasets, QCOC achieves the highest mean accuracy among the compared compression and retrieval methods at both retained counts. Across 12 configurations on seven long tables, it ranks first among compressed methods in ten and averages 0.23 percentage points below full context. Compressing 8,192 in-context examples to 512 memory slots yields a 10.5x cache compression ratio; excluding one-time compilation, in a single-core CPU online-serving comparison over 1,000 queries, QCOC is up to 508x faster than dynamic retrieval baselines and 1.98x faster than full-context inference. These results show that QCOC enables compact-memory reuse and efficient inference across queries while retaining accuracy close to full context.

Figures & tables

Figure 1 : Reusable operator compilation with QCOC. The full in-context cache induces a query-dependent attention operator. Attention query vectors produced by the in-context examples are passed through the full cache to obtain target outputs, and QCOC compresses the cache into

(\widetilde{K}_{\ell},\widetilde{V}_{\ell}^{*},\log w_{\ell},N)

memory reused by future queries. Joint-KV clustering, cluster-count weighting, and anchored ridge fitting implement this process.

Global row selection
Method	$M=32$	$M=64$
Random	0.6819	0.7404
Input K-means	0.7321	0.7925
LUCoS	0.7348	0.7922
Dynamic row selection
MICP	0.7905	0.8137

Table 1 : Held-out CC18 classification. Mean accuracy across 64 datasets at retained counts

M=32

and

M=64

. Full uses all available in-context examples; bold marks the best compression or retrieval method. Appendix Table 6 reports additional direct-compression methods.

Global row selection
	$N=512$			$N=2{,}048$			$N=4{,}096$			$N=8{,}192$
Method	16	32	64	64	128	256	128	256	512	256	512	1,024
Random	.6003	.6276	.6763	.6716	.7120	.7254	.6854	.7421	.7494	.7319	.7487	.7596
Input K-means	.6656	.6846	.7370	.7153	.7440	.7573	.7389	.7610	.7759	.7399	.7793	.7871
LUCoS	.6463	.6721	.7323	.7067	.7266	.7696	.7374	.7626	.7773	.7656	.7670	.7900
Dynamic row selection

Table 2 : Accuracy across in-context example counts and retained counts. Means over seven CC18 datasets. Within each in-context example count block, all compressed methods use the same retained count

M=N/32,N/16,N/8

. Bold marks the highest mean accuracy among compressed methods.

\Delta

is QCOC minus the best other compressed method, including global selection, dynamic selection, and KV compression but excluding Full, in percentage points (pp); positive values favor QCOC.

Figure 2 : Accuracy and fidelity at

M=32

on 64 CC18 datasets. A: QCOC accuracy minus every Table 1 baseline on the full test split. Dots are datasets, diamonds means, and boxes interquartile ranges, not confidence intervals. B/C: B shows errors before (open) and after (filled) value fitting; C shows mean errors for Appendix Table 6 methods. B and C use logarithmic axes.

Figure 3 : Accuracy and online serving cost. A/B: Accuracy and matched single-core serving time at

N=8{,}192

M=512

. C: Serving-time scaling at

M=N/16

. Times average 1,000-query batches; B/C use logarithmic axes. One-time setup and QCOC compilation are excluded.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Method	$M=32$	$M=64$
Random	$+17.15$ $[14.74,19.55]$	$+11.62$ $[9.77,13.53]$
Input K-means	$+12.13$ $[9.88,14.49]$	$+6.41$ $[5.12,7.77]$
LUCoS	$+11.86$ $[9.70,14.07]$	$+6.44$ $[5.16,7.82]$
MICP	$+6.29$ $[5.02,7.65]$	$+4.29$ $[3.38,5.36]$
CRUMB	$+5.01$ $[3.79,6.38]$	$+2.70$ $[1.80,3.77]$
KATE	$+4.46$ $[3.38,5.65]$	$+2.22$ $[1.24,3.38]$

Appendix

Table 3 : Paired QCOC-minus-method accuracy differences for the complete 20% CC18 test partitions. Values are percentage points with 95% dataset-bootstrap intervals; positive values favor QCOC.

Global row selection
Method	$M=32$	$M=64$
Input K-means	.7306	.7919
K-Medoids (Euclidean)	.7475	.8009
K-Medoids (cosine)	.7413	.7919
LUCoS (Euclidean)	.7278	.7922
LUCoS (cosine)	.7306	.7934

Appendix

Table 4 : Additional row-selection baselines on the same 64 held-out CC18 datasets under the diagnostic protocol. LUCoS uses the released TabClustPFN encoder followed by K-Medoids; raw K-Medoids uses standardized inputs. Global MMD uses one query-distribution-matched context.

	$M=32$			$M=64$
Method	Acc.	Passes	Slots	Acc.	Passes	Slots
Dynamic row selection
MICP	.7941	12.38	396	.8181	6.81	436
Centroid-nearest batching	.8069	20	640	.8313	20	1280
Voronoi-uniform batching	.8069	20	640	.8369	20	1280
CRUMB	.8175	20	640	.8378	20	1280

Appendix

Table 5 : Context reuse accounting under the diagnostic protocol. “Passes” is the mean number of TabICL context fits for each dataset; “slots” sums the in-context examples used across those fits. QCOC stores no selected raw rows and instead uses

M

KV slots per layer and head.

Figure 4 : Accuracy and context reuse. Rows align methods at

M=32

and

M=64

. A: Mean accuracy. B: Mean number of context fits per dataset (log scale). C: Total raw in-context examples processed across those fits. QCOC processes no raw rows after compilation; its labels report the 32 or 64 retained KV slots rather than raw-row consumption.

Global row selection
Method	Accuracy	Prob. MAE	Attention MSE	QCOC gain [95% CI]
Input K-means	.7306	—	—	+.1184 [.0922,.1462]
Dynamic row selection
KATE	.8200	—	—	+.0291 [.0137,.0444]
KV compression
Key centroids	.7800	.10079	.008627	+.0691 [.0350,.1112]

Appendix

Table 6 : Matched compression baselines on 64 CC18 datasets at

M=32

under the diagnostic protocol. Selection and merger choices are fixed before held-out evaluation. Intervals compare QCOC with the corresponding method.

Inputs	Use original $N$	Cluster-count weight	Value fit	Acc.	Prob. MAE	Attention MSE
Key only	$\times$	$\times$	$\times$	.7581	.1329	.008627
Key only	✓	$\times$	$\times$	.7800	.1008	.008627
Key only	$\times$	✓	$\times$	.7922	.1023	.003293
Key only	✓	✓	$\times$	.8259	.0507	.003293
Key only	✓	✓	✓	.8469	.0210	.001641
Joint KV	$\times$	$\times$	$\times$	.7428	.1752	.010907

Appendix

Table 7 : Factorial component analysis at

M=32

. Mean results over 64 held-out CC18 datasets in the 50-query analysis setting. The last row in each block applies the same anchored value fit. “Use original

N

” evaluates count-dependent query transformations at the original in-context example count; crossing it out evaluates them at the compressed slot count

M

. Attention MSE measures KV reconstruction on fixed attention query vectors from the in-context examples, scaled by

N

, and therefore does not measure changes in query scaling.

Figure 5 : Component-wise accuracy and fidelity. The stages are initial joint-KV prototypes, restoration of original

N

and cluster-count weighting, and anchored value fitting. A: Mean accuracy. B: Mean probability MAE to Full. Error bars are dataset-bootstrap 95% confidence intervals. Lines connect discrete constructions rather than intermediate optimization steps.

Method	Accuracy	Prob. MAE $\downarrow$	Attention-output MSE $\downarrow$	$\Delta$ accuracy
Class-balanced random	.6922	—	—	+.1528 [.1234,.1841]
Input K-means	.7184	—	—	+.1266 [.0937,.1613]
LUCoS (Euclidean)	.7244	—	—	+.1206 [.0897,.1525]
LUCoS (cosine)	.7366	—	—	+.1084 [.0784,.1406]
MICP	.7966	—	—	+.0484 [.0312,.0659]
CRUMB	.8263	—	—	+.0187 [.0066,.0312]

Appendix

Table 8 : Matched baselines on TabPFN v3. Probability MAE and attention-output MSE apply only to direct cache-compression methods and measure error relative to dequantized full-context inference. The final column gives the paired QCOC-minus-baseline accuracy difference with a 95% dataset-bootstrap interval.

Figure 6 : Cross-model accuracy comparison. Points show mean QCOC-minus-baseline accuracy differences and bars show 95% dataset-bootstrap intervals over the same 64 CC18 datasets. TabICL uses the diagnostic protocol and TabPFN the transfer protocol; both panels share the same horizontal scale.

Budget	Method	Norm. RMSE $\downarrow$	$R^{2}\uparrow$
32	Full context	.4582	.7398
	KATE	.5261	.6728
	Weighted joint-KV	.5796	.6204
	QCOC	.5127	.6864
64	Full context	.4582	.7398
	KATE	.4946	.7074

Appendix

Table 9 : TabArena regression over 13 datasets and three official splits per dataset. Lower normalized RMSE and higher

R^{2}

are better; bold marks the best compressed or retrieval method at each retained count. KATE uses the same encoder-space retrieval protocol as in classification.

Dataset	Full	Input K-means	MICP	CRUMB	KATE	Weighted KV	QCOC
adult	.871	.841	.850	.863	.859	.865	.870
bank-marketing	.758	.740	.748	.739	.737	.763	.754
connect-4	.810	.734	.779	.770	.793	.762	.807
electricity	.878	.880	.854	.869	.883	.880	.871
jungle chess	.878	.809	.868	.885	.884	.868	.879
nomao	.968	.943	.954	.960	.971	.959	.971

Appendix

Table 10 : Results for the seven CC18 long-table datasets listed in Appendix C . electricity and bank-marketing use chronological holdouts; the other five use fixed stratified splits. All methods use 8,192 in-context examples and 1,000 test rows; compressed methods use

M=512

Benchmark	Full	MICP	CRUMB	KATE	Weighted KV	QCOC	Cache reduction
CC18-Long (7 datasets)	.8077	.7930	.7976	.8019	.7979	.8081	10.53 $\times$
SUSY	.8067	.7930	.7890	.7893	.7953	.8077	12.09 $\times$
HIGGS	.7167	.6803	.6853	.7007	.7090	.7110	10.94 $\times$
Covertype	.8580	.8130	.8157	.8530	.7707	.8153	9.03 $\times$

Appendix

Table 11 : Long-context accuracy and cache reduction. All results use

N=8{,}192

and

M=512

; external-dataset accuracies average three splits. Cache reduction compares aggregate cache bytes with Full, not memory use against row-selection methods. Bold marks the best compressed method.

$M$	Accuracy	Gap to Full	Agreement	Cache compression
256	.8056	$-.0021$	.9626	15.50 $\times$
512	.8081	$+.0004$	.9751	10.53 $\times$
1,024	.8069	$-.0009$	.9780	6.42 $\times$

Appendix

Table 12 : Memory-budget sweep on the same seven long tables. Cache compression is the ratio of aggregate full-cache bytes to aggregate QCOC cache bytes; it is not a comparison with row-selection memory.

Global row selection
	$N=512$			$N=2{,}048$			$N=4{,}096$			$N=8{,}192$
Method	16	32	64	64	128	256	128	256	512	256	512	1,024
Random	1.83	2.05	2.14	2.18	2.36	2.69	2.38	2.76	3.57	2.76	3.51	5.21
Input K-means	2.06	2.15	2.21	2.27	2.42	2.75	2.44	2.73	3.50	2.82	3.59	4.92
LUCoS	1.96	2.05	2.18	2.15	2.42	2.78	2.42	2.68	3.42	2.77	3.59	4.94
Dynamic row selection

Appendix

Table 13 : Single-core CPU serving milliseconds per query across all in-context example count and retained-count settings. Means are over the same seven datasets and input-size settings as Table 2 . Rows are methods with a complete matched CPU serving sweep; each

N

block uses

M=N/32,N/16,N/8

from left to right. Values use each method’s intended per-query, routed-batch, or fixed-batch execution and exclude selection and compilation.

Method	$512/32$	$2{,}048/128$	$4{,}096/256$	$8{,}192/512$
QCOC	1.86	21.01	74.36	283.40

Appendix

Table 14 : QCOC single-core CPU compilation time (seconds). One run on the adult dataset per setting on an Intel Xeon 6767P, with

M=N/16

. Times begin from the constructed full cache and include prototype construction, count estimation, and value fitting.

Dataset	Split	Full	QCOC	KATE	Weighted KV	CRUMB	MICP
SUSY	1	.797	.797	.773	.774	.779	.780
	2	.813	.813	.797	.812	.788	.795
	3	.810	.813	.798	.800	.800	.804
	Mean	.8067	.8077	.7893	.7953	.7890	.7930
Covertype	1	.855	.821	.854	.782	.823	.815
	2	.885	.835	.873	.788	.835	.827

Appendix

Table 15 : External long-table accuracy by split. All runs use

N=8{,}192

M=512

, and 1,000 test queries.

$M$	Accuracy	Gap to Full	Agreement	$\rho(M)$
512	.8153	$-4.27$ pt	.8973	.511
1,024	.8317	$-2.63$ pt	.9357	.467

Appendix

Table 16 : Covertype budget sensitivity. Means over three splits at

N=8{,}192

. Agreement is measured against Full, and

\rho(M)

is the normalized attention-output residual defined in Appendix C .

Figure 7 : Reconstruction residual and accuracy loss. Each point is one QCOC-compressed long table; circles denote seven CC18 tables and squares three external tables. The dashed line is a descriptive least-squares fit (

r=0.74

; dataset-bootstrap 95% confidence interval

[-0.20,0.99]

), not a confidence band or a calibrated decision rule.

Explore similar work

Aug 11, 2026cs.LG

TACTICL: Task-Aware Compression of Tabular ICL Models

The strong performance of foundation models for tabular tasks comes at substantial inference costs. Distilling models into task-specific architectures reduces model size and computational demands but also sacrifices in-context adaptability. Here we introduce TACTICL, an automated task-aware compression framework for tabular in-context learning models that jointly prunes transformer layers and replaces them with lightweight adapters trained on downstream tasks, thus blending in-context with in-weight learning. We study TACTICL on 47 benchmark datasets and show that we can substitute up to 85% of layers without substantial performance drop on a given downstream task. We further show that TACTICL maintains robustness to data shifts, leaving its in-context ability intact. Overall, TACTICL provides a robust framework for exploiting the depth-wise redundancy of tabular foundation models by combining task-specific adaptation and structured compression. We provide the code at: https://github.com/Hebog/tfm_compression

Mykhailo Koshil, Matthias Feurer, Katharina Eggensperger

TU Dortmund University, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany

Model Compression Tabular Foundation Models Adapter Tuning Tabular ICL Neural Network Pruning

Ask PDF

Oct 5, 2026cs.LG

Closing the Context Gap: Activation Alignment for Tabular In-Context Learning

Tabular foundation models perform in-context learning (ICL) by conditioning predictions on labeled training examples provided as context. Unlike traditional models that separate training from inference, these models must process all training examples in every forward pass, making each prediction expensive. Restricting the number of training examples reduces this cost but substantially degrades performance. Instead of discarding context, we propose activation alignment, a method that leverages the full context to teach a model how to behave when seeing only a subset. This is achieved by training a lightweight linear transformation on synthetic unlabeled data to map the intermediate activations of a data-constrained "student" (using partial context) toward those of a full-context "teacher" (using all data). Training the aligner requires no GPU and converges in seconds to minutes on commodity hardware. We evaluate on 38 classification datasets from the TabArena benchmark using the leading two tabular foundation models, TabPFN-3 and TabFM. Across all context budgets, the aligned student yields broad, statistically significant improvements over the unaligned baseline for both models. In low-data regimes, alignment recovers nearly half of the teacher's predictive advantage. The method provides a practical, low-overhead approach to achieving the inference speed of compact contexts while closing a significant fraction of the performance gap to the full-context teacher.

Yoel Zeldes

Independent Researcher

Tabular Foundation Models Tabular ICL Feature-Based Knowledge Distillation Representation Alignment

Ask PDF

Sep 24, 2026cs.CL

MILO: Efficient Many-shot In-Context Learning with Block-wise Low-rank Compression

Many-shot in-context learning (ICL) enables large language models (LLMs) to adapt to complex tasks by conditioning on thousands of demonstration examples, but this paradigm shifts the inference efficiency bottleneck to the key-value (KV) cache memory. Due to the linear scaling behavior of the KV cache, storing these intermediate tensors has become a paramount challenge for both online serving and on-device deployment. To address this issue, we propose a novel compression framework, termed MILO, that exploits the low-rank redundancy inherent in many-shot contexts. Specifically, MILO features a block-wise low-rank compression strategy that compresses the KV cache at the block granularity, where each block contains multiple many-shot examples. Furthermore, to handle the heterogeneous context density across different blocks, MILO dynamically allocates rank budgets based on the information entropy, preserving the fidelity of critical blocks while aggressively compressing redundant ones. Experimental results on Qwen2.5 models demonstrate that our method achieves up to 50% reduction in KV cache memory and 1.8x throughput improvement, with negligible performance degradation on classification and reasoning benchmarks, significantly outperforming prior baselines.

Youpeng Zhao, Tian Tan, Liqian Peng +2

Google · University of Central Florida

Memory-Efficient Inference LLM Inference Acceleration In-Context Learning Low-Rank Compression KV-Cache Compression

Ask PDF