Beyond the Vote: When Do LLMs Add Value to Retrieval-Based Tabular Classification?

Ziquan Wang; Yuxuan Nie; Jian Ma; Yingjie Li

cs.IROct 21, 2026

Beyond the Vote: When Do LLMs Add Value to Retrieval-Based Tabular Classification?

Authors: Ziquan Wang, Yuxuan Nie, Jian Ma, Yingjie Li

Organizations: Hangzhou International Innovation Institute, Beihang University, Hangzhou, China · Peking University, Beijing, China

Abstract

Retrieval-based in-context learning (ICL) enables large language models (LLMs) to predict tabular labels from similar examples, but it remains unclear whether LLM inference improves on direct voting over those same examples. We address this question by comparing four general-purpose LLMs with distance-weighted nearest-neighbor voting (Distance-KNN) using identical retrieved neighbors across 19 tabular classification datasets. With the number of demonstrations fixed, enlarging the retrieval candidate pool improves accuracy for all predictors, yet Distance-KNN achieves the highest mean accuracy at every pool size. More detailed analyses reveal that LLM gains are concentrated where retrieved neighbors exhibit low label consensus and vary across models. These gains further depend on the average feature-space distance to the test query: under low label consensus, some LLMs outperform Distance-KNN in close and intermediate neighborhoods, but all fall behind on average in the most distant group. Complementary experiments on the number of demonstrations, prompt formulation, retrieval representation, and label-balanced demonstration selection largely preserve the overall findings. These results characterize when LLM inference adds predictive value beyond distance-weighted voting in retrieval-based tabular ICL.

Figures & tables

Figure 1: Illustration of shared-neighbor prediction in tabular classification. Distance-KNN and the LLM use the same eight retrieved neighbors. Label consensus measures the majority-label fraction, while feature proximity is measured by the mean query-to-neighbor distance. In this example, Distance-KNN correctly predicts class A, whereas the LLM predicts class B.

Figure 2: Shared-neighbor comparison of Distance-KNN and LLMs for tabular classification. For each test query, a common retriever selects $k=8$ labeled neighbors from the candidate pool. Distance-KNN uses their labels and distances for weighted voting, while the LLM uses the same neighbors as serialized demonstrations, followed by the query features. Retrieval distances are excluded from the LLM prompt. Varying the candidate-pool size while fixing $k$ enables a controlled comparison on identical retrieved examples.

Figure 3: Performance of Distance-KNN and four LLMs using identical retrieved examples (

k=8

) across 19 tabular prediction datasets with five seeds. (a) Mean accuracy as candidate-pool size increases, with shaded bands indicating

\pm 1

standard deviation across seeds. (b) Mean rank across datasets at full pools (lower is better), with methods ranked by seed-averaged accuracy and ties assigned average ranks. (c) Number of wins at full pools (higher is better), with each dataset contributing one win divided equally among methods tied for the highest accuracy.

Model	$\Delta\mathrm{Acc}$	95% CI	$p_{\rm Holm}$
Gemma3-12B	-1.93	[-3.31, -0.79]	0.001
Gemma3-27B	-0.24	[-1.12, 0.53]	0.625
Qwen3-8B	-3.53	[-5.42, -1.84]	$<0.001$
Qwen3-32B	-1.95	[-3.30, -0.77]	0.012

Table 1: Statistical comparison of full-pool accuracy between Distance-KNN and four LLMs across 19 datasets. Paired differences

\Delta\mathrm{Acc}=\mathrm{Acc}_{\mathrm{LLM}}-\mathrm{Acc}_{\mathrm{KNN}}

are computed after averaging five seeds. We report 95% paired percentile-bootstrap confidence intervals over datasets and exact sign-flip

p

-values adjusted for four comparisons using the Holm correction.

Figure 4: Accuracy differences between LLMs and Distance-KNN by retrieved-label consensus at full training pools (

k=8

). Consensus is the majority-label fraction among retrieved neighbors. Curves show mean

\Delta\mathrm{Acc}=\mathrm{Acc}_{\mathrm{LLM}}-\mathrm{Acc}_{\mathrm{KNN}}

across 19 datasets and five seeds. Shaded bands denote

\pm 1

standard deviation.

Figure 5: Accuracy differences between LLMs and Distance-KNN across retrieved-label consensus ( $k=8$ ) and feature proximity at full training pools. Cells show mean $\Delta\mathrm{Acc}=\mathrm{Acc}_{\mathrm{LLM}}-\mathrm{Acc}_{\mathrm{KNN}}$ across datasets. Columns indicate label consensus, while rows group test queries by distance percentile based on mean neighbor distance per dataset and seed (lower values indicate closer neighborhoods).

Figure 6: Effect of the number of demonstrations on classification accuracy across candidate-pool sizes. We compare (a)

k=4

, (b)

k=8

, and (c)

k=16

, with Distance-KNN and LLMs using the same retrieved neighbors at each

k

. Curves show mean accuracy across 19 datasets and three seeds. Shaded bands denote

\pm 1

standard deviation.

Figure 7: Effect of prompt formulation on classification accuracy for (a) Gemma3-12B and (b) Qwen3-8B across KV, NL and Rule-Augmented NL using identical retrieved examples (

k=8

). Curves show mean accuracy across 19 datasets and three seeds as candidate-pool size increases. Shaded bands denote

\pm 1

standard deviation.

Model	Feat-Space	Text-Embed	$\Delta\mathrm{Acc}$
Distance-KNN	85.75	69.89	+15.86
Gemma3-12B	84.19	68.51	+15.68
Qwen3-8B	82.59	58.39	+24.20

Table 2: Effect of retrieval representation on classification accuracy for Distance-KNN and two LLMs at full training pools. Feature-space retrieval is compared against text-embedding retrieval (

k=8

) across 19 datasets and three seeds.

Model	Nearest	Label-Balanced	$\Delta\mathrm{Acc}$
Gemma3-12B	84.19	76.70	-7.49
Qwen3-8B	82.59	72.01	-10.58

Table 3: Effect of demonstration selection on classification accuracy for two LLMs at full training pools. The eight nearest neighbors are compared with eight label-balanced examples selected from the 32 nearest candidates (

k=8

) across 19 datasets and three seeds.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Rows	Features	Classes	Train / validation / test
AI4I Predictive Maintenance	10,000	6	2	8,000 / 1,000 / 1,000
Car	1,728	6	4	1,382 / 173 / 173
Cardiotocography	2,126	21	3	1,700 / 213 / 213
Credit G	1,000	20	2	800 / 100 / 100
Diabetes	768	8	2	614 / 77 / 77
Dry Bean	13,611	16	7	10,888 / 1,361 / 1,362

Appendix

Table 4: Characteristics and train/validation/test split sizes of the 19 tabular classification datasets. Splits are stratified by class in an 8:1:1 ratio, and feature counts refer to the processed inputs.

Dataset	Distance-KNN	Gemma3-12B	Gemma3-27B	Qwen3-8B	Qwen3-32B
AI4I Maintenance	97.58	97.56	97.32	96.78	97.04
Car	90.17	89.25	93.06	89.83	90.87
Cardiotocography	91.92	91.27	92.02	90.42	93.52
Credit G	73.60	73.60	73.00	73.20	71.40
Diabetes	74.29	63.90	68.31	63.12	69.35
Dry Bean	92.29	91.78	92.13	92.20	92.35

Appendix

Table 5: Full-pool classification accuracy (%) of Distance-KNN and four LLMs on 19 datasets using shared retrieved neighbors (

k=8

). Each entry averages five seeds. Bold marks the highest displayed accuracy within each dataset, including ties.

Figure 8: Schematic illustration of beneficial and harmful overrides on shared queries and retrieved neighbors. Beneficial overrides occur when an LLM predicts correctly and Distance-KNN predicts incorrectly, while harmful overrides reverse these outcomes.

Figure 9: Beneficial (

B

) and harmful (

H

) override rates (%) by retrieved-label consensus at full training pools (

k=8

). Each rate uses all queries within its dataset–seed–consensus group as the denominator. Rates are averaged equally over non-empty dataset groups within each seed and then over five seeds. As shown in Figure 4 , the mean accuracy difference satisfies

\text{Acc}_{\text{LLM}}-\text{Acc}_{\text{KNN}}=B-H

	Gemma3-12B		Gemma3-27B		Qwen3-8B		Qwen3-32B
Dataset	B	H	B	H	B	H	B	H
AI4I Maintenance	0.58	0.60	0.44	0.70	0.58	1.38	0.66	1.20
Car	5.66	6.59	5.55	2.66	4.28	4.62	5.55	4.86
Cardiotocography	1.69	2.35	1.60	1.50	1.41	2.91	2.91	1.31
Credit G	4.80	4.80	5.00	5.60	4.80	5.20	5.40	7.60
Diabetes	11.17	21.56	8.57	14.55	12.21	23.38	7.79	12.73

Appendix

Table 6: Full-pool beneficial (

B

) and harmful (

H

) override rates (%) for four LLMs relative to Distance-KNN across 19 datasets (

k=8

). Within each dataset, rates use all test queries as the denominator for each seed and are averaged over five seeds. Note that

\text{Acc}_{\text{LLM}}-\text{Acc}_{\text{KNN}}=B-H

(before rounding).

Figure 10: Schematic comparison of label consensus and feature proximity. The two neighborhoods have the same label composition, with five of eight neighbors in the majority class, but differ in their distances to the query.

	Feature-space retrieval			Text-embedding retrieval
Pool	Distance- KNN	Gemma3-12B	Qwen3-8B	Distance- KNN	Gemma3-12B	Qwen3-8B
16	73.03	70.44	67.45	67.79	67.04	60.12
32	75.95	73.80	70.10	67.46	66.43	60.80
64	77.96	75.38	71.69	68.07	66.86	60.14
128	80.67	78.30	75.31	67.99	67.05	59.26
256	82.38	79.49	77.13	68.33	67.52	57.02

Appendix

Table 7: Effect of retrieval representation on classification accuracy (%) across candidate-pool sizes. Feature-space and text-embedding retrieval are compared for Distance-KNN, Gemma3-12B, and Qwen3-8B with

k=8

. Entries report mean accuracy across 19 datasets and three seeds.

Figure 11: Dataset-level classification accuracy (%) under text-embedding retrieval at full training pools (

k=8

). Results compare Distance-KNN, Gemma3-12B, and Qwen3-8B on 19 datasets, with each value averaged over three seeds.

	Gemma3-12B			Qwen3-8B
Dataset	Nearest	Balanced	$\Delta\mathrm{Acc}$	Nearest	Balanced	$\Delta\mathrm{Acc}$
AI4I Predictive Maintenance	97.57	95.60	-1.97	96.63	90.17	-6.47
Car	89.40	86.51	-2.89	90.75	84.39	-6.36
Cardiotocography	90.92	89.51	-1.41	90.45	84.98	-5.48
Credit G	75.00	69.33	-5.67	74.67	72.67	-2.00
Diabetes	64.50	64.07	-0.43	61.90	50.65	-11.26

Appendix

Table 8: Effect of demonstration selection on full-pool classification accuracy (%) for Gemma3-12B and Qwen3-8B across 19 datasets. Nearest uses the eight closest examples, while Balanced selects eight label-balanced examples from the nearest 32 candidates. Dataset entries average three seeds.

\Delta\mathrm{Acc}=\mathrm{Acc}_{\mathrm{Balanced}}-\mathrm{Acc}_{\mathrm{Nearest}}

negative values favor nearest-neighbor selection.

AI4I Predictive Maintenance
Pool	Distance-KNN	Gemma3-12B	Gemma3-27B	Qwen3-8B	Qwen3-32B
16	96.60	94.06	93.72	94.06	90.30
32	96.60	94.90	95.46	95.52	93.42
64	96.60	95.66	95.52	95.60	94.24
128	96.58	95.62	95.72	95.46	94.72
256	96.54	96.34	96.08	95.76	95.02

Appendix

Table 9: Classification accuracy (%) of Distance-KNN and four LLMs by dataset and candidate-pool size (part 1 of 3,

k=8

). Values average five seeds. Within each dataset and pool, bold and underlined values mark the highest and second-highest distinct displayed accuracies, respectively, including ties.

Iranian Churn
Pool	Distance-KNN	Gemma3-12B	Gemma3-27B	Qwen3-8B	Qwen3-32B
16	84.13	82.29	84.32	73.65	81.84
32	84.70	86.35	87.94	78.98	83.75
64	85.14	86.29	87.81	81.78	85.65
128	86.98	87.56	88.00	83.81	88.57
256	89.14	87.62	89.27	82.92	90.60

Appendix

Table 10: Classification accuracy (%) of Distance-KNN and four LLMs by dataset and candidate-pool size (part 2 of 3,

k=8

). Values average five seeds. Within each dataset and pool, bold and underlined values mark the highest and second-highest distinct displayed accuracies, respectively, including ties.

Phishing Websites
Pool	Distance-KNN	Gemma3-12B	Gemma3-27B	Qwen3-8B	Qwen3-32B
16	74.32	79.64	77.79	56.98	79.62
32	82.95	84.09	83.74	59.11	83.78
64	85.75	85.66	86.06	62.15	85.55
128	89.19	88.12	88.73	65.17	88.79
256	90.25	89.24	90.16	69.46	89.58

Appendix

Table 11: Classification accuracy (%) of Distance-KNN and four LLMs by dataset and candidate-pool size (part 3 of 3,

k=8

). Values average five seeds. Within each dataset and pool, bold and underlined values mark the highest and second-highest distinct displayed accuracies, respectively, including ties.

Tabular ICL

Tabular in-context learning (ICL) uses examples in a table to adapt pretrained models to prediction tasks without task-specific weight updates.

See all 108 papers

EdgeLM: Edge Demonstrations for Language Models' Table Understanding
Aug 5, 2026Soroush Omidvartehrani, Mohammadamin Habibollah, Mohammadreza Daviran +1In-Context Example Selection In-Context Learning
University of Alberta, Canada
Localized TabICLv2: Scaling Tabular In-Context Learning through k-NN
Aug 17, 2026Beimnet Bekele GutaTabular ICL Tabular Classification
University of Cambridge, Cambridge, United Kingdom.
Do You Even Need More Data? Rethinking LLM Deployment for Label-Scarce Tabular Classification with TabLens
May 23, 2026Daria Grushina, Kseniia Kuvshinova, Alina Kostromina +3Zero-Shot Learning LLM Evaluation
Sb AI Lab, HSE University, NES, Russia · MBZUAI, United Arab Emirates · Sb AI Lab, HSE University, Russia +1

Aug 5, 2026J/K move · Enter open · S save

EdgeLM: Edge Demonstrations for Language Models' Table Understanding

Soroush Omidvartehrani, Mohammadamin Habibollah, Mohammadreza Daviran, Davood Rafiei

University of Alberta, Canada

In-Context Example Selection In-Context Learning Tabular ICL

Open paper AskPDF arXiv

Large language models (LLMs) perform table-centric prediction through in-context learning, making demonstration selection critical to performance. Existing retrieval methods prioritize similarity to the query, but similar demonstrations often reinforce the model's likely prediction rather than reveal the distinctions needed for difficult decisions. We propose EdgeLM, a retrieval framework that instead selects edge evidence, demonstrations that are both relevant to the query and informative about the decision boundary. EdgeLM retrieves two complementary forms of edge evidence by selecting data edges, nearby examples with different ground-truth labels, and model edges, similar examples previously misclassified by the deployed model. EdgeLM requires neither model retraining nor task-specific engineering. Across five data wrangling tasks, fifteen datasets, and five open-weight and proprietary LLMs, EdgeLM consistently achieves the best or near-best performance in every setting, while ablations show that the two forms of edge evidence provide complementary benefits. Our code and datasets are publicly available at https://github.com/soroushomidvar/EdgeLM.

Abstract

Tabular ICL

Explore similar work

EdgeLM: Edge Demonstrations for Language Models' Table Understanding

Localized TabICLv2: Scaling Tabular In-Context Learning through k-NN

Do You Even Need More Data? Rethinking LLM Deployment for Label-Scarce Tabular Classification with TabLens