CONE: Embeddings for Complex Numerical Data Preserving Unit and Variable Semantics

Gyanendra Shrestha; Anna Pyayt; Michael Gubanov

cs.AIMar 5, 2026

CONE: Embeddings for Complex Numerical Data Preserving Unit and Variable Semantics

Authors: Gyanendra Shrestha, Anna Pyayt, Michael Gubanov

Organizations: Florida State University · University of South Florida

Abstract

Large pre-trained models (LMs) and Large Language Models (LLMs) are typically effective at capturing language semantics and contextual relationships. However, these models encounter challenges in maintaining optimal performance on tasks involving numbers. Blindly treating numerical or structured data as terms is inadequate -- their semantics must be well understood and encoded by the models. In this paper, we propose CONE, a hybrid transformer encoder pre-trained model that encodes numbers, ranges, and gaussians into an embedding vector space preserving distance. We introduce a novel composite embedding construction algorithm that integrates numerical values, ranges or gaussians together with their associated units and attribute names to precisely capture their intricate semantics. We conduct extensive experimental evaluation on large-scale datasets across diverse domains (web, medical, finance, and government) that justifies CONE's strong numerical reasoning capabilities, achieving an F1 score of 87.28% on DROP, a remarkable improvement of up to 9.37% in F1 over state-of-the-art (SOTA) baselines, and outperforming major SOTA models with a significant Recall@10 gain of up to 25%.

Figures & tables

Figure 1. Distance Between Age and Other Attributes, Calculated Using BioBERT and CoNE .

Features	BERT	ELMo	NumBERT	BioBERT	DICE	AeNER	GenBERT	NumNet	CONE
Numeration	limited	limited	yes	limited	yes	yes	yes	yes	yes
Magnitude	yes	yes	yes	yes	yes	yes	yes	yes	yes
List maximum	limited	better than BERT	-	limited	yes	yes	yes	yes	yes
Decoding	limited	better than BERT	-	Limited	yes	yes	yes	yes	yes
Addition	limited	limited	-	limited	yes	yes	yes	yes	yes
Scalar Probing	some	limited	good	limited	-	-	-	-	yes

Table 1. CONE versus SOTA Models in Key Numerical Reasoning Capabilities and Support for Complex Numerical Data.

Age	Sex	Tumor Stage	Operating time	Blood loss	Follow-up (months)	BP (mmHg)	BMI (kg/ $\text{m}^{2}$ )
28	F	S1–3	7 hrs	3,000 mL	30	76-118	21.8 $\pm$ 2.9
34	F	S1–3	580 min	5 L	32	80-122	23.4 $\pm$ 3.2
36	F	S1–3	6 hrs	3.5 L	28	82-125	22.5 $\pm$ 3.17
42	M	S1–2	500 min	3,000 mL	25	85-130	24.1 $\pm$ 3.5
33	F	S1–3	400 min	3.2 L	20	78-120	22.2 $\pm$ 3.0
31	M	S1–2	500 min	3,000 mL	15	84-128	23.6 $\pm$ 3.4

Figure 2. A Table with Numerical Data Including Ranges and Gaussians Accompanied by Units.

Figure 3. CONE Architecture including the Encoder and Numerical Value Embeddings. The Encoder Embeddings for Numbers (

M^{E}

, shown in blue) are Fused with Numerical Value Embeddings (

M^{N}

, pink) and Refined Through a Transformer to Produce the Contextualized Representations (

M^{O}

, green).

Figure 4. Composite Embedding Structure for (a) Complex Numerical Data, (b) Numerical Ranges, and (c) Gaussians. T: Text, N: Numerical Value, SD: Standard Deviation.

	Numbers		Ranges				Gaussians
	Abs. Diff.		Euclidean		Jaccard IoU		Wasserstein
Method	$r$	$\rho$	$r$	$\rho$	$r$	$\rho$	$r$	$\rho$
BioBERT	0.067	0.064	0.398	0.355	0.267	0.248	0.038	0.039
CONE	0.989	0.798	0.997	0.786	0.498	0.465	0.689	0.663

Table 2. Correlation of Analytic Distances (Section 5.1 ) with Embedding Cosine Distances via Pearson’s

r

and Spearman’s

\rho

Figure 5. Euclidean Distance from Embedding of 0 to the Embeddings of Numbers from 0 to 100

Figure 6. PCA-transformed CONE Embedding for Numerical Ranges $[1,20]$ without Units.

Figure 7. Attributes Similarity with CONE Embeddings.

Figure 8. Units Similarity with CONE Embeddings.

Figure 9. Sample Numerical Ranges and Gaussians.

Panel of Figure 10 — Figure 10. Distance Between CONE Composite Embedding for Numerical (a) Ranges and (b) Gaussians with Attributes and Units in Figure 9 .

Panel of Figure 11 — Figure 11. Effect of "Rotating" (see Section 5.2.6 ) Individual Components in CONE Composite Embeddings: (a) Illustration of Component-wise "Rotation"; (b) Change in Cosine Similarity Relative to the Original Composite Embedding.

List Maximum (accuracy)

Decoding (RMSE)

Addition (RMSE)

Integer range

[0,10^{2}]

[0,10^{3}]

[0,10^{4}]

[0,10^{5}]

[0,10^{6}]

[0,10^{2}]

[0,10^{3}]

[0,10^{4}]

[0,10^{5}]

[0,10^{6}]

[0,10^{2}]

[0,10^{3}]

[0,10^{4}]

[0,10^{5}]

[0,10^{6}]

Random vectors

0.17

\pm

0.030

0.22

\pm

0.030

0.20

\pm

0.035

0.18

\pm

0.038

0.16

\pm

0.039

30.1

\pm

1.80

295.88

\pm

2888.62

\pm

28k

\pm

280k

\pm

41.80

\pm

0.50

412.33

\pm

4400.39

\pm

40k

\pm

405k

\pm

BioBERT

0.94

\pm

0.025

0.63

\pm

0.028

0.50

\pm

0.022

0.47

\pm

0.030

0.42

\pm

0.032

3.28

\pm

0.10

29.7

\pm

0.80

434

\pm

4.1k

\pm

0.2k

40.5k

\pm

4.61

\pm

0.20

69.2

\pm

2.10

461

\pm

4.2k

\pm

0.3k

42.1k

\pm

DICE

0.97

\pm

0.002

0.96

\pm

0.001

0.96

\pm

0.002

0.95

\pm

0.002

0.94

\pm

0.003

0.45

\pm

0.03

0.83

\pm

0.03

3.02

\pm

0.08

5.42

\pm

0.15

11.2

\pm

0.4

0.64

\pm

0.02

3.80

\pm

0.15

29.2

\pm

1.2

55.1

\pm

2.1

108.4

\pm

NTL-WAS

0.95

\pm

0.003

0.94

\pm

0.003

0.92

\pm

0.002

0.91

\pm

0.002

0.88

\pm

0.003

0.30

\pm

0.03

0.49

\pm

0.12

2.75

\pm

0.15

18.4

\pm

1.20

45.0

\pm

2.5

1.26

\pm

0.03

4.25

\pm

0.25

36.4

\pm

1.8

82.4

\pm

4.1

158.2

\pm

CONE

0.98

\pm

0.002

0.95

\pm

0.002

0.97

\pm

0.001

0.96

\pm

0.001

0.95

\pm

0.003

0.42

\pm

0.02

0.73

\pm

0.03

2.71

\pm

0.05

4.89

\pm

0.10

9.85

\pm

0.3

0.64

\pm

0.02

2.18

\pm

0.08

26.10

\pm

0.9

48.9

\pm

1.8

95.1

\pm

Table 3. Experimental Results on List Maximum, Decoding, and Addition.

Method	Dev		Test
	EM	F1	EM	F1
NumNet ( Ran et al., 2019 )	83.34 $\pm$ 0.2	86.00 $\pm$ 0.38	83.40 $\pm$ 0.18	86.42 $\pm$ 0.4
NC-BERT ( Kim et al., 2022 )	75.18 $\pm$ 1.2	77.68 $\pm$ 0.75	76.02 $\pm$ 0.9	77.91 $\pm$ 0.8
AeNER ( Yarullin and Isaev, 2023 )	83.72	86.56	83.69	86.98
CONE	83.71 $\pm$ 0.12	86.70 $\pm$ 0.2	83.74 $\pm$ 0.12	87.28 $\pm$ 0.3

Table 4. Results (in %) on the DROP Dataset 6.2.1

Method	Task	CancerKG			CovidKG			WebTable			CIUS			SAUS
		R	MAP	MRR	R	MAP	MRR	R	MAP	MRR	R	MAP	MRR	R	MAP	MRR
TAPAS	CM	0.783	0.728	0.742	0.734	0.669	0.688	0.800	0.736	0.753	0.820	0.771	0.789	0.820	0.774	0.787
TAPAS	TM	0.736	0.683	0.701	0.700	0.642	0.658	0.800	0.733	0.745	0.800	0.751	0.764	0.800	0.752	0.767
NumNet	CM	0.650	0.581	0.596	0.700	0.628	0.641	0.700	0.612	0.628	0.758	0.686	0.703	0.780	0.707	0.724
NumNet	TM	0.700	0.619	0.634	0.667	0.598	0.611	0.723	0.629	0.639	0.750	0.675	0.686	0.700	0.624	0.641
NC-BERT	CM	0.792	0.758	0.768	0.750	0.694	0.712	0.850	0.788	0.795	0.850	0.792	0.809	0.850	0.793	0.804

Table 5. Average Recall@10 (R), MAP@10 and MRR@10 for Column Matching (CM) and Tuple Matching (TM) 6.2.1 .

Dataset	ISResMat		SimFlooding		Magneto		CONE
	R	MRR	R	MRR	R	MRR	R	MRR
GDC	0.24	0.30	0.24	0.29	0.50	0.87	0.65	0.85
Magellan	0.98	0.99	1.00	1.00	1.00	1.00	1.00	1.00
WikiData	0.85	0.96	0.65	0.85	0.90	0.95	0.94	0.95
Open Data	0.60	0.78	0.45	0.75	0.75	0.95	0.88	0.90
ChEMBL	0.65	0.82	0.45	0.68	0.86	0.97	0.91	0.95

Table 6. Schema Matching (SM) Accuracy; Datasets ( Liu et al., 2025 ; Koutras et al., 2021 ) .

		CONE 1			CONE 2			CONE 3			CONE 4			CONE
Dataset	Task	R	MAP	MRR	R	MAP	MRR	R	MAP	MRR	R	MAP	MRR	R	MAP	MRR
CancerKG	CM	0.800	0.789	0.808	0.766	0.764	0.772	0.790	0.773	0.786	0.797	0.782	0.790	0.833	0.817	0.831
CancerKG	TM	0.833	0.800	0.804	0.700	0.783	0.791	0.820	0.790	0.802	0.830	0.796	0.807	0.867	0.844	0.853
CovidKG	CM	0.767	0.752	0.770	0.733	0.732	0.746	0.760	0.748	0.771	0.765	0.754	0.768	0.800	0.782	0.798
CovidKG	TM	0.700	0.724	0.736	0.700	0.658	0.688	0.720	0.705	0.725	0.752	0.723	0.755	0.800	0.781	0.792
WebTable	CM	0.900	0.806	0.814	0.900	0.801	0.802	0.920	0.818	0.832	0.920	0.821	0.835	0.950	0.842	0.858

Table 7. Ablation Study on Column Matching (CM) and Tuple Matching (TM): Average Recall@10 (R), MAP@10 and MRR@10 6.2.1 .

Figure 12. Examples of CONE Composite Embeddings Correctly Matching Complex Semantically Similar (a) Columns; (b) Tuples; and (c) A Challenging Case of Column Matching.

Numerical Reasoning in Language Models

Numerical reasoning in language models studies how systems represent, calculate with, and explain numbers across mathematical and real-world tasks.

See all 150 papers

ANI: Adaptive Numerical Injection for Unifying Semantic and Arithmetic Representations in Numerical Reasoning
Sep 30, 2026Jinsung Jeon, Seung-won HwangNumerical Reasoning in Language Models LLM Mathematical Reasoning
KAIST InnoCORE LLM · Seoul National University · Korea University
MoMaTune: Model, Match, Then Tune Language Models for Numerical Records
Oct 11, 2026Weimin Lyu, Jingfen Zhang, Lingdao Sha +4LLM Post-Training Incomplete Data Imputation
Amazon
Numeracy in Large Language Models: Fundamental Limitations and Paths to Improvement
Aug 13, 2026Aoxin NiNumerical Reasoning in Language Models LLM Grounding
Department of Computer Science University of Chinese Academy of Sciences

Sep 30, 2026J/K move · Enter open · S save

ANI: Adaptive Numerical Injection for Unifying Semantic and Arithmetic Representations in Numerical Reasoning

Jinsung Jeon, Seung-won Hwang

KAIST InnoCORE LLM · Seoul National University · Korea University

Numerical Reasoning in Language Models LLM Mathematical Reasoning

Open paper AskPDF arXiv

Precise numerical reasoning with Large Language Models (LLMs) is essential for expanding their applicability to complex real-world tasks. However, text-based tokenization often fragments numbers, significantly hindering precise arithmetic reasoning. Meanwhile, numerical embeddings, despite arithmetic precision, rely on context-agnostic substitution that disregards the semantic role of numbers as identifiers. To combine the complementary strengths, we propose ANI (Adaptive Numerical Injection), a hybrid framework that governs the selective injection of numerical features based on the semantic context. By employing a context-aware gating mechanism, we selectively inject numerical embeddings (specifically FoNE) into the latent space, explicitly preserving nominal identifiers while enhancing quantitative operands. Through extensive evaluations across various LLMs, we demonstrate that ANI enhances MATH performance by 9.5 points over the official reference model, while maintaining robust performance on general linguistic benchmarks.

Abstract

Numerical Reasoning in Language Models

Explore similar work

ANI: Adaptive Numerical Injection for Unifying Semantic and Arithmetic Representations in Numerical Reasoning

MoMaTune: Model, Match, Then Tune Language Models for Numerical Records

Numeracy in Large Language Models: Fundamental Limitations and Paths to Improvement