Semantic map models (SMMs) construct a graph that reflects conceptual similarity by comparing cross-lingual form-meaning mappings. Experts build such models not only by obeying the data constraint, i.e., the connectivity hypothesis, which states that conceptual nodes belonging to the same form must be connected, but also by incorporating their prior semantic knowledge. Current automatic methods ignore this knowledge, which harms both accuracy and efficiency by enlarging the search space of candidate graphs. In this paper, we fuse the distributional semantic knowledge hidden in the internal representations of language models into a top-down, data-driven algorithm. This expert-like behavior re-sorts the candidates in a huge search space, guided by conceptual semantics similarity. Experiments on a self-collected benchmark of 10 datasets and comprehensive metrics show that our method builds SMMs both effectively and efficiently.
Figures & tables
Figure 1: An example of SMM construction over concepts related to Tree .
Figure 2: Semantic fusion of OCSMST. Three conceptual nodes, A, B, and C, are highlighted as the focus. In (a), the model is confused about which edge to choose because A–B and A–C have the same weight. In (b), a graph built from semantic similarity forms a semantic space that differentiates the weights of A–B and A–C. The fusion of these two graphs in (c) selects the correct edge.
Id
Name (Domain)
Source
#L
#F
#Fc
Sparsity
Avg_D
Div_D
#E
Function words
1
SUPPLEMENTARY adv.
Guo (2010)
37
82
18
0.816
2.111
1.560
19
2
PERFECTIVE marker De
Liu (2015)
14
14
9
0.737
2.00
1.25
9
3
AND Hé
Zhai (2015)
55
55
29
0.801
2.000
1.509
29
4
FINALLY Dàodǐ
Yu (2023)
-
31
8
0.605
2.25
1.56
9
5
QUANTIFIER
Li (2011)
90
90
14
0.590
1.857
1.059
13
Table 1: Well-established cases from typological literature. Three grammatical categories are split into three parts. The core concept in each case is capitalized with the core Chinese word in italics. L, F, Fc, E, Avg_D and Div_D represent languages, forms, functions, average of degrees, standard derivation of degrees.
Metric
Description
Trend
Intrinsic
Div_D
Standard deviation of node degrees
↓
Cov
Proportion of covered instances
↑
Extrinsic
Acc
Matched / total elements in G
↑
Prec
Matched edges / generated edges
↑
Table 2: Metrics for evaluating SMMs. Arrows indicate the preferred direction.
Strategy
Input and extraction
Def-only
Definition D only; sentence embedding.
Exm-only
Examples E only, indexed by target word form; last hidden state at target token (averaged if split).
DefExm-sim
Concatenation of D and E ; sentence embedding.
DefExm-com
Structured combination of domain A , definition D , and examples E ; sentence embedding.
Table 3: Four information aggregation strategies.
Method
Cov
Div_D ↓
Acc
Prec
Rec
F1
F1@5
F1 @5 ∗
F1@10
F1@10 ∗
Time ↓
OCMST
97.96
1.43
88.83
62.31
58.65
60.31
54.68
65.31
52.62
66.84
14.53
OCMST_J
74.10
0.84
82.87
43.67
41.05
42.25
44.06
48.86
43.18
50.13
0.22
OCMST_D
74.10
0.84
82.84
43.37
40.75
41.94
44.45
49.21
44.43
52.81
2.60
Qwen
86.81
2.04
81.02
40.82
45.51
42.49
-
-
-
-
357.4
Codex
100
2.54
87.95
56.73
56.19
56.44
-
-
-
-
490
OCSMST
97.59
1.43
92.25
71.84
68.01
69.76
66.88
74.69
66.11
75.81
0.09
Table 4: Overall performance for baselines and our method OCSMST along with its post-processing version (OCSMST w. m). Except Div_D and Time with seconds, other metrics are shown the percent.
Model
Type
F1
OCMST
–
60.31
Qwen3-Embedding 0.6B
Def-only
64.13
Exm-only
54.09
DefExm-sim
64.79
DefExm-com
59.31
Qwen3-Embedding 4B
Def-only
67.51
Table 5: Average Top-1 F1 (%) over 10 cases under different settings of semantic fusion with λ=0.1 .
Figure 3: Average F1 score and the number of datasets achieving the optimal λ . The tradeoff is indicated by the red dotted line, and the optimal λ is marked with a star.
Figure 4: SMMs for Dataset 9: OCMST vs. OCSMST, where OCSMST coincides with the GT. Semantic fusion identifies the correct edges for nodes W (WRITE), Fm (FORM), and U (UNDERTAKE).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Maximum F1 score among the top 5 candidates for OCMST and our method OCSMST_m.
Dialect
E
P
U
H
F
W
Fm
A
Yinchuan
1
1
1
1
0
0
0
0
Taiyuan
1
1
1
1
0
0
0
0
Wanrong
1
1
0
1
0
0
0
0
Yangzhou
1
1
0
1
1
0
1
0
Xuzhou
1
1
0
0
0
1
0
0
Xi’an
1
1
0
1
1
0
0
0
Appendix
Table 6: Form-function table for Dataset 7, whose domain is MAKE (Chinese zuò ). Column abbreviations: E = Engage, P = Produce, U = Undertake, H = Hold, F = Feign, W = Write, Fm = Form, A = Act As.
Language model benchmarking is a difficult task. Outcome reasoning alone does not test the model's conceptualization of language and popular open-source benchmarks are quickly saturated or ingested as training data. It is important to test the model's output, but augmenting these tests by characterizing semantic structure gives more insight to how models relate abstract concepts. However, the high dimensional embedding spaces are not easy to interpret. This work demonstrates how topological methods can be used to rigorously compare these spaces to low dimensional and interpretable baselines like ontologies and curated knowledge graphs. These multi-modal alignment tests make it possible to track model adaptations and test phrase understanding across multiple languages.
Tyler Ashoff, Jordan Rodu
Department of Statistics, University of Virginia. · Department of Statistics, University of Virginia
Structure-mapping forms analogies by aligning systems of relationally connected elements based on shared structure instead of surface features. We introduce a new structure-mapping algorithm: Structure-Mapping with Tight Bounds (SMTB) that is 5--15x faster than the structure-mapping engine (SME) and about 50% better at finding mappings in large nested domains. SMTB is part of the broader Cognitive Rule Engine (CRE) project, a flexible multi-language-compatible framework with an accessible Python interface to state-of-the-art C++ implementations of core algorithms commonly used in cognitive systems such as pattern matching, planning, and structure-mapping. CRE and SMTB are designed to work with a wide range of representation choices. Unlike SME, which biases higher-order correspondences in tree-like predicate logic, SMTB maximizes relational connectivity without privileging higher-order relations. This allows SMTB to work just as well over arbitrary relational graphs as it does in tree-like domains of nested predicate logic. We discuss situations where privileging "higher-orderness" in structure-mapping can cause issues, and illustrate how SMTB avoids failure modes that SME would encounter in these situations. We also provide an evaluation comparing SMTB to SME v4 over 5845 domain pairs from the SME corpus.
Daniel Weitekamp, Christopher MacLellan
School of Interactive Computing, Georgia Tech, Atlanta, GA 30332 USA
Knowledge graphs describe reality in crisp assertions, while the systems now consuming them, foundation models and autonomous agents, reason natively in probabilities. We argue that this mismatch is why the integration of language models and knowledge graphs remains a data-feeding pipeline rather than a unified reasoning architecture. We envision Semantic Bayesian World Models (SBWMs): a Web that describes the world not as a database of facts but as a shared, evolving fabric of beliefs over knowledge graphs, where ontological axioms constrain priors, observations update beliefs by Bayesian conditioning, and actions intervene upon the world. We work through what an agent gains from such a model: a home-security agent deciding whether the figure at the gate is a courier or a burglar, an actuarial estimate aggregated by entailment rather than by string frequency, a planning task that language models reliably fail, and the estimation of quantities that no document has ever stated. We then set out what the community must build to make them possible: belief annotation over RDF~1.2, probabilistic entailment regimes, semantic calibration layers, and protocols by which agents that have never met can exchange, and disagree over, calibrated beliefs.