Rethinking Evaluation for Principled Generalization Using a Kripkean Challenge to Meaning as Use

Niyar R Barman; Omar Naim; Nicholas Asher

cs.CLOct 26, 2026

Rethinking Evaluation for Principled Generalization Using a Kripkean Challenge to Meaning as Use

Authors: Niyar R Barman, Omar Naim, Nicholas Asher

Organizations: Université de Toulouse

Abstract

Large language models operationalize an intuitive, view of meaning as use. We argue that it faces a rule-following challenge. Using Kripke's interpretation of Wittgenstein, we argue that finite histories of use do not determine a unique rule for future extension. This reframes the debate about OOD generalization but also about NLP evaluation; benchmarks that measure performance on fixed use cases or local robustness, we argue, do not by themselves establish semantic competence. To evaluate whether models preserve rules and concepts under principled domain extension, we propose to evaluate two capacities, central to this ability: abstraction and recursion; and we offer an experimental paradigm for testing both.

Figures & tables

Figure 1: Exact match on addition vs. answer length relative to each model’s maximum training operand length

n

. All models are near ceiling in distribution but collapse beyond it.

Source

Janet produces 16 eggs per day, eats 3, uses 4 for baking, and sells the remainder for $2 each. What is her daily revenue?

Target Eval 1

A laboratory produces 347 ceramic blanks per day, reserves 89 for testing, uses 53 for calibration, and sells the remainder for $17 each. What is its daily revenue?

Abstraction Eval 2

\displaystyle f(a,b,c,d)=(a-b-c)d

Transfer Eval 3

Table 1: Illustrative GSM8K transfer example. Eval 1 measures direct target solving, Eval 2 abstraction, and Eval 3 transfer through the induced abstraction.

Figure 2: Original and abstracted ASDiv problems restricted to addition and subtraction (

n=190

per condition). Original scores measure numerical-answer accuracy; abstracted scores measure formula exact match.

Figure 3: Direct solving, abstraction, and transfer in the baseline condition on 214 matched GSM8K problem and variant pairs. The two models obtain similar direct-solving scores but differ substantially in transfer accuracy.

Figure 4: Direct-solving and transfer accuracy (%) for ten models on 214 matched GSM8K problem and variant pairs.

Figure 5: (a) Exact match (EM). (b) EM vs. dimension

d

(

n=20

per point).

Figure 6: Addition with a supplied recursive rule (100 problems per point). (a) EM vs. operand length, which equals the number of recursive calls. (b) EM vs. verified EM (correct sum and correct state sequence) at five digits.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

	Model input	Answer produced by
Plain	372 + 158 =	LM head, text tokens 530
Col	<num> + <num> =	number decoder, LSB-first
	values $372,158$ via encoder	$0,3,5\rightarrow 530$
	columns $(2,7,3),(8,5,1)$
Aux	372 + 158 =	LM head, text tokens 530
	$e^{\prime}_{t}=e_{t}+g\cdot\mathrm{num}(x)$

Appendix

Table 2: The same problem under the three interfaces. Col changes the numeric interface, replacing numbers in the text stream and decoding the answer outside the LM head. Aux leaves that interface intact and changes only what the hidden states are encouraged to represent.

Backbone	$n$	Method	ID	Near-OOD	Far-OOD
GPT-2 small	6	Plain	30.18%	0.00%	0.00%
GPT-2 small	6	Col	99.50%	58.34%	1.36%
OPT-125M	6	Plain	36.42%	0.00%	0.00%
OPT-125M	6	Col	100.00%	58.96%	1.47%
SmolLM-135M	6	Plain	94.67%	15.85%	0.00%
SmolLM-135M	6	Col	99.30%	59.27%	1.47%

Appendix

Table 3: Addition under digit-length shift.

n

is the maximum operand digit length seen in training; ID uses operands of at most

n

digits, Near-OOD uses

n{+}1,n{+}2

, and Far-OOD uses

n{+}3,n{+}4

Model	Method	$n$	ID	Near-OOD	Far-OOD
Qwen3-0.6B	Plain	20	76.12%	0.00%	0.00%
Qwen3-0.6B	Abacus	20	91.80%	72.85%	34.20%

Appendix

Table 4: Qwen3-0.6B addition results comparing plain with an abacus-style positional scaffold

Model	Method	$a\bmod 10$	$b\bmod 10$	$(a+b)\bmod 10$
Llama-3.2-1B	plain	87.0%	98.1%	12.7%
Llama-3.2-1B	aux	99.6%	99.8%	84.8%

Appendix

Table 5: Linear-probe accuracy for base-10 residue information at the answer-context hidden state.

Figure 7: Direct supervision trains only on input-output mappings; trace supervision trains on state updates.

Figure 8: Per-operand-length OOD accuracy trends for addition. The x-axis shows operand length relative to the maximum training operand length: ID,

n+1

n+2

n+3

, and

n+4

. Accuracy drops sharply as operand length moves farther beyond the training regime.

Model	$a\bmod 10$	$b\bmod 10$	$(a+b)\bmod 2$	$(a+b)\bmod 5$	$(a+b)\bmod 10$	$(a+b)\bmod 100$
Llama-3.2-1B + plain FT	87.0%	98.1%	62.5%	25.4%	12.7%	1.4%
Llama-3.2-1B + aux FT	99.6%	99.8%	100.0%	95.8%	84.8%	9.6%

Appendix

Table 6: Held-out linear-probe accuracy for residue information at the answer-context hidden state. The probe asks whether base-10 arithmetic information is linearly decodable from the model state. Operand residues are already decodable after ordinary arithmetic fine-tuning, but numerical-auxiliary training makes residues of the sum much more decodable. Chance accuracy is

1/T

for a modulo-

T

residue.

Figure 9: Complete direct-solving, abstraction, and transfer results on 214 matched pairs per condition. Scaled, perturbed, and extended denote the conditions in Table 7 . Source prompts remain unchanged across columns; the scaled and perturbed conditions share generated abstractions. Values are exact numerical or algebraic-equivalence accuracies, as appropriate.

Condition	Largest target input	Construction
Baseline	Unscaled	Retains the target quantities assigned when the paired problems were constructed.
Scaled quantities	$[10^{4},10^{6})$	Places the largest input in the five- or six-digit range while preserving structural ratios, fractions, and domain constraints.
Fractional perturbation	$[10^{4},10^{6})$	Uses the scaled condition, adding 0.01–0.99 to one eligible price or measurement in 66 problems. The other 148 problems remain unchanged.
Extended scale	$[10^{12},10^{13})$	Places the largest input in the 13-digit range while preserving the reference expression and domain constraints.

Appendix

Table 7: GSM8K transfer dataset variants, each containing the same 214 pairs. Ranges describe the largest bound input per target problem, not every operand or the answer. Baseline and scaled quantities can overlap; the conditions are not disjoint magnitude bins.

Figure 10: Symbolic and numerical composition accuracy on 1,000 problems per condition. Direct prompting requests a final output; provided and model-generated decomposition request intermediate results. Scores measure presentation-normalized final-answer correctness, not correctness of every intermediate step.

Figure 11: Symbolic composition by operation count with provided and model-generated decomposition. Each point averages 200 instances from ten rules. Direct prompting is omitted. These bounded explicit-rule tests do not establish unrestricted compositional generalization.

	Zero-shot		Five-shot
Model	Orig.	Abstr.	Orig.	Abstr.
Qwen3-8B	90.0	64.7	91.6	76.3
Llama-3.1-8B	87.9	56.8	88.4	63.2
Qwen2.5-0.5B	14.2	7.9	52.1	58.9
Qwen3-0.6B	56.3	24.2	52.6	52.6
SmolLM2-360M	10.5	0.5	20.0	45.8

Appendix

Table 8: Complete ASDiv addition/subtraction scores (%), with 190 problems per condition. Original scores measure numerical-answer accuracy; abstracted scores measure formula exact match. Llama, Qwen2.5, and SmolLM2 entries use Instruct variants.

Model	EM	Verified EM	Unverified
Llama-3.1-70B-Instruct	80.0	69.0	11.0
Qwen3-8B	66.2	64.8	1.4
Qwen3-32B	62.0	61.6	0.4
Llama-3.1-8B-Instruct	58.2	37.2	21.0
OLMo-2-0325-32B-Instruct	57.4	17.6	39.8
Qwen3-0.6B	51.2	8.6	42.6

Appendix

Table 9: Geometry results (% of 500). EM is final-answer exact match. Verified EM and Unverified partition EM: unverified responses have a correct answer but a derivation with at least one error or no adequate derivation.

Unit sphere (3D) yes

Do all points lie on the unit sphere? Test

x^{2}+y^{2}+z^{2}=1

for every point.

\left(-\tfrac{7}{27},\tfrac{2}{27},-\tfrac{26}{27}\right)

\left(-\tfrac{2}{7},-\tfrac{6}{35},-\tfrac{33}{35}\right)

\left(-\tfrac{4}{13},-\tfrac{3}{13},-\tfrac{12}{13}\right)

Convex hull (2D) no

q

in the convex hull of

V_{1},V_{2},V_{3}

? Solve

q=\sum_{i}\lambda_{i}V_{i}

\sum_{i}\lambda_{i}=1

; membership holds iff every

\lambda_{i}\geq 0

q=\left(\tfrac{46}{7},\tfrac{19}{3}\right)

;

V=(3,-4),(13,-4),(3,6)

Collinearity (

d=16

, sparse) yes

Appendix

Table 10: Example problems. Each prompt states the rule and asks for the substitution and exact calculation before a final yes/no. Sparse vectors list nonzero coordinates only.

ADD(A, B, C, O): record the entry state (A, B, C, O) if A == 0 and B == 0: if C == 1: O = "1" + O return O with leading zeros removed, or "0" T = (A % 10) + (B % 10) + C D = T % 10 return ADD(A // 10, B // 10, T // 10, str(D) + O) Start with ADD(A, B, 0, "").

95 + 17 : (95,17,0,"")

\rightarrow

(9,1,1,"2")

\rightarrow

(0,0,1,"12")

\rightarrow

112

Appendix

Table 11: The recursive rule supplied in the prompt (abridged from the prompt’s register definitions), with an example execution.

A,B

are integer input registers,

C

a carry bit, and

O

a string that keeps its leading zeros until the base case.

Figure 12: Answer EM (%) for direct prompting, CoT and the supplied recursive rule on the same 100 problems per length.

Step	$A$	$B$	Ref. $(C,O)$	Qwen3-32B
0	55900	40560	0, ""	0, ""
1	5590	4056	0, "0"	0, "0"
2	559	405	0, "60"	0, "00"
3	55	40	1, "460"	0, "000"
4	5	4	0, "6460"	0, "0000"
5	0	0	0, "96460"	0, "00000"

Appendix

Table 12: Qwen3-32B on

55900+40560

. It updates

A

and

B

correctly but prepends 0 to

O

at every step, adds a state after the base case, and still gives the correct sum.

Language Model Generalization

See all 29 papers

Competence-Based Evaluation of Logical Reasoning in Large Language Models
Oct 16, 2026Jize Jiang, Chengxiang ZhaiLLM Evaluation LLM Reasoning
University of Illinois Urbana-Champaign
Neither Stochastic Parroting nor AGI: LLMs Solve Tasks through Context-Directed Extrapolation from Training Data Priors
May 29, 2025Harish Tayyar Madabushi, Melissa Torgbi, Claire BonialLanguage Model Reasoning Evaluation Language Model Generalization
University of Bath Bath, United Kingdom · DEVCOM Army Research Laboratory United States of America
Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMs
Oct 1, 2026Nagham Omar, Mahmoud Jabarin, Maya Rozenshtein +3Prompt Sensitivity LLM Evaluation
Technion – Israel Institute of Technology

Oct 16, 2026J/K move · Enter open · S save

Competence-Based Evaluation of Logical Reasoning in Large Language Models

Jize Jiang, Chengxiang Zhai

University of Illinois Urbana-Champaign

LLM Evaluation LLM Reasoning Reasoning Consistency in Language Models

Open paper AskPDF arXiv

Recent advances in large language models (LLMs) have brought impressive performance across diverse tasks; however, shortcomings remain in logical reasoning, including inconsistencies under linguistic variation. To improve the reasoning capacity of LLMs, we must be able to accurately measure their reasoning capacity. Current benchmarks primarily report single accuracy numbers on selected reasoning tasks. Such a performance-based evaluation methodology says little about a model's generalization capacity in reasoning. To address this limitation, we argue for a different evaluation paradigm, called competence-based evaluation (CBE), which tests whether models reliably apply the same competence across a family of problems. A parameterized generator scales complexity while keeping the required competence fixed, and matched answer-preserving presentations test correctness beyond one formulation. We instantiate CBE on elementary ordering tasks, scaling input size while varying presentations of the same ordering rule. Across six GPT and Qwen models, performance is reasonably good over bounded complexity ranges, but even strong models fail the competence condition across the tested family. Chain-of-thought and GRPO extend strong performance without resolving all tested failures; supervised LoRA shows little held-out transfer. CBE provides diagnostic profiles and an evaluation protocol that can grow with future models.

Abstract

Language Model Generalization

Explore similar work

Competence-Based Evaluation of Logical Reasoning in Large Language Models

Neither Stochastic Parroting nor AGI: LLMs Solve Tasks through Context-Directed Extrapolation from Training Data Priors

Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMs