A Pre-Specified Construction-Confirmation Test of Operation-Level Causal Transfer Across Finite Isomorphic Symbolic Domains

Xinyi Shan

cs.LGAug 16, 2026

A Pre-Specified Construction-Confirmation Test of Operation-Level Causal Transfer Across Finite Isomorphic Symbolic Domains

Authors: Xinyi Shan

Organizations: Independent Researcher

Abstract

Behavioral accuracy, linear decodability, and successful activation interventions do not by themselves show that a model carries an operation-level structure from one symbolic domain to another. We ask a narrower question in finite isomorphic state spaces: if the hidden-state difference between two operations is estimated separately for each source input, does adding that difference to a mapped recipient input move the model toward the corresponding recipient answer? The design compares this input-specific intervention with wrong-operation, norm-matched random, and no-op controls, and separates candidate construction from an independently isolated confirmation split. On a frozen Qwen2.5-7B-Instruct model at layers 20--21, one route--domain--operation candidate from a family pre-specified and frozen before confirmation access, transparent | integer_mod16--letters16 | successor->predecessor, passed both PyVene splits; its confirmation intersection--union p-value was 0.000198 and its 36-family Holm-adjusted p-value was 0.006943. A subsequent NNsight 0.7.0 experiment, pre-specified and frozen before its confirmation access, tested only this selected prompt route, without candidate or layer reselection. It reproduced all 12 confirmation effect estimates, confidence intervals, and exact sign-flip p-values numerically; its 36-family Holm-adjusted p-value was 0.007141. The result is therefore limited to one prompt route and one candidate, replicated across two intervention implementations on one model revision and one layer interval. It does not establish cross-model generalization, full-family backend independence, domain-general transfer, or algebraic invariance.

Figures & tables

Figure 1: Evidence ladder used by the study. Behavioral success, decodability, causal efficacy, and transfer are distinct evidential levels; no lower level automatically establishes the next. The current P3C result reaches route-bound causal-transfer evidence for one candidate replicated across PyVene and NNsight intervention implementations on one frozen model revision and layer interval.

Stage	Model	Primary role	Evidential status
P3A	Qwen2.5-1.5B	Capability and measurement gate	Gate open
P3B	Qwen2.5-7B	Domain/operation clean screen	Strict gate closed
P3C	Qwen2.5-7B	Causal construction and confirmation	Phase success

Table 1: Roles of the three stages. Earlier stages do not provide causal evidence for P3C.

Figure 2: Result-blind schematic of the recipient-per-input operation contrast. A source-domain difference between operations

B

and

A

is extracted separately for each state

x

, then added to the recipient-domain representation at

\pi(x)=(x+5)\bmod 16

. The schematic does not select a route or candidate from the observed results.

Figure 3: PyVene full-family construction and confirmation flow. Eligibility is recomputed in each split:

36\rightarrow 12\rightarrow 5

in construction and

36\rightarrow 6\rightarrow 3

in confirmation. Only the pass sets are then intersected, yielding one both-split candidate. Four candidates are construction-only; two confirmation-only positives remain frozen, uninterpreted, and unpromoted. No funnel, inheritance, or subset relation between split-specific eligibility sets is implied.

Record class	Count
Real	336
Shuffled operation	336
Matched random	10,752
Zero delta	336
Identity patch	336
Inline self-replacement	336

Table 2: Confirmation causal and control records.

Figure 4: Frozen confirmation estimates and 95% confidence intervals for all 12 pre-specified components of transparent|integer_mod16--letters16|successor->predecessor . Points are frozen estimates and bars are frozen intervals; no bootstrap was recomputed for this figure. The four no-op classes have exact zero effect. The 32 matched-random replicates are controls, not independent samples. The NNsight selected-route confirmation reproduced this complete 12-component effect object numerically. The in-figure Holm label is the PyVene full-family value; the NNsight selected-route family adjustment is reported separately below.

Figure 5: Current stage boundary. PyVene full-family construction and independently isolated confirmation produced one both-split candidate. A separate NNsight 0.7.0 selected-route replication, pre-specified and frozen before confirmation access, then passed construction and independently isolated confirmation for the same candidate, model revision, and layers. NNsight did not repeat 36-candidate screening; the 35 unmeasured family members remained at

p=1

. The scientific terminal is S13_FULL_PASS_SINGLE_ROUTE . This does not establish backend independence or full-family cross-backend replication.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Model	Role	Causal stage entered
P3A	Qwen/Qwen2.5-1.5B-Instruct	1.5B behavioral capability qualification only	no
P3B	Qwen/Qwen2.5-7B-Instruct	7B clean screen and design narrowing; no downstream causal execution	no
P3C	Qwen/Qwen2.5-7B-Instruct	7B formal causal construction and independent confirmation	yes

Appendix

Table 3: Stage roles and model separation.

Field	Value
model_id	Qwen/Qwen2.5-7B-Instruct
model_revision	a09a35458c702b33eeacc393d103063234e8bc28
snapshot_file_count	14
snapshot_file_manifest_sha256	4ae733cf244e1ae3715c9f79a95e804d2c08fa352d5bd90230efa8f967200336
backend	pyvene
backend_version	0.1.8

Appendix

Table 4: Selected frozen execution identities.

Type	Per bucket	Total	Generation rule
real	1	336	measured source operation contrast
matched_random	32	10752	deterministic seeded random directions with exact FP16 registered-norm match
shuffled_operation	1	336	no-fixed-point mapping over six ordered operation transitions
zero_delta	1	336	zero intervention payload
identity_patch	1	336	identity patch
inline_self_replacement	1	336	inline replacement by the same activation

Appendix

Table 5: Per-bucket and total control roster.

Class	Subclass	Rows	Cells/buckets	Exact	Eligible
clean	formal	288	18	243	6
clean	diagnostic	144	9	95	0
causal_control	all	12768	336
causal_control	cached_self_replacement	336	336
causal_control	identity_patch	336	336
causal_control	inline_self_replacement	336	336

Appendix

Table 6: Clean and causal/control record composition.

Route	Domains	Transition	C pass	Conf. elig.	Conf. pass	Class	IUT $p$	Holm $p$
transparent	binary4--integer_mod16	successor->predecessor	no	no	no	neither	1	1
transparent	binary4--integer_mod16	successor->reflection	no	no	no	neither	1	1
transparent	binary4--integer_mod16	predecessor->successor	no	no	no	neither	1	1
transparent	binary4--integer_mod16	predecessor->reflection	no	no	no	neither	1	1
transparent	binary4--integer_mod16	reflection->successor	no	no	no	neither	1	1
transparent	binary4--integer_mod16	reflection->predecessor	no	no	no	neither	1	1

Appendix

Table 7: Construction and confirmation status for all 36 formal candidates.

Scope	Metric	Estimate	Lower	Upper	Exact $p$
D1 to D2	real dM	19.690188	15.485354	23.678714	1.5258789e-05
D1 to D2	normalized recovery	0.601868	0.484423	0.711212	1.5258789e-05
D1 to D2	real - mean matched random	0.549932	0.431464	0.655376	1.5258789e-05
D1 to D2	real - shuffled operation	0.374984	0.256848	0.492048	6.1035156e-05
D2 to D1	real dM	15.799307	11.270405	20.154126	1.5258789e-05
D2 to D1	normalized recovery	0.505288	0.350213	0.656517	1.5258789e-05

Appendix

Table 8: Frozen component estimates, 95% intervals, and exact one-sided sign-flip p-values.

Class	Evidence status	Permitted or prohibited wording
supported	transparent\|integer_mod16–letters16\|successor->predecessor	route-bound, single-candidate replication across two intervention implementations, on one frozen model revision and one frozen layer interval
protocol terminal success	MET WITHIN FROZEN S13 SINGLE-ROUTE SCOPE — S13_FULL_PASS_SINGLE_ROUTE	May describe terminal success only for the frozen selected route and claim boundary.
cross backend replication	SUPPORTED ONLY AS SINGLE-ROUTE REPLICATION ACROSS TWO INTERVENTION IMPLEMENTATIONS — NNsight did not repeat full 36-candidate screening	May claim selected-route replication across PyVene and NNsight implementations; do not claim backend independence or full-family cross-backend replication.
cross model generalization	NOT TESTED — formal P3C is bound to the frozen Qwen2.5-7B-Instruct revision	Do not claim cross-model generalization.
all domains all operations	NOT SUPPORTED — only one complete candidate is split-replicated	Do not claim support across all domains or all operations.
abstract route causal confirmation	NOT SUPPORTED — the diagnostic route is non-promotional and had zero eligible candidates	Do not claim abstract-route causal confirmation.

Appendix

Table 9: Supported claim and prohibited extrapolations.

Field	Value
comparison	exact int16 bit-pattern equality
control_vector_persisted_on_failure	false
failure_signature	fp16_registered_norm_match_failed
precise_norm_accumulation_dtype	float32
registered_scalar_dtype	float16
scientific_effect_value_persisted_on_failure	false

Appendix

Table 10: Final registered-norm contract. Retry-development history is not part of the scientific claim.

Symbolic Reasoning in Language Models

See all 18 papers

Is Inference Mediated by Distinct Semantic Structures in LLMs? A Mechanistic Interpretation
May 25, 2026Nura Aljaafari, Marco Valentino, André FreitasMechanistic Interpretability Language Modeling
University of Manchester, United Kingdom · University of Sheffield, United Kingdom · Idiap Research Institute, Switzerland +1
State Transfer Reveals Reuse in Controlled Routing
Apr 20, 2026Yanzhen Lu, Zhicheng Qian, Muchen Jiang +1Prompt Learning Transfer Learning
Shanghai Qizhi Institute · Southeast University · Nanjing Whale Cloud +1
A Calibrated Test of Internal Action Maps: State Signals Without Global Affine Closure
Aug 13, 2026Dekun YangHidden-State Probing State Representation Learning
Zhejiang University

May 25, 2026J/K move · Enter open · S save

Is Inference Mediated by Distinct Semantic Structures in LLMs? A Mechanistic Interpretation

Nura Aljaafari, Marco Valentino, André Freitas

University of Manchester, United Kingdom · University of Sheffield, United Kingdom · Idiap Research Institute, Switzerland+1

Mechanistic Interpretability Language Modeling Natural Language Inference

Open paper AskPDF arXiv

Predicting a label correctly does not necessarily require representing the operation that produces it. Transformer representations are known to carry label-level information, but whether they encode semantic operations producing those labels is unclear. We investigate this in Natural Language Inference using controlled premise-hypothesis pairs that differ by a single semantic transformation. Using layer-wise activations, we estimate operation-level subspaces via SVD and test their causal relevance through activation steering in four open-weight decoder models. Transformation effects are decodable with

84.8

99\%

accuracy and occupy partially distinct but overlapping subspaces, exceeding random-subspace baselines. Steering experiments show that these directions causally influence predictions, though steerability varies across models; cross-operation steering further reveals structured interference and a dissociation between subspace selectivity and cross-operation independence. These findings indicate that the models encode not only that a hypothesis relates to a premise but also, in part, how it does so, implying that mechanistic analysis and control should operate at the level of semantic operations rather than predicted labels alone.

Abstract

Symbolic Reasoning in Language Models

Explore similar work

Is Inference Mediated by Distinct Semantic Structures in LLMs? A Mechanistic Interpretation

State Transfer Reveals Reuse in Controlled Routing

A Calibrated Test of Internal Action Maps: State Signals Without Global Affine Closure