cs.LGOct 31, 2026

Same Function, Different Mechanism: Implementation-Local Geometry Beyond Accuracy

Authors: Miroslav Lžičař

Organizations: Mereology Labs

Abstract

Two models can make exactly the same predictions yet respond differently when their implementations are perturbed. A lookup table can change one case independently; a shared rule couples changes across cases. We compare these mechanisms through an implementation chart: a specified local family of logit perturbations around a fixed predictive distribution. Our main result counts the independent prediction changes allowed by exact finite-group sharing: the active dimension is the number of input-output orbits minus the number of input orbits. For softened modular addition, lookup and shared cyclic charts therefore have ranks p2(p−1)p^2(p-1) and p−1p-1 despite identical predictions. The classical Fisher pullback connects these dimensions to local KL curvature; mechanism-preserving reparameterizations leave the rank unchanged. We also bound the spectral tail under approximate sharing and separate quadratic rank from higher-order learning coefficients. Trained modular-addition models verify that chart rank is measurable, while controlled compiled transformers distinguish compiler-supported intermediate charts from matched alternatives. Negative controls show where this discrimination fails. The framework evaluates specified mechanism hypotheses; it does not infer a unique mechanism from predictions or predict training dynamics.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Aug 12, 2026

Function Lives Where Variance Doesn't: Task-Weighted Charts of a Language Model's Computation

How many dimensions does a language model's computation actually use? The question is ill-posed until one names a functional. Task-weighted charts make it well-posed: low-dimensional coordinate systems fit against a chosen functional of the representation, under the functional's own metric, turning distillation into plain least squares. Across six models from three families, spanning 70m to 7B parameters, next-token prediction needs 70--90% of the residual stream's width to stay within 5% of intact perplexity, a width consumed by the rare tail of language, and the variance profile predicts none of it: two directions carry 90% of GPT-2's activation variance and almost none of its function. Dimension is per-functional: the model's own uncertainty reads from six coordinates where the full predictive distribution needs hundreds; and it grows with depth. The dissociation is exploitable: when only a few dimensions can be kept, charts trained under the functional's metric preserve the model's predictions better than variance-based or optimal linear compression.
May 14, 2026

LoMETab: Beyond Rank-1 Ensembles for Tabular Deep Learning

Recent tabular learning benchmarks increasingly show a tight performance cluster rather than a clear hierarchy among leading methods, spanning gradient boosted decision trees, attention-based architectures, and implicit ensembles such as TabM. As benchmark gains plateau, a complementary goal is to understand and control the mechanisms that make simple neural tabular models competitive. We propose LoMETab, a rank-rr generalization of multiplicative implicit ensembles. LoMETab lifts the rank-1 BatchEnsemble/TabM modulation to a rank-rr identity-residual Hadamard family by parameterizing each member weight as Wk=W⊙(1+AkBk⊤)W_k = W \odot (1 + A_kB_k^\top), where WW is shared and (Ak,Bk)(A_k, B_k) are member-specific low-rank factors. This exposes two practical diversity-control axes: the adapter rank rr and the initialization scale σinitσ_{\mathrm{init}}, and we prove that for r≥2r \ge 2 this generalization strictly enlarges BatchEnsemble's hypothesis class. Empirically, we show that this added capacity manifests as measurable predictive diversity after training: on representative classification datasets, LoMETab sustains higher pairwise KL than an additive low-rank ablation, and (r,σinit)(r, σ_{\mathrm{init}}) provides broad control over pairwise KL, varying by up to several orders of magnitude across configurations. The induced diversity is reflected in task-appropriate output-level measures: argmax disagreement for classification and ambiguity for regression, indicating that the control extends beyond pairwise KL to decision- and output-level member variation. Finally, experiments sweeping over adapter rank rr and initialization scale σinitσ_{\mathrm{init}} reveal that predictive performance is dataset-dependent over the (r,σinit)(r, σ_{\mathrm{init}}) grid, supporting LoMETab as a controllable family of implicit ensembles rather than a fixed rank-1 construction.
Jul 8, 2026

Multiplication Beyond Groups: Stratified Fourier Mechanisms in Transformer Circuits

Transformers have demonstrated a remarkable ability to learn algorithmic reasoning, yet mechanistic analyses have mostly focused on globally invertible operations such as cyclic addition and group composition. In this work, we investigate how small transformers learn modular integer multiplication over composite moduli, a fundamentally non-invertible operation due to the presence of zero-divisors. We propose the monoid extension: a localized generalization of Group Composition via Representation (GCR) that suggests the learned computation does not rely on a single global representation space. Instead, the model partitions the input space into local hierarchical algebraic regions, where group-like structure survives and Fourier mechanisms can be applied. In transformers trained on square-free modular multiplication, we find that embeddings organize around these regions, attention exhibits class-sensitive routing and low-rank write directions, and local character features explain a large fraction of the model's output logits. Our results suggest that representation-theoretic mechanisms previously identified for group operations can extend beyond groups to more general structures.