Inducing Fourier Geometry: A Smoothness Prior Imposes Approximate Translation Equivariance in Multimodal Stellar Embeddings

Ilay Kamai; Alex M. Bronstein; Hagai B. Perets

cs.LGOct 16, 2026

Inducing Fourier Geometry: A Smoothness Prior Imposes Approximate Translation Equivariance in Multimodal Stellar Embeddings

Authors: Ilay Kamai, Alex M. Bronstein, Hagai B. Perets

Organizations: Technion - Israel Institute of Technology · ISTA - Institute of Science and Technology Austria

Abstract

Learned representations often contain geometric structure induced by symmetries in the underlying data. We ask whether a known continuous parameter space can instead be used to induce such structure when the corresponding symmetry is only approximate. We introduce a graph-Laplacian smoothness prior that encourages representations to vary smoothly over a physical parameter space without specifying a representation geometry; in the continuum limit it selects Laplacian eigenfunctions and therefore predicts the Fourier-like geometry associated with translation symmetry. On multimodal stellar representations spanning six astronomical surveys, the prior induces Toeplitz Gram structure and sinusoidal modes along stellar parameters, including in modalities whose native representations lack this geometry, while modality-specific residuals preserve structure outside the imposed coordinates. Of the exact-symmetry predictions, stationarity in a recoverable warped coordinate survives; diagnostics assuming exact quantization or asymptotic rates do not. The prior also improves downstream linear-probe performance and label efficiency, suggesting a general strategy: where a meaningful continuous parameter space exists, its local geometry can serve as a data-informed prior.

Figures & tables

Figure 1: Translation symmetry in the shared latent space. Upper row: our full model, with $\mathcal{L}_{\mathrm{smooth}}$ . Fourier modes in $T_{\mathrm{eff}}$ with frequency matching eigenmode degree, and principal-component manifolds matching the prediction of Karkada et al. (2026) . Lower row: the same model without $\mathcal{L}_{\mathrm{smooth}}$ , showing neither.

Figure 2: Effect of $\mathcal{L}_{\mathrm{smooth}}$ on downstream tasks. Linear probes on the frozen shared latent, with $\mathcal{L}_{\mathrm{smooth}}$ (black) and without (magenta), against the number of labelled training stars.

	Kepler	TESS	LAMOST	APOGEE	GALAH
Kepler	91,213	69,701	66,615	20,712	0
TESS		1,247,706	935,262	226,347	188,513
LAMOST			1,053,269	135,429	60,388
APOGEE				310,347	55,623
GALAH					247,444

Table 1: Survey census and pairwise overlaps in the cross-match master (unique stars; Gaia is the anchor and covers essentially all 1,386,210 stars). The diagonal gives each survey’s total; off-diagonal entries give the number of stars observed by both surveys.

Modality	Architecture	Params	Input Length	$d_{i}$
Gaia XP	Spectral ViT	38.5 M	$340$	512
LAMOST	Spectral ViT	615.3 M	$4096$	2048
APOGEE	Spectral ViT	154.2 M	$8576$	1024
GALAH	Spectral ViT	154.4 M	$14{,}420$	1024
Kepler	Dual-view LC	129.6 M	$6\times 21{,}600$	768
TESS	Dual-branch LC	139.5 M	$4\times 1000$	512

Table 2: The six frozen unimodal encoders. Input Length is the number of flux bins (spectra) or channels and time samples (light curves).

d_{i}

is the feature dimensionality.

Figure 3: Where the smoothness penalty acts. Each star’s graph degree, relative to the batch median ( $\log_{10}$ ), binned in $(T_{\mathrm{eff}},\log g)$ . Left: without the density correction ( $\alpha{=}0$ ), the weight concentrates on the densest regions, the main sequence and red clump, and giants, hot stars and subgiants are nearly unweighted. Right: with it ( $\alpha{=}1$ , our model), the weight spreads across a wider part of the HR diagram; the few very high-degree bins at the sparse edges are stars the correction up-weights where the density estimate is thin.

Figure 4: Same as Figure 1 , but for $\log g$ .

Figure 5: Same as Figure 1 , but for [Fe/H].

Figure 6: Fourier geometry per modality and layers in the model for $T_{\rm{eff}}$ . Each column represents a specific modality embedding. Each row represents different parts in the model - raw pretrained features, $z_{i}$ , (upper), shared space (middle), and residual space (lower). We see that the shared space improves Fourier geometry compared to $z_{i}$ across all modalities. The residual stream preserves the geometry of $z_{i}$ .

Figure 7: same as Figure 6 , for $\log g$

Figure 8: same as Figure 6 , for [Fe/H]. Note that this is not a graph coordinate.

Figure 9: Effect of $\mathcal{L}_{\mathrm{smooth}}$ on the latent space. Left: UMAP of the shared latent space with $\mathcal{L}_{\mathrm{smooth}}$ . Right: the same model without it. The prior yields a more continuous latent space with a clean $\log g$ gradient.

Graph	$T_{\mathrm{eff}}$	$\log g$	[Fe/H]	Age	Act.	$v\sin i$	$R^{2}_{\mathrm{Toep}}$
Physical (rerun)	0.872	0.873	0.530	0.557	0.165	0.481	0.852
Physical (paper)	0.860	0.875	0.502	0.562	0.167	0.509	0.889
No Laplacian	0.781	0.762	0.435	0.490	0.123	0.474	0.759
Permuted	0.548	0.449	0.180	0.302	0.005	0.399	0.394
Latent self-graph	0.705	0.504	0.204	0.448	0.109	0.442	0.775
Input $z_{i}$	0.826	0.869	0.326	0.478	0.124	0.486	0.846

Table 3: Graph controls: linear-probe

R^{2}

on the shared token (

n_{\mathrm{train}}=1000

, 30 paired draws) and

R^{2}_{\mathrm{Toep}}

along

T_{\mathrm{eff}}

11{,}347

clean main-sequence singles with LAMOST, APOGEE and Gaia , fixed so that survey coverage cannot drive the comparison. All arms share data, seed and recipe; only the Laplacian’s graph differs. The two FAME rows are the same recipe trained twice and set the noise level (

\lesssim 0.03

in probe

R^{2}

0.04

R^{2}_{\mathrm{Toep}}

). One training seed per arm; best checkpoints fall at different epochs (3 to 79).

	Kepler	LAMOST	APOGEE	TESS	Gaia XP	GALAH
$n$	1,819	14,161	20,000	20,000	20,000	20,000
Raw $z_{i}$	$+449.38_{0}$	$\mathbf{-1.16}_{2}$	$\mathbf{-1.56}_{1}$	$+22.44_{0}$	$+4.21_{0}$	$\mathbf{-1.69}_{1}$
Vanilla VICReg	$-0.95_{0}$	$-0.19_{0}$	$-0.24_{0}$	$-0.05_{0}$	$-0.31_{0}$	$+0.34_{0}$
Shared, no $\mathcal{L}_{\mathrm{smooth}}$	$\mathbf{-1.26}_{1}$	$\mathbf{-1.13}_{2}$	$\mathbf{-1.00}_{2}$	$+0.04_{0}$	$\mathbf{-1.07}_{2}$	$+0.82_{0}$
Shared, with $\mathcal{L}_{\mathrm{smooth}}$	$\mathbf{-1.40}_{2}$	$\mathbf{-1.37}_{2}$	$\mathbf{-1.50}_{2}$	$\mathbf{-1.33}_{2}$	$\mathbf{-1.38}_{2}$	$\mathbf{-1.54}_{1}$
Residual, with $\mathcal{L}_{\mathrm{smooth}}$	$+446.25_{0}$	$\mathbf{-1.10}_{2}$	$-0.89_{0}$	$+5.73_{0}$	$+3.35_{0}$	$\mathbf{-1.64}_{1}$

Table 4: Marginal shape by survey block: excess kurtosis of PC1 (

\gamma_{2}

; pure 1-D arcsine limit

-1.5

, Gaussian

0

), with strong depth as a subscript: the number of leading contiguous PCs with

\gamma_{2}<-1

. Bold marks

\gamma_{2}(\mathrm{PC1})<-1

. Star sets differ per block (

n

in the header row), so this table reports coverage , not cross-arm magnitude; for the latter see Table 5 . Kepler is the minimum stratum (

n=1{,}819

) and its verdict is the least secure.

\mathcal{L}_{\mathrm{smooth}}

is the density-normalised (

\alpha{=}1

) form throughout: a fixed-

\epsilon

kernel divided by

q_{a}q_{b}

, with the VICReg variance and covariance statistics importance-weighted by

1/q

. The prior takes the signature from three blocks to six and tightens it onto the arcsine value; without the prior the same architecture retains it in four blocks, marginally, and loses TESS and GALAH.

Arm	Panel	$\gamma_{2}$ (PC1)	$\gamma_{2}$ (PC2)	Strong depth	Med. PC1–8	Bulk
Raw $z_{i}$	LAMOST	$-1.38$	$-1.05$	2	$+0.10$	0.00
	APOGEE	$-1.53$	$-0.45$	1	$+1.04$	0.00
Vanilla VICReg	LAMOST	$-0.29$	$-0.32$	0	$-0.02$	0.00
	APOGEE	$-0.42$	$-0.52$	0	$-0.08$	0.00
	pooled	$-0.48$	$-0.60$	0	$-0.08$	0.00
Shared, no $\mathcal{L}_{\mathrm{smooth}}$	LAMOST	$-0.86$	$-1.10$	0	$+1.52$	0.00

Table 5: Common star set and calibration. Upper block: tier cross (clean main-sequence stars with both LAMOST and APOGEE present,

n=10{,}214

; pooled is the modality-averaged token on the largest exact availability cell,

n=4{,}584

). Identical stars in every row, so this is the only cross-arm comparison we make. Lower block: synthetic manifolds pushed through the identical pipeline at the measured signal fraction (

0.899

). The density-matched control is the honest reference: a perfect Fourier manifold at this sample’s own

T_{\mathrm{eff}}

density reaches strong depth

2

, so the achievable depth here is two to three directions.

\mathcal{L}_{\mathrm{smooth}}

is the density-normalised (

\alpha{=}1

) form throughout. Single seed,

\varepsilon=0.0139

Geometric Representation Learning

Geometric representation learning encodes data to preserve its intrinsic shape, distances, symmetries, and relationships.

See all 259 papers

Group-Invariant Statistics Determine Embedding Geometry: Harmonic Analysis of Representations from Bach to the Night Sky
Sep 30, 2026Liam Storan, Andreas Tolias, Nina MiolaneWord Embeddings Representation Geometry in Language Models
Stanford University · University of California, Santa Barbara
Symmetry in language statistics shapes the geometry of model representations
Feb 16, 2026Dhruva Karkada, Daniel J. Korchinski, Andres Nava +2Representation Learning Neural Representation Geometry
1UC Berkeley · 2EPFL · 3Johns Hopkins +1
Fisher-IRG: Fisher-Induced Local Invariant Representation Geometry across Language and Vision Models
Sep 29, 2026Abdullah All Tanvir, Xin ZhongInformation Geometry Neural Representation Geometry
Department of Computer Science University of Nebraska Omaha, Omaha, NE, USA

Sep 30, 2026J/K move · Enter open · S save

Group-Invariant Statistics Determine Embedding Geometry: Harmonic Analysis of Representations from Bach to the Night Sky

Liam Storan, Andreas Tolias, Nina Miolane

Stanford University · University of California, Santa Barbara

Word Embeddings Representation Geometry in Language Models Latent Space Geometry

Open paper AskPDF arXiv

The representations that language models learn for concepts such as months, weekdays, and places display consistent geometric structure: circles and saddle-shaped "Pringle" manifolds. Recent work traced these structures to

\textit{translation symmetry}

in word co-occurrence statistics, deriving the observed Fourier geometry when co-occurrence depends only on distance on an abelian lattice of concepts. We demonstrate that more general notions of symmetry lead to equally structured predictions. Considering symmetries defined by arbitrary finite groups, compact groups, and homogeneous spaces, we prove that whenever the co-occurrence statistics of a word family are invariant under a group

G

, the learned word embeddings consist of matrix elements of the irreducible representations (irreps) of

G

. Circles and Pringles arise when

G

is cyclic, in which case the irreps are Fourier modes. We verify the irrep structure in three experimental settings. (i) The cyclic group

\mathbb{Z}_{12}

: for the months of the year we recover the known circular geometry. (ii) A dihedral group acting on the major and minor triads: we unify two classical observations -- that transposition and chord inversion form a group (

T/I

) acting on chords (music theory), which

\textit{implies}

that the well-known "circle of fifths" emerges in learned chord embeddings (machine learning). (iii) We explain and reproduce a recently discovered spherical representation of celestial objects in large language models (LLMs) as a spherical-harmonic embedding derived from our theory. Our results demonstrate that the geometry of learned representations is often a consequence of the statistical symmetry of underlying data.

Abstract

Geometric Representation Learning

Explore similar work

Group-Invariant Statistics Determine Embedding Geometry: Harmonic Analysis of Representations from Bach to the Night Sky

Symmetry in language statistics shapes the geometry of model representations

Fisher-IRG: Fisher-Induced Local Invariant Representation Geometry across Language and Vision Models