IsoAct: Geometry-Aware Post-hoc Debiasing via Isometric Actions

Sumin Park; Taero Kim; Minhyeong Cho; Subeen Park; Jong Hwan Lee; Kyungwoo Song

cs.LGOct 26, 2026

IsoAct: Geometry-Aware Post-hoc Debiasing via Isometric Actions

Authors: Sumin Park, Taero Kim, Minhyeong Cho, Subeen Park, Jong Hwan Lee, Kyungwoo Song

Organizations: University of Wisconsin–Madison · Yonsei University · SK Shieldus · Arist Inc.

Abstract

Learning representations that are robust to nuisance factors is essential for reliable deployment, but remains challenging when the representations lie on non-Euclidean manifolds. Existing debiasing methods typically assume Euclidean representations or apply linear interventions, which can distort the geometry of structured latent spaces. We propose IsoAct, a post-hoc framework that accounts for geometry in both nuisance identification and representation editing. It identifies nuisance-predictive directions using Fisher discriminant analysis (FDA) in tangent coordinates and applies corrections through manifold-native actions that preserve manifold membership. Controlled and real-world experiments on hyperspherical, hyperbolic, and $\mathrm{SE}(3)$ representations evaluate nuisance reduction, utility preservation, and manifold validity jointly. IsoAct reduces nuisance recoverability while retaining competitive utility. Its clearest joint result is on KLIFS, combining reduced kinase-group recoverability, strong structural prediction, and valid rigid poses.

Figures & tables

Manifold	Representation	Constraints
Hypersphere	$x\in\mathbb{S}^{d-1}$	1) $\\|x\\|_{2}=1$
Lorentz hyperboloid	$x\in\mathbb{H}^{d}_{c}$	1) $\langle x,x\rangle_{L}=-1/c$	2) $x_{0}>0$
$\mathrm{SE}(3)$	$x=(R,t)\in\mathrm{SE}(3)$	1) $R^{\top}R=I_{3}$	2) $\det(R)=1$

Table 1: Target manifolds and the membership constraints that admissible edits must preserve. The Lorentz model has curvature

-c<0

, with

\langle x,x\rangle_{L}=-x_{0}^{2}+\sum_{j=1}^{d}x_{j}^{2}

and time coordinate

x_{0}

Figure 1: Overview of IsoAct on the hypersphere. FDA identifies nuisance directions in tangent coordinates around a coordinate reference $r$ constructed from training data. A neutral target $n$ from low-confidence training examples supplies the correction reference. The right panel shows one update along $u_{k}$ while preserving unit norm. Lorentz and $\mathrm{SE}(3)$ instances use the same pipeline with geometry-specific actions.

Figure 2: Controlled debiasing on three manifolds. IsoAct combines Oracle-matched nuisance-probe scores with preservation of the designated utility proxy and near-zero constraint residuals. All axes improve outward; constraint scales are panel-specific. $\dagger$ marks the learning-based CMF adaptation.

Figure 3: $\mathrm{SE}(3)$ mechanism check. Colors denote nuisance groups; the dashed line marks the neutral angle. The light-green band is the shared translation-magnitude range used as a utility proxy. IsoAct removes angular group separation while keeping all samples in this band, matching the Oracle pattern. In contrast, CMF and SFID move many samples outside the band. Rotation residuals and detailed results are given in Figure 2 and Appendix C.1 .

		Debiasing Capability		Utility Preservation				Manifold Constraint Violation
	Method	Kinase Group		$\alpha$ C-helix State		Pocket Occupancy		Constraint 1	Constraint 2
		Acc $\downarrow$	F1 $\downarrow$	Recall $\uparrow$	AUROC $\uparrow$	Recall $\uparrow$	AUROC $\uparrow$	Mean $\downarrow$	Mean $\downarrow$
KLIFS	Original	0.1398	0.1409	0.3362	0.7163	0.7357	0.9708	6.5065e-08	2.2077e-08
	Null	0.0827	0.0679	—	—	—	—	—	—
	CMF	0.1988	0.1858	0.3046	0.7087	0.8173	0.9732	3.6900e-01	1.1368e-01
	SFID	0.1398	0.1409	0.3247	0.7176	0.7388	0.9712	6.5065e-08	2.2077e-08

Table 2: KLIFS results. IsoAct combines reduced kinase-group predictability with strong structural utility and numerical-precision rotation residuals. Constraints 1 and 2 report mean rotation orthogonality and determinant errors, respectively. Bold/underline mark best/second-best values among debiasing methods; Original and Null are reference rows excluded from ranking.

Figure 4: KLIFS debiasing-utility trade-offs. Each point is one configuration’s test result; higher and farther right is better. Debiasing gain $D$ measures progress from Original ( $0$ ) toward the shuffled-label Null ( $1$ ). Mean utility retention quantifies utility preservation by averaging Recall and AUROC ratios to Original across both structural tasks; $1$ denotes unchanged aggregate utility. Outlined diamonds mark the IsoAct Pareto frontier. Its high-utility point combines $D\approx 0.51$ with mean retention of $1.07$ , while other frontier points achieve stronger debiasing at lower retention. Definitions and configuration pools are detailed in Appendix C.3 .

		Age			Gender			Race
Model	Method	Debias $\uparrow$	Utility $\uparrow$	Constraint $\downarrow$	Debias $\uparrow$	Utility $\uparrow$	Constraint $\downarrow$	Debias $\uparrow$	Utility $\uparrow$	Constraint $\downarrow$
CLIP	Original	0.0000	1.0000	2.2612e-08	0.0000	1.0000	2.2935e-08	0.0000	1.0000	2.2181e-08
	Null	1.0000	—	—	1.0000	—	—	1.0000	—	—
	CMF	0.6339	0.0096	8.8207e-02	0.4481	0.0130	7.0113e-02	0.6890	0.0115	1.1123e-01
	SFID	0.0117	1.0005	3.8070e-03	0.0389	0.9788	8.1287e-03	-0.0085	0.9975	8.3426e-03
	SPD	0.7536	0.9867	1.8327e-02	0.8243	0.9938	1.0768e-02	0.9656	0.9891	5.3183e-02

Table 3: CLIP and MERU results across protected attributes. IsoAct combines high retrieval retention with the strongest MERU debiasing and attribute-dependent CLIP gains. Debias averages normalized Accuracy/macro-F1 proximity to Null; Utility averages Original-normalized R@1/5/10. Constraint reports CLIP unit-norm or aggregated MERU Lorentz residuals. Bold/underline mark best/second-best method values, excluding reference rows and numerical-precision MERU residuals.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Acc	F1	$D_{\mathrm{toy}}$	$U$	Mean $C_{1}$	Mean $C_{2}$
Hypersphere
CMF	0.50000000	0.49944383	0.98280340	0.10500000	0.02109016	—
SFID	0.50000000	0.49776786	0.98110009	0.05250000	0.08129425	—
SPD	0.50000000	0.49986107	0.98322745	0.30500000	0.02415661	—
IsoAct	0.50833333	0.50802585	1.00000000	1.00000000	2.33146835e-17	—
Lorentz

Appendix

Table 4: Detailed results underlying the toy starplots. Acc and macro-F1 are ranked by proximity to Oracle; higher

D_{\mathrm{toy}}

and

U

and lower residuals are better. Bold and underline mark the best and second-best edited methods within each geometry; ties share a rank.

	Nuisance metric / Oracle discrepancy				Task		Residual mean
Method	Acc	Recall	F1	AUROC	MAE	$U$	$C_{1}$	$C_{2}$
Hypersphere
Original	1.0000	1.0000	1.0000	1.0000	0.0000	1.0000	0.0000	—
Oracle	0.5083	0.5083	0.5080	0.4956	0.0000	1.0000	0.0000	—
CMF	0.0083	0.0083	0.0086	0.0023	3.2653	0.1050	0.0211	—
SFID	0.0083	0.0083	0.0103	0.0000	13.1276	0.0525	0.0813	—

Appendix

Table 5: Additional controlled results. Nuisance columns show raw scores for Original/Oracle and absolute Oracle discrepancies for edited methods. Lower discrepancies, MAE, and residuals and higher

U

are better. Bold/underline mark best/second-best edited methods within each geometry; ties share a rank at displayed precision. Table 4 gives higher-precision residuals.

Figure 5: Hyperspherical mechanism check. The horizontal angle carries nuisance and the polar angle is the shared utility proxy. IsoAct collapses group separation while preserving the proxy. Shading is a visual guide; utility uses each sample’s original-angle $2^{\circ}$ tolerance.

Figure 6: Lorentz mechanism check. IsoAct removes angular group separation while retaining the shared radial utility range. The dashed line marks the neutral angle and shading marks the shared range; the accompanying residuals assess manifold validity.

Model	Attribute	$K$	$\rho$	Passes	Strengths	Within-axis order
CLIP	Age	7	0.1	1	$\alpha=1$	—
	Gender	1	0.01	1	$\alpha=1$	—
	Race	6	0.1	2	$\alpha=1$	—
MERU	Age	8	0.01	1	$\alpha_{\mathrm{ang}}=0.75$ , $\alpha_{\mathrm{tr}}=1$	Ang. $\to$ Trans.
	Gender	1	0.03	1	$\alpha_{\mathrm{ang}}=0.5$ , $\alpha_{\mathrm{tr}}=1$	Trans. $\to$ Ang.
	Race	6	0.1	1	$\alpha_{\mathrm{ang}}=0.5$ , $\alpha_{\mathrm{tr}}=1$	Trans. $\to$ Ang.

Appendix

Table 6: Selected IsoAct settings for CLIP and MERU.

K

is the retained rank and

\rho

the FDA ridge. Ang. and Trans. denote angular and translation actions composed within each direction; a dash indicates a single spherical action.

Method	Acc	F1	R@1	R@5	R@10
Age
Original	0.2987	0.2614	0.5876	0.8350	0.9004
Null	0.1477	0.1060	—	—	—
CMF	0.1920	0.1742	0.0012	0.0070	0.0166
SFID	0.2967	0.2599	0.5892	0.8354	0.8990
SPD	0.1860	0.1431	0.5764	0.8268	0.8906

Appendix

Table 7: CLIP raw test probe and retrieval metrics. F1 denotes macro-F1. Bold/underline mark best/second-best methods within each attribute: smaller absolute distance to Null for Acc/F1, higher values for retrieval. Original and Null are unranked; ties share a rank. Table 3 reports normalized summaries.

Method	Acc	F1	R@1	R@5	R@10
Age
Original	0.2747	0.2465	0.2096	0.4242	0.5216
Null	0.1439	0.1055	—	—	—
CMF	0.2573	0.2359	0.0372	0.1188	0.1778
SFID	0.2587	0.2424	0.2030	0.4194	0.5160
SPD	0.1373	0.1236	0.1986	0.4146	0.5094

Appendix

Table 8: MERU raw test probe and retrieval metrics. F1 denotes macro-F1. Bold/underline mark best/second-best methods within each attribute: smaller absolute distance to Null for Acc/F1, higher values for retrieval. Original and Null are unranked; ties share a rank. Table 3 reports normalized summaries.

Setting	Method	Mean $C_{1}$	Mean $C_{2}$
KLIFS	Original	6.5065e-08	2.2077e-08
	CMF	3.6900e-01	1.1368e-01
	SFID	6.5065e-08	2.2077e-08
	SPD	9.4948e-01	3.7450e-01
	IsoAct	6.9510e-16	3.5723e-16
CLIP age	Original	2.2612e-08	—

Appendix

Table 9: Mean manifold-constraint residuals on selected test representations. IsoAct maintains unit-norm and rotation constraints to numerical precision. Lower is better. Bold/underline mark best/second-best methods within each setting, excluding Original and all MERU residuals. MERU’s Lorentz quadratic and upper-sheet residuals are reported separately.

Geometric Representation Learning

Geometric representation learning encodes data to preserve its intrinsic shape, distances, symmetries, and relationships.

See all 259 papers

Fisher-IRG: Fisher-Induced Local Invariant Representation Geometry across Language and Vision Models
Sep 29, 2026Abdullah All Tanvir, Xin ZhongInformation Geometry Neural Representation Geometry
Department of Computer Science University of Nebraska Omaha, Omaha, NE, USA
Beyond Gaussian Worlds: Latent Geometry Matters for JEPAs
Sep 18, 2026Léo Nicollier, Enric Meinhardt-Llopis, Marc Pic +2Joint-Embedding Predictive Architecture Latent Space Geometry
CB, ATT · Université Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli, France · Advanced Track and Trace +5
Discovering Data Manifold Geometry through Geometric Properties
Feb 2, 2026David Vigouroux, Lucas Drumetz, Ronan Fablet +1Unsupervised Representation Learning Geometric Representation Learning
ANITI, IMT Atlantique - DSD, LaTIM · IRT Saint Exupery, ANITI, Toulouse, France · IMT Atlantique, LaTIM UMR 1101 INSERM, Brest, France +5

Sep 29, 2026J/K move · Enter open · S save

Fisher-IRG: Fisher-Induced Local Invariant Representation Geometry across Language and Vision Models

Abdullah All Tanvir, Xin Zhong

Department of Computer Science University of Nebraska Omaha, Omaha, NE, USA

Information Geometry Neural Representation Geometry Representation Geometry in VLMs Invariant Representation Learning Representation Geometry in Language Models

Open paper AskPDF arXiv

Semantic-preserving transformations can induce substantial motion in learned representations, while small changes may strongly affect model predictions, raising a basic question: what local metric best captures semantically consequential variation? We propose Fisher-induced invariant representation geometry (Fisher-IRG), which measures local representation directions through their predictive sensitivity. Around each representation, we construct semantic-preserving and semantic-changing neighborhoods, aggregate their local Fisher information, and recover invariant directions through a contrastive generalized eigenvalue problem. Controlled displacement analyses first show that comparable Euclidean motion can have substantially different predictive consequences, supporting the need for a predictive geometry. Across language and vision models, Fisher-IRG yields stronger semantic-versus-nuisance predictive selectivity and generally more reproducible subspaces than covariance-based geometry, while recovering systematically distinct local directions. Representation interventions further localize semantic effects to the Fisher-derived subspace, and held-out separation and retrieval show that the recovered geometry generalizes beyond the discovery neighborhoods. These results support Fisher-IRG as a principled framework for characterizing local invariant representation geometry.

Abstract

Geometric Representation Learning

Explore similar work

Fisher-IRG: Fisher-Induced Local Invariant Representation Geometry across Language and Vision Models

Beyond Gaussian Worlds: Latent Geometry Matters for JEPAs

Discovering Data Manifold Geometry through Geometric Properties