cs.LGOct 30, 2026

A Framework for Task-Agnostic Counterfactual Fairness in Transformers through Activation Steering

Authors: Joshua Wolff Anderson, Yanshan Wang, Shyam Visweswaran

Organizations: University of Pittsburgh, USA

Abstract

Bias in transformer-based models is a growing concern as these architectures become more prevalent in automated decision-making systems, such as prediction models for hiring, lending, and clinical treatment recommendation. Mechanistic interpretability (MI) is a growing body of work that measures meaningful causal effects of activations on model output by identifying human-interpretable features encoded in model activations. When such features correspond to sensitive attributes, they provide a direct way to quantify the bias those attributes introduce and to causally intervene to mitigate it. However, counterfactual fairness (CF) methods have not been systematically adapted to this setting of intervening on interpretable mechanisms in transformers. We propose a framework, Mechanistic Interpretability for Counterfactual Fairness (MICF), that utilizes MI tools to reformulate CF constraints in terms of transformer activations, unifying former approaches into a single pipeline for fairness analysis in transformers. MICF does not require a prior causal model or specific transformer architecture. We evaluate MICF on real-world tabular data and show that CF can be improved across different prediction tasks.

Figures & tables

Explore similar work

CardsList
  1. Efficient and Interpretable Transformer for Counterfactual Fairness

    Apr 29, 2026Panyi Dong, Zhiyu QuanTransformer InterpretabilityTransformer

  2. CAFP: A Post-Processing Framework for Group Fairness via Counterfactual Model Averaging

    Apr 8, 2026Irina Arévalo, Marcos OlivaAlgorithmic FairnessGroup Fairness

  3. Do Fair Models Reason Fairly? Counterfactual Explanation Consistency for Procedural Fairness in Credit Decisions

    May 12, 2026Gideon Popoola, John SheppardFeature AttributionCounterfactual Explanations