cs.LGDec 15, 2023

Integrating Fairness and Model Pruning Through Bi-level Optimization

Authors: Yucong Dai, Gen Li, Feng Luo, Xiaolong Ma, Yongkai Wu

Organizations: Clemson University

Abstract

Deep neural networks achieve high predictive performance but often require substantial computational and storage resources, which motivates model compression techniques such as neural network pruning. Conventional pruning methods, however, can introduce or amplify algorithmic bias and produce disparate outcomes across demographic groups. We study fair model pruning, which seeks sparse models that satisfy fairness criteria while maintaining predictive performance. We characterize fairness using both group accuracy disparity and differences in compression-induced accuracy degradation across demographic groups. We propose a framework that jointly optimizes pruning masks and model weights under fairness constraints, thereby integrating accuracy and fairness into the compression process. We formulate fair pruning as a constrained bi-level optimization problem and derive an efficient optimization procedure. Experiments across multiple datasets, predictive tasks, model architectures, and sparsity levels show that our framework improves the trade-off among fairness, accuracy, and training efficiency relative to the baseline pruning strategies. Compared with the strongest evaluated baselines, BiFP maintains competitive accuracy and fairness while requiring more than 94% fewer training iterations in the real-world experiments.

Figures & tables

Explore similar work

Sep 15, 2026

FairCompressAgent: An Agentic Framework for Fairness-Aware Model Compression for FPGA Deployment

Fairness-aware model compression requires selecting methods and configurations that balance accuracy, fairness, and deployment cost. These decisions become more difficult when compression methods are composed or the user's requirements change. In this paper, we propose FairCompressAgent (FCA), an agentic framework that integrates fairness-aware pruning, incremental quantization, and sparse low-rank factorization through a common operator interface. A language-model planner uses model profiles and measured outcomes to select compression configurations, while an execution layer performs compression, fine-tuning, evaluation, and constraint-based selection. FCA also supports requirement updates and reports the remaining violation when a request cannot be satisfied. Experiments on Fitzpatrick-17k with VGG-11 compare four search methods over 40 measured configurations. Under the accuracy-constrained request, FCA selects a compressed model with 59.54% less inference tensor storage, while validation average precision increases from 0.5141 to 0.5233 and equalized opportunity (EOpp) decreases from 0.2251 to 0.2168. It reaches the same final selection as one-shot planning with 7.33 versus 12 candidate evaluations on average, under their respective stopping policies. Repeated fine-tuning, held-out testing, and online requirement updates characterize the stability and interactive use of this compression workflow. The results demonstrate how measured feedback and explicit constraints support the selection and interactive refinement of fairness-aware compression configurations.
May 16, 2026

Differentiable Optimization Layers for Guaranteed Fairness in Deep Learning

Differentiable optimization layers are traditionally integrated in predict-then-optimize frameworks where a neural model estimates parameters that subsequently serve as fixed inputs to downstream decision-making optimization problems. In this work, we introduce the concept of a "fairness layer": a differentiable optimization layer appended to a model's output layer that guarantees a chosen notion of output parity is satisfied when integrated into a neural network. Additionally, we introduce an online primal-dual inference algorithm that provides provable aggregate fairness guarantees for streaming predictions with arbitrarily small batch sizes, where traditional per-batch constraints become overly restrictive. Numerical experiments demonstrate the effectiveness of the fairness layer and associated algorithm, and theoretical analysis characterizes the layer's differentiability and stability properties during model training and backpropagation. Our code for these experiments is publicly available on GitHub (https://github.com/dtroxell19/FairDL-ICML-2026.git) and our public Python package documentation can be found online: https://dtroxell19.github.io/fairness_training/.
Jul 30, 2026

Fairness Pruning: Locating Demographic Bias in GLU-MLP Layers via Differential Activations

This work presents Fairness Pruning, a lightweight structural intervention method designed for the management and future mitigation of demographic bias in large language models (LLMs). As a foundational empirical validation of this method, this work focuses on causal bias localization. Using minimally contrastive prompt pairs and inference-time activation capture, the method identifies neurons that react differentially when processing demographic attributes in GLU architectures, evaluating the signal at the down_proj input. Empirical evaluation was conducted on models of up to 3 billion parameters (Llama-3.2 family and Salamandra-2B), combining standardized benchmark evaluation with qualitative text generation experiments. Results demonstrate that zeroing the identified neurons alters how the model responds to associated demographic variables. However, rather than producing flat mitigation, the intervention causes bidirectional bias destabilization: because BiasScore is unsigned, candidate sets mix neurons that push toward and against the stereotype, and the net effect on aggregate bias depends on which sign dominates. The intervention is extremely surgical: zeroing at most 40 neurons in Llama-3.2-1B (less than 0.031% of total MLP width) achieves a mean retention of 99.49% in reasoning and general knowledge capabilities. These findings empirically confirm that demographic bias processing and model capabilities operate on dissociable circuits, establishing the methodological foundations for transitioning from blind zeroing toward directional behavior modulation.