Auditing and Learning Fair Classifers Under Distribution Shift

Machine learning systems are widely used in daily lives, naturally making fairness an important concern when designing and deploying these systems. Most bias mitigation techniques often hinge on two critical assumptions: firstly, train and test data consist of i.i.d samples from the same distribution;and secondly, the labels are reliable indicators of the ground truth. This work scrutinizes two distinct aspects of distribution shift—label shift and covariate shift. Label shift constitutes the alteration in the distribution of labels, challenging the assumption that the observed labels faithfully represent the true ground truth. On the other hand, covariate shift addresses changes in the distribution of features over time, challenging the same distribution assumption. To tackle this problem, a framework is proposed to audit and learn fair classifiers by using a probabilistic model to infer the hidden fair labels and estimating the expected fairness criteria under this distribution. In particular, (i) a ``data clean-up'' method that replaces biased labels with fair ones---which can be used as pre-processing at train time or for better auditing at test time---and (ii) a fair classifier learning method that aims to directly enforce statistical fairness notions with respect to the inferred fair labels. By explicitly considering these components of distribution shift, this work provides an exploration of the challenges posed by evolving data distributions, offering insights into the robustness and adaptability of bias mitigation techniques in dynamic real-world scenarios