Institut für Data Science Datenbanken und Informationssysteme News
Submission accepted for ACM International Conference on Fairness, Accountability, and Transparency (FAccT) 2022

Submission accepted for ACM International Conference on Fairness, Accountability, and Transparency (FAccT) 2022

Our paper in collaboration with Amazon colleagues was accepted for the ACM International Conference on Fairness, Accountability, and Transparency.

Our paper in collaboration with Amazon colleagues was accepted for the ACM International Conference on Fairness, Accountability, and Transparency.

 

Title: Measuring Fairness of Rankings under Noisy Sensitive Information

Authors: Azin Ghazimatin, Matthäus Kleindessner, Chris Russell, Ziawasch Abedjan, Jacek Golebiowski

Abstract: Metrics commonly used to assess group fairness in ranking require the knowledge of group membership labels (e.g., whether a job applicant is male or female). Obtaining accurate group membership labels, however, may be costly, operationally difficult, or even infeasible. Where it is not possible to obtain these labels, one common solution is to use proxy labels in their place, which are typically predicted by machine learning models. However, proxy labels are susceptible to systematic biases, and using them for fairness estimation can thus lead to unreliable assessments. We investigate the problem of measuring group fairness in ranking for a suite of divergence-based metrics in the presence of proxy labels. We show that under certain assumptions, fairness of a ranking can reliably be measured from the proxy labels. We formalize two assumptions and provide a theoretical analysis for either of them that shows how the true metric values can be derived from the estimates based on proxy labels. We prove that without such assumptions fairness assessment based on proxy labels is impossible. Through extensive experiments on both synthetic and real datasets, we demonstrate the effectiveness of our proposed methods for recovering reliable fairness assessments.

Verfasst von Ziawasch Abedjan