Automated Fairness Testing with Representative Sampling

The issue of fairness testing in machine learning models has become popular due to rising concerns about potential bias and discrimination, as these models continue to permeate end-user applications. However, achieving an accurate and reliable measurement of the fairness performance of machine learning models remains a substantial challenge. Representative sampling plays a pivotal role in ensuring accurate fairness assessments and providing insight into the underlying dynamics of data, unlike biased or random sampling approaches. In our study, we introduce our approach, namely RSFair, which adopts the representative sampling method to comprehensively evaluate the fairness performance of a trained machine learning model. Our research findings on two datasets indicate that RSFair yields more accurate and reliable results, thus improving the efficiency of subsequent search steps, and ultimately the fairness performance of the model. With the usage of Orthogonal Matching Pursuit (OMP) and K-Singular Value Decomposition (K-SVD) algorithms for representative sampling, RSFair significantly improves the detection of discriminatory inputs by 76% and the fairness performance by 53% compared to other search-based approaches in the literature.