“Is your AI model fair?” There are 10 ways to answer that.
Many people don’t know this, but there are at least 10 ways to answer the question:
“Is your AI model fair?”
To illustrate this quickly, I created a simple logistic regression model trained on the UCI Adult Income dataset, which predicts whether an individual’s income exceeds $50,000 based on features such as age, education, and hours per week, with gender as the sensitive feature.
I then used the `fairlearn` library in Python, which includes the `MetricFrame` class for calculating various metrics such as accuracy, selection rate, and true positive rate, as well as functions for measuring fairness like Demographic Parity Difference and Equalized Odds Difference. (If you’d like the code, comment below.)
As you can see, the results from the fairness metrics indicate considerable disparities across different groups.
The Demographic Parity Difference and True Positive Rate Difference both show a 19% discrepancy between males and females, suggesting a moderate bias in both selection rates and correct identification of high earners.
The Equalized Odds Difference and False Positive Rate Difference highlight more substantial discrepancies, each showing a 40% difference, indicating that the model performs quite differently for the two groups.
The Disparate Impact Ratio is particularly low at 0.09 (the smaller, the more biased), indicating a severe bias where one group is selected at only 9% the rate of the other group.
These metrics illustrate how different fairness measures can highlight various aspects of model bias, with some metrics indicating severe bias and others showing moderate bias. This underscores the importance of considering multiple fairness metrics when evaluating model fairness.
The main takeaway, though, is to emphasize the importance of AI and data literacy for decision-makers as AI models become more and more ubiquitous.
When discussing an important topic like fairness, it is crucial to understand how multi-faceted it can be. By being aware of the various angles from which fairness can be assessed, we can better ensure that our AI models are equitable and just.
