The most surprising thing about AI bias is that it’s not about the AI being inherently prejudiced; it’s about the AI learning our own societal biases from the data we feed it.
Let’s see this in action. Imagine we’re building a model to predict loan eligibility. We feed it historical loan data, which, due to past discriminatory lending practices, shows fewer approvals for certain demographic groups.
Here’s a simplified look at the data we might feed it:
[
{"applicant_id": "A001", "income": 75000, "credit_score": 720, "loan_approved": true, "demographic_group": "White"},
{"applicant_id": "A002", "income": 60000, "credit_score": 680, "loan_approved": true, "demographic_group": "White"},
{"applicant_id": "A003", "income": 50000, "credit_score": 650, "loan_approved": false, "demographic_group": "Black"},
{"applicant_id": "A004", "income": 80000, "credit_score": 730, "loan_approved": true, "demographic_group": "Asian"},
{"applicant_id": "A005", "income": 55000, "credit_score": 660, "loan_approved": false, "demographic_group": "Hispanic"},
{"applicant_id": "A006", "income": 70000, "credit_score": 700, "loan_approved": true, "demographic_group": "White"},
{"applicant_id": "A007", "income": 48000, "credit_score": 630, "loan_approved": false, "demographic_group": "Black"}
]
If we train a model on this data without any intervention, it might learn to associate "Black" or "Hispanic" demographic groups with a higher probability of loan denial, even if other factors like income and credit score are comparable to approved applicants from other groups. The model isn’t "thinking" in discriminatory terms; it’s simply identifying patterns in the data that reflect historical societal inequities.
The problem this solves is fairness. AI systems are increasingly used for critical decisions like hiring, lending, and even criminal justice. If these systems perpetuate or amplify existing societal biases, they can lead to unfair outcomes, further marginalizing already disadvantaged groups and eroding trust in AI.
Internally, a biased AI model has learned a spurious correlation. It has assigned undue importance to a sensitive attribute (like race or gender) because that attribute is correlated with the outcome in the training data, often due to historical or systemic factors unrelated to the individual’s actual qualification for the outcome. The model’s decision-making process, when analyzed, would reveal that the sensitive attribute has a significant weight in its predictions, even when other predictive features are controlled.
The levers you control are primarily in the data preparation and model training phases. You can:
-
Data Auditing and Cleaning: This involves thoroughly examining your training data for imbalances and proxies for sensitive attributes. Tools like IBM’s AI Fairness 360 or Google’s What-If Tool can help identify disparities. For instance, if you find that "zip code" is highly correlated with race and also with loan approval, you might consider removing "zip code" as a feature if it doesn’t add independent predictive value.
-
Bias Mitigation Techniques (Pre-processing): This involves altering the training data to remove or reduce bias before feeding it to the model. Techniques include:
- Reweighing: Assigning different weights to data points to counteract imbalances. For example, if Black applicants are underrepresented and historically denied more often, you might increase the weight of data points from Black applicants to give them more influence during training.
- Sampling: Oversampling underrepresented groups or undersampling overrepresented groups. If your dataset has 90% White applicants and 10% Black applicants, you might randomly duplicate Black applicant records (oversampling) or randomly remove some White applicant records (undersampling) to create a more balanced dataset.
-
Bias Mitigation Techniques (In-processing): This involves modifying the learning algorithm itself to be aware of and counteract bias during training. Algorithms can be designed to optimize for both accuracy and fairness metrics simultaneously, penalizing the model when it exhibits bias. For example, adversarial debiasing uses a "competitor" network that tries to predict the sensitive attribute from the main model’s output; the main model is trained to fool this competitor, thereby learning representations that are independent of the sensitive attribute.
-
Bias Mitigation Techniques (Post-processing): This involves adjusting the model’s predictions after training to achieve fairness. For example, you might set different decision thresholds for different groups to ensure equal opportunity or equalized odds. If a model predicts a 70% loan approval probability for two applicants with identical financial profiles, but one is from a historically disadvantaged group, you might adjust the threshold for that group to ensure they have a fair chance.
-
Fairness Metrics: You need to define and measure fairness. Common metrics include:
- Demographic Parity: The proportion of positive outcomes should be the same across all groups. (e.g., loan approval rate is 50% for men and 50% for women).
- Equalized Odds: The true positive rate and false positive rate should be the same across all groups. (e.g., among qualified applicants, the approval rate is the same for all races; among unqualified applicants, the denial rate is the same for all races).
- Predictive Parity: The precision (positive predictive value) should be the same across all groups. (e.g., for all applicants predicted to be approved, the actual approval rate is the same across all genders).
The most subtle form of bias often arises not from direct demographic labels, but from proxies embedded in seemingly neutral features. For instance, a feature like "time spent on website" might, in some contexts, inadvertently correlate with socioeconomic status or access to technology, leading to biased outcomes if not carefully scrutinized.
The next step after mitigating bias is often ensuring the model’s robustness against adversarial attacks, where malicious actors try to manipulate its predictions.