Mitigating Negative Impacts from Socio-Technical AI

Abstract

Over the past decade, data-driven Machine Learning (ML) models have become more popular because of the advances in their capabilities driven by increases in compute power and available data. ML models allow for speedier analysis of and learning from data than human analysts typically could provide. Further, the ML models can spot patterns in historical data that human analysts might not see which makes them useful in decision-making contexts (e.g., in healthcare, spotting cancerous cells). It is well known that human decision-makers can make mistakes, especially when dealing with an overwhelming amount of data; ML models can be more accurate and consistent than human decision-makers. Despite these benefits of ML models, there are downsides to their use in high stakes decision-making contexts. For instance, since these ML models learn from historical data, they learn patterns of historical bias. As such, the ML models have been shown to replicate and exacerbate bias from our society through discriminatory model predictions. In this thesis, I evaluate the consequences experienced by affected groups and the legal implications of ML models with bias mitigation methods applied.

ML models have received much attention, especially when predictions have gone awry within high stakes domains like policing, finance, and healthcare. In these domains, taking an example, recidivism prediction from policing, there are often higher False Positive rates for historically disadvantaged groups. This rate reflects the incorrect assumption that individuals from that group will reoffend. These significant predictions the ML models make can result in serious consequences for people’s lives. Since the harms of these systems have become starkly clear, computer scientists have rapidly presented solutions to detect and alleviate bias so that discriminatory predictions can be minimized. To detect bias, fairness metrics have been used. These metrics usually measure discrepancies between ML model performance rates (e.g., selection rates) across different demographic groups. Fairness metrics can be used to constrain ML models to mitigate bias. When fairness metrics are used for this, they are called fairness constraints. An example of a bias mitigation method is the application of a fairness constraint to an ML model and then the model optimizes for both accuracy and the constraint. I refer to ML models that have had fairness constraints applied to them as fairness constrained models.

The algorithmic fairness literature typically assumes that optimizing for a fairness constraint will improve the outcome of the disadvantaged group. Throughout this thesis, I show that this is not necessarily the case. Conversely, I demonstrate that, in the case where the assumptions that underpin the constraints (e.g., positive model outcomes are better than negative model outcomes) do not hold, optimizing for these constraints leads to worsened impacts on the disadvantaged group. To show these limitations of fairness constraints, I run experiments with financial datasets where impact is tied to financial well-being (e.g., a credit score increase or monetary gains from a loan). The financial use cases reflect common loan repayment scenarios where a bank predicts if an applicant would repay them if given a loan and if a person is a high or low credit risk. For my first thesis contribution, I show the limitations of fairness constrained models when their assumptions about model outcomes do not hold for the actual impacts of those outcomes; in addition, disadvantaged groups are usually worse off due to fairness constrained models when those assumptions are not true. These results demonstrate the risk of blindly optimizing for fairness constraints.

For an alternative method to applying fairness constraints to ML models, I investigate the suitability of cost-sensitive learning for mitigating negative impacts and uplifting positive ones for the affected groups. Cost-sensitive learning comes from the machine learning domain. I argue that it can be considered a bias mitigation method. In high-stakes decision making settings, the different model errors usually have different levels of negative impacts on people. For instance, a False Positive outcome could be twice as bad as a False Negative outcome. Cost-sensitive learning allows for AI practitioners to specify those different error costs so the ML models account for them when training. For my second thesis contribution, I illustrate that cost-sensitive learning applied to ML models outperforms fairness constrained models for the disadvantaged groups’ impact results.

Since these ML models and bias mitigation methods exist within society, I consider a human aspect of these socio-technical AI, their legality. I argue that AI practitioners need more guidance on what discrimination implications there are when bias mitigation methods are applied to ML models. In this thesis, I study how ML models with bias mitigation methods applied to them abide by the United Kingdom’s non-discrimination law. For my third thesis contribution, I provide a categorization of bias mitigation methods and how they make ML models more or less reliant on protected characteristics; then, I highlight how different bias mitigation methods fit within this categorization and discuss how their application to ML models could violate the United Kingdom’s non-discrimination law through an immigration case study. Given that protected characteristics are at the heart of the United Kingdom’s non-discrimination law, if ML models are overly reliant on them, then there could be direct discrimination implications. My research into the impacts of high stakes decision-making ML models on affected people and the legal implications of them provides the groundwork for future work on human AI oversight (inspired by the EU AI Act’s Article 14) and what is unfair discrimination from a model.

Date of Award	1 Jul 2025
Original language	English
Awarding Institution	King's College London
Supervisor	Elizabeth Black (Supervisor), Jose Such (Supervisor) & Natalia Criado Pacheco (Supervisor)