Evaluating the Tradeoff Between Predictive Accuracy and Racial Fairness in Machine Learning-Based Recidivism Models

Abstract

With the increasing usage of predictive models for criminal risk assessment and predicting recidivism, concerns have grown due to the "black box" nature of these models and their implications with fairness and racial bias. Although these classifiers have shown the potential for high accuracy in terms of predicting criminal risk of recidivism, they may use sensitive factors such as race that can lead to racial unfairness in predictive outcomes. This research explores the impact of race being included or excluded as a feature for machine learning classifiers in predicting recidivism, with a goal of addressing the difficult task of minimizing bias while maximizing predictive accuracy.

Two binary Random Forest classifiers were trained on the COMPAS dataset: one including race as a feature and one excluding it. Both models used demographic and criminal history features and were evaluated using accuracy, precision, recall, F1-score, and flip rate, a measure of counterfactual fairness measuring the percentage of predictions that change when race is altered, where a greater flip rate means larger sensitivity to race changes and lower fairness.

The classifier including race achieved an accuracy of 77% with both the precision and recall being balanced at 77%, but had a flip rate of 36.83%, suggesting high sensitivity to racial variation. Removing race in the second classifier decreased accuracy to 74%, with precision and recall dropping to 74% and 73% respectively. These findings demonstrate the complexity of fairness and race in recidivism prediction, as excluding race reduces direct racial influence but reduces overall predictive performance. Future work will seek to develop a judicial assistant that can provide Chain-of-Thought reasoning to how it arrived at each prediction, enhancing transparency, and Reinforcement Learning with Human Feedback (RLHF) will be used to improve the model’s alignment with ethical judicial decision-making.