Predictive modeling is supposed to be neutral, a way to help remove personal prejudices from decision-making. But the algorithms are packed with the same biases that are built into the real-world data used to create them. Wharton statistics professor James Johndrow has developed a method to remove those biases. His latest research, “An Algorithm for Removing Sensitive Information: Application to Race-independent Recidivism Prediction,” focuses on removing information on race in data that predicts recidivism, but the method can be applied beyond the criminal justice system. He spoke to Knowledge at Wharton about his paper, which is co-authored with his wife, Kristian Lum, lead statistician with the Human Rights Data Analysis Group. (Listen to the podcast at the top of this page.)
An edited transcript of the conversation follows.
Knowledge at Wharton: Predictive modeling is becoming an increasingly popular way to assist human decision-makers, but it’s not perfect. What are some of the drawbacks?
James Johndrow: There has been a lot more attention lately about it, partly because things are being automated so much. There’s just more and more interest in having automatic scoring, automatic decision-making, or at least partly automatic decision-making. The area that I have been especially interested in — and this is a lot of work that I do with my wife — is criminal justice.
In criminal justice, there is a lot of use of algorithms for things like who will need to post bail to get out of jail pre-trial versus who will just be let out on their own recognizance, for example. At the heart of this is this idea of risk assessment and trying to see who is most likely, for example, to show up to their court dates. The potential problems with this are just that these algorithms are trained on data that is found in the real world. That data can be the by-product of this person’s history of interaction with the court systems or history of interaction with police. The algorithms and their predictions can bake in all of this human stuff that is going on, so there has been a lot more attention lately to making sure that certain groups aren’t discriminated against by these algorithms. For example, is the algorithm less accurate for certain groups, or is the algorithm recommending that minorities are released less often than white people? Those are the kinds of things that people pay a lot of attention to, and there was a particular literature on this where we were operating when we thought of writing his paper.
Knowledge at Wharton: Is it the idea that algorithms are created by humans, so they contain human biases?
Johndrow: That is part of it, but the real thing isn’t even so much the algorithm itself. It’s that algorithms, especially in machine learning and statistics, are things that allow us to take inputs and produce outputs. They allow us to take data and then transform that data into some kind of prediction.
What happens here is you take information on a bunch of people and what you have observed them doing. For example, you would take information on people and whether they showed up to their court dates. Then you have some kind of algorithm and use all of that historical data on people to train the algorithm, basically learn its parameters. But the problem is that data is not the result of some laboratory experiment; that data is the result of humans interacting with one another.
“We’re all imperfect, and those things show up in the data.”
We’re all imperfect, and those things show up in the data. Any biases that exist in the data, those things will eventually show up in the predictions. It is impossible to avoid that kind of stuff because the data that we care about in these areas is data that we find in the real world.
Knowledge at Wharton: You focus on recidivism, which is the likelihood of committing another crime after being released from prison. What did you set out to do in this paper?
Johndrow: In this field, what we have been trying to do is to create a fair algorithm. Nobody agrees on what that is. There are lots of different notions of a fair algorithm. I almost think of that as a philosophical question. I think of my role as a statistician or a machine learning person as, “Well, there are these different notions out there. Can I produce a method that people could use if they wanted to go with this notion?” There is this notion called statistical parity or demographic parity, and that is the idea that you would have certain protected groups. Those protected variables might be things like sex or race. What we want, what our notion of fairness is, is that the predictions of the algorithm don’t differ by race or sex. What that means statistically is that when you make the predictions, there is no information on race left in the prediction.
What our paper is really doing is a method for taking some data and removing all of the information about race or about sex or about whatever your protected variables are, and then handing that data off to anybody who wants to train an algorithm. We’re not the first ones to have done something like this. We are just following on this stream of research over the last few years.
The word that people use is that you have “repaired” the data. We have, I think, a more flexible or general way of repairing data. Anything that you produce with it later, it’s not going to have any information about those protected variables anymore.
Knowledge at Wharton: Can you talk more about how this method works and how it is different from what has been proposed before?
Johndrow: What had been proposed before, a lot of what people were able to do, is if you had what we call categorical variables, then everything would work out fine. We want something that works no matter what it is.
One of the things that people would have trouble with is if you had a numeric variable like your age. If you wanted to repair the data so that it removed information on lots and lots of variables, that was harder. Our method is designed to make that kind of thing possible. The way it works is just that you go through one variable at a time. You look at all of the other variables and strip out the information on the protected variables. Then you go to the next variable and do that.
It is similar to things that people have used in areas like privacy and so forth, like if you wanted to release a dataset and not be able to identify certain characteristics of people in the dataset. It has relationships to that kind of literature.
Knowledge at Wharton: Once you were able to do that, what was the result?
Johndrow: Again, we are not exactly trying to take a stand on what fairness is. We’ve picked this one notion and wanted to do an algorithm for it. I think the result was that the algorithm works pretty well. At least in the example that we have and the dataset that we are working with, you can see that after you use our method, there really is not any information about the protected variables left.
We have statistical tests, for example, to see whether there is information about a particular variable sitting in a dataset, and it passes all of those statistical tests. That is not a certainty, that we have definitely removed all information, but empirically it seems to work pretty well. And we have some theory about how it works, also.
“In this field, what we have been trying to do is to create a fair algorithm. Nobody agrees on what that is.”
Knowledge at Wharton: You did look at this in terms of recidivism rates and predicting someone’s likelihood to reoffend?
Johndrow: That’s right. I want to be really clear here: It’s not like anybody is putting a race variable in the model. It’s just that all of the data that you do use is correlated with that, so our procedure is trying to wipe out those correlations while leaving all of the remaining information behind.
What you can see is if you just naively take the data and try to predict recidivism, you are going to find really big differences by race and certainly by other variables as well. You are going to find really big differences group-wise in the distributions of the predictions, like how many people are flagged as high risk. Running our method on the data, the distributions of the predictions across all of the people in the dataset look very, very similar by race. Not only that, but the overall accuracy of the things you train on this data is basically the same. We lost almost no accuracy, but we were able to pretty much equalize things by race. We thought a pretty good selling point of this is that overall the predictions are still quite good. Well, they are as good as they can be on the data. But we are able to accomplish this demographic parity.
Knowledge at Wharton: A lot of industries are trying to use predictive modeling without bias. How can this research be applied beyond the criminal justice space?
Johndrow: One of the things that we were aiming for is that this method is totally general. You take these features or predictors, and you do this to them, and what you get out the other side is something that will have removed the information about the variables you wanted to get rid of. You could use this in lots of areas. You could use it in credit scoring if you want.
Anyone who wants to adopt or achieve this demographic parity notion can do that with our procedure. We think it is broadly useful. I think the question we have now is scaling it up to lots of predictors. We can do it with a dozen or something, but sometimes people will have hundreds or thousands. So, that is a future work type of thing.
Knowledge at Wharton: Are there other future lines for the research?
Johndrow: I think there are a lot of interesting things going on in this area now, and I tend to be driven more by the application. There are lots of other notions of fairness. What is really interesting is that, through a small grant that my wife and I have with my former Ph.D. adviser, we now have access to a much bigger criminal justice dataset. We are finding lots of interesting things with this data. The data are a mess. We are trying to understand what the different variables mean. There are all kinds of things going on about how people record the data and how that has changed over time. By being driven by that application and trying to do good modeling with that data, I think it is going to open up a lot more interesting areas that will be relevant to real world stuff, which is what we are excited about.
