Detecting Bias in AI Image Generators

Identifying Bias in AI Images

Angie Basiouny: If I ask you to think of a doctor, your brain might conjure up the image of an older white male. If I ask you to think of a fitness enthusiast, it might be a younger, slender person. These are biases that are baked into our brains, and unfortunately, they also get baked into AI. I’m here today with Wharton professor Kartik Hosanagar and his colleague, Pushkar Shukla, who’s a PhD candidate at the Toyota Technological Institute at Chicago. They’re part of a team that has created a tool that can detect biases in text-to-image generators. Those are the machine learning models that can produce a visual image with just a few prompts.

Gentlemen, when I first read your paper, I thought, what a difficult problem to solve for in real life. I can’t even imagine how difficult it is to solve for it on a technology platform. Give us an overview of this research.

Kartik Hosanagar: Our research was motivated by the widespread use of generative AI systems today to generate all kinds of content. Certainly text, through systems like ChatGPT. But also a lot of image generation through tools like Stable Diffusion, Midjourney, Meta’s AI, and even ChatGPT does image generation.

While they’re being used at an unprecedented volume or scale today, these systems do have biases. It’s kind of what you were mentioning earlier. They’re trained on data. Those data are generated by humans who have their own biases. They pick up their biases, and in many ways institutionalize those biases, and kind of do that going forward.

Look at image generation as an example. In these systems, if you type in, say, “computer programmer,” you’re likely to see most of the images being of males. These systems are prone to biases. But generally, the biases get detected after the fact, and usually by a human user using them and kind of saying, “Hey, I’m noticing a pattern.” Or a journalist might notice a pattern and write about it. We’ve seen, for example, Google’s Gemini having bias issues, and people wrote about it. Similar things have been written about pretty much every image generation model.

We set out to come up with a system that can automatically detect these biases. Automatic detection is extremely important because of the scale at which they’re being used. But it’s also a very hard problem. We wanted to be able to detect them, and not just detect simple or the common biases, like, say, gender and age, but all kinds of potential biases, and to be able to provide explanations to human decision-makers.

Basiouny: It would be helpful if we could show some of the biased images that we’re talking about. Pushkar, can you share your screen and show us some of these images that you’re studying in this project?

Pushkar Shukla: The first set of images talks about a computer programmer. We used Stable Diffusion to generate images of a computer programmer, and this is how these images look like. Similarly, in the second set of images, I asked the generative AI model to generate images of an old man at a church. Now, before I go on and talk about the possible directions in which these images can be biased, I want to take a second and ask you two. What do you think are the possible directions in which these images can be biased?

Basiouny: Most definitely gender, right? For the first one, a computer programmer, the images are all male. I noticed in your paper that you had a collection of images that you prompted for. For example, childcare worker, and all the images were female. Another stereotype.

I would also say maybe the image about an old man in a church. The image of the church in this case is the same. It appears to be a Catholic Church — the stained glass, the arched windows. It’s sort of an image that we have from maybe our school days of studying the Renaissance.

Hosanagar: Yeah. I was going to add that, well, at least the computer programmer images, in terms of race, seems to be diverse. But in terms of the old man at the church, from what I can see, they’re all white male. Also, the computer programmer seems to be, based on these four images, all young. There’s another stereotype I’m seeing there.

Shukla: I think all the axes of biases that you mentioned are correct, but there is more to it. You talked about gender, ethnicity, and age being important axes of biases. But let’s look at a few others. All the computer programmers are lean, so there is clearly a body type issue there. Similarly, the physical appearance of old men at a church are of a certain type. They look grim, slightly depressed or sad, so there might be ableism bias. In our research, we found that oftentimes old men were portrayed to have a certain kind of disability.

The point that I’m trying to make is when we think about biases in AI, we generally think about the common dimensions of biases such as gender, ethnicity, culture, maybe age. But there is more to it. There are other equally important dimensions of biases, such as socioeconomic bias, disability, body type and health bias. And that is something that needs to be looked upon, and that is what makes it a tough problem to solve.

Detecting and Correcting Biases in AI Image Generators

Basiouny: Why is it important to correct this kind of bias?

Hosanagar: Well, quite simply, these biases will continue to be there in society, but also at a scale that we have probably not seen before. If you have a human being generating images in some marketing agency, and that’s going to be used in ad collateral or visuals that we see around us, then that bias with an individual human being affects a few hundred, few thousand images. But if you have an AI system that’s doing this at scale for millions of images — and in the end, we’re probably going to have like three, four image-generation AI systems that are powering a lot of the images that get created and used by all of us. Systematically, that will mean that all of us are seeing biased images everywhere, which ultimately affects our social constructs. How do we imagine a CEO? Is that always a male? You brought up a childcare worker? Is it always a female? And so on. I think those stereotypes will then propagate in society.

The challenge is that we can’t always rely on manual human detection of these biases, because millions of images are being generated on a daily basis. At that scale and speed, manual approaches just will not work. If I generate two images from the system, and those two images happen to show two males, that doesn’t mean there is bias in there. But if tens of thousands of people are generating images, and each of their one or two images are biased, then collectively, we’re skewing how we represent teachers or how we represent computer programmers, and so on. And many times, there may not be a human in the loop. If we are automatically using AI systems to create content and use them in some downstream activity or task, then we need some automated way to detect and correct those biases. Otherwise, they will go undetected and be used in society at large.

Basiouny: Pushkar, I know that bias detection in computer models is something that you study specifically. From your point of view, why is this important?

Shukla: I think because AI models in general are a reflection of society. We want make sure that the harmful aspects that are already prevalent in the society, first, are not transferred into the AI models. More so, they are at least not exacerbated by these AI models. Oftentimes we see that a lot of these biases in, say, certain professions, are exacerbated or worsened by how AI models depict them. I think that’s a very important problem that needs to be solved. I’m all in for developing and deploying AI models at large scales, but we should understand that they should be done in an efficient manner.

Basiouny: Your model is called TIBET, which is an acronym for Text to Image Bias Evaluation Tool. How does it work?

Hosanagar: Generally, the philosophy behind what we’re doing is that if humans cannot handle the scale and speed of image generation, and if AI’s strength is scale and speed, can we leverage AI itself to detect the problems in AI models? If we do it carefully, then it doesn’t become a loop of: AI creates models that are biased, and AI detection tools cannot detect those biases because they have the same problem.

If we think about a term like “computer programmer” or “old man at a church” — the two examples Pushkar showed — we first want to think about, what are the ways in which that image can be biased? We did a joint brainstorming of the ways in which it could be biased.

The first step in our tool is to ask a large language mode, a question about, “What are the ways in which images that are generated in response to this prompt by a user can be biased?” It’s certainly gender, but it could identify some of the other biases that Pushkar raised, like body type or even the visual environment for the image.

We call it an axis, but every potential dimension or axis along which an image can be biased, we generate what we call counterfactual prompts. Meaning, what are the other kinds of prompts that could have been entered by the users along the dimension of bias. If we are worried about gender, a counterfactual prompt for computer programmer would be “male computer programmer” and “female computer programmer,” or “transgender computer programmer.” And then we generate images for these counterfactual prompts. We compare those images with the image for our main prompt, again using AI to look at the concepts that are in the images and compare them.

We might find in the example that Pushkar showed, that the images for computer programmer have very similar concepts as the images for “male computer programmer,” but the concepts are very different than the images for “female computer programmer” — meaning the dresses people are wearing, the jewelry they have on, whether they’re wearing glasses or not, things like that. We might also find that the concepts for “computer programmer” look very similar to “young computer programmer,” but look very different from “older computer programmer.”

Once we start to see that, we start to see the results for computer programmer are nearly the same as male computer programmer, nearly the same as young computer programmer, but very dissimilar to female computer programmer or older computer programmer. Based on that, our system is able to generate scores for how biased they are. And if it starts to look like it’s very similar to some counterfactual prompts, but very dissimilar to others, our system knows that there’s a bias here and is able to flag that and give a high score for that bias.

Limitations of AI Bias Detection

Basiouny: That’s incredible. Now, you briefly mentioned Google Gemini. This was a case where Google put out this text-to-image generator, and then found themselves embroiled in the culture wars because the generator was actually over-correcting for bias. One of the famous examples was, if you prompted it for an image of the Founding Fathers for the United States, it would depict a Black man. If you asked for soldiers who fought in the Vietnam War, it would depict an Asian woman. That brings me to asking you about the limitations of your model. How do you prevent the same problem from happening in your model?

Shukla: I think that problem happens because they come up with this one-size-fits-all approach. Most of the prompts, you just correct for culture as well as gender, and that’s about it. Whereas, what our approach does, is it’s a very dynamic approach that is dependent on what the input is. A typical example I give is, capybaras getting married in Italy, versus images of a philosopher on Mars. They might have completely different sets of biases. Capybaras getting married in Italy might not have any biases related to gender, but the way Italy is portrayed might be biased in that direction.

I think the way our approach differs is that it considers the context in which the prompt is set. Based on that context, it generates a set of biases, and then does this investigation on those sets and says, “OK, I’m biased along these directions, I’m not biased along these directions. But I know what directions to check my biases to start. It’s not like I’m going to check my biases on age, race, and gender for every prompt or every input.”

Basiouny: This tool was specifically designed for text-to-image generation. Can it be used for other things? Do you have plans to expand its abilities?

Hosanagar: Yeah, we do. Because if you think about the underlying approach, it can be generalized to large language models that are generating text. Because we are thinking about first asking an AI system to reason through, what are the ways in which it can be biased? And then, given the ways in which it can be biased, constructing counterfactual prompts, other ways in which to construct those prompts. For example, what output would be generated if we worded the prompt slightly differently along gender, around age, or along body type, and so on. And then looking at the output and comparing it and saying, “Hey, the output changed a lot when I changed the prompt in terms of gender.” That approach is easily applicable to other settings. In fact, we are currently working on applying the same kinds of ideas in the context of the use of large language models and enterprises. We don’t have a paper yet on it, but we are certainly extending our study in that direction.

Basiouny: Is the TIBET tool available to the public?

Shukla: It’s currently in progress. The paper is out. We are finishing our tool, but it’s going to be soon available. We’re just trying to wrap up the nitty gritty details and make it a tool that people can use.

Hosanagar: While the tool is not fully available for anyone to use, we do provide a lot of the code that anyone can use to also apply it in their settings. We provide a lot of the data explanations on our website. You’ll find our paper there. You’ll also find raw code that anyone can use in their settings. In terms of a user-friendly tool, that’s in the works, and hopefully down the road.