“I’m just lucky.” “I was in the right place at the right time.” “I’m not ready for that promotion.” According to Good Morning America reporter Claire Shipman and BBC anchor Katty Kay, these are phrases used almost exclusively by women when talking about their careers. In their new book, The Confidence Code: The Science and Art of Self-Assurance—What Women Should Know, they explore why women lack confidence and what it means for their careers.

Wharton management professor Adam M. Grant recently interviewed Kay about her new book when she visited campus as a guest lecturer in the Authors@Wharton series. In this interview, Kay discusses the research about this confidence gap, the importance of confidence and how it can be addressed.

An edited transcript of the conversation appears below.

Adam Grant: What drove you to write this book?

Katty Kay: [W]e wrote a book about six years ago on the value of women in the workforce. For that book, we interviewed a lot of senior women in business, in the military, in politics. We were struck by phrases that they would use, phrases like, “I’m just lucky to have got where I got to”; “I was in the right place at the right time”; or, “You know, I think I’m not quite ready for that promotion yet.” It occurred to us that we never heard men say things like this. 

Grant: How could that be?

Kay: [I]t just struck us that something was happening with women in the professional space that was not happening in their home lives. When you ask them about their kids or their friends, they think they are great. They are totally confident of their ability to make friendships or be great mothers or supportive wives. But get them into the professional space, and we wondered if it was just anecdotal or if there was actually data behind this.

Grant: What led you to the initial idea that it was a confidence gap as opposed to humility, let’s say?

“That’s what the confidence gap is. Women don’t believe they are as good as they are.”

Kay: Because it wasn’t just words. They weren’t saying one thing and doing another. They genuinely believed they weren’t good enough when you start looking into all of the data. Wharton’s done some of it. Columbia Business School has run numbers [to show] men overestimate their abilities by some 30%, [whereas] women routinely underestimate their abilities. We talked to a lot of psychologists who were working in business schools who put men and women in front of scientific reasoning quizzes. The women will routinely think they have done less well than they have done. The men will think they have done better than they have done. In reality, they have done about the same.

It’s that women’s perception of their ability skews below their actual ability. It’s not that they are just saying, “I’m not very good,” but actually thinking they are really good. They don’t believe they are as good as they are. That’s what the confidence gap is. Women don’t believe they are as good as they are.

Grant: You point out that it’s not just other women, but that as a BBC journalist, you had actually experienced this yourself.

Kay: Oh, yeah. I’m riddled with this, and have been for the last 30 years of my career. I spent years in America saying that the only reason I’ve been successful in America is because I speak the way I do. I mean, it can’t possibly be my talents, right? Or my ability or my hard work. That’s preposterous. It had to be some external factor. In my case, it was the fact that I speak with a British accent, which makes people think I’m smarter than I am. I actually believed this, Adam. For years I believed this. Claire [Shipman], my co-author, will tell you. She has been banging on about this one for years.

Claire thinks [she] became CNN’s Moscow correspondent because she happened to be in the right place. That is very common for women.

Grant: What did you do when you noticed that there were people with British accents here who weren’t successful? Because that would violate the theory at some level.

Kay: You always find some reason they must have some problem. 

Grant: OK, so, you identified the gap.

Kay: No one’s ever asked me that before.

Grant: Good. There is a lot of data behind the confidence gap, right? What do we do about it? Where does it come from, and how do we start to solve the problem?

Kay: Well, it led us then on this rather tangential quest to find out what confidence is. We thought that if we were going to try and grow confidence, we were better off if we knew what we were dealing with. So, we interviewed dozens of neurologists and psychologists for the book. We would always start off with this simple question, “Can you define confidence for us?” Inevitably, we would be met by a pause: “Hmm … well, it’s complicated.”

We went into this with a couple of misconceptions. First of all, we thought that confidence was the same as self-esteem — a general feeling that you’re a valuable person. I have high self-esteem. I think I’m a valuable person, the universe is a friendly place. It’s an almost sort of moral emotional quality that is pervasive to who you are as a human being. 

“Richard Petty, who is a psychologist at Ohio State, … said to us that he thought the best definition of confidence was this: ‘Confidence is the stuff that turns thoughts into action.’”

We were wrong. Confidence is not the same as self-esteem. We also thought — and this was wrong, too — that confidence is a manner, a mannerism, it’s a bravado, a sense of swagger. It’s that thing of dominating meetings or speaking loudest and longest. We were wrong [there] as well. Confidence has nothing to do with a manner. It’s to do with a belief that you can succeed at something. 

We would ask all these neurologists and psychologists, “What is confidence?” Finally, it was Richard Petty, who is a psychologist at Ohio State, who said to us that he thought the best definition of confidence was this: “Confidence is the stuff that turns thoughts into action,” which is beautifully simple. The great thing about that idea is that not only does it turn thoughts into action — “I want to introduce myself to that interesting-looking person at a party, but I feel nervous about doing so” — confidence gets you across the room to shake somebody’s hand and introduce yourself.

When you do it, when you take the action, you grow your confidence. So, it’s a wonderfully virtuous circle. Confidence is about action. 

Grant: Even when you take the action ineffectively, though? So, you make the introduction. It’s a complete disaster. What happens then?

Kay: Yes. Even if you fail, even if you meet hurdles when you try something new. It’s inevitable, right? You’re always going to meet hurdles. You introduce yourself to that person, they brush you off. Think to yourself: “What’s the worst that’s happened? Did the sky fall on your head? Did the earth open up and swallow you whole because that person brushed you off?” No, you’re still standing. You’re still there. And in a sense, even if you fail, you’ve learned that you can take a risk, try something hard, even if you fail, you’re still there.

Now, if you keep doing that, eventually you’ll succeed. If you introduce yourself to the next person, the chances are pretty slim that they are going to brush you off as well. You’ve learned that you can do something and that your world doesn’t fall apart because you try something that’s outside your comfort zone.

I read the news every night to millions of people around the world. It doesn’t test my confidence. Working at Wharton would terrify me. I’m absolutely convinced I would be useless at doing what you do. The only way I would ever find out would be to try. That’s how you build confidence. You take something that is challenging to you, that seems difficult, that is new and hard and outside your comfort zone — a small thing or a big thing — and you keep going, overcome hurdles and you succeed to some degree. And that’s how you build confidence.

Grant: It’s an interesting counterpoint to the self-esteem movement, which we all know from mountains of evidence did almost no good and a lot of harm. This is different, right? This isn’t about looking in the mirror and saying, “I’m really great.” It’s actually about increasing your ability to, as you said, convert thoughts into action. Do you have any other favorite strategies for boosting confidence?

“You take something that is challenging to you, that seems difficult … and you keep going, overcome hurdles and you succeed to some degree. And that’s how you build confidence.”

Kay: One of the reasons that there’s a confidence gap between men and women is women often find action harder than men because we are more risk-averse, because the fear of failure is enormous for us. It seems to be bigger than it is for men. 

The other thing that women do is we think a lot. We bounce around inside our own heads. “I sent Adam that email. He didn’t get back to me after half an hour. Maybe he’s mad at me…. Maybe all of Wharton is mad at me. Maybe everyone at Penn is mad at me because I didn’t get back to them.” That’s the way women work. We extrapolate — we take one small thing — a small slight, a small criticism, a small thing we’ve done wrong and it holds us back from acting and trying hard things because we’re running around in our own heads.

One of the things we [suggest] in The Confidence Code is you have to think less. You actually have to draw a line under those thoughts. Women dwell on the things they have done wrong. It’s what happens in review processes. It’s what happens in negotiations with our bosses. And what happens when there’s one piece of work we didn’t manage to hand in on time that day, even though we’ve done five other good pieces of work; we’ll remember the one piece of work we didn’t do so well. We need to find a way to draw a red line under that.

Grant: It’s interesting though, because everything you say, to me it sounds like a list of desirable attributes. So, isn’t a possible solution here just to get men down to the level of reasonable confidence?

Kay: [W]e’re asked, “Well, aren’t we at risk of pushing women into overconfidence?” I don’t think we’re at any risk of pushing women into overconfidence. I see no evidence that we’re going to suddenly become Lehman Brothers redux — all of us. I just don’t think that’s going to happen.

Ideally, everyone needs a little bit of overconfidence. It’s interesting that psychologists disagree about most things, but the one thing they actually do agree about is that a little bit of overconfidence is better than a little bit of under-confidence. Men probably have too much. And it might be better for all of us if some of it came down, particularly if you’re thinking of a cultural environment in the office space, right? If you’re thinking of meeting environments or employee/employer relations. That is, to some extent, a mannerism. 

“Get over the fear of failure — that techie buzz phrase ‘fail fast’ is a great one for women. Women hold themselves to a very high standard.”

But I think the priority is to get women over this hump of under-confidence because that’s part of what’s stopping them from taking action and getting to the next level.

Grant: So, you’ve talked a little bit about people stretching outside of their comfort zones. If you were going to create your wish list of a couple of steps that women and also men lacking confidence ought to take, what else would go on that list?

Kay: Be prepared to fail. Get over the fear of failure — that techie buzz phrase “fail fast” is a great one for women. Women hold themselves to a very high standard. We know this. Women are 25% more prone to perfectionism than men are. We’re perfectionists at work. We’re perfectionists as wives. We’re perfectionists as mothers. We’re perfectionists in the yoga studio. You name it, we want to be perfect at it. If you’re going to try and be perfect, you’re never going to get there. It’s an impossible standard, right? No one is ever going to be perfect. 

But the pursuit of perfection is something that holds us back from taking risks because it makes us very scared of failing. One of the first things that people who are under-confident need to do is give up trying to be perfect. It will be the single biggest thing that they can do to help them take risks and be prepared to fail.

Grant: In closing, is there a point that you feel has been misunderstood or oversimplified that you want to set the record straight on?

Kay: Yes. Some people have said to us, “Well, aren’t we just trying to make women like men?” We wrestled with this when we were writing the book. Basically, do you have to be a jerk to be confident? Because I think a lot of women look around them, and they see a very male model of confidence in the professional space that frankly is unappealing and inaccessible to us because it’s downright foreign.

It was Christine Lagarde, the head of the [International Monetary Fund], who was really helpful to us in explaining this. She said it’s essential for women to be authentic. Don’t give up the very qualities that make you valuable — an ability to listen, ability to build consensus, a high EQ that’s good at reading a room, warmth. Warmth is an amazing quality to have. It’s a very powerful quality. Don’t give up all of that in the pursuit of a mannerism — bravado and swagger — which doesn’t really suit you, and when you try it, doesn’t work for you anyway.

You want to be confident in that you want your voice to be heard. You don’t want to apologize. You don’t, as Sheryl Sandberg says, want to lean back. But you want to do it in a way that is authentic to whom you are as a woman. That’s critical. We’re not asking people to become somebody different. We are just asking them to bring their perception of their abilities in line with their abilities. When you’re there, you’re in the sweet spot.