In his new bestseller, David and Goliath: Underdogs, Misfits, and the Art of Battling Giants, Malcolm Gladwell looks at what happens when ordinary people confront powerful opponents. He starts the book by dissecting the classic tale of David and Goliath, challenging our beliefs about what the story tells us regarding underdogs and giants, and ultimately, our fundamental assumptions about power.

Wharton management professor Adam M. Grant recently interviewed Gladwell about his new book when he visited campus as a guest lecturer in the Authors@Wharton series. Gladwell shared why he never roots for the underdog, where he comes up with the ideas for his books and sets the record straight on the biggest misunderstandings about his work.

An edited transcript of the conversation follows.

Adam Grant: Let’s start talking about your latest blockbuster: David and Goliath. What’s the core message and idea?

Malcolm Gladwell: It’s an examination of the idea of advantage and, particularly, it looks at asymmetrical conflicts – conflicts between one very large and one not-so-large party. How do we account for the unusual number of successes that underdogs have in those situations? The book takes off from there to try and figure out whether our assumptions about what makes for an advantage are accurate.

Grant: Could we just be wrong about who has the advantage in the first place? We may have labeled somebody an underdog and, in fact, they are not. Or is it more complicated than that?

Gladwell: The opening chapter in the book is about the actual retelling of the Biblical story of David and Goliath. There, it is very clear. David is not in any sense the underdog. Properly understood, once he has decided to change the rules of the conflict, the sling in his hand is such a devastating weapon that no contemporary observer of that battle would have thought David was a long shot. Once they realized he was winding up with his sling, they would have realized that he had all of the cards. We are misled by the narrowness of our assumptions about what constitutes an advantage in any given situation.

Grant: That plays out in a wide range of circumstances in the book. So, talk to us a little bit about desirable difficulties.

Gladwell: “Desirable difficulties” is a notion taken straight from the psychological literature from the work of [Robert and Elizabeth Bjork at the University of California, Los Angeles]. They were interested in that [idea] in the context of learning theory. It is not always the case that if I make the task of learning something easier for you, your performance will improve. There are sometimes cases where your performance will improve if I make the task of learning more difficult for you. Not always, but what they do is draw a line between difficulties that are ultimately desirable and those that are not.

“There is some sense in which David and Goliath is a mild rebuke to some of the more sweeping conclusions of Outliers. I am fine with that. No book is the last word on a subject.”

I play with that idea in a number of contexts and [look at whether there are] cases when having dyslexia is a desirable difficulty – that is to say, where you end up being better off than you were before. The answer is, there is a small number of cases where it is plainly the case, at least according to those who have dyslexia and who achieved enormous success — particularly entrepreneurs. That’s the group that is most interesting here: We see so many entrepreneurs who have dyslexia. When you talk to them, they will tell you that they succeeded not in spite of their disability, but because of it. For them, they view their disability as desirable, ultimately. That’s interesting. That suggests that the distribution of responses to an obstacle are profoundly bimodal. We pretend they are not. Similarly, I look at this weird association between very successful people and having lost a parent in childhood.

Grant: Not a desirable difficulty.

Gladwell: For some small number of people, a parental loss appears to be, ultimately, a desirable difficulty – again, not a large number. But there does seem to be a class of obstacles that for some people – for whatever reason – has an advantageous outcome.

Grant: Where do you draw the line? What is it that differentiates people who end up on one side of the distribution versus those for whom the obstacle is insurmountable?

Gladwell: You can’t draw a bright line. We can speculate. For example, if you look at the class of dyslexics who end up as successful entrepreneurs, they obviously have certain things in common. They tend to be highly intelligent. I interviewed maybe a dozen of them. In almost every case, the successful dyslexic had one family member who always believed in them.

Maybe one way of saying it is if your only obstacle is dyslexia, then it could be desirable. But for a child who grows up in a low-income neighborhood, who has an average IQ, who has a troubled family life, and has dyslexia, it is not going to be desirable. You have too many obstacles to deal with. But if we start limiting the number of obstacles, then maybe it is different. So that is one idea.

Another has to do with attitude. For whatever reason, some people choose to interpret their circumstances differently. One of the chapters is all about a famous oncologist named Emil Freireich who has a Dickensian childhood and then goes on to achieve enormous things as an oncologist. There was a moment in my conversation with him when he describes a horrendous childhood. He says, “So there I am. I am sixteen years old and I am wildly optimistic.”

You realize it was a complete non sequitur, but not for him. He was orphaned. He grew up in poverty on the streets. But he just thought that was an occasion to look on the bright side. So where does that come from? I have no idea.

Grant: One of the [concepts] that you bring to light is … this personality trait of disagreeableness. It’s something with which I struggle, and I know you have commented it is not your forte as well. How does that figure into the story?

Gladwell: There is a wonderful psychologist at the University of Toronto called Jordan Peterson, whom I had a long conversation with about this. He says that if you look at the big five personality traits, entrepreneurs are characterized by openness — which is obvious — creativity; conscientiousness – again, obvious — diligence [and being] disagreeable. That is to say, they are not people who require the social approval of their peers. He makes a very compelling argument. I agree with that typology.

Grant: Theoretically.

Gladwell: Theoretically, yes. If you are going to do something truly innovative, you have to be someone who does not value social approval. You can’t need social approval to go forward. Otherwise, how would you ever do the thing that you are doing? I give an example in my book of Ingvar Kamprad, the founder of IKEA, one of the great entrepreneurs of the 20th century. At a crucial point in building IKEA, in the middle of the Cold War, he starts outsourcing to Poland, which was an unthinkable act in 1961. It would be like outsourcing to North Korea today. You would have to have so much audacity to do that…. Imagine if the head of Walmart said, “We are going to start sourcing from North Korea.”

Kamprad, who by the way is dyslexic, is just one of those people for whom it just doesn’t bother him. He won’t lose sleep over it. Or if he lost sleep about it, he would be more concerned about the health of his company than he was about his public reputation. To be able to do that is not easy at all. I see that trait, though, time and time again in innovators.

Grant: We have talked a little bit then about some of the advantages of disadvantages. Let’s flip this to the other side. A couple years ago, Barry Schwartz and I noticed, as we looked across lots of different studies in almost every domain we could find, that there could be too much of a good thing. Everything we thought might be valuable – whether it is practice or generosity or pretty much any virtue – if you had too much of it, it turned negative. How does that figure into the story of David and Goliath?

Gladwell: Your paper was hugely influential to my thinking. I read that paper and [thought that] it’s the best kind of insight. It is the most obvious. It is the thing your mom told you, which is, there [can be] too much of a good thing. But it is also [difficult to] wrap our minds around. We understand linear relationships. We understand diminishing marginal returns. We cannot understand the idea of the inverted U: The same thing that is positive at one level can turn negative at the other – with hugely deleterious consequences.

That’s the mistake that people in positions of privilege make. It is what dooms the favorite. The favorite assumes that they can extend their advantages indefinitely. If what makes me better than you at the beginning is that I have more resources, if I keep spending resources, I will always be ahead of you. It’s just not true. General Motors is not a more nimble, innovative company … than when it was at the height of its size and dominance in the 1970s. It is in profound decline. Microsoft is not more innovative today than it was when it was a fraction of its size. The American health care system is not better than other health care systems in the world by virtue of the fact that we spend 50% more per patient. In fact, I think you can very clearly make the argument that our health care system is as bad as it is because we spend so much money.

I gave a talk once at Columbia’s psychology department when I was writing my book, in which I presented the problem and I asked the audience – since they were all psychologists – to give me reasons why we struggle with the inverted U. People emailed them in. I got about 50 [responses]. Is it because for most of our history, on some evolutionary level, we never could get too much of a good thing, so we never got to that part of the curve? If you are living on the savanna and there is a drought every three months, there is no such thing as too much food. Maybe that’s just so baked into our system that maximizing surplus is the only way to get through life…. In the Western world, surplus is a condition of our lives. We are just woefully unequipped on some level for dealing with that.

“If the underdog loses, the underdog feels very little distress because [he] expected to lose. If the favorite loses, he feels a great deal of distress because every expectation was that he was supposed to win.”

Grant: It is interesting, though, because when we are in the observer position we have a different reaction. We don’t want to be the underdog necessarily, but we love to root for the underdog. Why?

Gladwell: This is paradoxical for the following reason. I understand on one level why. It is a version of the just-world hypothesis. It is that the world seems more just to us if material advantage does not automatically translate to dominance. We need the belief that those without obvious resources can win in order for the world to seem fair for those of us who are not in positions of power to feel we have a chance.

But the paradox is — this is slightly tongue-in-cheek — this is why I do not cheer for the underdog. That is, in a contest between a favorite and an underdog, if the underdog loses, the underdog feels very little distress because [he] expected to lose. If the favorite loses, he feels a great deal of distress because every expectation was that he was supposed to win. The empathetic position then, as an observer, is to cheer for the favorite, right? If your job as an empathic human being is to want to minimize human suffering, the suffering comes when the favorite loses. I remember as a kid watching one of the Olympic games, and I was cheering for a big track athlete. He was the favorite to win, and he lost. I realized in that moment the pain he felt was so much greater than the pain that those who never thought they were going to win would have felt had they lost. From then on, I felt that I have no choice as a human being but to root for the favorite. Rooting for the underdog requires that we be indifferent to the emotional distress of the person who is expected to win.

Grant: Potentially. Or, is it that we expect the joy of the underdog and those rooting for the underdog to outweigh the distress of the favorite?

Gladwell: This is interesting. This gives you an insight into my psychology. I am far more distress-avoidant than I am joy-seeking.

Grant: So bad is really stronger than good for you, as it so often is in psychology.

Gladwell: Yes. The fact that the underdog is happy means very little to me next to the distress of the favorite.

Grant: Okay.

Gladwell: But you’re right. Actually, it is so weird. I had never even framed it that way. That is how distress-oriented I am.

Grant: Let’s build on that distress then. One of the things that was striking to me about David and Goliath is the courage it took to, in some ways, challenge one of the core messages of Outliers. I walked away from Outliers with deep distress around how these early advantages could come even from a birthday and just build on themselves and accumulate and create a massively unfair set of circumstances. Then here you come in with David and Goliath and say, wait a minute, what you thought was an advantage is actually a disadvantage and vice-a-versa. How do you think about the reconciliation of those two?

Gladwell: There is some sense in which David and Goliath is an addendum to Outliers. It simply says, “Let’s complicate our understanding of advantage.” There is some sense in which David and Goliath is a mild rebuke to some of the more sweeping conclusions of Outliers. I am fine with that. No book is the last word on a subject. As a writer, you should — at least partially — contradict yourself on a routine basis if you are going to remain interesting.

“People have simplified the 10,000-hours thing ridiculously. I have never said that 10,000 hours was sufficient to achieve mastery. People have caricatured that claim over and over again.”

There is a great moment when Dick Nisbett made a turn in his career as a psychologist. In the beginning, [he thought] the fundamental attribution error was fundamental. Then he realized, actually, no, it’s cultural. It doesn’t refute the fundamental attribution error. It deepens our understanding of it to say, oh, actually it has roots in Western culture…. He got two great books out of going back and correcting his earlier position. That to me is the model for how you ought to behave in the intellectual world. You should always double back and say, “Well, now, wait a minute. This is more complicated here.” “Contradiction” is too strong a word. But you should be constantly revising your conclusions.

Grant: That’s the mark of an intellectual, right? To constantly be asking the questions as opposed to just fixing on an answer. It is interesting, though, because as a social scientist and as a writer very much inspired by your work, I have been waiting for somebody to go around and do the story of how Malcolm Gladwell generates his ideas. I am curious: If someone were to follow you from the inception to picking a story or identifying a study to the full book, what happens along the way?

Gladwell: I don’t really know. About five or six times a year I go to the NYU library, and I spend a couple of days – “browsing” is too mild a term – wandering around. I go through millions of journals in the most serendipitous way I can. Just to see what’s out there and see if I can stumble on something.

Grant: Without a clear goal or direction. Just to explore.

Gladwell: No goal whatsoever. Then I do a fair amount of speaking, and I always try and have conversations with people well outside my world. Today, I gave a talk in Philadelphia. One of the guys [who attended] runs a medical device company – a very, very small one. So I started talking with him because I’ve always had this idea in my head that it would be really, really fun to write to compare the way dogs are treated with the way humans are treated because they are not that dissimilar as problems for medical science. But the systems that surround doggy health care and human health care are profoundly different.

Analogous devices are used. Only, if you do a complex knee surgery on your dog, it is $7,000, and if you do it on a human being, it is $100,000. Now, is a human really 15 times more complex than a dog? No. But, anyway, I had this vague thought. Then I met this guy, and I started asking about this. He started riffing on it and gave me his card. That’s sort of how it works…. You take advantage of a little thought you had in your head when you meet someone by accident who happens to have specialized knowledge. You make sure you get his card.

Grant: It’s fascinating to think about it being so non-linear.

Gladwell: The non-linearity and the serendipity of it is what makes it fun. If it is too organized, it can fall flat on the page. I want my books to feel like there is a random element. It is supposed to be kind of like these accidental wanderings through the world. There is not supposed to be a grand plan. If there were, I think the books would lose some of their life.

Grant: You have done five books now. It is always interesting to look at what has changed about the way that you think about the world and, particularly, since this is a Wharton conversation, about the world of work and leadership in organizations.

Gladwell: I realize now that an effective leader or manager can come in a virtually infinite number of forms. I have way more respect for the heterogeneity of excellence. That took a long time because it is so tempting to try and paint a very specific picture of what you think effective leadership is or what an effective organization looks like. The older I get and the more I see, I realize high performers of one sort or another have certain things in common. But they are almost more distinguished by what they don’t have in common than what they do.

Understanding fit is a much more important issue than defining the characteristics of excellence — understanding the combination of individual and organization and why at different points in your life cycle you might want a very, very different kind of person. The purest example of this is in sports, where the notion of fit between the athletes that you have and the coach that you hire is only occasionally considered. You will read that they brought in a coach whose plotting style is ill-suited to the athletes that he has. And then you wonder: Why did they bring in that coach? Why do a plotting style if no one on your team wants to play the plotting style? It is interesting how hard that notion is. Maybe it’s because it renders the task of defining what you want a lot more complicated, and we would rather not deal with that.

Grant: One of the fundamental contributions that you have made to the world is to take people who have very simple ideas and get them to complicate them and question them and turn them upside down. What idea have you put out into the world that you think has been most misunderstood and that you would like to set the record straight on?

Gladwell: Well, there are quite a lot. I get misunderstood a fair amount. The thing that is said about my work that irritates me the most is that I cherry pick. I don’t think I do that at all…. We require those who construct arguments to do a reasonable survey of the literature and choose the evidence that is most relevant to their argument. That doesn’t mean evidence that is in agreement with their argument, but rather that it is relevant to their argument. You have to sift. I sift like anyone else sifts. But for some reason, that’s become a cliché about my work – that I simply zero in on things that accord with my preconceptions. But I don’t think I do at all. I try to be pretty good about it. So there is that.

Then, people have simplified the 10,000-hours thing ridiculously. I have never said that 10,000 hours was sufficient to achieve mastery. People have caricatured that claim over and over again, to my distress.

Grant: For me, it is an interesting example that challenges us to think differently about expertise. Part of what is so exciting about David and Goliath is that it shows just how clearly you are willing to say, look, there is one side of an argument, but wait there is another side of an argument. That is a great case for saying, look, the world is a lot messier than we think it is. So, thank you for joining us today.

Gladwell: Thank you.