How Is AI Affecting Innovation Management?

Wharton’s Christian Terwiesch and Valery Yakubovich join Eric Bradlow, vice dean of Analytics at Wharton, to discuss AI’s influence on innovation management, from risk assessment and strategy to organization and value capture. How can businesses use AI to ideate and better meet customer needs? This interview is part of a special 10-part series called “AI in Focus.”

Watch the video or read the full transcript below.

Transcript

Eric Bradlow: Welcome to the Analytics at Wharton and AI at Wharton podcast series on artificial intelligence. My name’s Eric Bradlow, professor of marketing and statistics here at the Wharton School. I’m also the vice dean of Analytics. What we’re doing in this series is to explore the role of artificial intelligence in various aspects of business. And today certainly is no exception. Maybe [this topic is] one that most people consider the most exciting, which is artificial intelligence in innovation management.

I’m joined by two of my colleagues. First is Valery Yakubovich. Valery is executive director of the Mack Institute for Innovation Management. The Mack Institute focuses on creating synergies between research teaching and the practice of innovation management within the school. So Valery, welcome to our podcast.

Valery Yakubovich: Thank you. Glad to be here.

Bradlow: I’m also joined by my friend and colleague Christian Terwiesch. Christian is the Andrew M. Heller Professor at the Wharton School. He’s a professor and chair of the Wharton Operations, Information and Decisions Department here. He’s also co-director of the Mack Institute. And he also holds a faculty appointment in the Perelman School of Medicine. Christian, welcome to the show.

Christian Terwiesch: Thanks for having us.

Bradlow: It’s great to have you both. So, let’s start. I hate it when people use jargon. Valery, maybe I’ll start with you. What is innovation management, and what does the Mack Institute do? And then we’ll dive into what role AI may have to play in that.

Yakubovich: Let me try. Also, I actually rely on Christian’s favorite definition of innovation, which I learned from my faculty co-directors. Basically, it’s about matching customer needs with the technological solutions we have out there. And what we do, basically, our priority— our departure point— is faculty research. We fund it, in innovation entrepreneurship. And then we try to translate it into experiential learning for students, and business practice. For that purpose, we have a course with students that is project based. We have corporate partners with whom we work trying to identify their problems and provide some kind of guidance, thought leadership.

And over the years, we identified basically four areas which are critical for us. It’s about opportunities and risks, how we discover them and analyze them— strategy development, organizing for innovation. What kind of organizational structures, teams, and so on, you set up and employ. And finally, value capture from innovation. And so in my view, today’s conversation is about how different types of AI, and in particular generative AI, affect all these areas.

Bradlow: Christian, all hell must be breaking loose at the Mack Institute. I mean, you guys are focusing on innovation, and I think most people would argue one of the big areas of application of these large language models like Bard and ChatGPT is innovation. How do you get started? How do you as a scholar think about your research? How do you think about it as a center director? How do you think about it as the chair of the department? With all the hats you wear, where do you even get started, and how do you think about it?

Terwiesch: I would narrow it down to three dimensions of innovation that, when I teach executive education, when I teach our MBA students, I would want to focus on. Right? There is the initial idea, and there’s this combination of solution and need that hopefully creates some form of value. That is something that a student needs to be able to manage from. I have an idea towards launching a venture. And I think Wharton has been wonderful at doing that. AI is helpful at that level, because a lot of the things that used to be very expensive, very difficult to do, such as market research, such as prototyping, through generative AI, got a lot better.

The second thing that everyone talks about is, in bigger organizations, established organizations, there’s a pipeline of ideas flowing. They don’t have just one idea, they have thousands of ideas. And that’s a process that needs to be managed. And AI has been— as we showed in some of our research—really good at fueling that process. Of filling that pipeline of ideas. And then the third thing is that process needs to have a direction. I mean, most organizations, unless they are venture capital firms, they have some form of strategic intent. And I have to help managers think about possible future states of the world, possible disruptive threats. And again, ChatGPT or other generative models can help me imagine a world for which I should be prepared that I myself would have not imagined.

Bradlow: Let me ask you, Valery, as well, whether it’s in general or the two specific uses that Christian mentioned, which sit in my area of the world too, which is marketing research and prototyping. As you’re talking to them, would firms really replace direct marketing research or creating minimal viable products and doing prototyping? Would they really trust artificial intelligence with this crucial step of the— let’s call it “new development process”? Or, what are you seeing out there?

Yakubovich: Well, actually, they definitely don’t transfer decision-making to these large language models. But what we do see, they bring them in and are trying to take all their corpus of knowledge they accumulated over years, bring it into interactions with these models, and try to automate some parts of the process. And actually, we are doing the same here. The consistent request we have, for example, from the medical school, engineering school, when we work with them on their inventions and try to develop Penn’s innovation ecosystem— they ask us for market research. And the demand is so substantial, we can’t meet it with our available tools and available resources. So what we are doing now, we are trying to actually figure out how to use large language models in assessing, let’s say, commercialization potential of new inventions in the school of engineering.

Bradlow: Christian, I know you’ve written a number of books. You’ve written stuff on innovation and innovation tournaments. You’ve been an entrepreneur. Can you take our listeners through this step-by-step process? Imagine the Mack Institute wanted to partner with Pennovation to try to help think about the economic value, or how to best launch projects. How would you use artificial intelligence to help support that process?

Terwiesch: I think in any innovation process, there are two key functions. There is the generation of opportunities. Create more, better, and higher variance opportunities. It’s a very Darwinian process, so we need to create variety first.

Bradlow: Let me start with that one. A lot of argument against generative AI is that it tends to work for the center mass of the distribution. It doesn’t give you very good ideas in the long tail. If it’s not in that trained corpus, you won’t see it. So, how do you think it does in that first phase?

Terwiesch: I’m glad that you asked. So, we have done a study based on MBA-generated ideas, of which we have thousands in our database after teaching innovation for many years. And we compare this with large language models. And to our surprise, the large language models are actually better at creating what we call “high variance innovations.” Of polarizing innovations. Of innovations where the payoff distribution has a high variance, which is good, because innovations, ideas, have this real option flavor — that if the idea is bad, we just cancel it. We don’t execute it.

Bradlow: Just for our listeners— as measured by whom? How do you measure the variance? Do you use humans, or do you ask an AI engine to score them on that?

Terwiesch: That would be magic. We’ll get to your second part in a moment when I get to my second part, which is the selection step. So, how do we evaluate the quality of the ideas? We do purchase intent studies, ascendant market research, where we go on MTurk, Prolific, or other platforms. We showcase textual descriptions of the ideas. We ask for purchase intent probabilities of large crowds, which is not perfect, but again, best practice of what you guys in marketing do very successfully.

Bradlow: So before you get to the selection piece, let’s imagine there’s a world where people are doing what you’re doing, and thousands of these types of— let’s call it “generative studies” are done. And MTurk and all of that’s done. Eventually, won’t an AI engine be able to forecast the stuff that we’re using humans for, to evaluate right now? Like, right now, we just have a data problem. But if we had the data, AI could do that, too.

Terwiesch: It’s a really interesting question, right? I mean, what we’ve seen with humans over many, many years, is the selection decision is always the hard one.

Coming up with ideas is— I don’t want to say “easy” but it’s something that humans can do, and now AI can do. When we try to use AI to predict the quality of an idea, it still struggles. And again, it’s not too surprising, in the sense that humans— even venture capitalists— are really having a hard time predicting the odds.

Bradlow: Valery, any thoughts about that? I’ll call it the “idea generation”—by the way, I love this bifurcation into the idea generation stage, versus the selection phase. And as a statistician, I would imagine with enough data, eventually, and enough variation— you know, as we always say, you need variation in X to be able to have good selection models. And you need to, of course, observe outcomes over time. I would imagine that if we’re sitting here five years from now, AI engines may be able to do better, a lot better, on the selection phase than they do now. But what are your thoughts?

Yakubovich: Well, actually, I think if we look a couple years back, we thought AI will never be creative. We always thought AI will be predictive. But creativity, based on existing data and so on. And suddenly, we’re surprised. We find that generative AI is quite creative. But if you think more about it, it’s not so surprising. One, how do we define creativity? We go back to Schumpeter, the famous economist. He said it’s about recombination of existing ideas. And because these large language models are trained on such a huge volume of information, which encompasses all kinds of diverse opinions— if the task is not very well-defined, it actually does better. Because it can produce all kinds of opinions, you can imagine. All kinds of customer profiles, you can imagine. And this very ability becomes very helpful for recombination, right? That’s why these findings are quite consistent with what we know how creativity operates.

Bradlow: And by the way, the nice thing that Christian is doing, he’s sharing a lot of his findings with the press and social media. So I had read that article. And I’m glad that I read it.

Terwiesch: Thank you!

Bradlow: No, I give myself a good grade, because that is what I read from the article. So I’m glad that I interpreted the findings appropriately.

Terwiesch: Yeah.

Bradlow: So, there’s the technology piece. There’s the innovation piece. But what about the company adoption piece? What are you guys seeing in the Mack Institute part of the world? Are companies embracing this as the next great opportunity, or as companies thinking, “My God. This is a threat to my business model.” What do you see happening out there in the world?

Terwiesch: So, we had just on Wednesday evening a session in Executive Education with the Customer Analytics Program that you’re well-familiar with, of course.

Bradlow: Which I taught on a Monday.

Terwiesch: Right. And so we were talking with the participants about what it means for their business. And I think many of them are struggling, making sense of the technology. They know it’s big, but they have a hard time. Like, “What do I do next? Where do I get started?” And I think— I hate to say this to a marketing professor, right? But, I mean, you start with your customer journey. You start with your customer pain points. It’s not the right strategy to say, “Let’s AI everything.” You look for the customer journey. What are the customer needs, where are the pain points? And then identify those and think along those customer journeys, where could AI be the right solution? You now have a new set of tools, and you can go through your existing pain points that you might have known for many years and fix those. Plus, you can find through the sensing technology of AI, by having it read customer reviews, by interviewing it, you can find new pain points along the way that you might have not been aware of.

Bradlow: I see. Valery, what are you seeing since as the executive director of the Mack Institute, a lot of your role is to interface with companies? Us as faculty directors, we obviously do a lot of research. We also interface with companies. But you’re really on the front lines. What are you seeing today, and how are companies thinking that the Mack Institute can help them?

Yakubovich: Well, I think right now that’s the major disruptive technology that preoccupies a manager’s attention. Basically, I mentioned briefly that we have this experiential learning piece at the Mack Institute. We do projects with companies. This semester, out of seven projects, four are about generative AI. And I looked at them before coming here.

Bradlow: Can you tell our listeners, without giving the companies, what are those projects? What are they on?

Yakubovich: Yeah. I can give you pretty much their names, their titles. Smart Supply Chain Management Using Generative AI. AI Synergies, Strategizing and Operationalizing Intelligent Transformation. It’s a very general kind of topic. One is directly relevant to AI in innovation management. It’s intelligent software testing with AI-ML innovations. So, software testing is one major part of making innovation. Or testing, prototyping in general. And finally, Evaluating the Potential for Disrupting the Mortgage Title Industry Through AI Technology.

What I also see— I was last week in Silicon Valley, meeting with our corporate partners, meeting with startups. Some of them gaining a lot of traction. For example, going back to innovation management, there’s one company that pretty much automated the process of patent writing, which is extremely labor intensive.

Bradlow: I saw an academic talk on that, maybe about two or three months ago, where I thought it was remarkable.

Yakubovich: Yeah. I mean, you can think it’s very well structured language, right? And very specific, very hard-to-understand language, if you read the patent. But I can imagine it’s pretty straightforward to train machine-learning AI, generative AI, large language model, to do it. And what I know is when companies deal with these things, what they need to do— they obviously take an existing large language model, foundation large language models. They don’t develop their own. But then they need safeguards. A wall between the vendor of the model, and their own knowledge base. And vendors now are willing to provide it.

At the same time, also, a number of startups emerged that actually are offering these companies security, privacy, and other tools to also not only safeguard their own knowledge base, but the knowledge base of their clients that they are going to use in order to deliver value to clients. So I think this privacy, confidentiality, are key issues, and security of these models. What we see going on.

And another example of very interesting creative application for innovation management— I encounter it. It’s a company that has a huge database of cancer patients. And now, they are trying to engage large language models to match it with the FDA’s database of clinical trials that are going on, in order to find the right subjects for the right trials. So apparently, it’s a huge value added that can be done now at large scale using these models.

Bradlow: Christian, let me ask you, on the biggest opportunity side and the biggest area where you still don’t see AI being used much, what are you seeing as, like— if you had to give a lecture tomorrow to your MBA students and say, “This is the most sophisticated, interesting, value-added application of AI. This is what I’ve seen in the last three years.” And if you also had to give the same lecture and say, “And here’s an example where I think there’s an opportunity, but I haven’t seen anything yet,” what would those be?

Terwiesch: In terms of what works, I think anything that is simple text writing, and simple, all the way to a new pattern—when it’s a writing task, AI is amazing. And I think that nut has been cracked. It’s only going to get better. I think as faculty, we all get these inquiries. “Professor Bradlow, could you summarize this paper for me?” I mean, don’t do this. AI can do this.

Bradlow: Part of both of our jobs is acting as journal editors and reviewers. The journals have a policy against this, potentially, right now. Maybe not in the future. But let’s say there was no policy. Should I take an article, jam it into an AI engine, ask it to give me a summary of it, in addition to my reading of it?

Terwiesch: As a reviewer, you should not. But as somebody who wants to stay current with the literature, having basically AI give you, every morning, a two-minute summary of a paper that otherwise takes an hour or two hours to read, I think that would be a very healthy thing.

Bradlow: Forget the ethical or moral parts of it. Why do you say, as a reviewer, I should not? I could give a reason, but I’m here to interview you. Who cares what I think? What do you think?

Terwiesch: As a reviewer, you have to turn over every stone to make sure that there’s not a flaw in the paper, in the methodology. I think that is something which is a corner that I don’t think AI is ready to do. I’ve tried this. AI is doing a decent job, when you feed it a PDF of a paper saying, “Look. There might be some issues with endogeneity,” or what have you. Some pretty generic ones. I don’t think it has the precision to dial in and say, “In equation seven, the arrow term is correlated with the explanatory variable.”

So, I think there, we still have to do the homework. The second type of work is the one that puzzles me the most, analytical types of things. Right? So especially at the beginning, when I gave GPT my MBA exam a year ago.

Bradlow: Please tell our listeners about this. By the way, I would guess it is probably the most publicized article that has come out of Wharton in the last five years. No, I’m just saying, it was on every major news network. It was republished everywhere. So please tell people about the study, what you found, and then what you’re still puzzled about.

Terwiesch: Over the winter break, my kids and I were sitting together. My kids are in college or through college. We were talking about GPT, like everybody else, probably, in the world. And so the question came up. “Dad, you’re teaching in this MBA course. Do you think that GPT could take your exam?” And so we literally took my MBA exam and fed the questions, cut and pasted them in the prompt line. And it did really well. It did what I would have given a solid B, if not a B-plus.

Over the subsequent months, then, with GPT4 coming out, it is now well in the A range. Basically, the type of questions— my questions are mini-cases, so to say. Ten lines of text, with some computations in there. “Find the bottleneck, compute the inventory cost. Do some queueing analysis.” GPT is amazing at that. Which again, is counterintuitive, because it’s a language model. Right?

Bradlow: Yeah.

Terwiesch: It has no representation insight of what capacity even is. You tell it to find a route through a traveling salesman problem, connect cities in the right sequence to minimize transportation time, it does a pretty decent job already, right out of the box. What is new since with GPT4— also, we have these plug-ins now. One of the set of plug-ins is from Wolfram, the power side for analytics.

Bradlow: That’s going to help.

Terwiesch: And that’s going to help, right? But even the plain vanilla GPT out of the box has gotten pretty good at doing analytics tasks. So I imagine, just as a lay person, most of our listeners here would have not used Wolfram analytics, unless they are nerds like you and me. Right? I mean, that’s an insider type of tool. You now have, basically, a user interface that lets you do sophisticated analytics that lay persons can do, and make inquiries into exploring, into analyzing hard mathematical problems, data sets, operations research type of problems. I think that is super exciting.

Bradlow: So, Valery, can you tell us about, what role do students play at the Mack Institute? And what do you think— when students, I’m sure, ask you all the time— they ask me all the time, especially around machine learning. “Professor Bradlow, what should I be studying now?” What do you tell students? Should they focus on becoming great prompt engineers? Should they focus on being able to take the output of ChatGPT, integrate it with their own beliefs, and then help in decision-making? How do you see that?

Yakubovich: Well, basically, they have a challenge now. And Christian’s research showed what the challenge is. You have to figure out, ChatGPT can do better, some tasks. You have to figure out where you belong. And there is actually another recent study done by a large group of researchers, some of them Ethan Mollick, our colleague in management. It was done at the Boston Consulting Group, where they really, in their own demise experiment, looked what consultants can do or can’t do with generative AI. And roughly speaking, my reading of the paper is that on exploration, their productivity drastically increases with generative AI, large language models. On problem solving, more specific contextual tasks which require more precision and understanding of the context and so on— those who use generative AI do worse.

So basically, we know— actually, Ethan and his colleagues talk about this kind of changing frontier, which tasks can be done or cannot be done by generative AI. And it’s hard to identify, and it’s a moving target. Right? But they need to experiment themselves. They have to innovate and reinvent their careers in some sense, so disruptive this technology is.

Bradlow: Yeah. So Christian, let me ask you. If we were sitting here five years from now, what do you think will have changed in those five years? Is it the language models will get better at prediction? Which, I’m sure the answer to that is yes. The application areas that we haven’t even thought of will get done? That’s probably yes. But what do you think are the big changes our listeners should know that is coming in the next five or so years?

Terwiesch: I think we have to stop thinking about the substitution game, where, will the MBA students be replaced by GPT? The amount of work that is going around in the world is not constant. If we make it efficient and cheap enough, there’s new work that is going to bubble up. I just wrote my latest paper, a paper on ethical advice an AI— can AI give me ethical advice? And we show in the paper that it is basically as good as The Ethicist in The New York Times on providing advice to readers, or people who face ethical dilemmas.

So what does this mean? One hypothesis, it puts the ethics advisors out of business. But that’s not what I think. What I think is going to happen is, we’re going to go to a virtual, zero-marginal cost ethics advisor a lot more, and get more ethical advice. Right? And so this productivity gain is not putting people out of work. The amount of work that we can productively serve is going to go up. And so the effect on employment is highly ambivalent.

Bradlow: Yeah, please, Valery.

Yakubovich: Just to add to this, I just worked on a paper with Peter Cappelli and Sonny Tambe, on trying to think through these organizational implications of generative AI. And one point we are trying to make is, the jobs consist of multiple tasks. And what we see so far, some tasks indeed can be automated. But then the amount of information you have to process, produced by generative AI, is at such a scale that sometimes it becomes costly. You need to now make sense of this.

And the problem is that these models, the expandability is still a big problem. So I talked to some engineers in this area related to health care. And they’re saying that new types of models now, they believe might replace large language models. Which actually can understand things at a large, more conceptual level. When, instead of predicting the next word, as these models do, you kind of, based on what you know so far, you predict the next high-level concept. And the text will follow from that concept. This is the way we think.

And so apparently, there is quite a bit of traction in that area. So we’ll see. Large language models, is it the last word? Or, we’ll have a totally, or quite different, technology which is already out there for, for example, visual images. Basically, when you see a part of the image, you reconstruct the second part completely, instead of specific pixels, right? So there is a lot of development. Again, it’s a moving target which— it is exciting to watch these developments. And we need to adjust quickly, to explore these things as they appear.