Brought to you by the Wharton School in collaboration with Accenture, Where AI Works explores AI’s real-world impact on business. Each season takes a fresh approach, led by a different Wharton faculty expert who brings their own AI-focused expertise to the conversation, alongside practitioners actively shaping AI’s role in innovation, strategy, and transformation.
“AI is rewriting the playbook for business and society. In Season 1 of Where AI Works, I’m excited to explore what that means for decision-making, marketing, and beyond.
Kartik Hosanagar
Professor of Technology & Digital Business,
Co-Director of Wharton Human-AI Research
Host, Season 1: The Impact of AI in Marketing
Brought to you by the Wharton School in collaboration with Accenture, Where AI Works explores AI’s real-world impact on business. Each season takes a fresh approach, led by a different Wharton faculty expert who brings their own AI-focused expertise to the conversation, alongside practitioners actively shaping AI’s role in innovation, strategy, and transformation.
“AI is rewriting the playbook for business and society. In Season 1 of Where AI Works, I’m excited to explore what that means for decision-making, marketing, and beyond.
Kartik Hosanagar
Professor of Technology & Digital Business,
Co-Director of Wharton Human-AI Research
Host, Season 1: The Impact of AI in Marketing
New episodes will drop every two weeks
Season 3: AI & Workforce Transformation
In Season 3, Wharton Professor Peter Cappelli unpacks how AI is reshaping the workforce—shifting job roles, redefining skills, and driving human-AI collaboration while businesses navigate job displacement and ethical AI challenges.
Episode 1
Guest: Karalee Close, Global Lead Talent & Organization, Accenture
Transcript:
S3E01 — Work, Rewired: Rethinking Talent, Strategy, and AI
Karalee Close 00:01
Quite oftentimes in the literature, you have Gen AI pitted against humans and jobs, and I was really interested in what’s the context that drives a behavioral shift? What do we have to do with skills and learning? What do we have to do with change management practices that creates a more positive tissue connection between people and technology, as opposed to just deploying technology and expecting that the world’s going to change differently.
Peter Cappelli 00:25
Hello and welcome to season three of Where AI Works, conversations at the intersection of AI and industry, brought to you by the Wharton School in collaboration with Accenture. My name is Peter Cappelli. I’m a Professor of Management and Director of the Center for Human Resources at the Wharton School. It’s my pleasure to be your host for this season of the podcast, as we tackle the big questions that are shaping AI’s role in the world of business today and into the future. Our goal here is to cut through some of the noise and to get to actionable insights for business leaders. We’re going to combine cutting-edge research with real-world case studies. Things are changing fast, so let’s dive in.
News Clip One 01:11
Organizations are racing to adopt AI to improve business processes. But the biggest opportunity…
News Clip Two 01:18
…change management, change management, change management.
News Clip Three 01:21
This is about humans and agents working together as a team.
News Clip Four 01:24
Any company that’s not going fully into agentic AI is going to have of a hard time navigating next few years.
Peter Cappelli 01:31
So in this season, here’s what we want to do, which is maybe a little different. We want to focus, not so much on conversations about what AI will do, but we want to talk about what it is actually doing now. We want to focus on the real world impact of AI in the workplace. We’re going to hear from business leaders who’ve been in the thick of it, trying to solve real problems, some messy process, often, of managing change. We want to learn about that as well. In my own research, I’ve been looking at the complications of trying to introduce AI and the fundamental issues of conflict between management and the laws on which it’s based, which are rooted on fairness questions about human psychology, frankly, and the issues of AI, which are rooted on sort of optimization thinking, and often those come into conflict. So to begin our season, we’re delighted to have as our guest, Karalee Close. Karalee is the Global Lead, which means the head of, if you’re outside of that business, Talent and Organization practice at Accenture. She is based in Victoria, Canada. Karalee, welcome to Where AI Works.
Karalee Close 02:39
Thank you for inviting me to join you, Peter.
Peter Cappelli 02:41
What I would like to start with, Karalee, because you’ve had a long career now, starting on the management consulting side, before coming to Accenture. And so I guess my question for you, the first question is, what was it like to drop into the IT world from management consulting? What seemed different?
Karalee Close 03:01
Yeah, so I started my career in healthcare, and I worked in strategy consulting and technology-enabled transformation for 23 years with global companies and across industries. I think, dropping into Accenture and into the technology world. You know, a lot of it’s familiar because I led a technology practice at a strategy firm. What’s different, I think, is we’re actually really working with technology and changing companies in parallel. And so my work actually has been not only working with global companies to transform with AI, but also using AI within our own operations to change the way we work with technology. So that is our corporate functions, marketing and sales and finance and HR, really deploying the best technology, data and AI and how we work. Accenture is a little bit interesting, because we have strategy and consulting as well as technology. It’s all about people. So especially this next era of technology-enabled change is going to be much more about changing how people work with technology than it is about just deploying the technology.
Peter Cappelli 04:12
So let’s see if we could jump into what we really would like to talk to you about, and that is at Accenture, what you did there with AI to change your own management practices. This is an enormous company now, isn’t it? How big is Accenture?
Karalee Close 04:26
Accenture has just over 800,000 people.
Peter Cappelli 04:29
800,000 people. Are you the biggest company in the world, sort of like this, consulting management per se? Anybody bigger? I don’t think so, right.
Karalee Close 04:36
Yeah, it’s one of the biggest companies, and it’s also uniquely integrated across strategy and consulting, technology and operations. So it is a moment where you can really think differently, and not only about the technology and how it’s deployed, but actually how you fundamentally change the way human organizations actually work with technology.
Peter Cappelli 04:56
So let me ask you about the particular problem that you undertook the AI solution to a problem, but what was the problem before you got to AI, or what was it you were trying to solve?
Karalee Close 05:07
Fair to say here then that it wasn’t like there was a particular immediate problem. It was more an opportunity that you had all this Gen AI ability and stuff, and let’s see what we could do with it.
Karalee Close 05:07
So most people start with an ambition to drive efficiency, speed and cost savings, is absolutely what we need to do within the way that we do marketing, the way we do finance, the way we do HR. So the bigger question, I think, is, what are the new things that we can do with Gen AI that we couldn’t necessarily do before? So for marketing and communications, the function that I was really working with, with our chief marketing officer, we were really trying to say, how could we use the best tech, data and AI as well as operational setups, to improve the way that our marketing function works. So to be the best marketing company in the world, to improve efficiency, but also to do new things that we couldn’t do before. So it’s the combination of understanding how work happens and also how the technology works that I think is an interesting one to create impact. The other interesting thing is to think differently about how to drive change, which is what I was really interested in pioneering. Quite oftentimes, in the literature, you have Gen AI pitted against humans and jobs, and I was really interested in what’s the context that drives a behavioral shift? What do we have to do with skills and learning? What do we have to do with change management practices that creates a more positive tissue connection between people and technology, as opposed to just deploying technology and expecting that the world’s going to change differently so. And in that, I think we have a lot of disciplines around change management. Thinking about adopting technology isn’t sufficient in this next era. It’s really about getting people to work differently. So we got right into, how do you really redesign the work, and how do you reshape the workforce, so tasks and skills and jobs that people do as marketeers as well as the organization design and the change management. The change management isn’t deploy the technology. The change management is continuous change, because for the first time, we have a technology that learns, and getting that co-learning and co-working to happen between the Gen AI agents, we have 14 now working in our end to end campaign optimization, getting those technologies to work with people and getting marketeers to embrace a new way of working was the change challenge.
Karalee Close 05:21
Well, there’s always an opportunity to get better at what we do, right? So we’re over 800,000 people, and the marketing and communication function was really looking to reimagine how we operated to support the needs of the business going forward.
Peter Cappelli 05:45
Okay.
Karalee Close 05:45
If I take marketing campaigns as an example, they were taking upwards of 150 days to launch. So think about any time you see Accenture’s brand in the market, there’s 83 different steps within the process. We were really looking to say, could we do, you know, a better job of supporting our brand and our marketing activities with fewer resources? And could we do some new things that we couldn’t do before?
Peter Cappelli 05:45
It starts with being faster, and that would be more efficient, too. And are there examples of things that you couldn’t do before and you knew you wanted to do them, that you could share with us, the marketing people? We would like to be able to do this.
Karalee Close 05:45
Well, it started with a classic sort of set of things around understanding where the work happens and how the work happens, and actually thinking about organizing the work in a different way. But one of the examples of what we could do with Gen AI that we couldn’t do before was to sort of redesign the way that we thought about understanding competitive intelligence. So if you think about at any point in time, any number of our competitors would be talking about any topic, because we work across 14 industries, the ability to understand what other organizations are talking about, what our brand message is and our new IP is, and the ability to frame our own thought leadership in the context of what our brand promise is, and also in the context of what would be different from other competitors, we can now do, you know, not with a whole pile of people doing analysis, but actually having a workforce of AI agents that go out and instantaneously look at what’s just been published, look at our brand guidelines, and also look at providing feedback to people who would be writing different thought leadership pieces.
Peter Cappelli 09:29
Right, so the point about that for other organizations is that you could get intelligence in real time, cheaper, better, faster.
Karalee Close 09:37
Well, and we could provide feedback to humans. So if I had an idea that I wanted to talk about the feature workforce, I can now get instantaneously, a pulse of what’s being said by my clients as well as by my competitors. I could have that checked against other things being written that I wouldn’t know about in our organization, and I could get input. So strategic advice effectively, or writing advice effectively, that is impossible with just humans.
Peter Cappelli 10:08
So get us into the agent context here a little bit. So a lot of folks hear that, they don’t know exactly what we’re talking about. So what is the AI agent here?
Karalee Close 10:18
It’s an agentic workforce, in fact. So there’s combination of utility agents that do basic tasks, orchestrator agents that work across those utility agents, and super agents that provide insight and recommendations on the activities. So if I think about a super agent that is a marketing companion that will provide insights through an end to end program, it’ll make recommendations on which channels and what tactics could be used. And we have a critical thinker agent that analyzes the validity of potential biases and strategies, provides some critical insight into the market tiers as they’re planning campaigns. We have the base level of agents that can do things like booking things in an editorial calendar, so that they can plan and program launch in a much more efficient way.
Peter Cappelli 11:08
So in addition to having software that can handle each of these tasks, being able to put them together in sequences is really something new.
Karalee Close 11:18
Well, it is, and you actually don’t have to define the sequence, you define a strategic intent. The interesting thing is, you have to define a strategic intent in an organization as well. If you’re an HR professional, you do think about that as an intent for teams and KPIs and the like. And we found ourselves sort of saying: okay, well, rather than telling the agents what needs to happen, we’re going to give them a strategic intent, right? So the strategic agents will we give them an intent around a campaign, an outcome that we’re expecting, and then they work to achieve that campaign, much like you would expect a human to do.
Peter Cappelli 11:53
Walk us through, if you would, a campaign intent. So you’ve come up with a description of a campaign and what the goal of it is. What happens next?
Karalee Close 12:02
Well, so you have the intent of the campaign, and then you have the different utility agents actually working through the execution. So you have the original sort of planning. So you have the strategic insights agent providing advice on, you know, other things that would be happening in the market. Keep in mind that the world changes quite regularly, and new things are happening everywhere. And so the agents would provide input into the strategy, and then they would also provide input into the specific tactics that would be associated with a campaign.
Peter Cappelli 12:36
So let’s shift a little bit here. The hardest part of getting this done. What would you say the hardest part was?
Karalee Close 12:43
The first thing is the ability to get the leadership to embrace a new way of working. So the change delta, we often talk about, what is the ambition? What is the step change in performance that you’re expecting? We needed our marketing team to think differently about the application of the technology. You can’t do that if you don’t understand the technology. So a base fluency around what agentic AI actually really means. But at the time, this was started last summer, right? So at the time, it was all still quite new through our Nvidia partnership, right? So it was all a little bit about learning and doing at the same time. With the leadership team, our Chief Marketing Officer, did an amazing job of setting a context for the change, setting a level of ambition, that was one.Two was a base fluency, like an understanding of okay, the work has to happen differently. People often redesign work around the way that they think work should happen, or the way they think a technology will work and core, like IT, doesn’t necessarily always understand some of this new intelligent agent. So we needed a team approach, right? A team that understood the data and AI aspects of this, and a team that understood our underlying core systems and the data sets that the models would be trained on, we also needed really new approaches to understanding the roles in the organization and this tasks that were involved and the skills that are involved in producing certain types of work. So when you think about redesigning work more fundamentally, you’re basically saying: If I lay out today’s process, I could actually fundamentally reimagine the way the work happens. I can automate some things traditionally with existing technology. I can also augment the work that people are doing with agents. And so what we needed was a combination of technology disciplines, and I’ll use the plural, right, so core technology as well as data and AI. And then we needed some HR disciplines that were really different. Those HR disciplines aren’t just, well, tell me everybody in the organization, and then tell me what the technology is going to do, and then I’ll just work the change. But it was to say we had to decompose people’s rules, which the unit of organization is, roles, right? The first time we looked at this, we actually didn’t come up with a massive amount of savings because we hadn’t unpicked the unit of job. So deconstructing jobs became the next challenge to say: Can this agent take over this particular role? And oftentimes, no. Oftentimes it takes over a set of tasks, but not necessarily the role. So we had to work very iteratively between what the technology can do and how it could be trained to what the people can do and how they can be trained. We had a philosophy of like progress over perfection, right? So we tried a few things and then learned across the technology as well as the people. And had to fit the thing together. And actually it’s a continuously evolving thing. So our agents are actually getting even more and more proficient, and our marketeers are also getting more proficient at working with the agents. And so there’s this new concept, I think, of co-learning, co-working with agents that we’re not pushing far enough as an industry, right? So this is not from a change perspective. I have a new thing, and I want to roll it out, but I have a new capability, and I need to figure out how to get people to embrace that capability and how to change the way that work happens, you know, with it. And so what we found was providing two-way communication between the people and the agents. So if an agent comes back with a recommendation and it’s actually not good, giving people the ability to say, yeah, that was not so good, right? But equally, getting that feedback helped the agent to learn where we could see where the gaps were. But most people don’t want to teach something that’s going to take a colleague’s job. So we did learn also that, well, what’s the context that actually creates that tissue connection is an interesting one. So we went back to the purpose of the organization, which I think is really important. One of the learnings was it’s not about change and adoption, it’s about creating a movement where an organization embraces a new way of working, where an organization embraces working with the technology in a new way and isn’t afraid to experiment and to grow and to learn. And we brought in a series of people who were really skeptical. We called it an influencer network, to say, well, is this technology actually ready to do this job, and what would it take to get it ready, and are you also learning? So we also had a skill development program with the marketeers to teach them prompt engineering as the base level, which everyone talks about, but also dialoguing with data like you’re getting fed a different set of recommendations, so the ability to like, interact and explore new ways of processing information was actually really interesting.
Peter Cappelli 17:42
So many interesting things there. But maybe just to call out a couple, this idea of deconstructing jobs is, as you know, in the business press, there’s fascination with whether this will eliminate individual jobs or not. And you know, the issue is, individual jobs are full of separate tasks, and AI can take over some of those. Probably those, probably not all of them for any one job, because they’re so different, right? Then you have to recombine and think about those jobs differently. And it takes us to this question of resistance to this kind of change, which probably you saw at some point. What kind of resistance do you face in doing something like this? You’ve probably seen it elsewhere as well.
Karalee Close 18:21
Yeah, I see it in clients all the time, and we faced it internally as well. So the first resistance is, well, what is this thing? Is it gonna take my job-
Peter Cappelli 18:30
…take my job, right.
Karalee Close 18:31
And then it’s like, was it gonna take my colleagues job, or is it gonna change my KPI for my organization? Like, how am I going to be measured? And quite oftentimes, people are like, well, if this takes away half of my team and my reason for being, then what do I do? And we had to set a context for those that are learning AI effectively are going to be the ones that are going to be more marketable and more useful going forward anyway. So you need to be on the journey. You need to be embracing the change is one thing. And then two, you need to be learning how you might work with this technology in new ways, and you need to be innovating so those that actually have the curiosity and courage and connection to be able to make that happen. The other thing is the two way feedback between the agents and the people. So we talked even about, you know, do you have a performance review cycle that goes with agents and people. Do you actually even recognize people’s role as being someone who trains and onboards and develops agents just like you would a trainee. We also went to onboarding practices, which is a little bit interesting. So I went to the HR practices to say, right now, if you don’t do anything different, IT is defining a lot of the onboarding of agents, because it’s a technical practice, and we started exploring, well, if you’re onboarding a trainee, or if you’re onboarding a new employee, you would have a set of things that would sign off that employee to do the work. So we’re looking at the same thing for agents, because you actually need to onboard and train agents, and you need to reward people for their role in providing feedback to the agents and also helping them to learn. And I think that’s a really interesting area, even do you personify some of these agents so they do learn the behavioral aspects of what it takes to work effectively in the organization.
Peter Cappelli 20:21
That’s really interesting, isn’t it? Because it certainly sounds like you should have the agents aware of the values of the organization and the culture and how people respond. And does it get easier to do that if we think of them as human like? Maybe it does. Does it get as creepier for people who are afraid of it? Maybe it does, right? So thinking about that issue is really fascinating. We’re just about out of time, but I did want to ask you the sort of takeaway question for business people here, what kind of advice generally would you give them if they’re trying to do something like this?
Karalee Close 20:55
At the 30,000 foot level, this is an incredibly powerful set of technologies that will change work. You need to start not with the technology, but you need to start with the impact and the value creation ambition of the company or the function. Lead with value. Lead with the strategy. Second one is, unless you’re really redesigning the work, you’re not going to get the impact that you expect. So you know, our estimates are 44% of working hours will change as a result of Gen AI, but unless you’re really redesigning the way that people work with the technology, you’re not going to get the impact. And so most organizations are doing quite a lot there, but only 13% are getting real value at scale. Those 13% that are getting value are 88% more likely to be addressing the human dimensions. So skills, tasks, roles, organization, change, all those kind of things. So it’s actually a people question more than it is a technology question. And then the next thing is really around the maturity of the digital core technologies, as well as the data. You can’t train models on things that don’t exist. So the ability to have, you know, a mature and responsible technology foundation, is also an important thing. And then finally, it’s continuous change, so building new muscles around helping people to embrace the technology as well as change the way that they work with the technology is really important.
Peter Cappelli 22:24
Those are great points, and I think that’s a great way to end. My sense of the conversation with you is that this is about trying to do new things and do things better. It’s not necessarily just about replacing people, for sure. This is a great way to start our season here. So thank you very much for being with us.
Karalee Close 22:41
Thank you for having me.
Peter Cappelli 22:43
So now let’s review a little bit of our conversation with Karalee, and what I think are the takeaways from it. Most certainly you heard the importance of people, which is something we don’t hear very much about in discussions of AI and data science as well. And I think a couple of things came through strongly to me. One was Karalee’s point about the need to think about the overall work process. Build the case for why you’re doing this as an organization. Understand that people and their acceptance are going to be crucial to whether things can work or not. And some of that begins with the fact that you have to take jobs apart. Jobs are made up of lots of different tasks. The AI agents can do some of those tasks, but each task almost requires a different agent. So it’s not going to be the case that you easily replace people. Once you do that, you have to recombine people and tasks into jobs that look different, and how to figure out how to work with those AI agents is going to be the key thing. Maybe the most fascinating part of this conversation was, how do we think about the interaction between the people and the AI agents? Do we try to teach them more as employees, or do we try to think of them as really something quite separate?Because the employees have to help train, they have to help the AI agents learn too. So it’s a really important issue that is getting, in a really practical way, to the heart of the human-agent intersection.
Peter Cappelli 24:17
That brings us to the end of our season premiere. Thank you so much for listening. Please follow us so you don’t miss an episode. Be sure to tune in next time I sit down with Vivian Sun. She is the Senior Director for Data and AI Enterprise Architecture and IT Transformation at Jabil Incorporated. This has been Where AI Works, conversations at the intersection of AI and industry brought to you by the Wharton School in collaboration with Accenture. I’m Peter Cappelli. See you soon.
Episode 2
Guest: Vivian Sun, Senior Director for Data and AI, Enterprise Architecture and IT Transformation, Jabil Incorporated
Transcript:
S3E02 — Start Small, Think Big: Making AI Stick
Vivian Sun 00:01
There are going to be digital employees working next to us, and we are already seeing that they have actually started to identify AI agents as a virtual employee, because just like human beings, the agents that coming to work for a company needs to understand the specific terms and the policies, so we are trying to make them go through the same validation, same training, or same management as a human employee.
Peter Cappelli 00:33
Hello and welcome back to Where AI Works, conversations at the intersection of AI and industry brought to you by the Wharton School in collaboration with Accenture. And I’m your host, Peter Cappelli. I’m a Professor of Management at the Wharton School and Director of our Center for Human Resources. It’s our goal to cut through the noise and deliver insights that are actionable for business leaders. We’re going to do this by combining research with real world case studies. Things are changing fast, so without further ado, let’s dive in.
News Clip One 01:05
AI impacts not just operational processes, but it also impacts the workforce.
News Clip Two 01:10
64% of workers are confident now in their ability to use AI on the job.
News Clip Three 01:16
I think that AI is going to make a big impact on jobs and on workplaces.
News Clip Four 01:21
These tools are incredible. They are so powerful. And if you aren’t adopting or using it, you are falling behind.
Peter Cappelli 01:26
On this episode, I’m excited to get us into the issue of, how do changes when we introduce AI really work. What’s different? What’s the effects on employees? In particular, what we’ve been learning here is, increasingly, that’s kind of where the action is. It’s not the AI per se. It’s this intersection. Joining me for today’s conversation is Vivian Sun, Senior Director for Data and AI, Enterprise Architecture and IT Transformation at Jabil Incorporated. Vivian, thanks for being here, and welcome to Where AI Works.
Vivian Sun 01:59
Thank you, Peter. It’s such an honor to speak in this podcast, especially coming back to the school I studied at, Wharton. So I’m very, very happy and this pleasure to come back.
Peter Cappelli 02:12
Thank you. I was going to point out, but Vivian beat me to it that she’s a Wharton alumnus, but she’s also got three different degrees from the University of Pennsylvania, two engineering degrees as well as the Wharton degree. So the big question, Vivian, is, when you get two degrees, is the third one free?
Vivian Sun 02:27
Actually, yeah, but it was, but it was not paid by- it was not waived. It was actually paid by the company I worked for.
Peter Cappelli 02:37
Oh, very nice.
Vivian Sun 02:38
I might be getting the fourth one too.
Peter Cappelli 02:40
Oh, wonderful.
Vivian Sun 02:40
In the future.
Peter Cappelli 02:41
Well, that’s good to know. We’ll keep the door open. I wanted to just ask you a little bit about your own career, because you are right at the heart of what everything is doing now. Everybody’s talking about this intersection. Did you start out thinking that you would be at this intersection? Or tell me a little about your career, because I know you worked in Hong Kong and other places before landing back here in the US with Jabil. We’ll talk about Jabil in a minute, but tell me about your path.
Vivian Sun 03:08
I actually never imagined I would be working with artificial intelligence, because back in engineering school, there was actually a course called artificial intelligence, but we didn’t know much about it. It was also not a course that a lot of people didn’t take the course. So what led me to where I’m doing now is the transformation characteristic of my career, because even though I worked at different companies and different locations in Asia, in the US, there has always been the notion of managing transformation at different companies. So it naturally leads to where I’m transforming now, which is utilizing AI to improve our business, to bring the business values right. So I have held many different positions. I managed large teams, I managed the big operations, but what always have interested me is managing the change itself, because we know the only thing that’s not going to change is the change itself. So that’s how I started AI, transformed myself and the company, utilizing AI about five years ago, starting from a very first use case until now, where we are scaling AI solutions for Jabil.
Peter Cappelli 04:27
Let’s talk about Jabil for a bit. This is the biggest company that no one has heard of, or, I confess I didn’t know who they were. 140,000 employees, and it’s US based, right? But it’s all over the world now. Is that right?
Vivian Sun 04:39
It’s the largest listed company in Florida, and we have 100 plants worldwide. We have over 400 customers, and we employee has 140,000 people.
Peter Cappelli 04:51
And what you do as a company is also at the heart of what people are talking about today, at least in the US and policy, right? And this is manufacturing. So you’re a company that makes stuff and helps other companies manufacture stuff. Have I got that? Right?
Vivian Sun 05:07
Yep, we’re a contract manufacturing.
Peter Cappelli 05:09
Contract manufacturer, right? So you’re at the heart of all this discussion about making manufacturing back in the US and building it up and all that stuff. So you’re at the heart of this AI discussion in a company that’s at the heart of the manufacturing conversation, so you’re just at the heart of everything. So let’s maybe back you up a little bit you were saying before that five years ago is when this transformation with AI started. What was the transformation around AI that began this path, at least at Jabil?
Vivian Sun 05:40
So in early ages, even before the five years, we started on machine learning, trying to understand more stories from big amount of data. Around five years ago, we feel like everybody’s talking about AI, so we as a group sat down in the conference room trying to define what is our strategy. Yes, we want to embrace AI, but where should we start? So in that meeting, we started to identify three different AI technologies we should be concentrating on. The first one is AI computer vision. The reason we identify AI computer vision is the capability where we want to get into because one it is very widely utilized in the manufacturing environment for inspections and also because it’s more deterministic than other AI technologies. And the second group is machine learning, where we want to understand what the data will bring to us. Help us transform, help us conduct preventive maintenance. And the third area is on generative AI. At the time, it’s more of the big language models that we’re looking at to help us be able to talk to AI utilizing the model.
Peter Cappelli 06:52
Can I back you up on this section? Just on the first one. So machine learning listeners probably know this is kind of data science is statistics for engineers, right where you’re trying to usually look at big data to solve problems which are not well defined, and you’re looking for causation, not necessarily theoretical explanations. The first one was, again, computer vision. So can you just make sure we all know what computer vision means in this context? This is optical scanning stuff. What do we mean by that?
Vivian Sun 07:22
It is optical, but it is using AI models to improve the optical technologies. For example, understanding the color of a product. A traditional optical technology might understand blue versus yellow, but there are subtle differences between more a little bit blue or less blue, right? Human eyes can recognize that, but maybe not traditional optical technology.
Peter Cappelli 07:45
And so where that is useful might be if you are producing stuff that’s supposed to be colored, and you want to make sure that it’s exactly right?
Vivian Sun 07:53
Right, quality control and quality inspections. And we also utilize it in inspecting the cosmetic errors. So the products that we make, we actually hire many inspectors to look at if there are scratches, if there are dirt or dent on the product. And traditionally, we have used humans to do that. Now we are in utilizing AI in many of the cases.
Peter Cappelli 08:19
And as you’re saying this, it strikes me one of the interesting things about manufacturing here is, unlike other areas, trying to use AI, is that you know what the right answer is in manufacturing, right?
Vivian Sun 08:31
Exactly.
Peter Cappelli 08:31
You know the color that you want. You know what the image should look like, and that makes it conceptually simpler, I guess, but you’ve also got to be right.
Vivian Sun 08:39
Exactly. That’s why I said it’s more deterministic, because you know what is not a good product, what is a good product. So that AI computer vision becomes a entry point for us to get into AI, right? Because normally AI, it’s difficult to understand how AI made decisions, but using AI computer vision when we know the results and we don’t have to understand how AI made the decisions, but if the results is good or not.
Peter Cappelli 09:09
So Vivian, can you just walk us through maybe the first execution of this at Jabil when you started with computer vision, what did it look like before and what did it look like after?
Vivian Sun 09:20
So in manufacturing, most of the people hear about is how we produce things. In the process, the visual inspection is a very crucial step. Before the product goes out of the door, every single product has to go through many steps of inspections. Some are testing out the capabilities of the machine, making sure it’s functional. And there is another group of inspections that we need to make sure the product cosmetically is up to the quality. We make sure that the right labels are sticked onto the product in the right arrangement. All of those were done previously by people. Although it’s very important, it’s also very tedious and tiring for people to do. We used to hire inspectors who have to spend 20 seconds on the same product looking at where the defects are. It is not an easy job. Because of that, it’s also very difficult to hire people to come here and inspect those products. So it made a job much more difficult for our HR, especially during the peak seasons. So after we implemented the AI solution, of course, we were able to remove people from those repetitive and non value at work. They were able to then concentrate on jobs that they’d like to do, and also it improves our inspection qualities and saved time for our HR department. Now they don’t have to look for as many inspectors that they used to have.
Peter Cappelli 10:55
And just to point out to our listeners, the psychological process of doing repetitive and boring tasks is called habituation, and that means that you just start to tune it out when you see the same thing over and over and over and over. It’s really, really hard to pay attention. It’s not, as you say, something that humans, we’re very good at. So when you introduced this, and you were saying earlier, this was kind of an entryway into AI. Was this because the leadership and your stakeholders could see the benefits of it? Is that why it was kind of the entry point?
Vivian Sun 11:31
Yes, we were actually implementing solutions using other technologies at the same time, for example, machine learning, for example, big language models. But this area, it was more of the shorter wing for us, comparing to other technologies, because it was so direct impact to our business, so people can see the result immediately, and that brings the confidence of the AI technology. So it was a live case to educate everybody the possibility of AI with this very small capability that AI has, so we were able to gain more attention from our executives, right, even our customers, and so they trust us in continuing the pursuing of AI technology into other manufacturing or other core functional teams.
Peter Cappelli 12:21
Yeah, it’s a great point that this is an organizational challenge. It’s not just a technical challenge. You have to persuade the finance department to give you the money for this. You have to persuade the investors that it’s going to be worth the money. And if you’re waving hands, it’s a harder act to sell. But if they can literally start seeing examples, you’re off to be able to do other things. Can we shift a little and talk about machine learning? And that is what example was, maybe the first of the applications that you folks at Jabil used.
Vivian Sun 12:52
So a very first one, I have to go back to the product color, because we have this one customer is very strict on having the exact color they want. So when we dye the color in the solutions, because of the environmental collection, we have to put the solutions outdoor. And because it’s outdoor, chemicals in there is prone to changes of weather. For example, if it’s raining, the solution composite is different, so we have been using our employees who are very, very familiar with the process, and they can tell, as experts, with what temperature or level of raining, amount of chemical deposits they put into the solution. So what we did is we installed sensor into the solutions and collect the data, and we use the data to analyze the weather information to be able to tell or suggest correct amount of solutions to guide our workers so we can get the exact color.
Peter Cappelli 13:56
So let me make sure I got that. It’s a great point. So we have in many organizations, what our colleague, Sid Winter at Wharton, used to call tacit knowledge. And that is some understanding which is hard to explain to other people. And in your context, here it was understanding how the weather will change the color. What you did is gather as much information as you could about the weather and about the compounds you’re working with, then look for relationships between them, and then at the end, you get an algorithm that says, if the weather is doing this, adjust the color this way. How long did it take for you to figure that out, to get to the point where the algorithm was about as good as your employees?
Vivian Sun 14:40
The project actually lasted for two years, because every color is different, so we can make a better prediction for certain colors, but not all of the colors right. That leads back to my argument about AI is really to make people more intelligent. It’s assisting people, but it cannot replace people in many senses today, so we need to be very conscious, after learning for a long time, understand where we should apply the technology and where we should not right. I think that’s a very important learning that we also obtained from that project.
Peter Cappelli 15:17
It sounds to me like the point is sometimes you could do it with machine learning and other tools, but it might be really expensive and time consuming if you have to build a different model for every color, for example, right? Can you give us another machine learning example?
Vivian Sun 15:34
We are a manufacturing company, so we have hundreds, thousands of parts that’s shifting across the world all the time, and it’s very important for us to understand the export-import policies of every single country, especially in the situation that we have now. So it’s very important for our supply chain people to be able to identify the right code that we should apply to the product. Before we had people, 10s of people, trying to identify that code, and it’s very time consuming, and it’s very prone to mistakes, inconsistencies. So originally, what we did is we used the machine learning and learn about the decisions that people made so we can predict and suggest to our employees what might be a possible code that they can utilize. We spent a long time to do this project. As we’re saying, machine learning actually takes a lot of time because data collection alone, we had come to the point that its accurate enough that our business team decided to accept and make it a copilot of our supply chain people, which is good. But we are in the process of even making it more accurate, right? That’s where we apply generative AI, which is the third pillar that of AI technologies we use here. So in addition to the prediction, we’re also utilizing generative AI to be able to interrogate the documents using a chat bot.
Peter Cappelli 17:06
Tell me if I’ve got this right. So you’re trying to figure out what are the relevant trade policies and approaches. You look at what your employees have decided to do, and you look at the context around it. And so from that, you kind of build a model that say, when the context looks like this, this is to write code. And so you built that, and you can see how that would not just save a lot of time, but would be far more accurate. So now you can move us to the generative AI part from there.
Vivian Sun 17:38
So now, because the technology has advanced. Everybody knows about chatbot, so we have added one more solution on top of our original utilizing machine learning. Even though machine learning can do some good predictions, but we cannot validate, we cannot be absolutely sure that is a right import, export code. So we added on utilizing the generative AI technology to interrogate the document to see what the chat bot would return. So if we get the same answer from the machine learning and also the chat bot, there’s a higher chance that it’s the correct one.
Peter Cappelli 18:17
Good. So this is kind of using the generative AI tools to interrogate. Look at it again, look at the document, see if the algorithm is right or if you’ve missed something. So it’s a check on the algorithm, but it’s also a way to see whether maybe there’s something new in the document or something that we didn’t catch when we originally coded it up. So it’s kind of a quality thing, and it’s an interesting idea, right? That you might get disagreements between machine learning and generative AI, which somebody’s got to step in and decide.
Peter Cappelli 18:52
So if you were to look out, say, two years from now, what things are you working on now in the company that you think within a year or two, we’re going to see something, a different application, a different innovation here.
Vivian Sun 19:05
I think it’s the explosion of AI agents. There are going to be digital employees working next to us, and we are already seeing that, giving an example of Workday, which is a SaaS software, providing HR services, they have actually started to identify AI agents as a virtual employee, because just like human beings, the agents that come into work for a company needs to understand the specific terms and the policies. So by identifying that agent in Workday, we are trying to make them to go through the same validation, same training, or same management as a human employee. I believe that’s going to happen, because if you look at how agentic AI is transforming the world, it’s going to start to take over the decision makings. It’s going to take actions instead of human to replace repetitive actions, being able to communicate with people internally and externally. And it’s going to be everywhere, in every single industry, in our work, in our life. So because of the impact it brings to business, there will be a huge explosion.
Peter Cappelli 20:21
Just so our listeners are clear what we mean here. When we’re talking about agents, we’re not talking about robots sitting at desks. We’re talking about large language tools or generative AI tools that do specific things that we can plug into the work stream. It doesn’t necessarily even take the employee out of the chair that they’re in now, but it takes over some of those tasks, and the great trick will be to figure out which agents you can insert in which jobs in order to make the work maybe even easier, but more accurate and perform better. Yeah. And a last question, Vivian, since you’ve been through this process in a company where you’re using it a lot already, if you had advice for other companies that are just getting started on this, what would you tell them from your own experience?
Vivian Sun 21:09
Yeah, I have a few suggestions. I think number one suggestion is to start from the business value, instead of the technology. We’re trying to use the technology to solve a problem. We’re not trying to build a use case, because we want to use the technology. And it is also very important to keep track of the value, because as we iteratively build or train an AI model, there will be many factors that could change, like continuously making sure we keep up the value would be able to guarantee that at the end, the AI solution will meet the expectation of value generation. That’s one. Start small, but think big. I talked about starting small and prioritize lower hanging fruits so we can prove the value to the organization, our employees and then the customers, but don’t always stay there. Design the projects in with smaller steps, but eventually will lead to a bigger goal. That’s my second suggestion. And also transformation. It is no different from other transformation or revolutions. We need to communicate well. We need to mind about organizational change management. We need to train our people. We need to mind about responsible AI, how it’s impacting the society. Just like how model three was very initially introduced to the world, we had to design how the light how the traffic light works, what are the traffic laws that we should be designing. So don’t only concentrate on the technology, but look at this whole initiative, whole revolution, as a transformation. There is a lot of other factors we always have to keep in mind and build. So that’s also very, very important, and we have to do it at the beginning.
Peter Cappelli 22:59
Those are great points, and I think it’s a terrific discussion about organizational change as well, and how to use little AI to get to bigger AI and to work through the transformation of work in the process. Vivian, thank you very much for being with us today.
Vivian Sun 23:15
Thank you, Peter.
Peter Cappelli 23:18
So let’s review some of the big takeaways from our conversation with Vivian. I thought the most important thing was her discussion about how small wins in ways that people could easily see build the case for doing bigger things. I thought it was also important to hear how sometimes maybe you don’t need to automate everything that employees can respond quicker to things which are changing all the time. But thinking about this as a transformation that has to be managed, and not just as tools that we’re popping in to the workplace is really an important thing.
Peter Cappelli 23:56
So that’s all for today. Thanks so much for listening and being with us. Follow us so you don’t miss an episode. Next time, I’ll be sitting down with Greg Ulrich, the Chief AI and Data Officer at Mastercard. This has been Where AI Works conversations at the intersection of AI and industry, brought to you by the Wharton School in collaboration with Accenture. I’m Peter Cappelli. See you again soon.
Episode 3
Guest: Greg Ulrich, Chief AI and Data Officer, Mastercard
Transcript:
S3E03 — Trust and Transactions: How Mastercard Is Doing AI Differently
Greg Ulrich 00:02
The challenge we have is we’ve spent a lot of time trying to prevent bots from making transactions because we’ve seen them as malicious. We’ve seen them as likely fraudulent actors. So we need to create transparency in this ecosystem that there’s another party that’s transacting, and make sure that’s done with the same transparency and security and trust as you expect when you’re buying online today.
Peter Cappelli 00:29
Hello and welcome back to Where AI Works, conversations at the intersection of AI and industry brought to you by the Wharton School in collaboration with Accenture. I’m your host, Peter Cappelli. I’m a Professor of Management at the Wharton School and Director of our Center for Human Resources. It’s our goal here to cut through the noise, to deliver actionable insights for business leaders, combining cutting edge research and real world case studies. We’re doing the case studies today. Things are changing fast, so let’s dive in.
Peter Cappelli 01:01
On this episode, I’m excited to explore the world of using AI agents, inserting them into some customer service work, the kind of thing that we thought was going to happen a lot with AI, and we don’t have very many examples, so today, we’re going to jump in on an example and see what is going on with it. And joining me for today’s conversation is Greg Ulrich. He’s the chief AI and Data Officer at Mastercard. Greg, thanks so much for being here. Welcome to Where AI Works.
Greg Ulrich 01:31
Thank you so much for having me, Peter.
Peter Cappelli 01:33
So first question is, how did you get to where you are now? Were you a data science guy who became a manager? Were you a manager who became a data science guy or none of the above.
Greg Ulrich 01:45
A little bit of both. I originally came from Wharton, for business school, so it’s great to be here with you.
Peter Cappelli 01:50
Were you a student of mine? Did you get an A I hope?
Greg Ulrich 01:52
I was not a student of yours, unfortunately, I missed you.
Peter Cappelli 01:55
I’m safe. Okay, good.
Greg Ulrich 01:57
I did consulting sort of before and after, but afterwards, I focused in the nonprofit sector. I was really intrigued by why certain interventions in the social sector worked and some didn’t. And that got me into data analytics from there, and I joined a company called APT. And long story short, Mastercard acquired that company 10 years ago.
Peter Cappelli 02:15
Was this an intention on the part of Mastercard to pull the talent out of your of your organization, was that partly what they wanted?
Greg Ulrich 02:23
I think all of our acquisitions have some degree of talent acquisition, but the primary goal, maybe I’ll give you just two seconds on Mastercard’s business model, how it’s evolved, and why a data analytics company fits into this ecosystem. Mastercard is known as a global payments network, but we’re not just trying to process transactions, we’re trying to add security and intelligence to those transactions. And the business, you could think almost as a virtuous flywheel. The larger our network gets, the more data we get from that network, the more we’re able to scale services on top of that. Fraud scoring cybersecurity insights to help our customers, and therefore we make that the ecosystem safer, more secure, more intelligent. The more value we add to our customers with those services, it helps us win share and grow our payments network. The more that happens, the more data and scale we get for those services. And the business has really evolved that way. Our Services Division, which if you can think of as the insights, the fraud scoring, open finance, things around customer engagement and marketing and loyalty, is now 38% of the business. It’s an $11 billion business, and it’s been growing much more rapidly than the core of the business. And it’s enabling us to add a lot more value to the entire ecosystem and to our core. And so as we went down this path, Mastercard went down this path, they realized they needed to do a lot of acquisitions to bolster these capabilities. Some around data analytics, that’s where APT came in. Some around cybersecurity and threat intelligence. And the most recent acquisition we made, a company called Recorded Future, was done about a year ago, and is in the threat intelligence space, really trying to get additional information so we can provide insights to help people understand where threats are coming from before they’re on their doorstep.
Peter Cappelli 04:06
It’s a really interesting story to hear about how AI, broadly defined in data analytics, generates new business for you and new opportunities. That’s an interesting thing. We want to talk about one of the things that you’ve done in particular, and that is to put agents — and by that we mean large language model type of AI agents — to work on the customer facing sort of side, and the tool that you developed. And so if you could tell us a little about what it looked like before you introduced your tool, how did work get done?
Greg Ulrich 04:38
We think about AI deployment at Mastercard in four ways. One of which is around, how we just help our company get stronger, how we help it get better. And then three of the applications are external for our customers, how we make commerce safer, but on where agents come in and where we’re trying to deploy this is in a few places. One of those is in customer service. And one of the things that we notice that we could use some help with was how our customers onboard our products, how they bring it into their ecosystem. It is a lot of technical manuals, they’re hundreds of pages long. We have a support team that provides support to our customers to enable this. And what we did was we created agents that were trained on all the Q and A back and forth, all these technical documents, and it’s really an agent that enables our customer support agent to give much faster and better support to the end customer.
Peter Cappelli 05:30
So let’s ask you to drill down on one of these that I guess you are responsible for, this agent pay thing. And how did that work before agent pay was introduced? So how did the money flow? What did it look like before?
Greg Ulrich 05:43
Sure, this whole ecosystem of agentic commerce is incredibly new. It’s happening in the moment. The phases that one goes through on this it starts with using ChatGPT, or whatever my favorite, to do research on running shoes that I might want to buy. And I’m getting a different interaction mechanism than I’m getting in traditional search. I’m not going to the Hoka website, I’m not going to the Asics website. I’m getting all of this in a recommendation in one spot, a natural manifestation of that is we’re seeing more and more people move search to these AI agents, it’s actually execute a transaction in that ecosystem, enabling a consumer a small business, to execute in that same workflow. I’ve decided I need stability issues. I’ve decided I like this brand the best. I’ve had this back and forth interaction that’s faster and gives me better information than I had previously. And now I know I want the Hoka Bondi in size 11. I want to click a button and say, buy that for me. And that’s the next step where these agents also want to go. The challenge we have is that’s a new player in the ecosystem, and a lot of the things that we’ve developed, I talked about how we use AI for all these fraud prevention, we’ve spent a lot of time trying to prevent bots from making transactions because we’ve seen them as malicious. We’ve seen them as likely fraudulent actors. So we need to create transparency in this ecosystem that there’s another party that’s transacting, and make sure that’s done with the same transparency and security and trust as you expect when you’re buying online today. And so what we’re doing is trying to put the tools in place to do that by basically certifying who these agents are. So they become a part of the Mastercard network, we know that they’re a legitimate actor, and we can pass information back and forth to the merchant, to the issuer, that we can certify the user, the individual, Peter is associated with this agent, that you’re a real authenticated user. So we can create this audit trail in case challenges come in in the future, and that we create the security and tokenize the transaction to make sure we can pass the right information along this chain in a way that’s safe and secure.
Peter Cappelli 07:48
Really relevant for me, because I just did this exercise in trying to buy shoes, and I ended up I was actually placing an order to some place in China that I was not aware of. It’s exactly the problem you’re describing, right? This was a transaction that was actually fraudulent, and I couldn’t really tell, and it also had connections to credit cards that looked legitimate. So that’s the kind of problem you’re trying to deal with here.
Greg Ulrich 08:11
We’re trying to prevent the fraud in the first case, and then, like we have with any transaction, be it agentic or not, we’re trying to make sure that there’s a clear, transparent, and simple process, so the consumer stays protected in this ecosystem. If you buy something that you weren’t responsible for, there’s a dispute resolution process. The consumer is protected. And that happens because we can see that, hey, you were trying to buy and actually, the wrong thing was shipped, and there was a there was a mistake, and we can adjudicate that process through dispute resolution. And we need that same audit trail to happen so we can make sure that in the case something goes wrong, we have the information through the ecosystem to handle that in agentic commerce as well.
Peter Cappelli 08:55
Before you guys went to this or before your colleagues probably elsewhere were trying to do similar things, did it just not work? Or did they have checks in place to try to prevent fraud? I mean, what happened before?
Greg Ulrich 09:07
So this is an interesting example, because you haven’t traditionally seen a lot of the actual execution of, I want to be able to buy these [unintelligible] through these agents. Perplexity was the first one that did this, and if you’re signed up with perplexity, they have payment information for you. And they were sort of, it’s almost a manual process of saying, okay well, if you want to buy this, we’ll go through and enter your information into the ecosystem. But it’s a little bit of a cumbersome process, and they’re trying to make sure that there’s a market here. As this comes to scale, you want that on an automated basis, and that’s the technology we’re trying to enable. But also without this, the challenge you will have is the authorization rate on these transactions will be lower than people expect, because there’ll be too many transactions that won’t be recognized where it’s coming from. They’ll be flagged as fraudulent or mysterious when they shouldn’t be. And by providing that transparency, you might have a much higher hit rate of transactions coming through. Without that transparency, there’s too many challenges in that ecosystem, and the ecosystem doesn’t work. The beauty of paying anywhere with a credit card is it’s very high secure, but also a high throughput system. It works. When you tap your card, it works, when you swipe your card, it works. And you always have to balance fraud and the convenience. So that’s what we’re doing here as well, and without the information, because we’re worried about fraud, I think the convenience would go down for all the parties in the ecosystem.
Peter Cappelli 10:31
So just to illustrate this one, the problem is, because I’ve certainly had this, particularly recently, if you travel far, I was in Europe, you know, and your credit card is turned down, which is kind of embarrassing, and you can get irritated at your credit card company, but it could be any part in that chain where they decide to block it because they’re not sure, right?
Greg Ulrich 10:50
That’s right.
Peter Cappelli 10:51
And so the goal for you folks is to do in real time enough checking and certification that we don’t have false negatives, right?
Greg Ulrich 11:01
That’s right.
Peter Cappelli 11:01
And we don’t get ripped off either.
Peter Cappelli 11:03
The amount of data that you use now to make those scores, give us a sense of how big that is. I mean, how many variables are you looking at?
Greg Ulrich 11:03
Sure. So we switch 159 billion transactions a year for the decision intelligence product I was talking about. That’s the one that gives a score using the information we have as well as these new merchant graphs. There’s a trillion data points, effectively, that’s going into these models that are helping make these decisions.
Peter Cappelli 11:03
Wow, big numbers.
Greg Ulrich 11:03
Big numbers.
Greg Ulrich 11:03
That’s right. Every transaction that comes through our network gets a score. It can score from 1 to 999, based on how likely we think that this is legitimate or fraudulent. And that’s based on AI that we’ve developed over the years, and based on transaction history and what we see happening in the network. You take GenAI, I’m shifting a little bit from agentic commerce, but to another way in which GenAI is enabling this challenge that you have now, we can use GenAI to bring in a lot more information than we could before, right? We can use an edge inference graph and look at all the different merchants in the ecosystem, even if you haven’t shopped at them, to understand how likely it would be that you would have shopped at these different merchants. So even if you would show up at something that you’ve never been there before in our current model, that would reduce your score. But if we see it’s actually pretty likely that other people that have the same shopping history as Peter and shop at these types of retailers actually also shop there, it’s much more likely that that’s a legitimate transaction that’ll help increase your score, so we provide a score through to the bank. The bank is the one that makes the decision ultimately on whether or not that transaction goes up or down. But we’re trying to provide the best possible information, and GenAI is enabling us to bring in much more information than we could previously, to make sure we don’t get those inappropriate declines, and make sure that when there is something inappropriate that we actually are able to stop it.
Peter Cappelli 11:56
So that’s that’s fascinating in itself. It’s presumably something you’ve always been trying to do, and it’s just in the last few years, your ability to incorporate so much more data and generate a single number from it, rather than a series of numbers from each individual measure, which is probably what at some point they were trying to do, is quite something, right?
Greg Ulrich 13:10
Yeah, using AI machine learning, not the GenAI capabilities that have come out in the last several years, but traditional machine learning on this has been something that’s been a part of this for over two decades. And we were producing a score, a single score out of that, but we’re now, with Gen AI, able to bring in a lot more information, increase the efficacy of some of these solutions, and produce new solutions for customers that we couldn’t without this new technology. So we were in a good spot, because we had such a deep bench of expertise and capabilities of using data and using AI, GenAI now brings additional capabilities that allows you to sort of turbo charge that in many ways.
Peter Cappelli 13:48
So working us toward the agent pay tool, maybe you could walk us through the problems of trying to get that thing to work. Some of them are not technical problems. Some of them are kind of maybe managerial problems. Maybe start with the managerial ones. You got these different stakeholders. Presumably you all have to sort of sign up on this. What would that look like? That problem?
Greg Ulrich 14:09
This is an example where I think our organization came together very, very quickly and coalesced on a pretty clear answer about how we were going to enable this. We’re fortunate that we had some of the technology in place that we thought would enable these to happen, and the immediate pain points around the transparency and the fraud on the transactions were pretty clear.
Peter Cappelli 14:30
When you get to the stakeholders that once you get outside of Mastercard, were there any particular challenges there in terms of rolling this out?
Greg Ulrich 14:38
On the fundamental technology, I think we’re having very productive and good discussions. I do think that there will continue to be discussions around different parties, because you’re basically adding another party into the flow. Instead of just a consumer, a bank, the acquirer for the merchant, and the merchant, you now have an agent in the middle. And I think the questions around. How economics are distributed with another party, how liability are distributed with another party, how visibility and ownership of the consumer exists with another party are still things that people have concerns about and we’re working through.
Peter Cappelli 15:12
Yeah, so let’s talk about that a little bit. So Mastercard was already there in the middle, but now you’re making judgments that are more sophisticated and you’re passing along more information than you were passing along before. Is it different in any way for the users, or is it sort of for them, transparent?
Greg Ulrich 15:31
So for a consumer, our goal was to make this seem just as it would for any other transaction you’re making online. You click buy, you’re going to see a checkout page show up. It’s just going to end up being in the AI agent, as opposed to on your retailer site. But we’re trying to make it so for the consumer, it’s a simple experience, it has the same security, but also it’s the same acknowledgement. You’ll see, if it’s a Mastercard transaction, the trust marks and the branding that are associated with, okay, I know that’s my card. I know this is trusted. I could see parties that I recognize in this ecosystem. So this is an easy and simple experience for the consumer. I mean, that’s what everyone in this ecosystem wants to create. That’s why the agents are doing this. We have people that are coming to us and doing all this work. The best thing for a consumer, the best thing for Peter here, now that he’s figured out what running shoes is, just buy them. Why are we then sending them somewhere else, and why are we breaking up the flow? Why are we creating work for Peter on this, we want to make this simple and easy.
Peter Cappelli 16:29
It sounds like there’s a lot involved in trying to develop this agent pay inside Mastercard. Do you think of it as one thing, or is it a whole series of little decisions and a bunch of agents. How do you look at this thing?
Greg Ulrich 16:44
It’s a little bit different from some of the other things that we’ve developed. So I’ll answer your question with a little bit of context before it. There are a whole host of ways in which we’re deploying AI agents to help solve different problems. Most of them exist sort of within a single vertical or area of our company. So we have 1000s of consultants in the enterprise. We’ve created a series of agents that enable them to access material, summarize research, do additional research, all within the context that you could break their job out into the component pieces. And we’ve created different AI agents to enable different elements, and we’ve done some, and we’re going down the list and continuing to add more information and improve life. But it’s contained in the ecosystem, if you will, of the consulting workflow. We’ve done the same for our franchise. We have violations around our franchise, so we’ve created a bunch of agents. Some are checking the internet for information, some of which are looking at transaction data to figure out if there’s a merchant that is violating rules or is miscoded for some reason and therefore not getting the appropriate economics. It’s a series of agents, but again, contained within the franchise group. Agent pay is interesting because it’s bringing in tokenization technology that is part of our core payments. It’s bringing in the ability for us to capture consent, which is part of our developer experience, we’re creating an intent API so we understand what the consumer is asking for and making sure we’re collecting that information. It creates the framework for disputes which sits in our security group around, do we have the right post-purchase experience, and can we enable that? So unlike a lot of the other applications, it really touches on all the different elements in the company, all the different services and the different elements of our core technology to come together to create a seamless experience. And so, whereas most of these have a pretty defined working group within an area, this is a much broader team across the enterprise.
Peter Cappelli 18:41
It is a big challenge, and you guys have pulled it off, I think, right?
Greg Ulrich 18:45
Yeah, well, it’s not a finish line, and you run 26.2 miles, and then you get to be done. This is going to be a space that evolves quite a lot. Right now, we’re talking about this example with the running shoe. A more complicated one is you say, hey I like this running shoe, but you know what? Don’t need them right now. If it goes on sale for under $99 in the next three months, then buy it, as long as there’s free shipping and free returns. And when you get into planning to travel, okay, book that, but you have to go to airline X here, and hotel Y here, and car rental Z here, and restaurants here, and all of a sudden, the hotel doesn’t have availability. And does that change it? Like the complexity of managing this and why that intent matters and the information flow matters so much, this ecosystem will continue to evolve, I believe, and it won’t just be for consumers, it will be in the B2B context as well. And so there’s a lot more that we have in front of us to continue to innovate to make sure we address the challenges in front of us.
Peter Cappelli 19:43
Greg Ulrich is the head of Data Science and AI stuff for Mastercard, and ran a really interesting project here. Greg, thanks very much.
Greg Ulrich 19:52
Thank you so much for having me.
Peter Cappelli 19:55
Let’s review some key takeaways from my conversation with Greg here. I think what’s interesting and important about this conversation is it really illustrates the intersection between management issues and AI technology. And in Greg’s story, the most important thing to me was the ability of Mastercard to get all these different groups, that may have sat at one point in different silos, to work together on a big collective AI tool that was going to change in a fundamental way, how business is done. How do you get everybody to cooperate and be aligned on a mission? Well, that is a big task. It’s not about AI per se, but it is about things as he was describing: leadership, organizational culture, and priorities. The fact that everybody understands this is a big deal for the organization. That’s what makes it work. It is a good reminder, I think, a lesson for other organizations. If you’re going to take on something big in AI and that is complicated, you know, it’s going to span organizational issues, you better start with the management problems first, because otherwise they could block the success of your effort, and then you end up with with nothing.
Peter Cappelli 21:06
So that’s all for today. Thanks very much for listening. Please follow us so you don’t miss a thing, and be sure to tune in next time for our final regular episode of the season. This has been Where AI Works, conversations at the intersection of AI and industry, brought to you by the Wharton School in collaboration with Accenture. I’m Peter Cappelli, see you again soon.
Intersection of AI and industry brought to you by the Wharton School in collaboration with Accenture. I’m Peter Cappelli. See you soon.
Episode 4
Guest: Sohaib Perwaiz, Group Business Engagement Operation Lead, RBC Borealis (Royal Bank of Canada)
Transcript:
S3E04 — From Call Volume to Client Value: AI in the Banking Sector
Sohaib Perwaiz 00:02
Being one of the largest financial institutions in the world, we have a lot of data. We have about 17 million clients, which means that every day we get a lot of transactions where they tell us a lot about our clients. What the advisors were looking for is an asset we inherently have. We just try and figure out a way to surface it in a way that the advisors would actually use that kind of knowledge. And we were doing that in our other products, where we were leveraging all this transaction data making more informed decisions about the right products for a client, but we hadn’t really cracked the nut on how do we surface this for the advisor so they can have a better conversation with the client?
Peter Cappelli 00:28
Hello and welcome back to Where AI Works, conversations at the intersection of AI and industry brought to you by the Wharton School in collaboration with Accenture. I’m your host, Peter Cappelli, Professor of Management at the Wharton School and Director of the Center for Human Resources here. It’s our goal to cut through in these podcasts the noise to deliver actionable insights for business leaders by combining cutting edge research with real world case studies. As the great Elvis Presley said in a different context, a little less talk and little more action. We’re going to try to get practical here very quickly. Things are changing fast. So let’s dive in.
Peter Cappelli 01:08
In this episode, I’m excited to explore the use of artificial intelligence in financial services, particularly in banking, really deep into white collar work. One of the things I’ve learned in our own research about AI is that there’s an enormous amount of conceptual discussion about how AI could be used, and there’s very little really practical stories or examples about how it’s actually being used. And so that’s we want to talk about today in the financial sector, and joining me for today’s conversation is Sohaib Perewaiz. He’s the lead for the group business engagement operation of Borealis, which is part of Royal Bank of Canada, that is the center of excellence for introducing AI. Thanks so much for being here. And welcome to Where AI Works.
Sohaib Perwaiz 01:58
Thank you for having me.
Peter Cappelli 01:59
He also reminds me he was a former student of mine here at the Wharton School. Before that, he had degrees at Harvard and Princeton. And he also reminded me the I gave him a good grade in the class, which I’m sure he deserves. So, when you left Wharton, just want to know, and I think we’re trying to ask everybody this question, how you ended up in this topic, which was something that didn’t exist. This, this function that you’re in now didn’t exist when you left Wharton. So from here to McKinsey, and then from McKinsey, I guess, to RBC. But how did you end up in this AI area?
Sohaib Perwaiz 02:33
Yeah, it was actually quite coincidental. I left Wharton thinking I’d be a management consultant, which is why I ended up at McKinsey and at McKinsey, I just happened to find my home in banking. So I got really interested in the banking industry, and worked across the globe with McKinsey on banking topics. I also got really interested in business building, like how to build new things that can change banking. And that’s where I started getting exposure to AI through my clients, actually back then. And what I was observing was that a lot of our clients, and perhaps, you know, this inspired the topic of this podcast, treated AI as a shiny new thing that was for marketing and branding. I wasn’t necessarily an expert in AI, but I was more and more keen on seeing how AI was actually being turned into value and dollars in banking. And that’s kind of what drew me to RBC Borealis, and then also AI. It’s more the interest of like how banking is changing and the role AI is playing, and RBC seemed to be a place where they’re taking value to heart and trying to do real things with AI, as opposed to just marketing and branding tactics.
Peter Cappelli 03:32
Do you think it helped to be not near Silicon Valley? I mean, Canada’s got its own tech world. But did it help to be a little step removed and not so caught up in the shiny stuff.
Sohaib Perwaiz 03:42
I think it did help. I think it helped because I approached the problems that we’re trying to solve with AI, not just purely from the lens of cool technology that could be deployed and finding a place to deploy it, but more from the core problems banks were facing that I saw as a consultant serving those clients, but then also being a client of banks, the problems that I had to deal with as a client. So to me, it helped, because I didn’t take the lens of what’s the new cool tech I can bring to this problem. It was like, how do I just solve these day to day business problems that exist in this industry, for the banks and for their clients?
Peter Cappelli 04:10
I think that’s also a core finding from the podcast so far, is that if you start with a problem in search of a solution, rather than starting with the solution, looking at places you can apply it, you’re going to do much better. And it sounds like you have so let’s see if I could get you to tell us about something that you fixed at the bank using AI?
Sohaib Perwaiz 04:28
Yeah, for sure. I’ll get very specific, but just maybe to give you a bit of a context, we are doing AI across the bank. We have a program that covers almost every part of the bank. We touch frontline productivity. We do general employee productivity, we do pricing, we do everything. I’ll zoom in on an example I think that’s just really easy to make tangible, which is in our advice center. So RBC has an advice center that’s one of our major channels, where clients can call us with questions, concerns, to get a new product, to change their product, et cetera. We have about 4000 colleagues that work in our advice center. So, starting with the pandemic, but more recently, with our acquisition of HSBC bank in Canada, we were finding that the volume of calls coming into our advice centers was decreasing dramatically. So clients, perhaps because their preference for calling in rather than going into the branch or using online tools, we just saw volumes of call increase, and the complexity of those questions that clients were asking our advisors increase. That was compounded by the fact that we generally have a high attrition and turnover within our advice centers. So on the one hand, we’re seeing increasing volume, increasing call complexity. On the other hand, we have to keep reskilling and upskilling our advisors, because they’re just naturally changing. So we got to a point where, and this is, I guess, the crux of the business problem is the status quo became unsustainable, and we had tried the traditional digital tools. We had digitized a lot of our workflows. We had digitized a lot of like our policies and procedure libraries that often our advisors go to to answer client questions. But none of that was able to like surmount the wave that was coming at us in terms of the call volume and complexity. So that was the business problem. Where we started was, how do we empower our advisors, or give them more skills and tools so that a they can do more calls, but also that those calls aren’t just transactional question-answer or sell you a product kind of type of conversations. How do we enrich those conversations with clients so advisors also feel more excited and engaged about their day-to-day and the conversations they’re having with our clients? So on the one hand, we wanted to reduce the time it takes to answer a client question or engage with the client over these calls. On the other hand, we just wanted to make it a better experience for both the client and the advisor. So we started with the basic stuff. When a client calls, what is happening most of the time in that call? So I say, like, I have missed a credit card payment, what are my options? Or I have lost my credit card, what do I do next? And we found that 80% of the time in that call, our advisors are actually like searching for the answer through our policies and procedure libraries, because often in banking, as you can imagine, heavily regulated industry, lots of rules to follow. Our advisors want to make sure they give the client the right answer, but that process was taking a lot of time, so we thought, okay, like, let’s start there. Let’s start making that part easier, because that seems to be taking a lot of time. So that’s where we started our first application of generative AI in our advice center. And this was about almost two years ago now, and where we started with like, an LLM that could actually just help answer those questions in natural language with an advisor. So rather than the like, if you call and say, I’ve missed a credit card payment, rather than the advisor going to like, search for, like, the exact policy or procedure that would help address that client concern, it’s almost like a Google search bar where they could say that the client has missed a credit card payment. They missed it yesterday. They have this kind of card. How can I help them? And then the response that LLM would give back was: Okay, here are the three steps you should tell the client, oh, by the way, clients generally like to hear that it’s all going to be okay as well. So maybe, like, throw that into the conversation. So that was where we started, and we found that started having a huge impact in terms of call time. We started seeing that go down, and we started seeing that within like, six or seven months of deploying the solution. So we started scaling this across our advice center. Then we started honing in on the other problem. How do we enrich this conversation? So it’s not just about the client saying I lost my card or I missed a payment. What do I do next?
Peter Cappelli 07:51
So can I pause you just a second to make sure we’ve got this? So this is not a chat bot. This is helping the agent, the human, give quicker answers. And then from there, we’re going to talk about expanding what they can say. So it’s assisting the agent. It’s not replacing the employee?
Sohaib Perwaiz 08:07
Definitely we’re getting feedback from the agents. We collect active and market feedback from our clients as well, and we actively monitor that, but we have some inherent guardrails on that too. Our advisors can only know the standard information that our clients know our advisors have. So when a client authenticates, they’ve already given us permission to know what their account numbers are, what account holdings they have, so that’s the information that the client, the advisor can see directly. All the other insights are not super specific enough, I would say, to be creepy. It’s generalized enough, and this is what at least we’re seeing from the feedback we’re getting, that it is not creepy. So it’s not as if I’m going to say, like, unless you tell me that you missed a payment, I’m not going to be the one who tells you you’ve missed a payment, and in fact, our advisors won’t be able to see that until the client tells them that it doesn’t come off as creepy because it’s based on data that the client knows we have and that they’ve also given us permission to surface for the advisor.
Sohaib Perwaiz 08:07
100%. And then the second part was, how do we enrich the conversation? We really went out there and listened to the advisors and said, what’s your ideal conversation with the client? And we heard consistently that it’s like a conversation where they understand the client. They can get to know the client better. They can go deeper into the problem that the client is bringing to them. So our advisors were saying, we just want to better serve the client and understand the client better. Now the good news is, at RBC being one of the largest financial institutions in the world, we have a lot of data. We have about 17 million clients across North America and the world, which means that every day we get a lot of transactions where they tell us a lot about our clients. And if you’ve been a client with RBC for, let’s say, even half a year, or, like, even a couple of months, we can learn a lot about you through how you interact with us. So what the advisors were looking for is asset we inherently have. We just try and figure out a way to surface it in a way that the advisors could actually use that kind of knowledge. And we were doing that in our other products, where we were leveraging all this transaction data making more informed decisions about the right products for a client, but we hadn’t really cracked the nut on how do we surface this for the advisor so they could have a better conversation with the client. So what we started doing was, as soon as a client calls, we recognize who the client is based on authentication they do with the advisor, and then immediately, on one screen, they can actually search for our policies, procedure library, to the tool I just mentioned. The second screen is basically a client profile. This is what the client is. This is how long they’ve been with RBC. This is the products they hold, et cetera, et cetera. That this client has been with RBC for three years, they have a first home savings account, they’re probably thinking about getting a mortgage. You may want to ask them about if they’re interested, and if they are, you may want to send them to our mortgage specialist. Or this client has been saving a lot in their savings account, they should maybe consider a GIC or a term deposit where they can make more interest. The innovation here was just surfacing this for the advisor so they could engage with and interact with it and have that deeper dialog with our clients while they were on that call.
Peter Cappelli 09:40
I imagine one of the things you have to think about a little bit is how much information do you want to give the agents to pass on to the client and yet not make the client feel creeped out about this. Yeah, like, like, I see you missed a payment. Did you have to market test this or something to figure out, you know, what the boundaries would be, what’s too much, what’s not enough? The agents were helping you with this I guess?
Peter Cappelli 11:03
Do you see a next step in this? I mean, one could imagine, for example, the large language model could suggest ways to phrase the conversation, and they could draw on some marketing tools and things. Do you want to go down that path? Or you think what you’ve done already is enough, and let’s leave the human dimension here?
Sohaib Perwaiz 11:21
I think the human dimension will stay just because we believe it’s important for the client relationship. We are definitely open and are exploring how we take this to the next level. We actually have a training tool now that we’re using with generative AI to like coach our advisors on how they can have more engaging dialog with our clients answer certain kinds of questions. So we’re certainly open to that. We’re just very hesitant to take this kind of technology to market, up in front of clients until we’ve gone through rigorous testing.
Peter Cappelli 11:44
Do you think, I mean, there are people who would say, boy well this is great, just dial it up a little bit further, make an enhanced chat bot and get rid of the agents altogether. I’m sure somebody’s thought about that approach, but you guys aren’t going in that direction. Any sense as to why not?
Sohaib Perwaiz 12:00
Our hypothesis and the industry may, like, prove us wrong right now is that the human connection is important for the client. The human layer actually adds a bunch of value that purely AI couldn’t and again, like there’s a world in which we can have, like, an agentic AI system that can orchestrate it all. But where we are right now, where our clients are right now, we feel like the human connection, the human overlay, is actually very important for establishing that relationship.
Peter Cappelli 12:22
But even given this and you’re going to keep the agents, it sounds like productivity, in terms of calls handled, is probably gone up, because they can do all this stuff faster, right?
Sohaib Perwaiz 12:31
100%. And that’s not to say that we’re never going to change how we’re structured, how our workforce is structured. Our intent right now was upskilling so they can have more engaged dialog with our clients. Those are the business problems we’re trying to solve today, and that didn’t require a massive overhaul of our workforce.
Peter Cappelli 12:47
Just on that one what’s the reaction to the agents when you rolled this out? Do they, were they nervous about it? Were they happy once they got it? What was that process like?
Sohaib Perwaiz 12:56
It’s been very positive. And there are a couple of reasons like why we think that was. One, because of the pervasiveness of tools like GPT as a consumer product in the market, a lot of folks know how to engage with LLMs. So like, the usability was very high. So like, our tool that enables our advisors to, like, go in and like, search for answers. It’s the kind of tooling they’re already using at home with ChatGPT, or Perplexity, or the other tools out there. The second part of it was it was it actually elevated the type of conversations they were having with our, with our clients. So like our client engagement went up, our NPS went up. I feel like our advisors were experiencing that more positive client reaction firsthand, which was also just, I guess, making that experience more rewarding for them.
Peter Cappelli 13:35
That’s a good outcome. Shall we switch now and ask you to tell us about another one. And we had our druthers, maybe something different. Yeah. What else have you done that you think would be an interesting variation here?
Sohaib Perwaiz 13:48
It’s definitely something different, and it’s in our mortgage business. And the technology is also quite different, because it’s more in the background than like, you know what I just described as something like our advisors can touch and feel and work with. The use case I’m going to walk you through now is a bit more behind the scenes, so to speak. So maybe just to give you a bit of context, RBC is obviously one of the biggest banks in Canada. Also we have one of the biggest mortgage businesses in Canada. It’s very important for us as a business, but it’s also very important for our clients, because home buying, as you can imagine, is one of the major anchors of a relationship any client has with their bank. Historically, we’ve done pretty well on our mortgage business, our retention rates. So that’s when a client mortgage is coming up for renewal. And by the way, maybe just for context, since most of the listeners may be American, the American mortgage market is unique where you have thirty year mortgages. In Canada, our mortgages are usually three to five years. Which means for banks, renewals are a big deal. Getting new mortgages on the book is important, but it’s almost equally important that we keep those mortgages on the book when they come for renewal. RBC has a pretty good track record of doing that historically. Like 90% ish plus minus retention rates. A couple of years ago, we started to see a bit of a change there, where our renewal rates weren’t as positive as they used to be, and we started to see a bit of a dip. Generally, how it has worked in the mortgage industry in Canada is once you hit 180 days before renew, you get a letter from your bank saying you’re up for renewal, and then different banks have different strategies. They’ll try to reach out to you throughout that process. And the hope is, before that 180 day window runs out, you renew with that bank. If you opt not to renew, obviously you will go to another bank. But our goal is within that 180 day window to get a client to renew. We started seeing that our retention rate started to go down a few years ago, A, because the market became very competitive in terms of pricing. So the housing market was, I think, across North America, but particularly in Canada, pretty hot. Lenders were very aggressive with pricing, and lenders were also very aggressive with how they were engaging clients. RBC was beginning to feel a bit of that pressure. So once we, like, dug a little deeper, we found, well, how is our process set up right now? And does it like, if you do first principles, does it make sense? And the way our process was set up was we sent a letter, and then we basically, like, have a proof-based approach of during the 180 day window, like, here are the four or five ways in which you can engage with a client. We’ll try calling you. We’ll send you an email. If you go to a branch, then we’ll remind you. That doesn’t work in today’s world. I may never go to a branch. I may never want to talk to someone on the phone about on the phone about my mortgage. And each client is different that way. Some clients will make their decision just 30 days before the renewal window. Some clients are more conservative. They want to do it three or four months in advance. So our hypothesis here was this, like, one size fits all approach for how we engage with our clients when it comes to the renewal window is not meeting the needs of the day and what our clients expect. So we started building an AI solution that could help again, because we know clients very well, these are clients that have been with us at least for a few years, given they have a mortgage with us for a few years, and often tend to have other products. We know these clients pretty well. We understand that we have a lot of data on them, so there’s no reason we can’t be a bit more tailored and a bit more custom in how we engage with them in that renewal window. So what we started doing was we started looking at that data, surfacing insights about, like, how a client wants to be engaged in that mortgage window. Some clients, like I said earlier, only want to hear from us 30 days before. Some clients only want to be engaged on the mobile app. Some clients want an email. Some clients want to go to the branch. And we started, like, making those active, live predictions through this model that we built. So like our clients are being engaged in the way exactly in the way they want when they hit that 180 day window.
Peter Cappelli 17:06
So was this a machine learning type exercise where you’re looking at renewal as the outcome, and then you’re trying to see what differentiates why some clients renew and some don’t, and you’re going from there is that there’s kind of an inductive data science machine learning exercise, rather than a large language model thing, yeah?
Sohaib Perwaiz 17:24
You got it. At a high level, that’s exactly it. The outcome we were optimizing for was retention, and that’s the business outcome that we’re tracking. And we basically look at what are the different drivers of retention for a client, and then basically started optimizing for retention, for around those features.
Peter Cappelli 17:38
That kind of approach is pretty data intensive, right? And kind of expensive in computer power to do? Did you find that? Was it harder than, for example, the earlier one with the agents to do in terms of the computer science and the programming tasks?
Sohaib Perwaiz 17:53
In terms of the data, absolutely, but not necessarily the compute. I won’t get into specifics of the comparison and like, how much like the compute was between the two solutions, but the data was far more complicated. I think the defining thing was the different channels, because the way our bank is set up with our legacy tech, each channel has a different monolithic tech system, and then marketing sits outside of it. So, for example, like, part of our personalization strategy was also customizing which marketing copy goes to which client, and our marketing systems are not set up to do that. They’re not set up to like operate at a client level. And our channels are not set up to like, pass around client actions seamlessly, either. So like, for example, when we send a batch of leads to a branch, we will then send a batch of leads via email, or we’ll send a reminder message through email. Like, we have all these batch processes in somewhat siloed, monolithic systems trying to traverse that was the big challenge here, like, how do you, like, actually get to a client specific action or recommendation? It has to happen through all these various systems and data sources we have.
Peter Cappelli 18:51
Well, actually, that’s the, maybe the last question for you, but it is a big one. That is the way you folks are handling this RBC, but particularly, you know your Borealis group. You’re the kind of center of expertise that is going around different parts of the bank where you can’t be experts in advance on how each of these little particular tasks work. How would you describe that to folks? I suppose the alternative is to try to, I don’t know, what is the alternative to that? To distribute the expertise around the bank? Or what do you think the pros and cons of your approach is? It’s not one that we see everybody doing.
Sohaib Perwaiz 19:29
Yeah, and like, I mean, from my consulting days, I saw a lot of like, big banks take the federated approach. I mean, there are, like, a couple of like models you could do a pick one of the federated approach where, like, each business kind of has its own AI unit that is closest to the business, understands that business. Then you have a hyper-centralized approach, which is you have a center of excellence that formulates AI, AI strategy, and then either, like, implements it directly, or works with the businesses or other technology partners to implement it. We have a bit of a hybrid model in Borealis. So a couple of things on how we’re set up, and I guess the learnings from that. So we are centralized. We are a central enterprise function. But we also build. So we don’t just pontificate on AI strategy, we build. And the other thing that we try to do very like, distinctively, is to like, work very directly with the businesses. In fact, my entire team is set up to be that interface with the business and our technical teams. And we try to embed ourselves within the business, even though we are a central team. So once we say, like, we work with a business leader to figure out this is the problem you wanna solve. It’s not as we go away and just start building it and leave the business owner on the sidelines, we try to really partner with them on the ground and get their resources and our resources working together to solve the problem.
Peter Cappelli 20:30
So is this very different than the management consulting work that you did before? Or it’s it rhymes as they say, it’s similar.
Sohaib Perwaiz 20:37
I would certainly say there are similarities. It’s similar because part of my job, and my team’s job is going deep into a business, understanding their problems and translating that into some sort of practical solution. Where I think it diverges the long term ownership that we often end up taking with these solutions and these partnerships. And this is something like we’re having an active debate on whether we should move to a like a consulting type model, where we actually build a solution, hand it over to someone, and then walk away and do something new. But traditionally, we don’t just, like, set it up, framework it, and walk away. We kind of stick around for the journey till the very end. And now there’s a question about, like, maintenance and stuff, which we’re still like sorting through, which part of the organization owns that. But as of now, the way Borealis works is we don’t just, like, start we take it to the very conclusion. We make sure the business outcomes are there before we walk away.
Peter Cappelli 21:21
And they know your phone number, so it’s not like you can walk away. If there’s a problem, they’re going to follow up, right? Yeah, well, this is terrifically interesting discussion, and it raises a lot of important questions for everybody to think about who’s using AI. Sohaib, thanks very much for coming back to Wharton here and talking with us on this topic today, and sharing what you guys have been doing, it’s really interesting to see.
Sohaib Perwaiz 21:45
Thank you so much for having me.
Peter Cappelli 21:48
So let’s see if we can do some quick takeaways on this. A couple of things that struck me. The first and then maybe the general one is just the extent to which the introduction of AI successfully is very much deeply embedded in management. Management issues, right? And particularly how to use the human agents and the AI agents together is an important decision, and you could imagine organizations have different views on that, maybe depending on the kind of customers and the way they want to interact with them. It’s kind of a strategy choice. And the issue at the end, about how you want to carry out this organizational change, which is a big deal right? To change how work gets done and how employees have to deal with different technology. How do you get that started? How do you manage that process? A lot of important choices there, one of which, of course, is to outsource the whole thing, which is one extreme, and doing it internally, you still have lots of choices. So it’s a really interesting facet of the discussion about introducing AI that has not gotten enough attention.
Peter Cappelli 22:52
That’s all for today. Thank you all very much for listening. Please follow us so you don’t miss an episode. Be sure to tune in next time for a look back at our highlights and the biggest lessons for this season from the podcast. This has been Where AI Works, conversations at the intersection of AI and industry brought to you by the Wharton School in collaboration with Accenture. I’m Peter Cappelli. See you again soon.
Episode 5
Host: Peter Cappelli, Reflections on Season Three
Transcript:
S3E05 Reflection Episode — Host’s Cut: Reflections on Season Three
Karalee Close 00:01
Karalee Close 00:01
Thinking about adopting technology isn’t sufficient in this next era. It’s really about getting people to work differently.
Vivian Sun 00:08
It’s going to be everywhere, in every single industry, in our work, in our life, there will be a huge explosion.
Greg Ulrich 00:17
This whole ecosystem of agentic commerce is incredibly new. It’s happening in the moment.
Sohaib Perwaiz 00:22
A lot of our clients treated AI as a shiny new thing that was for marketing and branding, but I was more and more keen on seeing how AI was actually being turned into value and dollars.
Peter Cappelli 00:36
Hello and welcome to Where AI Works, conversations at the intersection of AI and industry brought to you by the Wharton School in collaboration with Accenture. I’m Peter Cappelli back in the chair one last time for a recap and review of our third season. Our guests shared a lot of interesting insights. So I want to share some highlights and break down my key takeaways for you. Things are changing fast, so let’s dive in.
Peter Cappelli 01:02
We kicked off the season by speaking with Karalee Close, the Global Lead for Talent and Organization at Accenture, and she told us about an effort that Accenture had made internally to introduce AI into their own effort to produce proposals for marketing. Here’s what they did. The first thing they had to do was to identify all the separate steps involved in the process of producing these proposals, and then figure out where agents, that means large language models, could be introduced, task by task to improve things.
Karalee Close 01:37
There’s always an opportunity to get better at what we do, right? So we’re over 800,000 people, and the marketing and communication function was really looking to reimagine how we operated to support the needs of the business going forward. So if I take marketing campaigns as an example, they were taking upwards of 150 days to launch. Any time you see Accenture’s brand in the market, there’s 83 different steps within the process. We were really looking to say, could we do a better job of supporting our brand and our marketing activities with fewer resources? And could we do some new things that we couldn’t do before?
Peter Cappelli 02:14
So the key thing there is that the employees who are involved in this have to really help out. They have to help with the agent. When the agent is introduced, to make sure what the agent is saying is true, they have to try to improve what the agent is saying along the way. Once they get that all done, then they still have to put the jobs back together, probably in a different way, because different agents have been placed into different human jobs in different places, so the jobs have to be reconfigured.
Karalee Close 02:46
We also needed really new approaches to understanding the roles in the organization and the tasks that were involved, and the skills that are involved in producing certain types of work. So when you think about redesigning work more fundamentally, you’re basically saying, if I lay out today’s process, I could actually fundamentally reimagine the way the work happens. I can automate some things traditionally, with existing technology. I can also augment the work that people are doing with agents.
Peter Cappelli 03:16
Our second guest of the season was Vivian Sun, who’s the Senior Director of Data and AI Enterprise Architecture and IT Transformation at Jabil Incorporated, that is a manufacturing firm. One thing that I thought was particularly interesting there is that they were looking at a task to automate, basically to learn from it. They were going to automate the task of quality control. Are we producing parts with the right color?
Vivian Sun 03:44
A traditional optical technology might understand blue versus yellow, but there are subtle differences between more a little bit blue or less blue. Right, human eyes can recognize that, but maybe not traditional optical technology, and we also utilize it in inspecting the cosmetic errors. So the products that we make, we actually hire many inspectors to look at if there are scratches, if there are dirt or dent on the product, and traditionally, we have used humans to do that. Now you we are in utilizing AI.
Peter Cappelli 04:18
The key thing about this was it wasn’t so much that this particular intervention was super important for the company. This was kind of a proof of concept for the leaders of the company and the board to show them that, yes, you could automate stuff, and yes, we could do it in a way that should open up opportunities to do it company wide.
Vivian Sun 04:38
We were actually implementing solutions using other technologies at the same time, but this area, it was more of the shorter wing for us, comparing to other technologies, because it was so direct impact to our business, so people can see the result immediately, and that brings the confidence of the AI technology. So it was a live case to educate everybody the possibility of AI. So we were able to gain more attention from our executives, right, even our customers, and so they trust us in continuing the pursuing of AI technology into other manufacturing or other core functional teams.
Peter Cappelli 05:21
For the third episode, we looked at the intersection of AI in financial services and the work there. And we sat with Greg Ulrich, who’s the Chief AI and Data Officer at Mastercard, also a Wharton alumnus. I was particularly interested to see that what they were doing there was not necessarily a single AI introduction. They already had lots of separate agents and some machine learning models along the way of trying to figure out the following problem: Is a transaction on our credit card legitimate or not?
Greg Ulrich 05:53
Every transaction that comes through our network gets a score. It can score from 1 to 999, based on how likely we think that this is legitimate or fraudulent, and that’s based on AI that we’ve developed over the years, and based on transaction history and what we see happening in the network. Now we can use Gen AI to bring in a lot more information than we could previously to make sure we don’t get those inappropriate declines, and make sure that when there is something inappropriate, that we actually are able to stop it.
Peter Cappelli 06:21
And the key insight here was how important organization change was. That is the key problem for Greg was to get all these different players, each with their own AI tool, to agree to come together and create one simple but very big measure that would say whether a particular transaction is valid or not.
Greg Ulrich 06:42
I think our organization came together very, very quickly and coalesced on a pretty clear answer about how we were going to enable this. I do think that there will continue to be discussions around different parties, because you’re basically adding another party into a flow. Instead of just a consumer, a bank, the acquirer, for the merchant and the merchant, you now have an agent in the middle. And I think the questions around how economics are distributed with another party, how liability are distributed with another party, how visibility and ownership of the consumer exists with another party are still things that people have concerns about, and we’re working through.
Peter Cappelli 07:18
The fourth and final guest of the season was Sohaib Pereaiz, who was a student of mine at Wharton and is now the Group Business Engagement Lead for Royal Bank of Canada, the Borealis group, which is the AI Center of Excellence for the bank. And he talked to us about some interventions they had made to improve the delivery of customer service at the level of the individual employee, agent.
Sohaib Perwaiz 07:45
We started with an LLM that could actually just help answer questions in natural language with advisors. If you call and say, have missed a credit card payment, rather than the advisor going to like search for the exact policy or procedure that would help address that client concern. It’s almost like a Google search bar where they could say that the client has missed a credit card payment. They have this kind of card. How can I help them? And then the response the LLM would give back was okay. Here are the three steps you should tell the client, oh, by the way, clients generally like to hear that. It’s all going to be okay as well. So maybe, like, throw that into the conversation.
Peter Cappelli 08:13
So the key thing here, for me was that the intervention they had made did not cut any jobs. It just made each agent a better delivery of customer service to each client. In particular, helping to figure out before the client even picked up the phone what it is that client might need so the agent could talk to them about it.
Sohaib Perwaiz 08:36
Our hypothesis and the industry may prove us wrong right now is that the human connection is important for the client. The human layer actually adds a bunch of value that purely AI couldn’t and again, like there’s a world in which we can have like an identical AI system that can orchestrate it all. But where we are right now, where our clients are right now, we feel like the human connection, the human overlay, is actually very important for establishing that relationship.
Peter Cappelli 09:01
Our guests shared a lot of valuable advice and insights over the course of these conversations. My main, maybe overriding take away from these business leaders sort of looks like this. The first and maybe most important thing is that head count was not cut in any of these interventions, that basically they found ways to make employees more valuable. The second point is that change management was super important in trying to get these things to work. The big gains came from trying to be more productive and allow employees to do new things, not necessarily replacing any of them with AI.
Peter Cappelli 09:40
This has been season three of Where AI Works, conversations at the intersection of AI and industry, brought to you by Wharton in collaboration with Accenture. Thank you so much for listening. Please follow us so you don’t miss as episode. Season four is coming in the new year, hosted by my colleague, Christian Terwiesch. And of course, if you haven’t listened to every episode from season one or two yet, I really hope you’ll do so. I’m Peter Cappelli, goodbye for now.
Season 2: AI & Business Model Innovation
In Season 2, Wharton Vice Dean Serguei Netessine explores how AI is reshaping business models—helping companies unlock new sources of value, rethink competitive advantage, and navigate disruption through innovation.
Episode 1
Guest: Alan Lee, Chief Technology Officer, Analog Devices Incorporated
Transcript: Episode 1
Alan Lee 00:01
There’s still plenty of hype going on about things that AI can do, and I also think there’s underestimations by a fair number of people, particularly senior leaders, that they’ve heard about the hype and they’re worried about it, and they’ve seen many deployments that have failed for a variety of reasons, and so they’re not as willing to place large term investments, and this could place some of these firms at risk, because they’re not going to be on the leading edge.
Serguei Netessine 00:31
Hello and welcome to Season Two of Where AI Works, conversations at the intersection of AI and industry brought to you by Wharton in collaboration with Accenture. My name is Serguei Netessine. I’m a Professor of Operations, Information and Decisions at the Wharton School, and I’m also Senior Vice Dean for Innovation and Global Initiatives. It is my great pleasure to be a host for this season of the podcast as we tackle the big questions shaping AI’s role in the world of business today and into the future. It’s our goal to cut through the noise and to deliver actionable insights for business leaders by combining cutting edge research with real world case studies. Things are changing fast, so without
further ado, let’s dive in.
News Clip One 01: 7
We’re also monetizing AI in a variety of different ways.
News Clip Two 01:20
If you want to think big picture and you want to think AI, and that trend that has really been the key driver of technology, still talking about ways in which they can monetize AI.
News Clip Three 01:30
Clearly, there are massive applications and demand for AI out there. You then start to have to focus a lot more on the monetization.
Serguei Netessine 01:37
For this season of the podcast, I’m going to focus primarily on business models and monetization of AI. After all, there are a lot of AI-powered technologies and solutions in the market, but if a big company wants to offer AI capabilities to its customers, it had better know how to make money from them. In my own research at Wharton, I’ve learned that most of the time, new business models come later after technologies have already come in. And first, we often see new technology like AI to improve productivity of already existing business models. And only later we see that the new business models are emerging. To help me discuss this topic today, in greater detail, I’m thrilled to introduce my first guest of the season. Alan Lee is a fortune 500 CTO currently at Analog Devices. Alan, welcome to Where AI Works.
Alan Lee 02:29
It’s great to be here, Serguei, thanks for having me.
Serguei Netessine 02:31
Before we talk about AI specifically, I wonder if you could share a little bit about yourself and your career path, your organizations, so listeners can learn a little bit more about you.
Alan Lee 02:42
I’m a Senior Executive focused on leadership, innovation, finance, technology and strategy. I’ve worked at a variety of corporations; IBM, Intel, AMD, most recently at ADI, in addition to a few startups. One of the things I’m most passionate about is really co-innovation, that simultaneous development of technology along with the business models to really achieve the maximum possible value.
Serguei Netessine 03:07
Fantastic. Thank you, Alan. First, I would like to discuss how AI is transforming traditional business models. So ADI, for example, has a long history of producing, selling hardware, microchips. How do you think about AI challenging this old business model of selling hardware, and what new business models do you see evolving across multiple industry verticals that you have experience in?
Alan Lee 03:35
Yeah, there’s been a trend from the edge, but also, certainly for mid-size players, in terms of automotive, also the core players, your larger AI to move more from just selling the chips themselves towards a more integrated model that clearly includes software algorithms and other things. Now, different companies have, of course, done this in a variety of different ways, but the move is really more from companies selling components to selling integrated platforms that create solutions for customers all the way up through creating systems or subsystems. So that has really been a trend, certainly in terms of the larger hardware suppliers that started, maybe say 10 or 15 years ago, but it’s really
pervasive now and accelerating.
Serguei Netessine 04:23
Interesting. How do you do this? Do you do this with customers? Do you do this with suppliers? Can you give us some examples of how this kind of a collaboration, co-innovation works in practice?
Alan Lee 04:34
Yeah, it really needs to be done internally first. First of all, it has to have particularly with traditional hardware companies, that understanding of the importance of software, the importance of firmware, the importance of algorithms. And once you start bringing these things together into a systemic whole understanding, and then working with the customers directly, not in the sense of, “hey, we’re producing something”, but rather, “how should we produce this platform or system that will actually create solutions for those customers?” And I say you start internally first, because you have to have some guideposts, some vision about what you might do. But so much of that once you get that initially kicked off is really about working directly with customers, looking across the supply chain to even your customer’s customers, and finding ways that solutions can be brought about more efficiently or more cheaply.
Serguei Netessine 05:32
So it sounds like this will change your entire relationship with OEMs and integrators. Does that sound right? Or how does this happen?
Alan Lee 05:42
Absolutely, and sometimes, of course, as one thinks of, say, verticalization or other things that are going on, there’s natural tensions that occur there. But what I’ve seen more often than not, in the AI arena, particularly as one moves from the core, by the core, I mean your larger AI or large scale out deployment platforms, to the edge, by which I mean things from the automobiles, smaller things that are in handheld wearables, those types of things, I believe there’s actually less tension occurring there. And because there’s such a broad desire to increase the amount of intelligence as one moves out, I see really more of a willingness from customers that might if you went through traditional verticalization, they might say: “Oh no, that’s kind of our business. We do it.” But in the AI space, I’m seeing more of a willingness to partner, because I believe that artificial intelligence in many ways, pardon the cliche, but it’s really increasing the pie, if you will, and they understand that, so they’re more willing to say: “Okay, I’m willing to give up a few things here”, but to have a much larger market to bring in far more customers, or a much higher value add.
Serguei Netessine 06:53
Would you say that customers are responding positively to this idea of embedding intelligence at the edge, rather than relying solely on something that sits somewhere in the cloud?
Alan Lee 07:05
It’s very early days. People aren’t used to it yet. We saw some early misses, even of course, on the larger scale, AI side, from some major companies. And as we move to the edge, it’s going to be trial by learning process. But for those that have gotten it right, and we’ve seen some huge successes, say in robotics and factories, in automotive driving cars and safety features, that people are very much embracing it, because they’re they’re not just saying: “Hey, we’re doing things a little bit better”, they’re seeing capabilities and solutions that were never possible before, and that excites anyone across the world.
Serguei Netessine 07:43
It sounds like companies that used to sell hardware and make money that way are moving increasingly to business models which sell integrated systems, subsystems, which are designed with AI at the edge, which allows you to deploy smaller, maybe AI models, which take data from Internet of Things, kind of devices, and allow you to do things that you couldn’t do before.
Alan Lee 08:08
Absolutely, I usually use an eighth grade Earth science model to describe this. In the center, you have the core, and that’s the large scale, scale-out massive AI systems. In between, you have the mantle, and you could think of this as industrial AI, or AI for automotive, where you have enough compute power there, it’s not a supercomputer, but enough compute power to do some really viable AI. And then at the edge you have sensors, where you have phones, you have not as much compute power, usually small form factor, but you can indeed do some intelligence there. You can do some basic language work. You can do some basic prediction work there, and if nothing else, you can do some basic data analysis to decide the bottleneck in most modern machine isn’t the computing, it’s actually the memory. So what do you actually transmit? What do you actually store? And it’s not that one of these will be dominant over the other in the end, although the core certainly dominates now, is that what we need and what we will achieve is a world where this artificial intelligence will have the right intelligence at the right place, at the right time, and that’s when we’ll really be starting to extract full value from these complex systems.
Serguei Netessine 09:18
I really like this mantle model of mantle versus, kind of, the edge intelligence, and that at the end of the day, we will have to have intelligence throughout those layers, and that not every query has to go to ChatGPT or OpenAI or whatever it is, right? Some of the compute you can do just at the edge and your server start doesn’t need to talk to ChatGPT and constantly pump data back and forth into the cloud.
Serguei Netessine 09:44
I want to move on and talk a little bit about your objectives and your work and potential obstacles on the road to implementation of AI in a big organization. So, what do you see kind of as key challenges? How do you make AI work for a big company? Is it about your people? Is it about your customers? Is it about
your processes?
Alan Lee 10:12
It’s a great question. I think we finally reached the point that when we actually consider people, and the number of people that are working at, say, modern tech firms, about half of the people either grew up with or are fairly familiar with AI, and about half are not. It was something that’s new to them. Now many of those have, of course, done some upskilling and some other work, but right now, what I’m seeing is more of a cultural conflict in the sense of, what does it mean to be a modern technology firm. There’s the group that says we don’t need artificial intelligence. Human intelligence was good enough for us, and we can build these technologies. And there’s the group that grew up using AI for the most part, and they’ve seen firsthand the value, whether that was in school or whether they developed some of these basic skills working with their colleagues early on, and I think as we move ahead, that cultural element, certainly over the course of the next few years, shouldn’t be neglected. Everyone understands these days that big data is big, right? It’s growing, and we see all these exponential graphs, et cetera. But we need to think the same way of how do we get overall entire corporations bought in two AI models and how they’re used, and the models of yes, some things are very large, some things are medium-sized, some things are very small. And what is the best use of AI for a particular application, at the time? There’s still plenty of hype going on about things that AI can do. And I also think there’s underestimations by a fair number of people, particularly senior leaders, that they’ve heard about the hype and they’re worried about it, and they’ve seen many deployments that have failed for a variety of reasons, and so they’re not as willing to place large-term investments. And this could place some of these firms at risk, because they’re not going to be on the leading edge when the tides eventually turn and as we move through the next few years of graduation and things, their body of engineers and body of scientific employees really has that as a basis or foundation, which is only naturally going to lead to more and more efficiencies in AI, much higher understandings and, in turn, much more value extracted from the technology.
Serguei Netessine 12:28
How do you think about upskilling employees? What is the most effective approach to that?
Alan Lee 12:35
That’s a tough question. It’s really about balance. What you need to do is you think of not just upskilling, but rolling out an AI program of some sort, is to understand that, yes, it is fantastic to have some of these lofty goals, and you must indeed invest in those lofty goals and what supports them, things like ITinfrastructure, things like making sure you have the right data and the quality of data that you need to back it. But also provide some early wins and tie in and have a very high level of communication on both sides of the fence, both those that are pro and those that say: “Yes, we see some value, but not the value that’s going to cause us to invest, say X amount of dollars in this”, they see it as a fool’s errand. Call them the AI skeptics, if you will. And I think that gradual iteration of the shorter term, investing in the longer term, and then doing things in between, will eventually get us where we want to be. Companies that I’ve seen that just try to do things that are small, they don’t see the value in it. And those companies that I’ve seen that only want to invest and push all their chips into the table before the company’s ready for it, I’ve seen a huge allergic reaction, if you will, if some cases, that causes people to pull back and say “Well, that didn’t work, so we’re not going to use it.” And I don’t think either of those approaches really work. So achieving that balance between the two, I believe, is critical.
Serguei Netessine 13:21
So, thinking about this balance between internal units, right? So obviously, you’re a CTO, you have more advanced users, probably fewer AI skeptics on your team. How do you work with business units on developing your AI applications, developing new business models, because those other units might have lots of skeptics, right?
Alan Lee 14:26
It’s surprising. It’s actually, from my personal experience, been the opposite. The technologists who believe that they understand technology and the way that the world is and it will continue going on that way, often are some of the largest skeptics, seeing more pushback. Whereas those, for instance, I’ve seen in the legal profession, those in HR, those in finance, they’re honestly dying to get their hands on the new technology, and more willing to explore because they don’t have the technical doubts around it. It’s the old science fiction and aphorism, any sufficiently advanced technology appears like magic, and it is, in many cases, magical to them. It can truly change the way that they’re doing things. So I’ve seen, in many cases, more pull from, say, the SGNA functions, than I have seen on the technology side, where they believe, no, we’ll just keep making this 3% better every year, and that will get us where we need to be.
Serguei Netessine 15:24
Interesting, that’s unexpected to me. So when it comes to reworking internal processes, is there some change that you have made that had the biggest impact so far to help adopt this AI-forward thinking?
Alan Lee 15:40
You need to make it a part of people’s actual workflow in the organization itself. If you make it something separate, then you’ll have very low adoption. And already it’s hard to measure productivity in this area. Mainly because, as with many of these new technologies, you’re not going to ask somebody, say, a program, they’re asking something to do it, and they do it with AI, and they do it certain role, but you’re not going to ask them “Okay, now go and redo that entire software program without AI.” And so that makes, in some cases, metrics hard to come by, at least some of the harder metrics, but you can do it in aggregate. So if you only do some of these small experiments and things, and yes, of course, you should start with some small experiments. But if you’re not willing to dive in in some areas and provide at least bit by bit, the large scale training and make it a part of the actual workflow, people won’t use it. We’ve seen this a lot in terms of basic engineering tools at the companies I’ve worked with, if you say: “Okay, well, you can use your standard engineering tool, which you’ve been using for 15 or 20 years, or you can use this new thing”, people tend to gravitate towards what they know. But if you turn off the old thing for a bit and you say: “Hey, why don’t you try using this”, people are up in arms for about two or three weeks, but once they see the value in it, once they see the possibility, they’ll turn around, they will be your biggest supporters.
Serguei Netessine 17:13
I want to move on and talk about the future a little bit. So what do you see around the corner. What do you think AI will do to alter your products that you offer? How do you think about future development of AI? Where will it focus?
Alan Lee 17:30
I really see, as we move ahead, over the course of the next 5 to 10 years, a world of pervasive AI, where intelligence, machine intelligence of some sort will exist everywhere, in our cars, in our homes, in our irregular systems, in our schools, in the way that we interact with our media, our phones, et cetera. Maybe it helps you make a better selection of tomatoes at a supermarket, say, or something like that. As we move towards more autonomous driving and things, as you move towards more efficient robotics in warehouses, in factories, and that’s going to increase in efficiency, which will in turn increase the amount of AI and things that we use, and, of course, something that’s already present today, and very much in the top of people’s minds, the very large scale AI trying to move towards, in some sense, general artificial intelligence, right? That idea of sentience, that idea of truly thinking, and what that can mean. And to me, the future is about: What is the balance between these things? What does that spectrum look like? Where do we need more intelligence? How is this going to help humankind get rid of more medium tasks? But just as importantly, what are the people that are doing, for instance, more menial tasks, what is their next role as the world evolves? And how do we bring society forward? So I see both real beauty and real possibility in what we can do, but I also see real areas of caution where we need to think and bring people along.
Serguei Netessine 17:31
What do you see in this space evolving as a kind of a business model? Do you charge per use of AI? Do you charge per money that the AI makes for you somehow? Do you charge something for data?
Alan Lee 18:36
We’ve seen a lot of missteps here, right? We’ve seen a lot of people say, because theoretically, AI is great to solve a huge number of the world’s problems, but not many of those have actually come to pass yet. So I think it’s important to get over that hill, or to have the activation energy to look first internally, as I mentioned before, and get more of those folks involved and engaged. Because when you do that’s when you’ll start seeing that interaction at the large-scale corporate levels, about “hey, what does the solution really mean?” What AI is enabling us to do is to think about the world and interact with the world in brand new ways. It hasn’t created, you know the old jokes, I mean, you’re dean of a business school, “oh well, this is a new business or a new way of doing business.” No, in essence, you buy low, you sell high in any particular order, right, you know, that you can. But what you’re doing is looking at solutions from a different perspective. And I think that by co-developing these new technology along with new business models. We’re not going to change the “buy low, sell high” parts-portions of the business, but we will see new models about how we monetize, particularly as we move AI forward. We’ve seen some of those, for instance, let’s take subscription models. First of all, subscription models were the rage, and lately, if you follow a lot of the commercials on TV, you’ll see a lot of business models that are based on actually getting rid of your subscription models, which you’re paying for, because you’re not actually using them. And you know, I’ve seen things that are creative, such as AI could go through and maybe attenuate that somewhat, right? And say “Okay, well, we’ll keep you on some kind of basic maintenance plans. You’re not paying as much.” People say “Oh, wow, that’s great.” This company I’m using is looking out for me when I need it, it’ll be there, it’ll have it available. So I’m seeing a lot more areas that used to be black and white, that are a lot more nuanced, that I think are really ripe for the picking in terms of AI, where if you had to make a decision early on, and you didn’t have any of that feedback, you would have to make a digital decision, yes or no, zero or one. But adding AI into the mix, along with, of course, big data, large computation and AI at the edge, it really is going to enable a more nuanced understanding of where and when we can charge customers, and where do they see value, and how can we monetize and capitalize on that. And the other part of that model, you know, they say the customer is always right, and they still are. But now we’re in an era in particular where innovation is king, and understanding what is the art of the possible and what the customer can possibly have. It’s again that balance as we bring things forward.
Serguei Netessine 2:07
Finally, I want to ask you, just for your kind of a top AI related recommendation or advice to business leaders who are listening to this podcast.
Alan Lee 22:17
I would say it comes back to balance, the yin and the yang of AI, particularly at this particular time in human history and where AI stands. It’s such a new technology to some, and it’s such a powerful technology that applying some balance being both cautious about how you apply it, but also willing to invest heavily when you see a true and viable market where you can capitalize on it, that’s where I really see the huge value and the best advice that I can give business leaders right now.
Serguei Netessine 22:49
Fantastic. What a great way to kick off season two of the podcast. We’ve covered a lot of ground today. Alan, thank you for sharing your time with us here on Where AI Works.
Alan Lee 22:59
Thank you for having me, Serguei, it was a real pleasure to be here and speak with you today.
Serguei Netessine 23:02
Let me now review a few key takeaways from my conversation with Alan. So there are so many, it’s a little hard to summarize them all, but as I teach a program on business model innovation in the age of AI at Wharton, and I hear from many executives that struggle with AI implementation at their companies, we see here, there are actually many ways to monetize AI. Some of it just comes from demand, your customers, demand AI from hardware manufacturers. And increasingly, hardware manufacturers are moving into subsystem manufacturing, which is combined with AI in some ways, and the whole kind of creation of ecosystems where hardware interacts with AI. Other monetizations may involve AI for internal efficiency driving internal processes within the organization. And yet, another monetization could come from external optimization, connecting better with customers, learning about customers needs through AI, and the new business model can come out of that. But overall, we both believe in pervasiveness of AI, it will be everywhere, which is why we need balance right now. We need to be careful and cautious about investing in AI and experimenting with AI. It always starts with experiment. As I teach in my classes at the same time, you need to invest decisively. You need to invest heavily.
Alan Lee 23:05
Start with multiple small experiments, see what works. But if you’re playing in Texas hold ’em. There comes a time and you need to push your chips into the center of the table.
Serguei Netessine 24:45
That’s a great way to summarize this, Alan. That brings us to the end of our season premiere. Thanks so much for listening. Please follow us so you don’t miss an episode, and be sure to tune in next time when I’m speaking with Elad Walach, the CEO Aidoc, a trailblazer in clinical AI applications. This has been Where AI Works, conversations at the intersection of AI and industry, brought to you by Wharton in collaboration with Accenture. I am Serguei Netessine. Bye for now
Episode 2
Guest: Elad Walach, CEO, Aidoc
Transcript: Episode 2
Elad Walach 00:02
I think this whole space is going to be explosive in the next few years. Every patient interaction would have an AI guardian angel looking over it, ensuring quality of care is good, you’re not missing anything important, collecting information, ensuring the patient gets the care that they need at the time they need it. This is a world where diagnostic errors I have mentioned 370,000 deaths, on top of 400,000 permanent disabilities. All of that could be at least halved if we do this correctly. The impact on human lives and society as a whole is going to be immense.
Serguei Netessine 00:39
Hello and welcome back to Where AI Works, conversations at the intersection of AI and industry brought to you by Wharton in collaboration with Accenture. I’m your host, Serguei Netessine, I’m a Professor of Operations, Information and Decisions at the Wharton School, and I’m a Senior Vice Dean for Innovation and Global Initiatives. It is our goal to cut through the noise and to deliver actionable insights for business leaders by combining cutting edge research with real world case studies. Things are changing fast, so without further ado, let’s dive in.
News Clip One 01:14
AI has been called medicine’s biggest moment since antibiotics
News Clip Two 01:19
A task that can take specialist doctors hours now being done in seconds through artificial intelligence.
News Clip Three 01:25
AI is not replacing your doctor. It’s just another tool in the toolbox to help take care of you.
Serguei Netessine 01:30
On this episode, I’m excited to discuss AI transformation and implementation in a healthcare context. We’ll explore what it takes to go from a single-use algorithm to an enterprise-grade platform that hospitals rely on every day, and we will find out how real world ROI, not hype, is driving adoption. In my own research at Wharton, I learned that it is particularly difficult to implement innovation in an industry like healthcare. Healthcare is heavily regulated, and as such, this usually slows down fundamental innovations in particular platforms. To explain how this company cracked the code on monetization and what historically has been a fairly slow moving sector. I’m thrilled now to introduce the CEO of Aidoc, Elad Walach. Elad, welcome to Where AI Works.
Elad Walach 02:24
Serguei, thanks for having me.
Serguei Netessine 02:25
So before we dive into the AI conversation, I wonder if you could give our listeners a little bit of a background on Aidoc and what led you to create this company back in 2016.
Elad Walach 02:37
Well, first of all, wow, nine years. That is a long time. But you know, we started the company, really with the mission to reduce diagnostic errors. I had a few family stories of deaths and injuries with diagnostic errors, and I think I became immersed in this world of diagnostic quality. There is ample data that we need to improve quality of care. If you look at Hopkins University, for example, published about a year ago a study about the burden of diagnostic errors on the US. Every year there are about 370,000 deaths due to diagnostic errors.
Serguei Netessine 03:12 Incredible.
Elad Walach 03:13
How do you go from something that just offers this point solution for radiologists who are doing diagnoses to an AI tool that is a platform for hospitals?
Elad Walach 03:13
370,000, that is the third largest cause of death. That is 10 times more than car accidents and gun violence and all of those, right? We’re talking about a massive problem that, by the way, if you look at the economical cost of it, this could easily be half a trillion to a trillion dollars a year that is caused by this plague. So when we came to the AI scene, we saw incredibly amazing and passionate physicians that we’re really passionate about the care they deliver, but a healthcare system that is really bursting at the seams and having all these quality issues caused by this immense pressure we have on the system, the labor shortage, the over specialization of care. And we wanted to help really with clinical AI. So I was an AI researcher, originally, not from healthcare, but from defense, and we came in to spend a bunch of time in hospitals and saw that, really, clinicians are now doing all their work manually. You mentioned that regulated industries are slow to adopt technology. That is absolutely the case, and AI seemed to be like this very obvious tool to help them, right? If you look at, let’s say, a radiologist. They could look at a CT scan of the head, which is 500 images, and they need to find the most subtle of signals in that scan. It’s like a Where’s Waldo? but Waldo is somewhere in a movie, right? So you need to somehow do that, and we’re expecting them to spend maybe a minute on a scan to find the subtlest of findings. Obviously, AI can help them triage through the noise. And that was really the vision of Aidoc, is: Can we build something that is comprehensive and can transform the way that people deliver care, not just on a single disease, but in a broader sense.
Elad Walach 04:57
That’s a really good question, because obviously, if it was easy to build a platform, then anybody would do it. So the first three years of the company, we actually didn’t focus almost at all on the application, but on the infrastructure to build it at scale. And that was a very important lesson for us, because we saw that indeed our assumption was true, that if you just build point solution, you will get stuck really, really quickly. The second thing that we’ve learned was actually something that we’ve learned by, I would say, a mistake or errors we did along the way. When we started selling point solutions, it was
very easy for us to identify the buyer of the solution and the beneficiary. We had the same who’s the user, who is the same as the buyer, it was very easy. We always had for this initial solution, the radiologist at heart. Over time, we started learning that actually you have to think about it, not from an individual perspective, but from a system perspective, especially as you build a platform. And let me explain what I mean. One of the value propositions for the AI tool is ability to triage patients. So imagine you have a backlog of 1000 patients. We can tell you, hey, patient number 950 actually has something acute. You should take a look at that patient right now. If you look at how healthcare works, and we could dive deeper into it, radiologists don’t get paid more to read that patient on time. They’re only incentivized to just read through their like whole backlog. So why would they even invest in AI that helps them do that. Well, the reason is not in radiology, it’s actually downstream from radiology, because the emergency department is solely dependent on imaging. So if you want to improve your ED through-put, you better know how to triage those patients. So what we’ve learned is that you have not to think from an individual perspective, but from an enterprise perspective, and that allows us to unlock the possibilities of a platform, because then we could go and say: Look, these are these 50 use cases that different parts of the organization cares about, but you have 1 c-suite owner that is going to do the AI adoption. And now it makes sense to think from a platform perspective versus a point solution.
Serguei Netessine 07:19
Yeah, essentially, it sounds like what you saw is that the benefits of your solution might accrue to someone who is not the user of the solution, but someone who is upstream or downstream.
Elad Walach 07:30
Absolutely. And actually think it’s broader than healthcare. I think a lot of people, when they think about AI, they think about efficiency. Oh, can I do things faster? And it’s absolutely a value proposition. I’m not arguing against efficiency, but I think we need to think about from a system efficiency, you can unlock so much bigger gains. Somebody asked me, for example, when I implement AI for my sales team, do I really care about my sales team efficiency? I mean, a bit, but if they can increase revenue: Oh my God, I care so much more. So thinking about quality and system efficiency. It was a really important observation for us, at least in healthcare.
Serguei Netessine 08:04
Fascinating. So how did you go about discovering all these learnings? Was it through some experimentation? Was it through customer push, customer discovery, or through trying a few things and showing customers what’s possible with AI, because they didn’t know for example.
Elad Walach 08:23
We found that, in healthcare specifically, AI was- is a new category. So it’s very hard for people to imagine the possibilities, especially true if, again, as you mentioned, the user is not the main beneficiary sometimes. So a lot of our work was data-driven extraction of value. You know, you have this saying: If you build it, they will come. I found that to be untrue. You have to build it, but you also have to show the value and be able to extract it. So if I think about AI, the loop for AI for healthcare, it’s you have to find the patients, then you have to generate an action on this patient. Just finding the patient isn’t sufficient, so find act, but then measure. Find, act, and measure. And if you don’t have the full loop, then you can either find the patients, but nothing happens to them, so nobody cares, or you actually act, but nobody knows what the value is. So the ability to measure the value was a very important facet of the adoption.
Serguei Netessine 09:22
So I’m hearing three things so far. So you talked about necessity to focus on scale from day one, and that kind of immediately leads you away from point solutions and into something bigger, into system solutions. The second point that you mentioned is always have a systems perspective, because you might be doing lots of good, but somebody who is a user of the solution does not necessarily benefit from the solution. So you need to look at the entire system, improve entire system. And finally, in order to be able to do that, you need to find, act, and then measure. So in order to demonstrate benefits of AI to someone who is not even using AI, then you need to have very measurable outcome of your implementation of AI.
Elad Walach 10:10
Absolutely, and especially on that last point, if you think about healthcare specifically. So the burden on healthcare is massive. Right now, if you look at margin, those who don’t know health systems have very lousy margin. Typically in the one, you know, single digit percentage, below five would be a typical health system. So on the one hand, they’re very passionate about improving care, and these are big organizations, typically billions of dollars in revenue, right 10s of 1000s of employees, but on the other, very thin margins. So even if they want to invest in quality and doing what’s right for patients, they need to find sustainable innovation models. So the ability to measure the outcomes, both from a clinical perspective, in our case, because we do the clinical side, but also the financial is really key. It’s very complex, but I think that’s very important, and maybe different from some other areas of enterprise where they’re saying: Oh, I’ll just adopt AI and I’ll assume there will be benefits. Here, you have to be a lot more educated about how they operate and how you can actually drive financial value.
Serguei Netessine 11:12
I want to talk about specific challenges that you faced around monetization. I know that many AI startups struggle to monetize because ROI is very, very hard to measure. And you know, in healthcare, it’s particularly difficult because adoption and budget cycles are so notoriously slow, right? Waiting for this insurance reimbursement for, you know, paperwork to go through the hospital and so on. So measuring things should be triple hard in healthcare. So how did you do it?
Elad Walach 11:42
So the first thing is that there is a bit of a like, let’s say, lack of awareness on how the different pieces connect. So healthcare as a whole is a very fragmented industry. For example, when you go to a hospital, you could have the hospital being one entity and the physician giving you the treatment be a whole different entity. One way you solve it in healthcare is by getting reimbursement, as you mentioned, Serguei. So you get, you ask for the government, you get a code. That is incredibly slow. There are some companies that do that, but because of we, we want to do, as mentioned, hundreds of applications. It was just too slow for us. So we had to find ways where, like a health system as an entity, would generate this ROI. And the way we would find that ROI was a very laborious process. We had to actually connect to all the data sources and with our partners and customers together, actually create hypothesis as to where the value is and then investigate it. I’ll give an example. We have this capability of getting faster to patients, and our initial hypothesis was: Well, okay, why do you care about that? It’s better care. You’ll reduce morbidity. That’s valuable. But how do you operationalize it? We’ve actually found that if you get to the right patient faster, there is a massive link into how the whole ED or in- patient stay for the patient continues. So then we had to measure that, but when we saw it like one health system, second health system, third health system. Oh, we have a clear link. So now we know we’ve established in a new hypothesis, right? That you reduce time to diagnosis, you improve the throughput for the whole hospital. And we did many of those over time, such that we were able to
identify these gaps and then quantify them. One of the interesting areas we found is that there is a lot of value between the cracks, especially in healthcare, but I think in many enterprises. For example, one, there is one use case that I found fascinating for pulmonary embolism. So in recent years, so pulmonary embolism, I’m simplifying is basically a clot in the arteries of the lung. It could kill you, so you definitely want to find it and treat it. And what we found was that even if you find it, then, for certain
high-risk pulmonary embolism patients, you want to actually do an intervention. You go in and remove the clot physically. To do that, you need a specialist team. So the ED physician needs to say: Hey, I have a patient. It’s a high risk one. Let me get a team on board to evaluate this patient. That process is broken. To understand how broken, in great health systems, university hospitals in Cleveland, NYP in New York, right? Amazing Health Systems. Over 50% of the high-risk patients did not even reach the team.
Serguei Netessine 14:14 Wow.
Elad Walach 14:14
So a lot of people fall between the cracks. So we found AI can have massive impact on those areas of information gaps, because what they do is that they have an AI solution that can actually pick up on those higher risk patients and alert the team immediately to evaluate those patients. Doesn’t replace a physician at all, but making sure you give consistent care. And we see massive increase in the number of appropriate procedures that they do because of that.
Serguei Netessine 14:39
It sounds like you don’t really get reimbursed for the use of AI, right? Because there is no such thing as AI-based reimbursement in healthcare, right? So instead, you focus on KPIs, which are meaningful for patients or for the doctors.
Elad Walach 14:56
Exactly, or for the health system. Patient, doctor or health system.
Serguei Netessine 14:59
And how do you discover those KPIs, those kind of a immeasurable outcomes to begin with, they must come from the customer, be it the final patient or or a doctor, or is it something that is just consistently tracked in healthcare so it’s kind of easy to pinpoint?
Elad Walach 15:17
Honestly, I would say neither. I would say it’s a discovery process where you work together with the customer, but it’s not obvious to the customer, if you’d ask them, they may not know. So it’s this iterative process where you as like the entrepreneur or product team, need to come up with these hypotheses and keep testing them all the time and even look at the data. Sometimes they don’t even know that these links exist. Sometimes they do, and I think we have really good people in healthcare, but something we just lack the data. It’s something you can’t really outsource or assume somebody would know. It’s a lot of active discovery that needs to happen. And by the way, let me share on this point a mistake we made in the early days. So we wanted to monetize it on a per-patient basis. So we said, every patient you’ll analyze with AI, they’ll pay us whatever amount for the analysis. And this was a disaster. And the reason why it was disaster it because it encouraged your customers to do cherry picking, and because there is no clear understanding of where the value is in advance, then this cherry
picking is disastrous to the adoption of value so, and as an example, in healthcare, a lot of the value comes from incidental findings. So we gave this pulmonary embolism example prior, just to stay on point, pulmonary embolism can be found where you’re expecting it and you have shortness of breath, but it can also be found in cancer patients with almost no symptoms. Only a small portion of those patients, but from the whole population of your cancer diagnosis. So you have to run it on every patient, almost like a safety net, which is a lot of the power of AI, right? You can have this automated screening. So we found that if you’re encouraging customers to do cherry picking, it’s very counterproductive to the actual value realization. So thinking about the actual scalable factor, like, what do you scale so the customer would use? Is it user? Is it patients? Or how do you build this was actually another really key task we had to do in the early days.
Serguei Netessine 17:09
So let’s move to the third part of the discussion where I want to look a little bit into the future. You are now positioning Aidoc as an operating system for the modern hospital, that’s a very bold shift. What does that mean for your business model going forward?
Elad Walach 17:27
If you look at the world today, we have the most FDA clearances in our category for clinical AI. But even with all of those, we only have maybe 30 solutions on the platform that is out of 700 different admission codes, or 70,000 different disease codes. So there’s a lot more ground to cover that is only enabled now with the advent of new technologies. In the past five- six months, we’ve actually partnered with Amazon and Nvidia to launch the first clinical-grade foundation model for medical imaging. We’ve already gotten the first FDA clearance on an application that we built on top of this foundation model.
But it’s going to be a massive undertaking in the next few years, which we’re really excited about. The reason why I’m saying all of that is: I think this whole space is going to be explosive in the next few years. If today, a good health system that, let’s say, is leaning in with AI, would adopt 20, 25 use-cases. I think a reasonable like a medium of the PAC health system, 3 years from now, will be adopting 100 to
- Every patient interaction would have an AI guardian angel looking over it, ensuring quality of care is good, you’re not missing anything important and collecting information, ensuring the patient gets the care that they need at the time they need it. And I believe this is not decades into the future. This is the next 3 to 5 years, we will have that AI running and providing this improvement in care. And I think this is a world where, in theory, all these, you know, diagnostic errors I’ve mentioned: 370,000 deaths, and on top of that, 400,000 permanent disabilities. All of that could be at least halved if we do this correctly. It’s a pretty bold vision, but I think the impact on human lives and society as a whole is going to be immense.
Serguei Netessine 19:11
That’s an incredibly bold vision, but you have to, I guess, prioritize somehow, because there are so many diseases that you are not currently addressing. So do you have some kind of a plan in your mind? This is where we’re going to go, first, second, and third. How do you think about this process?
Elad Walach 19:28
Obviously, we have to prioritize, and we pick the areas based on, you know, acuity and disease burden. And every year we have a list of the things we need to do. But the biggest thing I I would like to emphasize is the increase in innovation that we have with comprehensive models. Think about ChatGPT for a second. NLP in the past was: you build one use case, you ask one question, right? You do the spell check here for the A words. ChatGPT does everything, right, you can ask it any question
you want. And it basically allows us, the users,to build applications in rapid speed that are very generic. That is the future. So these foundation models are not going to be disease-specific. They’re going to be generic. It’s going to take time to build. It’s going to be a massive investment, but I think this would allow us to really capture and democratize access to disease detection, such that A) Aidoc could generate a lot of those. But also we have a whole ecosystem of people, health systems and other companies building on these models and integrating it into the workflow. So yes, we obviously have a plan, and I can, I can mention, like, the first areas of focus we’re going to start with acute, then cancer. But in my mind, the more exciting part is that we’re going to have a whole industry moving towards this, allowing massive proliferation of these use-cases.
Serguei Netessine 20:45
That makes absolutely total sense. And it’s kind of on a parallel question: How do you think about evolution of the business model of monetizing this AI? Is that going to change? Or are you kind of thinking about the same, tracking some KPIs, and syncing from those KPIs, measuring your impact.
Elad Walach 21:06
I think it will change just because of the sheer volume of indications. I sometimes use the Netflix analogy, right? So when you had not many movies, you could pick a movie, you would rent it. That was the model. And you would think about on a movie-by-movie basis, and the ROI you would get from watching that movie. That is no longer impossible. When you’re talking about hundreds of indications. Even thinking for a second, what do you want and how to evaluate it, that will be too costly, and nobody would really want to do that. We’ll think about bigger adoption. So I think over time, we’ll move to
Netflix-type subscription, where you’re running a whole suite of AI tools on every patient, and you do it based on more, not by an AI solution, but really just like on your whole patient population. I believe that’s a model we’re gonna go into over time, just because we have so many solutions running, but it’s gonna be a few years.
Serguei Netessine 21:57
Fantastic. This has been a fascinating and insightful conversation Elad. Thank you so much for sharing your passion and vision with us here on Where AI Works.
Elad Walach 22:05
Thank you so much for having me.
Serguei Netessine 22:08
So now, let’s review the key takeaways from my conversation with Elad today. So Aidoc isn’t just building a product, it’s setting what I would call a new standard for clinical AI by delivering the infrastructure hospitals need to combine clinical logic, operational intelligence, and system wide accountability. And the way the company does it is by starting with a scalable solution, not thinking about point solutions, but taking a system perspective and then running a sequence of experiments to discover the right KPIs to track impact of AI. The end goal is to achieve a healthcare ecosystem which will allow other players to plug in and offer a democratized, comprehensive solution, kind of like ChatGPT that we use nowadays, but for healthcare. And the way the system is going to be monetized eventually, there’s going to be basically some kind of a subscription, sort of like a Netflix business model, where users are just going to pay subscription fee and enjoy all the benefits of this amazing platform.
Serguei Netessine 23:25
That’s all for today. Thank you so much for listening. Please follow us so you don’t miss an episode and be sure to tune in the next time, I’ll be speaking with Tereza Nemessanyi, an executive at Microsoft, who is focused on AI and digital transformation. This has been Where AI Works, conversations at the intersection of AI and industry, brought to you by Wharton in collaboration with Accenture. I am Serguei Netessine, goodbye for now.
Episode 3
Guest: Tereza Nemessanyi, Worldwide Director, Private Equity and Venture Capital Partnerships, Microsoft
Transcript: Episode 3
Tereza Nemessanyi 00:01
Speaking with a very innovative, digital, native CEO, he’s of the mind that there’s about a six month window to transform their services, their products and fundamentally the give-get value of their customer relationships with AI. And once some type of product market fit hits, they will be moving at such a velocity that it becomes impossible for competitors number 234, to catch up.
Serguei Netessine 00:34
Hello and welcome back to Where AI Works, conversations at the intersection of AI and industry, brought to you by Wharton in collaboration with Accenture.I’m your host, Serguei Netessine, Professor of Operations, Information and Decisions at the Wharton School, and I’m a Senior Vice Dean for Innovation and Global Initiatives. It’s our goal to cut through the noise and to deliver actionable insights for business leaders by combining cutting-edge research with real-world case studies. Things are changing fast, so without further ado, let’s dive in.
News Clip One 01:09
Open source is at the core of GitHub, and we are taking this next big step.
News Clip Two 01:16
And for CIOs, IT pros, business decision makers, we know compliance, measurement, and observability are paramount.
News Clip Three 01:24
What does that mean for vertical SaaS companies? What, how do they prepare themselves for this future?
Serguei Netessine 01:30
On this episode, I’m keen to find out how AI is delivering measurable business results as organizations move fast pilots and turn AI initiatives into scalable, sustainable value. We will explore the sectors and functions that are seeing tangible returns and address the critical factors that separate successful AI deployment from failed ones. In my own research at Wharton, in the upcoming report on startup and corporate collaboration, we highlighted that a lot of AI implementations are happening at this edge of big company, collaborating with sometimes very tiny startups. To help me dig deeper into this topic, I am very excited to highlight a company that has been a cornerstone of the tech industry for decades. I have with me today, Tereza Nemessanyi, a worldwide Director of Private Equity and Venture Capital Partnerships at Microsoft. Tereza, welcome to Where AI Works.
Tereza Nemessanyi 02:27
Thank you, Serguei. It’s terrific to be here.
Serguei Netessine 02:29
Tereza, so we’ve known each other for a few years, and you spoke in my classes. You are obviously a Wharton graduate. Could you give us a little bit of overview of your role at Microsoft and how your career brought you to this wonderful company.
Tereza Nemessanyi 02:44
So first of all, I’m part of a team called worldwide private capital. We are part of Microsoft corporately. The notion behind our team is a lot of world class investors in venture capital and private equity want to know what Microsoft is doing, what we’re investing in, what our roadmaps are, what we’re interested in acquiring. With that in mind, in order to kind of reduce noise, we launched this team to equip our executives with that more focused signal and manage relationships with a small list of the very top. I manage a small number of very large venture capital and private equity firms, and on a very specific basis, we are both driving the conversation between our companies and then also looking at their portfolios and helping them ideate around ways that they cancreate really outsized value by taking advantage of what Microsoft has.
Serguei Netessine 03:44
How has your role evolved as AI became more and more central to Microsoft’s mission?
Tereza Nemessanyi 03:50
I have to tell you, I’ve been at Microsoft for almost 12 years, and nothing has hit as hard as AI in terms of our conversation. You know, the cloud generally has been on the table for quite a long time. Digital transformation has been core, and that intersection between cutting-edge startup innovation and scalability,always on the table. But AI is bringing it to a completely different level. How my career brought me to this? You know, it’s funny. I think when I wrote my Wharton application, they said “Well, what do you want to do?” And I said, “Well, I want to be a really senior person” at a at the time, I called it a media company, globally, that’s driving innovation. And at the time, I thought that that meant television and things like that. That was back in the early-to-mid 90s. Obviously, the world has completely transformed. And I’m someone who, as a daughter of an engineer and someone in sales, I’m always obsessed with that intersection of the very newest of the new and how we turn them into business models, and do that in a way that is completely scalable, and, in fact, so scalable that it will change the world.
Serguei Netessine 05:00
I’m very glad to hear that AI completely changed your conversation with your clients, conversations within Microsoft. So if you look from your vantage point, where do you see AI truly driving business value? There is, of course, lots of experimentation going on, lots of hype, but how about driving real value?
Tereza Nemessanyi 05:21
So our conversation, pretty typically, is going to focus on three main areas. Early out of the gate, as you know, as developer productivity through GitHub and GitHub co pilot, our company quickly saw that new capabilities were going to be very well-served within the developer space. So that was a clear, pointy area where the pain is there, meaning it’s hard to hire great developers. Everybody has a really huge backlog. So that was not really a monetization play, but for sure, a productivity play, helping software companies, or companies that develop software, which is pretty much everybody these days, be more strategic about what they’re doing. So that would be number one. Number two, I would be remiss to not talk about Microsoft365, and our whole modern work suite, as AI has been pulled through the veins of that. It really is putting capability into people’s hands to figure out what their workflows can and should be, so that they can do their work so much better when you’re democratizing AI that way, meaning their actual everyday tools, ability to do analysis, ability to draft emails, or whatever it might be, that can be incredibly powerful very, very quickly. The third piece that underpins everything that we do is security. So there’s no question that the velocity and complexity of threat is greater than it ever has been, and Microsoft made very clear decisions very early on to prioritize security above all else by leveraging AI to keep our customers safe and secure. I would also add to that, innovating products with the customers and partners that I do work with.
Serguei Netessine 07:17
So what I hear is number one, two and three is really about driving productivity. And you know, software development, but the fourth one is very interesting. It’s innovating products. So that sounds like more about increasing revenue for your clients, right? Rather than driving productivity. Can you elaborate on those products? What are those products that you are collaborating on?
Tereza Nemessanyi 07:39
When I speak to the customers and software companies that we do business with, and investors in those software companies, the holy grail is to be monetizing, creating fresh revenue. The reality is that the market isn’t fully there yet on how to do it. So to transform a workflow and then charging more for it:tricky, because you need to bring something new to the market that wasn’t there before. What we’re seeing, by and large, is a lot of innovation in that gray space between revenue and cost. You know, things like customer centers, call centers, things like revenue renewals, you know, places where there’s a lot of leakage, and the cost to serve that revenue is very high. Because this provides companies a really tangible sandbox to experiment, learn how to use the tools, get immediate feedback from their customer and end users, and really every single point of goodness that comes back will hit the top line. So that’s where we’re seeing it right now. Also, I will say that the revenue innovation is tending more towards long tail. So providing ability to do product extensions or versioning of products that will not require people in the same way that they would for a high-end, you know, enterprise client, let’s say. So in this case, it’s right sizing, if you will, a product in a way that is scalable for distribution to a customer base that otherwise you wouldn’t have gone after in a purely analog world
Serguei Netessine 09:21
I see, okay, so it basically enables large scale personalization to a variety of smaller clients, it sounds like?
Tereza Nemessanyi 09:29
Yeah, that’s right, that’s right. I do have a number that really are thinking about taking that thread that you just said of personalization and layering on top of that with AI that is that much more data driven. Epecially the more data sources that we can draw on, that have traditionally lived in silos, but now we can start pulling them in. And not only, huge plug for Azure Data Fabric as an example, but being able to pull in from a wide variety of data sources, and to do it on the fly, and to do it within the context of an agentic conversation, let’s say. Lot of power there.
Serguei Netessine 10:11
Is there a particular vertical that you would highlight where this adoption is translating into ROI faster? Is it more finance function, sales function? From your previous description, it sounds like IT function is definitely one of them where you’re seeing significant ROI?
Tereza Nemessanyi 10:30
Yeah, so we do both horizontal and vertical. Personally, I tend to be a horizontal person, especially in the agentic world. I’m kind of looking at software development companies that serve other software companies. So one bit of work that I just had a great pleasure of participating in is with Gainsight, which is leading software as a service company for Customer Success Management, which in and of itself, is a pretty new function, or it’s a new function to the SaaS world. So imagine that you have an OS, if you will, or a dashboard for managing customer success. Well, then you can start going “Okay, well, what workflows in this can become agents?”, and that will, you know, give a pretty straight line of sight to ROI on revenue. So these are, you know, let’s say, horizontal tools being built on Microsoft products that are then being offered to other large companies that aren’t going to build their own Customer Success Management system, if you will. So that’s also where the scalability comes into place. So that was a horizontal example. A lot of my colleagues are working on very deep, vertical solutions. And I think what I would say, generally speaking, is we’re talking about ones that are really, really, really data intense, you know. So we talk about and hear about drug discovery. We hear about oil and gas, just very data intensive, computationally intensive sources of data, if you will. So these types of workflows require deep, deep domain expertise in order to do it. So you will never have a software developer working on that on their own. You know, they are in service to the scientists, the chemists, the you-name-it, who really have the vision for where that particular unique asset and knowledge base needs to go.
Serguei Netessine 12:23
I want to dive deeper into monetization. Obviously, Microsoft invested very heavily in AI platforms, Azure and Copilot and so on. How do your clients monetize AI? Do you see some patterns and pricing models evolving? Is it usage based? Is it seed based? Is it outcome based?
Tereza Nemessanyi 12:44
I have to tell you, from that perspective, it is still really early days. You know, speaking with a very innovative, digital native CEO shed some insight that I thought would be really interesting here. He’s of the mind that there’s about a six month window to transform their services, their products and fundamentally, the give-get value of their customer relationships with AI. There’s a really, really limited window, and once some type of product market fit hits, they will be moving at such a velocity that it becomes impossible for competitors number 234 to catch up. So that’s the way that I would be thinking about it. I’m hearing people talk more about market share within their competitive set and their positioning, and that it is historically during times of great disruption where massive market share gains can be made and lost.
Serguei Netessine 13:43
Interesting. Let me ask you maybe a somewhat different question. You already started alluding to the fact that you see some success with AI in high-cost-to-serve applications like contact centers, customer service and so on. Do you see any patterns in how companies are proving that AI actually works, that it delivers value? Do you see a particular ROI, some performance metrics that companies are trying to track to prove that AI is actually improving productivity or growing revenues, or doing both of those things?
Tereza Nemessanyi 14:21
I know you want numbers.
Serguei Netessine 14:23
Always.
Tereza Nemessanyi 14:24
Always, and it’s tricky in the sense that we’re talking about very, very different workflows. We know that, let’s say, with GitHub, Copilot, we saw quite quickly a 30% improvement in productivity. That’s a nice, solid, well-researched kind of a number for just taking a workflow and dropping some steps, leapfrogging some steps, compressing some steps, that sort of thing. And the example that I gave to you before, where you’d be talking about creating or versioning products in order to get at the long tail, those will probably be almost completely automated products. So the margin on that is completely different, right? I mean, that’s where you’re you’re absolutely looking for scale. Another place where I see a lot of success is in business applications that draw on a very, very large corpus of data. So let’s say things like regulatory you know, where you have lots of different rules coming from lots of different places, and those need to be evaluated for compliance and possibly for some sort of monetization or whatnot, you know, on the fly. And AI, in the form that we have today, is fantastic for that. When you think about what large language models, and then also small language models are really, really good at it’s, it’s drawing on, you know, information that’s already there and that you have confidence in its applicability to the problem at hand. Which is a reason why I give that example of regulatory because it is an existing set of data. It’s just more than a human being can handle.
Serguei Netessine 16:03
Makes a lot of sense. So from monetization for your clients, I want to ask you about monetization for Microsoft. So how do you think about monetizing AI across all of your products? Are there any lessons you can share on, you know, bundling, up-selling, creating an ecosystem. What are you seeing there?
Tereza Nemessanyi 16:24
It’s our job to empower developers and business users to create what they need in order to achieve more. They are the experts in it. We are not. So we imagine a world and an ecosystem of best-in-class tools that are constantly up-to-date. You know, a place you can go and leverage the very best, the very newest. So, for example, we have over 1800 models in Azure AI foundry today. You know, I hear a lot on Wharton chats and whatnot about “What about this model? What about that model?” It drives me crazy. The real question that should be asked is: :What is a company’s AI capability? What’s that capability over time, as the world of AI changes, and then, what is that impact on the business?” So, imagine managing AI as a capability. Imagine a world where there is not a single model to rule them all. Managing AI means picking the best models for a particular point in time. Now, the efficacy of any model could be shifting from one second to the next as well. So you probably need an overlay of AB testing of models that are in use at any time. Like just managing all of that can get really, really complex. And so that’s where Microsoft is playing really big. That’s where we’re making our biggest investments. We don’t want to be the king maker of this model or that model. We want to say: “Developers, you tell us which of the best models. We’ll make them available. We’ll make them delightful to access.” So it’s absolutely an ecosystem play. There’s no question about it. You know, we have a lot of people are creating their own Copilots, their own agents, for their own processes, and that is as it should be. So let’s say in M365 you know, we get paid on usage of the platform. So the more useful The platforms are, the better that is for us. It’s a really elegant alignment between, you know, who our customers are, and who is paying us for our value and providing super, super tight feedback that they’re getting what they need.
Serguei Netessine 18:32
I like this KPI of usefulness. So you know, people pay you for being useful, and the more useful you are, the more they pay. And AI, of course, makes Microsoft as an ecosystem more useful. I want to move to our final segment, where I want to discuss challenges on the road to monetization. So of course, many companies struggle with turning AI into scalable and repeatable systems. What kind of blockers do you see to monetize AI effectively for your clients?
Tereza Nemessanyi 19:05
I think the ones that have the best grasp on their customer, and what delivers truly a differentiated end customer experience provides that true north for working backwards and then you know, you apply the tools like you would apply any other tools toward it. When you know your customer and when you know what differentiation means for them, you’re getting close to the data around “What will they pay more for? What are they not paying for that they would”, you know, et cetera. So it’s very easy in this highly technical world to look for quick fixes by just diving straight into the data. And that’s not wrong, but that that true north of customer understanding, there’s no substitute, and I find, more often than not, banging up quantitative versus qualitative data back and forth is kind of where the magic starts to happen. A second piece, I would say is that, generally speaking, these kinds of initiatives are highly cross matrix from a functional perspective, and it is really hard to get specialists from different areas to speak the same language and rally against a similar goal. They’re generally very busy doing their day jobs, so freeing them up to ask these provocative and bigger questions, not so easy, especially when you know management is already looking to define an ROI in order to deploy the person to spend time on it. So it becomes a self-fulfilling prophecy going after cost savings or reduce revenue leakage, if you will. It’s just measurable. So the ROI is there.
Serguei Netessine 20:42
Okay, wow, this is a wonderful and powerful list of things to worry about. Does Microsoft play some role in helping your clients to move kind of from a pilot to something that actually scales?
Tereza Nemessanyi 20:56
Oh, yeah! We do a lot with envisioning sessions, with sharing examples of capability and facilitating them with management teams to find relevance for these freshly new capabilities within the context of their business. But they’re the ones that know their business, so they know if that’s going to become the source of differentiation, if you will. The second piece, so monetization and scaling, that’s a big yes. This is really, really important to Microsoft, and this keys into, you know, the ecosystem play. Our partners who’ve built amazing products with Microsoft Cloud, it’s completely in our interest to have them succeed. It is in our interest to make them available, create pull-through to our customers, to our shared customers, as well as our customers whom those partners haven’t reached yet. We’re all about marketplaces, and this is not unique to Microsoft, but our view of it is extremely partner-centric and hyper aligned towards shared positive results. So in my area, which is the partnership area, I mean, we’re essentially looking for the next generation of such partners to be in our ecosystem, because they bring either the horizontal or the vertical expertise that we don’t have, but which our shared customers require. It’s our greatest privilege to work with the best partners in the world to drive amazing outcomes. So if there’s anyone listening to this podcast that would like to think hard and deeply about the world of the possible, we would love to work with you.
Serguei Netessine 22:39
Wonderful. That’s, that’s a great ending. It’s been a real pleasure speaking with you today, Tereza, and thank you for giving us a peek behind the curtain of one of the world’s most iconic tech companies.
Tereza Nemessanyi 22:51
It’s been a pleasure.
Serguei Netessine 22:53
Let me review the key takeaways from my conversation with Tereza today. We’ve heard again that AI deployment is still in very nascent stages. Nobody really has figured out how to monetize AI in kind of, in a one way. Every company is trying different approaches. You have to experiment, you have to have a proper culture for this. You have to run through short sprints and see what works, see which vertical or horizontal application seems to be the most fruitful. There is not going to be one solution, one business model that works in every single application. We saw some emerging successes. We saw big productivity gains using GitHub. We saw some successes in, for example, handling large sets of data, which is very difficult to do for humans. But beyond that, you have to really look for some high cost to serve areas where there is just an inherent pain point that can be solved by AI and again, work backwards from your customer.
Serguei Netessine 24:03
That’s all for today. Thanks so much for listening. Please follow us so you don’t miss an episode like our upcoming season finale featuring Ajay Anand, the SVP of Global Services Strategy and Business Services at Johnson and Johnson. This has been Where AI Works, conversations at the intersection of AI and industry, brought to you by Wharton in collaboration with Accenture. I am Serguei Netessine, goodbye for now.
Episode 4
Guest: Ajay Anand, Sr. Vice President, Johnson & Johnson
Transcript: Episode 4
Ajay Anand 00:01
Our employees have to navigate roughly 10 million interactions across a very wide web of systems and applications, which we believe is somewhat fragmented and also adds complexity. So we started out with a question or a proof of concept or experiment, and what we found was that the degree of accuracy that we were able to get by deploying Gen AI was very high, and that it could truly act as a good assist for our employees who are a part of our contact center. And as we then further looked into our processes, is when we decided that we wanted to use this as a capability across all of our employees.
Serguei Netessine 00:44
Hello and welcome to the season finale of Where AI Works, conversations at the intersection of AI and industry, brought to you by Wharton in collaboration with Accenture. I’m your host, Serguei Netessine, Professor of Operations, Information and Decisions at the Wharton School, and I’m also Senior Vice Dean for Innovation and Global Initiatives. It’s our goal to cut through the noise, to deliver actionable insights for business leaders by combining cutting edge research with real-world case studies. Things are changing fast, so without further ado, let’s dive in.
News Clip One 01:22
AI has the potential to change many ways in which we’ve thought about society, about what we’re able to do.
News Clip Two 01:29
We have the opportunity to leverage this change, to innovate, to be creative and to find the opportunities for growth.
News Clip Three 01:36
This is going to impact every product, across every company, and we are just in early days.
Serguei Netessine 01:42
On this final episode of season two, we are going to discuss how artificial intelligence is transforming service infrastructure in one of the world’s most complex and highly regulated industries, pharmaceuticals and medical technology. AI has the potential not just to automate tasks, but to augment workflows, advance decision making and reimagine how people and processes come together at scale. In my own research at Wharton and the research of others, we’ve learned that AI works in applications like coding. There is lots of evidence for that. There is some evidence that it works in consulting, but there is really limited evidence that we’ve seen so far in the context of such a large and complex organization. Which is why today, joining me for this conversation, it’s my pleasure to introduce a man who oversees strategic planning, data management, intelligent automation, and much more at one of the world’s most influential med-tech and pharmaceutical companies. Ajay Anand is the Senior Vice President of Global Services, Strategy and Business Services at Johnson and Johnson. Ajay, welcome to Where AI Works.
Ajay Anand 02:57
Serguei, great to be here and look forward to talking to you about AI.
Serguei Netessine 03:01
It’s great to have you here to wrap up our season of the podcast. Before we dive into AI conversation. Could you tell us a little bit about yourself and your role at Johnson and Johnson?
Ajay Anand 03:12
So as you said, I’m a part of our shared services organization in Johnson and Johnson called Global Services. As a shared services organization, we deliver HR, procurement, finance, and commercial services to our employees, customers, and suppliers. We are focused on helping increasing and improving J and J’s competitive advantage, simplifying ways of working for all of our employees, and continuing to improve experience for our customers, suppliers, and employees. We were launched about 10 years back. In addition to all the great work that we are doing in the services space, we also have some really strong capabilities, whether it’s around data management, continuous improvement, transformation, to name a few, and digital is one of our core capabilities. And we take pride in being a company that prioritizes and operates with a digital-first mindset. What that means is integrating digital into our ways of working and constantly and continuously reimagining all of our processes and services. And today, as we continue to talk, I’m gonna talk about one of the key examples called JAIDA, which is J and J’s AI digital assistant, which has been launched to our 138,000 employees, and something that we’re very actively involved in and very proud of.
Serguei Netessine 04:38
So really, it sounds like your organization is kind of a backbone of Johnson and Johnson, if you will.
Ajay Anand 04:44
Yeah, it’s very well said, Serguei. Our CEO actually calls Global Services the backbone of Johnson and Johnson, because of our role across many of our processes that serve employees, customers, and suppliers. Well said.
Serguei Netessine 04:56
Okay, thank you. Thank you for this background. So let’s jump right into AI and tell us a little bit about how AI is being integrated in those Global Services, and how AI is contributing to the broader use at Johnson and Johnson.
Ajay Anand 05:12
So artificial intelligence is integrated into many of our services and our processes. As I said earlier, I’m gonna use the J and J AI Digital Assistant, also called JAIDA, that we use to serve our 138,000 employees, as an example to go deeper. Our employees have to navigate roughly 10 million interactions across a very wide web of systems and applications, which we believe is somewhat fragmented, and also adds complexity in the journey of our employees. And what we are doing with JAIDA Gen AI, which we have in production at this point, with a few aspects of how we serve our employees and our customers, is:number one, simplify the ways of working for our employees. As you said when you were kicking it off, we also see Gen AI and AI as a very good way of assisting our employees when they’re dealing with a lot of complexity. We are using data in our contact center, where an agent has to navigate through a lot of different policies when they’re responding to an employee as an example. So we use data as a way to assist them in commercial we are using it for order management so that the associates are working with our customers are able to do it. We also view it as a very good capability to augment our employees, where you have to navigate a lot of information, where a lot of complexities, where AI can help them with a good starting point. And we have seen this in contracting as an example. And what we are striving for is maximizing the degree to which we are able to enable our employees with self service in the future, leveraging AI. So that’s one example that I would share with you that we are super excited about and continue to integrate deeper into the experience of our employees in J and J.
Serguei Netessine 07:00
So how did you start on this journey? Was it immediately kind of an enterprise-wide implementation, or was it a series of small experiments with one kind of a service at a time, and then going deeper and deeper?
Ajay Anand 07:14
So we started out with a question, or a proof of concept or experiment, as you may want to call it, where we wanted to look at what the real power of AI and Gen AI would be in this specific example. As I said, we are applying AI very broadly across all of our services and processes. And what we found was that in our early experiments, the degree of accuracy that we were able to get by deploying Gen AI was very high, and we found that it could truly act as a good assist for our employees who are a part of our contact center. And as we then further looked into our processes and these complexities that I talked about, specifically the 10 million interactions that are fragmented, is when we decided that we wanted to use this as a capability across all of our employees. So it started with an experiment and a pilot.
Serguei Netessine 08:07
What kind of KPIs did you use to assess how much help AI offers, how much productivity is enhanced by using AI tools?
Ajay Anand 08:17
Serguei, great question. So in Johnson and Johnson and in Global Services, we use a framework, we call it our 3 E framework, where one of the E stands for experience. How are we able to use AI, in this case, to help transform the experience for our employees, our customers, our suppliers? Are we able to show that the experience became simpler, it became faster, it became optimized, and we have metrics associated with it. We are able to measure their satisfaction using KPIs like CSAT or net promoter score. The second E stands for effectiveness, our ability to show that we have taken some of our core processes and they are now able to deliver better outcomes and better results, and we have made the process better. A good example would be, have we improved the forecasting capability that we have? And then the third one is around efficiency. Are we able to show that by utilizing AI, we are able to do things faster, we are able to do things with a lower cost to serve. We are seeing fewer errors associated with the work, and I’m very happy to share that what we have found is, in all cases, with the work that we are doing with AI, we have seen tremendous impact across all three of those E’s, and then we are able to connect them to some of our mega KPIs at J and J level, which is what has led to very senior levels of sponsorship and excitement for this effort.
Serguei Netessine 09:46
So to wrap up the first part, we hear that implementation of AI went in this sequence of small experiments, targeted experiments, and then now it consumed entire company. Started with maybe contact centers as kind of an assistant, and now it’s kind of weaved through the fabric of the company. The reason this decision was made is because really on all the KPIs that you measure, Net Promoter Score, effectiveness, cost to serve, reduction of errors, AI outperformed your expectations, which is why a decision was made to go all the way forward with the digital-first implementation of AI.
Serguei Netessine 10:34
I want to ask you, Ajay, how did implementation of AI impact workforce dynamics? How do you now think about metrics performance? How did the nature of work evolve for your employees and most of your employees, I imagine, these are white collar employees, so they have intellectual work. So how do you think about this collaboration between these transformative technologies and your employees?
Ajay Anand 11:02
Great question, Serguei. What I would highlight is that we saw a very strong adoption from our employees, mainly based on one key aspect associated with automation and AI as a capability, and it’s specifically the ability for employees to shift their time to more value creation, their ability to work on more high impact and strategic work for Johnson and Johnson. And as I said earlier, what we saw is that this is acting as a capability that can assist the employees. It can augment them in a way to amplify their overall impact, and that then leads to increased level of engagement in our employees and us just seeing increased effectiveness. What we have also seen is, I’ll use contact center as an example, this has led to our contact center associates being able to acquire new digital skills, and that’s just driven a very high degree of excitement and engagement from them. And as you pointed out, we are also seeing that this is leading to creation of new roles, which is requiring us to work on upskilling strategies for our employees.
Serguei Netessine 12:16
That’s a really tremendous impact. Can you do a bit of a deep dive on metrics that you used for assessing effectiveness of employees now that they are supplemented with this AI tools. Did this change? Or could you just continue using the same effectiveness metrics as you used before?
Ajay Anand 12:34
Yeah, I’ll speak to some macro level themes associated with it so that I don’t have to get into any specific metrics associated with it. We look at the ability for our employees to have capacity to do more strategic work. We have seen that improve. We have also seen this creating, and again, I’ll stick with the contact center example, our employees now being able to get exposed to more cross functional work than what they have done in the past. And as a result, their cross-functional acumen is increasing. We have seen the ability of our employees to start working with more complex cases, if you will, as a result of how AI has been able to augment them.
Serguei Netessine 13:15
Can you say anything about how you now think in terms of hiring people, selecting people, interviewing people. Do you require any new skills from your employees, or is it kind of more or less unchanged?
Ajay Anand 13:28
So there are some changes, I’m not going to be able to get into the specifics. The one part that I will say is that we are certainly using AI as a capability within what we do in Global Services to also assist our Talent Acquisition team with doing the same thing that I talked about with contact center. How are we going to use AI to assist them, augment them in the work that they do around talent acquisition.
Serguei Netessine 13:53
Okay, wonderful. So what I am hearing is that there was a tremendous impact of AI on employees. Employees are now able to spend more time on creating value and spending less time on non-value-adding tasks. We see increased engagement from employees. We see employees developing new digital skills. In particular, there is more cross-functional training and cross-functional work that became possible because it’s easier to learn adjacent areas for an employee using AI tools. And of course, there is an impact on on talent acquisition as well. So it sounds like the workforce was really affected in positive ways.
Serguei Netessine 14:39
What can you tell us about your outlook for the future for these AI tools that J and J is now using. What lessons can you share with other business leaders who are just thinking maybe about implementing those AI tools? And also, what are you thinking about next step in Global Services? How will you continue evolving the role of AI for J and J?
Ajay Anand 15:01
I’ll answer that in two parts, just to make sure that I share my thoughts effectively. So as I think about the future of AI for us in Global Services at J and J, as you also said, we are the backbone of Jand J, the breadth and depth of everything we do in Global Services plays a very, very crucial role in value creation for Johnson and Johnson. And keeping that in mind, our goal is to continue to transform into an AI first enabled digital operating model, and also embracing intelligent operations and everything we do so that we are able to shorten cycle time, improve our decision making, continue to make strides in what I shared around our three E’s, improved experience, improved effectiveness, and improved efficiency. What we have also seen with the breadth and depth of all the work that we do across many of our key processes that serve our employees, suppliers, customers and other stakeholders is Global Services has to play a very key role in ensuring that our content and data quality is where it needs to be. So our ability to play a role in driving AI-ready data and content is significant. What we have also seen is that we have to continue to play a key role in bringing in what I call external lens with our peers and many partners to help drive ideation, help with focusing on the right priorities across all the various partners that we work with to serve these stakeholders. And then finally, I’ll say our ability to really measure and trace value, right? ROI is critical in continuing to drive traction from our senior leaders and all stakeholders. So that’s what I would share around future vision. As it relates to part two of your question around what might be the advice that I would have for our peers and others as they engage in the space, whether it’s with traditional AI, generative AI, or agentic AI. Number one, developing a strategic mindset. As opposed to looking at AI as a technology or as a tool, how do you go about integrating it into your business model? How do you use it as a way to reimagine your business model, your strategies and processes, is number one. Number two is getting very strong leadership support. So we are fortunate to have the support from our senior most leaders on all the work that we are doing in this space. But that becomes extremely critical. Focusing on change management is another topic, right? So this is where you’re introducing new ways of thinking, new ways of working. Having a very strong narrative that can be embraced by your employees and other stakeholders is critical. So focusing on change management becomes really key. I talked about responsible use of AI earlier, which is a foundational element of being successful, creating a culture of learning an experiment. You talked about how our journey started with us wanting to do some experiment and POC with this new space. I’ll then get into storytelling. What we have found is that when you have good impact with some of these use cases, how do you go about sharing that success story in a way that momentum breeds more momentum, which has been really key as we have continued to scale this across all of our processes. And then finally, just staying on top in a very proactive way to manage all of your external risks becomes extremely key.
Serguei Netessine 18:41
That’s a great list. Yeah, of course there are, you know, technological part of innovation, but then there is organizational change and culture change and making sure that stakeholders are engaged and involved, and that very often, is the hardest part for a big company like Johnson and Johnson, I imagine. Quick follow up on this, as you develop these AI capabilities, are you continuing to do this internally? Are there any thoughts of on partnering with startups versus doing these advancements internally, versus partnering maybe with some big tech companies? How do you think about the splitting innovation between inside the company and outside the company? Because, obviously, for a pharmaceutical company, it’s very common to co-develop drugs, co-develop technology, with other companies, with other startups. Do you think the same way about AI, or is it different in your mind?
Ajay Anand 19:35
Great question, and you also gave me a point that I can use to respond. Very similar to what we do in our normal ways of working as a business, as you pointed out, we obviously look to leverage our overall ecosystem, but by being very mindful of responsible use of AI and managing all the risk exposure and all the other factors that are very critical in this space. So it is at leveraging the full power of our ecosystem.
Serguei Netessine 20:05
Yeah, that’s great to hear. We just finished the report on collaboration between big companies and startups, and certainly it looks like in the AI space, a lot of big companies are going in the way of collaborating a lot with external organizations and thinking about leveraging ecosystems. So that is great to hear. Any last thoughts, any last advice to executives at other companies?
Ajay Anand 20:29
My only point that comes to mind, beyond what I just shared as some thoughts for other peers and senior leaders is it’s a very dynamic and agile space, and staying connected with what’s going on in the industry with your peers and other partners in the ecosystem becomes extremely critical and just the need to be highly adaptive as a result of how dynamic this environment is.
Serguei Netessine 20:53
Wonderful. Thank you. Ajay. This has been a really insightful conversation and a great way to wrap up the second season of the podcast. Thank you, Ajay, for taking time to share your expertise with us here on Where AI Works.
Ajay Anand 21:08
Thank you, Serguei.
Serguei Netessine 21:12
Now let me review a couple of takeaways from conversation with Ajay. So once again, we see that AI implementation is a very dynamic space, and you really need to stay connected. You really need to stay involved in the industry and understand what are the new tools coming up and beyond all, you need to experiment. As you saw once again, Johnson and Johnson started AI implementation from smaller experiments, looking at a number of KPIs, assessing them, and then moving forward, area by area, job function by job function. And what we see in case of Johnson and Johnson is a great example where AI lives in harmony with humans. AI lives in harmony with white collar workers who became more efficient, who became more productive. They spend more time on value creation. They develop new digital skills and cross-functional training. And this is likely a blueprint for implementation for other large companies, which will need to think carefully about processes like change management, think about stakeholder involvement, think about ability to experiment and getting support from the very top of the company. Otherwise, a big initiative like this would probably fail.
Serguei Netessine 22:39
That’s a wrap up on season two of the podcast. Thank you so much for listening. I hope you enjoyed those conversations and learned something along the way. Season three is coming soon. It will be hosted by my colleague, Professor Peter Cappelli. Be sure to follow us so you don’t miss an episode. This has been Where AI Works, conversations at the intersection of AI and industry, brought to you by the Wharton School in collaboration with Accenture. I am Serguei Netessine. Bye for now.
Episode 5
Host: Serguei Netessine, Reflections on Season Two
Transcript: Episode 5
Alan Lee 00:01
Steve adding AI into the mix is gonna enable a more nuanced understanding of where and when we can charge customers.
Elad Walach 00:09
There’s so much more value to unlock from an enterprise perspective, as we think about system efficiency, not individual efficiency.
Tereza Nemessanyi 00:16
We imagine a world and an ecosystem of Best in Class tools. It’s absolutely an ecosystem play. There’s no question about it.
Ajay Anand 00:25
By utilizing AI, we are able to do things faster. We are able to do things with a lower cost to serve.
Serguei Netessine 00:36
Hello and welcome to Where AI Works, conversations at the intersection of AI and industry, brought to you by Wharton in collaboration with Accenture. I am Serguei Netessine, back in the hosting chair one last time for a recap and review of our second season. Our guests shared a lot of interesting insights in those four episodes. So I want to share some highlights and break down my key takeaways for you. As I’ve been saying all along, things are changing fast, so let’s dive in.
Serguei Netessine 01:09
We started the season by speaking with Alan Lee, the CTO of Analog Devices. He emphasized that while a lot of focus is on the innovations at the core of AI that is huge, large language models offered by companies such as OpenAI and Anthropic, much more innovation in the business models will be happening at the edge, where monetization involves interactions between Internet of Things, devices and robotics, with smaller AI models more efficient, consuming less power.
Alan Lee 01:43
I usually use an eighth grade earth science model to describe this. In the center, you have the core, and that’s the large scale mass of AI systems. In between, you have the mantle, and you could think of this as industrial AI, or AI for automotive where you have enough compute power there, it’s not a supercomputer, but enough compute power to do some really viable AI. And then at the edge you have sensors, where you have phones, you have not as much compute power, usually small form factor, but you can indeed do some intelligence there. And it’s not that one of these will be dominant over the other in the end, we’ll have the right intelligence at the right place at the right time, and that’s when we’ll really be starting to extract full value from these complex systems.
Serguei Netessine 02:28
In my own research, I demonstrated that contracting on performance is often beneficial, and as Alan points out, with the Internet of Things paired up with the AI, this is finally becoming easy to accomplish, and it will lead to some new business models.
Alan Lee 02:46
Adding AI into the mix, along with, of course, big data, large computation and AI at the edge, it really is going to enable a more nuanced understanding of where and when we can charge customers, and where do they see value, and how can we monetize and capitalize on that?
Serguei Netessine 03:05
In episode two, I discussed AI implementation in healthcare with Elad Walach, CEO of Aidoc. Elad insisted that to realize true value from AI, simple applications, so called point solutions, will need to be completely replaced by AI first business models such as the entire organization becomes AI-enabled. And the reason is that the benefits of AI often accrue to some part of the company other than the direct user of AI.
Elad Walach 03:37
When we started selling point solutions, it was very easy for us to identify the buyer of the solution and the beneficiary. We had the same who’s the user, who is the same as the buyer. It was very easy, we always had for this initial solution, the radiologist at heart. Over time, we started learning that actually you have to think about it, not from an individual perspective, but from a system perspective, especially to build a platform.
Serguei Netessine 04:01
So this message is consistent with the key finding of my book on business model innovation, where we conclude that misaligned incentives in a business model often lead to inefficiencies. AI can serve as a layer that aligns all the players in the business model and leads to a superior efficiency of the entire business model.
Elad Walach 04:25
There’s so much more value to unlock from an enterprise perspective, as we think about system efficiency, not individual efficiency. As an example, if a radiologist needs to diagnose a brain bleed, he doesn’t get paid more or less if he diagnoses it maybe an hour later. But you know who does care about it? The ED physician that is now reliant on this diagnosis to discharge the patient. So sometimes you have to think from a system perspective, the user may not be the main beneficiary.
Serguei Netessine 04:56
My third guest this season was Tereza Nemessanyi, from Microsoft, who acknowledged that we see lots of gains from AI in writing code, individual productivity, but also in security and product innovation through data driven product personalization. Tereza said, and I completely agree, that there is a limited window to make progress with AI for most companies out there.
Tereza Nemessanyi 05:22
Speaking with a very innovative, digital, native CEO, he’s of the mind that there’s about a six-month window to transform their services, their products and fundamentally, the give-get value of their customer relationships with AI, and once some type of product market fit hits, they will be moving at such a velocity that it becomes impossible for competitors, number 234, to catch up.
Serguei Netessine 05:51
Tereza also told me is that for all companies, managing AI should be a capability and an ecosystem play, whereby different startups and large companies come together to make AI useful.
Tereza Nemessanyi 06:04
We imagine a world and an ecosystem of Best in Class tools that are constantly up to date, you know, a place you can go and leverage the very best, the very newest, and so that’s where we’re making our biggest investment. We don’t want to be the king maker of this model or that model. We want to say developers, you tell us which are the best models, we’ll make them delightful to access. It’s absolutely an ecosystem play. There’s no question about it.
Serguei Netessine 06:33
My final guest of the season was Ajay Anand from Johnson and Johnson. He explained that experimentation is the key with AI, and we are now in the experimentation stage for most companies, mostly in very, very early stages of this experimentation.
Ajay Anand 06:51
We started out with a question or a proof of concept or experiment, and what we found was that the degree of accuracy that we were able to get by deploying Gen AI was very high, and it could truly act as a good assist for our employees who are a part of our contact center. And as we then further looked into our processes, specifically the 10 million interactions that are fragmented, is when we decided that we wanted to use this as a capability across all of our employees. So it started with an experiment and a pilot.
Serguei Netessine 07:24
Ajay also referred to triple E, EEE measurement framework. Experience, Effectiveness and Efficiency, which is used at Johnson and Johnson to produce specific KPIs for implementation of AI.
Ajay Anand 07:39
Are we able to show that, by utilizing AI, we are able to do things faster, we are able to do things with a lower cost to serve, we are seeing fewer errors associated with the work. And I’m very happy to share that what we have found is, in all cases, with the work that we are doing with AI, we have seen tremendous impact across all three of those Es, and then we are able to connect them to some of our mega KPIs at J and J level, which is what has led to very senior levels of sponsorship and excitement for this effort.
Serguei Netessine 08:14
Our guests shared a lot of valuable advice and insights over the course of those conversations, but my main takeaway for business leaders is something that all of my speakers mentioned. AI implementation is not a technical problem. It is beyond all a leadership problem. It requires change management, it requires company to have superior experimentation capabilities, and those are essential parts of any technologically-enabled transition, and in that sense, AI is not all that different from other technologies that we’ve seen before. If the organization lacks this fundamental capability to experiment, it will not succeed in creating an AI-driven business model.
Serguei Netessine 09:00
This has been Season Two of Where AI Works, conversations at the intersection of AI and industry, brought to you by Wharton, in collaboration with Accenture. Thanks so much for listening. Please follow us so you don’t miss an episode in just two weeks, we’ll be launching season three, hosted by my great colleague, Peter Cappelli at Wharton. He will be focusing on how AI is transforming workplaces. And of course, if you haven’t listened to every episode from season 1 and 2, I’d really encourage you to do so. I’m Serguei Netessine, goodbye for now.
Season 1: The Impact of AI in Marketing
In Season 1, Wharton Professor Kartik Hosanagar explores AI’s transformative role in marketing—how AI-powered social listening, digital advertising, and creative automation are reshaping strategy, customer engagement, and brand trust.
Episode 1
Guest: Jonathan Halvorson, SVP, Consumer Experience; Mondelēz International
Transcript: Episode 1
Jonathan Halvorson 00:01
With everyone having these tools to be able to produce creative, what differentiates people? I think what matters a lot is going to be the differentiation of the brand and the foundations. What is the product truth that is core to the brand? What is the brand purpose? What is the tension it plays against? And if you define those things clearly, then I don’t think that you’re going to have that swell to the average and to the mean. But for brands who aren’t clear on exactly who they are in their foundations, I think they will find themselves in a sea of sameness.
Kartik Hosanagar 00:34
Hello and welcome to the premiere episode of Where AI Works, conversations at the intersection of AI and industry, brought to you by the Wharton School and sponsored by Accenture. I’m Kartik Hosanagar. I’m a professor of technology and digital business at the Wharton School and also co-Director of our AI research center, and it’s my pleasure to be your host on this season of the podcast, we’re going to tackle some big questions about how AI is shaping the world of business today and where it’s going to take us. It’s our goal to cut through the noise and to deliver actionable insights for business leaders. On every episode, we’ll combine cutting edge research with real world case studies and uncover how the best companies are using AI to upskill their workforce and transform their business. Things are changing fast, so without further ado, let’s dive in.
News Clip One 01:24
We are definitely in an AI hype cycle, and artificial intelligence is a very big and very important macro trend.
News Clip Two 01:31
The race is no longer just about scale or speed, it’s about intelligence that can think and reason.
News Clip Three 01:37
AI is replacing human tasks faster than you might think.
Kartik Hosanagar 01:41
AI has been around since the 1950s and computer scientists have been working on it for several decades now, but it’s really taken off in the last two years, and all of a sudden it’s become extremely important for the C suite. It’s likely to impact everything from marketing and sales to software development, R and D and other functions, and likely across all industries. This inaugural episode of Where AI Works starts with marketing, and in particular, how is AI super charging the way brands are creating content to help me unpack the answers, I am pleased to introduce the global Senior Vice President of consumer experience at Mondelez International. Jonathan Halvorson, Jon, welcome to Where AI Works.
Jonathan Halvorson 02:25
Thank you so much for having me.
Kartik Hosanagar 02:26
Not everyone is likely familiar with Mondelez, but I suspect the brands that are within Mondelez are known to all of us. So maybe I’ll ask you to first just introduce Mondelez and tell us a little bit more about the company.
Jonathan Halvorson 02:39
Sure, Mondelez is the global leader in snacking. We operate in a few core categories that you’re very familiar with: chocolate, biscuits, baked snacks, you’d be familiar with: Oreo, Milka, Cadbury, Ritz. So, a true combination of global and local brands operating everywhere around the world,
Kartik Hosanagar 02:57
From what I understand, your background in college was investigative journalism. So I’m kind of curious how you go from investigative journalism to advertising, and in particular, Mondelez as well.
Jonathan Halvorson 03:08
So I attended the University of Missouri undergraduate program in school of journalism, and why I was there, I was sitting in history of American journalism, and they were doing a preview of all the sections. It’s that moment where you decide, am I going to be a newspaper or a broadcast? And a professor, Steve Capch, came in to present what the vision was for STRATCOM, and he had an unbelievably brilliant articulation of exactly what STRATCOM was. And it was the just really the compelling nature of it, how he talked about writing, the clarity of thought, and just really building brands. And I sat there, and I scrapped all my dreams of being an investigative reporter or working a broadcast beat, and I signed up for advertising. A few short years later, I would go to work at Leo Burnett and later on, in media organizations, OMD, Publicis, working around the world with a lot of blue chip clients. And fortunately, I had Mondelez as a client, and later on, had a chance to join them about eight years ago in the capacity I’m in now running global media, digital data, creative and digital commerce. For us, I think ultimately AI is going to work from the bottom of the funnel and come up. So the beginning uses of AI that we see are really in D commerce, building product display pages. So, the images for that, the text optimizing that, for search engine optimization inside retailers. The second use case that we see is really in producing everyday advertising content, the display assets and banners you see that are meant to drive conversion and are slowly working way up to social assets. So lifestyle imagery that you would see and Instagram posts and Tiktok, and we continue to progress up, and we see a clear path to doing video and even our above the line TV assets.
Kartik Hosanagar 04:09
Got it and in fact, I think advertising and content creation is a great place for us to kick off this series of conversations we’ll have on the show on AI and marketing. And I want to start by sharing a study that some of my colleagues at Wharton did where they were looking at which industries and job types are going to be most impacted by AI, and the top two in that list were media and entertainment, and legal. And of course, media and entertainment includes advertising. Where I want to begin the conversation is perhaps even the very basics, which is, where exactly can you apply AI within the content creation space in advertising? Where are you folks using it today?And it’s interesting, you said you’re starting at the bottom and working way up, meaning like these asset creation and hopefully impacting strategic choices over the long run.
Jonathan Halvorson 05:37
Yeah, you’re starting with lower funnel assets that are more conversion, but I think the systems you’re building are really end to end, and I think that’s where you’re going to get the value. And I think your Wharton colleagues are speaking to that. That’s why it’s not augmentation, but true disruption. Because I think AI is going to help you do everything from creation of the brief all the way to actually traffic that asset and put it out into market.
Kartik Hosanagar 06:00
That makes sense. But one question I have for you is, in this early phase, are you already seeing any evidence of ROI? And I ask that because there’s so much hype around AI, but many skeptics are also starting to raise these questions about all the spend, where’s the return? What are you folks saying?
Jonathan Halvorson 06:17
Over the last two years, we have done 40 campaign based AI activations. So this is example wearing campaigns. We’re using AI technology to augment and deliver a higher efficacy or a higher efficiency of our assets in those campaigns. We can clearly point to that not only being some of our strongest work, but also in those campaigns, delivering higher brand awareness, higher market share gains and higher net revenue. I think that you see a lot of the best of that work inside Mondelez on Cadbury. I’d point to two specific examples. First is give a cheer for a volunteer in Australia. Last year, our top performing BU was Australia, and the top performing brand Cadbury, at the center of their campaign activation was a program that allowed consumers to create an AI film that celebrated a volunteer in their life. And it’s something that wouldn’t have been possible without AI. We can’t hire an artist or a film director to produce all those films and be able to make that possible. Similarly, inside India, last year for Valentine’s Day, we created an ability, through packtivation, for consumers to create Valentine’s stories and tell the entire story of their love. It was really geared towards people who are, perhaps not the planners, but rather last minute gift givers and a little bit of also helpless romantics who want to tell those stories. Those campaigns aren’t possible without it, and there are two of our highest performing campaigns, both internally as well as externally celebrated.
Kartik Hosanagar 07:39
Jon, what I really like about the examples you just shared is that the examples are not just about cost cutting with AI use, but it’s about meaningfully getting a lift in outcomes like consumer engagement or attention and things like that. And we’ve seen that actually, we organize an annual generative AI workshop where researchers share the latest research in these spaces, and multiple studies in our workshop have shown that it’s, again, not just a cost reduction standpoint. For example, one study by researchers at MIT and Indian Institute of Management showed that there was a 6 to 9% lift and click through rate of ads, and another one by researchers at University of Hamburg and Emory University compared human-generated ads against AI-created ads for automotive companies, and they used purchase funnel metrics like: Does the ad get attention? Does it generate interest? Does it generate desire? Does it create action, like downloading a brochure and so on. And on a seven point scale, they found those purchase funnel metrics went up by nearly a point, about 0.75 points. So strong evidence. Have you guys tried to measure this as well?
Jonathan Halvorson 08:47
Yeah, the same thing. I think in the very beginning you see higher ROIs, higher conversion rates. But I think it’s also important to note, this is just the beginning. We’re in the early rounds of the ball game. What I think the potential that gets people excited about AI is for that to only improve over time. So today it’s seven points. And just to be clear to any of the listeners, that’s material to the business, when you start delivering 250 trillion impressions in a year, and if you can do that 1% better, that adds up to significant volume gains and significant growth in terms of net revenue, but then it’s just the beginning. Over time, AI is going to cumulatively learn and get better and get better. And I think that’s what really excites us. If we can start from that point, have really good success signals and train the systems, we’re only going to achieve higher levels of efficacy and excellence.
Kartik Hosanagar 09:37
It’s great to see that kind of impact with AI optimized or AI-generated content. I’m also curious about what is the role of the human in this? So what is the human in the loop process here? Because I’m assuming it’s not like a bunch of AI agents doing everything there are humans prompting and humans selecting and so on. So tell me about that role.
Jonathan Halvorson 09:57
A lot of people have a different point of view about this, and I’ve had a lot of good debates with practitioners, colleagues, peers, academics, about where it’s at. And I think where I’m at on this is AI is going to impact every single part of the value chain. And I think it’s going to help humans do their jobs better. And if we look at it the very beginning, like we use AI internally to help us write better briefs. So do we write good creative briefs today? Yes. Can they be better? Yeah. I want them to be perfect every single time. And I think with AI inputs, we’re raising the quality of our briefs. The second thing is coming up with concepts. So we use our tool, our proprietary tool at Mondelez is called Aida, AI D A. It stands for artificial intelligence, data and analytics, and that tool allows us to generate concepts. Are any of those concepts perfect? No, but they help inspire our creative teams and our marketers around the world as to what’s possible. So I think that then gives a jumping off point, and this is where our agency creative directors, at WPP, Publicis, the Martin agency, VCCP, provide their own ideas, elevate those ideas to get to really good scripts, then we can quickly test those scripts, ultimately, by generating a series of images that represent that we test it. What used to take two weeks, can now take a few hours, get a score back, and then we can go produce that asset at a higher quality. So I think the human in the loop really is in a few key areas, like it is giving humans a lot more time to spend on creative excellence, to elevate the ideas and a lot of the place where that uniquely provides value.
Kartik Hosanagar 11:27
Yes, that makes so much sense. I think AI can be a great tool from an idea generation standpoint, idea enhancement and, of course, the actual execution pieces as well. We actually did a study recently where we were looking at AI use in creative writing. So we had a bunch of students who wrote without AI and with AI, and we noticed that AI use, of course, reduced the time it took to write, it increased the quality of the writing as well. But one of the interesting side effects we found was that human-created content in aggregate was more diverse than AI-created content. When everyone is using the same AI tools to write, they’re coming up with similar ideas, whereas all of us have very different biological neural networks in our head, your influences are different from my influence. So we come up with different ideas, but if we both use the same kinds of tools, you end up with similar ideas. We see it now with like PowerPoint presentations, like everyone’s PowerPoint is starting to look similar. Do you worry that if all your people are using these same tools, the diversity of ideas might go down? And eventually what you’re seeing as a lift in performance might actually go down, because everything starts to look the same?
Jonathan Halvorson 12:42
Yeah. Look, this is the huge [unintelligible] challenge. Like everyone’s getting ice cream. It’s the best ice cream, but everyone’s getting vanilla is essentially a good summary of the problem. I think about this a lot, because there will be a massive democratization of AI tools, and anybody, any marketer, any company, no matter how big or small, will be able to produce Super Bowl level creative instantly off of their laptop, if not their phone. 12, 18, 24 months at most from today. So then, with everyone having these tools to be able to produce creative, what differentiates people? And I think what matters a lot is going to be the differentiation of the brand and the foundations. At Mondelez, we’re really blessed. We have an amazing portfolio of over 250 brands that are iconic taste of nation. But what will be increasingly important in the journey we’ve gone on over the last eight years is to very clearly define those brands. What is the product truth that is core to the brand? What is the brand purpose, what is the tension it plays against? And if you define those things clearly, then I don’t think that you’re going to have that swell to the average and to the mean. But for brands who aren’t clear on exactly who they are in their foundations, I think they will find themselves in a sea of sameness. So I spend a lot of time thinking about this, and I think the safeguard on it is: one, the clear brand foundations;two, humans in the loop; and three, just the discipline you put in, just the intentionality of how you use AI and just recognizing the concern.
Kartik Hosanagar 14:10
Yeah, I think that’s a really important point. And I just want to dig a little deeper into that. How do you in practice, implement brand distinctive features in your AI augmented or AI generated content?
Jonathan Halvorson 14:23
There’s a few things. First, it starts by being really clear on what are the distinct features of your brand. And that sounds like a very obvious thing, but it’s a very intentional exercise that has to take place before you ever open up AI or open up a laptop, you know. And so in the case of Oreo, it is a black and white sandwich cookie. It has a very clear embossment. There’s a very clear logo, and look to that brand that has to be clear, no matter what the instance is. Then having defined all those assets that are truly distinct, then you have to train a large language model, and you have to explain it. And we’ve had a lot of work that we’ve done on this across all of our agency networks. Training for Milka, training for Oreo. And it just requires time to essentially upload the entire brand history. And really it’s a technical exercise of training the system as to exactly who you are. So it starts by uploading all those awful brand playbooks that we’ve built over the past few years. Those have immense value. Two, is your archive. You know, the really great brands keep very good archives of all the work that they’ve done. And that’s because often, when you’re in a challenge, you can find your solution in the archive. But also that becomes really powerful training data to explain to the AI system your performance over time, what’s worked, what’s not. Then it’s a matter of as you train your large language model, sitting down and giving it back and forth. Just like you train in machine learning or any other application, there is a back and forth exchange of when a system is showing you something, and you have to provide feedback on that. Yes, that’s right. And no, it’s not. And through all that training exercises, you will create a custom LLM that represents your brand.
Kartik Hosanagar 15:59
Yeah, thanks for that helpful peek under the hood. It’s important to understand that process that goes into that and it’s not just going to ChatGPT and writing a prompt and voila, you have your ad. So we’ve talked a little bit about the process. Let’s talk a little bit about the outcomes. I know you’ve run dozens of campaigns that are AI-supported, but I’d love to hear about one or two in particular. What was actually done? What was the new customer experience and things like that?
Jonathan Halvorson 16:27
Let me answer that question in two parts. First, let’s talk about the journey, and then let’s talk about a specific example. So let’s talk about the journey.
Kartik Hosanagar 16:34
Great.
Jonathan Halvorson 16:34
So 18 months ago, we start doing our first work on this. We trained a large language model. I’m sitting with the agency, and I’m super excited. We’ve done a little pilot, and the first thing it spits out is humans that have six fingers and people who have giraffe necks. And then we would see Milka bars that were missized. Everything was wrong. We tried to prompt it on Halloween, and instead, I get things that looked like Christmas. There were people who were joined at the shoulder. I could have cried right then and there, and I was like, man, maybe this isn’t for us.And the early thing that we’ve done in the small little pilot just showed how hard this was gonna be when you talk about some specific examples, and what does that really enable? I like to go to the India BU, because I think they’ve done some of our most advanced work on Cadbury. The first work that they did was actually in music, and it was for Cadbury birthday song. And so the idea behind Cadbury birthday song was a simple insight: There are literally billions of people in this world, and yet there’s only one song to celebrate them. Happy birthday to you. And so their insight was, wouldn’t it be powerful if you could create a custom birthday song? But the thing is, is you can’t hire a musician to write a billion birthday songs for any iteration, but using AI and some large language models, in partnership with the Ogilvy team, they created a tool where, if you uploaded some basic information about yourself, you got to pick a type of music, you got to pick a language. Ultimately, you would produce a custom birthday song for you. We did work promoting it and above the line, as well as below the line, and it became huge hit across India, with literally millions of people participating, driving incremental sales celebrations outside of the core festive season, when typically almost all of our chocolate is sold. So really extending the chocolate season from just Diwali and Valentine’s Day to really being something that’s 365 days a year. And that’s a powerful example.
Kartik Hosanagar 17:08
Right.Yeah, and by the way, I grew up in India, and so Cadbury was a huge brand in India when I was growing up.
Jonathan Halvorson 18:38
I have to thank Brot Purdy and many of the great marketers that come before us for the great tradition we have in India in marketing, and it’s laid a great foundation.
Kartik Hosanagar 18:46
It’s been great to hear about what’s gone well, how things are producing positive impact and results for you. But let’s talk a little bit about lessons learned, because a lot of companies are just getting started with their AI and content journeys. And I’m curious what’s been the hard part of all this? What have you learned in terms of both what’s been hard that you’ve solved, but also what you’re still trying to solve?
Jonathan Halvorson 19:11
So many lessons learned. I mean, what started as an insight at a moment where we had the flash of brilliance to being here today, has been a long road and a lot of work. I think, you know, lessons learned, and things I would tell one is, one is find a strategic partner to be your sounding board and stick with one all the way consistently. I would say, throughout this entire journey, I have had really good strategic partners, and it’s the one voice in my head and that makes life really simple. The second thing is, I think it’s very important for you to have a clear narrative. Every company will approach AI differently and even the use case of content. I’m very focused on increasing the effectiveness of that content and the quality of it. Others, their business case is more built around speed. How can they take global assets, localize them, get to market faster? Other people are all about cost savings. What I think is important is that you have a very clear business case and narrative around that that you consistently talk about through the organization, and that becomes really important. Third is, most of this is change management. And I’d say that the change management is probably half of the cost and definitely more than half of the time. And so I think it’s important to get people in early to see this and talk about where you’re going. The last thing is, I think you learn from us, is the power of a demo. We did a small little pilot. We had a little example of a Milka large language model that simulated the brand, and we produced this video. And so when we went to go talk to the marketing organization, we went to talk to the C suite, it was an asset we could show. And it was powerful, because often when you start talking about AI, large language models, you can get very quickly lost in all the junk in a series of acronyms that aren’t really going to help you. But this little example became a real shining light, and for people who weren’t even marketers or weren’t familiar with AI, they saw it and it changed their mood. There was a moment, a pivotal moment, where we were deciding if this was gonna be part of our strategy or not, and one of our region presidents thought,he goes, “that’s the future”.
Kartik Hosanagar 21:06
Change management, to bring people along, demos to win people over. The interesting thing, and that’s where I wanna take this last question, is that advertising, at the end of day is all about people. It’s about people with creative ideas who can help brands connect with real people, and now you have AI, native agencies, AI automating work and all of that. I’m curious, what are your hopes, or what’s your vision for how people and human creativity can still be at the center of this conversation around AI.
Jonathan Halvorson 21:40
There is an amazing tension and cycle that happens between technology and creativity. Technology gets out ahead, and it inspires new level of creativity, that inspires the next level of technology, and you get a very virtuous cycle, and I think you see that playing out over the last several decades. And I always love moments where technology is out ahead, because it means that the things I dream up in my willy wonka mind are possible. And whereas, when creativity is ahead of technology, you’re like, this would be really cool, but we can’t do it. And I find that to be a lot more frustrating. But my hope and dream is that I always believe that the best advertising is yet to be done. And therefore there’s a constant way to do better and cooler things. And every time we do this, I think for us, it was our Shah Rukh Khan, your ad moment.
Kartik Hosanagar 22:29
By the way, for our listeners who don’t know, maybe we should say who is Shah Rukh Khan is. He’s, because I grew up in India, I can say he’s like the Brad Pitt of India, only 10 times bigger in India, or something like that, right?
Jonathan Halvorson 22:39
Tom Brady, plus Tom Cruise, plus every good imaginable human like, you know, all rolled into one. I mean, and this asset we just we shot with him for a day, and it allowed you to create personalized versions of it for every local retailer. And it was just a dynamic creative optimization on steroids. And when you saw that, you were like, wow, that wouldn’t be great if everything we did was that good. I think you have moments like that where you see a piece of work, it inspires you as to what could be. And I think every day, I hope that the work that I produce at Mondelez inspires other marketers to create things. Because I know as Mita at L’Oreal, Tamara at Heliot, they’re constantly producing things that are inspiring me, and I go to my teams, I go look at this, and wouldn’t that be so cool? And I’m pushing that to the next frontier.
Kartik Hosanagar 23:28
Jon, this has been great. I think we created some good podcast content here, and we didn’t even need AI for this.
Jonathan Halvorson 23:34
Nope.
Kartik Hosanagar 23:36
Thank you so much for being guest number one on Where AI Works.
Jonathan Halvorson 23:40
Thanks so much for having me.
Kartik Hosanagar 23:42
Now let’s review the key learnings from my conversation with Jonathan. A big initiative like AI driven change is likely to throw some early challenges. Like Jon was talking about images where people had giraffe necks, or six or seven fingers. What’s really important is one needs to have patience, because you’re investing in AI for the long run. The other learning was that AI value is not just about cost reduction. It’s also about consumer engagement and revenue impact and increasing top line increasing consumer retention and things like that. And finally, it’s important for companies to find the right and consistent strategic partner, because AI driven transformation is as much about change management. How do you convince people? How do you bring them along for the ride? How do you create demos to get some early excitement about what you’re trying to do, and how do you make sure your people are with you in this process? That brings us to the end of the series premiere. Thanks so much for listening. Please follow us so you don’t miss an episode, and be sure to tune in next time when I sit down with Jill Kramer, CMO at Accenture. We’re going to explore how AI is reshaping marketing. This has been Where AI Works, brought to you by Wharton and sponsored by Accenture. I’m Kartik Hosanagar. See you next time.
Episode 2
Guest: Jill Kramer, Chief Marketing and Communications Officer; Accenture
Transcript: Episode 2
Jill Kramer00:01
Curiosity and creativity is driven by exposure, by options, by the least restrained possibilities, and Gen AI unleashes that. So if you allow it to be short-handed too “Can’t Gen AI just write that for you?”, it’s an incredible disservice to the technology and the potential of it being applied to a function as important to growth and brand and the strategy of any given company as marketing is.
Kartik Hosanagar 00:31
Hello, and welcome back to Where AI Works, Conversations at the Intersection of AI and Industry. Brought to you by Wharton and sponsored by Accenture, I’m your host, Kartik Hosanagar. I’m a professor at the Wharton School, and my work focuses on AI and business transformation. It is our goal to cut through the noise and deliver actionable insights for business leaders by combining cutting edge research with real world case studies. Things are changing fast, so without further ado, let’s dive in.
News Clip One 01:02
I’m getting the same question a lot nowadays. Should we be scared of AI?
News Clip Two 01:06
When we’re talking about trust, it means so many things, and it’s a whole world that everybody will have a different interpretation of it.
News Clip Three 01:12
One of the challenges that we have is that the space is moving incredibly quickly.
Kartik Hosanagar 01:16
On this episode, I’m excited to explore how global brands are harnessing AI and also what it takes to take large teams along on that journey. Multiple studies have shown that AI dramatically improves human productivity. These studies range from settings like software to sales, consulting, customer support, professional writing and many more. Now, while we have that evidence at the individual level, hard evidence on return to AI investments at the organizational level haven’t yet emerged, and I think that’s because there’s a difference between a few individuals adopting and seeing their returns on that versus large, complex organizations, because you’ve got to manage cultural change. Re-skilling, up-skilling, strategic prioritization; lots of issues here. To help us think through the challenges and the opportunities around AI, it’s my great pleasure to welcome Jill Kramer. Jill is the Chief Marketing and Communications Officer at Accenture. Jill, welcome to the podcast.
Jill Kramer 02:15
Hello, Kartik. Thank you so much for having me. I’m excited for our conversation.
Kartik Hosanagar 02:19
Likewise, super excited to have you here. I guess even before I get to the topic of returns from AI and how are you managing AI driven change at Accenture, I’m really curious about what’s been your relationship with technology.
Jill Kramer 02:34
So my personal technology journey is one of avoidance and fear that turned to a deep, unwavering love. I would love to tell you that I was fearless, that I was first in line with my hand up, saying, let’s do this specifically with the most recent wave with Gen AI and the speed at which it’s moving. I did start this thinking, wow. Like, is this an existential crisis, and I made the decision, and I was very transparent about it with the full marketing and communications team at Accenture that I desperately wanted to be in the driver’s seat. And the only way you can do that is by getting really close, getting very hands on, deeply understanding, and taking some risks on behalf of yourself, your company, your brand, your team. So that’s where I went over to the love side. And it is something that I truly believe now is invigorating, exciting, drives more creativity than I ever could have imagined.
Kartik Hosanagar 03:35
Yeah, that’s great to hear. And in fact, later in the conversation, I’d love to explore how you have undertaken that journey. But before I get into that, perhaps where I want to start off is maybe at the organizational level, before we get into like the marketing team and then you individually. Accenture has been talking about being early and aggressive with transforming its business around AI, so help us understand what progress has Accenture made in that AI transformation journey, and where is it currently positioned relative to others in the market?
Jill Kramer 04:08
The journey for us is really the journey we’ve taken with previous technologies, really, which is understanding each wave of technology and the impact it can have on work and workers, and in this particular wave, one thing that was evident very early in the process with Gen AI was that if you apply it to existing work, existing processes, you would see incremental improvement. But if you said, I want to look at the work in a zero based way, now that I know that this technology exists, how would I do it given these new circumstances? And that is something we have done across the board. With every function that helps run this corporation, we have looked at rethink the work now that you know that Gen AI exists. Because one of the things you learn, when you embrace technology like this, is those areas of overlaps and adjacency, there’s magic in reimagining those handoffs, those shared services, those processes from legal to HR to operations to marketing to sales. So it’s something that has to be done intentionally. It’s something that has to be done with a clean sheet of paper, you have to rethink and reinvent, and it’s something that has to be done very collaboratively across all of the functions that run the enterprise. Let’s go from Accenture as an organization to your marketing organization. There are many things under your purview which AI could touch upon and does indeed impact and influence today, whether it’s advertising, content, digital marketing, social media, corporate communications, AI’s applications in all of those areas. I’m curious, for you, what have been the first few areas that you have brought AI into, and why have you picked these specific applications or use cases for AI? So I’m actually going to tell you a quick recap of the chapter that preceded our application of Gen AI.
Kartik Hosanagar 06:01
Okay.
Jill Kramer 06:01
And that is when you run a marketing function at a large multinational company, you have projects being asked from every part of the organization, lines of business geographies, you name it, you have requests. A lot of marketing organizations execute those requests, yes, in service of a business strategy, a brand strategy, a marketing strategy, but you don’t necessarily know every project that’s going on at any one moment.
Kartik Hosanagar 06:28
Right.
Jill Kramer 06:29
And you can’t necessarily reconcile, prioritize. So I wanted to be able to say at any point in time, how many events have we done this year, which are the most successful? How many pieces of content have we put in the market? How many new pieces of content have I put on our website? What’s performing the most? So the first thing we did unknowing about the Gen AI movement that was about to happen, we really got our act together in terms of knowing the work that happens in marketing using free Gen AI AI, so regular AI, to create algorithms and prioritize and sort work in a very simple but strategic and intentional fashion. We also brought all of our data together, because if I wanted to look at the work across the function, I had to look at the KPIs. That meant that when Gen AI started ramping up, we were able to be pretty confident about which we were going to adopt first. So that brings me back to where you began with your question. Some of the first things we did was we had really done a very rigorous reinvention of our internal communications. We had created a very highly personalized approach to communicating with the then 700,000 or so people of Accenture. So one of the first things we did was we applied AI to our writing so that, as we were highly personalizing, we could use Gen AI to quickly version. Similarly, we did it with content, not the entire content supply chain, but the content versioning. We did that very quickly. So here’s the long form report, using Gen AI to assist in the medium size and the short size, the quick synops that would appear on the website, the quick synops that might appear on our app. What really changed was when we decided that we were going to go all the way to agentic AI. When you do that, you have to go all the way back to your process reinvention and say that if I looked at the steps in the marketing process, and I knew I had an agent or agents who could do work simultaneously with me going to parallel work streams, instead of everything having to be so linear and sequential, how might I design the work? We ramped that up this past September, and we now have 1000 marketers all onboarded using agents in their daily work.
Kartik Hosanagar 08:47
For our listeners who are not as familiar with agenetic AI and where it’s headed, I think the way to think about it is that lot of AI systems today can get individual tasks done. Agent-based systems are systems that can function autonomously. They can communicate with each other, they can interact with the outside world, get information from there. They can transact. And so once you start to build agentic systems, you’re not talking about automation of individual tasks. You’re talking about automation of full workflows where an agent based system can go fetch information from some other place, maybe interact with a vendor or a partner, negotiate with them, complete full tasks. So that’s an interesting direction that AI is headed. And it’s great to hear that you’re talking about some of these implementation of agentic systems. We’ll come back to that in a moment, but coming back to what you were saying with this journey from starting with AI for writing and then for image-based content, to other workflow products and now to agentic systems. Are you seeing any results in terms of costs or business agility, or what are you seeing in terms of what it’s doing for the business or for your org?
Jill Kramer 09:55
What we’re tracking very specifically is speed to market.And we’re tracking reduction of steps taken in a given process across the function, but also within sub-functions. Then there’s two waves of KPIs that I’m tracking. One is the prioritization of the work, which is Gen AI assisted, because the way we’re processing our reports and our measurement, it allows us to make better decisions, which has led to, like, 50% content reduction. Using Gen AI allows me to now do what we call Good Morning Accenture, which is a highly personalized multi-channel way of greeting each person within Accenture and letting them know what they need to know and what they need to do for that day in order to be productive, focused, etc. On the flip side, when you look at the longer flow processes, we’re seeing about a 35% reduction in steps taken within the application of the agentic workflows, and as much as 30 to 50% increase in speed to market.
Kartik Hosanagar 09:59
Yep. Wow. Those are very impressive numbers, I’m sure, an organization of Accenture’s scale, many of these systems are often built in house, but at the same time, there’s probably tons of AI products that you’re consuming. One of the questions I have for you, which is something I keep hearing from a lot of CMOS, and in fact, across different functions, but certainly from CMOS as well is that there’s so many different vendors out there, and so when you’re bombarded with so many tools to potentially use or pilot, how do you approach the decision of which tools to prioritize? How do you pilot them? And there’s possibly IT approvals and AI committees that need to approve things, that slows things down. So how do you stay nimble within your marketing org, where you can experiment, try new tools, but at the same time be compliant within the organization’s broader mandates and restrictions about what AI is allowed, what’s not allowed, and things like that?
Jill Kramer 11:56
You bring up so many important things and points, but you bring me back to my driver’s seat decision. There are so many solutions. They’re exciting. People want to get their hands on them, and so I’ve got to be in the driver’s seat for myself and for my team on this because people were adopting solutions, bringing them forward. We can use this over here. This is now embedded in this solution we already use, and it felt like a beautiful kind of chaos, but I realized that it was going to vary. We’d just gone through all the work to get our tech stack cleaned up, our data cleaned up, our processes cleaned up. So it was like, how do we bring our people with us? Use everything that they’re understanding? I recently had another CMO saying to me, the people in your marketing organization and or your customers are going to start using these tools, whether you sanction them and intentionally do them or not. And that’s a very, very true statement. And we made pretty bold decisions, like giving access to writer.ai, to every marketer. We could have done a much smaller group. We could have trialed, but we said, you know what, we didn’t want people to feel left behind. And then in other cases, we said, this is going to be the landscape. So we created a leader on my team who was going to oversee all the Gen AI ideas. Let all those beautiful ideas bloom, bring them back to our team and our IT advisors, because no CMO should be doing this without right your IT advisor. They’re your best friend.
Kartik Hosanagar 13:22
Yep.
Jill Kramer 13:22
The other thing about these AI systems is that they are designed to be more modular. You know, like, whether it’s the ability to switch between LLM models, whether it’s the ability to click in new modules of agent types or support with out of the box solutions. There’s an agility and a flexibility if you design your approach to AI and Gen AI in the right way. My advice is set up a system where your people are bringing you their best ideas, but you do have a formal way to assess pilot and scale. You must change the work, because if you don’t change the work, the scale and the actual benefit from going to scale won’t be there. And then the last one is that you need to have your IT department with you, just like we wouldn’t want anybody coming up with their own brand tagline. So you have to merge the craft. Then what is exciting about the technology for the craft of marketing and communications with the expertise of technology experts who can say, here’s how you build this, and don’t worry, because it’s modular, it’s agile, and you’re going to want that flexibility to evolve as you go forward.
Kartik Hosanagar 14:26
I want to move the conversation from the technology to the people, because a lot of this is really a story of people, how comfortable they are with how empowered they feel with these technologies and so on. I’m curious how big is your marketing org, Jill? How many people are we talking about?
Jill Kramer 14:45
We’re at about 17, 18 hundred people.
Kartik Hosanagar 14:47
Wow. Okay, yeah, that’s a lot of people you’re sort of bringing into this AI journey. I feel like when it comes to an org with as many people, you’re going to have some people who are maybe early adopters, who are jumping in trying things out, and then there are going to be people who feel threatened by this. With my previous AI startup, you know, I had these conversations with the C-suite about the products, and I remember one conversation in particular where one of the C-suite leaders was talking about how this tool that we had could potentially be threatening for people in their org. And specifically, he kind of said that, you know, I don’t see our employees using this at all. And the reality was, we had several dozen of the employees already using the free version of the tool, and then we had to walk them through. Hey, listen, you know, your employees are actually further ahead than you think they are. So tell me a little bit about both sides of this coin. One being there are many instances where the employees are ahead of where the leadership thinks they are. And then there are also instances or individuals who are feeling very threatened and feeling left behind because the pace of change is so fast. How has it played out in your arc, both in terms of people feeling threatened, but also empowering people who are moving faster?
Jill Kramer 16:04
Very similar. I mean, a lot of people were proactively coming to us saying “Hey, I’ve been using this tool on the side, and look at how much faster it helps me do this brief” and “look at how much more information I was able to collect”, or “Look at how quickly I did these” and you want to embrace those things, but also make sure you’re within your responsible AI guidelines. Is everything legally sanctioned? On the flip side, I always talk about marketing is a very generous function, right? You give your ideas, your words, your perspective on imagery, to internal and external clients, right, to form marketing materials. But a lot of people think they are marketers, right? Oh, I would have written the headline this way, or I shouldn’t have done this this way. So when you end up with a technology who now says, oh, it can write the headline, it can do the image. It was a really natural reaction to protect a craft, to protect the function, and not allow it to be unintentionally devalued by saying anybody or anything could do this craft that we all love so dearly. So my take on this is, and I was very clear with my journey with the team was that a good marketer is a curious marketer. If you watch someone write a brief, they will spend as much time as you will give them, researching competitors, looking back at everything we have done and not done. What was the best thing we ever did? What was the biggest failure? Same thing with creatives, they want to write, rewrite. I mean, how many times have you had to say to a creative team “okay, it’s time to put the pen down. We got to go produce this.” They want to find every image they can.When you think about Gen AI through that lens, these agents we’re using help you go out and get every bit of analysis, competitive intel, activation schema that you could possibly think of. If you’re using it as a creative, the ability to seek images, to change parameters, to look at the same thing 10 different ways, is now at your fingertips.So I think that’s the biggest thing is curiosity and creativity is driven by exposure, by options, by the least-restrained possibilities. And Gen AI unleashes that on behalf of the human, on behalf of the craft. So if you allow it to be shorthanded too “Can’t Gen AI just write that for you?”, it’s an incredible disservice to the technology and the potential of it being applied to a function as important to growth and brand, and you know, the strategy of any given company as marketing is.
Kartik Hosanagar 18:04
Yeah.Yeah, I’m gonna remember that statement you said, which is, curiosity and creativity are fueled by having options, having this information and so on, and AI enabling that. So that’s great. Walk us through the up-skilling and re-skilling process. You know, 1800 people who you’re now trying to train on new systems, new tools like writer.ai, new ways to approach work where you’re giving up part of what is core to your craft to an AI system, and focusing on other pieces where you can add unique and differentiated value as a human creative. What has that re-skilling process been like?
Jill Kramer 19:13
So there’s a couple of things that are absolutely core to it. The first one is the concept of creating cohorts. When you are bringing a large team along on a very new journey, don’t try to do it all at once. Don’t try to spread yourself, your ability to be attentive, to deeply train and to listen to people by trying to do it all in one fell swoop.So we’ve created cohorts for every wave of transformation. Each cohort takes the next one with them. They create a bigger community of people who can listen, who can help, who can energize and who can address problems. The second thing is communication. Very often, as a leader, you’re already sold and you know all of the reasons in your head. It’s very easy to quickly think “Well, I’ve already told them this, like they know this, they know the why.” You have to say it over and over and over again, and you have to gather together regularly to listen, to communicate progress, communicate advancements, be incredibly transparent, and take every question that comes your way. And the only way that that changes the culture is by repetition and by people realizing they truly can ask any question that they want to or need to. And then the last one are use cases. When the first two things go well, the use cases that are created by your people will be better than the ones you ever imagined. Mark my words. That’s where the curiosity and the creativity is on steroids, so you need to find a way to bring those use cases back, because that’s the very essence of change management.
Kartik Hosanagar 19:34
Right?Right. Makes sense. So, Jill, we’ve talked about the employee side of things. Now, employees are part of the journey, but so are customers, and one of the interesting things is customer behavior is changing. So for example, many of us are no longer just going to search engines and searching and finding our answers. We’re going to LLMs and finding answers, including, what should I buy, or what are the best brands for, whatever the use is. So how do you think about that, and in particular, how does the marketing org prepare for and keep up in a world where customers are embracing AI and making decisions informed by AI?
Jill Kramer 21:27
This is a case where I think it is similar to previous waves of technology, when customers were going to digital, if you decided not to understand that, you would get left behind. When search became a big component if you were not intelligently constructing your content and your digital experiences for search results, you would get left behind. It’s very similar. So, you know, one of the beautiful things about Gen AI is, for example, when you are creating something or writing something, you ask it for a summary.
Kartik Hosanagar 21:54
Yep.
Jill Kramer 21:54
And it makes you realize, oh, the points I thought I was making, I actually obscured. And some points were taken away as top level, and I meant them to be second or third level. If you’re like us, B2B professional services, and you’re creating long form content, what’s the first thing a person’s going to do now that they have these tools at their disposal? They’re going to create summaries, they’re going to merge documents. They’re going to go across and look for the best. So how do you create the experience that’s going to do that for them and with them? It’s just like anything else, you recognize a new condition and you create for it.
Kartik Hosanagar 22:31
Jill, my last question for you is where we started this conversation. You talked about how you started from a place of fear, and over time, it’s been one where you feel like you’re in the driver’s seat, in control and loving it. Tell us how that change happened, and in particular, what advice do you have for other marketing leaders that are facing this change and are perhaps a little behind in that journey and facing some of those fears themselves?
Jill Kramer 22:57
It has to do with human nature and how you overcome fear and misperceptions in anything, and that is through proximity, through exposure, through empathy and through a desire to learn. I will give this single piece of advice Kartik, and that is be hands on with this technology. It is not the same as previous waves, where someone else created a digital environment, and we just had to put the content into it.
Kartik Hosanagar 23:24
Right.
Jill Kramer 23:24
You need to know this intimately and deeply, and that comes from being very close and very hands on.
Kartik Hosanagar 23:32
That’s great advice. I mean, that’s what I discuss with my students as well, that you have to get into the black box, you have to use it, you have to touch it, feel it, and you can’t really poke at it from a distance.
Jill Kramer 23:43
It’s a lot less scary and a lot more exciting when you do that.
Kartik Hosanagar 23:46
Jill, we’ve covered a lot of good ground here today. Thank you so much for your time, and thank you for sharing your insights here on Where AI Works.
Jill Kramer 23:54
Thank you. I’ve enjoyed every minute of it.
Kartik Hosanagar 23:57
One interesting takeaway for me is about how we need to approach AI, not with a lens of fear, but with a lens of curiosity. And her point about how curiosity and creativity are both fueled when we approach AI with that lens. The other interesting takeaway for me is Jill’s framework of cohorts, communication and use cases. When you’re trying to pull off change in large organizations, doing so in stages with cohorts, makes it more achievable, realistic, at the same time, creating use cases, communicating that celebrating them helps bring people forward, and I think that’s a very practical, easy to implement framework for our leaders today. That’s all for today. Thank you so much for listening. Please follow us so you don’t miss an episode, and be sure to tune in next time when I sit down with David Droga, the founder of ad agency Droga5, and CEO of Accenture Song, which is Accenture’s creative agency. We’ll be discussing AI’s transformative impact on the creative industry. This has been Where AI Works, brought to you by Wharton and sponsored by Accenture. I’m Kartik Hosanagar. Bye for now.
Episode 3
Guest: David Droga, Chief Executive Officer; Accenture Song
Transcript: Episode 3
David Droga00:00
I think people always expect me as a creative person, even though I’m CEO, to sort of be defensive and push back and think that AI is going to gobble up everybody’s jobs and all that. I’m like, look, history’s proven us over and over that it moves us forward, and then the creative minds, it’s what they do with it, their interpretation, misinterpretation of it, that moves us forward. I also think that, which is controversial, but I mean it when I say it, not all creativity is worth saving.
Kartik Hosanagar00:31
Hello and welcome back to Where AI Works, Conversations at the Intersection of AI and Industry, brought to you by Wharton in collaboration with Accenture, I’m your host, Karthik Hosanagar, Professor of Operations, Information and Decisions at Wharton. My work focuses on AI and business transformation and how AI is affecting creative industry. This is the show where we explore the implications of AI for business leaders by combining cutting edge research with real world case studies. Things are changing fast, so without further ado, let’s dive in.
News Clip One01:04
We are at an inflection point for human creativity. These happen every so often, usually caused by some sort of technology innovation. The innovation this time is artificial intelligence.
News Clip Two01:14
Generative AI is transforming the core of marketing activities.
News Clip Three01:19
It’s not just how things are changing and how we work, but how we create, connect, and tell stories.
Kartik Hosanagar01:25
On this episode, we’ll discuss AI’s transformative impact on creative industries with a focus on how AI is redefining the value proposition for agencies and their clients. The motivating question for today really is: is AI the best creative partner for agencies and creative workers, or is it potentially a threat in certain ways. To help us explore that, I am thrilled to introduce our guest, the CEO of Accenture song and the founder of creative agency, Droga5: David Droga. David, welcome to Where AI Works.
David Droga01:57
Thank you very much. Very happy to be here.
Kartik Hosanagar01:59
Before we even jump in and talk about AI, I’d like to just start off with your decision to start your own agency and become an entrepreneur. I’m just curious about that piece, being an entrepreneur myself. What was the gap you saw in the market, or what was the need for you personally that took you in the direction of starting your own agency, and you ran it for over a decade before it was acquired by Accenture.Absolutely. Yeah. And by the way, that makes the two of us. My most recent company, I gave myself the Chairman title, and it was AI for creative industry, specifically Hollywood. So there’s some of what you’re saying that resonates.
David Droga02:22
Great question, and I wish that it was built around a spectacular business plan. Now, you can probably tell from my accent that I’m from Australia. So, you know, I sort of grew up looking at both what was happening in the US and it was influenced by the UK. And my background is as a writer. So I sort of moved into the advertising industry just because I wanted to be a writer. And I moved up the ranks rather quickly, and I ended up running Saatchi and Saatchi in Asia, and then Saatchi and Saatchi in London as their Chief Creative Officer, and that brought me to the US. And for all the successes that I had and the learnings that I had along the way, and worked with some spectacular people, as someone who was a very passionate, creative person, I kept on feeling a love of the industry, but a sort of a disdain of the industry, because I felt like advertising had become almost like a tax for people, and most of it felt quite pollutive. But I could also see the power of it when it was done very, very well. And I always sort of go to this, this go-to observation that, you know, people in Hollywood and the music industry, people invent technology to steal what they create. But people were inventing technology to avoid what we were creating. And that, for me, was such a stark wake up call that we need to create better, more relevant, more inspiring, more effective things. I didn’t like working for other people so much, and I decided that, you know, if you’re going to start something, New York is the place. I didn’t have to be for everybody, but I knew I could be for somebody. The scale makes that possible. And my business plan, which really wasn’t a business plan, was I just wanted to make work that I thought was relevant and was making the most of the technology, because this was sort of the start of the digital age, and I wanted to work with people that I wanted to work with. And right from the get go, we just sort of started doing work that was amplified by digital, because digital suddenly made it that you were judged by the caliber of your ideas, not by the media budget that you had to spend. So it sort of gave opportunity and advantage to people that were putting the consumer first, not the media budgets first. And I did that for about 15 years as Creative Chairman, which was a made-up title, because I didn’t like the idea of being a CEO, the irony of now that I’m a CEO. So that was sort of my backstory, but it was I just wanted to make work that I believed in, and technology was an enabler, which is relevant to what we’re sort of talking abouttoday.Exactly.
Kartik Hosanagar04:03
You just said that one of the motivations was to make sure what you’re making is relevant in making use of the technology that was becoming available. Obviously, over the last couple years, there’s been a big shift in the technology that’s available. We’re starting to see, even in these early two, three years, AI’s impact in creative industries, and projecting out 5/10, years, the impact could be huge. And generally, I’m seeing its impact in many places. Now, you can talk about, say, in Hollywood, somebody with a great idea can go make short films or make movies without the blessings of the gatekeepers. It’s already affecting jobs for freelancers and simple creative tasks, like, say, logo design and so on. So I have a two part question for you. I think the first question really is, what does this technology mean for creative folks? Is it a tool that you see that’s going to augment and enable them, or is it a tool that’s going to displace them, and we can talk about it generally or in the context of advertising, which is where I want to focus this discussion.
David Droga05:46
My response to that is, I don’t think it’s quite as simple as that, because I think really it’s going to change jobs, but not the necessity of what the output of the jobs themselves. There’s always going to be a need for communication and brand and design and storytelling and, you know, all these things, how we get there, that is definitely changing. So when we stop holding on to a business model or a role and understand that creativity, taste, strategy, they’re liquid. They go wherever they’re needed. You know what I mean? And that’s why I’m very bullish and optimistic about it. You know, I know we’re in the infant stages, but I think people always expect me as a creative person, even though I’m a CEO, to sort of be defensive and push back and think that, you know, AI is going to gobble up everybody’s jobs and all that. I’m like, look, history has proven us over and over, you know, that it moves us forward. And then the creative minds, it’s what they do with it, their interpretation, misinterpretation of it, that moves us forward. I also think that, which is controversial, but I mean it when I say it, not all creativity is worth saving.
Kartik Hosanagar06:47
Interesting.
David Droga06:48
Now, that’s a provocative thing to say, but there’s so much things that are deemed creative that are just not. You know, I look at architecture, I look at majority of advertising, I look at a lot of journalism, that can be better. That, you know, that’s quite formulaic. I mean, they’re informed and influenced by something far more scary, which is conformity and research. Playing it safe. So you know, if we’ve created tools of get rid of the mediocre middle then that’s going to accelerate the people that actually have the talent and the know-how, and the ambition and the creativity to do more with it. Because every time creativity has been enhanced by a tool, yes, we all get intimidated by it, and obviously I understand the scale and pace of this is unlike anything we’ve seen. But I think of even when I started in the industry, I’m not looking back thinking, I feel terrible that we don’t have rooms full of typographers or still have rooms filled with illustrators who are doing storyboards. The output is what matters. And when you focus on the output, creatives will always be relevant. The tasks are changing, not the necessity of what people need.
Kartik Hosanagar07:47
Yeah, I like the way you put that, and I have the same belief that we can break down work into a set of tasks. And no doubt some tasks will go and some tasks will stay, and some new tasks will get created. You brought up the point of taste, I think that’s really interesting to me. It reminds me of a conversation I had with producer some time back, and that context was filmmaking, as I mentioned earlier, and when it came to AI tools, say, for producers to analyze scripts and so on. Initially, the reaction from some studios was: there’s no way our creative people will use AI because AI cannot be creative. Then it was: they feel threatened by it. But then you actually look at the job that’s happening, and you talk to the people. Turns out a typical person is overwhelmed with the number of scripts coming in, four books, six scripts to read in a weekend. And AI actually is great for them to understand. What is this about? Does this fit your mandate? Is it too similar to what you’re working with? And when it comes to what does this make me feel, and where do I want stories to go in the future that taste-making role, it’s very hard to not see humans being relevant in that setting.
David Droga08:54
Completely. I mean, again, a lot of what AI is doing is sort of bringing together all the best practices and all the knowledge that exists to formulate something new. And again, it could probably write scripts and movies that are on point with some of the average films that are out there. Of course it could. But it’s never going to create something and you take some people who push into new territories or have context and references that only a human that’s gone through the human experience can get to in certain pieces of dialog and and I think that’s why I feel I’m cautious and I care about when I say, you know, not all creativity and not all jobs are worth preserving. I don’t say that in a cold-hearted way. I mean, everyone deserves to have a job. I just think it’s going to give people new opportunities, and it’s going to industrialize the imagination. Because when everyone has access to best practices, then something new has to happen. You know what I mean, to sort of elevate beyond that. And we look out in the real world. As I said, taste is actually a real thing. It really is. Understanding and context are real things, making connections are real things. And our past is littered with things that allowed us to do things faster and more. And at the moment, it’s just allowing us to do things cheaper and faster. There was a big to do about some big fashion brand that had spent $600,000 making a esoteric commercial with someone diving in the water and swimming and whales and all this sort of stuff. And it was for, you know, a perfume or something like that. And I think they spent two months making the ad or three months, and it cost $600,000 and someone had used AI and recreated it in a matter of hours for 100 bucks. Which is extraordinary.
Kartik Hosanagar10:19
Right.
David Droga10:19
What no one’s talking about is both of them were crap. Now, what the advantage of that is you can get to crap faster, so you can move on from that. But the parlor trick of the technology is never going to be the thing that just makes the connection. It still has to have strategy and insight, particularly if you talk about things that motivate people and you want to move people and stuff like that, but it’s same with architecture, same with literature, same, you know, it can do great things, and we shouldn’t begrudge that. But it’s what we do with it, that’s the great thing about humanity, our interpretation of things, as I said, and we’re the ones who enforce ourselves into that.
Kartik Hosanagar10:53
Yeah, and I’m curious how you infuse that taste, that originality, in your own work, and I’m asking specifically whether there’s a trade off, especially when it comes to AI use, but there’s a trade off between efficiency and originality.
David Droga11:08
Well, at this stage, majority of people are still very much in their table stakes, and so they should be. There’s great efficiencies to be made: time, money and all those things. And so a lot of people think that, oh, this is going to solve everything. I’m never going to need to deal with a creative organization or a design company or get, you know, of course. So they’re sort of enjoying the- reaping the benefits of that, but that’s only going to get you so far. Again, our world is to sort of connect with consumers from a emotional point of view, from a commercial point of view, and brands have to be distinctive, they have to be meaningful. They have to be memorable. AI, can get you so far. But if everyone has access to the same tools, you’ll get the same outcomes all the time. So we sort of add that layer of context and differentiation.
Kartik Hosanagar11:51
I think your point about everyone having access to the same tools, you’ll have the same outcome if you’re not careful, is an important one. We actually had a study recently in our research group where we had 250 plus participants engage in creative tasks, mostly creative writing, some with AI, some without AI. And what you observe is that AI used reduced task completion time. It also increased the quality of the writing as rated by independent evaluators. But when all of them were using the same kinds of AI. What we observed was that the diversity of the writing went down.
David Droga12:25
Oh, of course. I mean, you know, think about when search first started. Everyone thought that everyone was going to be lazy and wouldn’t be able to do anything. And you know that, for instance, suddenly everyone had access to everything. And, you know, people cut corners and all that sort of stuff. But it still opens up new avenues. It opens up new quadrants of exploration. And I think, I also have to believe this sort of being transparent, you know, my wife is in film, my eldest son graduated USC film schools. My daughter got accepted last night into USC film school.
Kartik Hosanagar12:51
Oh, wow.
David Droga12:52
So I have to believe. Now, the cynic in me would be, and I’m a creative thinking, oh, are they going into an industry that’s going to be irrelevant? I’m like, the business model of the industry is maybe going to be irrelevant, but the need for storytelling and movies and evocative and emotional things is never going to be. So I really want them to understand the technology, work out how to bend it and use it and break it and sometimes ignore it. But I’m not so bullish that I’m not aware of the pitfalls. And you know that we have to be very thoughtful about it and consider- we have to have a little bit of paranoia that’s also important, to make sure that we sort of utilize it and use it in a way that’s responsible and fair, and it’s, you know, because there’s a lot of things that could go wrong as well.
Kartik Hosanagar13:32
So we’ve talked a little bit about what AI means for creativity. I want to talk about what it means for the business of creativity. Let’s start with the client side of things. What are you seeing in terms of how technology in general and AI in particular, is making people rethink roles and responsibilities within the enterprise. For example, what does it mean for CMOs, and how is it changing what they need from their marketing partners?
David Droga13:57
The CMO’s role has changed. It was already morphing before the emergence of AI, and that sort of moved from being, you know, responsible for just growing brand, to sort of growing companies. And I think that the voice of the CMO carries a lot more weight than it used to, or should, definitely it should need to be, sort of, you know, really, in the C suite. I think what AI is allowing us to do, and I’m seeing it firsthand with Song, which is part of the reason I sold my company to Accenture, and sort of the belief about the necessity of technology just to weave together all the facets of the world I’d come from. You know, we can actually do things. But the excitement of the output is tapered a little bit in the beginning, because it’s sort of exposing people like: Are they are they ready for it? Is their corporation ready for it? Is their data clean enough? And at the moment, it’s all fragmented and in silos. So there’s a lot of enthusiasm for it, but it’s also revealing, when you have the conversations about just how ready is your organization, your company, to do that. What’s happened is every company out there needs to have sort of a unified customer experience across every facet of it. You can’t separate marketing from the commerce or from the sales, orr from the products. You know, they have to be united, and AI is a way to sort of make that tangible and a reality. But as I said, there’s a lot of questions. It exposes a lot of flaws that people need to fix their digital core and sort of get their heads around what their data capabilities and is it ready for it?
Kartik Hosanagar15:15
Yeah. So if the challenge for a lot of CMOs is quite heavily in the data and technology camp, meaning, help me get a unified view of my customer, help me understand the customer journey and so on. Does it change anything in terms of how they work with partners like you, like agencies? Does it change anything in terms of what agencies do? Because what you’re describing primarily sounds like a technology issue, a data issue, more for the CMO, interacting with their CIO, CTO, versus with their agency partner.
David Droga15:46
Well, the thing is, and I say to you, as someone who built a well known creative agency, the model of that just being by itself from any agency can no longer exist, or it’s going to be sort of not irrelevant, but its power and influence is going to shrink.
Kartik Hosanagar15:57
Oh, I see.
David Droga15:58
So that’s why they have to morph it in and again, being transparent, 100% why I sold my company to Accenture and why even why I said yes to this job because I believed that a creative person should be at the head of a company like this, making decisions that are about brand and consumer, but make sure that strategy and creativity is woven into the data and AI, as opposed to a nice-to-have or an addition.
Kartik Hosanagar16:20
I see.
David Droga16:20
So there is consistencies, and there is ambition, and there is an understanding of humanity. Because, you know, creativity needs technology to scale, and technology needs creativity be more human and relatable.
Kartik Hosanagar16:31
You know, it makes me wonder what you think an ad agency will look like in five years. The kind of vision that you are espousing, I don’t mean to put words in your mouth, but at least what I’m taking away from what you’re saying is that it is hard to separate the creative insight and the creative strategy from the data and the technology piece. And part of what you’re doing is, you know, coming together with Accenture and now heading Accenture Song is you’re bridging the two together simultaneously and taking it to your clients.
David Droga17:01
100%. I mean, that’s our point of difference. I think that’s why we’ve grown exponentially. I’d love to think it was my genius, but it’s not. It’s sort of the alchemy of what we have, the sort of unified customer positioning and understanding. And it doesn’t sort of leave something on the sidelines. It’s consistent. Complexity is something that is one of the great stiflers, particularly at the pace that everyone’s moving now, and it does take out a lot of the complexity as well. Everyone’s still in the learning phase. I mean, I’m not pretending that we have all the answers, but we have a lot of the people that have the answers. And know, you know, we spend as much time trying to have clients get business ready, as well as customer relevance. You know, we’re putting those two things together. Sometimes they’ve been separated, but I think some of the Holdcos understand customer relevance, and I think some of the other consultancies understand business readiness. I think we’re the ones who sort of interlock those together because they influence each other and they need each other. And having a creative person at the helm, really, that means I’m just a product person. So I’m thinking about not just the integration and the seamless nature of it, but the output. You know, I always start with what’s going to work for the consumer, and that’s the best thing you can do for your client, is care about their consumer.
Kartik Hosanagar18:05
Yeah, it’s interesting to hear the blurring of these lines between the creative person and the product person. I think that’s quite fascinating.
David Droga18:11
It’s necessary, and again, I bang this drum too often. I’m not down on creative agencies, as I said, as a creative person, I’m bullish on creative people. I just keep on telling the agencies out there: change your business model, embrace the thing. You know, some of the companies we bought, a great company called Work and Co, and they’re a digital design products company. You know, they’ve created this new model where the designers and the engineers are partners in crime, so that just that sort of harmony between that. And then you’ve got the sort of agentic thing, which is going to change so much as well, because we spend a lot of time building that for our clients and making sure that they’re built in a way that has understanding in it, you know what I mean.And, you know, look, who knows in five years what we will be. But I like the idea that people who understand people, and not just engineers, are making decisions.
Kartik Hosanagar18:53
You’ve talked about, strategically, structurally, what changes you see coming in this space. Let’s talk about operationally, like inside an agency, day-to-day work, what are the changes you imagine? And one of the things that AI is allowing us to do broadly is reimagine work in general. And we’re seeing tons of money being poured in by VCs, and entire industries and functions are being re-imagined. With all these investments coming in, no doubt that many of these experiments will fail, but also many will succeed. And so I’m kind of wondering, what are the experiments you’re running in-house? You talked about agents, for example. So I’m kind of very curious internally, what are the changes that you see in terms of how the creative work gets done?
David Droga19:39
Well, I think that people who can ideate and have taste, I’m going to belabor those things are still necessary. You know, do I need reams of people who can do translations and storyboards and, you know, things that are time consuming? No, it allows people to get stuff out of their head faster, good or bad, right? So we can just operate at a different speed. And what’s the biggest change that I’m seeing in the agency space is, you know, there was that classic triangle that we used to say to our clients about speed, quality and cost. You can only pick two, that whole thing. What AI has allowed us is to take that out of the equation. You can actually do all three. Now, you can’t outsource to it all completely and just think that it’s going to solve everything but the repeatable, scalable part as one now. You know what I mean. Part of what I said to Droga5 is, you know, part of their superpower now is, yes, they still have the DNA core of being creatively, strategically-led. But the conversations and the people that have a voice at the table are technologists as well, and data scientists and people that actually understand how things are made and scaled anddeployed. And that makes a big difference. And then once creatives get over the nervousness of this is going to make me redundant or compromise me. And they go actually, I can do things that I could never do before. Yeah, that’s where it gets really exciting. And we’re still at that early stages of that. As I said, everyone out there is still at the, mainly at the efficiencies and speed phase of things. But like with all technology, soon we’re on the cusp of getting to the “I could never dream of doing this before”, now we can actually do that.
Kartik Hosanagar21:03
I guess, one closing question for you, David, is: what’s your advice for creative folks who are starting their career now? The world is so different than when you were starting so for most industries, most jobs, the old ways no longer work. So what would you ask them to do? What kinds of skills or what kinds of approaches they should be using early in their career?
David Droga21:23
Well, success is not nostalgic, right? We are, but success is not. Okay. You can be nostalgic for emotional things and again, the ramifications of things, but don’t look back and think, okay, you know, I have to be making a TV commercial, or the best team is a writer and an art director sitting together, or be open to experimenting more, as I said, have more faith in creativity, but also insert yourself in more places and learn more about it. When I took this job, it was the first time I’d been intimidated and learning in the last 20 years, and it was interesting and fascinating, because once I got through the sort of understanding and sort of I don’t pretend to have a grasp of everything, but I’ve moved past the instruction manual phase into the oh, what this means, and what could we do with it? Because I know that I, or young creatives even more so, look at things and see the world in different ways. You know, remember, engineers make things and expect the world to work out how to use them. Yeah, creatives and designers make things around how the people work, and what works for people, and it’s just a different mindset. So I just, I keep on saying to them, just know that you’re more relevant in more places than you’ve ever been before, and put yourself out there, push your boat out. Becausecreative voices, we ask different questions. You know, we don’t just give different answers, we give different questions, ask different questions. And that’s what sets us apart. So I’m the one who’s saying the model may be broken, but the need for your role, if you’re a genuine, creative person or a strategist, has never been more in need.
Kartik Hosanagar22:48
We’ll end on that very positive, uplifting, inspirational note. I think that’s a good call to arms for creative folks. David, thank you so much for joining. It’s been a great pleasure to have you on the show.
David Droga22:59
Really loved chatting to you, Kartik, thank you.
Kartik Hosanagar23:02
I thought there were two really interesting takeaways from this conversation with David. The first thing is that while AI is helping automate and augment a lot of creative tasks, the role of the creative still remains. This is because while AI can add a lot of efficiency into generating drafts, into creating early mock-ups, even creating ultimate visual designs; there’s an important role for humans to be taste-makers, to see what actually gets emotional response from the audience and so on. The other takeaway is the blending together of the creative and the product person. David talked about how in many ways, the creative person has to almost be the doer as well. I brought up the analogy that entrepreneurs need to be able to build product prototypes themselves in other domains. Similarly, I think in creative domains, it’s the creative, the idea generator, who should be able to use the right tools and create prototypes of their ideas. And ultimately, that ensures that they can go from idea to the final product in a more effective manner.
That’s it for today’s conversation. Thank you so much for listening. Please follow us so you don’t miss an episode, and be sure to tune in next time for our season finale. I’ll be chatting with Lan Guan, who is Accenture’s Chief AI Officer, and we will look at how AI is transforming enterprise strategy, customer interactions and business growth. This has been Where AI Works, Conversations at the Intersection of AI and Industry, brought to you by Wharton in collaboration with Accenture. I’m Kartik Hosanagar. Bye for now.
Kartik Hosanagar 23:42
Now let’s review the key learnings from my conversation with Jonathan. A big initiative like AI driven change is likely to throw some early challenges. Like Jon was talking about images where people had giraffe necks, or six or seven fingers. What’s really important is one needs to have patience, because you’re investing in AI for the long run. The other learning was that AI value is not just about cost reduction. It’s also about consumer engagement and revenue impact and increasing top line increasing consumer retention and things like that. And finally, it’s important for companies to find the right and consistent strategic partner, because AI driven transformation is as much about change management. How do you convince people? How do you bring them along for the ride? How do you create demos to get some early excitement about what you’re trying to do, and how do you make sure your people are with you in this process? That brings us to the end of the series premiere. Thanks so much for listening. Please follow us so you don’t miss an episode, and be sure to tune in next time when I sit down with Jill Kramer, CMO at Accenture. We’re going to explore how AI is reshaping marketing. This has been Where AI Works, brought to you by Wharton and sponsored by Accenture. I’m Kartik Hosanagar. See you next time.
Episode 4
Guest: Lan Guan, Chief A.I. Officer; Accenture
Transcript: Episode 4
Lan Guan 00:01
When ChatGPT Phenomenal first happened at the beginning of 2023,everybody was scratching their head, oh, wow, this is so disruptive, there’s no playbook. But I think now, two and a half years later, we have learned a lot. We’re still learning. Now it’s our job to go to all these clients to say, okay, let’s focus on the how.
Kartik Hosanagar 00:25
Hello and welcome to the season finale of Where AI Works, Conversations at the Intersection of AI and Industry, brought to you by Wharton, in collaboration with Accenture. I’m your host, Kartik Hosanagar, Professor of Operations, Information and Decisions at Wharton and Co-Director of Wharton Human AI Research Initiative. It is our goal to cut through the noise, to deliver actionable insights for business leaders by combining cutting edge research with real world case studies. Things are changing fast, so without further ado, let’s dive in.
News Clip One 01:01
What we see in the data is that the executive urgency to incorporate AI is at an all time high.
News Clip Two 01:06
There’s been a lot of buzz, maybe even hype, around if you don’t adopt this now, you’re going to be left behind.
News Clip Three 01:13
Business is a game, and if you want to compete, if you want to win, you have to use AI the right way.
Kartik Hosanagar 01:19
On this final episode of our season, we’re going to transition from AI strategy and use cases in marketing, to implementation at scale. To help me unpack the challenges and opportunities for business leaders, I’m thrilled to introduce someone who has helped lead her company to its fastest growth in its history by booking over $3 billion in generative AI related business last year. Lan Guan is the chief AI officer at Accenture. Lan, welcome to Where AI Works.
Lan Guan 01:49
Thank you for having me. I’m so excited to talk to your audience about AI
Kartik Hosanagar 01:56
There’s really a lot to talk about today, but where I want to start off is with your role, your title, Chief AI Officer. It’s a relatively new role in organizations. It’s not too long back, I remember even the CTO and CIO roles were new in organizations. Until then, IT was like a back-office enabling function, and then it suddenly became strategic to drive business transformation. So what is the role all about, and how did it get formed within Accenture?
Lan Guan 02:24
So I’ve been playing this role for a little bit over two years now. First and foremost, I’m responsible for our company’s overall AI strategy in the market. So that means things like: What do we want to be famous for? Where do we invest and how do we work with ecosystem partners? Who do we work with? I’m also responsible for developing highly differentiated industry solutions, assets, platforms, accelerators for Accenture, so that our practitioners can be bringing this kind of tangible capabilities to our cross-industry clients. Last but not the least, I’m also responsible for leading 65,000 data and AI practitioners. It’s different from the typical IT role, because I always talk about AI, comparing to data, comparing to cloud, is so much closer to the business.
Kartik Hosanagar 03:23
Makes sense. I think it’s so exciting that you get to help define what this role ends up being, not only within your organization, but over time, other companies will draw inspiration, and they’ll figure out what that role means for their firms as well. Where I wanted to really take this conversation too is understanding what is the current status of AI implementation today? Where are enterprise AI investments currently being directed? What kinds of applications are companies buying and building? I was wondering if you can talk about that both in terms of within Accenture, but you work with so many clients with their AI implementation. So what does it look like across hundreds of thousands of companies?
Lan Guan 04:03
Yeah, sure. You know, oftentimes we get asked, is this reality or hype, right. So I think I can be very upfront here. In this case, given the results that we have seen, this is quite real. You mentioned 3 billion in revenue that we have been booking for gen AI. In fact, it’s 5.6 billion over the last 18 months. Okay.
Kartik Hosanagar 04:25
Oh, amazing. That’s amazing.
Lan Guan 04:27
This is all public.
Kartik Hosanagar 04:28
Good for you.
Lan Guan 04:28
And also, we have completed 2000 gen AI now agentic AI projects. The progress that we’ve been making in the market is just enormous. Let me also give you a couple of data points here. So our latest research shows that 86% of the C-suite clients, they’re actually planning to increase their investment in AI. 60% actually expect their gen AI solutions to be scaled across their organization, up from 36% in 2024. So I think the point here is this is real. They are spending money on this. They’re shifting their spend from other areas or into AI, or they are just increasing the net new spend. Another piece of the research we have done also indicated less than 10% of the clients have actually scaled. In fact, the number is 8% so everybody’s asking, okay, what happened to the other 92%? So it’s our responsibility to collectively go help them. I think the second part of your question is, what are some common use cases, right? So we actually tell our clients that you should be thinking about your AI investments in two categories. The first category is what we call table stake, right? Think about those no-brainer areas that every company should be investing in. For example, in marketing space. The second category is what we mean by more strategic bets, right? So these are the areas that tend to be associated with the client’s industry value chain. Okay, like, for example, in life science, one strategic bet that we’ve been investing in is actually helping a lot of life science companies to speed up their clinical review process, leveraging AI, leveraging data, so that they can release the new drug faster into the market. In telcos case, using gen AI and deep dura network to help improve their network operations. In banking industry, for example, deepen the bank’s ability in anti-money-laundering, so all these areas. So I think holistically, we believe this technology is ripe for enterprise reinvention, and we go to clients very strategically. Help them identify low hanging fruit area, table stake areas, so that you can reap the benefits faster, but at the same time, also help them develop this kind of a roadmap so that they can see continuous reinvention, continuous streams of ROI coming in by also tapping into strategic bets.
Kartik Hosanagar 07:09
I think one of the things that companies have to contend with is a problem of plenty in this space, meaning that almost every exec, especially CTOs I talk to, talk about how they’re being bombarded by lots of new vendors who have new AI-powered substitutes for software that already use, or new workflow automation kinds of solutions. But how does one think about where to prioritize one’s attention with adoption decisions, and especially on the strategic side. I’m understanding that on the table stakes side, yes, you know you have opportunities to increase efficiency. You should certainly go pursue those. But I’m really curious how you advise your clients, or how should one think about how to prioritize the attention when you’re being bombarded on a daily basis with new AI software?
Lan Guan 07:59
Yeah, people can easily get lost, right? Even just in the year of 2024 we counted more than 200 large language models being developed by all kinds of companies, not to mention, now it’s all about agentic AI. So I think from the C-suite perspective, we basically asked them the question, right? What is your strategy? What are you trying to achieve? Are you looking at AI as a competency-enhancing technology or something that is actually competency-diminishing? So this kind of question, it’s almost like the light bulb moment to them. Because it’s very easy for people to get enamored by this kind of a tool conversation and then forget about, what is the “so what?” That’s why we lay down five tricks that everybody should be talking about. I think the first one is, like I said, focus on the problem, not the tool. What is that strategic needle that you want to move? What is the competitive advantage that you want to be famous for? The second one not to get carried away by a lot of this technology conversation. So you stay calm, you stay focused. It’s actually about taking advantage of this kind of platform-based approach, as opposed to chasing all kinds of use cases. So let me tell you one example, a very large energy client that I was working with in the Middle East, okay, they’re basically telling me that, within their company, because of the size, because everybody is so interested in using gen AI, on a monthly basis, they are getting more than 800 use cases coming from all parts of the company. Everybody’s submitting this kind of use cases. So they came to us saying: Hey, should we be catering to all these use cases? So their approach is actually quite smart. They look under the cover, with our help, across all these use cases, they realize that there are a lot of the common building blocks. For example, most of these use cases involving working with some kind of unstructured data, so they had that light bulb moment. Hey, now in this case, why can I just invest in some kind of vector database? Why can I just go build, on the enterprise level, some kind of vectorized embedding pipeline, not just working with the textual data, also working with multimodal data, so that this kind of capabilities become the foundation of the platform. Then every single use case from this repository that they’re getting from their employees and different business functions is leveraging this industrialized, commonly governed capabilities. So we’re seeing this across the board. Now, sometimes I say, okay, when chatGPT Phenomenal first happened at the beginning of 2023, everybody was scratching their head, oh, wow, this is so disruptive, there’s no playbook. But I think now, two and a half years later, we have learned a lot. We’re still learning. Now it’s our job to go to all these clients to say, okay, let’s focus on the how, and look at this kind of platform-based approach, look at this kind of a reference architecture, and start investing in this kind of strategic bet. I think that’s something that is super, super important.
Kartik Hosanagar 09:29
So, you mentioned, focus on the problem, don’t get carried away by the technology. So that’s two. I think you said there were five.
Lan Guan 11:25
Yeah, I’ll quickly go through the rest of them. Also be pragmatic about the embedded AI. A lot of CIO/CTO clients, the first question they tell me is: I’ve been investing in a lot of technology in the past. So how do you make sure that Gen AI, or agentic AI is working with my existing technology landscape. If they have the enterprise platform, if they are working with Salesforce, if they are working with SAP or ServiceNow, they all have embedded AI capabilities. So they work quite well with the data that is specific to the platform. So I always tell my client, get started, turn on those capabilities. So that’s the third one. The fourth one is that you need to be demanding substance from the vendors, okay, meaning, it’s not just the buzzword, it’s focused on asking the tough questions, right? How did it works? How is that trained? What are the differentiators between your agents and the other agents? So I think asking these kinds of questions from all kinds of vendors is super crucial. The last one is actually about aligning with your data and talent readiness. The old phrase of garbage in, garbage out, is becoming even more relevant in this case. So I have seen so many examples, so many clients, they got excited to start AI, they thought that they are on the right track, and then they hit the road bump because their data is not ready. Their talent is not ready. Let me give you one quick example. One of the very large telco clients I was were personally working with, this is exactly what’s happening to them, right? They are very anxious to implement contact center agent solution. And then right at the moment of pushing this into production, they realized the model performance degraded, right, drifted. And then they realized: Oh, it was because the training data that they used did not have the good quality. They actually have 37 versions of standard operating procedures for their human employees on the floor. If it’s a standard operating procedure, SOP, how can you have 37 versions? It’s supposed to be one single source of truth. What we are dealing with every day is this enterprise messiness, meaning, data is messy, process is messy because of all kinds of merger acquisition that is happening, right. Talent shortages, right. I work with a lot of clients, they basically have been investing in the last two decades, building data science talents, building data engineers. Now I’m going to them say: Did you know your data science talents need to get re-skilled? Because now, in order for talent to thrive in the age of AI, they need to learn software engineering, they need to learn design, they need to gain this kind of a full stack capabilities, right? So a lot of times clients are like: Okay, that is a gap. How can you help me?
Kartik Hosanagar 14:16
Yeah, it’s so interesting you say that. It makes sense, but at the same time, is something that companies haven’t necessarily paid a lot of attention to. This point about to get those returns in AI investments, you need first the investments in data and the talent. And in fact, at Wharton in our center, we have several studies. My colleague Sonny Tambe has studies that have looked at AI investments by public companies and then their returns in terms of stock market returns, their revenues, profitability and so on. And it’s pretty clear that firms that are getting those returns, their AI investments are preceded by the data infrastructure investments, and they also tend to be companies that are in regions or geographies where there’s a large talent base that they can recruit from. Lan, I heard you mention multiple times agents. Now, in the previous conversations that we’ve had on this show, it’s come up, but nowhere near as many times as you, which makes sense, you’re the Chief AI officer. So we should spend some time talking about that. A lot of people believe agents will transform software as a service itself, because now you can replace traditional software applications with agents that are able to take action, listen to a sales call, analyze the call, and take some follow up action, like sending a follow up email to the customer. So even the human interaction with software sometimes becomes irrelevant. There’s so much hype around this, but it’s also not a mature space, it’s very early. So I’m really curious to hear, what kinds of experiments are you running in the agent space, and what are the big challenges and roadblocks to deploying large scale agent-based systems in organizations.
Lan Guan 15:58
So Kartik, we are the early adopter of HFT AI. So one of the best examples is actually the marketing reinvention that we were doing to ourself, because you know, as a large organization, we have lots of marketers, B2B marketing is something that is very important to us. Last year, we built AI refinery together with one of our partners in media, in this case, and we built 15 agents to basically reinvent the strategic planning, because we believe that’s the area that is almost like the crux of the marketing, right? Because it involves a lot of a cognitive task. It involves a lot of a permanent task, a lot of coordination, right? All these characteristics that matches the agent’s capabilities. So let me give you one example of the agent, utility agent level, like calendar agent. Calendar agent, why is that important? Because in B2B marketing’s case, we always have multiple campaigns coming out, multiple campaigns in multiple geographies. So how do you avoid the crashing calendar, so that we don’t have campaigns with the different objective, different messagings hitting the same audience. Today, it’s usually done by people, manually creating marketing campaign calendar. Oh, okay, this sounds like a classic assignment problem. Can we build operational research mini agent to solve, resolve a lot of this calendar conflicts?To optimize audience impression, optimize the marketing budget, to make this much more comprehensive and much more optimal. In terms of the challenges, right, the second part of your question, I think there’s also three things that I think is slowing us down. The first part is because of this confusion around what is agent, how is that different from RPA, should I be using agent together with large language models, especially thinking about reasoning capabilities? I think we, as an industry, we need to come together and actually put the stake in the ground to say: Okay, this is the definition: Agent needs to have these kinds of four characteristics. Agent needs to be able to perceive the outside world. Agent needs to have the cognition, reasoning capabilities to troubleshooting, do diagnostic, breaking down larger problems into smaller chunks. But that’s not just it. Also taking action, to using a function calling, giving the tooling part of the agentic capability is super critical, right? Connecting with your enterprise systems. Last one is learning, right? Because ultimately, agent is not perfect. So how do you institute this kind of self reflection so the agent can continuously learn, be environmentally aware. We need a clean way, very crystal clear way, to describe the power of agents, so that a lot of noise will actually be not there. I think that’s one challenge. The second challenge, I would say, is agent interoperability. I still think that there are a lot of gaps that we need to be closing. We don’t want this technology, super powerful, to also create fragmentation, to create silos. To me, that’s more tangibly, how do we solve the agent interoperability is a big challenge. The third one is, I would say the agent trust. Yes, we talk about responsible AI, explainable AI, probably for the last two decades, it became another buzzword when LLM came out, everybody started talking about AI safety. But Kartik, I can tell you, the importance of AI trust is paramount now, simply because of the proliferation of AI agents, not everybody will be creating LLMs, not everybody will be creating SLM, right? Model Customization still seems to be limited to the technical resources, but that’s not the case with AI agents. I have business users creating AI agents. I have power users who want to create co-pilot agents. Then you have the out-of-box agents. I’m just telling you, the proliferation of the agents make this so critical to actually bring the trustworthiness to everything we do. Because how do I know this agent created by Mary in supply chain should have the right role level access. Is this the authoritative one for research? All these other questions that my client is asking.
Kartik Hosanagar 20:12
It brings up a lot of questions about governance. I mean, you brought up proliferation of agents. You brought up how agents are being created and used by business teams and so on. But let’s talk about what can go wrong. And one, I guess, news story that is very fresh in my mind was, I don’t know if you saw the one about Cursor AI, they had the customer support agent Sam that was automating a bunch of customer support functions and sent people emails saying that you can only use our tool on one device which was incorrect, caused a bunch of people to unsubscribe, and then the company had to then follow up and say “No, that’s not our official policy. Our AI agent did that incorrectly.” So the question here is, there’s a huge opportunity around AI, but also a lot of things can go wrong when you do this level of automation, and AI has access to resources, can make decisions, and so on. How do you think about that governance? What needs to be in any governance framework for these companies?
Lan Guan 21:09
That’s a very good question. I think the example that you gave is very real, right? One of my similar examples I always talk about is a terrible story, or horror story that, okay, now in contact centers case, now you have agent out of the control. Agent just give refund to every customer. So to me, that’s not the kind of things we want to see happening. AI is surely becoming so autonomous. But how do we actually not slow down the progress, but also put a guardrail around this and basically say “Okay, how do we balance the risk and reward for this kind of imperfect technology?” So our point of view on AI governance, I think the entire playbook needs to be rewritten simply because of all the things that we’ve been talking about. A lot of cases, we tell the client, this is the area that you need the professionals help, doing this homegrown things can get out of control very easily. Professionalism of the gen AI, agentic AI, is quite important, right. Which means asking for outside help, right, don’t try to do all this by yourself. I think the second one is, how do you automate a lot of this instrumentation? How do you use algorithmic way to actually measure agent trustworthiness, right? How do you break down something like this into things like explainability, things like access control, security, safety, ethics. To me, that is the new way of addressing this kind of AI trust, and how do you do that in the highly automated way, so that you’re not just talking about theory, it’s not just talking about the guardrail, it’s not just talking about the design principle, because this is what every single client would need. It’s another huge gap area that I think we, as an industry, need to come together to close.
Kartik Hosanagar 22:49
Lan, this has been a great way to wrap up our first season. Thank you so much for joining us and sharing your insights here on Where AI Works.
Lan Guan 22:58
My pleasure, Kartik.
Kartik Hosanagar 22:59
Now I would like to highlight a few takeaways from my conversation with Lan. Lan brought up the fact that many companies are investing heavily in AI, especially generative AI, but many struggle with execution. The biggest roadblocks include integrating AI with proprietary data, building the right data and talent infrastructure in-house, and scaling beyond small experiments. We also talked about AI governance and trust, and there are many dimensions to it. Companies must prioritize explainability, transparency and ethical considerations to build trust, especially as AI becomes more autonomous with agent-based systems, organizations will need to rethink their trust models and establish clear guardrails. That’s a wrap on Season One of the podcast. Thanks so much for listening. It’s been a great pleasure taking part in these conversations, and I hope you have enjoyed them as much as I have. Season Two is just around the corner, so be sure to follow us so you don’t miss an episode. This has been Where AI Works, Conversations at the Intersection of AI and Industry, brought to you by Wharton in collaboration with Accenture. I’m Kartik Hosanagar, bye for now.
Episode 5
Host: Kartik Hosanagar, John C. Hower Professor; Professor of Operations, Information and Decisions; Co-Director, Wharton Human-AI Research; Professor of Marketing
Transcript: Episode 5
Jonathan Halvorson 00:02
For brands who aren’t clear on exactly who they are in their foundations, I think they will findthemselves in a sea of sameness.
Lan Guan 00:09
People can easily get lost. Focus on the problem, not the tool. What is that strategic needle that you want to move?
Speaker 1 00:17
Cohorts, communication, and use cases. They’re my quick tips to bringing people along on this AI journey.
David Droga 00:25
Creatives will always be relevant. The tasks are changing, not the necessity of what people need.
Kartik Hosanagar 00:33
Hello and welcome to Where AI Works, conversations at the intersection of AI and industry, brought to you by Wharton in collaboration with Accenture. I’m Karthik Hosanagar, back in the hosting chair one last time for a short recap and review of our inaugural season. We covered a lot of ground in those first four episodes, so I want to look at some highlights and break down my key takeaways for you. As I’ve been saying all season, things are changing fast, so let’s dive in. We kicked off the season by speaking with Jonathan Halvorson, the global SVP of consumer experience at Mondelez International. We talked about AI-driven content creation from a marketing perspective. My research shows that while AI can boost content efficiency, it may also lead to more similarity in content as everyone uses the same tools.
Jonathan Halvorson 01:22
I think in marketing there are some really exciting and clear use cases, and I think that’s why a lot of companies are starting in marketing. We get excited about focusing on content because we see the size of the opportunity. One of the biggest line items in any company’s budget is going to be what they spend on media to place advertising and what they spend on content creation. And if a company can improve the effectiveness of that content in media, or can improve the cost that they purchase that with, it’s going to have a big impact on the P and L.
Kartik Hosanagar 01:52
Jonathan emphasized the importance of embedding brand distinctiveness in AI-powered content, meaning that you want to make sure that your brand’s truths don’t get lost in AI-enabled content creation.
Jonathan Halvorson 02:05
What is the product truth that is core to the brand? What is the brand purpose? What is the tension it plays against? And if you define those things clearly, then I don’t think that you’re going to have that swell to the average and to the mean. But for brands who aren’t clear on exactly who they are in their foundations, I think they will find themselves in a sea of sameness. So I think the safeguard on it is: one, the clear brand foundations; two, humans in the loop; and three, just the discipline you put in, just the intentionality of how you use AI, and just recognizing the concern.
Kartik Hosanagar 02:40
On episode two, I sat down with Accenture’s Chief Marketing and Communications Officer, Jill Kramer. Jill talked about moving from fear and avoidance to excitement and curiosity about AI. Her journey felt deeply human to me.
Jill Kramer 02:54
I would love to tell you that I was fearless, that I was first in line with my hand up, saying let’s do this. I did start this thinking, wow, like, is this an existential crisis? And I made the decision, and I was very transparent about it with the full marketing and communications team at Accenture that I desperately wanted to be in the driver’s seat. And the only way you can do that is by getting really close, getting very hands on, deeply understanding, and taking some risks on behalf of yourself, your company, your brand, your team.
Kartik Hosanagar 03:27
We also got into the nuts and bolts of re-skilling teams at scale. A few of Jill’s insights: one, start small, move in cohorts. Don’t try to boil the ocean. Two, over communicate. The vision may be clear to you, but it takes repetition to make it resonate across the org. Let me repeat that. The vision may be clear to you, but it takes repetition to make it resonate across the org. And three, choose early use-cases, carefully. Nail the first two and employing curiosity and creativity, it will take off from there.
Jill Kramer 04:01
Curiosity and creativity is driven by exposure, by options, by the least restrained possibilities. And Gen AI unleashes that on behalf of the human, on behalf of the craft. So if you allow it to be shorthanded too, can’t Gen AI just write that for you. It’s an incredible disservice to the technology and the potential of it being applied to a function as important to growth and brand and, you know, the strategy of any given company as marketing is.
David Droga 04:33
Not all creativity is worth saving. Now, that’s a provocative thing to say, but there’s so much things that are deemed creative that are just not. You know, I look at architecture, I look at majority of advertising, I look at a lot, a lot of journalism – that could be better, that’s quite formulaic. I mean, they’re informed and influenced by something far more scary, which is conformity and research. So if we’ve created tools that allow us to sort of raise up and get rid of the mediocre middle, then that’s going to accelerate the people that actually have the talent and the know how and the ambition and the creativity to do more with it.
Kartik Hosanagar 04:33
For episode three, it was a real pleasure to chat with David Droga, the founder of Droga5, and the CEO of Accenture song. We talked about how AI changes the game for creative teams. David also mentioned that it’s natural to be fearful, but he doesn’t think AI will replace or eliminate human creativity.
David Droga 05:28
Every time creativity has been enhanced by a tool, yes, we all get intimidated by it, and obviously I understand the scale and pace that this is unlike anything we’ve seen, but I think of even when I started in the industry, I’m not looking back thinking I feel terrible that we don’t have rooms full of typographers or illustrators who are doing storyboards. The output is what matters, and when you focus on the output, creatives will always be relevant. The tasks are changing, not the necessity of what people need.
Kartik Hosanagar 05:54
And for the fourth episode, I was joined by Accenture’s Chief AI Officer, Lan Guan, to discuss AI at scale. She mentioned that many companies are investing heavily in AI, especially generative AI, but that execution can still be a struggle.
Lan Guan 06:09
Our latest research shows that 86% of the C-suite clients, they’re actually planning to increase their investment in AI. So this is real, they are spending money on this. They’re shifting their spend from other areas into AI, or they are just increasing the net new spend. Another piece of the research we have done also indicated less than 10% of the clients have actually scaled, in fact the number is 8%. So everybody’s asking: Okay, what happened to the other 92%? So it’s our responsibility to collectively go help them.
Kartik Hosanagar 06:44
We also talked about the biggest roadblocks to implementation, including issues around trust and governance, as AI becomes more autonomous with the rise of agentic systems.
Lan Guan 06:53
Yes, we talk about, you know, responsible AI, Explainable AI, probably for the last two decades it became another buzzword. When LLM came out, everybody started talking about AI safety, but I can tell you, the importance of AI trust is paramount now, simply because of the proliferation of AI agents. Not everybody will be creating LLMs, not everybody will be creating SLMs. Model customization still seems to be limited to the technical resources, but that’s not the case with AI agents.
Kartik Hosanagar 07:27
Our guests shared a lot of valuable advice and insights over the course of these conversations, but my main takeaway for business leaders is that AI-driven business transformation is not just a technology- based exercise. In fact, I would argue it’s primarily a people exercise, because one of the things that emerged from these conversations is, while the technology is there, we need to make sure that our people can embrace the change without fear. We need to make sure that culture change is feasible, and all of that requires much more from an organization than turning on a switch of technology. And of course, if you haven’t listened to the first four episodes in full, I would also very much encourage you to do so. This has been Season One of Where AI Works, conversations at the intersection of AI and industry. Brought to you by Wharton, sponsored by Accenture. Thanks so much for listening. Please follow so you don’t miss an episode. In just two weeks, we’ll be kicking off season two, hosted by my very own colleague, Serguei Netessine, and he’ll be focusing on the challenges surrounding the monetization of AI, an extremely important topic. I’m Kartik Hosanagar, bye for now.