Shawndra Hill, a senior fellow at the Wharton Customer Analytics Initiative, likes to dig into the details. As someone who studies data mining, she looks for new ways to apply what she finds to solve business problems. Hill’s latest research paper, “Television and Digital Advertising: Second Screen Response and Coordination with Sponsored Search,” focuses on TV ads, online search and the connections between them. The paper was co-authored with Gordon Burtch from the University of Minnesota and Michael Barto, a data scientist at Microsoft. Hill recently spoke with Knowledge at Wharton about what she found.
An edited transcript of the conversation follows.
Knowledge at Wharton: What is the focus of this research?
Shawndra Hill: What we aim to achieve is to find new ways to measure TV ad effectiveness. Let me take a step back and talk about how people typically measure effectiveness. Large brand advertisers will usually ask another company to survey consumers and ask them questions like, “Did you see the ad? Would you like to recommend the product that was advertised to your friends? How did you feel about the ad?” So, questions about their attitudes.
They might also look at sales data and correlate that with the amount of spend that they’ve made. What we hope to do is look at more granular data that reveals itself in the searches people post on large search engines. What we’re hoping to do, or have done, is link TV ad data at the aggregate level, where it can tell us precisely which television show, what time and which locations an ad was shown. Then we look at search data around that TV ad, before and after, to see whether there was an impact on the search behavior.
We’re trying to look at the ability to coordinate advertising efforts, not just on television but also on digital platforms like sponsored searches. What we do is combine data from TV ads and then link that to the search data. Not just the searches, but also conditioned on somebody making a search — did they click on a sponsored search ad or not? We combined data from all of these sources to make causal claims about the impact of TV ads on digital behaviors, towards measuring the effectiveness of TV ads.
Knowledge at Wharton: This research capitalizes on a phenomenon that’s grown in the last couple of years called second screening. No longer do we just watch TV, but we’re often sitting on the couch, looking at the TV and scrolling through our phones at the same time. When you looked at this a little more closely, what were some of the key takeaways that you found?
Hill: That’s a great observation. I should probably take a step back and tell you the research questions that we were interested in. The No. 1 research question we’re interested in is just that: How do behaviors in response to TV ads manifest themselves via these second screens? We do in fact see that there’s an increase in search behavior after a TV ad is shown. But that’s manifesting itself primarily on smartphones. The smaller the device, the more likely someone is to respond directly after a TV ad, digitally.
Because we have very granular-level search data, we were also interested in this interaction with the sponsored search ads. Finally, we wanted to look at how the TV ads impacted different users in various ways. For instance, we were interested in heterogenous effects on demographics — age and gender. Do certain genders respond differently to a particular creative that’s shown in a particular television show? Similarly, we looked at device. That’s how we were able to discern that the response was coming primarily from the mobile phones.
Knowledge at Wharton: What were some of the findings that were most surprising to you? One thing that stood out to me is that you found when this increase in searching on your phone is going on, it only amounts to about three minutes.
“The smaller the device, the more likely someone is to respond directly after a TV ad, digitally.”
Hill: That’s right. You hit the nail on the head in terms of the surprising findings. There were two that I think are obvious in hindsight, but we didn’t necessarily anticipate. The first one was one that we already talked about. By disaggregating the data and looking at different cohorts — people searching from smartphones versus tablets versus PCs — we were able to see that the significant effect in terms of the bounce in searches after a TV ad was happening only on mobile phones. That’s the first thing that was surprising to us. Although in hindsight, it makes sense. If you’re sitting in front of the television, you’re not going to bring your desktop to watch television, right?
The second one was, because we’re looking at very fine-grain windows, we were able to see minute by minute the dynamic change in how people search after a TV ad. We found that we’re seeing it either in the first, second or third minute after the TV ad. At first, we thought, “Wow, we expected this thing to sort of slow down, but maybe tail off.” The reason, we suspect, is that TV ad segments are almost always exactly three minutes. So, people are probably switching their attention back to the television show after the TV ads are aired.
Knowledge at Wharton: If I am an advertiser, then I have this three-minute window. People are on their phones, looking at these ads. What can I do to capitalize on this information that you found?
Hill: The implications of our work, I think, are many. The first one is that we’re finding that the search response to television ads is manifesting itself primarily on mobile phones, and from prior research — not ours — we know that people are more likely to click on the first ad [in a search listing] only on a mobile phone when compared to a PC or desktop. That’s primarily because of the footprint, right? You only see the first ad.
What that suggests is, if people really are moving to mobile phones when they’re watching television, then if you’re an advertiser and you really want to keep their attention, you should spend the money to make sure you’re the first ad that shows up.
But I think the work has even broader implications because we can see who is responding. Let’s take two examples. Let’s say you have only one ad creative — one TV ad, one commercial. Let’s say it’s a new product and you want to know who’s responding. You can launch that TV ad and basically look at the response in the way that we have and see which types of customers are responding and where. That can help you optimize your other advertising efforts, to do more here or less there, depending upon what you find.
The other example I wanted to point out is, if instead you have many ad creatives — maybe you’ve done some focus tests and know which one small groups like. But you don’t know in advance what the broader ad response will be. You can launch all four of those, or however many ads, see who is responding the best, and then adjust how you present those ads over time.
What this approach allows for is to do near real-time optimization of ads with very aggregate level data. You asked the question of, “What are people doing now?” For the most part, people are measuring advertising effectiveness in the ways that I mentioned when we first started. So, asking people via surveys, “Did you see the ad?” Or looking at sales data. But the future is quite different in that now there are solutions for TV networks and even solutions that sit outside of TV networks that allow people to buy advertisement programmatically.
“If you’re sitting in front of the television, you’re not going to bring your desktop to watch television.”
Right now, we’re not all the way there. There’s going to be programmatic buys that more advertisers will do, as well as something called addressable TV, where people can actually advertise to individual households. If you could do that, then looking at this aggregate-level data is kind of unnecessary. But until we get there, this way is a good way to begin to optimize campaigns.
Knowledge at Wharton: Were there some interesting things that you found in terms of demographic differences in how people reacted to this?
Hill: We did, and that was one thing that we thought would be most interesting to advertisers. We looked at just age and gender, but still that’s enough to give an advertiser insights. We could match the demographics of the TV show. For instance, if you look at sporting events, those tend to skew male. And then [you can] ask, when an advertisement is shown in a sporting event, what audience members are most likely to respond? We found — perhaps obviously — that when a TV ad is shown during a sporting event, men are much more likely to respond to it…. Women really don’t respond more than they would otherwise.
Knowledge at Wharton: The idea is maybe that the audience you want to go after with these ads are the people that are already going to be watching anyway, which I assume they know. But it transfers over to online searches as well.
Hill: That’s right. You can check that the people that you are targeting are actually the ones responding. That’s kind of like a validity check that your strategy is a good one. In addition to that, if you have two types of shows that men are likely to watch, you can compare and see which type of show, when an ad is placed in it, are men most likely to respond to that ad.
There could be all kinds of things going on. Perhaps in some shows, people are more engaged with the show and are less likely to turn away from the commercials, for instance. Or maybe it’s a longer show and they get up and do other things. You can look at the match between the type of show and the audience that’s responding.
Knowledge at Wharton: Does this research also play into the idea that more people are turning away from broadcast TV and towards streaming? When I watch Hulu with ads, for example, it’s asking me, “Do you want the experience of this? Or do you want a travel video about California, or do you want something about a cleaner?” Can this also be applied to beyond broadcast TV?
Hill: It can be applied to other advertising strategies where you have a specific time stamp associated with the event. That could be placing a billboard in a particular location and then taking it away. It could be that you have a radio advertisement, and it has a certain time. The type of methodology that we actually advocate for is one that allows us to tease out the causality between an event that has a specific time and behavior that happens after that event by comparing it not only to what happened before, but also to some control group that we come up with.
“What this approach allows for is to do near real-time optimization of ads with very aggregate level data.”
But in any event-based advertising, it would work. To answer your question about Hulu and let’s say Netflix, those solutions for media consumption are a little bit different in that they know who you are. They have your account information.
Knowledge at Wharton: They know what I’m watching.
Hill: What they can do is closer to the addressable television example that I mentioned earlier, where people can now advertise directly to individual households. Companies like Hulu and Netflix have the ability already to do one-on-one advertising. And they can use your behavior either on their own site or by matching their data with third parties to target to you directly.
Knowledge at Wharton: What sets this research apart from other research that’s been conducted on this topic?
Hill: There are a few things that set it apart. The one thing is the granularity of the data. Because of the scale of the data, we were able to look at minute-by-minute response, by different locations. The second thing is that for these searchers, we also have demographics. Very crude, high-level demographics. But we were able to then look at these heterogeneous treatment effects for demographics and then also by device type, which no one’s done before.
Then finally, this combination between not just looking at search response but looking at the clicks. So, conditioned on a search, looking at the sponsored ad clicks is something that’s also novel. Looking at how firms might begin to coordinate their advertising efforts is something that hasn’t been done before when looking at response to TV ads.
Knowledge at Wharton: What’s next for this research? I know you’ve done a lot with social TV in the past couple of years, but where are you going to go with this next?
Hill: There are a number of obvious extensions. We have already started to bring in just organic search. We focused in the first paper on sponsored search, and we can look at what an organic search response actually is for the brand. Does that make a difference in their likelihood to click after an ad?
We’re also looking at other types of digital responses, not just search. So, we’re looking at clicks on web pages associated with the brand and where people are coming in from when they make those clicks. We’re asking questions around which specific advertising platforms might be most effective right after a TV ad campaign.
“We found– perhaps obviously — that when a TV ad is shown during a sporting event, men are much more likely to respond to it….”
After those things that we’re already working on, what we plan to look at is assigning people to different categories. Instead of assigning them to demographics like male or female, we assign them to a place on the purchase funnel. Are they ready to buy? Are they just seeking information? Are they comparison shopping? We want to know whether the TV ad is more or less effective, depending upon where people are in that purchase funnel.
We have used observational data techniques, and we feel pretty strongly that our method for teasing out the causal relationship between the TV ads and the search response is pretty solid. However, what we’d really like to do is run an experiment while TV ads are running to make sure that what we’re finding with the sponsored search response is really true. That, in fact, there is the impact on sponsored search results that we’re seeing.
I guess the pie-in-the-sky future work would be to actually run experiments using addressable TV. Our work, I think, will last for quite a while. Because although addressable TV exists today, there are not that many advertisers that have adopted it right away. What we want to see is whether these different combinations of advertising lead to more or less spend, more or less clicks, more or less search for information. And in using addressable TV solutions in combination with experiments on sponsored search, you can get precisely at that answer.
