Summer is still a few weeks away, but people are already talking about what will be 2018’s “song of summer.” New research from Wharton marketing professor Jonah Berger might help out the oddsmakers. Along with co-author Grant Packard, a marketing professor at Wilfrid Laurier University, Berger used natural language processing to study hundreds of songs and their lyrics to see what made some catch on while others failed to climb the charts. Berger recently spoke to Knowledge at Wharton about his findings, which are outlined in the paper, “Are Atypical Songs More Popular?”
An edited transcript of the conversation follows.
Knowledge at Wharton: A lot of your past research focuses on virality or why certain things catch on while others don’t. What inspired you to focus on songs this time?
Jonah Berger: We hear songs all the time. We’re in our car on the way to work, or we’re at home listening to YouTube. We like some songs and don’t like others. Some climb the Billboard charts and some fail. One question I wondered was why. Obviously, songs are tough to study. There are a lot of different factors that shape song success, from who sings the songs to what sort of melodies they have. We thought it would be interesting to look at a slightly more unexplored place, and that is the lyrics. Just by looking at the lyrics alone, could we pick up some traction on why songs succeed and fail?
Knowledge at Wharton: You examined songs within their particular genres — country songs with other country songs, R&B songs with other R&B songs. Why did you decide on this format?
Berger: We had a simple, and I think interesting, hypothesis, which is that the success of songs doesn’t just depend on their lyrics in general, it depends on how similar that song is from other songs that are popular recently. Take a country song, for example. A country song could be very like most country songs that are out there already. The lyrics could be very similar to what people usually sing about in country music, or the lyrics could be more different, more novel, more new for a country song. We wondered whether songs that sound more new, because their lyrics are different than most songs in the space, could be more popular with listeners.
Knowledge at Wharton: You found some interesting patterns among these song lyrics. For example, if somebody thinks that all country songs are about girls and trucks, they are not too far off, based on this research.
“There is this notion out there that certain genres care more about lyrics than others.”
Berger: We used natural language processing, which is a way to analyze text using software. We did something called LDA (latent Dirichlet allocation), which helps us figure out the underlying themes or topics in the songs. It can take a whole bunch of different songs with all of their different lyrics and find general themes. We found that there were 10 themes across all songs that tended to pop. Some songs talk about love, others talk about girls and cars, some talk about dancing. We asked, what part of each song is in each of these themes? And we found, like you mentioned, that country songs don’t only sing about girls and cars, but they sing a good bit about that theme.
Knowledge at Wharton: What were your key findings about how lyrical differentiation impacted song popularity?
Berger: We found that even though many things affect song popularity, like who sings it and the melody, we could understand what songs become successful just by looking at the lyrics alone and just by looking at how different a song is from its genre. We found that, on average, the more different a song was from its genre, the more atypical a country song was for country music, the more successful it was on the Billboard chart, the higher ranked it was. Even controlling for things like who sang the song, when it was released, etc., the mere fact that it was different from most other songs in its genre was connected to it being more successful.
Knowledge at Wharton: Is there a particular song that is an especially good example of these overall findings?
Berger: What is interesting is it is hard to tell the difference between songs just by listening to them. I bet if you were listening to a radio and I asked you how different this country song is from country songs that usually come on, you wouldn’t be very good at telling me, and I wouldn’t be very good at telling you. We often don’t consciously pay attention to all of the lyrics or even recognize all of the lyrics, but natural language processing allows us to find these implicit or underlying themes that drive success. So, it wasn’t driven by one song in particular; it was looking at it across all of these songs. Lyrics shaped whether they were successful or not.
Knowledge at Wharton: There were a couple of outliers here, and one of those was pop songs. Can you talk to us about that?
Berger: One thing we wanted to try to do is say whether this effect is causal. It is neat that successful songs tend to have more different lyrics, but are the different lyrics causing the songs to be successful, or might it be something else? We did a lot of work to show a causal effect, and one thing we tried to do was see whether it varies by genre. There is this notion out there, if you think about songs, that certain genres care more about lyrics than others.
You might imagine, for example, that lyrics don’t matter so much in dance songs because there aren’t many lyrics, if there are any at all. If we think about pop songs compared with country or rock, success in pop is often more about being the same rather than being different. We wondered whether in those two genres — pop and dance — we might see different effects, and indeed we did.
We found that lyrics didn’t matter much at all in dance songs, and similarity was better than difference in pop songs. It suggested it is not just about the lyrics themselves, it is about how the lyrics relate to difference and how that matters in the specific genre being examined.
Knowledge at Wharton: Could this technique also be used to predict a summer blockbuster or most popular beach read, for example?
“We found that, on average, the more different a song was from its genre, the more successful it was on the Billboard chart.”
Berger: That is exactly right, and that is what we are trying to do now. We are looking at thousands of movies to see whether we can predict how successful movies are going to be, in terms of box office sales as well as ratings online, based on their scripts. We’re looking at emotional trajectories, for example, in the scripts. We’re doing more work with music lyrics, we’re doing some work with content and text of books, and we’re also doing work with customer service calls.
Imagine you call an airline or an online retailer. How do the words the customer service representative uses, as well as how they use those words, affect how satisfied the customers are? Across a bunch of different domains, we are interested in words even though we don’t always pay attention to them. How might those words affect success or failure?
Knowledge at Wharton: How does this research demonstrate the value of natural language processing, and how do you see that helping to develop this area of research?
Berger: There is a lot of attention these days around artificial intelligence and machine learning, though most people in the general population don’t necessarily understand what that means. But one way that these tools are being used is to pull behavioral insight from text. There are all sorts of textual data out there from online reviews to things like song lyrics and movie scripts. Textual analysis or natural language processing allows us to pull behavioral insights from those reams of data, not just to predict what is going to succeed and fail, but also to use it to understand human and customer behavior. That is really the power of these tools — using them to understand things we might not have been able to understand before.
People may tell you they liked a song or a movie, but they may not know why. What this allows us to do is actually quantify what makes a song or a book or a movie successful, even if it is hard to study otherwise.
Knowledge at Wharton: Do you have a prediction for what is going to be the song of this summer?
Berger: I don’t have a specific prediction, but I bet it will be atypical. I bet it will be something unusual rather than normal.
