In January 2007, Uri Simonsohn, a Wharton professor of operations and information management, received an e-mail containing a New York Times article about a new rating from Consumer Reports, the venerable U.S. magazine that publishes monthly product reviews. The rating assessed various models of infant car seats and how well each fared in high-impact car crashes. The key findings in this ranking were dramatically different than a similar evaluation the magazine administered two years earlier. According to the new ranking, seats that had previously done well were now failing when cars were hit on the side. Simonsohn made a mental note to replace the seat he and his wife had purchased for their baby at a time when the product performed the best in Consumer Reports testing. But one thing led to another, and he didn’t buy a new one as quickly as he intended.
The delay turned out to be a good thing. Two weeks later, Consumer Reports editors admitted they made a major error in one of the magazine’s testing procedures, largely rendering the data invalid. “That gave me an idea,” says Simonsohn, whose previous research has focused on how online auction services and lotteries can determine consumer behavior. “How could I test people’s ability to ignore this information?” He ultimately devised a way to use data from an auction website to see whether news coverage of the initial results and the subsequent retraction influenced consumers.
Simonsohn discovered that car seat buyers reacted quickly to reports about the initial rating — with online shoppers bidding up the top-rated models while reducing their offers for those with poor ratings. But they also readjusted their behavior as soon as the results were retracted. This seemed to contradict a considerable body of research showing the reverse being the case — that most people don’t disregard information that they subsequently learn is incorrect. “All the evidence [indicates that I] should have found that people are unable to correct [their opinions after information is deemed wrong],” notes Simonsohn.
His report — titled, “Lessons from an Oops at Consumer Reports: Consumers Follow Experts; Ignore Invalid Information” — is scheduled to be published in a forthcoming issue of the American Marketing Association’s Journal of Marketing Research. As the study notes, the results of his research could be useful for companies, or even politicians, who can now look for reassurance that acting quickly and aggressively to disseminate corrected information, as Consumer Reports did, can counteract the effects of an earlier blunder. Simonsohn acknowledges, however, that an incident involving both a trusted news source such as Consumer Reports and parents concerned about their children’s safety may also provide unique circumstances.
When the award-winning magazine, published by the non-profit Consumers Union, reported in early 2007 that nine of the 12 car seats it tested “failed disastrously” in high-impact crashes — with some of the seats twisting violently or flying off their bases upon collision — it quickly became national news. That even the most popular seat models failed to make the grade was shocking enough to garner significant attention.
But there was a problem. For the 2007 rating, Consumer Reports did not have the equipment needed to carry out an additional test so cars were hit not only head-on but also from the side, which is why the magazine enlisted the help of a federal agency — the National Highway Traffic Safety Administration (NHTSA). But the NHTSA made a major blunder. Although Consumer Reports asked the agency to simulate a side collision at 38.5 miles an hour, the test crashes were nearly twice that velocity — at 70 miles per hour, a speed at which most infant seats would fail. As soon as Consumer Reports learned about the error, the publication quickly retracted the original findings and began alerting consumers. Editors published an apology on the publication’s website and sent postcards to print subscribers. The magazine made national news once again — but this time, for the wrong reason.
Better Safe Than Sorry?
What interested Simonsohn most as the events unfolded was something that didn’t receive much attention from the media: How did the original article and the later retraction affect consumer behavior? At first, he predicted that despite the widely publicized correction, the initial rankings would have a lingering impact on the car seat market because often “it’s particularly hard to ignore misinformation.” One reason is that although the initial report was later denounced as wrong, consumers may ignore a particular car seat model nonetheless just to be overly cautious, especially when something as emotionally charged as child safety is involved.
Indeed, numerous pieces of research have covered the ways consumers act on bad information. Simonsohn’s paper discusses what other researchers call the “debriefing paradigm,” in which participants in studies who receive false feedback continue to act on the statements even after they have been retracted. Another phenomenon is the “dilution effect,” in which the presence of irrelevant information reduces the weight that a study’s participants place on relevant facts.
Simonsohn set out to use hard data on prices for car seats from an online auction service as a way to test whether and how consumers acted on incorrect information, or on its retraction. The data the professor obtained covered the three months prior to, and the three months after, the Consumer Reports article. Notably, he examined bidding on six types of seats that had been ranked by the magazine in 2005 and 2007. The analysis covered 5,471 auctions. Of the six brands of car seats, four fell in the 2007 ratings when compared with the previous test because of the bad data, and two achieved better rankings. The data obtained by Simonsohn showed that for every position gained or lost in the ratings, the average auction price of a seat either increased or decreased by three percentage points. In other words, if a product dropped four places in the ratings — which was the average move for the brands Simonsohn studied — its bidding price fell by 12%.
According to Simonsohn, the impact on the different brands was dramatic because the erroneous results were so different from those that consumers had received before the 2007 article. “The fact that Consumer Reports provided wrong information that was uncorrelated with existing beliefs created a unique opportunity,” he writes in the paper. “The large impact of the new safety assessment on auction prices provides direct evidence that [perceived] experts [such as Consumer Reports] causally influence consumer demand.” Indeed, the reputation of Consumer Reports for being a voice of authority on consumer safety, along with the ubiquity of news accounts about the initial tests and the retraction, all but assured greater insight into the longstanding debate about expert influence on consumers, Simonsohn notes. The paper cites data from NewsBank, an aggregator of articles from local newspapers, finding that while 51 articles covered the initial study — which the service says is a comparatively high number — there were 118 stories published about the retraction.
The widespread coverage of the error helped Simonsohn when he looked at the seats’ auction prices following the correction. The data showed that bidding prices for the six brands he was studying reverted almost immediately to what they had been before the first article — with one exception. One of the seats, manufactured by Evenflo, had received a failing grade correctly in the initial article. “This was the car seat that not only failed the 70 mph test but also the standard 30 mph one, and hence its negative information was not invalidated,” the paper notes. “This suggests, again, that after the retraction, people still had access to the original information — otherwise they wouldn’t know the Evenflo was [still considered] unsafe.”
Beware of Bad PR
Simonsohn’s report also grapples with broader issues: Are there lessons from the highly publicized case that could help other companies that are dealing with public relations crises related to bad information? “Marketers frequently encounter situations where they wished consumers would ignore invalid information,” he writes. “For example, companies are often interested in eliminating negative associations with their brand (e.g. ‘American cars are unreliable’), dismissing rumors about their products (e.g. ‘the food sold at that restaurant is made from mutant chickens’) or eliminating wrong beliefs that have been debunked by scientific studies (e.g., ‘drinking coffee while pregnant leads to fetal development problems’).” What’s more, this conundrum can also include politics, which is dominated today by a fast-moving, 24-hour news cycle that rarely lingers on a story. In several well-documented cases — such as that of the late Richard Jewell, a security guard falsely linked in the news to a fatal bombing at the 1996 Olympics in Atlanta — people wrongly associated with crimes have struggled to regain their reputations.
There are aspects in the car seat episode that are fairly unique, Simonsohn concedes. When shopping for a seat, parents are heavily influenced by safety information over and above other considerations, such as price. Also, he asserts that no single source of information in the United States is as influential as Consumer Reports. “The fact that Consumer Reports gave a substantial and compelling reason to ignore the safety information [i.e. that it was incorrect] is likely to have contributed to the success consumers had in ignoring it,” he writes. In addition, consumers were able to validate their early conceptions about which seats were safe from other sources, such as buyer comments on websites like Amazon.com.
Although previous research asserts that consumers struggle to disregard faulty information, Simonsohn’s findings show that they can make corrections when given time and authoritative new information. The report notes: “In the context of studying people’s ability to ignore information, for example, lab studies have not allowed subjects to obtain additional information, postpone the decision or physically eliminate the irrelevant information.” Decision-making experts may argue “that people can’t ignore information, even if you’ve retracted it, that it will stick in people’s minds,” Simonsohn adds. “But not always — here’s an example that’s different.”
