How Credible Are Polls? Is There a Better Way to Predict Outcomes in Politics and Business

Nota A los Lectores:
Vaya aquí a leer en español a de Universia-Knowledge at Wharton

This election year’s hard-fought presidential race has brought increasing focus on the credibility and methodology of polls – a focus that could have implications for politics, but also for business forecasting, according to Wharton faculty.

With growing uncertainty about the value of polls, people are looking more closely at new ways to predict election outcomes – including the use of aggregate poll results, expert opinion surveys and betting markets. “It’s interesting that there is so much attention paid to polls, but my guess is that polls are the least accurate way of gauging the election,” says Wharton marketing professor J. Scott Armstrong.

Polling and the Information Age

Betting odds were published routinely as election forecasts before they were displaced by polls in the 19^th century, says Wharton statistics professor Abraham Wyner. Since then, pollsters – such as The Gallup Organization and national television networks – have become adept at designing accurate, and frequently expensive, polls. Recently, he adds, new information technology has made it easier to conduct polls that reach many people faster, but are not as accurate as a well-designed, smaller sampling. “The question is, how will the information age affect polling? Can you use a computer or a cell phone to do an effective poll?”

Many of the new polls, such as instant Internet and television polls, are meaningless, Wyner says. “A television station can say, ‘Vote on our web site, or use your cell phone to call this number for free.’ These kinds of high-tech polls are cheap to do, but they are as valid as making up numbers. They are nonsense.” The worst part about that kind of poll, adds Wyner, is the disclaimer about its scientific validity. “They should just say, ‘We make this up.'”

Meanwhile, new forms of communication are amplifying the impact of polls, Wyner adds. “Weblogs are constantly referring to the different polls. There are all kinds of discussions in real time, and high-profile chats going on, about this information. People are connecting in a way they never could before, and information is proliferating in a way it never could before. The polling organizations are just a cog in a very big network that includes bloggers, individual bettors and newspapers. It’s all part of a big system. It’s not simply ‘the polls’ anymore.”

Criticism of polling is nothing new, according to Frank Newport, editor-in-chief of The Gallup Poll. “We have seen polls become matters of controversy in each election, in some more than others. It may be accelerated this year because the election is particularly intense. Our data show people feel strongly – on either side – about this race, and because it is so close, there may be higher sensitivity to all kinds of information.”

Newport says pollsters have been struggling against declining response rates for a number of years, in part because of new technology. Caller-ID, cell phone-only homes and a general reluctance by potential respondents to answer any calls after years of intrusion by telemarketers are all obstacles to an accurate, randomized poll. “The work has become more challenging, no question about that. So far we think we are able to do good, valid, projectable polls and we continue to monitor these changes.”

Yet the rise of quickie opinion polls does concern survey research professionals, Newport adds. “It’s important that we make sure people understand that legitimate polls are random – where we select people randomly.” Instant call-in polls may be interesting, but these have no scientific validity, he says. “Television likes them from a marketing standpoint because it gets viewers involved.”

Pollyvote

As part of an experiment in forecasting, Armstrong has created a web page tracking what he calls the Pollyvote, based on a parrot character named Polly. The parrot averages different ways of forecasting and comes up with her own predictions about the presidential race.

Polly’s diet of information includes traditional polls, including those by Gallup, ABC News, and the Washington Post. From Oct. 2 to Oct. 18, data from 39 polls were integrated into the Pollyvote. In addition, Polly takes into account 10 quantitative models developed by political scientists and forecasters and assesses the odds on the Iowa Electronic Markets, an online futures exchange where people can trade on economic and political events.

With two weeks to go to the election, Polly says President Bush will capture 51.8% of the two-party vote – less than what the quantitative models suggest, but better than the odds given by the experts and the Iowa futures market.

Polly has been forecasting since March and her views are coming in closer synch with the more traditional polls. Armstrong says that is predictable. “As we get closer we expect things to converge and that’s happening. We have always expected the polls would be inaccurate further out, but as you get closer you expect them to be more valid.” Research shows that an aggregate of different methods of predicting outcomes reduces error, Armstrong explains. Under ideal conditions, combined forecasts are at times even more accurate than their most accurate components.

Armstrong’s goal for the project is to use election predictions as a hook to show corporate managers that they can build better business forecasts for their companies using an aggregate of inputs. “What I’m looking for is some way to demonstrate to management that the science has a payoff in real life,” says Armstrong. The Blue Chip Economic Indicators work on a similar consensus theory, he points out, but few business people have adopted the method when building their own internal forecasts.

The Polly page also includes a measure called the Delphi survey, a consensus of 16 experts published periodically and melded into the Pollyvote. “We suspect that 16 experts will be more accurate than 1,000 voters being interviewed,” says Armstrong.

He tends Polly with two other academics, Alfred G. Cuzan, professor of political science at the University of West Florida, and Randall Jones, professor of political science at the University of Central Oklahoma.

Like Polly, a parrot who is only capable of repeating what she hears, the professors are trying to keep their own leanings off the web page. The three have agreed not to share whom they are planning to vote for. “The idea is that Polly has no opinion. She hardly has a brain,” says Armstrong. “She’s the perfect one to report back on what people are saying.” The experiment will continue after the election, and Armstrong and his partners will evaluate which methods were most successful. “We think the techniques are useful in business forecasting and will be useful in other elections around the world and in other voter issues in the states. There are many applications.”

The biggest problem with most forecasting is that people rely too heavily on their own judgment and experience, he says, adding that in many cases people turn to polls or other forms of forecasting merely to support the view they have already formulated.

Armstrong describes attending a corporate meeting in Bangkok several years ago during which company leaders projected that a new product would drive sales up 20%. Armstrong asked each of the people in the room to make his or her own projections. No one forecast a sales increase of more than 5%. “If you want to know the sales of a new product, have everybody in the office bet on what the sales will be and use those results,” he suggests. “The results produced in a betting market will be more effective than a traditional meeting where everybody is listening to what the boss thinks.”

Red Sox vs. Yankees

Betting odds historically have been used to predict elections and are emerging as a new form of forecasting, according to Wharton professor of business and public policy Justin Wolfers. He examined various methods of predicting the outcome of the 2001 federal elections in Australia and found that polls were fairly accurate predictors of election results in a short-to medium time frame of about six months. Economic models worked better for a longer horizon. He then compared those results against betting odds from one of Australia’s largest bookmakers, Centrebet. The bookmaker “not only correctly forecast the election outcome, but also provided very precise estimates of outcomes” in a host of individual electorates. “Particularly in marginal seats,” Wolfers writes, “the press may have better served its readers by reporting betting odds than by conducting polls.”

Betting markets act as a way of distilling masses of information, the same way that markets compress information to determine prices, Wolfers explains. He acknowledges that perhaps some people would be willing to lose money by betting with their hearts, rather than their head, as a way to boost their chosen candidate. For example, he admits that he placed some emotionally-driven playoff bets on the Red Sox against the Yankees in this year’s baseball playoffs.

“You might think guys like me are crazy, but the question is whether the odds are wrong as a result of us,” says Wolfers. “This is where you need to think of a betting market more like the stock market. I might buy [the stock of] a particular firm for any reason, but investors like Warren Buffett care only about profits. If Red Sox fans bid up the price, then Buffett will buy Yankees stock until it gets to equilibrium.”

Wolfers is working with Tradesports.com – an Irish betting site that where buyers and sellers can sell contracts on potential political outcomes – to examine how betting markets function when they are asked to determine conditional outcomes. The site offers wagers on a Bush victory given several scenarios, including a red homeland security alert in place on Nov. 2, and a situation where Osama bin Laden is “neutralized” by Nov 2. “This allows us to get a handle on a market’s conditional expectations, or what the market says is the correlation between the two events,” Wolfers says. “We can get the expectation without having the event occur. We can have serious punditry in which people are putting their money where their mouth is. That’s rare on Sunday mornings” when, he notes, the talk shows abound in “cheap talk.”

As for the Nov. 2 election, Adam Grossman, a second-year MBA at Wharton, has been wagering on Tradesports.com since July. He says the online betting markets are still immature and slower to react to information than established capital markets. For example, it took 24 to 36 hours to reflect the presidential debates. “That’s a lot of opportunity.”