The stock markets climbed all summer, and were nearing record highs before the Dow Jones Industrial Average fell 243.36 points on Tuesday, down almost 4% from a five-year peak. Yet even before the drop, many investors didn’t trust the rally.

“This is the most disliked rally I can ever remember,” Charles Rotblut, vice president of the Chicago-based American Association of Individual Investors, said last week, referring to the recent stock run up, which ended abruptly after a string of dismal third quarter earnings reports. Washington gridlock — including speculation about the upcoming Presidential election, stubborn unemployment and the stalemate in Europe were making investors uncertain, Rotblut notes. Weekly surveys of the sentiments of the Association’s 150,000 members (with the option to pick “bearish,” “bullish” or “neutral”) have revealed bullish outlooks to be below average for 28 of the past 29 weeks. Bond allocations among surveyed members have been above average for more than three years.

Many investors are frustrated with low yields, they don’t have confidence in the market and they are worried about “another shoe dropping,” Rotblut says. “And nobody counts how many people have just checked out of the market,” he adds. “They tend to just very quietly move away.”

Wharton professors disagree about which way stock markets are headed and whether investors should be in or out. Some have pulled their own investments out of equities and don’t plan to return. Others say bearish investors are hurting themselves by withdrawing at the worst possible time and missing out on the market’s upside.

Chris Goolgasian doesn’t need surveys to tell him how retail investors feel about the stock market these days. As head of the U.S. portfolio management and investment solutions group for Boston-based State Street Corp., he meets regularly with brokers and advisors who manage individual client portfolios. In as many as 80 meetings since April, they have all said the same thing: How do I get my clients out of cash and bonds? “This is repeated at every meeting,” Goolgasian notes. “These advisors are meeting clients who are extremely bearish.”

Investors have moved about $2 trillion into cash and fixed income since the financial crisis, Goolgasian says. And despite a number of indicators that suggest the market today is “a fairly low risk environment,” many retail investors “don’t want to hear it. Money flow weekly is still out of equity funds and into bonds and cash.”

Investors who have swapped equities for bonds have “exchanged nominal safety for a real loss,” Goolgasian points out. “They are getting ‘safety’ in their accounts in that they are not seeing volatility…. But they are going to suffer real losses because the bonds and cash they bought are earning less than the inflation rate.”

Institutional investors are less likely to divest from equities, given spending goals that can’t be met with bonds earning 1.5%, Goolgasian notes. Individual investors, on the other hand, are suffering from a number of ‘isms’: “Pessimism about the job market, skepticism about the equity rally and cynicism about our political leaders and their ability to pull us out of this.”

Many investors pulled their money out of the market in 2008 and haven’t come back, scared of volatility and still reeling from the drama of the Flash Crash in 2010. That’s been a mistake, according to Wharton finance professor Jeremy Siegel. Even factoring in Tuesday’s drop, the market’s value has doubled since March 2009. Volatility has eased. For equity investors, Siegel sees sunny days ahead.

“The public unfortunately lags [behind] what’s going on in the market,” says Siegel, noting that rank-and-file investors tend to be bullish at the top of the market and bearish when the market is about to recover. “It is not unusual for the public to miss the first half to two-thirds of a bull market. Then they get in at the end, and they ride it down.”

To sophisticated market watchers, the fact that most retail investors are staying out of the market right now is actually a positive indicator, since history shows that public flows in and out of the market are usually badly timed, Siegel says. “I still think this bull market definitely has room to run,” he notes. “I can easily see stocks up another 20% to 30% from these levels in a year or two.”

Wharton finance professor Franklin Allen holds the opposite view — as well as opposite investments. “I don’t have much in the stock market at all anymore,” notes Allen, who describes himself as risk-averse. “It’s in Treasuries and real estate.”

Allen says he doesn’t know why the stock market is as high as it has been lately, and wonders if it’s a bubble. The recent rally may have more to do with quantitative easing by the Federal Reserve (QE3) than a positive economic outlook, he notes. Despite low returns on bonds and Treasuries, Allen says he wouldn’t go back into the market unless equity prices dropped by about 30%. “If you think the market is going to drop, then zero [return] is better than minus-20,” Allen points out. “That’s the perspective that people who are pulling out are using.”

Wharton finance professor Richard J. Herring says the market outlook remains uncertain. “It is hard to imagine earnings continuing to grow since they are already at an historical high relative to GDP, and the economic outlook is tepid at best,” he notes. “Many experts believe this is a market pumped up by QE3 and little else.”

Market volatility over the past few years may be keeping investors away, Herring suggests. “Although we are now back to about where we were in 2006, it has been a horrifying ride, and some investors bailed out at precisely the wrong time because they were terrified it could go still lower.”

A Crisis of Confidence?

Confidence in the markets has been damaged in a variety of other ways, with a number of incidents causing individual investors to wonder how well they are being protected by institutions such as the Securities and Exchange Commission, Herring adds. “The Madoff scandal, MF Global, the flash crash, the software error at Knight Capital, the NYSE settlement with the SEC over information sold preferentially to a program trader, the bizarre spectacle of the Facebook IPO and scandals at several major banks all have undermined the confidence of individual investors that they can be treated fairly,” he says. “It seems like a game rigged against [them].”

This is costly to society, because it means that less capital will be available for risk-taking and the liquidity of markets will diminish, Herring notes. “Although many overlook the point, we should realize that confidence in the markets is an enormously valuable public good. If investors trust that they will be treated fairly, they will be much more likely to place their savings in marketable instruments, and both the quality and quantity of capital formation will be improved.”

About 46% of American households held investments in stocks or stock funds at the end of 2011, down from 53% in 2001, according to the Investment Company Institute, a Washington, D.C.-based association of U.S. investment companies. “We have seen outflows from domestic mutual funds since 2007,” says Investment Company Institute senior economist Shelly Antoniewicz. Yet she says it would be wrong to suggest that investors are simply running away from stocks. “We don’t think that investors are fleeing” domestic equity markets, she adds. “We think of it as diversification.”

For example, from the beginning of 2007 to August 2012, a total of $196 billion flowed out of domestic mutual funds and exchange traded funds. During that same period, $361 billion went into international equity mutual funds and ETFs, which “more than offsets all the flows out of domestic equity,” Antoniewicz says out.

Investors are also putting more money into products such as target date funds and hybrid mutual funds, which contain a mix of equities and bonds, she adds. During the first nine months of 2012, for example, $45 billion flowed into hybrid mutual funds, up from $30.7 billion for the year 2011 and $23.7 billion in 2010. With two bear markets in the past decade, the willingness to take risk has gone down across all age groups, Antoniewicz notes. “Even within age groups, people are a little more risk averse. They are more conscious about diversifying.”

A General Malaise

Reticence to invest in stocks may also reflect “just a general malaise and a growing distrust for essentially all of our institutions,” says Olivia S. Mitchell, a Wharton professor of business economics and public policy and executive director of Wharton’s Pension Research Council. A Gallup poll in June, for example, found that only 21% of Americans have confidence in banks, down from 60% in 1980.

“The other thing you see over and over again is that people today are extremely pessimistic,” Mitchell adds. Worries include the U.S. economy, the looming fiscal cliff, continuing turmoil in Europe and a slowing economy in China. “If China’s growth is tapering off, what does it mean for the rest of the world? People are skeptical,” she notes.

Mitchell cites the latest Chicago Booth/Kellogg School Financial Trust Index, a quarterly survey of public attitudes toward financial institutions, which found that only 15% of those surveyed said they trusted the stock market, and 47% said it was likely to drop 30% or more in the next 12 months.

Her daughters, both in their 20s, echo public sentiment, Mitchell says. They argue that there’s no point in putting money in their 401ks because they could lose it, and if they put it in the bank, they won’t earn enough to beat inflation, “so they might as well spend it.” It’s not exactly what Mitchell wants to hear: Long term, that approach translates into having to save more, consume less and most likely delay retirement.

Mitchell herself pulled out of the stock market entirely in 1999, investing her pension in Treasury Inflation-Protected Securities (TIPS) instead. “And I’ve made more money investing in TIPS between 1999 and today than if I had invested in the stock market,” she says. “Since the year 2000, the market hasn’t performed very well.”

The problem now is that TIPS are paying negative returns, and she is reluctant to put all of her money in real estate. “I don’t think there is any easy answer except work longer, save more and spend less,” Mitchell suggests.

Most middle class households probably held too much of their investments in stock anyway, notes Wharton business economics and public policy professor Kent Smetters, which is why he is “not too upset” with the apparent trend away from equities. “Stocks should not be the main workhorse vehicle for basic retirement needs,” Smetters says. “Bonds should be used for basic retirement and stock for goals where falling short is more acceptable to the household.”

Smetters recommends that portfolios include bonds, TIPS and some stocks, but adds a caveat: “People should not use the expected return on stocks when estimating how much saving they need,” he says. “They should use a bond yield to adjust for risk. That forces them to save but also better manages risk.”