It’s been nearly 35 years since the precursor to The Vanguard Group, now the country’s largest mutual fund company, offered the first index-style mutual fund to individual investors. Through mutual funds and exchange-traded funds, indexing now accounts for about $1.6 trillion in investor assets, and indexing has legions of diehard believers.

A parade of studies has shown why: Index funds, which try to simply match the performance of a broad market sector, have consistently beaten “actively managed” funds, where professional money managers attempt to outperform the market by picking the hottest stocks and bonds.

Over the 23 years ending in 2009, actively managed funds trailed their benchmarks by an average of one percentage point a year. If a benchmark like the Standard & Poor’s 500 returned 10%, the average managed fund investing in similar stocks would therefore have returned 9%, while an index fund would have returned 9.8% to 9.9%, giving up only a small amount for fees.

But if this is so, why do a majority of investors still choose active management? Indexing, or “passive management,” accounts for only about 13% of assets in mutual funds holding stocks. It doesn’t seem to make sense. Experts have offered a variety of explanations: Investors are duped by slick managed-fund marketing, they don’t know the facts or they believe “you get what you pay for” — that paying higher active-management fees should buy better results. Maybe they are deferential to “professionals,” or believe they are smart enough to pick the active managers who are better than average. All those explanations have one thing in common: They assume investors are not very bright.

But new research shows that investors who embrace active management may, in fact, be behaving perfectly rationally. “You don’t have to say investors are silly or unaware of the data in order to explain this,” says Wharton finance professor Robert F. Stambaugh. His research — presented in a paper titled, “On the Size of the Active Management Industry,” co-authored by Lubos Pastor, a finance professor at the University of Chicago Booth School of Business — shows that investors use active management in a kind of arms race to unearth a limited number of bargain-priced investments.

Sniffing Out Good Deals

A key factor, Stambaugh and Pastor argue, is investors’ rational belief that active managers have a better chance of sniffing out good deals if there are not too many managers looking. A rational investor will pull money out of actively managed funds if the results are disappointing, but he will not pull out entirely because he realizes that other investors will withdraw money, too. That will leave less money to chase the few bargains, making them easier to find. “We wanted to come up with a rational explanation for why, despite this rather mediocre track record for active management, the lion’s share of money is still managed that way, as opposed to passively,” Stambaugh says. “We think we have done that.”

Advocates for active and passive management are like meat eaters and vegetarians, each convinced their approach is best. Active investors bet that fund managers can use good research and judgment to find mispriced assets such as stocks and bonds. Passive investors believe it is nearly impossible to do that consistently over the long run, and many studies have shown them to be right.

Among the reasons: When all the active managers are taken together, their holdings are so large that they reflect the entire market, and their performance will therefore match the market’s. But the expenses they incur in their search for hot stocks and bonds undermine their results. The average actively managed stock fund, for example, incurs annual expenses of about 1.3%, or $1.30 for every $100 an investor has in the fund.

That means the manager must beat the market’s performance by 1.3 percentage points a year just to break even. If the market returned 8%, the fund would have to return 9.3%, a very large margin to achieve year after year. Put another way, if the fund manager selected investments that returned 8%, his investors would receive just 6.7%, or about 16% less.

Index-style funds incur much lower expenses because they don’t employ large numbers of analysts and stock pickers. They simply buy the stocks or bonds in the underlying index, making only minor adjustments as investors’ money moves in and out of the fund. Today, most index funds and exchange-traded funds have expenses below 0.2% a year, and many of them charge less than half that much. If the market returned 8%, the fund’s investors would enjoy returns of 7.8% to 7.9%, and in some cases, more.

In most years, only about a third of actively managed funds beat their benchmark indexes, such as the Standard & Poor’s 500. And managers who succeed in one year often fail the next, suggesting that many winning results are no more than luck. “Numerous studies report that these [actively managed] funds have provided investors with returns significantly below those on the passive benchmarks, on average,” Pastor and Stambaugh write in their paper.

Many investors who prefer active management understand the corrosive effect of higher expenses, and know that managed funds as a group do poorly over the long term. But many bet they can select the fund managers who are better than average. To do that, the manager must find assets that are “mispriced.” Their odds are better when there are fewer managers in the hunt, just as a gold prospector does better when he is the only one panning a stream.

“The more money you’ve got chasing opportunities to outperform passive benchmarks, the more elusive those opportunities become,” Stambaugh says. If a lot of active managers spot an under-priced stock, they will pour money into it, and its price will rise because of the high demand, eliminating the bargain.

Looking at Alpha

In their paper, the authors argue that investors are in a kind of eternal struggle, each hoping the others will withdraw to make it more likely that the remaining managers will find the nuggets.

For active managers, the Holy Grail is “alpha,” the industry’s term for a return that exceeds the market average when the investment’s risk is taken into account. There are two ways to conceive of alpha on an industry-wide basis, Stambaugh notes.

In the first view, alpha is a constant figure, a percentage by which active managers would beat or trail the market. If investors held this view, they would pull all their money out of actively managed funds as soon as they concluded that alpha was negative — that managers could not consistently match or beat the market. Stambaugh and Pastor studied active and passive fund returns from 1962 through 2006 in relation to the amount of money investors had committed to each type of strategy. Because active management trailed consistently, investors who viewed alpha as constant should have pulled all their money out of actively managed funds by 1969. But they didn’t.

In the second view, alpha is not constant. Instead, it depends on “industry size,” or the amount of investor money chasing bargains through active management. This, Pastor and Stambaugh believe, is the view that explains why investors remain committed to active management despite its poor track record. As a group, investors expected results for active management to improve when less money was committed to that investment strategy.

After a period of poor performance, the rational investor realizes that other investors will pull some of their money out of actively managed funds, making bargain finding easier. The investor will also pull only some of his own money out, because he wouldn’t want to miss the chance of improved results after others pull money out. “If some money is pulled out of the industry, that is going to allow more mispricing opportunities to exist,” Stambaugh says.

Because investors cannot tell exactly how alpha is affected by industry size, they cannot fine-tune their strategy, so they simply maintain a heavy allocation to active management. “Most of us think that if we could just get the other guy to take his money off the table, then there would be a better opportunity for me to outperform,” Stambaugh states, adding that “we realize that if we pull out a lot, all we are doing is leaving money on the table for the guys who don’t pull out.”

Under this view of alpha, it was a rational decision for investors to keep around 70% of their money in actively managed funds through 2006, rather than pulling out entirely in 1969, Pastor and Stambaugh conclude.

Stambaugh says he does not dispute those who see other factors in investors’ embrace of active management — namely industry marketing, pressure from brokers and financial advisors, ignorance and so forth. But his work with Pastor, he adds, provides a simpler explanation for investor behavior: “There’s no reason to resort to calling investors stupid if you can explain [their behavior] without doing that.”