For investors, the battle between actively managed mutual funds and their index-style competitors is like a decades-long sports rivalry. Index funds have been gaining ground for years, largely because their fees are lower and performance is often better, but managed funds still dominate the fund industry.

Now, new research by two Wharton professors and a colleague at the University of Chicago unearths an additional factor in the difficulty managed funds have in performance: the growing size of the managed-fund industry. The bigger it gets, the harder it is for any individual fund to beat its benchmarks — typically, an index like the Standard & Poor’s 500.

Since investors can match a benchmark’s performance with index-style funds, beating a managed fund’s return, the work underscores the uphill battle the managed-fund industry faces in hanging on to its share of the market. In the 10 years ending in 2013, the market share of indexed stock investments has grown to 35% from 17%, according to Morningstar, the market-data firm. Indexing has also become more popular with bond-market investors, and with institutional investors like pensions and endowments.

“The size of the industry in which they are competing does seem to play a role in the ability of [actively managed] funds to outperform the benchmark,” says Wharton finance professor Robert F. Stambaugh, who co-authored the study. “As the industry got bigger, the ability of the funds to outperform declined.”

The reasons are not entirely clear, he adds, though he notes it is possible that as the industry grows, more stocks receive heavy scrutiny, so there is less chance a fund manager can find mispriced gems.

Stambaugh and his colleagues, Wharton finance professor Luke Taylor and Lubos Pastor, a finance professor at the University of Chicago’s Booth School of Business, describe their findings in their paper titled, “Scale and Skill in Active Management.”

Fees Add Up

Studies have long shown that in the average year, the majority of managed funds fail to perform as well as their index-style competitors, largely due to costs. Because they must pay teams of researchers, analysts and stock and bond pickers, managed funds typically charge annual fees exceeding 1% of the fund’s assets. While that sounds small, it adds up over time, undermining the fund’s compounded gains. Put simply, if the fund had no gains over a 10-year period, it would lose 1% a year to fees, leaving the investor with a stinging loss.

“As the industry got bigger, the ability of the funds to outperform declined.”

Index funds don’t employ all those professionals. Instead, they buy and hold the stocks or bonds in a standard industry benchmark. Indexers tracking the Standard & Poor’s 500, for instance, buy the 500 stocks in that index. Many index funds can thus charge fees of 0.2% or less, sometimes only half of that. Active-fund managers have a hard time picking enough hot stocks and bonds to overcome the index funds’ low-cost edge.

The new research by Stambaugh, Pastor and Taylor is the first to look at the additional effect of fund-industry size. In recent years, researchers have examined evidence that a given fund’s performance deteriorates as the fund gets larger, or has more assets under management. That suggests a relationship between management skill and fund scale.

“If scale impacts performance, skill and scale interact: For example, a more skilled large fund can underperform a less skilled small fund,” Stambaugh and his colleagues write.

Though experts had not nailed down a reason for this, it is commonly believed that big funds have to invest in stocks of big companies, because small companies do not have enough shares in circulation to soak up the vast amounts of cash big funds must invest. Big funds therefore miss out on the small-stock gems that offer benchmark-beating performance. But it is not completely clear that this theory is correct.

Previous academic work has also identified “liquidity restraints” as a performance factor. As an individual fund, or the industry as a whole, grows larger, trades become so big they warp the balance between supply and demand. Big buy orders enhance demand enough to raise the stock’s price, so that the same fund’s subsequent purchases of that stock aren’t such good bargains. Similarly, a big sale causes an oversupply, which drives the price down, reducing the proceeds from subsequent sales.

“Consistent with such liquidity constraints, there is mounting evidence that trading by mutual funds is capable of exerting meaningful price pressure in equity markets,” Stambaugh and his colleagues write.

Funds try to minimize this effect by conducting trades piecemeal and stringing them out — but this strategy goes only so far because of the need to hurry before a good deal is gone.

Fund vs. Industry Size

While it is clear that the size of the fund or size of the industry can affect a fund’s performance, Stambaugh and his colleagues wondered which is more important.

In theory, if all managed funds followed the same strategy, buying or selling the same stocks at the same time, the funds’ combined “industry” size would matter more than their individual sizes. The same amount of stock would be bought or sold regardless of whether the industry was composed of a lot of small funds or a few large ones, and it is the total amount of a specific stock bought or sold that has the effect of moving the price up or down.

“There is mounting evidence that trading by mutual funds is capable of exerting meaningful price pressure in equity markets.”

At the other extreme, if each fund were to follow a completely different strategy, buying and selling stocks different from their competitors’ trades, stock prices would change more dramatically from large funds’ trades. Fund size, rather than industry size, would be the key factor in reduced returns as trading volume grew.

To see whether the greatest impact comes from fund size or industry size, the researchers examined 3,126 funds from 1979 through 2011, a period of dramatic growth for the industry. The study found mixed results when it looked at how the increase in an individual fund’s size affects performance, but it revealed a clear pattern as the industry grew: Performance got weaker. For example, if actively managed funds accounted for 60% of all stock market activity rather than 50%, active funds’ returns would be expected to decline by a fraction of a percentage point per month.

By filtering out the effects of greater fund and industry size, the researchers showed that the average managed fund has done better over time because the managers have become more skilled. One estimate of skill, for instance, found that average returns on the first dollar invested grew from 24 basis points per month in 1979 to 42 in 2011. (A basis point is 1/100th of a percentage point.) Much of the increase in skill is due to new funds, as younger ones tend to do better than older ones. “In short, active funds have become more skilled over time,” the researchers write. (These results exclude the effect of fees.)

But investors have not benefited because the managers’ greater skill is offset by the effects of the growing industry. “We argue that the growing industry size makes it harder for fund managers to outperform despite their improving skill,” the researchers report. “The active management industry today is bigger and more competitive than it was 30 years ago, so it takes more skill just to keep up with the rest of the pack.”

“The effect of greater skill is being offset by the fact that there is more money being managed,” Stambaugh notes. This is especially evident in funds that invest in small companies, as well as funds with higher-than-average turnover and higher volatility.

Aging Funds

While the young funds tend to outperform older ones, they lose that edge over time. But this is not because the managers lose their touch. Instead, the study found, it was due to the industry’s growth as the young funds aged.

“If skill were to stop rising, then any continued increase in the size of the industry would probably hurt performance going forward.”

“Taken together, our results are consistent with the following narrative,” Stambaugh, Pastor and Taylor write. “New funds entering the industry tend to be more skilled than the incumbent funds, perhaps due to better education or greater command of new technology. As a result of their superior skill, the new funds tend to outperform their benchmarks as well as older funds. As these funds grow older, though, their performance suffers as a result of the continued growth in industry size, which is associated with the steady arrival of skilled competition. Learning on the job might alleviate the negative impact of growing industry size on performance, but it does not eliminate it.”

Further research will be needed to pin down the reasons industry size affects fund performance, the researchers note, adding that it also is not yet clear what has caused fund managers to become more skilled over time. Also awaiting further study is the question of whether there are niches, such as foreign-stock funds, that are still small enough that active management can pay off significantly, perhaps because less competition makes it easier for managers to find bargains.

For the ordinary investor, the study underscores the fact that growing competition can hamper returns of actively managed funds, Stambaugh says. “It’s basically a reminder that competition matters.” In recent decades, growing skill among active-fund managers has helped to offset the damaging effects of industry growth, he notes. However, he cautions: “If skill were to stop rising, then any continued increase in the size of the industry would probably hurt performance going forward.”

The study excluded the effect of the higher fees investors pay for active management compared to indexing. Add the fees to the headwinds from industry growth, Stambaugh says, and it becomes even harder for active funds to keep up with the indexers.