California’s woes keep mounting. Now, on top of earthquakes, mudslides, grassfires and drought, there is the extraordinary electricity crisis. Prices have soared, shortages have taken parts of the state to the brink of blackouts, politicians are brawling and several major utilities are near bankruptcy.

Most agree that part of the problem is a simple run of bad luck. Prices of natural gas used by many California generators have soared. Economic growth in the late 1990s turned California from an electricity exporter into an importer. The winter has turned unusually severe, heightening electricity demand. Dry weather in the Pacific Northwest has reduced supplies of hydroelectric power. And one day earlier this month there was a true act of God: a Pacific storm created huge waves that drove kelp into the coolant intakes of a key nuclear plant.

But most observers also agree the situation has been exacerbated by the 1996 legislation that partially deregulated the electricity industry. Some say the crisis proves deregulation was a bad idea; others argue California’s deregulation simply didn’t go far enough, allowing wholesale prices to float while putting a cap on retail rates.

“If you’re going to deregulate something, you’ve got to deregulate everything,” says Gerald Faulhaber, a Wharton public policy and management professor on leave to serve as chief economist for the Federal Communications Commission, which is wrestling with phone deregulation issues. “If you deregulate half of it, it’s a recipe for disaster. There’s no middle ground.”

With many states considering electricity deregulation, what does the California case say about whether a free market for power can work? Not that deregulation is a bad idea, but that other regions can avoid the kinds of problems that have bedeviled California.

Paul R. Kleindorfer, who is co-director of Wharton’s Risk Management and Decision Processes Center and has studied deregulation in California, as well as more successful efforts in Pennsylvania, New England and overseas, cites five key problems in the golden state:

  • The state’s largest power companies were forced to sell their generating plants.
  • Intermediaries, the brokers who find power for utilities to distribute, cannot work effectively.
  • Supplies did not keep pace with growing demand.
  • Emissions restrictions cause a dramatic price increase when generators work at full capacity.
  • Flaws in the system’s design put generating companies in a position to drive up prices.

Historically, production, transmission and distribution of electricity was heavily regulated throughout the country because it was thought that only a monopoly could shoulder the enormous costs and difficulties of providing this vital service.

But other capital-intensive industries such as trucking, railroads and airlines were deregulated with relative success in the 1970s and 1980s, says Faulhaber. So attention naturally turned to electricity.

As free market fever ignited in the 1990s, and proponents argued that deregulation could make the power industry more efficient while encouraging competition that would give consumers the lowest possible prices.

At the same time, technological change made electricity deregulation more feasible, says Witold Henisz, a Wharton management professor who has studied deregulation. It is now possible to build relatively small and efficient gas-fired generators, he said, and generation, transmission and distribution no longer need to be handled by the same company. Electric power contracts can be traded in markets that work something like the stock exchange, allowing participants to nail down prices for future delivery and to hedge risks of extreme price changes.

But the power industry is quite different from most others. Unlike other commodities, such as grain, electricity cannot be stored. Hence, the system has to be capable of furnishing all that is needed at periods of peak demand, even if portions of it are idle much of the time. At the same time, most of the industry’s costs – for generators, transmission cables and distribution grids – are fixed. It’s not possible to quickly cut costs when demand is low, as a factory owner might do by going from three shifts to two.

Since it is hard to quickly increase capacity or to cut costs, imbalances in supply and demand can cause extraordinary swings in price in an unregulated system. If demand exceeds supply, a desperate utility is at the mercy of outside suppliers who can charge premiums to feed power into the grid. If demand falls below supply, generators facing fixed costs that cannot be reduced figure they might as well collect any revenue they can, even if it means selling at extraordinarily low prices.

In a January 2001 paper titled, “The Trouble With Electricity Markets (and some solutions),” Severin Borenstein, director of the University of California Energy Institute, points out that the deregulated airline industry is very different. Although empty seats are worthless once a plane takes off, just as unused electricity generating capacity is wasted, airlines can quickly move planes to different routes, constantly matching supply with demand. Electric utilities aren’t as agile.

Moreover, Borenstein writes, deregulated airlines can adjust prices on short notice, and customers will quickly react. California’s electricity deregulation, however, did not allow this. While wholesale prices are allowed to fluctuate on the “spot” market according to supply and demand, retail prices – those charged to customers – remain capped at low levels.

This was meant to protect consumers from price spikes on the spot market, keeping monthly bills steady. The risk seemed acceptable during the deregulation debate because California at that time could produce 30% more power than it needed, so utilities were not heavily reliant on the spot market. Surprisingly, though, despite the excess capacity, California’s regulated electricity rates in the mid-1990s were twice as high as those in many other states, largely because California utilities owned lots of expensive nuclear plants and regulators allowed them to pass those costs on to customers. It was easy to imagine that opening the market to competition would cause prices to settle to levels enjoyed elsewhere.

But when retail prices are capped, customers have little incentive to conserve, Borenstein writes. Most important, they have no incentive to conserve when the system needs it most – on a hot summer day or a cold winter one. In contrast, an airline with more customers than it can handle can raise prices, generating extra revenue to offset losses at other times when planes fly with empty seats. California’s utilities cannot do this.

California’s deregulation also prohibited the utilities from entering into long-term contracts with electricity suppliers. This was partly out of fear the companies would commit to pay high prices for years on end. And it was meant to force them to use the spot market, where power for immediate use is traded at constantly fluctuating prices. Reformers thought that a spot system with many users would unleash competitive forces that would keep prices low.

Without long-term contracts, producers don’t know how much power to generate, Kleindorfer says. Production cannot be ramped up instantaneously, as it may take a week to prepare to increase output. Relying on the spot market, then, aggravates imbalances between supply and demand, causing wild price swings.

In the past few months, high demand in California drove spot prices far above those that could have been locked in earlier with long-term contracts. According to some reports, on some days in December spot rates jumped to $1,500 per megawatt hour, which normally costs less than $50 to produce. That’s enough power to serve about 1,000 homes for an hour.

The California deregulation forced the state’s biggest utilities to sell many generating plants to other companies, which would then supply power to the wholesale market – in effect, selling it back to the utilities. This looked like a way to break up monopolies. But selling these plants put the utilities at the mercy of the wholesale market. Today, the three largest utilities control just 15% of the state’s generating capacity, down from 55% before deregulation.

If utilities own their own plants they have a hedge against price spikes, and can in fact enjoy higher revenue when prices go up, says Kleindorfer. In California, higher prices simply push utilities that don’t generate their own power into the red.

Meanwhile, the state’s economic growth overwhelmed the system, which has not added a new power plant in more than a decade, partly because of stringent environmental concerns, partly because utilities found new plants too big a gamble amid the uncertainty of deregulation.

Then gas prices jumped. Suddenly the system was stretched beyond its limits and overly dependent on the volatile spot market. Since last spring, two of the state’s biggest utilities, Southern California Edison and Pacific Gas & Electric, have spent an estimated $12 billion more to buy power from wholesalers than they have received through retail rates paid by customers.

Los Angeles and some other areas in California are under severe environmental restrictions governing release of nitrous oxides, which cause ozone pollution. Each plant has a federal permit to release a given amount, and the permits are traded at prices governed by supply and demand, says Kleindorfer. As the plants produced at higher and higher levels, demand for the permits soared, causing prices to rise from $1 per pound of nitrous oxide last spring to $30 this winter, he said. That added to the wholesale price.

Finally, he said, there is growing evidence that all of these conditions gave generating companies so much “market power” that they could drive prices higher by limiting production. They could gain more from higher prices than they would lose by selling less electricity, he said.

In hindsight, California’s electricity deregulation looks especially ill conceived. At about the same time, Pennsylvania, Texas and Wisconsin also deregulated, with much better results. Unlike California, these states took pains to ensure there would be plenty of supply. The utilities retained much of their generating capacity, and retail prices during the transition years were set high enough to draw competitors into the market. Also, Pennsylvania’s demand is not growing at anywhere near California’s pace; Texas has streamlined the approval process for new plants; Wisconsin has improved transmission lines so it can import power form neighboring states.

But, to be fair, these states also have been luckier than California, which is so heavily reliant on natural gas for electricity production. Pennsylvania, for instance, uses lots of cheap coal. “If coal prices had gone up the way gas prices did, I think Pennsylvania would have been the headline story, not California,” says Haas School of Business professor Catherine Wolfram.

The problem in California is too complex to suggest a simple solution, but most experts agree the state must hasten construction of new power plants to close the gap between supply and demand. Cutting red tape could help. It also is clear to most that it doesn’t work to let wholesale prices rise without limit while capping retail prices; retail caps may have to be lifted.

Borenstein says the system could be improved by allowing utilities to enter into long-term contracts with generators, making them less dependent on the volatile spot market. Henisz adds that utilities should be allowed to use derivative products to further hedge against price spikes, the way stock market investors use options contracts to insure portfolios.

Borenstein argues further that consumers should be given incentives to restrict electricity use during peak periods. If electricity costs fluctuated throughout the day, users would quickly see they could save money at peak times by turning down air conditioners or programming dishwashers to switch on in the middle of the night.

While it might be difficult to quickly install “real-time” electricity meters in millions of homes, most industrial and commercial customers already have these meters, which measure not only how much power is used, but when, says Wolfram. These non-residential customers account for about two-thirds of electricity use, so a real-time pricing system could have a significant effect quite soon.

The result: if all these changes worked, a truly deregulated market with a better chance of keeping supply and demand in equilibrium, which is the key to keeping prices on an even keel.