Risk managers armed with the most sophisticated quantitative tools available did not foresee the biggest development in a generation — the systematic breakdown and global contagion of financial markets. In an interview with Knowledge at Wharton, John Drzik, president and CEO of the Oliver Wyman Group, Richard J. Herring, a finance professor at Wharton, and Francis X. Diebold, a Wharton professor of economics, finance and statistics, discussed how to build a more informed risk management model. All three took part in the recent Wharton Financial Institutions Center and Oliver Wyman Institute 12th Annual Financial Risk Roundtable 2009.

An edited transcript follows: 

Knowledge at Wharton: Today we’re speaking about risk management with Wharton professors Dick Herring and Francis Diebold, and also with John Drzik, who is president and chief executive officer of Oliver Wyman Group.

I want to start with a very simple question. Can you really measure risk accurately?

Dick Herring: I think the last year shows that we can’t, that there are lots of things we can’t quantify very successfully and that we became overconfident in the things we could quantify. We’ve made great strides in risk analysis, risk measurement, and aggregating risk. But we’ve tended to focus most of those efforts on things that are relatively easy to manage. And even some of those relations broke down. We simply didn’t have enough data. Our techniques were not good enough. We weren’t using enough forward information and, unfortunately, this crisis has blame that can be shared across the entire spectrum of participants, from regulators to participants in securitizations and even to risk managers themselves.

Francis Diebold: I think that’s right. That reminds me of our project on the known, the unknown, and the unknowable that we’ve done here at Wharton at the Financial Institutions Center in conjunction with the Sloan Foundation. What we focused on and really came to realize more intensely was that there’s a whole spectrum of risks ranging from market risk to credit risk to operational risk to legal and reputational risk and things beyond that. Some are comparatively easy to model, which isn’t to say they are easy, but they’re comparatively easy. Others are really challenging and basically we’re not good at at all.

Knowledge at Wharton: Could you tell us what some of those are?

Diebold: As you go across that spectrum from market (market meaning risks associated with movements) and prices, to credit (meaning risk associated with defaults or bankruptcies, those sorts of things) to operations (meaning almost anything ranging from computer system failures to terrorist attacks) ….

Herring: Ice in the parking lot.

Diebold: And the thing is, ironically and somewhat depressingly, it’s not the easier-to-model risks in many cases, but the harder-to-model risks that really bring firms down as opposed to just swing earnings by 3% or 5% a year. So we need to be humble, but we have made progress.

John Drzik: I’m just going to add that there wasn’t enough attention on the unknown risks rule versus the known in that part of the problem. For people who are risk analysts or practitioners who were in the academic community who tracked risk there tends to be a focus on risk modeling where there’s data, rather than risk modeling where there’s risk. You can build much more sophisticated models where there’s lots of data to work with. That doesn’t mean you’re focusing on the biggest problems that the firms face, because that’s where you have thin data sets and often have to make judgment calls. People with an analytical bent are often uncomfortable going into that sphere. But that’s really where a lot of the real big risks that firms face are.

Knowledge at Wharton: So you swim in the known waters and you sort of ignore the unknown waters, but then one day you’ll get thrown into the deep end of the unknown waters.

Diebold: Absolutely. There’s the old joke about the guy who looked for his lost car keys under the lamppost because that’s where the light was.

Herring: I guess I would add (to the very good list of things we don’t know as much about as we think we do) one that is really perplexing because it’s how these things interact at a systemic level. It seems to be very dynamic. It can change rapidly. I think even those who understood the subprime crisis were totally amazed that it brought down virtually the entire international financial system. And these things are so inherently complex in terms of their interconnections and their knock-on effects that they may in fact be unknowable. And we may have to be able to think more about how you detach certain critical functions from the system.

Diebold: I think there are two key issues, or at least two, both of which are very difficult and related. The first is understanding correlations across banks, financial institutions …

Herring: And understanding how they vary over time.

Diebold: That’s number two. And they certainly are different in crises just as volatility is different in crises. And, of course, the way that they’re different is often in very adverse ways. Correlations rise just when you don’t want them to rise and you lose, for example, portfolio diversification benefits just when you need them the most.

Herring: The dirty little secret of diversification is that it disappears when you need it most. The only thing that rises in falling markets is correlations.

Knowledge at Wharton: Does that mean we have to tear everything down and start from the bottom to build back up? What is salvageable from the way things have been done?

Drzik: I don’t think it’s tearing everything down. I think it’s building new pieces of the infrastructure. There have been tremendous advances in market risk measurement and credit risk measurement. Those have their purpose within the organization. There’s nothing wrong per se with measuring daily value of risk and using it in certain respects in managing trading business or lending business. I think we over-relied exclusively on one type of risk measure and got a little false comfort that these sophisticated measures were protecting and measuring everything we needed to measure. So I think it’s more, keep what we have that can be refined, but add to it. Add things that are more focused on the unknown risk. More focused on stress and scenario analysis than just statistical analysis, which is more historical, and look back at nature. And to come to the point around correlation, if you look at historical patterns of things that haven’t had a life that long … What’s the correlation of different classes of subprime mortgages in the period that subprime existed would tell you something very different than if we just said let’s look at a stress scenario where house prices move and you know all the things are going to hook together in a scenario of that type. So it would give you a different answer than trying to precisely estimate correlations and volatilities in a bottom-up way and construct risk that way. Nothing wrong again with that bottom-up statistical measurement. It’s just that it has to be one element of a risk manager’s tool kit, not the only thing that’s relied upon for protection by an institution.

Knowledge at Wharton: So the old ways had a sort of overconfidence about them, which was unwarranted and seemed to justify taking on risks that in retrospect we know were too much. So that leaves us with the known unknowns, or the unknown knowns?

Diebold: Before punching into that, it’s helpful and important to distinguish between models and their successes or failures and uses or interpretations of models. It’s really important for users of models to maintain a healthy skepticism about those models. In other words, you don’t want to be celebrating your success and giving each other awards and thinking that these problems are solved. The right attitude is rather, “Surely this model is wrong. Where is it most likely to be wrong? When is it most likely to be wrong? How could I improve it? You need to have a healthy skepticism, and the converse of that is what John was describing, namely complacency or celebratory sort of feelings like, “We’ve nailed it now.” “We understand risk management.” “We’re calculating 5% value of risk so that problem is finished. We’ll move on.” It is clearly not the case, and it is important to do everything you can to heighten your awareness and the firm’s awareness that skepticism is important with models.

Herring: Another thing I’d like to highlight that follows on John’s comment is that there’s been a failure in most risk technology to look at forward-looking indicators. They’re out there. We saw, long before it happened, that Bear and Lehman and lots of other firms were going to fail because of the credit default swap spreads widening up. But if you looked at the backward statistics that you had to estimate from, you weren’t picking that up. Virtually every firm that failed over the last year had much more than the minimum required regulatory capital, which simply means the regulators aren’t measuring capital very well.

Knowledge at Wharton: When you talk about early warnings … Back in the ’90s, Long-Term Capital Management, LTCM, went up in flames, or went down in flames, in a way that resembled what happened today. Should there have been more lessons? It was seen at the time as an anomaly, but in retrospect maybe not.

Drzik: LTCM had some of the most sophisticated risk modelers in the world working for them. So it was …

Herring: So it was an omen.

Drzik: I think it’s another point reinforcing the general theme that all models are wrong and maybe some models are useful. But having the sense that they’re all wrong … I think there was a huge amount of confidence bordering on arrogance in LTCM that they had nailed the modeling and that there was a certain trend that markets always took and they were going to take a big bet in that direction. And then the model didn’t quite predict exactly where this was going to go. It’s a more complicated story than that, but I think there was a confidence in convergence that was based on very sophisticated thinking and modeling, but which wasn’t 100% right.

Herring: The other key thing was that they were relying on diversification. And suddenly after the Russian default everything that was not absolutely above suspicion, pristine quality, started to fall at the same time.

Diebold: There’s an interesting story that illustrates how judgment could have been much better blended with models in exactly this LTCM case told by Steve Ross at MIT. And that is, if one had simply gone to the data — a different set of data than the risk models were using, which might have just been 10 years of history on the particular assets being held, which as Dick and John have emphasized is woefully inadequately — and looked instead at Russian government bond issuances over the centuries, you would have seen that not a single such bond made it to maturity since, I think, it was 1840 or something. And Steve quite appropriately points out that if you want to do real risk management you have to take a much broader picture, think about putting things in perspective, and again ask what could be wrong with the statistics. How could I flesh it out with broader knowledge? Again, that’s not to say that the statistics are wrong or that we shouldn’t be using models. We certainly should. But we have to put them in context and try to think about where they might not be adequate.

Herring: It’s a very perplexing problem in a system that’s as innovative as ours. How much history do you use? If you had looked from LTCM’s viewpoint about spreads, and they were basically betting on convergence of spreads, they were looking in the beginning at spreads that over the past five to maybe 10 years were at historic highs … But if they’d looked 20 years ago, it was routine.

Diebold: That’s right.

Herring: And the question was, was it a special circumstance? Or had markets changed? Or had we developed new ways of bringing stability? Similarly, before the current process, we were all talking about the great moderation where volatilities were virtually nothing and credit spreads were pricing in perfection as far as the eye could see.

Drzik: And house prices always go up.

Herring: Yes.

Diebold: Just to amplify that tension a bit, simultaneously there are certain aspects of reality that are preserved over the centuries, and it pays to know what’s happened over the centuries. There are other things that are actually drifting and breaking and changing as institutions and laws and the complexity of financial markets evolves. So that on one hand, you want a long sample, a long perspective. On the other, you don’t. And where you are in that spectrum is a tricky business.

Knowledge at Wharton: It might paralyze you, might it not, if you start looking at Russia and saying, “Well, I’ll never invest in Russia with that record, because it’s a horrible record.”

Herring: Another source of insight that’s often overlooked, taking the housing crisis for example, is cross-country comparisons. Although it hadn’t happened to us in a long time, it’s happened literally scores of times to countries just like us. And it usually happens after a very long period of rising house prices. But sooner or later, reality catches up with you.

Knowledge at Wharton: Can you give some examples of those countries?

Herring: Sure. In fact it’s hard to figure a country that didn’t have the problem. Britain had the problem and they’ve had it again. New Zealand had the problem a long time ago and they may be headed for it once again. The Netherlands had the problem and, in fact, there’s a fabulous series you can look at in the Netherlands that is really a gem. It goes back at least 400 years along the Herengracht Canal. The Herengracht Canal, when it was built and now, is one of the very finest neighborhoods in the Netherlands. And the Dutch, being really good at keeping numbers, have the prices at which each one of those houses turned over for over the last 400 years. And you see these huge cycles up and down maybe once every 20 or 30 years, but if you had bought and held over the 400 years you’d have made less than you did on Treasury bills.

Knowledge at Wharton: It’s interesting how the Dutch seem to provide good examples because the tulip craze is often used as a good example of a bubble. John, let me ask you a question about, because you have both feet planted firmly in the business world, how much of this is behavioral crowd mentality? These correlations we talked about, the things that shouldn’t happen at the same time end up happening because there’s a stampede of some sort, and that’s what wasn’t predictable. Do you subscribe to that?

Drzik: That was definitely part of it. If you look at institutions getting into, say, something like the subprime market. One of the reasons institutions who didn’t initially go into it [eventually] did was because of pressure.

Herring: Everybody else was.

Drzik: Everybody else was in it. And if you were the one standing alone you got pressure from investors, analysts, internal pressure to be competitive and get into the market. Your stock price would suffer if you weren’t following the same earnings growth trajectory as your competitors. And potentially if you didn’t follow suit, you could end up being an acquisition target for one of them. So they might have been on an excessive risk track, but in the short term that was rewarded. It might have been enough to put you out of business in a different way. So I think there was that type of psychology.

Knowledge at Wharton: Not to mention your bonus was based on the next three months’ performance.

Drzik: Right. We had too short a time horizon where we — as a kind of very wide community, not just people within the financial institutions, but the analysts and investors and others who followed them, and the media and so forth — all focused on who was gaining at a particular point in time. It pulled the herd mentality along a little bit further than it would have otherwise been.

Herring: We had an interesting discussion this morning because we had on a panel Bob Chappell, who heads a mutual organization. It’s a mutual insurance company that’s been around for 150 years. And he was pointing out that because they are mutually organized he can keep his eye on the long rung. He has to. He doesn’t care about quarterly earnings. He doesn’t care about having more capital than hedge funds might think he ought to have. He simply does what he thinks he needs to do to keep his policyholders safe. Now I’m not suggesting everybody should mutualize all of a sudden. But there is a different dynamic at work for sure. The other issue that came up was do investment banks really make sense as publicly held corporations? They’re just about extinct at this point as such, now that they’ve become bank holding companies.

Diebold: Our point may be proven.

Herring: But as partnerships they might have been considerably more careful, although smaller, with regard to risk. [Bear Stearns] was undoubtedly a safer place when Ace Greenberg was on the floor watching every trader all the time.

Diebold: On this issue of behavioral effects and their importance, I would say clearly they’re partly responsible. It would be very hard to argue that they are absent in this situation. To make that argument is to say that markets are perfectly efficient at all times and I just don’t think it’s true. But having said that, I think the greater part of the crisis is linked to very sophisticated and highly rational economic agents responding optimally to the incentives that they are faced with. And a big part of the situation is misaligned incentives. Every time you turn over a stone, you find one or another example of that, whether it is loan originators with no skin in the game selling off these loans instantly, and no problem for them. Or whether it’s ratings agencies being paid by the people they’re rating, or just more generally the situation in which institutions know that explicitly, or at least implicitly, the gains they make are going to be privatized, and when they get into trouble it’s going to be socialized. So these are outcomes that are not due to irrational behavior in many cases. They’re actually highly rational. It’s just that they’re undesirable.

Knowledge at Wharton: Can I ask you each to take about 30 or 40 seconds to do the impossible task of suggesting what are the two or three things that might best help us in this situation?

Herring: I think the most important thing is for us to figure out a way to resolve large systemically important financial institutions in such a way that there are not unbearable spillovers. And that means having prompt corrective action measures so that you can intervene before they are absolutely bankrupt. It means probably requiring them to in some cases spin off some things that are just too difficult to manage. It means probably discouraging institutions from becoming too big to fail or too complex to fail.

Knowledge at Wharton: John, do you agree with that?

Drzik: I would focus at two levels. I think at the regulator level and then at the level of the individual institution. I think there are governance improvements in both. At the regulator level I tend to agree that the increase in focus should be on systemic risks. In individual institutions I think a lot of the improvement in governance could come from redesigning incentives as well as repositioning the role of risk management within the entity to have more influence on strategic decisions, because it’s not connected very well today.

Diebold: I would agree with Dick in the sense that effective resolution procedures are crucial moving forward, and we’re not there yet. For me, it’s not so much how to get through this crisis. We’re basically going to just muck our way through it. That’s what we’ve been doing. That’s what we are doing. The bigger question for me is how to avoid things like this in the future. How to put things in place now that make it easier next time around and hopefully reduce the probability of having this another time around. And there for me it’s finding ways to deal with the moral hazard in situations that involve rescuing financial institutions while still having effective ways to rescue them. There’s a tension there, but we need to deal with that.

Knowledge at Wharton: That’s a tough one. Just real quick, does anybody have any ideas on how you do that? How you handle this moral hazard issue, which is a constant?

Herring: I think it really is resolution policy. Resolution policy doesn’t mean you don’t rescue some parts of an institution, but you do impose discipline on the parts that are not systemically important. So you might well find that you have to keep the foreign exchange market going, and do what you need to do to keep that going. So a resolution authority needs resources. It’s not going to be able to intervene so perfectly that it can always stop a decline before it hits bankruptcy. But the idea is to have at least some of the creditors at risk so they’ll be helping the regulators and monitoring the institution and giving warnings that we can read on things like credit default swap spreads and other things.

Knowledge at Wharton: So just to be clear, when you talk about resolution, you’re talking about something similar to the Resolution Trust that helped clean up the S&L mess in the late ’80s?

Herring: Actually, I’m talking more like what the FDIC can do with bridge banks. That should have helped us in this instance at least with the big bank problems we have, except for the fact that when the legislation was enacted they were given authority only over the banks, not the bank holding companies. And the bank holding companies have responded by putting 20% to 40% of their assets in the holding company. They would have to go through bankruptcy, which would be another Lehman Brothers. So you’ve got to look at regulation as a regulatory dynamic.

Knowledge at Wharton: It’s an arms race.

Herring: Yes. And it’s not a very fair arms race either I have to say. The private sector is fleeter on its foot and it has more resources, and the regulators are always running to catch up.