AI and Financial Regulation

Itay Goldstein: I am Itay Goldstein, professor of finance at the Wharton School, currently the chair of the finance department. This is “Future of Finance,” a series where we explore the new advances in the world of finance and how the future of finance is shaping up.

This is the third season of the “Future of Finance” series. And in the third season, we are diving into the very important topic of AI and finance, [especially] how AI is changing the world of finance. We are examining the very important angle of the future of finance and AI in finance, which is how AI is going to affect financial stability and how we should regulate this technology when we are thinking about its introduction into financial markets and financial institutions.

This conversation pairs very nicely with the launch of a new initiative at the Wharton School. This is the Wharton Future of Finance Initiative. I am one of the academic directors of this new initiative, alongside my colleague, Patrick Harker, who is the former dean of Wharton and also the former president of the Philadelphia Federal Reserve. The Wharton Future of Finance Initiative studies the financial system and its support of the real economy, particularly focusing on two forces that shape it — the stability that sustains it and the innovation that transforms it. For more information about the Wharton Future of Finance initiative, I invite you to visit our website using the link in this episode’s description.

Speaking of financial stability and how it relates to innovation, I am very pleased to welcome our guest for today’s episode, who is the ideal guest to speak on the interaction of innovation and stability. This is Sarah Breeden, who is the deputy governor for financial stability at the Bank of England. Sarah leads the Bank of England’s work on financial stability, the supervision of financial market infrastructure, international issues, payment innovation, and fintech. Sarah and I spoke together this past June in a panel in the European Central Bank Forum in Sintra, Portugal. We discussed the many moving parts of regulating AI in finance. That will be the topic of the episode today. [Welcome], Sarah.

Sarah Breeden: It’s an absolute pleasure to be here. It’s hard to think of a more topical set of issues to be talking about, Itay.

Goldstein: Let’s dive right in. Financial regulators have adapted to many technological changes over the years. One could say that AI is just the latest version of that. Maybe you can take us a bit into this topic and tell us, is AI different? And if it’s different, then why?

Breeden: We’ve been looking at AI with the financial services firms here in London for ’five, six, seven years. We set up a public-private partnership with them where we were talking through how it was being used. And for many years, we’ve just thought, “Oh, that’s great. It’s being used to drive efficiencies. It’s being used to identify fraud. It’s being used to help defend from a cyber perspective.”

So, it’s been there and being used, but there hadn’t been anything that was flashing particularly red about how it was being used by the firms. The use cases fell into two categories. They’re either cyber or fraud, where you needed the latest technologies, and the best models, to be able to make sure you were doing your bit against the bad guys. Or, they were being used for relatively low-risk tasks, helping people do research, and those sorts of things.

For a long time, we’ve been working with firms, understanding how they’ve been using it, and happy that our regulatory regime was doing a decent job of managing the risks. But suffice it to say, I’m not sure that as we look ahead, we have got as much to be comfortable about with the arrival of agentic AI.

How Agentic AI Raises the Stakes

Goldstein: You say that agentic AI is really the thing about AI that worries you the most. Can you say a little more about that? What it is about agentic AI that we should worry about?

Breeden: Look, agentic AI is different, right? We’re moving beyond systems that help me make decisions towards systems that are increasingly taking actions for themselves. It’s not writing text, summarizing documents, answering questions, coming up with PowerPoint presentations. That’s not helping me do my job. Agentic AI is given an objective, and it works out for itself the sequence of steps needed to achieve it, whether it’s making financial transactions, rebalancing portfolios, or doing cyber defenses.

So, as we’re moving towards a world in which AI agents are acting autonomously in the financial system, we have to ask ourselves, “Is our regulatory framework set up for it?” The nature of the challenge has changed, right? We’re not regulating institutions where humans make decisions. We’re supervising systems that are operating at speed and at scale, and levels of complexity that it’s hard for a human to oversee fully in real time.

For us, we’re asking ourselves the question: if the financial system is operating autonomously at scale, at speed, are we prepared for that? Is our framework doing the best possible job it can to make sure that what’s happening in the system is safe?

A Partnership Humans-in-the-Loop and Machines

Goldstein: One of the things that people said can protect us potentially is to keep a human in the loop, so that it’s not going to be fully agentic, and fully autonomous. There will always be some human in the loop that knows what’s going on and can inspect and monitor and potentially prevent things from going in a different direction. But I think recently — and you spoke about it as well — it is becoming more and more difficult to keep a human in the loop. Do you think we will not be able to do that? If we can’t, then how are we going to address that?

Breeden: It’s really easy to present this as a choice between humans in control and machines in control. The reality is likely to be a much more complex partnership between the two. It’s important to remember this isn’t a one-zero thing. But if what we’re talking about is systems that are making thousands of decisions, or millions of decisions of machine speed, it is difficult to see how a human can meaningfully review every individual action.

Now, in many ways, we face similar challenges right now. Humans supervisors don’t review every transaction processed by a bank, right? They oversee the systems, the controls, and the governance that sits around those transactions. So, it’s on us to think about what is the analogous approach in the context of agentic AI? Who is responsible for the objectives of those agents being set? Who is responsible for monitoring performance? Who is responsible for testing the system under stress? And who intervenes when something goes wrong?

So, firms need to fully understand what systems are designed to do, what assumptions are relied upon, and where the system might fail. And that needs rigorous testing. It’s about knowing what inputs are going into the machines, what outcomes are happening, and being able to intervene if something looks like it’s going wrong, or if we get unintended behaviors. We need ongoing monitoring. Systems often behave differently in the real world than they do in controlled environments. That might be especially true for AI systems interacting with one another. So, how are firms going to observe and understand the behavior? Maybe AI ends up being a part of supervising AI and ensuring it’s working well.

It’s important also to remember about resilience. Let’s think about what happens when a failure occurs, and design frameworks capable of managing them safely. I guess we don’t need to approve every individual decision that an agentic AI [agent] might be making. But we do need to create a structure in which humans are accountable for actions. That accountability is meaningful because they understand the risks that are being generated and there’s an ability to intervene. We need to test. We need to monitor. And we need to set guardrails around the outcomes of AI systems. Let’s not just assume everything is going to be okay.

Maybe it’s worth just pausing on one thing that applied as much to normal AI as it does to agentic AI. But there is the issue of explainability, right? Financial regulation has typically been built about models being explainable. In the complex models we’re talking about here, that’s just not possible. So, we can’t be focused on the model. Like I say, we have to be thinking about the inputs and the outputs and thinking about what guardrails do we need to put around those.

How Regulators Are Keeping Up

Goldstein: How far along are regulators in understanding these issues and introducing those guardrails into the financial system? I mean, ’one thing about AI is that it’s fairly recent, at least in the current versions that we are seeing. It’s also moving very fast. Can regulators really keep track and be ahead of the curve? How far along are you?

Breeden: I think we are in the foothills of properly understanding what we need to do here. And I might mention the surprise that we had in the spring with the release of the new, very capable cyber models like Mythos and GPT-5.5-Cyber. What we are doing is working with industry. We have an AI consortium, for example, in the U.K. We and our fellow markets regulators at the FCA (Financial Conduct Authority) are involved in it. We’re talking to financial firms, talking to the hyperscalers, talking to the model developers, talking to academics like yourselves, to understand, try and get our arms around the uses of AI and what that means for regulators and firms in adopting them.

It’s fair to say that firms have been pretty cautious so far in their adoption. I don’t feel like we’re very far behind just yet. But what I am conscious of and what I think we really saw in the spring with the new models that were very capable of identifying cyber vulnerabilities is that these things move quickly and you need to be nimble.

One thing we talk about a lot at the Bank of England is being ready for tech surprise. I think we had a surprise in the spring, hearing of  OpenAI model escape from its sandbox environment’. So, I don’t think we’re behind, but I’m not sure that we’ve got terribly much time ahead of us before we need to be able to do a better job of thinking about where we want this tech to be used and what are the guidelines, guardrails we want to put around it.

I wonder if I might make one point. I didn’t say this at the start, and I probably should have done. Look, AI is a good thing, right? It should drive productivity improvements. It should enable me to have more capacity to do my job well. So, it’s on us as regulators not to say no, but to support its responsible adoption. That’s obviously the case in terms of cyber and fraud. If the bad guys have got these tools, you absolutely want regulated financial institutions to be able to use them. But I do think we need to think about where this might bring financial stability risks and how do we make sure that models are being used responsibly.

How Manipulation and Collusion Work

Goldstein: I think that’s an important point to keep in mind, that at the end of the day AI is a good thing and it can help many functions in the economy, many functions in the financial system. But of course we also need to be vigilant and make sure that it doesn’t get out of hand and create financial fragility. So, one of the things that I’ve been interested in in my own research is the interaction among different AI agents. We’ve been running experiments showing how when you have different autonomous AI agents, they learn to collude, potentially preventing the market from being competitive or even starting to create their own price path of pump-and-dump manipulation where the price goes up, then it goes down, and they manage to make profits on the way.

All of this sounds, of course, like science fiction. But in the lab, when you let these agents start trading against each other, they do find ways to coordinate and create this kind of collusion and manipulation. Now, when you think about how collusion and manipulation were treated by regulators in the past, I think the dominant thinking was that with humans, you’re going to look for signs of explicit communication or explicit intent before you’re going to pursue any anti-collusion or anti-manipulation enforcement. But with those machines, that’s probably not the way it works. So, do you think we need to worry about that and maybe think about collusion and manipulation differently going forward when we are dealing with those autonomous AI agents?

Breeden: It’s a great point. Your research has made us ask some really quite deep questions about how markets work. Exactly as you say, the research that you have done suggests that the manipulation or collusion that you normally think about people coordinating to deliver — conversations, agreements, intent — that is just not appropriate or not likely to capture everything that might happen in the context of AI. We get similar outcomes without explicit coordination.

All of the AI systems’ are trying to maximize returns, and they’re learning from market outcomes given enough time. They learn that certain forms of behavior are collectively profitable. Nobody told them to collude. They didn’t intend to collude. However, the market result may nevertheless look remarkably like collusion.

Now, I don’t want to make this sound over-dramatic. It’s early days and all that, but it does raise some pretty important questions as to whether the elements of our existing legal and regulatory frameworks that place a focus on intentional behavior are appropriately calibrated. Because what matters from a market stability perspective is not your intent. It’s what the outcome was, regardless of whether that was something you set out to achieve. So, I think we need to pay greater attention to behaviors of this type, behaviors that arise from interactions between agents, and not just the actions of any single participant.

The AI consortium that I mentioned earlier has this lovely phrase. We need to shift from thinking about whether any individual user of a model is managing it well, to thinking about: how can a market observe and control the collection of behaviors in a market? So, we need to flip our frame and the sorts of experiments that you’ve been doing and the sorts of experiments that we’re trying to do now. We’re thinking not just about how these behaviors arrive, but in what kind of market structures are they more likely. How might one be able to design ways of intervening to minimize the impact of such markets?

To the point that you raised with me earlier, I don’t think we have to do this by tomorrow. But we need to get our skates on if we are to understand these risks properly, to make sure that by the time this sort of agentic trading is common in financial markets, we’re ready with a financial stability response.

Keeping the Bad Guys in Check

Goldstein: You mentioned something like a kill switch that we can put on these AI agents such that if something goes out of whack, we can just stop it. How do you envision that working? Is that going to be a requirement from each particular firm or is that going to be at the level of the regulator to put a stop to all trading? How can we implement that?

Breeden: We’ve always built in mechanisms to help limit the consequences of stress in financial markets. We’ve got circuit breakers, trading holds, resolution frameworks, the failing institutions, and operational resilience arrangements. All of those are examples of trying to contain the impact of the stress once it has arisen. It’s on us to think through whether a deny becomes increasingly autonomous.

Suppose multiple agents are responding to the same signal in similar ways. Supposing we get powerful feedback loops. To be clear, we have those in markets now, but they might just happen at greater scale or speed in an agentic world. I think what we need to think about is how can we use AI to identify where those sorts of issues are arising and to think through what might AI be able to do to help us better manage those?

I talked earlier about how in a cyber or a fraud context, regulated financial institutions use AI to ensure that bad guys aren’t able to do bad things. In the same way, maybe in financial markets, I need agent Sarah Breeden on the pitch thinking about how we can use AI to stop financial market feedback loops happening in practice. Now, I haven’t got an answer to what that kill switch or circuit breaker might look like right now. That’s something that we’re hoping to test in the experiments that we’re doing. But the idea of having governance mechanisms, arrangements to pause things when financial markets are stressed, strikes me as a very sensible thing for us to explore, even if I don’t yet quite know exactly how it would operate in practice.

Goldstein: You mentioned using AI to regulate AI. I guess that brings us back to something we talked about in the beginning, keeping a human in the loop. There is a bit of a worry here, once you start using AI to regulate AI and you go further down the spiral of just AI all around and minimizing the role of humans. Do you worry about that?

Breeden: I always think that you need to have domain expertise, as well as model capability. We as central banks are information-intensive organizations. We spend a huge amount of time trying to understand what’s happening in the economy, in financial markets, what the risk landscape looks like. I’m pretty sure that AI is going to be able to help me do my job better. I’m not sure it’s going to be able to do it all on its own. It takes a huge chunk of judgment’ to process and properly understand what’s going on in the economy and the financial system. So, I do think it’s important that we use AI. We have the referee on the pitch. But at the same time, at least for a while, we’ve still got jobs to do.

Goldstein: Another aspect that we all think about and talk about with regard to AI and financial stability is the financing of AI and the possibility that right now we have a bubble in the financial market because of all the investment that went into AI. Do you have a view about that?

Breeden: This is one of the most important issues that we’ve been looking at recently — the degree of investment that is required to build the AI infrastructures that we need so that it can be used as the next general-purpose technology to drive productivity and growth. The pace of investment required is historically unprecedented. The scale might be below [that of] the railways, but the pace at which it is being pursued is historically unprecedented.

What it means, of course, is that a lot of institutions are raising equity capital and debt capital in order to be able to fund that investment. The numbers, as you well know, are very large, in trillions of dollars. We are seeing as well new and innovative financing structures coming through, right? We’ve got chip financing as well as data center financing.

What we have been trying to emphasize is people should think about what is the risk that they are potentially exposed to as they undertake these investments? Historically, the hyperscalers have been cash-flow-positive. But there’s some pretty clear graphs that show all of those cash flows are being pointed to infrastructure investment. There’s a chunk of debt being pointed to infrastructure investment as well. And the path to monetization, and to repaying that debt, is as yet unclear.

So look, is it a bubble? I don’t know. But what I am sure of is that there’s a chunk of investment that is being made. The payoff to that may take longer. It may accrue to different players. And so as this investment takes place, have an eye to downside risk as well as upside risk.

What Makes for Successful AI Regulation?

Goldstein: We’re coming to the end of the episode here. Let me take you maybe 10 years forward, trying to predict what the market is going to look like in 10 years. What would you think is a successful AI regulation at this point? First, do you think the market is going to look completely different in 10 years? And second, how would we then judge whether we’ve done a good job regulating the market and maintaining financial stability with all this innovation that is going on?

Breeden: I mentioned earlier the idea of tech surprise and how much I have been surprised by the rate at which capability in AI has progressed even over the last 12-18 months. So, I feel very nervous about projecting 10 years ahead.

I feel more comfortable in trying to articulate what success looks like. For me, success is not measured by how much AI we’ve regulated, but it’s measured by whether we’ve managed to adopt it across society in a way that captures the benefits — as we were talking about just before — productivity and growth, while maintaining trust and stability. So if we, in 10 years’ time, have seen AI contribute to stronger productivity growth, better financial services, greater economic opportunity, and we haven’t crushed the system on the way, that would be my definition of success.

That’s really easy to say, and I think we’ll have to be nimble and agile in how we get there. We’ve been really focused to date on cyber. We’ve also been focused on financial markets as AI is adopted there. But I’m sure there’s going to be many other use cases as well. I’m looking forward to when I can tell an AI agent to go and book my summer holiday for me. That’ll need some agentic payments to support that. How do we do that without there being fraud along the way? Supposing something happens and the wrong thing has been purchased. Who’s liable for that? Is it me? Is there a rogue agent involved?

It’s on us to think through, what are the use cases? What are the potential risks that each of those use cases brings and make sure that we get ahead of them? That is easy to talk about in high-level terms. We’ve got an awful lot of work ahead of us. In that respect, the community of Wharton with its research, with its students, with people leaving Wharton and coming into the world of central banking and regulation, I’m sure there’s a lot of interesting opportunities for us to work on together.