The Regulatory Challenges of AI in Finance

Itay Goldstein: The topic that we are going to dive into today is AI in finance. AI is promising to change the way that the financial industry works in many different ways. There are many opportunities, many risks, and regulatory concerns. We will talk about all this with Tobias Adrian, financial counselor and director of the Monetary and Capital Markets Department at the International Monetary Fund.

I look forward to hearing your insights, Tobias. Your division at the IMF has written a report about the use of AI in finance. Can you tell us a little more about what is at stake? What are the opportunities? What are the risks? What issues do you see?

Tobias Adrian: Our Global Financial Stability Report has a special focus on artificial intelligence or AI in finance. There are different modes of AI. Some AI techniques have been deployed in finance for many decades, and have reshaped the way that finance worked already. Here, I’m particularly thinking about algorithmic trading, which is primarily based on machine learning techniques.

When we’re looking at liquid capital markets today, we already see a tremendous impact of artificial intelligence on how those markets are working. [That relates to] trading activity in equity markets in the U.S. and other advanced economies, or treasury markets. The most liquid securities in those markets are already largely traded in an algorithmic fashion based on artificial intelligence; [they trade] at very high frequencies.

What is new in recent years is, of course, the arrival of generative artificial intelligence. That is quite different from the machine learning that has been previously deployed. So, the focus of our report is on understanding to what extent generative AI, including the large language models, could impact the financial industry. Our particular focus is on trading, or on capital markets activity. There are also other areas of finance that are already being impacted.

Challenges in Using Gen AI in the Financial Sector

Goldstein: One of the things that we think about when we think about AI is the agency – that now AI will be a decision-maker, and will start doing things on its own. I wonder if some of this is also reflected in the way you’re thinking about it and in your report.

Adrian: Yes, absolutely. It is certainly true that in algorithmic trading, you do see a lot of automated decision-making at very high frequencies. There could be thousands of millions of decisions that are that are fully automated. [Understanding] to what extent that will be changed or influenced by generative AI is still work-in-progress.

Let me dig a little bit deeper into what generative AI is, and how it is being used already and how it may be used going forward. Generative AI uses very large models that are calibrated to very large data in order to elicit answers that are somewhat similar to the way that human intelligence works. That is why the term “artificial intelligence” is used. In the large language models, the interaction is via natural language. The logic is quite different from [that with] traditional programming, where you write code, and then the computer generates output. [With Gen AI], you have a model that has been calibrated, and then you use information from that model to generate output.

The challenge here is two-fold. Number one, we don’t fully understand how these models really work. They are extremely powerful, but they’re a little bit of a black box at the moment. So, a lot of research today is discovering what kind of knowledge is embedded in the generative AI models. We are still learning about the technology that is very powerful, but that’s still very recent. As a result, the financial sector is still in the process of understanding how [it could use] those models across [its activities], and for trading activity in particular.

One policy concern is the lack of explainability and the unpredictability to some degree. Highly complex information is used in a way that may be working similar to the human brain. But we don’t fully understand how it performs in terms of trading, for example, when new situations are feeding into financial market activity. There’s a certain amount of unpredictability and lack of explainability. [Understanding] the impact of that is a work-in-progress.

Big Opportunities for Efficiencies, and New Risks

Our sense is that there are tremendous opportunities. We already see some areas where we think the new technology has a first-order impact. For example, when complex reports are being published — say, an SEC filing of a publicly-traded company. These can be very thick, 100-page-long reports. A generative AI model can very quickly extract information from [such a] report.

Another example is statements by monetary policy-makers like the Federal Open Market Committee of the Federal Reserve. When they publish their policy decision that comes alongside their statement, [that] statement now can be analyzed very quickly by generative AI – and so, the informational efficiency of markets can be increased, in principle. We see some evidence that that may be the case. It could be very good for market efficiency.

But then there are other areas where we don’t know exactly how much efficiency would be improved and what the potential pitfalls could be. So, there are efficiency gains, but there are also new risks that we have to take into account.

Goldstein: How widely used is it, at this point?

Adrian: Our understanding is that every financial firm is exploring it, and there are many startups that are being built around the technology. In some areas, we do see traction. For example, we hear [that Gen AI is being used] to do compliance checks or credit risk analysis – where complex data, some of it from the public domain like the internet, and some from reports such as credit reports – have to be combined, and activity has to be detected.

For example, for payments activity, financial institutions put a lot of effort into understanding whether a certain payment may be legitimate or illegitimate. We understand that in those areas, newer AI models can be very effective and the detection rate for fraudulent activity can increase tremendously. Our understanding is that there is already deployment [in those areas] that is very beneficial.

We think trading as a capital market activity is more of an emerging area. It is not clear to us that it’s very widely used at the moment, say, for high-frequency trading. What we have heard is that to venture into new potential trading activities, generative AI can be very good because it can generate, say, new trading strategies very quickly. It understands the potential trading strategies, and it can give you trading ideas. Then you can work with those ideas. You can literally generate code, but, we don’t fully know to what extent that is being used,. What we’re hearing from market intelligence is that it’s helpful, but how it’s being used is still somewhat a work-in-progress.

When you think about a risk manager in a financial firm, you would certainly have some concerns about accountability and decision-making accuracy, right? In traditional automated trading, you can go through the code, and you understand how the code works. In generative AI, you can’t trace back the code to decisions [that] are being taken. [However], if the model generates code, then that you can check. So, it is a bit of a complex undertaking. The lack of explainability, the magnitude of unpredictability and this lack of accountability in decision-making are certainly issues.

Let me flag two more things. There is some literature documenting some bias that’s not necessarily related to financial activity. But we don’t know whether the time series properties of what is coming out of the model has good econometric properties from a trading perspective. So, when you think about trading strategies or capital market-making, you want to make sure that your strategy is resilient relative to extreme events in the future and to structural changes. We don’t fully understand to what extent the [quality] of the data to which the model is calibrated is a good predictor for future performance.

Another issue for financial firms that we heard [of] is that the technology firms ultimately develop and operate the large models; the financial firms are users of the models. But it’s not [as if] they can possibly use variants of the model or calibrate the model to specific data. There’s a certain amount of reliance on outside parties developing models that are not specifically generated for the financial industry. Third party service providers that may not have the same challenges in mind when generating models which a financial firm or trading firm would have. That generates a whole new set of issues in terms of third party dependency, which the financial industry is trying to tackle.

A Black Box, with Risks of the Unknown

Goldstein: So, you’re saying there is potential here for better information processing that will contribute to market efficiency, and this is something we all want. On the negative side, or on the risk side, I hear you saying mostly that it’s hard to explain what’s going on. It’s a black box. It is delegated to other parties, and we don’t know exactly what they’re going to do with it. So it’s mostly the risk of the unknown – things that we can’t fully think about right now might happen when we deploy these new machines, and potentially [introduce] new agents in the trading process.

Adrian: Exactly. It is a bit of a black box. We may start to use it, but we may not fully understand where results are coming from, and how they’re going to behave in the future. So, the risk management around using generative AI is certainly a key priority.

I would add another aspect, which is the malicious use of AI. It is not easily verifiable for a user of AI [as to] what kind of information has been fed into the model and what is the relative importance of good data and malicious data. It is – in principle – possible that the models were calibrated to malicious data where some actors may want to manipulate the outcome of the model. It’s difficult to assess the extent [to which] your model is robust to such forms of manipulation. That is another area of concern, and I think the financial industry is working on that. But it’s certainly not an easy challenge to overcome due to the complexity and the magnitude of the models.

Preparing for Cyber Vulnerability

Another issue is cyber vulnerability. We all know that operational risk through cyber attacks is a first-order concern for the financial sector. In fact, regulators are asking financial services firms to hold capital and liquidity against cyber incidents. IT departments in financial sector firms put a huge amount of resources into protecting firms against cyber attacks.

Now, I already gave an example where generative AI is used to detect, say, malicious or criminal activity in, say, financial flows. The challenge is that generative AI can also be used for criminal purposes. You may use it for better defense, but it can also be used for more malicious and more effective attacks.

We worry that cyber incidents may become more effective. Let me give you an example of that. You and I and the listeners [of this talk] are probably receiving all kinds of emails that are trying to phish our confidential information, and are trying to gain access to our computer systems. In the past, oftentimes you could fairly easily detect these phishing attacks, say through emails or social media. There are oftentimes spelling mistakes, or things don’t look quite right. But generative AI could be very good at actually fixing those issues, and so make those phishing emails or text messages seem more real. In cyber attacks, it’s typically the interface between technology and the human element that is the vulnerability; i.e., cyber criminals tend to exploit human weaknesses in order to gain access to the technology. That could be much more effective [with Gen AI).

On the one hand, as a risk manager or an IT department [executive] in a financial firm, you have better tools to protect yourself, but probably the attacks are also more effective. So that’s another first-order concern that we hear from the industry.

What Regulators Could Do

Goldstein: Yes. Regulators are in a pretty tough position right now. On the one hand, opportunity is coming from AI that we all recognize. On the other hand, the risks are pretty big. But they’re not easy to define. They’re not easy to quantify. A lot of it is just that – “It’s hard to explain. It’s a black box.” We don’t know exactly what to expect. What do you think regulators should do in light of all this?

Adrian: The first-order work that regulators have to do is just to understand and do outreach to firms – meet with participants to understand what is happening. Our chapter is trying to do some of that. We did a lot of market outreach just to understand where financial sector firms are using generative AI, what they are exploring, and what they see as risks and challenges.

In many ways we’re at the junction time where the perspective of the policy maker and the perspective of the financial firms is not that different. Traditionally, in regulation, you often have a big conflict in between the policy objective and the individual firm perspective, because the policy objectives are not necessarily closely aligned with the individual firm perspective. But in this area, there’s actually a fairly close alignment, so that conversations are fairly straightforward.

Having said that, a lot of it is work in progress. We are getting updates [on technology] that are first-order improvements. The technology is evolving very quickly, and the applications are evolving very quickly. So, it is still a work-in-progress. For example, in the U.S., the Securities and Exchange Commission proposed a rule in July 2023 on the use of predictive analytics by broker-dealers and investment advisors which would require regulated entities to take steps to address conflicts of interest in the use of predictive data. This is still in the rule-making progress. But that would be one example of a regulatory step by a securities regulator.

Coordination Across Regulatory Jurisdictions

We’re working closely with the Financial Stability Board and the International Organization of Securities Commissions (IOSCO), as well as the Bank for International Settlements, in order to continue to assess the developments.

Goldstein: Is there room for more coordination across regulators in different countries, given the nature of technology [where] cross border spillovers are very significant? Is that something that you’re trying to do here?

Adrian: Yes. For financial regulators, there’s a very well-developed mode of cooperation across countries and across different types of regulatory bodies. For example, the Basel Committee, which works on regulating banks; and IOSCO for securities market regulators, are both members of the Financial Stability Board. The Financial Stability Board brings together all the different regulatory bodies — including insurance regulators, for example— in order to have an umbrella of policy framework. That is very well developed.

In this area of generative AI and large language models, there are other regulatory issues that don’t fall within the [current scope of] financial sector powers and mandates. [Consider], for example, the use of data. As a financial sector firm, you may use a model that has been calibrated to data where there could be, potentially, governance problems around the data in certain jurisdictions.

In some jurisdictions, for example, individuals have a right to withdraw their data from being used in the public domain. Now, what does that mean for the calibration of the model, if there’s an individual whose data was being used for the calibration that is then withdrawing that data? These [issues] fall to some degree outside of the policy framework and the powers of the financial regulators, but they are highly relevant for the financial regulatory sphere. It’s the interdependence between data models and the technological infrastructure that is at issue here. Data and intellectual property are some of the issues.

Could AI Trigger the Next Financial Crisis?

Goldstein: Should we worry that the next financial crisis will be a result of AI, or am I going too far with that?

Adrian: It’s a little bit too early to say. I do think we have made first-order progress in financial regulation, particularly in the regulation of banks and insurance companies, but also important securities market participants such as broker-dealers and funds – both open-end funds and closed-end funds. So, I do think we’re in a much better place than we were 10 or 15 years ago. For example, with respect to operational risk, there has been a lot of progress. That is true for banks, for funds, but also for infrastructures.

The challenge is really around those issues that are not readily captured in the existing regulatory approaches. Regulators themselves are working on upskilling their own technology in order to meet these technological challenges. In our regulatory upskilling [meets], we hear quite a bit about – for example – sup tech and reg tech, where supervisors and regulators are themselves using technologies to detect risks better. So, on the one hand, there could be new risks. On the other hand, policy makers could be more effective.

Let me be a little bit more specific about some regulatory initiatives that we have seen already. For example, in Hong Kong, the Hong Kong Monetary Authority has put out an AI governance framework which is about the risk management and the reliability of AI systems used in the banking sector. That’s one concrete example. In the U.K., the Financial Conduct Authority has implemented requirements for the explainability of AI-driven decision-making. For example, in credit scoring, it’s fine to use those models. But you need to have a rationale or explainability around that. That, in turn, helps mitigate the risks associated with the opaqueness of the models.

In Singapore, [the monetary authority has published] “the fairness, ethics, accountability and transparency” principles for AI and finance. That helps financial institutions in Singapore in the responsible usage of AI systems; it is particularly focused also on bias and discrimination. In the European Union, the General Data Protection Regulation (GDPR) has been adapted to the use of AI practices in financial services, particularly for focusing on data protection and privacy practices. Those are some of the examples I alluded to earlier, [which are] aimed at reducing the risk of data breaches and the misuse of personal information.

Goldstein: All of that is very fascinating. There is certainly a lot more to hear about and learn about from this AI revolution in financial markets and all the challenges it poses when thinking about risks and regulation. Thank you very much, Tobias, for all the great work that you’re doing at the IMF and for joining us for this podcast.

Adrian: Thanks so much, Itay. Very good to see you, and warm regards from Washington to Pennsylvania.