As artificial intelligence (AI) continues its expansion into the finance sector, a new report published on the International Monetary Fund (IMF) Publications web portal cautions about risks associated with the latest AI capabilities in the financial world.

The Rise of Thinking Machines: Report Warns of Risks From Generative AI in Finance

The report, authored by Ghiath Shabsigh and El Bachir Boukherouaa, focuses on generative artificial intelligence, often referred to as generative AI, which is a form of AI capable of creating original content such as text, images, and video. Notable systems like Openai’s Chatgpt, Anthropic’s Claude, and Midjourney have gained popularity for harnessing this technology in recent months.

According to the study, while generative AI holds promise for applications like automating processes and enhancing risk management, it also introduces new risks related to data privacy, bias, performance robustness, and cybersecurity.

Researchers point out that generative AI systems ingest massive amounts of online data, raising concerns about privacy breaches or the replication of biases from that data. Furthermore, the systems’ ability to generate entirely new content means they could produce plausible yet inaccurate information. The report states:

Publicly available [generative AI] systems pose significant privacy challenges for financial institutions wishing to incorporate their capabilities into their operations. This automation thus raises the possibility that sensitive financial data and personal information provided by financial institutions’ staff in their engagement with the [generative AI] could leak out.

The report warns that these issues could undermine public trust if generative AI is irresponsibly deployed in finance. For instance, AI-generated customer risk profiles could be both inaccurate and discriminatory. The researchers insist that popularized chatbots might provide flawed financial advice.

In August 2023, Gary Gensler, the current chairman of the U.S. Securities and Exchange Commission (SEC), issued a warning that AI technology will be at the core of future financial crises. The securities regulator has proposed addressing some AI model risks. Meanwhile, the SEC has recently approved a new AI-powered order type for Nasdaq, marking the first such approval for a traditional finance (tradfi) exchange.

The researchers emphasize that cybersecurity is another concern, with generative AI potentially being exploited to create more sophisticated phishing attempts. The report suggests that the full extent of vulnerabilities is not yet understood and may evolve over time. Additionally, the study further discusses alleged vulnerabilities, including generative AI “jailbreaking.”

“Current [generative AI] models are increasingly subject to successful ‘jailbreaking’ attacks,” the study’s authors noted. “These attacks rely on developing sets of carefully designed prompts (word sequences or sentences) to bypass [generative AI’s] rules and filters or even insert malicious data or instructions (the latter is sometimes referred to as ‘prompt injection attack’). These attacks could corrupt [generative AI] operations or siphon out sensitive data.”

To mitigate these risks, the report recommends close human monitoring of AI systems, improving the explainability of AI decision-making, strengthening regulatory capacity, and promoting international collaboration on AI governance. While acknowledging the transformative potential of generative AI, the authors emphasize the need for a cautious approach, particularly in a sensitive sector like finance.

What do you think about the fintech report published on the IMF Publications web portal? Share your thoughts and opinions about this subject in the comments section below.