Keeping an eye on AI in financial services: a focus on AI, cyber, and operational resilience

There is a lot of news lately in the interconnected worlds of AI, cyber, and operational resilience. We had the “shock” of Mythos. Now we have Hugging Face. The EU has lifted its cyber risk status to “severe”. The Bank of England is laser focused on systemic issues at large. Firms are mid-preparation for a new reporting framework and the UK's first four of likely more Critical Third Parties have now been designated. 

What is the regulatory position?

The regulators recently issued a joint warning to the financial services ecosystem, and we also have The Mills Review (TMR) to consider for regulatory guidance and positioning in this space.

This blog provides a brief overview of TMR with specific focus on the AI, cyber, and operational resilience themes which, in summary, are these:

• Frontier AI (the first example of which was Mythos) marks a step change in AI capabilities, with significant implications for cybersecurity and operational resilience at both firm and ecosystem levels;

• AI materially increases the speed, scale and connectivity of cyber and operational resilience risks; 

• Accountability and responsibility for harms arising from disruptions to important business services continues to rest firmly with the regulated firm and its senior management; and

• The Consumer Duty, the Senior Managers Regime, the operational resilience requirements and the wider conduct framework, are regarded the flexible foundations of the regulatory framework and will continue to apply subject to the need for the regulators to be mindful of the need to adapt them in line with the demands of fast-evolving technologies. 

The narrative

A new version of systemic risk: TMR tells us that the biggest change is not that AI creates an entirely separate set of risks (though it does of course have unique risks), but that it makes the location and exploitation of existing flaws or weaknesses more likely, more dangerous, faster, and much harder to contain within a single firm. Accordingly, it presents a new dimension of systemic risk to the financial services ecosystem.

A warning to get the fundamentals fortified: TMR (and the Bank of England's speech from the week prior) position Mythos as a warning signal. Its key capability of being able to autonomously identify and exploit previously undiscovered vulnerabilities in software and systems is what makes it so dangerous. As a result, any firm that has under-invested in its fundamental cybersecurity protections is more likely than ever before to be exposed to the risk of attack. A strengthening by all firms of their cybersecurity fundamentals has therefore become an immediate priority. 

Risk indicators: TMR highlights the following key factors that contribute to the present elevated risk status:

• shared dependencies on a small number of providers have created concentrations, and associated shared and correlated, system-wide risks because multiple firms rely on the same AI model providers. This means that an outage, security breach or model degradation, could affect multiple firms simultaneously;

• correlated behaviours may amplify market risks. For example, where firms use similar models for credit underwriting, investment recommendations or pricing, synchronised decision-making could amplify market movements or lead to coordinated withdrawal of services; 

• no single organisation (firm or regulator) currently has enough visibility across the infrastructure of the ecosystem to see all concentrations and identify all vulnerabilities. The ability of firm level resilience controls weakens as concentration risks increase. Co-ordinated capability at system level may be challenged and needs to evolve to pool specialist resources, share data and analytics, and collective approaches; and

• senior managers of regulated firms remain responsible for harmful outcomes, including where third-party models, cloud services or external infrastructure are involved.

A regulatory framework under strain: TMR does not call for any major changes to the existing regulatory framework which is positioned as a strong foundation. It does, however, call for the framework to be adaptive to the strains and frictions that will be placed on it by autonomous AI, which is a form of technology that the framework was not built for. 

Challenging autonomy: One key source of strain and friction is the increasing autonomy of AI to which it is very challenging to apply 'traditional' methods of regulatory analysis (including for example, controls and oversight) and for which the evidence (for example, of consumer understanding and of good outcomes) and audit trails will be fundamentally different to those that the industry currently understands. 

Upgrades to governance needed: TMR makes it clear that governance can no longer be static and must shift to continuous mode based on run-time monitoring, supported with suitably skilled humans “in the loop”, constant model drift detection, auditability, clear permissions frameworks, clear and robust escalation protocols, and clear lines of accountability to an SMF (or SMFs) with ultimate responsibility for the deployment in its entirety.

New core governance capabilities

All firms must now address the question of whether their cyber and control environment is strong enough for a world in which attackers (which includes AI-enabled systems as well as "human" attackers) can identify and exploit weaknesses at machine speed. TMR highlights that those firms that have under-invested cyber fundamentals are the most exposed and must quickly move to strengthen the foundations that are essential to protect themselves against AI-enabled attacks. 

The critical issues of concentration and system dependence dictate that firms must deepen their understanding of their dependencies and reliance upon third-party model, cloud and infrastructure providers, and must have a clear understanding of how they would be affected by an outage or security breach, including where multiple other firms could be affected at the same time. In practice this means, thorough and up-to-date mapping of AI and cloud dependencies across important business services, robust testing of a full range of failure scenarios, and confidence that escalation processes can reach the right kind of human oversight real-time in a world of fast-moving AI-enabled risks.

Practical tips

Weakness is magnified: AI will magnify existing cyber and resilience weaknesses. Firms should assume more frequent, faster, and harder-to-detect attacks and failures across a more connected ecosystem where resilience risks are ecosystem-wide and not just firm specific.

Speed up defences: Cyber basics are an immediate issue. Any firm with weak security posture or under-invested cyber fundamentals will become progressively more exposed to AI-enabled attacks which can discover and exploit flaws at speed and scale. Known weaknesses must be triaged and fixed at speed. This is likely to require automation to match the speed of the attack technology.

The right SMFs: It is not enough for a firm to say that it has a human “in the loop”. Senior leadership teams must have SMFs with specifically assigned responsibility and accountability for AI and the firm must be able to demonstrate that its SMFs are suitably skilled, what their responsibilities are, what they are expected to do, what they understand, when they can intervene and how, how their challenges of their AI systems are effected and evidenced, and how relevant escalation systems work in practice. 

Machine speed governance: Governance needs to become continuous. Firms that have embedded AI systems must be able to demonstrate how outcomes are delivered, how controls operate and how accountability is exercised, including how system behaviours are monitored, how controls detect when AI moves outside expected bounds, and how governance processes respond to issues identified in operation. 

SMCR is key: Senior management responsibilities cannot be removed or placed upon third-party AI models, cloud services or other external infrastructure, or the suppliers of these. Accountability rests squarely on a firm's senior managers.

Mapping your systems and Plan B: All firms should have an up-to-date map of their AI, cloud and specialist vendor dependencies, tested fallback arrangements and realistic escalation routes to cover the likelihood that a critical provider degrades, fails, or suffers a security incident. Firms need to understand what the cost of a provider failing will look like, have a clear recovery plan, and have tested alternative providers or solutions in place. 

Trust: Firms must acknowledge that their customers are likely to be worried about where and how AI is being used in relation to their financial journeys, about the misuse of their personal and financial data, and about a possible lack of protection if something goes wrong. It will be important to consider how customer trust can be gained and retained. Operational resilience and cyber security need to be treated as central components of risk management, fully embedded security into systems, culture and decision making, and firms must recognise that secure systems will be a key enabler of trust and growth.

Ongoing scrutiny: Firms should expect more continuous and system-wide regulatory scrutiny, including regulator requests for timely, structured and auditable data, evidence of how outcomes are delivered, of how controls operate in practice, and of how accountability is exercised. This will include being able to evidence the monitoring of system behaviour, testing that extends beyond deployment into operational performance, controls that detect when systems move outside set boundaries, and governance processes that respond to issues in real-time. 

Our thought leadership:

You can subscribe to our monthly financial services regulation update by clicking here, clicking here for our AI blog, and here for our AI newsletter. You can meet our financial services experts here and our AI experts here.

the incident made it clear organisations needed to "step up" their own defences and "treat cyber resilience as a core operational priority"

https://www.bbc.co.uk/news/articles/c3ek3gvdnj3o