
The famous quote from the IBM manual back in 1979 says that “a computer can never be held accountable, therefore a computer must never make a management decision”.
In a recent note from the RBNZ, they highlighted expectations on other sources of AI risks, namely increased cyber risk, third-party and concentration risk. “The core message today is simple: advances in AI are changing the operating environment for financial institutions. Entities are responsible for ensuring that they have the systems and controls in place to manage the risks arising from AI”, RBNZ Assistant Governor Financial Stability Mr McGregor concluded.
From an enterprise perspective, cyber risk is largely about how the organisation’s perimeter is defended. The underlying risk is not new, but AI is amplifying it:
The response to these heightened risks is not new either. It requires enterprises to apply their existing defences rigorously, including:

Click to enlarge
Not all of the new risks are AI-augmented cyber-attacks. Some arise from the enterprise’s increasing reliance on third-party AI models and systems, which can change in ways the enterprise does not control. Changes to underlying models, model updates or retraining, advances in model capabilities, or changes to third-party configurations and settings may cause AI-enabled or AI-supported business applications to behave differently or produce inconsistent outputs over time. As a result, the testing and system controls put in place for an AI-enabled business application on ‘day one’ may no longer be relevant or robust enough on ‘day two’.
Change is not always a problem; it is to be expected. Comparing a basic interaction with generative AI today with one from 2023, for example, shows clear progress: responses are of better quality, with fewer discernible patterns of machine-generated text. However, when AI is used for decision-making, automated classification or core enterprise processes, it should be subject to the same fundamental change-management principles as traditional software. If a software component is changed or reconfigured, good practice requires appropriate testing to confirm the application continues to meet business requirements and remains fit for purpose.
The same principle should apply where an underlying AI model, service or configuration changes. Customers and regulators would expect the enterprise, not the third-party provider, to remain accountable for ensuring that its AI-enabled or AI-supported business applications continue to operate as intended, remain fit for purpose and do not expose the enterprise or its customers to undue risk over time.
Concentration risk adds a further challenge. Where several business applications rely on the same third-party model or provider, a single change, outage or withdrawal of service can affect them all at once. Knowing (and documenting) which applications depend on which models, and having a credible fallback where an application supports a critical process or service, is as important as testing each application individually.
Ultimately, resilience against third-party and concentration risk starts with sound change and project management disciplines. Consistent with our views in our previous article, an AI-enabled or AI-supported business application should be implemented to meet a defined business requirement, just as traditional software would be. That requirement should remain the reference point over time: as the underlying AI changes, the enterprise needs to be satisfied that the application continues to meet the requirement for which it was implemented. Because changes to underlying AI models and services can occur rapidly and outside the enterprise’s direct control, this may require more frequent testing and revalidation than traditional software. Assurance obtained at implementation may not remain sufficient over the life of the application.
AI may be novel technology, but many of the disciplines needed to manage it are not. The real challenge for financial institutions is understanding where AI changes the assumptions on which their existing controls and resilience arrangements were built and ensuring that, when it does, accountability remains firmly within the enterprise.
If you would like to discuss how your governance, resilience and control frameworks can better support the management of AI-driven or AI-enhanced risks, please get in touch.
Sources

The famous quote from the IBM manual back in 1979 says that “a computer can never be held accountable, therefore a computer must never make a management decision”.
In a recent note from the RBNZ, they highlighted expectations on other sources of AI risks, namely increased cyber risk, third-party and concentration risk. “The core message today is simple: advances in AI are changing the operating environment for financial institutions. Entities are responsible for ensuring that they have the systems and controls in place to manage the risks arising from AI”, RBNZ Assistant Governor Financial Stability Mr McGregor concluded.
From an enterprise perspective, cyber risk is largely about how the organisation’s perimeter is defended. The underlying risk is not new, but AI is amplifying it:
The response to these heightened risks is not new either. It requires enterprises to apply their existing defences rigorously, including: