Several weeks ago, I had the privilege of attending an excellent training session about how business analysts can benefit from using Agentic AI.

There were several aha moments, including a conversation where we discussed the responsibility of an outcome from the agent which was building a prototype for the students. This prompted me to reflect on a broader question: as AI systems become increasingly autonomous, how do organisations ensure they remain accountable for the outcomes.

The Human Factor in IT Governance

When I was a budding information technology student, the BOFH (from the website of The Register (UK)) was a fascinating character. There were many funny tales about the divide between knowledgeable system administrators and clueless users; and a common theme was how, as far as computer issues are concerned, the error is often located between the chair and the keyboard. Indeed, system users can find fascinating ways to use technology resources, which readers will find to be an excellent transition to the specification and design: surely, if business analysis, solution architects and system designers implement all possible guardrails, then material issues will not happen.

Even if this was true – what about basic tools available in any corporate organisations, such as an access to the Internet and Excel? Safeguarding in this case comes with governance and risk controls on acceptable behaviours. Those controls were intended not to stifle innovation, but to establish the defence of the perimeter for the organisation.

The parallel is easy to make with AI – conceptually a powerful tool, how can we make sure that it is wielded safely? Traditional governance and project management practices used throughout software delivery lifecycle are still useful, but not sufficient to address the potential risks coming from using AI:

• In more traditional models and systems, it is possible to understand and explain to an interested party the logic used to get a specific outcome: after all, this logic was coded following a specification, which would be agreed and validated with business stakeholders. In the case of AI systems where a prediction or classification is made, it becomes important to be able to explain how the outcome was reached. This is the concept of explainability. If an AI model recommends investigating a transaction as suspicious or declining a loan application, the organisation will need to understand which factors influenced its decision and be able to explain the rationale to regulators, auditors or customers.
  ‍
• Automation is not a new concept, as scripts would be configured to orchestrate file movements and execution of mission critical programs at specific times. When a new script is introduced, support engineers would validate the behaviour by combing through log files and confirming it behaves as intended, despite an automation bias. As AI offers to automate part or all of a process which was manual before, it is necessary to be vigilant in the same way. This is the role of human oversight, often described as human in the loop or in the lead; effectively displacing some of the responsibilities from performing a task to supervising it.
  ‍
• Despite best intentions, bias can be introduced at various stages of the AI lifecycle, from requirements gathering and data selection, through to deployment and monitoring. Organisations therefore need to assess the fairness of outcomes on an ongoing basis, ensuring that decisions do not inadvertently discriminate against protected groups. I have talked about human oversight role above; another dimension to consider from a continued assurance perspective is to frequently test the fairness of the model. An example of fairness is demographic parity, where a given outcome for the overall population has the same chance of happening when scaling down to a subpopulation.
  ‍
• Regardless of whether a decision is made by a human, an automated process or an AI model, accountability can never be delegated to the technology. Ownership needs to be clearly defined for the development and operation of AI systems. An organisation may automate a decision, but it cannot automate accountability for the decision.

The extent to which effective governance should be built has to correspond to the risk appetite of the organisation. Organisations which are using sophisticated decision-making or classification AI models, or using GenAI to derive insights from internal data will need solid guardrails and governance. On the other hand, if the organisation allows use only for brainstorming and productivity, governance needs will be lighter touch, just focusing on appropriate usage of tools and data classification.

Adapting Governance, Not Reinventing It

There was a saying years ago that “Machine Learning is usually coded in Python, AI in Powerpoint”. Since then, AI capabilities have developed at a furious speed; and using them in the corporate world can be a significant competitive advantage in many situations such as reacting quickly to events or having a more meaningful relationship with customers – that is, as long as its use is governed effectively and its adoption is anchored in a strategy which evaluates any other commitments and impacts on risks, such as conduct, privacy or climate risk.

Perhaps the biggest misconception about AI governance is that it is something entirely new. In reality, many of its foundations, accountability, controls, risk management and oversight have existed for decades. What has changed is the scale, speed and autonomy with which AI systems can operate. The challenge for organisations is not to reinvent governance, but to adapt it.