ISITC TALKTIME:AI Rogue Agents and Explainability

Home » Forums » AI » ISITC TALKTIME:AI Rogue Agents and Explainability

ISITC TALKTIME:AI Rogue Agents and Explainability

Posted on

AI Rogue Agents & Explainability

AI Rogue Agents and Explainability: What Financial Services Firms Must Know

Listen to Gary Wright of ISITC Europe and Rob Morel of AIRTA Systems explain how to detect rogue AI agents and meet FCA expectations under SM&CR.

AI agents are already working inside financial services firms, whether compliance teams know it or not. In part two of the ISITC Europe AI podcast series, Gary Wright, Director of ISITC Europe, and Rob Morel of AIRTA Systems unpack AI, Rogue Agents and Explainability, and what it means for boards, senior managers and compliance officers in the UK.

What you will learn in this episode

  • What a rogue AI agent is and how a poisoned knowledge base or a mis-programmed KYC workflow can cause unintended actions.
  • Why explainability is the main control for AI governance, and what regulators expect you to be able to show.
  • How the FCA’s Senior Managers and Certification Regime (SM&CR) applies to AI: a lack of explainability does not excuse non-compliance.
  • Why firms need an inventory of every AI agent across HR, front office and back office.
  • How third-party suppliers can create an “explainability gap”, and the due diligence questions to update now.
  • Why boards need someone with real technical awareness of AI, not just a tick-box audit.

Why AI agent explainability matters for compliance

Rob Morel explains that explainability is the answer to the rogue agent problem. If an agent is fed bad data, it can give a customer wrong information and cause financial loss. In a Know Your Customer (KYC) pipeline, an agent that goes off-script at any step could delay a firm’s business by several days. Firms must be able to explain what data the agent received, why it made its decision, how it went rogue, what was done to contain it and how recurrence will be prevented.

As Gary Wrights stresses, this all sits within existing UK regulation. Regulators will not tell you how to architect your systems. It is up to each firm to decide how to manage its AI risk.

Key takeaways for boards and senior managers

  • Know your inventory: identify every AI agent, where it operates and what it does.
  • Put controls in place: monitor each step an agent takes so you can spot and fix rogue behaviour.
  • Assign accountability: a board member must own AI risk, or a delegate must report to the board, and both need technical understanding.
  • Challenge your suppliers: “if they’re using AI, you’re using AI, and you’re the one that’s regulated.” Review contracts and update due diligence questionnaires.
  • Be ready for the regulator: have evidence and explanations available on demand.

The top 10 AI risks for SM&CR

The conversation draws on the top 10 AI risks for senior management and certification regime in financial services. Each risk will be covered in the series.

  1. Unpredictable consumer outcomes
  2. Rogue agents and explainability (this episode)
  3. Excessive agency and improper delegation
  4. System integrity and adversarial manipulation
  5. Algorithmic bias and protected characteristics
  6. Third-party vendor risk
  7. Technical competence
  8. Shadow IT and stale inventory
  9. Post-market drift and degradation
  10. Evidence sprawl

 Viewing time: 18 mins


 

Bob Morel

Bob Morel is the CEO of AIRTA Systems, an AI Risk Management platform designed to facilitate market access for AI-driven applications, and Co-Founder of Genbounty.

With a robust background in thought leadership, Bob previously served as the Head of Application Security at Centrica and the Application Security Lead for a cryptocurrency exchange. He is an active contributor to the cybersecurity community as an author for Infosec, creating learning paths on topics such as Breaking LLMs and AI-driven Applications, and the ChatGPT for offensive security. His expertise is supported by a B.Sc. in Computer Science, and industry certifications including the (ISC)² CISSP, CompTIA Security+, and SecAI+.