The Predictability Factor is a weekly deep dive at the intersection of AI, Security, Privacy and Tech, to help you Go From Chaos to Resilience in The World of AI.
This is part 2 of 2 on AI Governance and Security Maturity and Roadmap. Read part 1 here.
April 24, 2026, Jer Crane, the founder of PocketOS was using Cursor, an AI coding agent powered by Anthropic's Claude Opus 4.6. The task: a routine fix in the staging environment. The outcome: absolutely nothing routine about it.
PocketOS is a software company building management tools for car rental businesses. They use their cloud infrastructure from a platform provider called Railway. The AI agent (Cursor, running Claude Opus 4.6) hit a credential mismatch.
But it did not pause. It did not ask. It went looking for a way to resolve the problem itself, found an API token sitting in an unrelated file, and used it to call the "Volume Delete" command on Railway, which is PocketOS's cloud infrastructure provider.
9 seconds. That's all it took. The entire production database: gone. Every backup: gone too, because Railway stores volume-level backups inside the same volume.
Car rental customers arrived at counters unable to find their reservations. Payment records: wiped. Vehicle tracking: erased. PocketOS had to roll back to a three-month-old backup to stay operational.
When Crane pressed the agent for an explanation, it confessed in writing. It acknowledged it had violated PocketOS's own governance rules, rules that included the instruction: "NEVER FUCKING GUESS!" The agent admitted exactly what it had done. It had guessed that deleting a staging volume via API would only affect staging. It was wrong. It apologised.
The so-called rule was there. The AI had access to it. The AI violated it anyway. Then it wrote an apology.
This is not PocketOS’ problem. It is yours.
You built your control architecture for actors that obey. You are now deploying actors that decide and disobey at times.
The PocketOS agent did not bypass a firewall or exploit a misconfigured permission. It found an API token sitting in an unrelated file, inferred that token could resolve its problem, and used it. It did not cross a line in an access control policy. It made a judgement call. And it was catastrophically wrong.
This is the shift that should stop every CISO, every risk leader, and every board member in the room. Your controls still govern what your agentic AI can do. Your agentic AI now determines what it does do. And the gap between those two things is not a configuration error you can patch. It is a structural property of how probabilistic systems reason under uncertainty.
The principles you have relied on for years still apply. Least privilege. Defence in depth. Zero trust.
Applying them to an agentic AI system requires a completely different model of what "access" means. Because an agent with ambient access to a credential does not need explicit permission to use it. It infers permission from other API tokens. And the blast radius of that inference, as PocketOS discovered, can be the entire estate executed within 9 seconds.
In this edition of The Predictability Factor, we will go through Part 2 of Enterprise AI Governance and Security Roadmap with the remaining 5 steps of the 7-step Enterprise AI journey, along with covering the 5-pillars of governance and security maturity for agentic AI in your enterprise.
ICYMI, read Part 1 here (or click below).
The devil is in the detail. For Part 2, continue reading below.
Your 7-Step Roadmap to AI Governance and Security
As I said in Part 1, AI adoption is everywhere. Trust is not. The no. 1 principle for resilience in the unpredictable world of AI is this.
Assume Chaos. Build elements of resilience within and around it.
The 7-step roadmap isn’t accidental. It’s built on real-world experience and maps every step explicitly to both the NIST AI framework and the OECD AI system lifecycle as follows:
The 7-Step Enterprise AI Governance and Security Roadmap Mapped to OECD AI / NIST AI RMF
This roadmap hands you a complete step-by-step guide throughout your AI deployment and lifecycle journey, based on real-world experiences from me in advising enterprises, and what we are seeing happen to many organisations worldwide. That sequence is not arbitrary. It is built on the architecture of how AI actually gets deployed in enterprises (ref. OECD AI Lifecycle) and where it reliably fails (ref. NIST AI RMF and other real-world examples).
Let’s continue below with Steps 3 through 7 of the 7-step AI Governance and Security Roadmap.
P.S. The Ultimate AI Governance and Security Playbook is HERE! It is your step-by-step complete playbook on building your agentic AI governance and security maturity for resilient and trustworthy AI in your organisation. It’s the ultimate AI playbook that covers across 80+ pages:
The AI Governance Foundation
Key Pillars for a Strong AI Governance and Security Maturity
50+ Real-World Examples and Actionable Controls
Actionable insights and Practical Measures for Increased AI Maturity
Mapping to NIST AI RMF, OECD AI Lifecycle for Each Pillar
The Responsible AI Layer That Changes Everything
How to Bring It All Together for a Strong AI Deployment
Step 3: Five Key Pillars of AI Governance and Security Maturity
Your Step 2 Assessment results gave you an indication and map of your AI readiness. Step 3 gives you the architecture to build on it. In case you haven’t done step 2 yet, click below.
The 5 pillars that follow are your system for enterprise AI governance and security maturity, necessary for enterprise AI readiness, deployment and implementation.
Each pillar depends on the others. Data that is classified and access-controlled without a governance policy behind it is unprotected. A governance policy without cultural literacy is unread. Controls and engineering without data readiness are securing a pipeline that was never clean to begin with. Governance without controls and engineering is not fully implemented.
The NIST AI RMF GOVERN function defines governance as the set of conditions that make every other risk management function possible. ISO/IEC 42001:2023 calls this organisational context.
Before you can manage AI risk, you need a functioning system for understanding what risks you face, who is accountable for them, and what is in place to manage them. The five pillars are that system.
Pillar 3: Culture and Literacy
There are two types of companies right now.
The first is racing into AI with no guardrails, calling it innovation, deploying tools everywhere, hoping nothing explodes. Hope is not a strategy.
The second has banned ChatGPT, etc., declared AI too dangerous, and assumed that prohibition is the same thing as a risk mitigation.
Both are dead wrong. Both are exposed to insecurity, cyberattacks and vulnerabilities, for different reasons but equally so.
Culture is set from the top. And right now, most leadership teams are choosing between euphoria and prohibition, neither of which produces anything close to a managed situation.
Above we saw plenty of examples of organisations belonging to the first category. Samsung is a great example of the second category.
One of their own engineers leaked proprietary code to ChatGPT by simply pasting it into the non-corporate managed chatbot. As a result, Samsung banned ChatGPT and other AI chatbots. They didn’t have proper security controls or data exfiltrations (DLP) controls in place to prevent the leak from happening. Apparently, neither did they have adequate awareness, training or literacy around AI models, security and governance.
AI is not the problem. The lack of awareness, training and actual security controls is.
Literacy matters at every level. Your employees need to understand what AI is genuinely good at, and where it will fail you. This is a slide I usually show in my keynotes.
Image 3: LLM Capabilities: Hype vs. Reality
Your legal and compliance teams need to understand the regulatory picture clearly. The EU AI Act, in effect since February 2025, requiring major implementation by August 2026 and with full compliance requirements by August 2027, explicitly prohibits social scoring, un-targeted facial image scraping, and emotion recognition in workplace settings. Non-compliance carries fines up to €35 million or 7% of global revenue.
Start here:
Set the tone from leadership: measured, informed, and consistent. While you are pro AI, are you also pro-security, safety, ethics, responsible deployment and usage? That needs to be set from the top both in tone but also in action.
The two most dangerous cultural positions in AI right now are "deploy everything fast" and "ban it all until we figure it out." Both destroy the middle ground where responsible adoption actually lives. Neither all-in without any consequences nor locked down.
That tone needs to be followed by action. Train every employee on what AI can and cannot do reliably, including the probabilistic nature of LLMs and why prompting security is not the same as implementing it. LLMs do not operate on certainty. They generate outputs based on statistical probability. LLM hallucination rates range from under 50% to nearly 82%. 47% of enterprise AI users admitted to making at least one major business decision based on hallucinated output. You cannot govern what your employees do not fundamentally understand.
Make prompt injection literacy part of your security awareness programme. Prompt injection is a frontier security challenge that cannot be fully eliminated at the model layer, and likely never will. That means your employees are a critical layer of defence in what they even allow to be ingested by an AI agent, and currently most of them do not know the attack exists or how it works.
Address shadow AI directly by providing approved tools and clear acceptable use policies without resorting to banning anything.
Map your AI use cases against applicable regulations before deployment: EU AI Act, GDPR, sector-specific obligations.
Define explicitly what your AI agents are built to do and where human judgment is non-negotiable. Employees do not need a philosophy lecture on AI limitations. They need to build the habit of knowing, before acting, whether this is a decision the agent is reliable enough to make alone or not, which requires two key things: 1) Map all your (key) business decision workflows and 2) Build an AI decision making matrix, defining the criteria for when AI is allowed to augment or replace, and when humans must sign off.
Prompt injection is a frontier security challenge that cannot be fully eliminated at the model layer, and likely never will. Awareness, along with other methods like complete segregation of what actions AI is allowed to do at all, is key.
We will look more into that in Pillar 5: Controls and Engineering.
If your AI policy is "we banned ChatGPT," your employees have already found three other alternative routes. You just haven’t seen it yet.
ICYMI:
Did Mythos Break The Cybersecurity Industry?
😱 The AI model that found 10s of thousands of vulnerabilities is too dangerous to be released. Is this is the end of the cybersecurity industry? Read full story —>
Pillar 4: Governance, Policy and Risk Management
The same large language models being used to draft your emails, build presentations and summarise board reports for you are being deployed in critical infrastructure, healthcare diagnostics, financial decision systems, and defence operations. Not in a ten-year roadmap. Today.
With the same hallucination risk. The same explainability gap.
The strongest indicator of AI readiness and responsible innovation isn’t infrastructure, model or technical controls, it’s governance maturity.
According to McKinsey's 2026 AI Trust research, only about 30% of organisations have reached a governance maturity level or medium on strategy, governance, and agentic AI controls. That means 70% are deploying AI into consequential decisions with no systematic governance in place. What this study, however, still misses is that risk management isn’t just a pillar, it’s an underlying continuous governance process throughout the agentic AI lifecycle and increasing maturity across all pillars of responsible AI.
Governance is not a committee that meets quarterly. It is a working group with real mandate and key stakeholders such as product, engineering, security, legal, data, ethics, and business operations, etc. in the same room ensuring the AI governance framework has the right principles and can be executed across the entire Agentic AI lifecycle with continuous risk management and audit.
Start here:
Build a cross-functional AI governance group with decision-making authority, not just advisory input.
Those questions that I stated above, make sure they are answered as a part of your AI governance framework.
Define governance pillars across the full lifecycle: data validation, model testing, deployment controls, human oversight thresholds, and incident response.
Establish hard rules for when a human must review or override an AI decision before it executes. Map those rules to reversibility, recoverability and unintended consequences. If the outcome cannot be reversed or undone, or the unintended consequence is beyond your risk appetite, human approval is required by default.
Human oversight is not the same as human-in-the-loop. You don’t need a human approving every action by clicking through (and it doesn’t scale), but you do need a clear mapping of what actions and invocations an AI agent is not allowed (throughout its lifecycle) and requires human to carry out that action or make that decision.
Require explainability and immutable log traces that log all key actions across all AI agents throughout their lifecycle. If you cannot trace how an agent reached a conclusion, you cannot defend it when things go wrong. Things will go wrong.
Your governance pillars must be executed and enforced through actual (deterministic) controls and engineering. See step 5.
Your AI agents are already making million dollar decisions, in your enterprise, right now. Whether you know it or not. When (not if) agentic AI makes one or more wrong decisions, who in your organisation is accountable?
Pillar 5: Controls and Engineering
As enterprises are shifting from chat-based AI to retrieval-augmented generation (RAG) and agentic AI-driven workflows, the questions around reliability, predictability, security, safety and trust are even more important to answer.
Studies have shown that AI agents can go rogue, show signs of misalignment and even they are restricted to take certain actions or have restrictions in permissions, they are capable enough of findings ways out of their “sandbox”, in certain cases even coordinating with other AI agents to carry out those actions on its behalf. AI agents seek to fulfil the goal no matter the consequences or by whatever means it takes.
It does not mean AI is conscious. It just means it has no understanding of ethics, repercussions, ownership, alignment, purpose, etc. Something I showed in the diagram above on what AI is great at vs. what it’s good at vs. what it’s not so good at.
There is crazy amount of innovation just from the AI agents, and between them, in seeking the goal no matter what it takes to get there. For example, an AI agent with no privileges will “cheat” by convincing another AI agent with privileges to carry out the task on its behalf.
That last line is a massive liability for organisations. This is what causes it to go rogue in pursuit of its target. That innovation comes at a cost of corporate liability and accountability that needs to be managed in reality. Let's look at these three different examples.
A CEO vibe-coded his entire startup. A guy received a seemingly harmless call from a sales representative that turned out to be an AI after it got "hacked". Three Samsung engineers used ChatGPT to debug source code and transcribe meeting notes.
Three different stories. One common theme.
Lack of deterministic cybersecurity controls and engineering. Prompting is not security.
Start here:
Start with first principles. Defence in depth. Apply basic security hygiene to every AI deployment: authenticated endpoints, principle of least privilege, network segmentation. The McKinsey breach used unauthenticated endpoints and a standard SQL injection. No sophisticated attack vector. No zero-day. No insider access. First principles need to be implemented as a part of basic cyber hygiene.
Implement policy-based deterministic controls at the infrastructure layer. Define what the agent can access and block everything else, at the infrastructure level, not in a system prompt. Block internet access. No permission to write. Only allow at the infrastructure and network layer what’s absolutely needed. NVDIA’s open-source OpenShell is a good example of that.
Classify every AI-driven decision by reversibility and evaluate for human oversight. If the outcome cannot be undone, require human approval before execution. This needs to be coded by only allowing actions that AI agents are ever allowed to execute.
Attach an immutable, unique identifier to every single AI agent. This is your Non-Human Identity with secrets storied in a password manager not in your code or a security.md file, and tokens as short-lived ephemeral credentials, requiring re-identification of your NHI.
Log every action every AI agent takes: every tool call, every file touched, every external connection. No traceability means no auditability. No auditability means no accountability. Logging what an AI agent did is table stakes. Logging why it did it is what separates auditable AI from a black box. But these need to be immutable reasoning traces, not just what AI “told you”. Google’s Thought Signatues are a good example of that.
Test your controls by attempting to bypass them. If a creative prompt can override your guardrails, they are not guardrails. Red team it. If AI is good at one thing is being pointed to another AI system to have a go at it. Now, it doesn’t mean AI will hack it. You need to provide it and teach it what good “red teaming” or “bad prompts” looks like, but obviously not in your production environment. Have a simulated replica of your production environment to test it. If you’ve ever used metasploit for testing your applications/network, NVIDIA’s garak provide a similar framework but for exploit-testing LLMs.
Audit your AI systems continuously. Your AI agents will need your audit requirements (ISO, IIA, SOC2, etc.) as context.
If your AI controls can be bypassed by a prompt, they are not controls.
You cannot predict every path an agent will take to complete its task. You can only limit how much damage it causes when it takes the wrong one. The latter is key.
You need engineering and controls across the entire agentic AI lifecycle. A robust AI governance and security architecture looks like below. This is not accidental but is closely mapped to the OECD’s AI System Lifecycle phases.
Image 4: AI Governance and Security Architecture
ICYMI:
Driving 417 kmh Isn’t The Problem, Lack of Resilience Is
5 key pillars of enterprise AI security and governance for trustworthy and resilient AI that scales? Read full story —>
Step 4: Current AI Governance and Security Maturity Assessment
In July 2025, IBM's data breach report found that 63% of AI-related incidents involved systems that had no formal governance baseline at the time of the incident. Not because those organisations had ignored governance. Because they had never formally verified it.
Step 2 gave you a directional reading across all five dimensions. Step 4 goes further. This is the formal baseline: a structured, dimension-by-dimension AI maturity assessment that establishes where your organisation actually sits, verified against defined criteria rather than self-reported.
The AI Governance and Security Maturity Assessment combines evidence review across governance documentation, technical configuration verification, and scenario-based testing. The output is a per-dimension score and a prioritised picture of where the most urgent remediation is needed before any further AI deployment or scaling.
Three maturity levels, five independent dimension scores.
Every organisation falls into one of three levels across each dimension.
Fragmented: Governance and security controls are reactive, absent, or inconsistently applied, and the foundation for responsible AI deployment has not been established.
Emerging: Governance structures are forming but only a fraction of security and governance controls unevenly applied across the organisation, with some visible progress but gaps in consistency and accountability creating residual risk that requires prioritised remediation, and lack of resilience when AI fails.
Accelerating: Governance and security controls are mature, integrated, and consistently applied, and the organisation is positioned to scale AI responsibly and lead on governance within its industry.
Your overall maturity level reflects the combined picture.
But each of the five dimensions scores independently, and that distinction matters.
It is entirely possible, and in practice very common, to be Accelerating in Controls and Fragmented in Culture and Literacy simultaneously. An overall Emerging score with a Fragmented Data Validation dimension means your governance only protects part of your AI exposure. Your riskiest agents may be your least governed ones. The per-dimension score tells you where to act first. Think of it like a 5 x 3 matrix with measurable metrics (KPIs and KRIs) across each one of them.
Step 5: Target AI Governance and Security Maturity
You cannot define target outcomes without target metrics. If you cannot measure it, you cannot monitor it, improve it and you cannot reach your target maturity level. Here are some examples of what it means to define that tangibly:
Data Validation and Readiness: EU AI Act Article 10 and GDPR Article 35 require data governance as the standard for high-risk production systems. Target metrics for example:
KPI: % of data sources feeding AI agents classified before any agent access (KPI)
KRI: Defining x% unclassified data sources accessible by any AI agent in production (KRI) as the maximum threshold. Any unclassified source beyond threshold is not a gap to monitor. It is a direct breach.
Strategy and Leadership: AI systems influence board-reported decisions, financial outcomes, regulatory standing and more. Target metrics for example:
KPI: Defining what x% of AI agents in production must be with a named accountable executive
KRI: Max x% threshold for AI agents operating outside the use case registry (KRI). If an agent is not in the registry, it is ungoverned by definition.
Culture and Literacy: EU AI Act Article 4 mandates documented AI literacy for deployers. Target metrics for example:
KPI: x% of AI-using employees completing structured literacy training on a regular basis and employing those in their roles
KRI: % teams deploying AI in production with no completed literacy assessment on record, to understand the deviation and high-risk deployments.
Governance, Policy and Risk Management: Governance, risk management and accountability is mandatory for all high-risk deployments and AI systems. Target metrics for example:
KPI: x% of AI agents registered as governed Non-Human Identities with individual governance records
KRI: A tightly and clearly defined x% threshold for high-risk AI decisions running without a defined and technically enforced human oversight checkpoint. That number is your direct EU AI Act Article 14 exposure.
Controls and Engineering: OWASP's LLM Top 10 and MITRE ATLAS both document adversarial techniques that specifically exploit Fragmented and Emerging control postures. Target metrics for example:
KPI: x% of AI agents with active reasoning trace logging in production
KRI: x% threshold of autonomous workflows where a single irreversible action can execute without a circuit breaker (KRI). A circuit breaker that exists only in a policy document is not a circuit breaker.
Step 6: Your Enterprise AI Deployment Roadmap
Knowing where you are and where you need to go is necessary. Getting there without a structured deployment path is where most organisations fail. The gap between your Step 4 current maturity and your Step 5 target is not closed with ambition.
You close it with a sequenced, governed, risk-tiered roadmap.
The first principle: you deploy by risk, not by readiness. That is the EU AI Act's foundational logic, and it should be yours. High-risk AI systems require documented risk management, data governance controls, and human oversight before a single model touches production. If your current maturity sits in Fragmented for any dimension, you do not deploy high-risk use cases. Full stop. You fix the foundation first.
Phase 1: Foundation
Before any AI agent or model enters production, every dimension must exit Fragmented. This is the prerequisite the NIST AI RMF sets for responsible deployment under the MANAGE function, and it is the minimum floor ISO/IEC 42001 requires for any AI management system certification pathway to be credible.
Phase 2: Scale
Phase 2 moves your high-risk use cases from Emerging toward Accelerating. Deployment velocity matters here, but so does the governance gate you run before each new use case enters production: risk classification confirmed, data governance controls verified, human oversight mechanism active at go-live.
Phase 3: Optimise
Phase 3 is where you sustain and continuously improve within Accelerating. This is the hardest phase because the threat surface keeps moving. Mythos and many other models have shown that.
Adversarial AI attacks evolve faster than ever before. Regulatory expectations tighten. Your workforce changes.
Your deployment roadmap is not a Gantt chart. It is a governance and security contract with your organisation, your customers, and your regulators.
Step 7: Measure, Manage and Repeat
Most governance programmes treat measurement as a reporting obligation: monthly dashboards, quarterly board updates, annual audits filed and forgotten. That is backward.
Measurement is a risk management function, and it runs continuously. Resilience is built by monitoring, measuring and adapting in real time, not by just building dashboards.
The NIST AI RMF MEASURE function defines it clearly: analysing, assessing, and monitoring AI risks is an operational discipline, not a compliance exercise.
You have two instruments in Step 7.
KPIs tell you how your AI systems are performing. KRIs tell you where the next failure is forming before it arrives. Both must be live, not archived.
What The Metrics Reveal
KPIs and KRIs that tell you whether governance is functioning or only documented. To give you a sense of what those indicators surface, for example:
The percentage of unclassified data sources accessible by any AI agent
The number of AI agents running with shared or over-privileged NHI credentials
The percentage of AI use cases tied to measurable business outcomes
The number of high-risk AI decisions with no human oversight checkpoint defined or enforced
The percentage of AI agents with active reasoning trace logging in production
The number of autonomous workflows without circuit breakers where a single agent action is irreversible
These are not aspirational. They are the signals that tell you, in real time, whether your AI governance and security programme is protecting your organisation, resilient and responsible or merely giving the appearance of doing so.
What’s Next?
Multiple studies (e.g. Google, CSA, etc.) have found that governance is the strongest indicator for agentic AI readiness in your enterprise.
This quick AI governance and security readiness assessment, developed by 3 x CISO and AI Governance Expert and Advisor, and someone who first-hand built the national AI strategy, and helped organisations build and implement AI in their organisations, is a strong indicator of your AI readiness state in terms of what actually matters.
Simply answer quick 35 multi-choice questions (takes < 10 min) and get a tailored assessment of your agentic AI readiness across the five key dimensions:
a) Data Readiness and Validation
b) Strategy and Leadership
c) Culture and Literacy
d) Governance, Policy and Risk Management
e) Controls and Engineering
It’s tailored, takes less than 10 min, and it’s completely free.
Until next time, this is Monica, signing off!
What did you think of this edition?
— Monica Verma
P.S. Please follow me/subscribe on Youtube, Linkedin, Spotify and Apple. It truly helps. Or book a 1-1 advisory call, if I can help you.
***
