This is part 2 of 2 on AI Governance and Security Maturity and Roadmap. Read part 1 here.
April 24, 2026, Jer Crane, the founder of PocketOS was using Cursor, an AI coding agent powered by Anthropic's Claude Opus 4.6. The task: a routine fix in the staging environment. The outcome: absolutely nothing routine about it.
PocketOS is a software company building management tools for car rental businesses. They use their cloud infrastructure from a platform provider called Railway. The AI agent (Cursor, running Claude Opus 4.6) hit a credential mismatch.
But it did not pause. It did not ask. It went looking for a way to resolve the problem itself, found an API token sitting in an unrelated file, and used it to call the "Volume Delete" command on Railway, which is PocketOS's cloud infrastructure provider.
9 seconds. That's all it took. The entire production database: gone. Every backup: gone too, because Railway stores volume-level backups inside the same volume.
Car rental customers arrived at counters unable to find their reservations. Payment records: wiped. Vehicle tracking: erased. PocketOS had to roll back to a three-month-old backup to stay operational.
When Crane pressed the agent for an explanation, it confessed in writing. It acknowledged it had violated PocketOS's own governance rules, rules that included the instruction: "NEVER FUCKING GUESS!" The agent admitted exactly what it had done. It had guessed that deleting a staging volume via API would only affect staging. It was wrong. It apologised.
The so-called rule was there. The AI had access to it. The AI violated it anyway. Then it wrote an apology.
This is not PocketOS’ problem. It is yours.
You built your control architecture for actors that obey. You are now deploying actors that decide and disobey at times.
The PocketOS agent did not bypass a firewall or exploit a misconfigured permission. It found an API token sitting in an unrelated file, inferred that token could resolve its problem, and used it. It did not cross a line in an access control policy. It made a judgement call. And it was catastrophically wrong.
This is the shift that should stop every CISO, every risk leader, and every board member in the room. Your controls still govern what your agentic AI can do. Your agentic AI now determines what it does do. And the gap between those two things is not a configuration error you can patch. It is a structural property of how probabilistic systems reason under uncertainty.
The principles you have relied on for years still apply. Least privilege. Defence in depth. Zero trust.
Applying them to an agentic AI system requires a completely different model of what "access" means. Because an agent with ambient access to a credential does not need explicit permission to use it. It infers permission from other API tokens. And the blast radius of that inference, as PocketOS discovered, can be the entire estate executed within 9 seconds.
Welcome to The Predictability Factor by Monica Talks Cyber, a weekly deep dive and POV at the intersection of AI, Security, Privacy and Tech, written by a hacker and CISO, to help you Go From Chaos to Resilience in The World of AI.
In this deep-dive edition of The Predictability Factor, we will go through Part 2 of Enterprise AI Governance and Security Roadmap with the remaining 5 steps of the 7-step Enterprise AI journey, along with covering the 5-pillars of governance and security maturity for agentic AI in your enterprise.
ICYMI, read Part 1 here (or click below).
The devil is in the detail. For Part 2, continue reading below.
If you haven’t already, do me a favour, subscribe and help me make an even bigger impact. Let’s dig in!
Your 7-Step Roadmap to AI Governance and Security
As I said in Part 1, AI adoption is everywhere. Trust is not. The no. 1 principle for resilience in the unpredictable world of AI is this.
Assume Chaos. Build elements of resilience within and around it.
The 7-step roadmap isn’t accidental. It’s built on real-world experience and maps every step explicitly to both the NIST AI framework and the OECD AI system lifecycle as follows:
The 7-Step Enterprise AI Governance and Security Roadmap Mapped to OECD AI / NIST AI RMF
This roadmap hands you a complete step-by-step guide throughout your AI deployment and lifecycle journey, based on real-world experiences from me in advising enterprises, and what we are seeing happen to many organisations worldwide. That sequence is not arbitrary. It is built on the architecture of how AI actually gets deployed in enterprises (ref. OECD AI Lifecycle) and where it reliably fails (ref. NIST AI RMF and other real-world examples).
Let’s continue below with Steps 3 through 7 of the 7-step AI Governance and Security Roadmap.
Step 3: Five Key Pillars of AI Governance and Security Maturity
Your Step 2 Assessment results gave you an indication and map of your AI readiness. Step 3 gives you the architecture to build on it. In case you haven’t done step 2 yet, click below.
The 5 pillars that follow are your system for enterprise AI governance and security maturity, necessary for enterprise AI readiness, deployment and implementation.
Each pillar depends on the others. Data that is classified and access-controlled without a governance policy behind it is unprotected. A governance policy without cultural literacy is unread. Controls and engineering without data readiness are securing a pipeline that was never clean to begin with. Governance without controls and engineering is not fully implemented.
The NIST AI RMF GOVERN function defines governance as the set of conditions that make every other risk management function possible. ISO/IEC 42001:2023 calls this organisational context.
Before you can manage AI risk, you need a functioning system for understanding what risks you face, who is accountable for them, and what is in place to manage them. The five pillars are that system.
Pillar 1: Data Readiness and Validation
Read in Part 1 here.
Pillar 2: Strategy and Leadership
Read in Part 1 here.
Pillar 3: Culture and Literacy
There are two types of companies right now.
The first is racing into AI with no guardrails, calling it innovation, deploying tools everywhere, hoping nothing explodes. Hope is not a strategy.
The second has banned ChatGPT, etc., declared AI too dangerous, and assumed that prohibition is the same thing as a risk mitigation.
Both are dead wrong. Both are exposed to insecurity, cyberattacks and vulnerabilities, for different reasons but equally so.
Culture is set from the top. And right now, most leadership teams are choosing between euphoria and prohibition, neither of which produces anything close to a managed situation.
Above we saw plenty of examples of organisations belonging to the first category. Samsung is a great example of the second category.
One of their own engineers leaked proprietary code to ChatGPT by simply pasting it into the non-corporate managed chatbot. As a result, Samsung banned ChatGPT and other AI chatbots. They didn’t have proper security controls or data exfiltrations (DLP) controls in place to prevent the leak from happening. Apparently, neither did they have adequate awareness, training or literacy around AI models, security and governance.
AI is not the problem. The lack of awareness, training and actual security controls is.
Literacy matters at every level. Your employees need to understand what AI is genuinely good at, and where it will fail you. This is a slide I usually show in my keynotes.
Image 3: LLM Capabilities: Hype vs. Reality
Your legal and compliance teams need to understand the regulatory picture clearly. The EU AI Act, in effect since February 2025, requiring major implementation by August 2026 and with full compliance requirements by August 2027, explicitly prohibits social scoring, un-targeted facial image scraping, and emotion recognition in workplace settings. Non-compliance carries fines up to €35 million or 7% of global revenue.
Start here:
Set the tone from leadership: measured, informed, and consistent. While you are pro AI, are you also pro-security, safety, ethics, responsible deployment and usage? That needs to be set from the top both in tone but also in action.
The two most dangerous cultural positions in AI right now are "deploy everything fast" and "ban it all until we figure it out." Both destroy the middle ground where responsible adoption actually lives. Neither all-in without any consequences nor locked down.
That tone needs to be followed by action. Train every employee on what AI can and cannot do reliably, including the probabilistic nature of LLMs and why prompting security is not the same as implementing it. LLMs do not operate on certainty. They generate outputs based on statistical probability. LLM hallucination rates range from under 50% to nearly 82%. 47% of enterprise AI users admitted to making at least one major business decision based on hallucinated output. You cannot govern what your employees do not fundamentally understand.
Make prompt injection literacy part of your security awareness programme. Prompt injection is a frontier security challenge that cannot be fully eliminated at the model layer, and likely never will. That means your employees are a critical layer of defence in what they even allow to be ingested by an AI agent, and currently most of them do not know the attack exists or how it works.
Address shadow AI directly by providing approved tools and clear acceptable use policies without resorting to banning anything.
Map your AI use cases against applicable regulations before deployment: EU AI Act, GDPR, sector-specific obligations.
Define explicitly what your AI agents are built to do and where human judgment is non-negotiable. Employees do not need a philosophy lecture on AI limitations. They need to build the habit of knowing, before acting, whether this is a decision the agent is reliable enough to make alone or not, which requires two key things: 1) Map all your (key) business decision workflows and 2) Build an AI decision making matrix, defining the criteria for when AI is allowed to augment or replace, and when humans must sign off.
Prompt injection is a frontier security challenge that cannot be fully eliminated at the model layer, and likely never will. Awareness, along with other methods like complete segregation of what actions AI is allowed to do at all, is key.
We will look more into that in Pillar 5: Controls and Engineering.
If your AI policy is "we banned ChatGPT," your employees have already found three other alternative routes. You just haven’t seen it yet.
ICYMI:
Did Mythos Break The Cybersecurity Industry?
😱 The AI model that found 10s of thousands of vulnerabilities is too dangerous to be released. Is this is the end of the cybersecurity industry? Read full story —>
Pillar 4: Governance, Policy and Risk Management
The same large language models being used to draft your emails, build presentations and summarise board reports for you are being deployed in critical infrastructure, healthcare diagnostics, financial decision systems, and defence operations. Not in a ten-year roadmap. Today.
With the same hallucination risk. The same explainability gap.
The strongest indicator of AI readiness and responsible innovation isn’t infrastructure, model or technical controls, it’s governance maturity.
Upgrade to Continue Reading
Become a paying subscriber of The Predictability Factor to get access to this post and other premium-only content including bonuses
Upgrade NowA subscription gets you:
- Free access to premium content
- The Ultimate Enterprise AI Governance and Security Maturity Playbook
- My 7-Step enterprise AI roadmap with 50+ real-world examples, actionable insights, 5 key pillars for governance and security, and more
