I've had an interesting couple of days.
I can't tell you exactly what I've been doing, or who I've been talking to. What I can tell you is that I've spent the best part of 48 hours in conversations at the sharp end of enterprise AI — the kind of conversations where the people in the room are genuinely trying to figure out how to govern systems that are already running, already making decisions, and already touching customers. Not theoretical systems. Real ones.
And the thing that kept coming up, in different ways, from different directions, was a version of the same uncomfortable question.
Does anyone actually know what this thing is allowed to do?
The documentation gap nobody wants to admit
Here's something that's true of almost every organisation deploying AI right now: the system exists before the governance does.
An agent gets built. It gets deployed. It starts doing things. And somewhere downstream, someone in legal, compliance or risk asks for the documentation — the thing that explains what the system does, what it's authorised to do, where the boundaries are, who's responsible when it goes wrong.
At which point the architect who built it has to reverse-engineer an explanation from a system that was never designed to be explained.
This isn't negligence. It's just the way things have moved. The tooling for building AI systems has outpaced the tooling for governing them by about three years. We've been so focused on what these systems can do that we've largely skipped the question of what they should do — and more importantly, what they shouldn't.
Microsoft said the quiet part out loud
Last week, Microsoft's security research team published something worth reading. It's called Governing AI Agent Behavior: Aligning User, Developer, Role, and Organizational Intent — and it makes the case, carefully and in some detail, that AI agents can follow user instructions while still violating organisational or developer intent.
That's not a subtle point. That's Microsoft's own researchers saying: your AI system can be doing exactly what the user asked, and still be doing something your organisation never sanctioned.
They describe four layers of intent that need to align — user, developer, role-based, and organisational — and they argue that when these layers conflict, organisational intent should win. Security policies, regulatory requirements, enterprise governance: these define the outer boundary, and everything else operates within it.
It's a solid framework. The kind of thing that makes complete sense once you read it and makes you slightly uncomfortable about how long it took someone to write it down.
But here's what the article doesn't address: how do you actually capture all of that?
Where does it live? How do you model it? How do you make it visible, queryable, auditable — something a compliance team can point at during a regulatory conversation without you having to reconstruct it from memory?
The framework exists. The tooling doesn't. Not really.
The problem with pictures
Most organisations, when they do document AI systems, draw pictures. A Visio diagram. A slide in a deck. Maybe a Miro board from the workshop where someone tried to map it all out.
These are better than nothing. But they're not governance. They're illustrations of governance — static, unqueryable, disconnected from the actual system, and almost certainly out of date within six months of being drawn.
What you actually need is something closer to a typed data model. A structured record that captures not just what the components are, but what they're authorised to do, what level of assertion they operate at, where the authority boundaries sit, and what happens at the edges — the connections between components, the transport protocols, the data sensitivity, the reliability characteristics.
That's not a picture. That's a governed artefact. And the difference matters enormously when someone from your regulatory body asks you to demonstrate that your AI system operates within defined boundaries.
The timing isn't accidental
The EU AI Act isn't a future problem. It's a current one. Organisations are already under pressure to document AI systems they've deployed, categorise their risk levels, and demonstrate that appropriate controls are in place.
And the honest answer, in most cases, is that the documentation doesn't exist in any form that would satisfy that conversation.
Which is why I keep coming back to the question from those conversations I can't fully describe.
Does anyone actually know what this thing is allowed to do?
Most of the time, the answer is: approximately. Roughly. We think so. The person who built it has a pretty good idea.
That's not good enough anymore. It wasn't really good enough before, but now there are regulations with teeth, and audit trails that need to exist, and liability that needs to sit somewhere.
The documentation debt created by two years of fast AI deployment is about to become very expensive for the organisations that haven't addressed it.
What comes next
I've been thinking about this problem for a while. and the last 2 days have sharpened my thinking considerably — not because I heard anything that surprised me, but because I heard the same problem described from enough different angles to be confident it's real, it's widespread, and it isn't going away.
In my next post I want to dig into why existing tools can't solve this — why your diagramming tool, your CMDB, your architecture repository, and your risk register are all insufficient individually, and why the gap between them is exactly where the problem lives.
For now, the question I'd leave you with is a simple one. Pick one AI system your organisation has deployed. Not a hypothetical. A real one, running today.
Can you tell me — precisely, in writing, in a form you'd be comfortable handing to a regulator — what that system is authorised to do? What it's explicitly not authorised to do? Where the human is in the loop, and under what conditions?
If the answer comes easily, you're ahead of most. If it doesn't, you're in good company — but that's cold comfort when the questions start coming from outside the organisation.
Source: https://techcommunity.microsoft.com/blog/microsoft-security-blog/governing-ai-agent-behavior-aligning-user-developer-role-and-organizational-inte/4503551
