The AI Security Model Nobody Designed

Your AI didn’t hack anything.

It didn’t bypass authentication.

It didn’t exploit a vulnerability.

It didn’t steal credentials.

Every permission check succeeded.

And somehow it still showed someone information they probably shouldn’t have seen.

Welcome to one of the more interesting problems with production AI.

Because sometimes the biggest AI security problem isn’t that your security model failed.

It’s that your security model worked exactly as designed.

We Built Security for Humans

Most enterprise security models were designed around a fairly simple assumption:

People access things.

• A user opens a report.
• A user queries a database.
• A user browses a SharePoint site.
• A user opens a document.
• A user runs an application.

Security answers whether the AI is allowed to retrieve something. It doesn’t answer whether that information is current, approved, or authoritative. That’s an entirely different production problem – and one we’ll get to next.

We decide whether that person should be allowed to perform that action.

Then we grant or deny access.

Over time, those permissions accumulate.

Someone joins a project.

They get access.

Someone changes departments.

They keep some access.

Someone needs temporary access.

Temporary becomes permanent.

A group gets nested inside another group.

A service account gets another role.

A SharePoint folder gets shared.

Then another.

Then another.

Eventually an employee may technically have access to an enormous amount of information.

But historically there was friction.

They had to know the information existed.

They had to know where it lived.

They had to know how to access it.

They had to know what to search for.

They had to understand enough of the surrounding systems to connect the pieces.

That friction quietly became part of our security model.

We just never called it security.

AI removes a lot of that friction.

Access Is Not the Same as Discovery

Imagine an employee has legitimate access to:

• A customer relationship system.
• A folder containing contracts.
• A Power BI report.
• A SharePoint site from an old project.
• Several operational databases.
• A collection of meeting notes.
• And some financial documents they were granted access to three years ago.

Before AI, finding a specific piece of information across all of that might take hours.

Maybe they wouldn’t even know it existed.

Now they ask: “Tell me everything we know about Acme Industrial.”

That’s a very different interaction.

The AI may search across every source it is allowed to access.

It can retrieve information.

Summarize it.

Compare it.

Connect it.

And present the useful parts in one place.

The user didn’t gain any new permissions.

They gained something potentially just as powerful.

Discovery.

The Sherpa’s Notebook

I’ve spent a lot of my career looking at permissions that made perfect sense individually.

A service account needed access to this database.

A reporting process needed access to that schema.

A user needed access to a folder for a project.

A developer needed elevated permissions temporarily.

A business team needed yet another dataset.

Every request had a reason.

Then you step back years later and look at the cumulative result.

Suddenly one identity can see half the organization.

Nothing necessarily happened because anyone was careless.

Permissions simply accumulated around real business needs.

That was always a security concern.

AI changes the consequences.

An identity with broad read access used to have the ability to retrieve enormous amounts of information.

Now an AI system can give that same identity the capability to understand and connect enormous amounts of information.

Those are not quite the same thing.

AI Can Connect Things Humans Never Did

Suppose an employee can legitimately access two documents.

Document A contains customer contract terms.

Document B contains internal margin targets.

The employee is allowed to read both.

Historically, those documents might live in completely different systems.

Different folders.

Different teams.

Different workflows.

Maybe the employee never had a reason to connect them.

AI does.

Ask the right question and suddenly information from both appears in the same answer.

Neither source violated its permissions.

The combination may still create a problem.

This is sometimes called an aggregation problem.

Information that is relatively harmless in isolation can become much more sensitive when combined.

AI is exceptionally good at combination.

That’s one of the reasons we’re deploying it.

It’s also one of the reasons we need to rethink security around it.

“The User Already Had Access” Isn’t Enough

This will be a tempting response when these situations happen.

“But the user already had access to those documents.”

Technically correct.

And sometimes that’s the end of the conversation.

If the employee is genuinely supposed to use all that information together, fantastic.

AI just made them more productive.

But sometimes the conversation reveals something else.

• Why does that person still have access to the old project folder?
• Why can this group see those contracts?
• Why does that service account have access to every schema?
• Why is this sensitive document available to everyone in the department?
• Why can this AI retrieve data from a system nobody remembered was connected?

AI didn’t create those permissions.

It made them useful.

And that can be uncomfortable.

Then We Add Service Accounts

Things get even more interesting when the AI doesn’t access information using the user’s identity.

Maybe the application uses a service account.

Or a service principal.

Or a shared database identity.

Or some wonderfully ancient account named something like SVC_APP_PROD_FINAL2.

That identity may have broad access because the application needs to serve many users.

Now we have a critical architectural question:

Whose permissions determine what the AI can see?

• The user’s?
• The application’s?
• The service account’s?
• Some combination thereof?

If the AI retrieves information using an identity with more access than the person asking the question, the application needs another mechanism to enforce the user’s authorization.

Otherwise we have built something very efficient.

A beautifully designed system for helping people discover information they weren’t supposed to discover.

Retrieval Is Part of Security

This is why production AI security can’t stop at authentication.

Who are you?

Still important.

Authorization matters too.

What are you allowed to access?

But AI introduces another important question:

What information is the system allowed to retrieve on your behalf?

Those rules need to survive the entire path.

• User.
• Application.
• Agent.
• Search.
• Database.
• Document repository.
• API.
• Model.
• Response.

It doesn’t matter if the front door checked the user’s badge if everything behind the door operates as SUPER_AI_ADMIN.

What About the Response?

There’s another boundary worth considering.

The model retrieves information the user is authorized to access.

Wonderful.

Then it generates a response.

• Can that response be copied?
• Downloaded?
• Emailed?
• Stored in conversation history?
• Passed to another agent?
• Written into another system?
• Used as input to an automated action?

Security doesn’t end when retrieval succeeds.

Information moves.

AI makes moving it extremely easy.

Production architecture needs to consider where information can go after the model touches it.

Agents Make This Much More Interesting

So far we’ve mostly talked about AI reading information.

Agents introduce actions.

Now the system can be used to:

• Read customer information.
• Create tickets.
• Send messages.
• Update records.
• Call APIs.
• Generate documents.
• Trigger workflows.
• Potentially execute operations across multiple systems.

At that point, identity and authorization become even more important.

• What identity is the agent using?
• What can it do?
• What can the user instruct it to do?
• What actions require approval?
• What happens if one tool has broader permissions than another?
• Can the agent combine permissions across systems in ways no human role ever could?

These aren’t theoretical security questions anymore.

They’re architecture questions.

And we’re going to have to answer them.

Soon.

Least Privilege Gets Harder

“Use least privilege” remains excellent advice.

It’s also easier to say than implement.

What is least privilege for an AI assistant expected to answer questions across the entire company?

If you grant it broad access, you increase risk.

If you restrict it too heavily, it becomes useless.

The answer isn’t simply: Give the AI everything.

Nor is it: Give the AI nothing.

The answer is designing authorization around the actual use cases, identities, information classifications, and actions the system needs.

And then continuously reviewing those decisions as the AI’s capabilities expand.

Because they will expand.

Nobody leaves a successful AI implementation alone.

Someone will eventually say:

“Could we also connect it to…”

That’s how architectures grow.

That’s also how permissions grow.

Your Existing Security Model Is the Starting Point

The good news is that organizations don’t need to throw away everything they’ve built.

• Your existing identity systems matter.
• Your roles matter.
• Your groups matter.
• Your data classifications matter.
• Your row-level security matters.
• Your object permissions matter.
• Your governance policies matter.
• Your audit logs matter.

AI doesn’t replace any of those.

It makes getting them right considerably more important.

Because AI can operate across them faster than humans ever could.

Ask the Question Before Production

Before deploying an enterprise AI system, pick a user.

Not an administrator.

Not the AI development team.

Jeannine.

Ask: “What can Jeannine get the AI to tell her?”

Then test it.

Not just what you intended her to access.

What she can actually retrieve.

What can be combined.

What can be inferred.

What old permissions still exist.

What sources the AI can reach through its own identity.

What happens when she asks unusual questions.

You may discover that your AI security problem has very little to do with AI.

You may simply be seeing your existing security model clearly for the first time.

The Sherpa’s Lesson

AI doesn’t necessarily need to break your security model to expose information.

Sometimes it only needs to make your existing permissions easier to use.

For years, complexity provided accidental protection.

People didn’t know where everything lived.

They didn’t know what they had access to.

They didn’t have the time or technical knowledge to connect information across dozens of systems.

AI removes those barriers.

That’s one of its greatest benefits.

It’s also a security risk we need to design for.

So don’t just ask: “Can this user access this data?”

Ask: “What can this user learn when AI connects everything they’re allowed to access?”

Because your permissions may have been designed years ago.

Your AI wasn’t.

And when the two finally meet in Production, you may discover you already had an AI security model.

No one designed it.