
AI After the Demo — Part 11
Monday morning.
Jeannine asks the AI a question.
“Which customers should I contact this week?”
The AI returns ten customers.
- Reasonable recommendations.
- Clear explanations.
- Everything looks good.
Tuesday morning.
Jeannine asks the same question.
The AI returns ten customers.
Seven are different.
Interesting.
She checks the reporting period.
Same.
She checks the filters.
Same.
She checks the customer data.
Nothing significant appears to have changed.
So she opens a ticket.
“The AI is giving different answers than yesterday.”
IT investigates.
No deployment.
No code changes.
No failed pipelines.
No infrastructure alerts.
The application is healthy.
The model endpoint is responding.
Everything is green.
Someone closes the ticket.
“Unable to reproduce. System operating normally.”
Jeannine reopens it.
Because the AI is still giving different answers.
Nobody deployed anything.
So what changed?
Welcome back to Production.
We Used to Know What a Deployment Looked Like
Traditional applications have taught us to look for certain things when behavior changes.
- Code deployments.
- Database changes.
- Configuration updates.
- New releases.
- Someone modified a stored procedure.
- Someone updated a report.
- Someone changed a business rule.
We investigate the change history.
We compare versions.
We identify what happened.
Sometimes we even discover who approved it.
But production AI introduces more moving parts.
The application code may remain unchanged while the behavior changes.
A model provider updates an underlying model.
A model alias points to a different version.
A retrieval index gets rebuilt.
New documents enter the search corpus.
An old document gets removed.
A prompt template changes.
A tool returns different information.
The business data changes.
The user asks the same question in a different conversation context.
Or the model simply produces a different response from the same inputs.
No application deployment doesn’t mean nothing changed.
Same Question. Different Answer.
Let’s return to Jeannine.
Yesterday, the AI recommended contacting Acme Manufacturing.
Today, Acme isn’t on the list.
Instead, it recommends another customer.
Is that wrong?
Not necessarily.
Maybe Acme placed a large order overnight.
Maybe someone resolved its service complaint.
Maybe the customer-risk definition changed.
Maybe the AI retrieved a different document.
Maybe the model interpreted the same evidence differently.
Or maybe the recommendations are both reasonable, but the ranking is unstable.
Those are very different explanations.
And they require very different responses.
The first question shouldn’t be, “Why did the AI change?”
It should be, “What part of the answer’s supply chain changed?”
The Answer Has a Supply Chain
We’ve talked before about how an AI answer doesn’t appear out of thin air.
It depends on an entire chain of components.
- The user’s question.
- Conversation context.
- Instructions and prompts.
- Model and model version.
- Retrieved documents.
- Data sources.
- Tool responses.
- Business rules.
- Filters.
- Permissions.
- And the logic used to combine everything.
Change any of those, and the answer could change.
But here’s the problem.
Most organizations aren’t recording enough information to reconstruct that chain.
They know the question was asked.
They know the model returned an answer.
Maybe they know how long it took.
But can they tell you which model version produced it?
Which prompt version was used?
Which documents were retrieved?
Which data snapshot was consulted?
Which tools ran?
Which business definitions were applied?
If not, we’re going to spend a lot of time investigating changes we can’t reproduce.
The Model Name Isn’t Always a Version
Here’s an easy trap.
Someone says, “We’re using Model X.”
Great.
Which version?
“The current one.”
Wonderful.
That narrows it down to whatever “current” means today.
Some model services allow deployments or aliases that can change their underlying model versions.
Others require explicit version selection.
The behavior depends on the provider and configuration.
But the architectural lesson is the same.
A model name isn’t necessarily enough to establish what produced an answer.
If the model can change independently of your application deployment, you need to understand that lifecycle.
Otherwise, you’re relying on a component whose behavior may change without appearing in your normal release history.
That’s not automatically bad.
It does mean your change-management process needs to account for it.
The Model Might Not Be the Problem
Now imagine the model version didn’t change.
Same model.
Same prompt.
Same application code.
Yet the answer is different.
Someone rebuilt the vector index overnight.
The underlying documents are mostly the same.
But chunking changed.
Or embeddings changed.
Or ranking changed.
Yesterday, the AI retrieved a current customer policy.
Today, it retrieves an older document that happens to be more semantically similar to the question.
The model answers based on what it received.
And the answer changes.
The model didn’t change. What we showed it did.
This is why treating the model as the entire AI system is such a mistake.
Data Changes Are Supposed to Change Answers
There’s another important distinction.
Sometimes the answer should change.
If a customer places a large order, that may affect its risk classification.
If Finance updates the reporting period, totals may change.
If a contract expires, its status should change.
We don’t want AI producing identical answers forever.
That would be a different problem.
So the objective isn’t to eliminate change.
It’s to understand it.
Expected changes should be explainable.
Unexpected changes should be investigated.
And changes that affect important business decisions should be evaluated before everyone simply assumes they’re fine.
The Sherpa’s Notebook
This isn’t entirely new territory for data professionals.
I’ve spent years working with SQL Server, Snowflake, and reporting systems where someone says, “This report was right yesterday.”
And then we start investigating.
- Did the source data change?
- Did a stored procedure change?
- Did the ETL process change?
- Did a business rule change?
- Did someone change a join?
- Did a late-arriving transaction appear?
- Did a customer record get corrected?
Sometimes the answer is obvious.
Sometimes it takes hours.
But at least we generally know where to look.
We have:
- Query definitions.
- Stored procedure versions.
- Execution history.
- Data timestamps.
- Source records.
With AI, we’re adding another layer of behavior on top of that environment.
Now the same data can produce different wording, different emphasis, or different recommendations.
And if we haven’t captured the context that produced the original answer, we may not be able to reconstruct what happened.
We didn’t eliminate change management. We made it more interesting.
Reproducibility Is Harder Than It Looks
Let’s say Jeannine gives us yesterday’s answer and today’s answer.
We try to reproduce yesterday’s result.
Same question.
Same model.
Same prompt.
Same data.
Still a different answer.
Why?
Because many generative models are not guaranteed to produce identical output on every invocation.
Even with settings intended to reduce randomness, exact reproducibility may not be guaranteed across all environments and providers.
That’s an important distinction.
We should not promise deterministic behavior where the underlying technology doesn’t guarantee it.
But we can still make the system more explainable.
Record the relevant versions.
Capture retrieval results.
Track tool executions.
Preserve the inputs necessary for investigation, subject to security and retention requirements.
Then evaluate whether the answer meets the business’s expectations.
Because exact wording isn’t always the thing that matters.
If the model recommends the same ten customers with slightly different explanations, that may be perfectly acceptable.
But if it recommends ten completely different customers with no meaningful change in the evidence?
That’s worth investigating.
Version More Than Your Code
Here’s where I think architecture teams need to expand their thinking.
We already version application code.
We version database schemas.
We version infrastructure configurations.
For production AI, we should also consider the versions of the components that influence answers.
The prompt template.
The model deployment.
The retrieval configuration.
The embedding model.
The indexed document collection.
The business rules.
The tool definitions.
The evaluation criteria.
Not everything needs to be stored in exactly the same way.
And we don’t need to preserve every byte of every system forever.
But we should be able to identify the relevant configuration and evidence for important interactions.
Otherwise, when someone asks why the AI behaved differently, our best explanation may be, “Well, something must have changed.”
Thank you, Detective.
Your Evaluations Need to Follow the Changes
Remember our previous article?
Your AI Is Working Perfectly. According to What?
We talked about building evaluations around real business questions.
This is where that work becomes useful.
Suppose we have an evaluation set containing 100 representative questions.
We update the model.
Run the evaluations.
Most still pass.
But the AI now handles ambiguous customer-risk questions differently.
Maybe it stops asking for clarification.
Maybe it starts presenting predictions as facts.
Maybe it omits important exceptions.
The application still works.
The model is faster.
The new version performs better on a public benchmark.
But our business evaluations reveal a regression.
That’s why the evaluation set matters.
Not because it guarantees perfection.
But because it gives us a way to determine whether changes affect the behavior we’ve agreed is acceptable.
Not Every Change Needs a Deployment
Here’s another nuance.
We shouldn’t force every business data update through a software release process.
That would be absurd.
Customer orders change every day.
Documents get approved.
Policies get updated.
Business activity continues.
Those changes should flow through the system.
But changes to the rules governing how AI interprets or uses that information deserve more attention.
Updating a customer’s address is not the same as changing the definition of an active customer.
Adding a newly approved policy is not the same as changing the retrieval rules so archived policies become eligible.
Updating a tool’s data is not the same as changing what actions that tool can perform.
Different changes carry different risks.
Our controls should reflect that.
The Ticket Nobody Wants
Let’s imagine Jeannine opens her ticket again.
“The AI recommended different customers today.”
This time, instead of closing it because no application deployment occurred, we can investigate.
Model version: unchanged.
Prompt version: unchanged.
Retrieval configuration: changed overnight.
Business data: unchanged.
Evaluation results: customer-risk scenario now fails.
Root cause identified.
The new retrieval configuration prioritized a different set of documents, changing the evidence used to produce the recommendation.
Now we have an explanation.
And a path forward.
- Compare the retrieved evidence.
- Correct the configuration.
- Rerun the evaluations.
- Verify the expected behavior.
- Close the ticket.
Not because everything was green.
But because we understood what changed.
The Sherpa’s Lesson
Production AI doesn’t live inside a single application deployment.
Its behavior depends on models, prompts, retrieval, data, tools, and business rules.
Some of those components change independently.
Some change constantly.
Some changes are expected.
Others are not.
And some answers can vary even when the underlying inputs appear unchanged.
The goal isn’t to freeze the entire system.
It’s to make meaningful changes visible and explainable.
Version what matters.
Capture the context needed for investigation.
Evaluate changes against real business expectations.
And distinguish harmless variation from material changes in behavior.
Because when Jeannine asks why the AI gave her a different answer, she doesn’t care that your deployment dashboard is green.
She wants to know what changed.
And if the best answer we can give her is, “Nobody deployed anything.”
Then we haven’t answered her question.
We’ve just explained why our change-management process didn’t notice.
Leave a Reply
You must be logged in to post a comment.