Your AI Is Working Perfectly. According to What?

AI After the Demo — Part 10

The dashboard was beautiful.

Green everywhere.

99.9% availability.

Response times under two seconds.

User satisfaction at 94%.

Thousands of questions answered.

The AI team was thrilled.

Management was impressed.

Someone probably put the dashboard in PowerPoint.

Then Jeannine asked a question.

“How do we know the answers are right?”

Silence.

Someone pointed at the satisfaction score.

Someone else mentioned that the model was grounded in company data.

A third person explained that the application hadn’t experienced an outage in weeks.

All wonderful things.

None of them answered Jeannine’s question.

Your AI is working perfectly, but according to what?

Welcome back to Production.

We Measured Everything Except the Answer

When organizations deploy AI, they tend to measure the things they’re already comfortable measuring.

Availability.
Response time.
Error rates.
Token usage.
User adoption.
Maybe thumbs-up and thumbs-down feedback.

Those are useful metrics.

They tell us whether the system is available, responsive, and being used.

But they don’t necessarily tell us whether the answers are good.

A model can respond in 800 milliseconds and be completely wrong.

It can retrieve an approved document and misunderstand it.

It can produce a perfectly formatted summary that leaves out the most important exception.

It can receive positive feedback from someone who doesn’t know that the underlying information is very incorrect.

And it can answer the question exactly as asked while completely missing what the business needed to understand.

We built excellent dashboards for the conversation. We forgot to define what a good conversation looks like.

Forty-Seven Customers. Four Different Answers.

Let’s return to Jeannine.

She asks, “Which customers are at risk of leaving us?”

The AI analyzes the available customer data and identifies 47 accounts.

The answer looks reasonable.

It includes explanations.

It references declining purchases, payment history, and customer service activity.

Awesome.

Then Sales reviews the list.

“That can’t be right. We only have 32 at-risk customers.”

Finance disagrees.

“We’ve identified 41.”

Customer Service joins the conversation.

“We’re tracking 53.”

Now we have a problem.

Four answers.

Four different numbers.

And four groups that believe they’re correct.

Is the AI wrong?

Maybe.

But let’s examine what happened.

Sales defines risk primarily through declining orders and customer engagement.

Finance focuses on overdue balances and payment behavior.

Customer Service looks at unresolved issues and repeated complaints.

The AI combined signals from all three.

Nobody agreed on what “at risk” meant.

Yet we expected the AI to produce the correct answer.

Correct according to whom?

AI Didn’t Invent the Disagreement

Here’s the uncomfortable part.

The disagreement truly existed before the AI arrived.

Sales had its report.

Finance had its report.

Customer Service had its report.

Each team worked from its own definition.

Nobody noticed the conflict because nobody put all four answers in the same meeting.

Then AI came along.

One interface.
One question.
One answer.

And suddenly everyone expected consistency.

AI didn’t create the disagreement. It just made the disagreement impossible to ignore.

Sound familiar?

We’ve seen this pattern throughout the series.

But this time, the problem isn’t a missing document, an undocumented process, or an unauthorized action.

It’s that nobody defined what a correct result should look like.

A Good Answer Needs a Definition

This is where we need to distinguish two things.

The question someone asks and the criteria we use to determine whether the answer is acceptable.

Those aren’t automatically the same.

Consider: “Which customers should I contact this week?”

What makes a good answer?

Is it the customers with the largest revenue declines?
The customers most likely to leave?
The customers with unresolved service problems?
The customers who haven’t been contacted recently?
The customers assigned to Jeannine?

Maybe some combination.

And perhaps the answer needs to explain why each customer appears on the list.

Perhaps it needs to distinguish confirmed facts from predictions.

Perhaps it must avoid presenting a risk score as a certainty.

Perhaps it should ask a clarifying question before producing a list.

Those are evaluation criteria.

And if we haven’t defined them, what exactly are we testing?

Sometimes the Best Answer Is a Question

This is one of my favorite complications.

We often assume a good AI system should answer immediately.

But consider Jeannine asking: “Show me our best customers.”

Best according to what?

Revenue?
Profitability?
Payment history?
Growth?
Customer lifetime value?
Number of purchases?
And over what period?

If the AI immediately produces a list of the ten largest customers by revenue, it might look impressive.

It might even be factually accurate.

But was that the right answer?

Not necessarily.

A better response might be: “Do you mean highest revenue, highest profitability, or strongest growth?”

That’s not the AI failing to answer.

That’s the AI recognizing that the question needs clarification.

But if our evaluation system rewards only complete answers, we may actually penalize the better behavior.

Sometimes a good answer is knowing not to answer yet.

We Need More Than an Answer Key

Traditional software testing is often wonderfully straightforward.

Input A.

Expected Output B.

Compare.

Pass or fail.

AI isn’t always that cooperative.

Ask an AI system to calculate total sales for a specific period, and we may have an exact expected answer.

Great.

Test it.

But ask, “Summarize the major reasons customer satisfaction declined last quarter.”

There may be several acceptable summaries.

Different wording.

Different ordering.

Different levels of detail.

All potentially valid.

So instead of testing only for exact answers, we may need to evaluate characteristics.

Did the answer use the appropriate reporting period?
Did it distinguish evidence from interpretation?
Did it include the major contributing factors?
Did it omit important exceptions?
Did it avoid inventing explanations?
Did it acknowledge uncertainty where the evidence was incomplete?
Did it answer the actual business question?

That’s a different kind of testing.

And it requires more thought than checking whether the model returned a response.

The Sherpa’s Notebook

I’ve spent a lot of time working with data systems where two reports showed different numbers.

The immediate assumption was usually that one report was broken.

Sometimes it was.

But often, both reports were doing exactly what they had been designed to do.

One included inactive customers.

The other excluded them.

One used invoice date.

The other used posting date.

One included returns.

The other didn’t.

One treated an account as active if it had purchased something in the past twelve months.

Another used twenty-four months.

The SQL was working.

The reports were working.

The definitions were different.

Now imagine putting AI in front of those same systems.

Someone asks, “How many active customers do we have?”

The AI returns a number.

Which definition did it use?

Does the user know?

Does the AI know?

Does anyone know which definition was intended?

If we can’t answer those questions, checking the AI’s response against a single number may simply validate somebody’s assumption.

A test can pass and still be testing the wrong thing.

Build Tests Around Real Business Questions

Here’s what I’d rather see.

Before declaring an AI application production-ready, sit down with the people who will actually use it.

Not just the developers.

Not just the data architects.

The business users.

Ask them for real questions.

Questions they ask every week.
Questions that are difficult to answer.
Questions where people regularly disagree.
Questions where missing an exception would matter.
Questions where the correct response might be: “I don’t have enough information to answer that.”

Then work together to define what an acceptable response looks like.

For some questions, that means an exact answer.

For others, it means required facts, explanations, caveats, and boundaries.

For ambiguous questions, it may mean identifying the ambiguity.

For questions the AI shouldn’t answer, it may mean refusing or directing the user to an appropriate source.

This becomes a business-owned evaluation set.

Not merely a collection of prompts.

A collection of expectations.

Let the Business Grade the Exam

Here’s where ownership becomes important.

The AI team can build the evaluation framework.

They can automate tests.

They can measure consistency.

They can track changes between model versions.

But should the AI team decide what constitutes a correct customer-risk assessment?

Probably not.

That belongs to the business.

Sales needs to explain its criteria.

Finance needs to explain its criteria.

Customer Service needs to explain its criteria.

And someone with appropriate authority needs to resolve disagreements or explicitly approve multiple definitions for different purposes.

Otherwise, the AI team is being asked to settle business policy through prompt engineering.

That’s not architecture. That’s an organizational disagreement disguised as a technical requirement.

A Passing Score Can Hide a Serious Failure

Let’s say we develop 100 evaluation questions.

The AI passes 95.

Excellent!

95%!

Someone puts that number on the dashboard.

But what were the five failures?

Maybe the AI got five minor formatting questions wrong.

That’s one thing.

Maybe it incorrectly calculated a customer’s credit exposure.

Or confidently reported a canceled contract as active.

Or recommended an action based on information outside the intended reporting period.

That’s another thing entirely.

Not all failures are equal.

We need to understand which errors are tolerable and which ones are unacceptable.

A missed comma in a summary isn’t the same as a materially incorrect financial figure.

And an overall percentage can hide that distinction beautifully.

We’ve all seen averages make terrible things look reasonable.

AI isn’t going to change that.

Don’t Let AI Grade Its Own Homework Unsupervised

There’s another temptation.

Use AI to evaluate AI.

And to be fair, model-based evaluation can be useful.

It can compare responses against criteria.

Identify missing elements.

Flag unsupported statements.

Help review large numbers of answers.

But here’s the catch.

If we ask another model, “Was this a good answer?”

What definition of good is it using?
Does it have the business context?
Does it know the approved rules?
Can it verify the underlying facts?
Or is it simply deciding that the answer sounds reasonable?

An AI evaluator can help apply a rubric.

It shouldn’t be allowed to invent the rubric.

And it shouldn’t become the unquestioned authority on whether the business answer is correct.

Otherwise, we’ve created a fascinating arrangement where one AI confidently answers the question and another AI confidently congratulates it.

Two confident models do not automatically equal one correct answer.

What Happens When the Model Changes?

Now suppose we finally have a good evaluation set.

The business has approved the criteria.

The AI is performing well.

Then someone changes the model.

Maybe it’s a newer version.

Maybe the prompt changes.

Maybe retrieval is adjusted.

Maybe the available data sources change.

The application still works.

The interface looks identical.

But the answers may change.

Sometimes for the better.

Sometimes not.

This is why those business-owned evaluations matter.

We need to be able to ask, “Does the system still behave acceptably on the questions that matter?”

Not just, “Did the deployment succeed?”

And not just, “Did the new model score higher on a public benchmark?”

A model can perform better on a benchmark and worse on your organization’s particular business questions.

Your business doesn’t operate inside a benchmark.

It operates on Tuesday.

The Goal Isn’t One Perfect Answer

There’s a subtle point here.

We shouldn’t expect every business question to have one universally correct answer.

Some questions are ambiguous.

Some depend on context.

Some require judgment.

Some have several legitimate interpretations.

And some cannot be answered reliably with the available evidence.

That’s fine.

The goal isn’t to force every question into a single predetermined response.

The goal is to establish what acceptable behavior looks like.

When should the AI answer?

When should it clarify?

When should it explain uncertainty?

When should it cite evidence?

When should it decline to make a recommendation?

And who gets to decide?

A good AI system isn’t one that always has an answer. It’s one that behaves appropriately when the answer isn’t straightforward.

Stop Celebrating Green Without Asking What It Means

I’m not suggesting we throw away our operational dashboards.

We still need to know whether systems are available.

We still need performance metrics.

We still need error monitoring.

Those things matter.

But they answer different questions.

Availability tells us the system responded.

Latency tells us how quickly.

User satisfaction tells us how people felt about the interaction.

None of those independently proves that the response met the business’s expectations.

And even a passing evaluation score means little unless we understand what was evaluated.

The real question isn’t: “Is the dashboard green?”

It’s, “What did we agree green means?”

The Sherpa’s Lesson

Your AI can be fast.

Available.

Popular.

Well-grounded.

Beautifully formatted.

And still give an answer the business shouldn’t accept.

Before you declare success, decide what a good answer looks like.

Use real business questions.

Define the criteria.

Include ambiguity.

Include exceptions.

Include situations where the AI should ask for clarification or admit uncertainty.

Let the people who understand the business help determine what’s acceptable.

Then test against those expectations.

And keep testing as the system changes.

Because an AI application isn’t working perfectly just because the technology says it is.

It’s working when its behavior meets the expectations the business actually agreed upon.