My tool reported success. The AI assistant said the job was done. The result was still wrong.
In Part 2, I shared how an AI assistant used one of my customer support tools and left out the customer's email address, and how my tool quietly filled in the bot's own address instead. Every reply went straight to the bot, while the real customer sat in the dark.
What bothered me most wasn't the mistake itself. What bothered me was why nobody caught it: every signal I had said everything was fine.
Nobody lied. Nothing crashed. The tool said "201 Created," the ticket existed, and the AI said done. All three proved that a ticket was created. None of them asked whether a customer could actually get a reply.
I didn't mean to stop checking. The green 201 and the AI's confident "done" were convenient, and I took them as the final answer. Every signal in that chain was written for a machine. Not one was written for me.
Two completely different sentences
I call this the trust gap.
"The checklist is complete" and "the right thing happened" are two very different sentences. When a machine checks another machine, it is brilliant at measuring the first sentence and completely blind to the second. The result is flawless execution of the wrong goal.
We fall into the trust gap because we confuse activity with outcomes:
- Activity: The form was submitted, the database was updated, the code compiled, and the status light turned green.
- Outcome: The customer got their answer, the refund arrived in the right bank account, or the patient received their schedule.
When humans ran every step of a process, we naturally connected activity to outcome. If a form went through with a robot's email address on it, a person would notice that it looked ridiculous. Automation strips away that context while keeping all the green checkmarks.
From a green light to a tripwire
Here is the check I wish I'd had. I call it a semantic assertion, which is a fancy name for a simple idea. Don't ask whether the system said "201 Created." Ask whether the customer can actually reply. Don't ask whether the report ran. Ask whether the ledger balances.
Does that cost more? Yes, if you run it on every tiny step, so don't. Put it on the two or three paths where being wrong hurts a real person, and treat it as a tripwire. For my ticket bug, one question would have done it: does this ticket carry a real customer's email? That would have caught it on day one.
Try the Green Light Experiment
As we hand more daily operations over to AI assistants, our dashboards are going to look greener and happier than ever. The machines will proudly report that 100% of their tasks succeeded. Our job is to make sure those green lights actually measure reality.
If you are using AI or automation in your workflow, try this experiment:
- Pick one report or dashboard that flashes a green "Success" light.
- Ask yourself what a successful outcome actually looks like for a real human being on the other end.
- Go check that real-world outcome, instead of the report.
I bet you will find a green light hiding a quiet mistake. What is one "Success" metric on your dashboard right now that only proves a step ran, not that a customer was helped? Tell me about it on LinkedIn.
Cheers,
Markus
No bug 🐛 too small, no syntax too weird 🔍 🧐 🔨 🦾