In Part 3, every check passed and the customer still got nothing. That raises an awkward question: where was the person who should have noticed?
When humans built and ran all our software, we had an invisible safety net. Someone, somewhere, would look at a screen, frown, and say, "Wait, that doesn't look right." Nobody had to engineer that net. It existed because a person was always in the room.
As more of the daily work fills with machines, that net quietly disappears.
Three machines in a trench coat
Today, AI often plays all three roles in a project: it writes the code, it checks the code, and it is the client that runs it.
- The AI that writes it tells me it's ready, even if it would confuse a real person the moment they touched it.
- The AI that checks it says all the quality checks passed.
- The AI client that runs it tells me the job is finished, even when the real-world outcome is a mess.
Each machine is confident, each one trusts the others, and nobody is actually looking.
I felt a sharp version of this recently. I had a screen with every icon in its proper place, and I asked an AI assistant to add one more tab. The tab appeared. Every icon on the rest of the screen had quietly vanished. A check that asked "is the new tab there?" would have passed with a green light. I only caught it because I happened to open the page myself.
Why machines have tunnel vision
Let me be fair to the machines first. They are much better than I am at the repetitive, exhausting parts of the job. They don't get tired on a Friday afternoon. They read every line they are handed, and they keep going long after a human's eyes would glaze over.
Still, they have tunnel vision. They only see what fits inside the instructions we give them. If a real-world problem isn't mentioned there, it doesn't exist for them. I am happy to hand over the brute-force checking, but there is one thing a machine cannot tell me: whether the box it ticked was the right box to care about.
Where the human job moves
The answer isn't to put a person back into every tiny step. That defeats the point of automation, and nobody misses reviewing code line by line at two in the morning.
Instead, the human job moves to the two ends of the process, and to setting tripwires in the middle.
At the start, intent
Somebody has to decide what is worth doing and what "success" looks like for a real person. A machine cannot define that for itself. When one agent orders work from another, "the system asked for it" is not a reason. A person still has to own the goal, and the bill.
In the middle, tripwires that outlive the moment
Every time I catch an automated mistake, I try to turn the lesson into a safeguard that keeps working when I'm not watching. After the ticket bug, the lesson was to check that the customer can actually get a reply, not just that a ticket exists. After the missing icons, it was to check the whole screen, not just the part I asked to change.
Some work has no outcome you can check right away, like an agent tidying data in the background for hours. For that I'd use a confidence budget that runs out unless a person looks. I'll show how it works in Part 7.
At the end, accountability
Somebody has to own the final outcome and decide who gets access, who can change what, and who is allowed to say "ship it." Agents can do the legwork, but people still have to answer when a silent shortcut lets a customer down. "The AI said it was done" isn't an excuse any of us can give.
Writing the code is getting cheap. Knowing whether it is right is becoming the premium skill. A few years from now, I expect the most valuable person on a team won't be the one who ships the most code. It will be the one who can tell when it's wrong.
Have you caught an AI passing its own test while quietly breaking everything right next to it? Share your best story with me on LinkedIn.
Cheers,
Markus
No bug 🐛 too small, no syntax too weird 🔍 🧐 🔨 🦾