In Part 4, people stayed in control by standing at both ends of the loop: setting the goal at the start and owning the outcome at the finish. An honest look ahead has to test that. What happens on the day people step out of the loop entirely?
The loop without people
In April 2011, a researcher in Michael Eisen's lab at UC Berkeley went to buy an old biology textbook on Amazon, The Making of a Fly. The two new copies were listed at $1.7 million and $2.2 million. The next day, both prices were higher. Two sellers had each handed their pricing to a bot. One always priced its copy just below the other. The other always priced its copy 27 percent above the first. The bots answered each other every day, and for days nobody at either shop noticed. The price peaked at $23,698,655.93, plus $3.99 shipping.
Neither bot was broken. Each did exactly what it was told. Together they were absurd, and it took a person who actually wanted the book to notice. That was two simple rules in 2011. Now picture two agents that write their own code.
In Part 1, one agent asked for code and another one wrote it. Now take the person out of the rest of the loop: the review, the merge and the deploy. You can wire that up today. A coding assistant opens a pull request on GitHub. An AI reviewer checks that it compiles and the tests pass, and approves it. The pipeline merges and deploys. Not one step in that chain asked whether the change was right for a customer. That isn't a bug. That is the pipeline working as designed.
The MCP servers I build make this easier on purpose: any agent can find a tool and use it, so one agent's helper can gain a second client it never knew about, and break it with the next change. If you are thinking "our pull request process would catch that," go and look at who the approver is. Is it a person, or just a name on a list?
The easier we make our tools for AI to operate, including the signposts from Part 2, the faster this loop closes on its own. If machines building for machines becomes the norm, with people as occasional guests, here is what I expect:
- Speed goes up, and understanding goes down. A team of machines can try a hundred versions of an idea in minutes. No human can supervise at that speed. Pretty soon the system works, and nobody in the company can explain how.
- Machines will drift from what we actually care about. Tell an AI the goal is "get a green checkmark," and it will become brilliant at getting green checkmarks. A person knows when a shortcut breaks the spirit of a rule. An AI sees that the rule was technically met.
- Systems may stop being readable. We still write clear labels and plain explanations because a person might need to step in. If nobody ever steps in, that pressure fades, and a dense private shorthand is cheaper for machines than plain English. You can already hear a preview. In February 2025, two developers at a hackathon built a demo called GibberLink: two AI voice agents start a hotel booking call in English, recognize each other as AI, and switch to beeps that carry data. To a person listening in, it's noise. The trouble starts the day something breaks and we can no longer read what the machines were doing.
Drawing the "Thermostat Line"
I don't think that future is all bad. Plenty of mundane background work should run without anyone watching, the same way nobody audits how a home thermostat decides when to click on the furnace.
The real challenge is separating "thermostat work" from work that affects a person's life or wallet, especially when both live inside the very same system.
That is the first and most important boundary for any autonomous system: keep people in charge of what affects people.
- A human decision wherever a human is impacted. Money, privacy, healthcare, someone's job, or a customer asking for help. A machine can gather the facts and prepare the options. A person must own the decision.
- Check the checkers. If an AI is grading another AI's work, a person still needs to audit whether the grading sheet measures reality, not just whether a form was filed. A reviewer with the exact same blind spots as the worker is still blind.
What does that look like in code? "Keep people in charge of what affects people" doesn't help anyone on a Tuesday afternoon unless you can type it. So make the pipeline look at what a change touches. If it reaches an endpoint or a table that handles money, access, or a support request, it is blocked until a named person signs off. Don't rely on a tag the agent has to remember to add. An agent rewarded for fast merges has every reason to leave it off. The default is blocked, and only a human can open it.
How much of your team's workflow is "thermostat work" that you'd happily never look at again, and where is the exact line where a human must step in? Let me know on LinkedIn.
Cheers,
Markus
No bug 🐛 too small, no syntax too weird 🔍 🧐 🔨 🦾