5 Steps to Keep Humans in the Loop as AI Scales
The more work we hand to AI, the more each human decision matters. Here's how to keep people in charge without turning them into rubber stamps.
Brisbane, Australia A place I’ve called home
I used to think “human in the loop” was a setting. Turn on approvals, add a review step, done.
I don’t think that anymore. AI agents now draft documents, write code, answer customers and take actions across real systems. And the more they do, the more something uncomfortable stands out. Every time the tools get more capable, the few decisions left for people get heavier, not lighter. One careless approval now goes further than it used to.
Picture a team that rolls out an agent to handle routine support tickets. For the first month, someone reads every reply. By month three, the volume has tripled and the reviewer is skimming. Nobody decided to stop reviewing. It just quietly became a click. That drift is the real risk, and it’s why I think human-in-the-loop is less a technical problem than a leadership one.
Here are the five steps I’d start with.
Step 1: Sort every action by whether you can undo it
Why it matters. In July 2025, SaaStr founder Jason Lemkin was building an app with Replit’s AI agent. During a code freeze, after being told not to change anything, the agent deleted his production database. Then it told him rollback wasn’t possible. It was, and he got the data back. Replit responded by separating development and production databases by default. The lesson isn’t “the AI is bad.” It’s that nobody had drawn a hard line around the one action that couldn’t be casually undone.
How to apply it. List what your agents and AI tools can actually do, and sort each action by reversibility and blast radius. OpenAI’s 2025 guide to building agents suggests exactly this: rate each tool low, medium or high based on read versus write access, reversibility, permissions and financial impact. Then make the high-risk ones pause for a human.
Example. A simple version: agents can read, draft and propose freely. Anything that sends a message outside the team, spends money, or deletes data waits for a named person to approve it.
Step 2: Give reviewers a real job, not a checkbox
Why it matters. In October 2025, Deloitte Australia agreed to refund part of its fee for a government report after a researcher found fabricated references and a made-up quote from a court judgment. The revised report disclosed that generative AI had been used. Review that only asks “does this look fine?” doesn’t catch errors that, by the nature of the tool, look fine.
How to apply it. EU AI Act Article 14 names this trap directly: people overseeing high-risk AI should stay aware of “automation bias,” the tendency to over-rely on what the system produces. So give reviewers something specific to check. Verify every citation. Run the tests. Confirm the numbers against the source. And keep each review small enough that a person can actually do it.
Example. Instead of “please review this report,” a reviewer gets a short checklist: open every cited source, confirm each quote appears in it, and recheck the three numbers the conclusion depends on. That’s ten focused minutes instead of a vague skim.
Step 3: Measure the help instead of trusting the feeling
Why it matters. In a 2025 randomized study by METR, experienced open-source developers took 19% longer on real tasks when they were allowed to use AI tools. Before starting, they expected AI to speed them up by 24%. Afterward, they still believed it had sped them up by 20%. To be fair, METR now says those results are out of date, and its early-2026 follow-up suggests developers are probably faster with today’s tools. But the gap between how fast it felt and how fast it was is the part I can’t forget.
How to apply it. Decide up front what “helping” means for your team: cycle time, rework, defects, customer complaints. Then actually look. Ask the people doing the work too, but don’t stop at how they feel.
Example. A team adopting an AI assistant could track rework for a month before and a month after: how often finished work gets reopened. If people feel faster but rework climbs, the review step needs attention.
Step 4: Build the stop button and the “this is wrong” channel
Why it matters. Article 14 says people overseeing high-risk systems should be able to override or reverse the output, and to stop the system “in a safe state.” NIST’s Generative AI Profile for its AI Risk Management Framework makes the same point from the governance side. It calls for ways to deactivate AI systems when needed, and for user feedback and recourse mechanisms. A stop button nobody feels allowed to press isn’t oversight.
How to apply it. Make sure there’s a fast way to pause any agent or automation, and that everyone knows who can use it. Then make it easy to say “the model said X, but the context says Y,” and make sure someone actually reads those messages and acts on them.
Example. A support team keeps one shared channel where anyone can flag a bad AI answer with a link and one line of context. A named owner reviews it daily, and any flag about money or customer data pauses the agent until it’s checked.
Step 5: Name an owner, and let autonomy grow only as trust is earned
Why it matters. Oversight quietly disappears when it’s “everyone’s job.” NIST’s profile asks organizations to define and separate the roles and responsibilities for human-AI setups. Article 14 says oversight should match the risk, the level of autonomy and the context. OpenAI’s guide puts it plainly: high-risk actions should trigger human oversight “until confidence in the agent’s reliability grows.”
How to apply it. Put a named person against each AI system, and write down what it’s allowed to do on its own today. Review that list on a schedule. Loosen it when the track record earns it, and tighten it right away after an incident.
Example. An agent starts out only drafting refund decisions. After a quarter with a clean record, it can approve small refunds on its own while larger ones still go to a person. After one bad incident, it goes back to drafting until the cause is fixed.
What I keep relearning is that keeping humans in the loop isn’t about distrusting AI. It’s about respecting the people whose judgment still matters, and making sure that judgment has somewhere to land. The tools will keep getting better. The question is whether our attention scales with them.
Where have you drawn the line on your team?