Human in the Loop Is Not Enough: Who Is Governing the Loop?

Human in the Loop Is Not Enough: Who Is Governing the Loop?

Human-in-the-loop is necessary—but in GMP environments, it may not be enough. Effective AI governance also requires humans above the loop: defining the boundaries, knowledge, controls, and conditions under which AI is allowed to support the work.

Human in the Loop Is a Good Start

“Keep a human in the loop.”

It has become one of the most common responses to concerns about Artificial Intelligence. On the surface, it makes sense. AI supports the work, a person reviews the output, and the person remains responsible for the decision.

I agree with that approach.

In Can You Write the Ship?, we raised a related concern: it is not enough for AI to produce a good answer if the person reviewing that answer no longer understands how to build, challenge, and defend the work themselves. Human judgment still matters.

But there is another question we need to ask.

Who is governing the loop?

Putting a person at the end of an AI-assisted process does not automatically mean the process itself is controlled.

The Human in the Loop Owns the Work

Think about a deviation investigation.

AI can help organize information, identify questions, improve structure, and help the investigator think through possible paths. But it should not independently decide the root cause or determine whether the available evidence is sufficient.

The investigator still has to understand the facts, challenge the logic, decide whether the conclusion is supported, and determine what moves forward.

The same principle applies to technical writing, batch record review, engineering documents, validation work, or other activities where AI may support GMP or Good Engineering Practice (GEP) work.

That is the human in the loop.

In Artificial Intelligence in GxP Environments, we described AI as decision support rather than decision authority. I still think that distinction is important. The value of AI is in improving how work and decisions are prepared—not transferring ownership of those decisions to the technology.

But Who Sets the Boundaries?

This is where I think the conversation needs to go one level higher.

Before a user ever interacts with an AI system, someone should have already decided what the system is allowed to do.

What information can it use? What information should it not assume? What happens when information is missing? When should the AI stop and ask a question instead of filling in the blank? What decisions must remain human decisions? What requires review or explicit acceptance?

Those are not decisions made by the human in the loop.

They are decisions made above the loop.

That is where governance sits.

The human in the loop owns the work and the decision.

The human above the loop owns the conditions under which AI is allowed to participate in that work.

Both matter.

An Approval Button Is Not Governance

One of the traps I think organizations need to avoid is believing that human oversight exists simply because somebody eventually clicks Approve.

Approval can certainly be part of governance, but it is not governance by itself.

If AI was allowed to fill in missing information, operate from the wrong source, move past unresolved questions, or make conclusions that should have remained with the user, the fact that somebody reviewed the finished output does not fix the problem.

The real question is whether the person was actually exercising judgment or simply reviewing something that already looked complete.

We have made a similar point about digital transformation in Digital Doesn’t Fix Broken Execution. Technology does not create discipline on its own. In many cases, it simply amplifies the system and behaviors already around it.

AI is no different.

GMP Environments Need Both

For regulated work, I believe we need both levels of human control.

The human in the loop needs to remain capable of understanding, challenging, changing, and ultimately owning the work.

The human above the loop needs to define how the AI is permitted to operate in the first place.

Without the first, we risk replacing human judgment.

Without the second, we risk putting good people inside a poorly governed system.

Neither is a particularly good outcome.

This becomes even more important as AI moves beyond simple drafting and begins supporting deviation investigations, batch record review, technical writing, Good Engineering Practice activities, validation, qualification, and other structured work.

The more capable the technology becomes, the more important it becomes to clearly define where its authority ends.

What This Looks Like in Practice

Consider that deviation investigation again.

The investigator may provide the known facts. AI may help organize those facts, identify gaps, ask questions, or help structure the investigation.

That is useful.

But what should happen when information is missing?

Should the AI make a reasonable assumption and keep going?

Should it create a root cause because one appears likely?

Should it fill in language because the user probably meant something?

In a regulated environment, those questions cannot be left entirely to the model. Someone has to establish the rules before the interaction begins. Do not invent missing facts. Do not quietly turn assumptions into evidence. Do not make consequential decisions for the investigator.

Require human review and acceptance where judgment matters. Those are examples of humans operating above the loop.

The investigator then works within those boundaries and remains responsible for the actual decisions being made inside the work.

That is the human in the loop.

Where GMPWit Fits

This distinction has directly influenced how we built GMPWit.

We are not simply putting a human approval step at the end of an AI workflow and calling that governance.

GMPWit is designed so users remain responsible for judgment, review, and acceptance. At the same time, the system is intended to operate within defined boundaries around knowledge, workflow, missing information, and decision authority.

For example, if required information is missing, the better answer is not necessarily for AI to generate something plausible. Sometimes the right answer is to stop and ask for what is missing.

If a decision belongs to the user, the system should support that decision—not quietly make it for them.

If content is being proposed, the user should know what is being proposed and explicitly decide whether to accept it.

That is what human-in-the-loop and human-above-the-loop begins to look like in practice.

Our GMPWit whitepaper describes GMPWit as a structured decision-support and work-preparation layer rather than a system of record or GMP decision authority. That distinction is intentional.

The goal is not to remove the human from GMP work.

The goal is to help the human do the work better while keeping ownership where it belongs.

AI Should Strengthen Judgment, Not Quietly Replace It

The discussion around AI in regulated environments is going to continue evolving.

The technology certainly will.

But I think the principle is fairly simple.

If AI is going to participate in GMP work, we should know what role it is allowed to play. We should know where human judgment is required. We should know what happens when information is incomplete. And we should know who is responsible for setting those rules.

As organizations adopt more AI, asking “Do we have a human in the loop?” is still important.

I just do not think it should be the last question.

We should also ask:

Who is above the loop?

Who defined the boundaries?

Who decided what the AI is allowed to do?

Who remains accountable when those boundaries need to change?

Human participation and human governance are not the same thing.

In GMP environments, we are going to need both.

Keep the human in the loop. But make sure somebody is still governing it.

Learn more about GMPWit: GMPWit was built around a simple idea—AI should strengthen human judgment, not quietly replace it. Visit the GMPWit page to learn more about how we are applying that principle to GMP learning and structured work.