A Checkmark Isn’t a Check
Plain-English premise: An approval record proves someone was at the gate. It doesn’t show what they checked, or whether they could have stopped the work.
Judgment Seat: The approver whose name is on the record, whether or not the gate gave them enough to judge.
Episode 1:
👋 Last week’s closing promised an episode called The Button That Used to Be a Person. The research changed my mind. The person didn’t disappear. They clicked. Episode One budgeted the reviewer’s attention. Episode Two asked what the sender owes before review begins. This week is about the moment in between: the approval itself, and what it can and can’t tell anyone who reads it later.
Episode 2:
Research Binder: the receipts (citations + source notes) are compiled in a PDF at the bottom of this post.
⚡ Pressure
AI can expand what arrives for approval, and the click that clears it records presence, not judgment.
🎯 Payoff
This issue gives you the Gate Honesty Check: five questions that test whether an approval gate can support the judgment its record implies.
Pressure Moment
In mid-September, a software engineer posted a short after-action note. He had approved a pull request an AI agent wrote for a system he designed. Tests passed. The diff looked reasonable. He merged it without fully tracing one change. Two days later, a background job quietly started double-processing records. It’s one person’s public account, from software, and it says nothing about how often this happens. What it shows is narrower and more useful. He did check things. The merge record would look identical whether he traced that change or not.
The log recorded a human. It didn’t record the check.
I’ve inherited plenty of “approved” decisions in enterprise environments where the approval was real, but the judgment behind it wasn’t recoverable. We could prove who signed and what moved forward. Reconstructing what they had actually checked sometimes required emails, meeting notes, and conversations with people who were still around. The problem wasn’t necessarily the review. It was that the record couldn’t carry the review forward.
None of this means the approval was empty. Sometimes a green checkmark is the compact end of an excellent review. Sometimes the review was thinner. Sometimes we simply can’t tell, and not knowing is different from knowing that no judgment occurred. An incomplete record is not proof of an empty mind. The useful question is narrower: what evidence justifies treating this particular approval as meaningful oversight?
Start by separating the three questions a single click tends to collapse into one.
Was a human present? Did meaningful judgment occur? Did it improve the outcome? The log can answer the first: who clicked, when, and what state the system recorded. The other two need more evidence: what the person inspected, what evidence they used, and what would have made them refuse.
The one number on that graphic shows why the design around a decision matters. In a 199-participant experiment (Buçinca, Malaya & Gajos, 2021), people using simple explainable AI accepted incorrect advice on one wrong-advice subdecision about 64 percent of the time, versus 48 percent under cognitive-forcing designs. That’s a conditional result, not a 16-point cut in overall failure. It still shows that how a decision is set up changes what people do with bad advice. A human’s presence alone doesn’t guarantee that.
What the Record Can Prove
Organizations like signatures because one field compresses hours of work into “Approved by Taylor Kim.” That compression is useful. It also loses a lot.
A meta-analysis of 106 experiments and 370 effect sizes (Vaccaro, Almaatouq & Malone, 2024) found that human-AI combinations beat humans working alone on average, yet underperformed the better of the two working alone, with results varying widely by task. That finding supports neither “humans make AI better” nor “get humans out of the way.” Performance depends on the task, the system, and how the collaboration is designed. So read the record for what it can support: a sign-off shows presence, a review trail offers observability, and only outcome data speaks to effect.
The gap between a record and what actually happened isn’t new, and it isn’t an AI problem. There’s useful evidence from a setting with no AI at all.
In one hospital audit (Salgado, Barber & Danic, Journal of Patient Safety), independent audio review of 100 surgical cases found lower checklist fidelity than the paper completion records indicated. That doesn’t mean the clinicians were lying, or that no safety work occurred. The audit compared spoken checklist items with paper records, not anyone’s private judgment. Missing detail doesn’t prove missing judgment. It proves the record can’t settle the question on its own.
The answer isn’t more paperwork or surveillance. It’s deciding what each record is for. Approval status tells you who acted. Review evidence tells you what was inspected. Outcome data tells you what happened next. One checkmark can’t do all three jobs.
Watching Isn’t Stopping
Now imagine the reviewer spots a problem. What happens next? Darren Kimura, a technology executive quoted in CIO this month, described many companies that claim a human in the loop as actually having “a human watching the loop”: someone who can flag a concern but can’t stop, change, reject, or escalate it. He’s a vendor voice, so treat the line as a sharp distinction, not a measurement.
The NIST AI Risk Management Framework tells organizations to assign mechanisms to supersede, disengage, or deactivate AI systems that perform inconsistently with their intended use. That’s a governance principle, not proof of outcome benefit. The closest outcome evidence comes from hospitals. A Cochrane review of escalation systems (McGaughey et al.) covered 4 randomized trials with 455,226 participants plus 7 nonrandomized studies. It couldn’t isolate authority as the active ingredient, because pathway design, recognition, communication, response, and implementation were bundled together.
That doesn’t mean authority is irrelevant. It means we should be precise about the pathway. Can the reviewer detect the problem? Can they act on it? Will the system honor the action? An approver without a usable pathway may be an observer with a better title.
Who Decided the Depth?
Episode One dealt with review capacity. This section asks a different question: when more work arrives, who decides how deep each review goes?
The forty concepts and the dozens of daily pull requests are illustrations, not prevalence estimates. We don’t have evidence that faster AI output broadly causes worse review. We do have a systematic review of 40 studies across 17 task types (Lyell & Coiera) that found automation-related omission and commission errors across different settings. Its methods were too varied to produce a product-team error rate. If the arrival rate changes while the review design stays fixed, something still decides which work gets scrutiny. It might be consequence, reversibility, or uncertainty. Or it might be whatever arrived while the reviewer still had patience. Only the first three are decisions. Don’t let arrival volume silently decide review depth.
Easy to Read Isn’t Easy to Verify
AI is very good at producing something humans are trained to reward: a coherent explanation.
Across controlled studies with 1,626 online participants (Bansal et al., 2021), AI explanations increased acceptance of both correct and incorrect advice, without significantly improving team accuracy over showing confidence alone. In maze experiments reporting 731 participants (Vasconcelos et al., 2023), making the proposed answer easier to check helped people reject wrong advice in some tested conditions, but not all. And the experiment behind the 64-to-48 figure showed no overall performance advantage for the forcing designs.
The goal isn’t more friction. It’s better verification. Show the claim, expose something the reviewer can inspect independently, and make the failure point checkable. That’s the same reason Episode Two’s Handoff Brief asked for an inspectable verification anchor.
Not Every Output Deserves the Same Gate
Uniformity feels safe. If one review is good, the same review for everything must be safer. That doesn’t follow either.
In the MASAI randomized trial (105,915 women analyzed), AI-supported selective single or double reading used 44.2 percent fewer screen readings than double reading everything. Sensitivity was 80.5 percent in the AI-supported arm and 73.8 percent with standard double reading, with specificity at 98.5 percent in both. Every examination still retained a human reader, and total labor savings weren’t established. That’s interesting, but it isn’t permission to copy a mammography workflow into Product Operations.
Routing can also miss. In a different paired mammography study (Elías-Cabot et al., 2026), 11 low-score cancers were auto-labeled normal by the AI strategy. Parallel standard reading found them, so they were policy-level routing misses rather than patients the trial left untreated. Selective review needs visible bypass risk. The Light, Standard, and Deep tiers are Episode One’s Review Budget applied at the gate: a design inference, not a clinical result.
Design review depth. Don’t default to it.
🧭 The Judgment Move
Orienting sentence: The Gate Honesty Check asks five questions about the gate, not the person standing at it. It’s a diagnostic, not a certificate, and it isn’t a lie detector for employees.
Question 1: Can the reviewer act? Is there a direct or delegated path to stop, change, defer, or escalate before the action moves forward? A routed escalation that reliably holds the work counts. Basis: institutional support (NIST); no study shows one form of authority outperforms another.
Question 2: Do they have enough to judge? Do they have the task-relevant time, context, and evidence to inspect the work? Basis: design inference, informed by evidence that verification difficulty is implicated in automation-related errors.
Question 3: What did they personally check? Name the specific artifact, constraint, evidence, or failure point inspected. Not what the AI summarized, and not what the workflow claims was reviewed. “I was there” isn’t an answer. Basis: design inference; a named check makes a gap visible but doesn’t prove the check happened.
Question 4: What would trigger refusal? Define the failure, change, or uncertainty that would require a stop, revision, deferral, or escalation. Basis: design inference.
Question 5: Will the intervention be enforced? What evidence shows that an appropriate stop, change, deferral, or escalation would actually be honored? Basis: design inference; escalation research shows pathways are bundled, not that any one element works.
There is deliberately no question asking whether the gate has ever said no. A refusal quota would turn the check into performance theater. A reviewer can approve ten excellent artifacts in a row and still be exercising excellent judgment, and can reject something for terrible reasons. The action is not the judgment.
Done state: the gate’s weakest answer is named and routed to someone who can fix it. Five completed fields is not the goal.
Constraint: the questions test the design of the gate, not the character of the reviewer. Once they’re used to score people, they stop producing honest answers.
I’ve seen gates where the reviewer did exactly what the process asked: they approved the work with explicit conditions attached. The judgment wasn’t the weak point. The weakness was downstream. Nothing in the workflow actually held the next step until those conditions were satisfied, so “approved with conditions” slowly became indistinguishable from “approved.” That taught me to ask a different question: not whether someone can raise a concern, but whether the system is built to honor it.
Limitation: the check doesn’t prove judgment. It helps show where judgment could break. It can’t verify that a stated check happened, and no question has been validated as reducing errors.
Run it on one real gate this week.
🧪 What Holds Up
Primary claim label: Evidence for the separable pieces; Design inference for the Gate Honesty Check as a bundle.
Basis: Records and observed execution can diverge (Salgado et al.). Automation-related errors appear across settings (Lyell & Coiera). Interaction design can change how people handle wrong advice (Buçinca et al.). Explanations alone don’t reliably improve accuracy (Bansal et al.).
Counterpoint, labeled Evidence: An approval can be the honest output of a good review. Human-AI teams outperformed humans alone on average (Vaccaro et al.), and selective review with a retained human reader performed well in MASAI. Enforcement has costs too: in one hospital trial, a hard-stop ordering restriction was stopped early after treatment delays.
Scope and transfer limit: this evidence comes from clinical screening, lab experiments, hospital escalation systems, and software practitioners. No study here directly tested AI approval gates in Product, Program, Design, or Research teams.
What remains unknown: whether the Gate Honesty Check changes gate behavior, catches more errors, or improves outcomes. It hasn’t been tested as a bundle.
The green checkmark isn’t the villain. Pretending it answers questions it was never designed to answer is.
🛑 Do Not Use When
You want to score, rank, or monitor individual reviewers. The questions test gates, not people.
The work is low-consequence and reversible, and a reliable automated check already covers it. Route it light instead of adding ceremony.
You need a compliance certificate. Neither NIST nor the EU AI Act validates this method.
Judgment Seat
I’ve been in roles where my name was attached to an approval even though I didn’t control every condition that made the work safe to move forward. In the moment, everyone understood where the dependencies and decision rights actually lived. Months later, the record remembered only the name and the word “approved.” That taught me that accountability and authority can travel separately, even when the system makes them look like the same thing.
Who generated the work: the submitter and whatever tool produced something ready to approve.
Who must understand it: the approver, to the depth the gate actually allows.
Who can stop it: whoever holds a usable pathway. If nobody does, the gate is observation.
Who pays if it is wrong: whoever carries the consequence. The approval record names someone, but it doesn’t decide that.
🎁 Your Gate Honesty Check
Gate Honesty Check PDF
The Gate Honesty Check helps us test whether an approval gate can support the judgment we expect from it. Five simple questions look at whether the reviewer can act, has enough information to judge, can show what was checked, knows what would trigger a refusal, and has a system that will honor a stop or escalation. There are no scores, rankings, or certificates. The goal is to find the weakest part of the gate, give it an owner, and decide what needs to change next.
Gate Honesty Check Workbook
The Gate Honesty Check Workbook turns the five questions into a working tool you can use on a real approval gate. It helps you name the gate, record what the approval is supposed to mean, test whether the reviewer can act, and see what the record does and does not prove. At the end, you name the weakest answer, choose who owns the fix, set the next action, and pick a date to check it again.
This Week
Three actions for the judgment owner: Pick one gate you approve through and answer Questions 1 and 5 in writing. On your next approval, write one line naming what you personally checked. Before your next click, name one thing that would have made you refuse.
One action for the system owner: Confirm in writing whether a flag at your highest-stakes gate can actually stop or defer the work, and who is responsible for honoring it.
One action for learning and provenance: For one gate, compare what the approval record captures with what it doesn’t show. Add one field that makes the check visible to whoever reads it next.
📎 Forward This
A green checkmark can record participation without proving scrutiny. Before trusting it, ask what the person inspected, what would have made them refuse, and whether they could actually stop the work.
🔚 Closing
The approval moved at the speed of a click.
Whether anyone judged the work still has to be shown, not assumed.
Next week, Episode Four looks at what leaders reach for when oversight is hard to see. The Scoreboard Is Lying asks what AI usage metrics can and can’t tell you about the work.
Because when a gate is hard to trust, the temptation is to count something instead.
This question is part of the larger challenge behind my upcoming book, Collaborate Better: From Silos to Synergy – How to Build Unstoppable Teams. Better collaboration begins when context, ownership, and judgment travel with the work.
Learn more at CollaborateBetter.us.
Regards,
Mark 👋
Previous:
More Content to Discover:
Half Your Team Is Using AI. You Don't Know Which Half.
Intro If you read the piece we published last week on Leadership in Change, you saw the structural diagnosis: mandate pressure, decision collapse, intake collapse. Three failures that sit underneath almost every AI pilot that goes sideways. That piece mapped the architecture. What it did not cover is what happens on the human side of the same problem, wh…
The most dangerous approval may be the one completed by a thoughtful reviewer. If the system keeps only the green checkmark, their judgment can disappear the moment the work moves on. Think of one gate you rely on: what does its approval record actually preserve, and what would you add so the next person can tell what was really checked?