The circle on the page

👋 Welcome to my paid subscriber-only edition of Empathy Engine (🔒 Leader’s Dispatch). Each week I build evidence-forward tools for product leads who need to say no, defend tradeoffs, and lock in decisions before they get rewritten later.

Maya Chen circled the line before she finished reading the page. (Maya and Dev are fictional, built to illustrate this series.)

It was Wednesday’s forecast draft, the one she had promised to go through with Dev. Three customer segments. Two had their source definitions confirmed, each logged in the model’s assumptions tab. The third, a new enterprise segment, had nothing. No confirmation. No note saying one was coming.

Maya had asked for the definitions check in a one-on-one. She had rewritten the ask so it covered every draft that leaves the team, internal stakeholders included. Dev had replied the same afternoon: understood, every draft that goes out. She had stopped repeating it on Fridays and named this draft as the next review point. The feedback was clear. It was specific. By any reasonable standard, it was well timed.

And there was the circle.

She opened her notes and typed: Agreed three times. Still not done. Then she sat with the cursor blinking after the last word, because the next word she wanted to type was accountability, and the sentence after that was going to be about Dev.

Then the other reflex arrived, the one last week had trained. The source-file queue. Dev had told her it was the hard part. Maybe the file never came. Maybe this one wasn’t on him at all.

I’ve written that note. The feedback was clear, the person said yes, the next piece of work came back looking the same, and my notes jumped straight from what I saw to why it happened. Sometimes the why I wrote down was right. I couldn’t have told you which times.

Research Binder: the receipts, methodology notes, and source boundaries are compiled at the bottom of this post.

Previous:

Last week I promised to look at why a clear, well-timed message can still go nowhere when the person receiving it can’t act on it. The research narrowed that promise, and I think it made it more useful. It doesn’t support “find the blocker.” It doesn’t support “find the excuse” either. It supports something less satisfying and more honest: before deciding why the work didn’t change, find out what the work can actually tell you.

Look at the three boxes in the middle. Feedback communicated. Acknowledgment expressed. Subsequent work observed. Each is a separate event, and each gives you a different kind of evidence. The first is what you said. The second is what they said back. Only the third is the work.

Maya’s circle sat on the third box. Her note had already filled the space between the boxes with an explanation. So had the queue theory, pointing the other way. That’s the banner across the middle: a result isn’t an explanation. It tells you what happened. It doesn’t tell you why.

The question at the bottom is where this episode goes. Before deciding why, ask what the next task actually showed. Was the expected behavior required, possible, observable, and relevant to this task? Maya hadn’t asked any of those four. By Friday, three of the answers had surprised her.

Feedback still isn’t the villain

In the 607 effect sizes from 131 papers Kluger and DeNisi reviewed (mostly laboratory studies published before 1996), feedback improved measured performance on average (d = 0.41), and over 38% of those effects were negative. About a third were still negative after the authors removed outlying data.

Two things follow, and Maya needs both. Feedback helps on average, so a disappointing result is no reason to stop giving it. And feedback doesn’t move the work in every case, so “I told him and he agreed” isn’t the end of the story.

That sets the boundary for this episode. Nothing below says employees are usually blocked, or usually unwilling. The research I found doesn’t say how often either explanation is the right one. It says they are different explanations, and that one result rarely chooses between them.

Said, intended, observed

Before her 2 p.m. review with Dev, Maya split her note into three lines.

• Said. In their one-on-one, Dev said the ask made sense. When she rewrote it, he replied: understood, every draft that goes out.

• Intended. He said he would run the check on every draft. She had no reason to think he didn’t mean it.

• Observed. One draft, sent to her for review, with two of three definitions confirmed.

Seen that way, her note had collapsed three different things into one.

The three boxes name what each line can hold. Acknowledgment is what someone said. Intention is what someone plans to do. Behavior is what the work may show. Notice the hedge in the third box: may. Even the work is evidence only when it was a fair place to look. We’ll come back to that, because it’s where Maya’s week turned.

I looked for a study testing whether an employee’s “understood,” after corrective feedback at work, predicts what the next piece of work looks like. I didn’t find one. So treat this as careful reasoning, not a research finding: acknowledgment alone isn’t evidence that the change happened. It isn’t evidence that it didn’t, either. It’s a different observation.

That’s the instruction at the bottom. Don’t treat the conversation as proof of the change. Maya had done exactly that, in reverse. Because Dev had agreed three times, one unconfirmed line felt like a broken promise instead of a data point.

It offers three better questions. What action was expected? Where could it be observed? What was actually observed?

Maya could answer the first: confirm the source definitions before any draft leaves the team. The circle answered the third. She skipped the second, where the change could be observed. It turned out to matter most.

A clear goal, a complicated task

Maya’s request passed every test Episode 2 set for it. It named the work, the standard and the next step. If clarity alone produced change, this request should have.

That assumption has a research line behind it, and the research is more careful than the assumption.

In 1987, Robert Wood, Anthony Mento and Edwin Locke published a meta-analysis of 125 goal-setting studies conducted between 1966 and 1985. Goal-setting effects were larger on less complex tasks, d = 0.76, than on more complex ones, d = 0.42. Both effects were positive. Both are standardized effect sizes, the same ruler from Episode 1, not percentages. A d of 0.76 does not mean 76% better.

Read the scope before the numbers. These are older studies, a mix of laboratory and workplace tasks, and none of them tested what happens after a manager’s corrective feedback. What they support is narrower and still useful. Specific goals can help. How much they help depends on the task.

That’s what the three boxes under the chart say: specific goals can help, complexity can modify effects, and clarity isn’t capability. A clear goal tells someone what to aim at. It doesn’t make the task any simpler to do.

Maya’s standard was simple to state. Dev had told her the check itself took about twenty minutes. But confirming a definition for a brand-new segment wasn’t a twenty-minute task. It needed a source file that lived with another team, a request in that team’s queue, and enough days between the request and the draft.

The instruction at the bottom is two moves, not one. Specify the goal. Examine the task. Maya had done the first carefully, twice. She had never done the second. So before the review, she answered its question for this draft: what skill, information, coordination and conditions does this work require?

• Skill. Tracing a segment back to its source definition. Dev had done it on the other two segments.

• Information. The source file for each segment, including the new one.

• Coordination. A request to the data team, and a turnaround she didn’t control.

• Conditions. Enough time between the scope change and the draft date.

What the request record showed

At 2 p.m., Maya started with a question instead of the circle. “Walk me through the enterprise segment.”

Dev did. The enterprise segment hadn’t been in the forecast until Monday’s stakeholder sync. He requested its source file that afternoon. The data team’s queue listed a three-business-day turnaround, which put the file at Thursday. “This version isn’t going anywhere until that file comes in,” he said.

Maya opened the request record herself. The timestamp matched: Monday, 3:40 p.m. Status: in progress. Dev’s account was now partly an observation. The other part, whether anything else had slowed the check, was still his account.

Share

The infographic lays out six conditions around the work: skill, information, authority, time, resources and dependencies. Maya’s draft touched three of them. Information, because the file hadn’t arrived. Dependencies, because the data team owned the turnaround. Time, because the Wednesday date came before Thursday’s file.

The research on conditions comes mostly from a neighboring field. In 2010, Brian Blume, Kevin Ford, Timothy Baldwin and Jason Huang published a meta-analysis of 89 empirical studies on training transfer, meaning whether people applied what they learned in training to their actual work. Transfer was positively related to person factors, including cognitive ability, conscientiousness and motivation, and to a supportive work environment. The relationships varied with the training’s objectives and with how transfer was measured. When the same source supplied both measures, relationships tended to look stronger.

Training transfer isn’t feedback follow-through. And “supportive work environment” is a much broader idea than one file in one queue. So this is adjacent evidence. What it supports is the infographic’s first box: conditions can matter, alongside the person.

A service-sector field study by Adkins and Naumann gets closer to the workplace. Situational constraints shaped how strongly people’s achievement orientation showed up in objective performance, with modest effects. Constraints shaped the person’s part. They didn’t replace it.

The second box says who controls it can differ. Put plainly: who actually controls the thing that’s missing? Maya mapped it for this draft.

• The data team controlled the turnaround.

• Dev controlled when he asked. He asked the same day the scope changed.

• Maya controlled the Wednesday date, and whether a draft could arrive with a gap marked as a gap.

That last line stung. She had set Wednesday the Friday before, before the scope changed, and never asked what the date required.

The third box keeps all of it honest: cause isn’t established. So does the red line under the headline. A missing condition isn’t automatically the cause. The missing file explained why the enterprise definition wasn’t confirmed on Wednesday. It didn’t explain why the draft carried no note saying so. Whether a note was expected, nobody had ever said.

Maya caught herself about to swap one verdict for another. “He didn’t follow through” had become “the queue did it.” Same jump, opposite direction.

The question at the bottom slowed her down. Which necessary conditions were available, unavailable or still unknown?

• Available. The skill. Two definitions confirmed and logged.

• Unavailable. The enterprise source file, until Thursday.

• Unknown. Whether an unconfirmed definition should have been marked in the draft. That one was a question about her standard, not about Dev.

“Every draft” is a goal, not a plan

Before the review ended, Maya asked one more question. “When the file lands, what tells you it’s time to run the check?”

That covered drafts he sent himself. It didn’t cover a version someone else forwarded, or a segment that changed after he had checked it. “Every draft that goes out” named the change. It didn’t name the moment.

The next two infographics use a project handoff, while Maya and Dev are working through a forecast. The task is different. The question is the same.

Read more

When the work looks complete but the shared result won’t move, don’t assume the fix is better communication. Don’t assume it’s a reorg either. Follow the work to where it changed hands, and look there first.

What if the biggest obstacle to collaboration isn’t the people, but what we’re overlooking between them?

My new book, Collaborate Better, explores how we can break through organizational silos, solve problems together, and turn individual contributions into shared success.

Your road to better collaboration starts here. Get your copy today at

https://CollaborateBetter.us

Research Binder: the receipts (citations + source notes) are compiled in a PDF at the bottom of this post.

The kickoff call starts on time.

The customer signed three weeks ago. Sales logged the deal. Product locked the scope. Engineering shipped the configuration. Marketing sent the welcome sequence. Customer Success staffed the account and booked this call.

Then the onboarding lead asks a friendly opening question. “So, what are you hoping to achieve with us?”

There’s a pause on the line. The customer answered that exact question during the sales cycle. More than once. They answer it again, a little slower this time.

Pull up the tracker and every box is green. Sales: done. Product: done. Engineering: done. Marketing: done. Customer Success: done. The thing everyone actually wanted, a customer getting value from what they bought, is still waiting.

A note on that scene before we go further. It’s an illustrative scenario, built from a kind of moment that people who work between Sales and Customer Success will recognize. It is not one company’s case file, and I’m not presenting it as one.

Now, the honest part. Maybe the account executive skipped the notes. Maybe Customer Success was short-staffed that month. Maybe the customer’s goals changed after signing. Any of those could be the answer.

Before you pick one, notice what the green checkmarks can’t tell you. Task status is not outcome status. The tracker doesn’t show what needed to cross the boundary between teams, or whether it arrived. And it doesn’t show what else could explain the stall.

That’s where this series starts. Not with blame. With the work.

Follow the work before assigning the blame.

Over the next eight Wednesdays, The Collaboration Trap chases one question: what is making capable people struggle to do good work together? Each week takes one piece of the answer and hands you one tool you can use without permission to redesign the company. This week’s piece is the gap between finished and delivered.

Marked done isn’t the same as delivered

The first trap hides in plain sight, inside the tool everyone trusts.

A completion record answers one question: did this task close? It says nothing about a second question: did the shared result arrive? We read one record and quietly assume it answers both.

Look at the two finish lines. The left one reads Task complete. The right one reads Outcome not yet verified. Same project, same week, two different states. Notice the right one isn’t labeled failed. It’s labeled not yet verified. A status report shows you only the left one.

Three things follow from that.

• A closed task is a status record. It tells you someone moved a card. It doesn’t tell you the next person can use what’s on it.

• Quality still needs evidence. “Done” means the assigned work was marked complete. It doesn’t mean the work was wrong. It also doesn’t prove the work was enough.

• Delivery needs its own check. The customer, the launch, the shared result: somebody has to look at that directly, not infer it from a column of green.

None of this accuses anyone of sloppy work. No one has to be dishonest or incompetent for two records to tell different parts of the story. Even if every person in our scene did exactly what was asked, the tracker alone couldn’t tell us the result arrived. Doing what was asked and delivering what was needed are two separate claims, and they need two separate checks.

Here’s a receipt on my own homework. Before writing this, I went looking for a study showing teams hitting their measured targets while the shared outcome stalled. I wanted a stated target, proof the target was met, and a measured result that still fell short. I didn’t find one that cleared all three bars. So I’m not going to tell you it’s proven. I’m telling you the distinction is real, and checking both sides costs you very little.

What needs to cross the boundary?

If the task closed and the result didn’t arrive, the next place to look is the seam where the work changed hands.

When work crosses a team boundary, the task can move without the context the receiver needs.

The sending team’s desk is covered in notes the receiving team never sees. Three of them matter more than the rest:

• What was decided? The commitments, trade-offs and promises made before the handoff.

• What dependencies remain? The things still owed by someone else before this work can finish.

• Who can clarify or escalate? The person to call when the receiver hits a question the ticket doesn’t answer.

Those three questions are my illustration, not a validated checklist. I’m being careful about that because the strongest evidence on handoffs comes from somewhere very different from your office.

In a multicenter study across nine pediatric residency programs, Starmer and colleagues introduced a structured handoff program called I-PASS. It combined a standard handoff format with training, faculty coaching and observation. After implementation, the programs saw fewer medical errors and fewer preventable adverse events, and handoffs didn’t take longer.

That’s real evidence, and it comes with real limits. It’s clinical. It tested a whole bundle, so nobody can say which part did the work. The study wasn’t randomized. It does not tell you what a better handoff is worth in a software launch or a customer onboarding. Corporate effects are not established.

What transfers is the idea, not the effect size. A handoff is a transfer, and a transfer can lose the very thing the receiver needs most.

So here’s a small check you can run this week. Ask the receiving person what they got, in their own words, before you call the handoff done. It’s a check, not a cure. It does tell you what arrived.

Check what arrived, not just what was sent.

When a decision stalls, check who has authority

Sometimes nothing gets lost in transit. The work arrives intact, the context is all there, and it still sits. The next move is a decision, and nobody in the meeting can make it.

You know this meeting. Eight smart people, a real problem, a lively discussion. Everyone contributes. Everyone leaves agreeing it matters. Next week the same item is back on the agenda.

Participation isn’t the same as authority.

The graphic splits the people around a pending decision into three illustrative roles. Some contribute ideas and information. Some review, because a legitimate control requires it. Someone decides. When the decision stalls, the useful question is where it’s stuck between those three.

Ask three things before you change anything:

• Who can make the decision? Not who cares most. Who is actually authorized.

• Which reviews are required? Some are there for good reasons: security, legal, safety, money. Know which ones before you call them delays.

• What is the authorized escalation route? When the decider is unclear or unavailable, there’s a sanctioned path upward. Find it and use it.

And keep the fourth question on the graphic in view: what else could explain the wait? The decision may be correctly reserved for someone senior. The data it needs may not exist yet. The decider may simply be out of capacity this month.

On evidence: Tubre and Collins pooled years of research on role ambiguity and job performance in a meta-analysis published in the Journal of Management. What they report is an association. It doesn’t show that unclear decision rights cause delays, and it doesn’t prove that naming one owner fixes them. Decision-delay effects are not established.

So don’t start by redrawing the process. Start by tracing the route the decision is supposed to take.

Check the decision route before changing the process.

Structure isn’t automatically the villain

By now you might want to blame the org chart. Departments, approval gates, review boards: tear them down and the work will flow.

I’d hold off on that. Not every boundary is a problem.

The pillar in the middle of the graphic is one boundary. Read it from the left and you see what it may protect:

• Expertise. Specialist teams give people room to go deep.

• Independent review. A second set of eyes only works when it’s actually independent.

• Risk controls. Security, compliance and safety checks guard against specific, known risks.

Read the same pillar from the right and you see what to examine: waiting, information loss and handoff gaps. Same boundary. Different questions. The goal isn’t to choose a side. It’s to ask both sets of questions before you touch anything.

Here’s why I’m wary of simple fixes. In Ontario, Canada, 101 hospitals introduced surgical safety checklists. Urbach and colleagues compared outcomes before and after. They found no statistically significant reduction in operative mortality or complications.

That doesn’t prove checklists are ineffective. It’s also a clinical finding, not a test of organizational hierarchy. I include it for one reason. Adopting a tool and making the work better are two different events. A boundary, a checklist or an approval step can look identical on paper and behave very differently in practice.

So before you remove anything, ask what it guards. Then look at the interface around it. Sometimes the control is fine and the handoff into it is the part that’s broken.

Check what a boundary protects before changing it. Then check whether the interface around it is working.

Operational effects are documented. The dollars are not.

You’ve seen the articles that put a price tag on bad collaboration. Billions lost. Hours wasted per employee per week. I went looking for a figure I could defend in front of a skeptical executive.

The evidence review behind this article didn’t find one that matched these claims. Unknown is not zero.

Share

I ran five of this article’s claims through a financial review: handoff gaps, unclear authority, the role of structure and controls, operational consequences, and the coordination fixes. The left side of the graphic shows what the evidence does document, mostly in clinical settings: information loss, errors and quality outcomes, and safety and reliability outcomes.

The right side shows what it doesn’t. For all five claims, the financial stakes came back unknown. Not low. Not trivial. Unmeasured. The studies counted errors and adverse events. They didn’t cost them in a way that transfers to your business. Unknown cuts both ways: it doesn’t mean the cost is zero, and it doesn’t mean a loss must exist.

That matters if you’re the one walking into an executive review to argue for a change. A borrowed dollar figure is the easiest number in your deck to knock down. Your own operational evidence is much harder to argue with.

Before you put a number on a slide, ask the three questions at the bottom of the graphic:

1. What was observed? The waits, the repeat questions, the rework you can actually point to.

2. What was actually costed? Money someone measured, or a model whose observations and assumptions are out on the table. A figure borrowed by analogy doesn’t count.

3. What remains unmeasured? Say so out loud. It makes everything else you say more credible.

Some remedies have evidence. Others are not yet established.

The same discipline applies to fixes. Every framework, consultant and conference talk has a remedy for cross-team friction. Some of those remedies have been tested. Others haven’t been tested in settings like yours.

A plausible fix isn’t a proven fix.

I sorted ten categories of remedy by how much evidence shows they actually work in the settings where they were studied.

• Moderate (3): structured handoff bundles, structured debriefing, and team-training bundles. Most of that evidence comes from healthcare and other high-reliability work.

• Weak (3): closed-loop confirmation, records and traceability, and boundary-preserving coordination.

• None established (4): readiness criteria, decision-rights interventions, synchronized dependency planning, and shared objectives or scorecards.

That last row stings a little for me. I’m an SPC- and RTE-certified enterprise agile coach. I teach and coach versions of some of those practices. I’m still putting them in the “none established” row, because that’s where the evidence I found puts them.

Read the label carefully, though. None means not established, not ineffective. The evidence I reviewed doesn’t show these remedies working for teams like yours. It doesn’t show them failing either. And across all ten categories, the direct effectiveness evidence for U.S. product, project and program work is weak.

That isn’t a reason to stop trying things. It’s a reason to try them like experiments, watch what happens, and stay honest about what you know. When someone pitches you a fix, including me, ask three questions.

Ask what was tested, where, and with what result.

Before you blame the people or blow up the org chart

So where does that leave you on Monday morning, with one stuck outcome and limited authority over the teams involved?

Not with a reorg. Not with a lecture about communication. With a better place to look.

Investigate one stalled outcome first.

This is the Stalled-Outcome Observation Guide: a practical observation guide for one stalled outcome. Try this with one outcome this week: pick something real from the last month and work through it.

1. Name the result. What was supposed to be delivered? Write the outcome, not the task list.

2. Trace the work. Where did roles or teams intersect? Mark every place the work changed hands.

3. Check decision authority. Who can decide or authorize the next step? Which reviews or controls are legitimately required?

4. Record what happened. What waiting, rework or information gaps, if any, did you actually observe? Write down what you saw, separately from what you think it means.

   1. Then check other explanations. Capacity? Upstream quality? Requirements? Complexity? Sequencing? Legitimate controls? Any of these may explain the stall better than the boundary does.

5. Choose one next step. What can be checked or clarified within your authority? If the decision belongs to someone else, your next step is a clear question or an authorized escalation. It might also be gathering missing evidence, or concluding that no process change is warranted right now.

A few honest guardrails, because I’d rather you trust this than oversell it. This is a practical observation tool, and its effectiveness has not been validated. It won’t diagnose the root cause, score your team, or prove what a fix is worth. It also doesn’t give anyone permission to skip an approval, remove a specialist, or weaken a security, legal, safety, financial or compliance control. If you finish it and still aren’t sure what’s going on, that’s a legitimate result. You’ll be unsure with much better information.

Subscribe now

The work was done. Now go find the outcome.

Let’s go back to that onboarding call. Every team marked its work complete, and the customer still had to repeat themselves. Nobody in that story needs to be the villain. We still don’t know what caused the stall. Maybe context didn’t travel. Maybe the goals changed. Maybe something else happened. What we do know is where to look first, and what else to rule out.

That’s the shift this whole series is about. When collaboration breaks, the reflex is to look at the people: who dropped the ball, who didn’t communicate, who needs coaching. Sometimes that’s right. But capable people can do their jobs well inside a system that still can’t deliver the result. Finishing your piece is not the same as delivering the result.

Go deeper with Collaborate Better

The Stalled-Outcome Observation Guide helps you find where the work stalled. Chapter Six of Collaborate Better, “Herding Cats, Breaking Silos, and Actually Getting Things Done,” picks up at the next question: once you know where to look, how do you get the right people into the right conversation at the right time to move it?

Inside the chapter:

• The Right People, the Right Conversation, the Right Time, and what goes wrong when you get only one or two of the three

• Psychological safety warning signs, including the “It’s fine” problem, and four ways to build it

• Tuckman’s Form, Storm, Norm, Perform, and why every roster change sends a team back to Forming

• The Tiger Team: how to build a small coalition of the willing to get stuck work moving

• The pre-meeting: how the real work gets done before anyone walks into the room

If you’re accountable for an outcome you can’t fully control, start there.

Collaborate Better is available now in paperback and Kindle, with the hardcover coming soon.

Get your copy at CollaborateBetter.us.

Next Wednesday

Part 2 takes on a strange contradiction: Your Company Has a Communication Problem. So Why Are There So Many Meetings? We’ll look at the gap between activity and decisions, and you’ll get a tool for reclaiming one decision from the meeting treadmill. Chapter Six has a line for the worst case: the twenty-seven-person meeting “where nothing happens except the slow erosion of morale and hope.”

Your turn

Hit reply and tell me about a time the tasks were all marked done and the outcome still got stuck. What needed to cross the boundary? I read every response, and the best stories (shared with permission and anonymized) help shape the rest of this series.

P.S. If this landed for you, forward it to one person who owns an outcome they don’t fully control. It might save them a status meeting.

Sources

Starmer, A. J., et al. (2014). “Changes in Medical Errors after Implementation of a Handoff Program.” New England Journal of Medicine, 371(19), 1803 to 1812. doi:10.1056/NEJMsa1405556. Checked against AHRQ PSNet and WashU Source.

Urbach, D. R., et al. (2014). “Introduction of Surgical Safety Checklists in Ontario, Canada.” New England Journal of Medicine, 370(11), 1029 to 1038. doi:10.1056/NEJMsa1308261. Checked against AHRQ PSNet.

Tubre, T. C., & Collins, J. M. (2000). “Jackson and Schuler (1985) Revisited: A Meta-Analysis of the Relationships Between Role Ambiguity, Role Conflict, and Job Performance.” Journal of Management, 26(1), 155 to 169. doi:10.1177/014920630002600104. Citation confirmed via QIC-WD summary.

Citations And Research Binder

525KB ∙ PDF file

Download

This Citations and Research Binder supports *The Collaboration Trap*, Episode 001: **The Tasks Were Marked Done. Why Was the Outcome Still Stuck?**, accompanying *Collaborate Better*. It examines research on task completion versus shared outcomes, cross-functional handoffs, decision authority, organizational boundaries, and coordination remedies. The binder distinguishes empirical findings from professional guidance, documents counterevidence and research gaps, and explains why financial stakes remain **Unknown** and ten remedies were classified **3 Moderate, 3 Weak, and 4 None Established**. It also supports the **Stalled-Outcome Observation Guide**, an unvalidated practical tool for examining one stalled result without presuming its cause. Its purpose is to make the evidence, limitations, and unanswered questions visible before readers assign blame or propose organizational changes.

Download

Previous:

More Content to Discover:

👋 Welcome to my paid subscriber-only edition of Empathy Engine (🔒 Leader’s Dispatch). Each week I build evidence-forward tools for product leads who need to say no, defend tradeoffs, and lock in decisions before they get rewritten later.

Maya Chen gave Dev feedback every Friday. By the third Friday in a row, Friday was the problem. (Maya and Dev are fictional, built to illustrate this series.)

Three weeks earlier she had sent Dev the rewritten message about the definitions check. He replied the same afternoon: he would run the check on every draft, internal ones included, and the source-file queue was the hard part. Maya put the definitions check on their Friday agenda so it wouldn’t drop.

The first Friday, she raised it. Dev said he was on it. The second Friday, she raised it again. No new forecast had gone out in between. On the third Friday, she opened her notes to the same line, in the same words: Definitions check, all drafts.

Dev listened, nodded and said he had been checking. After the meeting, he reopened the forecast he had already finished and rechecked every definition in it. Maybe the recheck caught something. Maybe it didn’t. Neither of them could say. Nothing in the request had changed. The only new thing that week was the meeting.

Maya didn’t think of it as repetition. She thought of it as follow-through. Good managers keep the thread alive. Good managers don’t let things drop. And the calendar kept offering Friday. The meeting itself was worth having. What she brought to it, week after week, was the part she hadn’t looked at.

I’ve run that meeting. A recurring one-on-one arrives, there’s no new work to look at, and the old correction is the only thing on my list, so I say it again. It feels like diligence. Looking back, the second and third times rarely told the other person anything they didn’t already know. Mostly they told them I was still watching.

Research Binder: the receipts, methodology notes, and source boundaries are compiled at the bottom of this post.

Previous:

Last week Maya turned a six-word judgment into a message Dev could check. This week the message isn’t the problem. The schedule is.

I promised to look at why more frequent feedback isn’t automatically better feedback, and how to tell whether a message is arriving too early, too late or too often. The research answers the first half more cleanly than the second. More conversations don’t guarantee better results. Nothing in the evidence turns that into a rule for less.

So here is the boundary for this whole episode, before any study.

Frequency is one of four questions on that page. The others ask what useful information or clarification the feedback adds, what the task needs and what should happen next. For three Fridays, Maya had been answering only the first one. She knew how often she was giving Dev feedback. She hadn’t asked what the next conversation needed to accomplish.

An average isn’t a cadence

First, the counterweight, same as the last two weeks. Feedback is not the villain of this series.

In 1996, Avraham Kluger and Angelo DeNisi published a meta-analysis of 131 papers and 607 effect sizes, each comparing a feedback group with a no-feedback control. Most of the studies were laboratory research. On average, feedback helped: d = 0.41, a standardized effect size, not a 41% improvement. Over a third of the effects went the other way.

The same paper tried to find out whether frequency explained that spread. The authors coded how many times feedback was given in each study. Across all the studies, the link between frequency and effect size was small and not statistically significant. Depending on which studies were included, its direction changed. The authors wrote that the frequency effect “may be an artifact.” They never coded how long feedback was delayed.

So the paper this series started with doesn’t settle frequency or timing in either direction. It doesn’t say Friday feedback helps. It doesn’t say Friday feedback hurts. Later research took the question on directly, and the answers depend on where you look.

More often isn’t the same as more useful

Three kinds of study point in three directions. Each answers a slightly different question.

• A laboratory experiment. In 2011, Chak Fu Lam, Scott DeRue, Elizabeth Karam and John Hollenbeck varied how often people received feedback on a task. Performance didn’t climb steadily as feedback increased. It rose, then fell: an inverted U. Their explanation is that very frequent feedback can overwhelm the attention people have available, so effort on the task drops. People’s mood also changed the shape of the curve. This was one laboratory task. The best frequency there says nothing about whether weekly or monthly is right for your team.

• A field experiment. Pablo Casas-Arce, Sofia Lourenço and Asís Martínez-Jerez randomly assigned about 800 home-repair professionals in Spain to weekly or monthly performance reports, either detailed or summarized, and tracked customer satisfaction. Detailed monthly reports produced the best results. Weekly reports, detailed or summarized, did no better than the comparison group, which received a monthly summary.

• A workplace observation. Shana Mertens, Eveline Schollaert and Frederik Anseel asked employees how many feedback conversations they’d had. The average was 3.8 over three weeks. More conversations went with better performance and satisfaction, with no drop at the highest frequencies.

Put those side by side and the honest summary is short. Frequency matters, but it doesn’t push results in one direction.

Look at what changes from row to row in that table: the setting. A laboratory task, a firm’s reporting system and a survey of employees aren’t the same place. None of them is a manager talking with a senior analyst on a Friday. That’s why the four boxes above the table matter more than any single row. The task, the information, the recipient and the setting all change what more feedback does, so the useful question isn’t “how often?” It’s “under what conditions?”

Same schedule, different conditions

The most useful studies don’t ask whether frequency helps. They ask when.

In four experiments, Nicholas Lurie and Jayashankar Swaminathan had people manage inventory and gave them feedback on results at different intervals. When demand jumped around, more frequent feedback made decisions worse. When demand was stable, frequency made no significant difference. Their analysis showed that under noisy conditions, recent results weighed too heavily in people’s decisions. Fresh numbers looked like signal when they were often noise.

The repair-network experiment offers a similar explanation. With weekly reports, the most recent week loomed largest, and the authors suggest people overweighted it. That’s their interpretation, not something they measured directly. What they did measure points the same way: detail mattered alongside frequency. Detailed monthly reports beat detailed weekly ones. When everyone moved to detailed monthly reports after the experiment, the gap disappeared.

The person receiving the feedback matters too. In one experiment, Joseph Johnson, Khim Kelly and Wioleta Olczak found that more frequent feedback hurt accuracy more for people high in what the researchers call narcissistic oversensitivity, a trait the authors say may cause especially strong reactions to performance feedback. A brief growth-mindset prompt reduced the effect. One experiment, one specifically defined trait, and no license to sort your team by personality.

The field experiment on that page carries a note worth repeating: performance reports, not live coaching conversations. A report arrives whether anyone discusses it or not. A Friday one-on-one involves two people, a relationship and whatever else is on the agenda. That difference cuts both ways. It’s also why none of these studies can tell Maya how often to meet with Dev. They can tell her that the calendar doesn’t tell the whole story.

Sometimes more helps

If you’re tempted to read all this as “cut the cadence,” the strongest workplace evidence points the other way.

The Mertens finding deserves more than a row in a table. In ordinary workplace ranges, employees who reported more feedback conversations also reported better outcomes. The authors suggest the relationship between employee and manager may help carry that link. But it’s an observational study. People reported their own feedback frequency, and better relationships may produce more conversations rather than the reverse. That’s an association, not proof that adding conversations improves anything.

Health care adds a second line. In the audit-and-feedback research I covered last week, one review found more frequent feedback was associated with stronger effects, and the Cochrane review says feedback may work better when it’s provided more than once. That’s clinical performance data, not one-on-ones. The two reviews also draw on overlapping evidence, so count them as one signal, not two.

And sometimes the person across the desk is the one asking. A request like “Could we review this?” may reflect a genuine need for clarification or support. It’s a reason to talk, whatever the calendar says.

Both findings matter, and neither cancels the other. More feedback may help. Its benefits can’t be assumed. For Maya, that rules out the easy fix. Dropping Dev’s Friday meeting would replace one rule with its opposite. The decision box on that page says it plainly: don’t prescribe less feedback by default.

Sooner isn’t a rule either

Frequency and timing are related, but they aren’t the same question. Frequency asks how often feedback happens. Timing asks when it arrives relative to the work. The other half of the cadence belief is about timing: give feedback right away, every time. The evidence here is thinner, and almost none of it comes from workplaces.

The most current synthesis I found is a 2026 meta-analysis by Erva Nihan Kandemir, Emanuele Esposito, Lilas Gurgand and Franck Ramus. It pooled 51 studies and 160 effect sizes comparing immediate and delayed feedback in computer-assisted learning. On average, timing made no meaningful difference: g = 0.03. Like d, g is a standardized ruler. A value of 0.03 is close to zero, and the plausible range ran from slightly negative to slightly positive.

The average hides a lot of variation. Results differed by educational level, by subject and by whether learners answered under time limits. Delays ranged from about a second to a week, and few lasted a day or more.

An older review by James and Chen-Lin Kulik found the answer depended on the kind of study. Applied classroom studies tended to favor immediate feedback. Some list-learning experiments favored delay. Getting an answer right today, remembering it next month and using it in a new situation are different outcomes, and timing doesn’t affect them all the same way.

I couldn’t find a study comparing a manager who corrects an analyst now with one who waits until Friday.

Share

Two lines on that page hold whatever the research says. A consequential error may need correcting promptly, and a person asking for clarification shouldn’t wait because of a timing rule. Everything else depends on the task, the purpose and what happens if you act now instead of later. Waiting isn’t automatically better. Neither is speed.

What Maya changed

That Friday evening, Maya scrolled back through her notes. The same line, three weeks running. Between the second and third mentions, Dev hadn’t produced a new forecast. There was nothing new to look at. The meeting wasn’t reviewing new work. Without anyone intending it, it had become a check on whether Dev still remembered.

Repeating a correction isn’t wrong in itself. If Dev had misunderstood the ask, if a draft had gone out unchecked, or if accountability needed a follow-up, saying it again would have been the job. Maya hadn’t asked which of those, if any, applied. The calendar had answered for her.

She kept the Friday one-on-one. It’s where Dev raises blockers, and it’s the one half-hour each week that isn’t about a deliverable. What she dropped was the repeated correction.

The next Friday, she asked a different question. “What have you tried with the definitions check, and where does it get stuck?”

Dev walked her through it. The check itself took about twenty minutes. Waiting on source files from the other team’s queue took days. Maya wrote the queue down as its own problem, separate from the feedback. That one needed a different conversation, with a different team.

Then she named the next review point out loud. “The forecast draft due Wednesday. I’ll go through that one with you. If something comes up before then, don’t wait for Friday.”

The same week showed the other side. On Tuesday, a figure in a forecast summary going to the executive staff the next morning didn’t match the model. Maya walked over to Dev’s desk that afternoon. Waiting for Friday, or for any checkpoint, would have been the wrong call.

And when Priya, another analyst on the team, stopped her after a meeting and asked, “Could we review this?”, Maya made the time that afternoon. The question was never how little she could say. It was what the next conversation needed to accomplish.

Read more

A researcher on an AI product team kept asking a question nobody wanted to answer. The team had a groundedness score for its large language model feature, roughly a measure of whether the model’s answers stay anchored to their sources. So the researcher asked the obvious next thing: “If we get a groundedness score of 0.5, what do we do?” The developers didn’t want to answer. An AI engineer kept pushing for a plan, and the best that came back was “we’ll monitor for trends.” Then came the question underneath all the others. “When I ask if we’d pull the product for bad scores, no one answers.”

That account comes from a study published at CHI 2026, in which researchers interviewed 19 people who evaluate LLM products in the wild. The participant is identified only as P15, someone working in research. I’m not going to dress it up as a tense meeting with a whiteboard and a long silence, because the paper doesn’t describe one. It describes something quieter, which P15 summed up this way: “We’re collecting data without a plan for what we’re going to do with it.” The researchers found that 17 of their 19 participants were affected by the same gap between gathering evaluation results and turning them into action.

👋 Welcome to this week’s 2nd edition of Empathy Engine. Every Wednesday, I publish a new article for paid subscribers first, then unlock the full piece for everyone late Thursday morning. Each week, I turn product leadership friction into practical tools, sharper language, and more defensible decisions.

Research Binder: the receipts (citations + source notes) are compiled in a PDF at the bottom of this post.

Notice what isn’t broken in that story. The feature works, at least well enough to show people. The team is measuring something. The engineer is asking exactly the right question. What’s missing is a person who says, out loud and on the record, what good enough means for this feature, what happens when the bar isn’t met, and who gets to decide. Nobody has measured how often this happens across teams or what it costs them, and I’m not going to pretend otherwise. What the research can show is why a working demo can’t answer the question P15 kept asking.

As an Agile coach, I’ve encountered situations where delivery dashboards showed encouraging numbers but offered little guidance about what leadership should do next. Velocity was stable, work was being completed, and the reports looked reassuring. Yet those measurements didn’t necessarily establish whether the team was delivering the outcomes stakeholders needed. The harder question was always what decision the evidence justified.

This is the seventh and final episode of The Judgment Layer. For six weeks we’ve been circling the same human moment from different angles: the person who reads the work, hands it off, clicks approve, reads the scoreboard, trains the next reviewer, and owns the voice on the slide. This last one is about the question that sits above all of them. When an AI feature appears to work, who decides it’s good enough, how is that judgment written down, and what can the next reviewer actually rely on?

A working example isn’t a release decision

A demo is one convincing result. It’s a real result, and it matters, because it tells you the feature can do the thing. But a release decision has to answer questions a single good run never touches. What counts as good for this use? What must not happen? Would another reviewer judge it the same way? And who is actually allowed to say yes?

Those four boxes on the right side of the board are the rest of this episode. Acceptable asks what counts as good. Failures asks what must not happen. Agreement asks whether other reviewers would see it the same way, and, as we’ll get to, whether it matters if they do. Authority asks who can decide. The demo on the left answers none of them, which is why the line across the middle is the one I’d want every product team to tape to a monitor: a working example isn’t a release decision.

That’s not a knock on demos. A demo starts the conversation. It just doesn’t settle whether the output is good enough, and the trouble starts when a team treats the first as if it were the second. The question in the bottom corner of the right panel is the one worth carrying into every review: what happens beyond that result?

The tests we choose change what we find

Start with the most basic problem. Whatever you put in front of a feature decides what you’ll learn about it. A demo is a test session too. Someone picked the inputs, someone sat at the keyboard, and the conditions were whatever happened to be on hand that afternoon. Change the conditions and you change what shows up.

One small study makes that visible. In 2020, Marco Tulio Ribeiro and colleagues gave 18 people with NLP experience the same model and the same task: a system that judges whether two questions mean the same thing. They had two hours to find its problems. Six people tested it unaided. Six were given a list of capabilities to check, things like whether the model handles negation or changes in names. Six got that list plus templates for writing tests quickly. Afterward, each participant rated the problems they had found on a five-point scale, where 3 meant a bug worth investigating and fixing.

The unaided group averaged 2.2 tests flagged as bugs rated 3 or higher. The capability-list group averaged 5.5. The list plus templates averaged 6.2. Look at where the jump happens. Most of it came from the list alone, from simply being told what to look at. The templates added a little more on top.

Now the honest part, which is printed right on the board. Participants rated their own bugs, so there was no ground truth about how serious each one really was, and the paper doesn’t report a significance test for these averages. It’s six people per condition, one model, one task, and a study that predates today’s LLM products. So I’m not telling you structured testing prevents failures. Finding more problems doesn’t prove fewer future failures. What the study shows is narrower and, for our purposes, more useful: the shape of a test session shapes what surfaces in it.

Which brings us to the takeaway at the bottom of the board. Before you trust what a test told you, ask what the test conditions didn’t examine. A demo examined one path through the feature, chosen by someone who wanted it to work. That’s worth knowing. It just isn’t the whole picture.

One prompt, more than one answer

Here’s the part that makes AI features different from the software most of us grew up approving. With ordinary code, if a test passes on Tuesday, it passes on Wednesday, unless someone changed something. With a generative model, you can send the exact same request twice and get two different answers back. That means a demo isn’t just one path chosen by a friendly operator. It’s one draw.

Shuyin Ouyang and colleagues measured this directly in a study published in ACM Transactions on Software Engineering and Methodology in 2025. They took 829 Python programming problems from three well-known benchmarks and asked ChatGPT to write code for each one, five separate times, with the same prompt. Then they ran the tests and compared the results.

At the default setting, the share of tasks where none of the five requests produced matching test outputs was 75.76% for the CodeContests problems, 51.00% for APPS, and 47.56% for HumanEval. The researchers also tried setting the temperature to 0, the dial that’s supposed to make a model’s answers more predictable. It reduced the variation. It didn’t eliminate it.

Two cautions keep this honest. First, different outputs aren’t automatically wrong outputs. Two pieces of code can both be acceptable while behaving differently on a test. Second, this is code generation with one model family, so I’m not claiming every AI feature varies this much. The narrower point holds anyway: if the same request can produce different results, one successful run is a sample of one.

The variation doesn’t stop at repeated requests, either. In a benchmark of AI agents completing customer service tasks with simulated users, the τ-bench study (ICLR 2025) found agents that succeeded once often failed when the same task was repeated, with consistency across eight attempts falling below 25% in the retail domain. And the model you approved may not stay the model you’re running. Lingjiao Chen, Matei Zaharia and James Zou compared the March and June 2023 versions of GPT-4 and found its accuracy at identifying prime numbers fell from 84% to 51% on the same questions. That’s one study of one hosted service, and it shows that a change like that can happen, not how often it does. But it’s enough to make the takeaway on this board stick: one successful run doesn’t establish repeatability, and a result from last quarter describes last quarter’s model.

Good can change while we’re checking it

The obvious fix is to write the standard down first, then test against it. Decide what good looks like, then go look. It’s sensible advice, and it runs into a problem that anyone who has reviewed real AI output will recognize. You often don’t know what you care about until you see what the system actually produces.

Shreya Shankar and colleagues watched this happen. In a study presented at UIST 2024, nine industry practitioners used a tool called EvalGen to define criteria for judging an LLM’s outputs and then grade those outputs. The researchers put the loop in one sentence: “users need criteria to grade outputs, but grading outputs helps users define criteria.”

Share

As participants worked through real outputs, their criteria evolved. They added new criteria when they ran into kinds of bad output they hadn’t anticipated, and they reinterpreted the ones they already had. Define, review, revise, and around again. The authors went as far as saying it is “impossible to completely determine evaluation criteria prior to human judging of LLM outputs.”

Keep the scale in view. That’s nine people, one tool, one task, and short sessions, so it can’t tell you how common this is. But it matches what showed up in the CHI 2026 interviews, where 12 of the 19 participants described beginning their evaluation with informal “vibe checks.” My read, and it is a read rather than a finding, is that those vibe checks aren’t laziness. They’re often where the real criteria first show up.

So the goal isn’t a standard that never moves. A standard written before anyone has looked at real outputs is a draft, and that’s fine. The problem is a standard that changes silently, so the next reviewer is grading against rules nobody remembers revising. That’s why the takeaway on this board is a habit rather than a rule: date each standard, and record why it changes.

Agreement is not correctness

If one person’s judgment shifts, the natural move is to bring in more people. Get the PM, the engineer, and the designer in a room, look at the outputs together, and see where everyone lands. When they all agree, it feels like the question has been answered. Agreement feels like proof. It isn’t, and there’s a study that shows the gap unusually cleanly.

In 2021, Elizabeth Clark and colleagues asked crowd workers in the United States to read short passages, stories, news articles, and recipes, and decide whether each was written by a person or by GPT-3. Some evaluators got training first, including annotated examples. Because this task has a known right answer, the researchers could measure two separate things: whether people agreed with each other, and whether they were actually correct.

Without example training, evaluators landed around 50%, which is chance for a two-choice task. With example training, accuracy rose to 55%. That gain was significant overall but not within the individual domains. Meanwhile, agreement between evaluators stayed low across every domain and training setting, with Krippendorff’s alpha at or below 0.11, where 0 means chance-level agreement and 1 means perfect agreement. And the sentence I keep coming back to: in the researchers’ words, “higher agreement did not correspond to higher accuracy.”

Be careful with what this does and doesn’t cover. Clark’s task was spotting AI-written text, not judging whether an output was good enough to ship, and the evaluators were crowd workers looking at GPT-3-era writing. But the distinction it isolates travels: whether reviewers judge alike and whether the judgment holds up are two different questions. A second study sharpens the point. When Sashank Santhanam and colleagues showed crowd workers a reference example before they rated chatbot responses (CHI 2020), ratings became more consistent in some conditions and also drifted toward the example’s score. The authors suggested the added consistency “may be a result of anchoring bias.” Examples can make reviewers more alike without making them more right.

That changes what disagreement means. Sometimes it’s a sign that nobody wrote the criteria down. Sometimes it’s a sign that people want legitimately different things. Back in the CHI 2026 interviews, P15 described the moment of realizing “we were all talking about completely different things actually,” with designers and product managers focused on user experience while the AI engineers meant something else entirely. Another participant, P7, recalled technical people calling an output a bug while the person using it was delighted. Neither side was wrong. They were answering different questions. That’s why the takeaway on this board doesn’t say “get to consensus.” It says preserve disagreement, and examine the reasons.

Someone still has to own the decision

If agreement doesn’t settle it, what does? Not a better rubric, and not a bigger spreadsheet. At some point a person with the authority to accept the risk has to say yes, no, or not yet, and put their name next to it. That’s the step P15 couldn’t get anyone to take, and it turns out to be a familiar gap. In a 2026 study of GenAI adoption at two software companies, Yu and colleagues reported that “ownership is fragmented across phases.” A separate set of interviews with responsible AI practitioners, published at CSCW 2021, captured the same frustration in four words: “Whose job is this?” That study was about ethics and risk concerns rather than quality sign-off, but the sound of the question is the same.

Guidance is clearer than practice here. The NIST AI Risk Management Framework, a voluntary U.S. framework, recommends that roles and responsibilities for managing AI risk be documented and clear (GOVERN 2.1), and that someone determine whether a system achieves its intended purpose and whether deployment should proceed (MANAGE 1.1). One practitioner put the human version of that plainly. Writing about AI features shipped into inspection apps, Lee Nguyen of Dev Station explained why the team pulled automated pass and fail decisions: “Nobody will sign a certificate that a model decided.”

The board splits the work into four jobs that often get blurred together. Someone specifies what counts as acceptable. Someone evaluates what was checked and, just as important, what wasn’t. Someone records what remains disputed. And someone authorizes, meaning they accept the remaining risk. Around those four sit three roles that are easy to conflate: the evidence owner who maintains the record, the decision owner who has authority, and the revisit trigger that says what would reopen the call.

I want to be straight about the status of that structure. It’s an Empathy Engine proposal, and it hasn’t been tested. NIST recommends separating these responsibilities, and interview studies keep finding that the separation is fuzzy in real teams, but no study I found shows that naming an owner makes AI quality decisions better. What naming an owner does is make the decision traceable to someone. That’s also why the orange line on this board matters so much: a completed record doesn’t grant approval authority. A signature records who decided. It doesn’t certify that the evidence was enough, and it doesn’t hand authority to someone who didn’t already have it.

The takeaway is two small acts. Name the authorized owner. Record what could change the decision.

About that number ten

A confession. The working title for this episode was “Ten Examples Before the Argument,” and the idea had real appeal. Before your team argues about whether a feature is ready, write down ten concrete examples of good and bad output, and argue about those instead. Starting small is sensible advice, and it’s common advice too. One evaluation vendor’s guide for product managers suggests five to ten examples to begin with, and on the same page says ten to two hundred is usually enough to identify and fix patterns. Starting is the easy part. The trouble is what ten clean results can and can’t tell you once you’ve got them.

So I ran the numbers. Imagine a feature with a problem that shows up on 1% of requests. You test it ten times, the tests are independent of each other, and every one passes. How likely was that?

With a 1% failure rate, ten independent trials find at least one failure only 9.6% of the time. The other 90.4% of the time, you see ten clean passes and nothing to worry about. Raise the failure rate to 5% and ten trials catch it 40.1% of the time. At 10%, they catch it 65.1% of the time. Put plainly, a problem that shows up once in every twenty requests would be missed by ten random checks more often than not. To have a 95% chance of catching a one-in-twenty problem, you’d need roughly 60 randomly chosen cases. That’s a different exercise from a starter set.

There’s an even bigger catch in the small print, and it’s the one I’d most like you to remember. Those numbers assume random, independent cases drawn from real use. Most teams don’t pick their examples at random. They pick the ones they know, the ones that matter, or the ones that were handy. That’s a perfectly good way to start a conversation, but it changes what the results can mean. Ten random passes can’t rule out an infrequent failure. Ten handpicked passes can’t tell you how often failures happen at all. They tell you about those ten cases. Passing ten examples tells us about ten examples.

Illustration only. Assumes cases drawn at random from real use, judged the same way, and independent of each other.

So here’s where I’ve landed on the number. Ten is a practical starting number, and nothing more. No study I found shows that ten, or any fixed number, is enough, and the right size depends on how rare a failure you need to catch, how precise you need to be, and what happens if you’re wrong. That’s why this episode isn’t called “Ten Examples” anymore. Ten passes aren’t proof of reliability.

Make the judgment inspectable

Put the pieces together and you can see what P15 was missing. Not a better score, and not more data. P15 already had data. What was missing was a record that said what the system was for, what had been checked, where people disagreed, what nobody had looked at, who could decide, and what would bring the question back. That’s what the next reviewer needs to know, and it’s what a demo never hands them.

Subscribe now

Each section of the record answers one of the questions this episode has been working through. For this use names the task and context, because good enough for an internal draft isn’t good enough for a customer email. Evidence checked records what passed, what failed, and who chose those cases and how, so nobody mistakes a handpicked set for a random one. Disagreement keeps the split views on the page instead of averaging them away. Not covered is the box I’d fight hardest for, because it’s where honesty lives: what remains unknown or untested. Decision owner names who authorizes use and accepts the remaining risk. Revisit trigger says what would reopen the call, whether that’s a new model version, a changed prompt, a revised standard, or an incident. And the versions and date line at the bottom keeps the whole thing anchored to the system that was actually checked.

That “not covered” box deserves a warning of its own. When Weixin Liang and colleagues analyzed 32,111 AI model cards for a 2024 paper in Nature Machine Intelligence, the Limitations section was among the least completed, filled out in only 17.4% of cards, with Evaluation close behind at 15.4%. That was public model documentation, not internal review records, but the pattern is worth borrowing as a caution. The fields people skip are often the fields that matter most. If your “not covered” box is empty, treat that as a question, not an answer.

One more line belongs in any record for a feature that takes actions rather than just producing text. An output that reviewers judged acceptable is not evidence that an agent actually did what it reported. An anonymous developer on the OpenAI Developer Community described their team’s biggest reliability problem as agents reporting work as done when nothing had happened, including a payment marked as sent when the underlying call had failed. For agents, the record needs evidence of what the system did, such as logs or state changes, kept separate from what it said. And even that evidence, as the same poster acknowledged, “doesn’t prove the answer was right.” Keep sensitive or regulated data out of the record unless you’re authorized to store it there.

Now the limits, which are printed on the board for a reason. This is a proposed, untested aid. Documentation isn’t validation. A record doesn’t prove coverage, settle disagreement, or authorize use, and high-stakes or regulated uses still require whatever testing, review, and approval apply to them. A documented decision can still be wrong. What the record does is make the judgment something the next person can read, question, and change. The takeaway is the question I’d want at the bottom of every one of these: what evidence would make us reconsider?

Your one bounded next decision

You don’t need a new platform or a governance committee to start. Pick one AI-assisted feature your team is about to approve, or already has. This week, write down six things for it: what it’s for, the real outputs you looked at (including the bad and borderline ones) and who picked them, where reviewers disagreed, what you didn’t check, who has the authority to accept the result, and what would make you look again. Date it. Put the model and prompt version next to the date. Then hand it to someone who wasn’t in the room and ask them what they’d rely on.

🎁 Complete Bundle Description

A great AI demo doesn’t mean the work is ready. Who decides what happens next?

The Quality Judgment Record gives teams a simple way to check AI work and write down what they find. It helps us see what worked, what failed, what wasn’t tested, and where people disagree. It also helps us name who can make the final decision and what might cause us to review it again.

This bundle includes a step-by-step guide, an Excel workbook, a two-page decision record, and a meeting field sheet. Each tool has a different job, but they work together to make our decisions easier to examine and explain.

The goal is simple: help the next person understand what we checked, what we don’t know, and why a decision was made.

The kit doesn’t test or approve AI systems for us. A completed record doesn’t prove that an AI system is safe, reliable, or ready to use.

Learn how to review AI work before making a decision.

This 12-page guide walks us through the questions behind a good review. It explains how to choose examples, spot missing tests, handle disagreements, and decide when another review is needed. It also includes research findings, a sample review meeting, and a filled-out example to show how the process works.

Quality Judgment Record Guide

763KB ∙ PDF file

Download

How it fits the bundle: Start here to learn the process. Then use the Workbook, Field Sheet, and Record to put it into practice.

Download

Keep the evidence organized so nothing important gets lost

This Excel workbook gives us one place to track AI test examples, expected results, known failures, and reviewer disagreements. We can also record changes to our standards, save model and prompt versions, and note when a decision needs another look. It includes four made-up examples to help us learn and a chart showing why a small set of passing tests can miss problems.

Quality Judgment Record Workbook

291KB ∙ XLSX file

Download

How it fits the bundle: Use the Workbook to collect and track the details. Then bring the key findings into the Record so the next reviewer can understand them.

Download

Put the evidence, open questions, and decision in one clear record

This two-page form captures what the AI feature is meant to do, what was checked, what failed, and what remains unknown. It also records where reviewers disagree, who has the right to decide, and what would cause another review. The result is a document the next person can read without having to start from scratch.

Quality Judgment Record Record

280KB ∙ PDF file

Download

How it fits the bundle: Use the Workbook for the full details and the Decision Record for one clear summary. It records a person’s decision but doesn’t grant approval or prove that the decision was right.

Download

Bring the right questions into the room before anyone says yes

This printable sheet helps guide an AI review meeting. The front has space to write down what was tested, where people disagree, what wasn’t covered, and who can decide. The back has eight questions that help the team challenge its first impressions and think about what evidence might change its mind.

Quality Judgment Record Field Sheet

265KB ∙ PDF file

Download

How it fits the bundle: Keep the Field Sheet on the table during the meeting. Afterward, save the important details in the Workbook and Decision Record.

Download

The last decision in the series

This series started with a simple question: who was the last human who actually read the thing? Every episode since has followed that person a little further. Into the handoff, where finished work still has to be owned by someone. To the approval button, where a click can happen without a judgment behind it. Past the scoreboard, where usage stands in for outcomes. Into the bench of reviewers we have to train, and onto the slide whose voice someone has to claim.

This last one lands where all of them were pointing. AI makes producing work cheap. It doesn’t make deciding cheap, and it doesn’t make the decision anyone’s job by default. If there’s one thing I’d want you to carry out of these seven weeks, it’s three sentences you can say in your next review. Agreement shows we share a view, not that the view is right. Passing ten examples tells us about ten examples. Who decides is a separate question from what the tests show.

The demo is not the decision. The decision is a person, a date, and a reason the next reviewer can read, question, and change.

As we close The Judgment Layer, I hope one idea stays with us: AI can help produce the work, but people still have to understand it, question it, and own the decisions that follow. That responsibility is also at the heart of my new book, Collaborate Better, which is finally available for purchase! The book explores how we can build stronger teams, make better decisions, and turn shared understanding into meaningful action. If this seven-part series has challenged how we think about leadership, teamwork, and accountability, I invite you to continue the journey at CollaborateBetter.us.

Regards,

Mark 👋

Citations And Research Binder

428KB ∙ PDF file

Download

This Citations and Research Binder supports Judgment Layer Episode 007: *The Demo Is Not the Decision*, the final episode of the series. It gathers the research behind the article's claims about how testing choices affect what we find, why the same AI prompt can produce different results, how quality standards can change during review, and why reviewer agreement doesn't guarantee correctness. It also explains the math showing how ten successful tests can still miss an infrequent failure. The binder separates published research, practitioner experiences, NIST governance guidance, and Empathy Engine's own recommendations, showing what each can and can't prove. It supports the **Quality Judgment Record** by explaining why teams should document their test examples, selection methods, known failures, disagreements, untested conditions, decision authority, and reasons to review a decision again. The Quality Judgment Record remains an untested documentation aid, not a proven way to improve decisions or certify AI systems. Ten passing examples don't establish reliability, a completed record doesn't grant approval authority, and high-stakes or regulated uses still require the testing and approval that apply to them. The binder's purpose is to keep the evidence, its limits, and the questions we haven't answered visible to the next reviewer.

Download

Previous:

More Content to Discover:

👋 Welcome to my paid subscriber-only edition of Empathy Engine (🔒 Leader’s Dispatch). Each week I build evidence-forward tools for product leads who need to say no, defend tradeoffs, and lock in decisions before they get rewritten later.

Maya Chen found the sentence three lines into Dev’s mid-cycle development notes. (Maya and Dev are fictional, built to illustrate this series.)

“You need to be more strategic.”

She had written it twenty minutes earlier, and it had felt finished. It was short, it was serious, and she believed it was true.

I’ve given feedback along the lines of “You need to take ownership.” Looking back, that sentence leaves too much for the other person to supply. If I mean “raise the dependency before the planning meeting,” I need to say that. I also need to check whether resolving it requires authority they don’t have. My concern about the unfinished work might still be valid. But I haven’t made the request clear just because I’ve named a quality I want to see in the person.

Research Binder: the receipts, methodology notes, and source boundaries are compiled at the bottom of this post.

Last week, Maya reopened one feedback conversation with Dev and found it less settled than it felt. One of her three examples was wrong, Dev had understood her ask differently, and the conditions around the work had shifted.

Previous:

Now she was writing again, and she noticed something about her own draft. The sentence named a judgment. It didn’t say what it was asking for.

Last week I promised to look at what a feedback message points at. The research narrowed that into a more useful question: what does the message itself specify?

Read Maya’s sentence again and ask what it leaves unstated.

• Which work? It names no forecast, no draft, no decision.

• What standard, and how was it set? “Strategic” names a quality without saying what it looks like in Dev’s role, or whether anyone ever said so.

• What evidence? Nothing in the sentence says what Maya saw.

• What next? Nothing says what should change in the next forecast.

Two cautions keep this check honest. First, missing from the message doesn’t mean unknown to Dev. This is a check on the words on the page, not on his mind. Maya hasn’t sent them, so there is no response to read into.

Second, the problem isn’t the word “you.” Rewrite the line as “More strategic thinking is needed” and the pronoun is gone. Every question is still open.

A clear judgment can leave the expected change unstated.

An average isn’t a forecast

First, the same counterweight as last week.

In 1996, Avraham Kluger and Angelo DeNisi published a meta-analysis of 131 papers and 607 effect sizes, each comparing a feedback group with a no-feedback control. The studies were published before 1996, and most were laboratory research.

Feedback helped on average. The weighted mean effect was d = 0.41: feedback groups outperformed no-feedback controls on average. That’s a standardized effect size, not a 41% improvement.

Study effects also varied. Over a third of the effect sizes were negative, meaning the feedback group performed worse than its control in those comparisons.

Both facts describe studies, not conversations. Negative effects are not a workplace failure rate. An average across studies cannot predict any one conversation, including Maya’s.

The same paper proposed one explanation for the variation: feedback may draw attention away from the task and toward the self. The authors noted that attention to the self was not directly measured, so I looked for tests that compare person-focused and task-focused wording directly.

Changing “you” isn’t the whole question

The cleanest test I found changed almost nothing. In 2022, researchers Gok and Fyfe ran two experiments in which people learned invented scripts and received feedback after each answer. The wording was either self-focused or task-focused, crossed with success or failure feedback.

After a wrong answer in Experiment 1, one version read “You answered this question incorrect!” The other read “The answer was incorrect!” In Experiment 2, the feedback also displayed the given and correct answers.

Neither experiment found a statistically significant overall effect of wording focus on learning. Experiment 1 involved 203 undergraduates; Experiment 2 involved 324 adults recruited online through Mechanical Turk. In Experiment 2, a self-reported measure of ego threat didn’t differ by wording focus either.

Two limits cut in opposite directions. No significant difference does not establish equivalence; a small effect could exist that these experiments couldn’t detect. And these were invented-script learning tasks, not managerial feedback conversations.

So these experiments don’t support a pronoun shortcut. Removing a pronoun is not a tested workplace remedy. Examine the judgment and the information, not “you” alone.

What did the record actually include?

How often does written feedback specify these things? The best evidence I found is narrow.

In 2021, Lee and colleagues analyzed the written narrative feedback records of one emergency-department training program: 928 records from 14 preceptors. The researchers coded each record against SBIA, a four-part structure: situation, behavior, impact and action.

Share

Seven of the 928 records, 0.75%, contained all four SBIA elements. The other 921 didn’t record the full set. More than half, 53.99%, described behavior only.

Read the scope before the number. This is one program’s written records, so it doesn’t establish prevalence elsewhere, and it doesn’t measure person-focused wording. Subsequent performance wasn’t measured. And written records may not capture everything discussed aloud.

The caution runs the other way, too. A complete record doesn’t establish an effective conversation. Check the recorded information. Don’t infer the outcome.

What the audit offers Maya is modest: measured against four parts, most of one program’s records named a behavior and stopped there.

The intervention isn’t the ingredient

The strongest outcome evidence I found comes from health care, where feedback is tested as a whole intervention.

Audit and feedback means giving health professionals summarized data on their own clinical performance, sometimes with benchmarks or action plans attached. It’s a category of interventions, not one identical intervention in every trial. A Cochrane review by Ivers and colleagues, updated in 2026, includes 292 studies; its main estimates draw on 558 outcomes in 177 of them. Searches ran to June 11, 2020.

Read more

Plain-English premise: AI can make customer voice sound finished before anyone can say where it came from. Someone still has to know what it can support.

Judgment Seat: The researcher or PM who presents customer voice they didn’t entirely produce, and everyone who inherits the slide after them

👋 Last week’s closing promised an episode about “customer voice” that may be partly or wholly synthetic. The research changed my mind again. It didn’t show that synthetic customer voice is fake, or that teams routinely pass it off as real. It showed that where material came from and what it can support are separate questions, and a slide’s appearance can’t answer either. Episode Five separated assisted output from independent capability; this week turns to the evidence the work stands on

Research Binder: the receipts (citations + source notes) are compiled in a PDF at the bottom of this post.

⚡ Pressure

AI can produce fluent customer voice faster than anyone can say where it came from.

🎯 Payoff

This issue gives you the Evidence Provenance Line: five fields that keep a customer-voice claim’s origin, AI role, checks, supported use, and limits attached to it

Pressure Moment

A research team gave an AI model 122 transcripts from 61 adults living with heart failure: one clinical consultation and one follow-up interview per person. The model proposed themes and picked representative quotations to illustrate each one. When the team compared its themes with an independent human-led analysis, every human theme had an AI counterpart. Then the first author opened the transcripts and checked the quotations by hand, one at a time. Most matched. Some quotations had been reworded, and others contained words no participant said. Finding a theme counterpart hadn’t verified the words beneath it

I remember reviewing a slide that said customers found onboarding confusing. There was a quotation beneath it, and we could trace that quotation to an interview. What we couldn’t trace was the move from one person’s experience to a statement about customers generally. I asked what else supported the heading. The quotation stayed useful, but the claim above it needed narrowing.

Two Questions on Every Slide

That study separates two questions a slide tends to blur: where did the material come from, and what can it support for this use? An interview, an AI summary, a simulated panel, and a blend of all three can produce sentences that look identical inside quotation marks. They aren’t identical as evidence.

A fluent slide still needs an inspectable source. That isn’t a verdict on any origin: a real interview can be unrepresentative, and a grounded simulation can perform well on a bounded task. Let the evidence travel with the claim, because origin alone doesn’t establish validity.

Name the Source

Each origin needs its own check. For direct human material, locate the original record. For AI-assisted synthesis, check the output against those records. For generated responses, ask what inputs and comparisons support them. For mixed material, ask what each part contributes and how each was checked.

The four-category split is my editorial framework, but survey research already draws the same line. AAPOR’s Code of Professional Ethics and Practices, revised June 2026, says AI-generated cases “are not research ‘participants’” and “must be identified as having been created through artificial intelligence.” When a sample mixes AI-generated and human data, the report must “clearly identify to the reader that not all participants were human.” Reports must also state the task AI performed and when its output was validated, and “disclose if this information is unknown or not available.”

The Code expects adherence from all public-opinion and survey researchers, regardless of AAPOR membership. It doesn’t name product teams. Applying its disclosure principles to product slides is my adaptation, and the Code doesn’t validate the proposed aid. A source label opens the evidence question. It doesn’t close it.

Every Human Theme Had a Counterpart

So Hyeon Bang and colleagues (JAMIA, August 2026) ran GPT-5.2 through an eleven-prompt protocol on 61 clinical consultations and 61 reflexive interviews, without the human codebook. All seven human themes had at least one similar counterpart among the AI’s nine, though overlap across theme pairs was moderate (0.44 by human raters, 0.51 by AI). Of 123 batch-level quotations, about 68% were verbatim, 20% paraphrased, 3% truncated, 6% partly hallucinated, and 3% fully hallucinated.

Theme counterparts don’t verify the quotes, and a slide can easily borrow confidence from one to cover the other. Calling the remaining 32% fake would erase distinctions the researchers made; a paraphrase can preserve meaning. But a reworded sentence inside quotation marks is presented as something a person said. The authors also noted that some verbatim quotations didn’t clearly support their assigned themes. A real sentence can still be the wrong evidence.

This was a researcher-led workflow, and the errors are a reason for its human checkpoints, not the result of skipping them; none of the nine final cross-cutting quotations was classified as hallucinated. It’s one study, one model, one clinical population. And even a quotation that passes both checks can’t show how widely an experience is shared. Check the quote against the record. Then check whether it supports the theme.

Plausible Answers Still Need Comparison

Generated responses may draw on real interview transcripts, but they aren’t recorded answers from participants. To assess how well they stand in for people, compare them with human answers on the same task. Pew Research Center ran that comparison on September 30, 2026.. It gave Claude Opus 4.6, the best-performing model it evaluated, the demographics and 2025 political typology answers of real panelists, then compared those “digital twins” with human results across three survey waves.

Even with real profile inputs, a gap remained: an average error of 12 percentage points per question across answer options (rounded from 12.4) over nearly 300 questions, and above 15 points on about 28% of them. Subgroup errors ran larger, and people chose “not sure” about four times as often as their twins. These are aggregate results on opinion questions, not individual predictions or product use. Ask for the comparison: what human results were used, how closely did the estimates agree, and what use does that comparison support?

Real Inputs Can Change the Result

The strongest counterpoint comes from Park, Zou, Kamphorst and colleagues (arXiv preprint, revised June 2026). They built GPT-4o agents from interviews averaging about two hours with 1,052 US adults, then tested each agent against its own person on General Social Survey attitude items held out of the agent’s inputs

Share

Interview-grounded agents matched about 66% of individuals’ held-out answers (65.67%), against a random-choice baseline near 30%. The “83%” you may see cited is relative to participants’ own two-week retest consistency: 0.83 for interview-grounded agents, 0.74 for demographics-only. Neither is raw accuracy, and the final sample skewed more educated, female, and Democratic than its census-quota targets.

Grounding helped here. The tested outcome still matters: these were attitude items, not product use or purchasing. Pew and Park don’t belong on one scoreboard, because their inputs, tasks, and units of evaluation differ. For any simulation, disappointing or impressive, inspect the inputs and the test, and ask what use the result supports.

A Bare Label Can Discount the Truth

The obvious fix is to stamp “AI” on anything AI touched and let readers adjust. Altay and Gilardi (PNAS Nexus, 2024) tested what a stamp like that does. In two preregistered experiments with 4,976 participants in the United States and United Kingdom, people rated news headlines, some labeled “AI-generated.”

The label lowered perceived accuracy for true and false headlines alike, including ones a human wrote. The effects were small on a 6-point scale, and the study didn’t directly test whether people got better at telling true from false, or anything about workplace readouts. People assumed the label meant full automation; when it explained a limited AI role, the penalty weakened. On a slide, that suggests a bare “AI” tag could discount a faithful synthesis as easily as it flags a weak simulation. That’s my inference, not a finding. Describe what AI actually did, and keep the evidence and limits visible.

A Pointer Isn’t a Tag

If a tag shifts trust without clearly helping judgment, what helps a reader check? Kambhamettu, Flores and Head (CHI EA ’25) linked each phrase of AI medical-record summaries to its source passage. Twenty analyzed readers used both interfaces, counterbalanced, after being warned about hallucinations, on deliberately tricky select-all-that-apply questions.

On hallucinated summaries, correct answers rose from 12.5% without links to 70% with them, and both conditions included the summary and the source. That’s a large effect in one small, constructed task, not a workplace error-reduction rate, and AI-generated links can themselves be wrong.

A source link invites checking. It doesn’t certify the claim. A tag tells you what touched the material; a pointer lets you go look. On a slide, keep the source or comparator traceable for authorized review. Privacy and consent still govern access, and an inspectable reference needn’t mean circulating raw interview records.

🧭 The Judgment Move

Orienting sentence: The Evidence Provenance Line is five fields that travel with a customer-voice claim before it informs a decision: origin, AI role, what it was checked against, supported use, and unresolved limits.

Subscribe now

Field 1, Origin: Is this direct human material, AI-assisted synthesis, generated responses, or mixed material? Where did each component come from? Basis: professional guidance (AAPOR Code) for identifying AI-generated and mixed material; the four-way split is an editorial adaptation.

Field 2, AI role: What did AI do: summarize, code, quote, or generate? Name the task, not just the tool. Basis: AAPOR Code (III.A.12); in headline experiments, explaining a limited AI role weakened the penalty a bare label created (Altay & Gilardi).

Field 3, Checked against: What source or comparator was used, and what was checked? A quotation checked for fidelity to its transcript and for support of the theme it illustrates. Simulated estimates against human results for the same task. Basis: Evidence; checking against the record is how quotation errors were found (Bang et al.), comparison with people is how simulation gaps became known (Pew), and phrase-level links improved correct answers in one tested setting (Kambhamettu et al.).

Field 4, Supported use: What does the evidence support for this use, and what is it not for? This field has to be able to say “not for this decision.” Basis: Design inference, adapted from AAPOR’s rule against claiming “greater confidence” than the data and methodology warrant.

Field 5, Unresolved limits: What remains unknown or unestablished? Unknown is an acceptable answer. Say so, and let it limit the supported use. Basis: AAPOR Code (III.A.11, III.A.12).

Here’s an illustrative provenance line based on the Bang study. The supported-use boundary is my interpretation, and the unresolved question is mine, not a finding the authors reported. Origin: AI-assisted synthesis of 61 clinical consultations and 61 reflexive interviews. AI role: GPT-5.2 proposed themes and selected representative quotations. Checked against: every quotation checked by hand against its transcript; themes compared with an independent human-led analysis. Supported use: naming themes worth examining in this population; not for estimating how common any experience is. Unresolved: whether quieter participants’ experiences were underweighted.

The study found verbatim quotations that didn’t clearly support their assigned themes. Field 3 therefore needs to record both whether the quotation matches the source and whether it supports the theme. Field 4 asks whether the checked evidence is sufficient for the intended use. A traced quotation doesn’t settle those questions by itself.

Done state: the claim carries its five fields and a source reference traceable for authorized review, or a named owner for each field still unknown. Whether the next reader understands them, or decides better because of them, is a separate question.

Constraint: Using it responsibly requires room to say “unknown” and “not for this decision” without treating either answer as an accusation against the person who produced the material

I’d recommended a change to how a team worked, and someone asked what supported it. My first instinct was to explain my experience. Then I realized they were asking what made the recommendation appropriate for this team. I needed to separate what I’d seen elsewhere from what we’d actually established here. Their question helped me make a more specific case.

Limitation: the line can’t establish that material is accurate, representative, or predictive, or that a decision improved. Disclosure makes a claim inspectable; it doesn’t establish it. In the research behind this issue, I found no direct evaluation of this bundle. It’s an unvalidated discussion aid, not a score, certification, or compliance check.

🧪 What Holds Up

Primary claim label: Evidence for the separable findings; Design inference for the Evidence Provenance Line as a bundle.

Basis: AI-selected quotations weren’t reliably verbatim even when every human theme had a counterpart (Bang et al.). Simulated estimates diverged from human results despite real profile inputs (Pew), and agreement depended on the inputs and the test (Park et al.). Bare AI labels lowered perceived accuracy for true and false headlines alike (Altay & Gilardi); phrase-level source links improved correct answers in one small study (Kambhamettu et al.)

Counterpoint, labeled Evidence: AI-assisted synthesis and grounded simulation performed well on specific tests. Interview-grounded agents matched about 66% of individuals’ held-out attitude answers against a 30% chance baseline (Park et al.), and every human theme had an AI counterpart under human checkpoints (Bang et al.). Usefulness belongs to a specified task and comparison, not to the label “AI” or “human.”

Scope and transfer limit: this evidence comes from clinical transcripts, US public-opinion questions, attitude survey items, news headlines, and medical-record summaries. None of it comes from product research readouts or the people who review them.

What remains unknown: how often synthetic, AI-assisted, or mixed customer voice reaches decisions, labeled or not; what it costs when it’s wrong; whether agreement transfers to observed product behavior; whether AI synthesis underweights minority voices; and whether this line improves anything.

🛑 Do Not Use When

You want to score, rank, or audit individual researchers. The line examines a claim, not a person.

You want a completed label to make a decision defensible. It can make a claim inspectable; it can’t make it right, certify it, or establish compliance.

Sharing the source would go beyond authorized access. Traceable for review doesn’t mean circulating raw interview records.

Judgment Seat

While facilitating a planning session, I summarized a customer finding someone else had supplied. A follow-up question came to me, and I realized my repetition had made it sound like I could vouch for the evidence. I brought the question back to the person who owned the research. Since then, I’ve tried to make the source clear when I carry someone else’s finding into a discussion.

Who generated the work: a researcher, a synthesis tool, a simulation, and increasingly more than one of them.

Who must understand it: the presenter owns the account of what produced the claim and what was checked. The person who inherits it owns the use they propose for it.

Who can stop it: the leader responsible for the decision, who has to make stopping an unsupported use an acceptable outcome.

Who pays if it is wrong: the team that builds on a voice nobody traced, and the presenter left defending a claim without its evidence. None of those responsibilities disappears when a slide is forwarded.

🎁 Your Entire package: Evidence Provenance Line + Slide Label

A real customer quote can still sit beneath a claim it can’t support. This two-part kit helps you ask where the evidence came from and what it can tell you.

The PDF guide explains the steps. The Excel workbook gives you a place to record five things: the source, AI’s role, what was checked, what the evidence supports, and what remains unknown. It also creates short labels you can copy beneath a claim on a slide, so its source and limits travel with it.

Use the pair when you build, review, or inherit a presentation. This is an untested discussion aid. Filling it out does not prove a claim is sound.

Practitioner Guide | PDF

This 10-page guide shows you how to use the Evidence Provenance Line and place its label on a slide. It includes a quick reference, step-by-step instructions, and six worked examples.

Learn how to describe human and AI material, trace a quote, record checks, and state what a claim is—and isn’t—for. The guide also covers missing sources, mixed material, and claims that need a fresh review. It explains how to keep sources traceable without sharing private records.

How it complements the workbook: Keep the guide beside you as you fill in the workbook or explain the five questions to a teammate.

Evidence Provenance Line + Slide Label Guide

806KB ∙ PDF file

Download

How it complements the workbook: Keep the guide beside you as you fill in the workbook or explain the five questions to a teammate.

Download

Evidence Provenance Workbook | Excel

This workbook turns the five questions into a record your team can revisit. Add one claim per row, then record its source, AI’s role, checks, supported use, and open questions.

Eight working tabs include space to trace quotes, record comparisons with human results, list sources, separate mixed material, and track changes. Six completed examples show you how to start. The workbook also creates short slide labels from your entries.

Evidence Provenance Line + Slide Label Workbook

969KB ∙ XLSX file

Download

How it complements the guide: Use the guide to understand each question, then use the workbook to keep your answers attached to the claim as it moves between slides and people.

Download

This Week

Three actions for the judgment owner: Pick one customer quotation in a current deck and trace it to its source record, noting whether it’s verbatim, paraphrased, or not found. For one customer-voice claim, write its origin and AI role in a single line beneath it. Before your next readout, review one claim’s supported use, and say “not for this decision” if that’s where the evidence lands.

One action for the system owner: Decide in writing that “unknown” and “not for this decision” are acceptable outcomes of review, and that no provenance line will be used to score a person.

One action for learning and provenance: Check that the source reference and unresolved limits stay attached in the version of the deck that actually gets shared.

📎 Forward This

Customer voice on a slide can be an interview, an AI summary, a simulation, or a mix. Before it informs a decision, ask where it came from, what it was checked against, and what it can support.

🔚 Closing

The voice arrived on the slide already sounding certain.

Knowing whose it is, and what it can carry, is still human work.

Next week, Episode Seven closes the season. Ten Examples Before the Argument asks what “good” means when output is probabilistic and a written requirement no longer settles quality. Because once we know where the evidence came from, we still need a definition of good.

This question is part of the larger challenge behind my upcoming book, Collaborate Better: From Silos to Synergy, How to Build Unstoppable Teams. Better collaboration begins when context, ownership, and judgment travel with the work.

Learn more at CollaborateBetter.us.

Regards,

Mark 👋

Sources

Bang, S. H., Han, S., Reading Turchioe, M., Ellison, M., Dai, S., Happ, M. B., Russell, D. & Masterson Creber, R. (2026). Developing and evaluating human-led and large language model–assisted hybrid deductive–inductive workflows for qualitative analysis. Journal of the American Medical Informatics Association. doi:10.1093/jamia/ocag123

Chapekis, A., Lau, A., Bestvater, S., Shah, S., Mercer, A. & Smith, A. (2026, September 30). Can AI Stand In for Human Survey-Takers? Not Really. Pew Research Center. pewresearch.org

Park, J. S., Zou, C. Q., Kamphorst, J. et al. (2026). LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals. arXiv:2411.10109v3, June 28. Preprint. arxiv.org/abs/2411.10109v3

Altay, S. & Gilardi, F. (2024). People are skeptical of headlines labeled as AI-generated, even if true or human-made, because they assume full AI automation. PNAS Nexus 3(10): pgae403. doi:10.1093/pnasnexus/pgae403

Kambhamettu, H., Flores, J. & Head, A. (2025). Traceable Texts and Their Effects: A Study of Source-Summary Links in AI-Generated Summaries. CHI EA ’25. doi:10.1145/3706599.3719830

American Association for Public Opinion Research (2026, revised June). AAPOR Code of Professional Ethics and Practices. aapor.org

Citations And Research Binder

486KB ∙ PDF file

Download

This Citations and Research Binder supports Judgment Layer Episode 006: *Whose Voice Is on the Slide?* It gathers the sources behind the article’s claims about quotation accuracy, comparisons between simulated and human answers, how AI labels affect readers’ ratings, and how source links can help readers check a summary. It distinguishes direct human material, AI-assisted synthesis, generated responses, and mixtures. The binder supports the Evidence Provenance Line’s five fields: origin, AI role, what was checked against, supported use, and unresolved limits. It separates published studies, research awaiting peer review, survey-research guidance, and Empathy Engine’s own design choices, while explaining what each can and can’t establish. The Evidence Provenance Line and Slide Label remain unvalidated discussion aids: naming a source doesn’t prove a claim is accurate or representative, agreement on one task doesn’t establish usefulness for another, and completing the record doesn’t show that a decision will improve. Unknowns remain visible and limit what the evidence can support.

Download

Previous:

More Content to Discover:

👋 Welcome to my paid subscriber-only edition of Empathy Engine (🔒 Leader’s Dispatch). Each week I build evidence-forward tools for product leads who need to say no, defend tradeoffs, and lock in decisions before they get rewritten later.

Maya Chen walked out of the one-on-one feeling good about it. (Maya and Dev are fictional, built to illustrate this series.)

She had prepared for two days. Dev, a senior analyst on her team, had been sending customer-segment forecasts to stakeholders before checking the assumptions underneath them. She brought three dated examples and one ask: confirm the source definitions before any draft leaves the team.

Dev asked a sharp question about the second example, then said, “That makes sense. I’ll check the definitions first.” Back at her desk, Maya typed two words into her notes: landed well.

Research Binder: the receipts, methodology notes, and source boundaries are compiled at the bottom of this post.

Eleven days later, the next forecast arrived. The assumption she had flagged was still in it. A new unchecked one sat in the section Dev had reworked.

On the drive home, Maya replayed the conversation, hunting for the sentence that went wrong. But the replay only reviewed the conversation. The question she needed was simpler and harder. What changed? That question has four parts:

• The message. What was said?

• The reaction. What did each person report?

• The work. What happened afterward?

• Other explanations. What else could explain it?

A finished meeting gives you a date, not the rest of the answers. Look at comparable work afterward, and where evidence is missing, write Unknown.

When someone agrees with me, I still need to understand what they think we agreed to.

A positive average, widely different results

Before going further: feedback is not the villain of this series.

In 1996, Avraham Kluger and Angelo DeNisi published a meta-analysis in Psychological Bulletin covering 131 papers and 607 effect sizes. Each effect compared performance with feedback against performance without it. The studies were published before 1996, and most were laboratory research.

On average, feedback helped: the weighted average effect was d = 0.41. That is a standardized effect size, not a 41% improvement in anything.

And over 38% of those 607 effects were negative. “Negative” here means the feedback group performed worse than the no-feedback comparison group. It does not mean an unhappy reaction, or a drop from someone’s own earlier performance.

Both facts are true at once, and neither forecasts Maya’s conversation. The 38% is a share of effect sizes from mostly older laboratory research, not a failure rate for today’s one-on-ones.

The tail wasn’t a fluke. When the authors removed outlying data, about a third of the remaining effects were still negative. They also found that, across studies, messages coded as discouraging showed lower average effects. They interpreted that as attention shifting from the task toward the self. Attention itself was never measured, so treat it as a theory about a pattern, not a window into Dev’s head.

Two checks, not one

Maya had a record of her impression. She did not have Dev’s account, and she had his reaction only as she remembered it: a sharp question, then “That makes sense.”

Separate doesn’t mean unrelated. In one field study of 145 leaders receiving 360-degree feedback, more negative immediate reactions went with worse ratings from others about a year later, after accounting for where each leader started. One study, ratings only, an association rather than a cause. Reactions matter. They just aren’t the same measurement as the work.

Accounts diverge, too. In several studies, people receiving negative feedback judged it as less accurate than the people who gave it. That is disagreement, not proof that either side is wrong. Someone can leave a meeting feeling respected and still be unclear about what was supposed to change.

A good feeling does not establish a better result, and a tense meeting does not establish a worse one. Describe the recipient’s reaction in their own words, without labels like defensive or unmotivated. Examine the work against stated criteria. Neither check establishes the cause.

When Maya asked for Dev’s account, it was specific: “I thought the definitions check was for anything going to the board. This one was an internal draft.” That is a different understanding of the same ask. Both accounts go on the page. Neither has to win.

Was the message accurate?

Then the question that can feel settled once you’ve prepared carefully: was the message accurate? Preparation measures effort, not correctness. When Maya reopened her three examples, two held up against the files. The third did not. That assumption had been checked, and the check was documented in a tab of the shared model she hadn’t opened. Dev knew, and he hadn’t mentioned it in the room. Two examples still support her concern. Whether they show a broader pattern, three documents can’t say.

Scullen, Mount and Goff studied developmental multi-rater ratings of managers in two large samples. Their model attributed 62% of rating variance in one sample and 53% in the other to idiosyncratic rater effects, meaning patterns specific to the person doing the rating. The managers’ measured performance accounted for 21% and 25%.

Share

Those percentages describe variation in ratings, not feedback conversations, and they don’t say what share of ratings were wrong. Rater-specific patterns can include valid differences in what people observed. Calling all of it bias would go beyond the finding.

What it does support: a rating is not a direct reading of performance, and neither is a manager’s summary of a draft. Make the judgment inspectable: the work, the criteria and both accounts. That is how Maya’s third example fell apart.

What else changed?

Maya’s first conclusion was that the feedback didn’t work. Maybe. But Kluger and DeNisi excluded simple before-and-after comparisons from their analysis because of serious threats to internal validity. A sequence tells you when two things happened, not why.

So Maya listed what else had moved in eleven days, separating what she observed from what she guessed.

• The task. Observed: the new forecast covered a segment with messier source data. Unknown: whether that made the check harder.

• The leader. Observed: Maya canceled their mid-week check-in while traveling.

• The conditions. Observed: a stakeholder pulled the deadline forward three days. Reported by Dev: source-file requests go through another team’s queue. Unknown: how long his took.

• Other changes. Observed: her deputy had reviewed Dev’s earlier drafts. Maya reviewed this one herself.

• Dev’s part. Observed: he didn’t raise the problem with example three, and the new unchecked assumption was his.

Read more

Plain-English premise: AI can make today’s work better without telling you whether the person doing it is getting better. That second question needs its own evidence.

Judgment Seat: The future reviewer: the newer colleague whose judgment is still forming, and the lead deciding when to trust it.

👋 Last week’s closing promised an episode about the tasks that once formed future judgment, and what happens when they’re the first ones automated. The research changed my mind again. It didn’t establish that AI is quietly removing the practice that turns a newcomer into a reviewer. It showed something more useful: better work with AI and a more capable person without it are two different results, and the way the help works shapes which one you get.

Research Binder: the receipts (citations + source notes) are compiled in a PDF at the bottom of this post.

⚡ Pressure

AI can finish the first pass before a newer colleague has learned how to judge one.

🎯 Payoff

This issue gives you the Reviewer Bench Inventory: six questions that locate where newer colleagues practice judgment, get feedback, and get checked without the tool.

Pressure Moment

In late August 2026, Business Insider and Forbes reported that Valon, an AI startup, told most of its new hires to learn the job without AI until their manager was confident they could tell when the AI was wrong. Engineers were exempt, because all of their code already goes through peer review. According to the reporting, new hires had started asking tenured colleagues for help again.

That’s one company’s account, with no outcome data, and nothing in this issue supports restricting AI for newer people. What interests me is the test inside the policy. It didn’t ask whether new hires could produce good work with AI. It asked whether they could recognize when the AI was wrong, and it treated peer review as a place that question already gets answered.

I’ve stepped into a review and rewritten a backlog item myself because we needed to keep moving. The acceptance criteria got clearer, and the team had something it could use. What I hadn’t checked was whether the person who wrote it understood why I’d made those changes. I’d helped finish the work, but I’d left their learning unchecked.

Three Windows on the Same Work

When the work improves, it’s natural to assume the person doing it improved too. That used to be a safe assumption, because the draft improved as the person drafting it improved. AI can separate the two, so every result in this issue answers one of three questions.

With AI, how well did the work perform? Without AI, what could the person do independently? After a delay, what capability remained? A successful output answers the first question. It doesn’t answer the other two, and when a team collects only output evidence, its development claim remains unchecked. Count improved work. Check what people can do independently.

This is where the episode separates from Episode Two, which asked whether the sender can explain the work at handoff. That’s still the right gate, but an explanation at handoff is evidence about that work on that day. It isn’t evidence that judgment is forming, and an explanation can be generated too.

The louder version of this story runs on hiring data: Stanford’s Digital Economy Lab found US employment for 22-to-25-year-olds in the most AI-exposed occupations about 19% below comparable less-exposed work, mostly through slower hiring, while a Danish study found no such effect. Those numbers describe who gets onto the bench, not what anyone on it is learning.

The Output Gain Is Real

Start with the first window, where the news is good. You’ve seen “Generative AI at Work” (Brynjolfsson, Li & Raymond, QJE, 2025) in earlier episodes; it’s back for a different reason. In a staggered rollout of an AI assistant to 5,172 customer-support agents, 89% of them working outside the US, resolutions per hour rose 15.2% across all agents and 36% in the lowest skill quintile, a subgroup of the same sample.

Every bar measures performance with AI assistance, though the study does hint at learning. Agents with AI reached in about two months the output untreated agents needed eight to ten months to reach, and during rare outages they stayed faster than their pre-AI baseline. The authors describe those outage results as consistent with durable worker learning.

That’s a real signal, but not a controlled retention test: outages were rare and may not have been random. The output gain is real. Retained capability is a separate question, so count the productivity gain and assess development separately.

The Task Ended. Understanding Still Had to Be Tested.

The second window opens when the AI leaves the room. Shen and Tamkin, “How AI Impacts Skill Formation,” a 2026 preprint by two Anthropic researchers, recruited 52 programmers from a crowd-worker platform, all new to Trio, an unfamiliar Python library. Everyone got two coding tasks and up to 35 minutes. Half could use a GPT-4o chat assistant and half couldn’t, and then everyone took an immediate quiz with AI not permitted.

On average, the AI-assisted group scored 4.15 points lower on the 27-point quiz (d = 0.738, p = 0.010) and wasn’t significantly faster. Three cautions travel with that number. The sample was experienced, not junior: 29 of the 52 had seven or more years of coding, and only two per arm had one to three years. The quiz was immediate, so it can’t say what stuck. And it’s an Anthropic-affiliated preprint.

Its six usage patterns, where conceptual questioners outscored delegators, came from unrandomized groups of two to seven people, so they’re a clue, not a training method. Check understanding after the task, and keep the conclusion tied to the test.

Then the Evidence Pointed the Other Way

If the coding study were the whole story, this would be a warning and nothing more. Contractor and Reyes, “Experimental Evidence on the Learning Impact of Generative AI” (2026 preprint), randomized a US undergraduate sample to learn with AI allowed or forbidden, then tested everyone with AI and other resources barred. 211 students attended the first session, and 204 returned about a week later to answer new test items.

The AI-allowed group was 6.7 percentage points ahead immediately after learning and 5.1 points ahead about a week later. There’s a caveat on the first number: rule violations during the immediate test increased in the AI arm, and the authors estimated they could explain up to about one-third of the initial gain. The two figures are separate between-group estimates, not a measure of how much each student retained.

This is one study and one week, with undergraduates in a lab, on a different task and population from the coding study, so the two shouldn’t be averaged into a slogan. It does show that AI-assisted learning can carry over, which is why the third window is worth opening: check what remains, and when.

The Help Changed. So Did the Result.

A third experiment adds a distinction within one setting: the design of the assistance changed the outcome. Bastani and colleagues (”Generative AI without guardrails can harm learning,” PNAS, 2025) randomized nearly 1,000 high-school mathematics students in Turkey by classroom. Students practiced with GPT Base, an unguided chat assistant; with GPT Tutor, which used teacher-written hints and guardrails designed to guide rather than hand over answers; or without AI. Everyone then sat an exam without AI.

Share

During practice, GPT Base students scored 48% higher than control and GPT Tutor students 127% higher. On the exam, GPT Base students scored 17% lower than control. GPT Tutor students showed no statistically detectable difference from control, and no demonstrated exam advantage either; the authors describe the guardrails as largely mitigating the harm. The biggest practice gain didn’t prove a learning gain.

One detail should bother anyone who picks tools by satisfaction scores: students preferred the unguided tool. It’s high-school math with no delayed result, so the percentages don’t transfer to a product team. The question does: examine how the help works, and check what people can do afterward.

Practice Needs the Right Support

If development isn’t automatic, what builds it? The nostalgic answer is to protect the grunt work, because it was the apprenticeship. Some of it was; some was repetition nobody ever looked at. Even review doesn’t teach by default: in a 2013 Microsoft study, Bacchelli and Bird coded only 12 of 570 code-review comments as knowledge transfer, though learning can happen in ways a comment code wouldn’t capture.

The chart’s three findings come from education, not workplaces. Sinha and Kapur’s 53-study meta-analysis found problem solving before instruction outperformed instruction first on average, g = 0.36, a standardized effect size rather than a 36% improvement, though instruction first did better for grades 2 to 5 and domain-general skills. Kalyuga’s expertise reversal research shows that support that helps a novice can become redundant as expertise grows; true beginners often learn more from a worked example. And Wisniewski, Zierer and Hattie’s 435-study meta-analysis found feedback helped on average and informative feedback helped more, while 17% of measured effects were negative, a share of effects rather than of people.

So the support has to match the learner and the task. The two sequences on the chart are illustrations, not a tested protocol. Counting repetitions doesn’t show what the practice teaches, and AI can either supply examples and feedback or do the attempt for the learner; only the workflow tells you which. Inspect the attempt, the support, and the feedback, and check what the learner can do next.

🧭 The Judgment Move

If the research can’t tell you whether your next reviewers are forming, your workflow can at least show you where they would be. The Reviewer Bench Inventory doesn’t measure anyone; it asks where, inside one consequential review task, a learner practices the judgment that task requires.

Orienting sentence: The Reviewer Bench Inventory is six questions about one consequential review task: what judgment it requires, where a learner attempts it, who gives feedback, what AI changed, where independent capability is checked, and who owns what’s unclear.

Judgment required: What tradeoff, error, or consequence must be recognized? Name the judgment, not the activity. “Review the pricing analysis” is a task; “notice when a comparison set quietly excludes the churned customers” is a judgment. Basis: Design inference.

Practice opportunity: Where does a learner attempt that judgment? The attempt might be independent, guided, or made right after studying a worked example; which fits depends on how far along the learner is. Watching someone else do it isn’t an attempt. Basis: Evidence from education (Sinha & Kapur; Kalyuga); applying it to workplace review is inference.

Feedback: Who explains what worked, failed, or was missed? A thumbs-up on the final version isn’t feedback on the attempt. Basis: Evidence from education (Wisniewski, Zierer & Hattie), with the Bacchelli and Bird review finding as context.

Workflow change: What has AI removed, added, or relocated? AI may remove practice, add it (an explainer, a faster feedback loop), or move it somewhere nobody’s looking. Basis: Design inference; the research reviewed for this issue didn’t establish how AI has changed these opportunities in professional review workflows.

Capability check: Where is independent capability checked, and when? Record the assessment conditions; if the only evidence is work produced with AI, write that down. Basis: Evidence; Shen & Tamkin and Bastani et al. show that assisted and unaided results can diverge, and Contractor & Reyes show that an immediate result and a one-week result are separate measurements.

Ownership: Who investigates an unclear or missing opportunity? Name a person, not a function. Basis: Design inference.

Two rules hold it together. “Not yet known” is a valid response. And an unknown field is a question, not a verdict about a person. The inventory locates development in the workflow and never grades the people inside it.

Done state: for one consequential task, the team can say where a learner attempts the judgment, where feedback happens, and where independent capability is checked, or it has named who will find out.

Constraint: this only works if “we don’t know where that happens” can be said out loud without anyone hearing it as a performance problem, theirs or the learner’s.

When I asked how newer colleagues were developing judgment, the answer was that they worked alongside experienced people. That sounded reasonable until I tried to locate an actual opportunity to make a call and get feedback on it. The experienced people were explaining decisions, correcting work, and keeping delivery moving. What I couldn’t find was a clear moment when the newer colleague offered their own assessment before someone supplied the answer.

Limitation: the inventory can’t prove anyone is learning, show what AI caused, or certify a reviewer as ready, and nobody has tested whether running it helps. It’s an unvalidated team discussion aid whose fields draw on separately studied practices, mostly from education. Investigate what’s unclear before prescribing a fix.

Run It on One Task

A heavy form is a form nobody finishes. Don’t start with the team or the career ladder; start with the next consequential review on the calendar.

Subscribe now

Choose the task. Locate the attempt and the feedback that followed it. Name what the assisted output demonstrates, and give it full credit. Record how and when independent capability is checked. Assign an owner for anything unknown.

The answers tend to point one of three ways. If development is visible, examine whether the opportunity is useful, because an attempt without informative feedback is just repetition. If it has changed, investigate what improved, shrank, or relocated; AI may have moved the practice somewhere better. If development is unknown, gather evidence before deciding on a policy; that beats a ban, a mandate, or a confident assumption. Better output doesn’t settle the development question. For the next consequential task, name where development happens, who will check, and the date they’ll check it.

🧪 What Holds Up

Primary claim label: Evidence for the separation between assisted output and independent capability; Design inference for the Reviewer Bench Inventory as a bundle.

Basis: In randomized studies, results measured with AI and without it can diverge: lower immediate unaided quiz scores after AI-assisted learning (Shen & Tamkin, Anthropic-affiliated preprint), and higher practice scores alongside lower exam scores under unguided AI (Bastani et al., PNAS). Large assisted gains (Brynjolfsson, Li & Raymond, QJE) were measured with AI in the room.

Counterpoint, labeled Evidence: Contractor & Reyes (preprint) found AI access raised unaided scores immediately and about one week later, and the support-agent outage results are consistent with durable learning on the job.

Scope and transfer limit: labs, classrooms, and crowd-worker samples, with follow-ups of about one week at most; the practice and feedback research is educational. None of it comes from Product, Program, Design, or Research teams.

What remains unknown: whether AI has changed attempts or feedback for newer professionals in real workplaces, what remains after months rather than days, and whether this inventory improves anything. It hasn’t been tested.

🛑 Do Not Use When

You want to rate, rank, certify, or monitor individual people. The inventory locates opportunities in a workflow; it says nothing about a person’s capability.

You’re looking for justification to restrict AI for newer staff, or to stop hiring them. The evidence here supports neither.

The task carries no consequential judgment. Inventory the reviews that matter, not every queue.

Judgment Seat

I’ve helped teams get work ready while also coaching the people producing it. Those responsibilities can pull against each other when a deadline makes it easier to supply an answer than explore someone’s reasoning. Once I’d stepped in, I needed to check what the colleague understood, because my contribution was now part of the finished work.

Who generated the work: increasingly, the tool, with a newer colleague steering it.

Who must understand it: that colleague, deeply enough to someday review someone else’s.

Who can stop it: the lead who decides whether the next consequential task includes an attempt, feedback, and an independent check.

Who pays if it is wrong: the lead who makes a readiness decision without knowing where the judgment was practiced or checked, and the colleague whose readiness was assumed.

🎁 Your Reviewer Bench Inventory

Your Reviewer Bench Inventory

Better work doesn’t tell us where people are learning to judge it. That’s why I’m gifting you this three-part kit: a guide, an editable workbook, and a printable field sheet. Use them to explore one important review task, find where practice and feedback happen, and plan what to check next. These tools guide a team discussion; they don’t score people or prove someone is ready to review.

The Practitioner Guide | PDF

This 10-page guide explains the six questions and how to use them with a team. It helps us choose a task where a missed error or poor choice would matter. Examples show the difference between a vague answer and one we can act on. There’s also guidance for running the meeting and handling gaps without blaming anyone.

Reviewer Bench Inventory Guide

474KB ∙ PDF file

Download

Keep it nearby when planning a session or explaining the inventory to a colleague.

Download

The Working Workbook | Excel

This workbook gives the team a place to record answers and keep track of open questions. Four filled-in examples show how the inventory can work across different types of reviews. Separate tabs help us record practice, feedback, and how skills are checked with AI, without it, later, or on a new kind of task. A follow-up log holds the next steps, owner roles, and dates. The Bench Note brings the recorded answers together for the next conversation.

Reviewer Bench Inventory Workbook

109KB ∙ XLSX file

Download

Use it to turn a useful discussion into a record the team can revisit.

Download

The Printable Field Sheet | A4 PDF

This one-page sheet puts all six questions within reach during a meeting. Print it and write down where people try making a judgment, get feedback, and have a chance to work on their own. It includes space for an owner role, an open question, and a follow-up date. “Not yet known” is a valid answer, so the team can name a gap without treating it as someone’s failure.

Reviewer Bench Inventory Field Sheet

279KB ∙ PDF file

Download

Bring it to the next review when you want to start with pen and paper.

Download

This Week

Three actions for the judgment owner: Name the judgment your next consequential review requires. Find the last time a newer colleague attempted it, and who gave feedback. Note where, if anywhere, their independent capability on that task gets checked without AI.

One action for the system owner: Decide in writing that “not yet known” is an acceptable answer on any development review, and that no inventory answer will be used to score an individual.

One action for learning and provenance: The next time AI-assisted work is praised in a review, add one line beneath it: what this shows about the work, and what it doesn’t show about the person.

📎 Forward This

Better work with AI doesn’t prove anyone got better without it. For your next consequential review, name where newer people practice, get feedback, and get checked without the tool.

🔚 Closing

The tool can improve the work on the first day.

Knowing whether the next reviewer is forming is still human work.

Next week, Episode Six widens the question from who can review to what counts as evidence. The working title, Whose Voice Is on the Slide?, points at “customer voice” that may be partly or wholly synthetic. Because knowing who can judge the work only matters if we also know where the evidence came from.

This question is part of the larger challenge behind my upcoming book, Collaborate Better: From Silos to Synergy, How to Build Unstoppable Teams. Better collaboration begins when context, ownership, and judgment travel with the work.

Learn more at CollaborateBetter.us.

Regards,

Mark 👋

Sources

Brynjolfsson, E., Li, D. & Raymond, L. (2025). Generative AI at Work. Quarterly Journal of Economics 140(2):889 to 942. https://academic.oup.com/qje/article/140/2/889/7990658

Shen, J. H. & Tamkin, A. (2026). How AI Impacts Skill Formation. arXiv:2601.20245v2. Anthropic-affiliated preprint. https://arxiv.org/html/2601.20245v2

Contractor, Z. & Reyes, G. (2026). Experimental Evidence on the Learning Impact of Generative AI. arXiv:2607.08849v1. Preprint. https://arxiv.org/html/2607.08849v1

Bastani, H. et al. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. PNAS. https://www.pnas.org/doi/10.1073/pnas.2422633122

Sinha, T. & Kapur, M. (2021). When Problem Solving Followed by Instruction Works: Evidence for Productive Failure. Review of Educational Research. https://eric.ed.gov/?id=EJ1308129

Kalyuga, S. (2007). Expertise Reversal Effect and Its Implications for Learner-Tailored Instruction. Educational Psychology Review 19, 509 to 539. https://www.uky.edu/~gmswan3/EDC608/Kalyuga2007_Article_ExpertiseReversalEffectAndItsI.pdf

Wisniewski, B., Zierer, K. & Hattie, J. (2020). The Power of Feedback Revisited. Frontiers in Psychology 10:3087. https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2019.03087/full

Bacchelli, A. & Bird, C. (2013). Expectations, Outcomes, and Challenges of Modern Code Review. ICSE 2013. https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/ICSE202013-codereview.pdf

Brynjolfsson, E., Chandar, B. & Chen, R. Canaries in the Coal Mine? Stanford Digital Economy Lab, August 2026 update. https://digitaleconomy.stanford.edu/news/canariesaug26/

Humlum, A. & Vestergaard, E. NBER Working Paper w33777, revised March 2026 (Denmark). https://www.nber.org/system/files/working_papers/w33777/w33777.pdf

Valon new-hire AI policy: Business Insider, August 29, 2026, https://www.businessinsider.com/valon-ceo-bans-ai-most-new-hires-job-skills-2026-8; Forbes, August 30, 2026, https://www.forbes.com/sites/rachelwells/2026/08/30/an-ai-startup-bans-ai-and-tells-new-hires-learn-your-job/

Citations And Research Binder

190KB ∙ PDF file

Download

This Citations and Research Binder supports Judgment Layer Episode 005: *The Work Got Better. What Did We Learn?* It gathers the sources behind the article’s claims about work produced with AI, what people can do without it, and what they can still do after a delay. It also covers practice, feedback, and how the right support can change as someone gains experience. The binder supports the Reviewer Bench Inventory’s six questions: what judgment a task requires, where people practice it, who gives feedback, what AI changed, how skills are checked, and who follows up on gaps. It separates published studies, research awaiting peer review, workplace findings, education research, news reports, and Empathy Engine’s own design choices, while explaining what each can and can’t tell us. The Inventory remains an untested team discussion aid: the research doesn’t show that AI always helps or harms learning, that better work proves stronger judgment, or that completing the Inventory proves anyone is ready to review.

Download

Previous:

More Content to Discover:

👋 Welcome to my paid subscriber-only edition of Empathy Engine (🔒 Leader’s Dispatch). Each week I build evidence-forward tools for product leads who need to say no, defend tradeoffs, and lock in decisions before they get rewritten later.

Nina finally had a market with pain, buyers, trust gaps, and workflow mess. The question was no longer whether it was interesting. The question was what it was asking of her.

A few months earlier, that list would have been enough. She would have seen the offer before she saw the evidence, named it, priced it, and started listening for the sound of a Stripe notification. Six episodes of inspection had made her suspicious of exactly that feeling, because the moment a market starts to look like a story is the moment the story starts filling in what the evidence hasn’t.

Her notes from the season covered the kitchen table, and they did not agree with each other. The oldest page held the broad AI consulting offer that owners nodded at and never bought. The newest held Dave’s Thursday, the morning his dispatcher rebuilt the day by hand, and the narrower question it had left her with. When an emergency reassigns a crew, what travels with the job, and who confirms it arrived? Dave had said, twice, that he would probably pay for help with that.

Her phone sat face up beside the pages, open to a post a friend had sent about twenty boring businesses that quietly print money. Nina read three of them before she noticed that none of the twenty mentioned a buyer.

She took a clean page and wrote one line at the top before she let herself open anything else. What I already want to do. Under it she wrote BUILD, and then she circled it, because she knew she would spend the rest of the night trying to prove it.

Nina, Dave, and Dave’s company are a fictional composite built from recurring patterns in the research, not real people or a real business.

During a transformation review, I watched a PI Planning board become the evidence for a verdict the room had already reached. The board was real. The dependencies were real. The progress markers were real. But somewhere between planning and review, those signals stopped being evidence about the work and became proof that the transformation itself was working. The question had quietly shifted from “What does this board actually tell us?” to “How does this board support the conclusion we already prefer?”

This episode assembles the season into one tool, the Boring Market Inspection Kit, and nothing in it is new to you. You have already asked where an opportunity’s confidence came from, where the value in a business actually lives, and whether a pain repeats often enough to show up on the Workflow Pain Index. You have asked what has to transfer when work changes hands, what you are allowed to do, and what would have to move with the work. The Kit puts those questions in one order and adds a record at the end.

Its decision logic isn’t new either, and it doesn’t pretend to be. Stage-gate reviews, kill criteria, and scientific-method approaches to entrepreneurship have handled the mechanics of continuing or stopping for years. The Kit adapts that practice for solo builders in unglamorous service markets and makes you write the decision down before you defend it. By the end you’ll have the Kit and Nina’s completed decision record. It won’t tell you whether a market will work. It will tell you which next move your current evidence can actually support.

Episode 6:

Research Binder: the receipts, methodology notes, and source boundaries are compiled at the bottom of this post.

Boring Isn’t the Signal

The post on Nina’s phone treats boring as the find. This whole season has argued the other way. Boring is scenery. It tells you where fewer people are looking and says nothing about whether anyone there will pay, let you in, trust you, or let you do the work. So the Kit starts from the premise printed across the top of its first graphic. Boring isn’t the signal. Evidence is what you inspect.

Nina stopped sorting her notes by episode and started sorting them by question, and the pile collapsed into five. What exact work hurts, and who actually pays? Can she reach the person who actually decides? What must move, transfer, or be believed when the work changes hands? What is she actually allowed to do? What result would change her course?

The order of the five fields is deliberate. Paid pain comes first, because nothing downstream matters if nobody pays to remove it. Access comes before trust, because a buyer you never reached can’t tell you whether they believe you. The review condition and cap come last, because they force the question every earlier field lets you postpone, which is what you would actually have to operate and what it would cost to find out.

The row across the middle collects the five confusions the season kept running into, each one a pair of things a builder wants to treat as one on the night they most want a yes. The three numbers below it each come from a different population, and the graphic labels them that way. None of them describes your market. Each describes a mechanism worth looking for in your market, and the next five sections take the fields one at a time, beginning with the one most founders answer too fast.

I’d Probably Pay for That

Episode 3 ended with a sentence Nina had underlined. The Workflow Pain Index can inspect the repeated, but paid still has to be validated downstream, and that worksheet was never built to validate it. Downstream is here, and Dave’s own words were the problem. He had said I’d probably pay for that twice, and he meant it both times. Nina had heard the same sentence from owners in Episode 1 about a different offer, and none of them paid.

A technician recalled a version of this on X in September. A modem, router, and server sat in an attic under a hole in the roof about two feet across, in standing water. He told the customer what the fix was, and the customer declined. Three months later the gear had been moved downstairs, and they still wouldn’t pay to replace equipment that had sat in the rain.¹ The pain was visible, expensive, and real, and it never became a purchase order.

The ladder on the graphic has a break in it on purpose, between interest expressed and budget committed, and most of what a builder hears in early conversations lives above that break. The label underneath matters as much as the ladder. It is an illustration, not a measured funnel, because nobody has measured how often small-business pain crosses that gap, and this article won’t pretend to.

What we do have is a caution from a neighboring field. In a meta-analysis of consumer-goods research, hypothetical willingness to pay averaged 21 percent above real, incentive-compatible willingness to pay.² The evidence boundary on the graphic says how far that number travels. It is not a B2B purchase rate or a conversion rate. It is a reason to treat I’d probably pay for that as a statement of intent, which is a different object from money changing hands.

The three questions under the ladder split one fact Nina had been treating as whole. Dave’s dispatcher feels the pain every Thursday. Dave is the one who can authorize spend. The problem already costs the shop rework when a tech arrives without notes and delay when a restaurant waits on a call back. A costly workaround like the dispatcher’s group chat shows the pain is real, and it says nothing about whether Dave will pay a stranger to change it. When Nina asked what Dave had actually committed, the honest answer was nothing yet. In the Workflow Pain Index’s vocabulary, his payment was Not Observed.

That is why the graphic ends on a different question. Asking whether they care gets you sympathy. Asking what they have done gets you evidence, because behavioral commitment already cost the buyer something. Sometimes, though, the first thing a builder learns is that the offer never reached anyone who could make that commitment at all.

Silence Is Not the Same Evidence as No

Dave was easy to reach. Nina had found him through a referral chain three people long, and by now he answered her texts from job sites, which made him a misleading sample of one. The second shop on her list had an office manager who took messages and an owner who was on a roof until six. Her offer, if it got that far, would be a forwarded email in that office manager’s inbox, the corporate void where PDFs go to die, scaled down to a twelve-person shop.

In August, an automation builder posted in a GoHighLevel community after cold-calling HVAC shops with a lead-conversion offer he had already built. The calls ended before the second sentence of his script, with some version of not interested, or I’ve got enough jobs.³ Commenters pointed to peak season and a labor shortage. Shops with full calendars weren’t short of what he was selling.

The map gives Nina’s second shop and the HVAC caller separate paths, because they produced different findings. Path A stops at the gatekeeper and never reaches the decider, which is where Nina stood, and demand is still untested there. Path B reaches the decider and gets a no, which is the HVAC call. That is evidence about this offer, and a no can still mean price, trust, timing, or fit. Peak season is timing.

The buying-center research behind the graphic is narrower than the distinction it supports. A meta-analysis of organizational buying studies found that more complex purchase situations were associated with larger buying centers, r = .40.⁴ That correlation comes from organizational buyers in studies run between 1979 and 1999, with firm size unreported, and none of those studies tested access as something separate from trust or demand. The graphic calls its own distinction an inspection heuristic, which is the right label. In owner-operated firms one person may hold several roles, and at Dave’s shop the dispatcher feels the pain while Dave authorizes the spend, so Dave’s interest and his dispatcher’s frustration are two separate pieces of evidence.

Nina ran the four steps at the bottom on her second shop. She could name the decider. She had no credible path to him, though Dave knew him from the supply house and that might become one. The offer hadn’t reached him, so there was nothing yet to record. No response is not the same as a no, and if you can’t reach the buyer, test access before concluding the market said anything at all. Access gets you into the room, though. It doesn’t make the room trust you.

The Book Walked Out With the Seller

By now Nina’s offer had shrunk to something she could say in one sentence. She would not run anyone’s Thursdays, because the last episode had settled that. She would sit with a shop’s dispatcher for two weeks and build what Dave’s shop was missing, a reassignment card that travels with an emergency job and a step where the tech confirms he has what he needs, and the shop would own it afterward. It sounded portable until she wrote down where the card’s contents currently lived, and most of the answers were a person.

Acquisition threads make this point more bluntly than research does. In a bookkeeping forum in 2025, someone planning to buy a practice with roughly $200,000 in revenue asked for advice, and the commenters barely mentioned price. They said the clients might be there for the owner, and one described a colleague whose seller reopened down the street and took the book back.⁵ In an accounting forum that December, a buyer of two firms said he wouldn’t do it again, and another commenter said clients are usually loyal to the owner.⁶ Nobody disputed that the practices had revenue. The question was whether the revenue would follow the work to a new name.

The work can move before confidence does. The middle figure on the graphic holds two pages, the process steps and a note about how a client usually likes things done, and the second page is the one that never makes it into the system. The spreadsheet may not contain the whole job, which is what Dave’s dispatcher demonstrated at 9:20 on a Thursday.

The four questions on the left are the question this season asked about where value lives in a business, turned on a single handoff. At Dave’s shop the dispatcher gets called, knows the exceptions, and knows what usually works, while Dave is the one who can calm the customer. Every one of those answers is a person Nina’s card would have to learn from without replacing.

The research supports the direction and stops short of the claim builders want. Relationship quality is linked to seller performance across a large body of relationship-marketing studies, with stronger effects in services and business markets.⁷ Among one supplier’s business customers, relational switching costs were associated with less switching,⁸ yet across the broader literature switching costs have only a weak effect on switching.⁹ Trust matters, and it is not a wall. The graphic’s evidence boundary is candid about the rest. No study cited here measured whether trust transfers in a handoff, and none tested solo or local providers, so the handoff map and its owner-dependence prompt are inspection tools.

That is why the graphic asks two questions instead of giving a verdict. Trust asks what the buyer must believe, and transition asks what must move or be relearned. For Nina, Dave had to believe a stranger could sit beside his dispatcher during a live week without breaking it, and the dispatcher had to believe the card would carry what she knew about the restaurant’s back door. When Nina reached the last inspection question, what could break during the switch, her honest answer was a Thursday, and two weeks of setup would include two of them. Then she reached the line on the card she wanted most and found it wasn’t hers to write.

A Barrier Can Be Real Without Being Your Advantage

The field she kept wanting to add was triage. When the restaurant calls with a tripped main, should the tech treat it as urgent or finish the current job first? The dispatcher answered that from experience every week, and a card that captured her answer would be the most useful line on it. It would also be a non-electrician writing guidance about when an electrical fault needs a licensed person on site. Episode 5 had taught Nina what that sentence is. It is a boundary question, and it does not get answered late at night by the person who wants the answer to be yes.

A bookkeeper described the same kind of gate from the other side in 2025. An accountant told her that accountants believe they can do the work themselves, so the path to competing with them was the license, and she didn’t want the hours or the tax work.¹⁰ The credential wasn’t protecting her market. It stood between her and the part of it she wanted.

Share

Entry friction doesn’t prove attractive economics, and the two boxes under the checkpoint show why. Regulation can establish who may do the work and what responsibilities attach to it. It does not establish margins, demand, easy access, or profit for a new entrant, and builders tend to read the first box and hear the second. The market doesn’t owe you an exception because the business model looks good in Notion.

The licensing evidence is the strongest on any graphic this season, and it still doesn’t say what builders hope. In a study of U.S. occupations, licensing was associated with a 17 to 27 percent reduction in labor supply.¹¹ A 2015 federal report found that about a quarter of U.S. workers held a state license and that only 2 of the 12 studies it reviewed found quality gains from stricter licensing.¹² Licensing can restrict entry, and it still doesn’t tell you what the margin will be. Outside licensing the evidence gets thinner and varies by field and state, which is why none of it substitutes for qualified review of your own facts.

Nina ran the four boundary questions on her card. The triage line might require a license, and it might create a duty if a tech relied on it. Customer addresses and site notes touched privacy, so she marked that to check. The triage line, for certain, needed qualified review. So she took the route at the bottom of the graphic. Dave’s licensed electrician would write the triage line in his own words, and the card would carry it. Nina would build the card without authoring the judgment.

That is the only meaning PARTNER has in this Kit. It isn’t the friendly option, and it isn’t knowing someone with a license. When a boundary gap appears, you route through a qualified party or seek qualified review, the move Episode 5 simply called qualified review, and then you return to the decision. The practical distinction at the bottom of the graphic keeps three questions apart, whether you can do this, whether you should build here, and what you actually know, and it refuses to let the first answer stand in for the second.

Precommit the Review. Don’t Precommit the Answer.

The HVAC builder from August ended his post caught between two things he didn’t want. He didn’t want to throw away the hours he had spent, and he didn’t want to keep grinding on calls that died before the second sentence.³ He had never written down what result would tell him which to do, so every new call reopened the same argument. Once we start, most of us become very talented lawyers for our previous decisions. The landing page is built, people have been told, six weeks are gone, and one more month starts to sound entirely reasonable.

I’ve sat in release-readiness reviews where “not ready” existed on the slide but not in the room. By then, communications were scheduled, teams were staffed, dependencies had moved, and leaders had already told people what was coming. The review became a search for mitigations instead of a genuine decision about readiness. The gate still existed. The option to fail it had disappeared.

A gate is only real if “not yet” is still allowed.

Otherwise it’s ceremony with a timestamp.

Read more

Plain-English premise: An AI usage dashboard can accurately show that people used the tool. By itself, it can’t show that the work got better.

Judgment Seat: The manager about to attach a usage number to a person, a team, a review, or a claim that the rollout worked.

Episode 1:

Episode 2:

👋 Last week’s closing promised an episode called The Scoreboard Is Lying. The research changed my mind again. The dashboards in this research weren’t lying. The ones whose definitions I could inspect were counting exactly what they were built to count. The trouble starts when we ask a count to testify about something it never observed. Episode One budgeted the reviewer’s attention. Episode Two asked what the sender owes before review begins. Episode Three asked what an approval can prove. This week moves up a level, to the number leaders reach for when they want to know whether any of it is working.

Episode 3:

Research Binder: the receipts (citations + source notes) are compiled in a PDF at the bottom of this post.

⚡ Pressure

AI usage is easy to count, and the count can quietly become the verdict on the work.

🎯 Payoff

This issue gives you the AI Proxy Audit: four questions to ask before a usage number becomes a verdict about people or progress.

Pressure Moment

In an April 2025 memo, Duolingo announced that AI use would be part of what it evaluated in performance reviews. In an interview published April 10, 2026, CEO Luis von Ahn said the company had backtracked. He said he didn’t know whether employees were using AI just to be seen using it. What he recounted was their question: “Do you just want us to use AI for AI’s sake?” His answer, in effect, was that the company had been pushing something that sometimes didn’t fit instead of holding people accountable for the outcome.

That’s one company, in its CEO’s account. No scored review field or effective date is in the public record. The employees weren’t asking whether AI mattered. They were asking what their use of it was supposed to prove.

I’ve had the experience of looking at completed work in Azure DevOps and then watching the same work return as follow-on tickets. The original story was still closed. The velocity was still real. The points had still been delivered. But the dashboard had stopped observing before the consequences did. That changed how I read activity metrics. The number can be perfectly accurate about what happened inside its measurement boundary and still be insufficient evidence that the work itself improved.

The Number Gets Promoted

An AI dashboard might accurately report how many people opened the tool this month, how many prompts were recorded, and how many suggestions were accepted. Those can be perfectly legitimate observations. Then someone carries the number into another room: the boardroom, the performance review, the transformation update, the compensation conversation. Somewhere along the way, an activity count acquires a promotion. It stops being evidence that the tool was used and starts standing in for evidence that the work improved.

That headline is deliberately provocative, so it’s worth being precise about it. An unsupported verdict can still turn out to be correct. What an accurate activity count can’t do is establish the stronger conclusion on its own. The gap between the two isn’t a flaw in the data. It’s an inference, and an inference needs its own evidence.

So the question this episode keeps returning to is easy to ask and harder to answer: What is this number allowed to prove? It’s the question underneath the whole series, applied one level up. Episodes One through Three asked who could actually defend a deck, a handoff, or an approval. This week, what needs defending is the conclusion someone draws from a number.

Four Kinds of Evidence on One Screen

Not every number on an AI dashboard belongs to the same evidentiary category. Some observe activity, some are estimates built on assumptions, some are perceptions users report, and some measure downstream work. Arranged in matching tiles, they look alike. The activity indicators, at least, are defined in public. Microsoft’s Copilot Dashboard documentation counts active users from recorded Copilot activity, and GitHub’s documentation calculates acceptance rate from counts of suggestions generated and accepted. Neither definition assesses whether the work that followed was any good. That isn’t a hidden flaw. It’s what those measures are for.

Estimates are where a number gets most persuasive. Microsoft’s dashboard also shows an estimated assisted value: estimated Copilot-assisted hours multiplied by a configurable hourly rate, with a documented default of $72 an hour. Change the rate and the dollar figure moves while nothing about the work does. That makes it a modeling input, not observed savings, and on its own it doesn’t establish realized cash savings, revenue, improved quality, or net ROI (Microsoft Copilot Dashboard documentation, as inspected September 2026).

Once the claim changes, the evidence question changes with it. “Did people use the tool?” is one question. “Did avoidable rework decline for comparable work?” is another, and “Did AI cause that decline?” is harder still. Moving between them isn’t a climb from bad data to good data. It’s a move between different claims, which is why the useful question about an AI metric isn’t whether it’s good. It’s good for what?

Sometimes Usage Is Exactly the Right Thing to Measure

There’s a lazy version of this argument that calls every usage metric a vanity metric. I don’t buy it. Organizations spend real money on AI tools, and leaders have a legitimate reason to know who has access, who started, who stopped, and where people are getting stuck. For those questions, usage data isn’t a weak proxy. It’s the right evidence.

Disney offers a documented example. Business Insider inspected a manager’s message to an engineer whose recorded AI use was low; the note asked about tool access, barriers, and what support would help. Separately, an unnamed Disney engineer said colleagues saw their name near the top of an internal usage dashboard and reached out to learn their methods. Other Disney employees described pressure around the same dashboard, so this is a documented check-in and a reported learning opportunity, not a verdict on the dashboard (Business Insider, April 30 and May 8, 2026).

JPMorganChase shows the same boundary drawn from the top. In the bank’s official February 23, 2026 Q&A, CEO Jamie Dimon discussed LLM usage and the time savings users perceive, while declining to count some of those savings as financial return: “That’s not in an NPV.” That’s a stated valuation boundary, not proof of bankwide practice. Business Insider’s April 2026 reporting on internal dashboards that track and rank engineers’ AI use belongs in the same picture, along with the bank’s statement that the data isn’t used in performance management and employee accounts of feeling pressure anyway.

None of this shows that monitoring improved anyone’s work, and a useful purpose doesn’t by itself settle privacy, fairness, or legal acceptability. What it shows is usage data doing the job usage data can do. A metric doesn’t become foolish when we change the question. It becomes insufficient when we ask it to answer a question beyond the support we have for that use.

When the Number Enters a Review

Usage information doesn’t become dangerous because someone places it near a review. The question changes. An adoption question asks whether people are trying the tool. A performance question asks what their usage says about whether they’re doing the job well, and that second question carries a much larger interpretive burden.

Share

Duolingo is the clearest public example of that shift, and its limits matter as much as its story. The memo announced AI use as an evaluation factor; the CEO’s later account describes backing away from it at a point he didn’t date. The record doesn’t establish a numerical usage score, exact workforce coverage, or any measured effect on performance, pay, or promotion.

Amazon supplies a narrower example. By May 29, 2026, Amazon had confirmed that KiroRank, an informal, employee-built AI-usage leaderboard, had been deprecated, describing it as a beta tool used by some employees rather than a formal or approved one. Reporting credited to the Financial Times quoted senior vice president Dave Treadwell telling staff, “Please don’t use AI just for the sake of using AI.” The explanations for the shutdown differ: 404 Media reported an internal announcement saying the goal had been accomplished, alongside employee accounts pointing to waste and gaming. The reporting doesn’t settle which was the real reason, and I won’t pretend it does. Amazon did tell Fortune that it discouraged measuring developer productivity through token utilization, while treating token monitoring for cost and efficiency as a separate matter.

KiroRank wasn’t a performance review, and that’s the point. Metrics don’t sit on a dashboard waiting to be read; organizations decide what they’re allowed to influence. A number can prompt curiosity, inform coaching, enter an evaluation, or move money. Those are different decisions, and they don’t automatically inherit the same evidentiary permission. A new decision use needs its own justification.

A Million Prompts

Shoosmiths is the best-documented case in this research, and it isn’t a failure story. That’s what makes it a fair test of the argument: here, adoption was the explicit goal.

In April 2025, the UK law firm announced a firm-wide target of 1,000,000 Microsoft Copilot prompts, linked to a £1,000,000 conditional staff bonus pool subject to other eligibility and financial conditions. City AM reported in December 2025 that the target had been reached more than four months early. Financial Times reporting from July 2026, republished by The Irish Times, referred to 1,200 staff who had “claimed their bonus,” with about 200 partners on a separate reward arrangement. A second £1,000,000 was reportedly set aside for AI-fluency accreditation, which the firm’s chief executive described as the plan all along: “We always knew that in year two it was going to be about AI fluency.” The exact cash distributed hasn’t been verified.

The detail I find most instructive isn’t the headline number. City AM reported higher usage among trainees and associates and lower usage among partners, which the chief executive regarded as consistent with their supervisory role. That’s usage read in the context of the role instead of ranked as a single score, and it still isn’t evidence that either group was doing better work.

So what did a million prompts establish? That the firm hit its prompt target, which answers a real adoption question. The sources examined don’t link the incentive to productivity, quality, or client value. That isn’t evidence those improvements didn’t happen. Unestablished isn’t the same as false. It means the stronger conclusion hasn’t been earned from the evidence in hand, and the practical fix is to define the intended outcome before setting the target, then assess what changed afterward.

When the Reward Changes What We Notice

There’s a famous shortcut available here: invoke Goodhart’s Law, declare that people game whatever gets measured, and move on. It would make a punchier article, and it would outrun the evidence. The AI-specific research I examined doesn’t establish that usage incentives generally cause gaming, that visible AI metrics reduce quality, or that leaderboards necessarily distort performance.

What we do have is adjacent laboratory research. In experiments published in The Accounting Review (Choi, Hecht & Tayler, 2012), participants in a strategy task were compensated on a narrow measure, and their later choices shifted in ways consistent with treating the measure as the goal it stood for. Researchers call this strategy surrogation. Paying on three measures instead of one reduced the overall surrogation score, though the evidence for reducing the most strategically costly choices was less consistent. These were graduate business students in a lab, not employees using AI. What transfers is a question worth testing: does a rewarded measure shape later judgment? What doesn’t transfer is any AI-specific effect size, prevalence, gaming rate, or harm.

That leaves four layers worth keeping separate: what’s visible, what gets attached to it, how people respond, and what happens to the work. Salience isn’t gaming. A measure can become more noticeable or more consequential without anyone manipulating it, and gaming is a behavioral claim that needs behavioral evidence. Incentives aren’t automatically bad, either. Shoosmiths used one to encourage adoption, and whether an incentive supports learning or distorts attention is an empirical question in each setting.

This is also where the promotion from the start of this issue gets a name. I use “the Proxy Transfer” as an editorial label for granting a metric authority beyond what its evidence supports. It’s related to strategy-surrogation research, and I’m not claiming a new or validated AI-specific mechanism. Before we reward a measure, the better question isn’t whether people will game it. It’s what response the reward might encourage, and how we’ll check.

Authority Belongs to the Use

A metric doesn’t carry one universal level of authority. It has an intended use, and the same AI-usage information can serve several: observing activity, informing where to ask questions, contributing to an evaluation of work, or feeding a consequential decision.

Those aren’t validated stages, and there’s no automatic climb. Interpretive reach and decision stakes are different dimensions. Deciding where to spend on training can be consequential while asking little of the number beyond what it observes. Claiming that a team’s work improved asks a great deal of the same number, even before anything rides on the claim. The more consequential the use, the more carefully its interpretation has to be justified, but the justification follows the inference, not just the size of the decision.

The principle is older than enterprise AI. The Society for Industrial and Organizational Psychology’s Principles for the Validation and Use of Personnel Selection Procedures (5th ed., 2018) hold that a procedure should be used only for purposes its validity evidence supports. NIST’s AI Risk Management Framework 1.0 (2023) ties validation to a specific intended use and documented limits. Neither prescribes this ladder or validates my four questions; applying their principle to AI-usage metrics is my adaptation.

That’s also why “more consequential” shouldn’t be translated into “collect more data.” More screenshots won’t rescue a weak inference, and neither will another tile on the same dashboard. Sometimes existing evidence fits the intended use. Sometimes the claim needs an outcome measure or a comparison. Sometimes the honest move is to narrow the claim, and sometimes the evidence stays insufficient. That’s allowed. Uncertainty is a better management outcome than false precision with a cyan progress bar.

Visibility Isn’t Authority

AI is giving organizations extraordinary visibility into activity, and that’s useful. It also creates a peculiar management hazard: the easiest things to count can become the easiest things to believe. The Judgment Layer isn’t asking anyone to distrust the number. It’s asking us to stop confusing visibility with authority, which takes a few deliberate questions before the number gets used.

🧭 The Judgment Move

Orienting sentence: The AI Proxy Audit is four questions to ask before a usage number becomes a verdict: what it observes, what it leaves unobserved, what we’re trying to establish, and what authority we’re about to give it.

Subscribe now

Question 1: What does this number directly observe? Stay literal. Name the activity the measure records: use, a defined activation event, prompts, tokens, recorded AI-suggestion acceptance. Stay inside its definitions and coverage. Basis: Evidence; vendor documentation defines these indicators from activity records.

Question 2: What does it leave unobserved? Name the parts of the claim the number doesn’t directly measure: quality, rework, cycle time, customer outcome, financial return. Unobserved doesn’t mean unchanged or unimportant; it means the number didn’t measure it. Basis: Evidence for what the definitions include; the gaps depend on your claim.

Question 3: What are we actually trying to establish? Name the claim before choosing the evidence. “Are people using the tool?” is an activity question, “Did the work improve?” is an outcome question, and “Did AI contribute to the change?” is a contribution question. A dashboard can answer the first beautifully and say very little about the other two. Basis: Design inference, informed by SIOP’s intended-use principle.

Question 4: What authority are we about to give this number? Name the decision use: an adoption check-in, a support conversation, a performance review, compensation, staffing, a transformation claim. Then ask what use, if any, the evidence justifies. Existing evidence may suffice if it fits. Basis: Design inference, informed by SIOP and NIST.

The answers point to one of three responses: use the measure for the bounded question it supports, seek outcome-relevant evidence for the stronger claim, or narrow the claim. If support stays insufficient, the stronger conclusion stays unestablished. Organizations hate leaving a box blank, but a blank box is often more honest than a number doing a job it was never qualified for.

Done state: the number moves forward with a named decision use and visible support for that use, or the claim attached to it is narrowed to what the number can carry.

Constraint: this only works if “we don’t have evidence for that yet” is an acceptable answer in the room where the number gets used.

I’ve been in leadership conversations where the dashboard looked healthy while some of the supposedly completed work was quietly returning as rework. The numbers weren’t wrong. Stories had closed. Velocity had been recorded. The mistake would’ve been treating those facts as proof that delivery had improved. Over time, I started asking a different question: what stayed closed? That small shift changed the conversation from whether the metric was accurate to whether we were asking it to prove something it had never measured.

Limitation: these questions can’t prove a metric is accurate, prove AI caused an outcome, prove a person performed well, or prove a decision is fair or legal. Nor can they prove that asking them improves decisions. The AI Proxy Audit is an unvalidated, principles-informed discussion aid. It can expose an inference gap; it can’t close one.

🧪 What Holds Up

Primary claim label: Evidence for the measurement distinction; Design inference for the AI Proxy Audit as a bundle.

Basis: Microsoft and GitHub documentation define adoption and usage indicators from activity records, without independent assessment of the associated work. SIOP’s Principles and NIST’s AI RMF both tie valid use to a specific intended purpose, supported by evidence and documented limits. Shoosmiths shows a named employer using a defined prompt-volume target tied to a bonus pool, with target achievement reported by independent outlets.

Counterpoint, labeled Evidence: usage-related information isn’t meaningless. You’ve seen “Generative AI at Work” (Brynjolfsson, Li & Raymond, Quarterly Journal of Economics, 2025) in Episodes One and Two; it’s here for a different reason. In a staggered rollout covering 5,172 customer-support agents, the study found larger estimated productivity gains among workers with higher initial adherence to the AI’s suggestions. Adherence isn’t raw prompt volume, and this isn’t a test of whether usage predicts work quality, but it blocks any claim that usage tells you nothing.

Scope and transfer limit: SIOP’s Principles were written for personnel selection; NIST’s framework is voluntary AI-system guidance; the surrogation evidence comes from a lab. The named cases come mostly through newsroom reporting and executive accounts, and none establishes how common usage-based evaluation is in Product, Program, Design, or Research teams.

What remains unknown: whether raw usage improves prediction of work quality once role and task are accounted for (no qualifying test was found); whether evaluative stakes on AI usage change behavior or outcomes in real workplaces; whether any case paired a usage-driven personnel decision with outcome evidence pointing the other way (none was found, which isn’t proof it never happens); and whether the AI Proxy Audit improves decisions. It hasn’t been tested.

🛑 Do Not Use When

• The question really is access, adoption, or usage cost. Usage data may be the right evidence; answer it directly instead of adding ceremony.

• You want to rate, rank, or monitor individual employees. These questions examine an inference, not a person.

• You need a compliance, fairness, or legal answer. Neither SIOP nor NIST prescribes this aid, and it can’t settle privacy or employment-policy questions.

Judgment Seat

I’ve spent much of my career sitting between the people doing the work and the people reading the dashboard. That seat changes how a number looks. At team level, the context is obvious: dependencies, rework, tradeoffs, and exceptions are part of the story. By the time the number travels upward, much of that context can disappear. I learned that interpreting metrics isn’t just a reporting task. It’s also a responsibility to preserve the boundary between what the number observed and what someone wants it to mean.

Who generated the number: the tool’s telemetry, recording what it was built to record.

Who must understand it: the manager about to use it, to the depth of the decision it will inform.

Who can stop it: whoever owns the decision the number is about to enter: the review process, the staffing call, or the transformation narrative.

Who pays if it is wrong: the person, team, or rollout judged by a number that couldn’t see the work, and the organization that believes it learned something it didn’t.

🎁 Your AI Proxy Audit

The AI Proxy Audit turns the four questions into a working page for the moment before a usage number enters a review, a staffing decision, or a board slide. It asks you to name what the number observes, what it leaves unobserved, what claim you’re trying to support, and what decision authority you’re about to give it. There are no scores, thresholds, or automatic verdicts. It’s an unvalidated, principles-informed judgment aid, offered so teams can start testing the practice: name the claim, match the evidence, and narrow what the number can’t carry.

📘 AI Proxy Audit — Practitioner Guide

376KB ∙ PDF file

Download

Learn what an AI number can really tell us before we let it make a bigger claim. This 7-page guide teaches a simple four-question method for checking AI usage metrics before they influence a review, budget, staffing choice, or claim of success. It helps separate what a number actually measures from what we may be tempted to believe it means. Inside, you’ll learn the difference between activity, estimates, self-reports, and real work outcomes. You’ll also see common traps, clear examples, a 15-minute way to run the Audit with a team, and real-world cases that show why the difference matters. The goal isn’t to decide whether an AI metric is “good” or “bad.” It’s to help us ask a better question: What is this number allowed to prove? Best for: learning the method, leading a team discussion, and keeping the Audit’s evidence rules close at hand.

Download

🛠️ AI Proxy Audit — Interactive Workbook

133KB ∙ XLSX file

Download

Turn the four questions into a repeatable decision process your team can actually use. This Excel workbook is the working version of the AI Proxy Audit. Add an AI metric once in the Metric Register, examine each proposed use in the Case Log, and record evidence that supports or challenges the claim in the Evidence Log. The workbook then brings the information together in a one-page Metric Use Brief that can be printed, saved as a PDF, or carried into a meeting. It also includes five completed examples, built-in definitions, guided choices, neutral notices that flag areas worth another look, and a reference section so teams don’t have to invent the process each time. There are no scores, rankings, pass/fail grades, or automatic verdicts. “Unknown” is even a valid answer. The workbook helps make the evidence and its limits visible while leaving the actual judgment where it belongs: with people. Best for: reviewing real AI metrics, preparing for leadership conversations, documenting why a metric is being used, and creating a record the team can revisit when new evidence appears.

Download

📄 AI Proxy Audit — Field Sheet

356KB ∙ PDF file

Download

Put the four questions where the decision is happening. This one-page field sheet is the quick-use version of the AI Proxy Audit. Keep it beside the dashboard, bring it into a meeting, or print it before a review where an AI number may shape a decision. It walks us through four simple questions: What does the number actually measure? What does it leave out? What are we trying to prove? And how much authority should we give it? The goal isn’t to tell us whether a metric is good or bad. It’s to help us use the metric for what it can support, look for more evidence when we need it, or admit when a stronger conclusion hasn’t been proven yet. The person still makes the judgment. The sheet just helps make that judgment more careful and visible. Best for: meetings, leadership reviews, quick metric checks, and keeping beside the full Practitioner Guide and Workbook.

Download

This Week

Three actions for the judgment owner: Pick one AI usage number you report or receive, and write one sentence naming what it directly observes. Write down which decision it currently informs. Before your next review cycle, check whether any AI-use expectation names the outcome it’s supposed to support.

One action for the system owner: Decide in writing which decisions AI usage data may inform on its own, such as access, support, and cost, and which require outcome-relevant evidence first.

One action for learning and provenance: The next time a usage number appears in a deck, add one line beneath it: what it observes, and what it doesn’t.

📎 Forward This

An AI usage number can be accurate and still be asked to prove too much. Before it judges a person or a program, ask what it’s allowed to prove.

🔚 Closing

The dashboard counted the activity correctly.

Deciding what that count is allowed to prove is still human work.

Next week, Episode Five asks the season’s longest-horizon question. Who Trains the Next Reviewer? looks at the tasks that once formed future judgment, and what happens when they’re the first ones automated. Because a usage number can show who’s using the tool. It can’t show who’s learning to judge the work.

This question is part of the larger challenge behind my upcoming book, Collaborate Better: From Silos to Synergy – How to Build Unstoppable Teams. Better collaboration begins when context, ownership, and judgment travel with the work.

Learn more at CollaborateBetter.us.

Regards,

Mark 👋

Citations And Research Binder

476KB ∙ PDF file

Download

This Citations and Research Binder supports Judgment Layer Episode 004: *The Tool Got Used. Did the Work Get Better?* It gathers the research behind the article’s claims about AI usage metrics, adoption signals, activity counts, estimated value, work outcomes, performance evaluation, incentive effects, metric authority, and the difference between observing activity and establishing that work improved. The binder also supports the AI Proxy Audit’s core principles: naming what a number directly observes, identifying what it leaves unobserved, clarifying what claim we are actually trying to establish, and examining what authority the evidence can justify for a particular decision. It distinguishes vendor documentation, company statements, newsroom reporting, experimental research, professional measurement standards, governance frameworks, counterevidence, adjacent-domain findings, and Empathy Engine synthesis while making the limits and transfer boundaries of each visible. The complete AI Proxy Audit remains a principles-informed design inference rather than a validated intervention, and the research does not establish that AI usage metrics are inherently misleading, that higher usage proves better work, that usage incentives generally cause gaming, that outcome improvement proves AI caused the change, or that completing the Audit proves a metric, interpretation, personnel decision, or transformation claim is accurate, fair, legal, or effective.

Download

Previous:

More Content to Discover:

Pain earns an inspection. Not a build.

👋 Welcome to my paid subscriber-only edition of Empathy Engine (🔒 Leader’s Dispatch). Each week I build evidence-forward tools for product leads who need to say no, defend tradeoffs, and lock in decisions before they get rewritten later.

Nina did not want to own the truck. She wanted to fix the handoff that made three trucks late every Thursday.

She had been coming back to Dave’s shop for most of a month. Same regional electrical and facilities company she shadowed in Episode 2, same six crews, same dispatcher running the week out of a spreadsheet and the day out of a group chat. The trucks were fine. The crews were good. The property manager waiting on a late truck at ten on a Thursday did not care about either.

This Thursday went the way the last three had. At 8:10 a commercial client canceled a panel inspection. At 9:05 a crew finished early two towns over. At 9:20 a restaurant called with a tripped main and a lunch rush forty minutes out. The dispatcher moved three jobs in eleven minutes, in the group chat, from memory. One tech got an address and no notes. Another got the right notes for the wrong building. By ten, Dave was answering his phone with the tailgate down, repairing the day.

Nina saw an offer in it before lunch and wrote it in one line. Remote dispatch and follow-up for small electrical and HVAC shops. They keep the trucks. I run the Thursdays.

Episode 2 Part A:

She had already crossed out the other option on that page. Starting a facilities-coordination company of her own meant owning the crews and the phone, and Episode 2 had shown her where that phone ends up. The overlay looked like the season finally paying out. Unglamorous, repeated, painful, and no van required.

Episode 2 Part B:

Nina, Dave, and Dave’s company are a fictional composite built from recurring patterns in the research, not real people or a real business.

It took her until Friday to name what she had watched. The shop wasn’t failing to schedule work. It was rescheduling reality, all morning, by hand. That is a different problem, and it raised the question that ruined the line in her notebook. If the Thursdays leave Dave’s building, what has to leave with them?

I have watched work look complete in the system before it was usable in the real world. A ticket moved to another team with the requirement, the due date, and the expected output all intact, but the decision history and operating assumptions stayed with the people who had been in the earlier conversations. The receiving team did not reject the handoff. They simply had to reopen the conversation to understand what they had supposedly inherited.

The task moved. The context did not.
The correction loop was the handoff telling us what we forgot to send.

I use Digital Overlay as a working label for the operating layer around a service business, the intake, scheduling, dispatch, estimates, documentation, closeout, billing, follow-up, and warranty work that surrounds the job. It is a practitioner label, not a proven market category. It names something to inspect, not something to build. This is the season’s thinnest evidence on the commercial question, so the episode leans on the inspection and leaves the opportunity open.

By the end you’ll have the Asset-Light Overlay Map, a way to inspect the work around a business before you offer to take any of it. It won’t tell you whether anyone will pay.

Episode 5:

Research Binder: the receipts, methodology notes, and source boundaries are compiled at the bottom of this post.

Last Week’s Six Questions, One Step From the Truck

Episode 5:

Episode 5 ended with a promise. The work that exists because regulated work exists would get the same six questions. Nina keeps it that night.

• What is the actual activity? Scheduling, dispatch, estimate follow-up, closeout, and the calls in between. Not the electrical work. She writes that sentence twice.

• What creates the entry friction? For the office work, she can’t find much, and she marks it to confirm rather than assume.

• What keeps recurring after entry? All of it. Every week has a Thursday.

• Where could a real boundary apply? Right next to her. The electrical work may sit behind licensing, permit, and inspection rules she doesn’t hold and can’t read from an outline. Anything she would schedule or promise about that work goes to qualified review, the way last week taught her.

• Who already owns the burden? The dispatcher, the office coordinator, and whoever answers Dave’s cell.

• What does the economic evidence actually say? Nothing yet.

The six questions did their job. They told Nina she can stand near the truck without pretending to drive it. They did not tell her whether the work around the truck can leave the building. The rest of this episode is the inspection it takes to answer the recurring-work, boundary, and ownership questions closely enough to stop guessing.

The Job Is the Visible Part

The truck, the technician, and the finished job are easy to point at. The ring of work around them is easy to treat as background, and it is where owners say the friction shows up. One operations consultant described a job marked complete on a Friday whose invoice didn’t go out until the following Wednesday, because sending it was still something a person had to remember.[1] He sells operations help, so discount the frame and keep the scene. Direct measurement of how much burden lives in that ring across small service firms is sparse. The people living with it are the best witnesses we have.

Visible pain ≠ separable work ≠ proven opportunity. Each sign is its own test. Pain tells you where to look. Whether the work can separate from the job is a second question. Whether anyone would pay for that separation is a third, and the first two don’t answer it.

Episode 3:

Episode 3’s Workflow Pain Index asked whether a pain repeats and costs something. This episode asks whether it can move without losing context, judgment, or accountability. Stays in the shop is the first destination on the graphic for a reason, because in the practitioner rooms behind this episode it is the answer owners describe most. The question marks sit beside scheduling, dispatch, and estimates, where Nina’s Thursday broke and where the research kept raising the hardest questions about judgment and context.

Nina can already answer two of the checklist questions for Dave’s shop. What surrounds the core job? All nine nodes. Who owns it now? The dispatcher owns the board, and Dave’s phone owns the exceptions. Leak or buffer, she can’t say yet.

Messy Doesn’t Mean Broken

In one small-business forum thread, a practitioner described scheduling crews in the same Google Sheet for more than fifteen years, texting assignments the afternoon before because the schedule kept changing until the end of the day.[2] One business, not a prevalence estimate. It still forces the right question. Why did the thing survive?

Nina’s first draft of the offer had a slide titled Replace the spreadsheet. She deleted it after asking Dave’s dispatcher why the sheet was still there. It predated her time in the job, and it bent when the day bent. Nothing they had tried since bent as well.

Information-systems research has a name for what she was looking at. Steven Alter’s theory of workarounds treats them as deliberate adaptations to obstacles in the system people were given, with consequences that can run either way.[3] That is the leak-or-buffer split. A leak creates work, through duplicate entry, lost handoff information, manual reconciliation, or a dispatcher who becomes the bottleneck. A buffer absorbs variability, through fast local adaptation, familiar crew habits, and quick exception handling.

The same workaround can do both in the same minute. Dave’s group chat was a buffer at 9:20, when the restaurant called and a truck was moving in under a minute. It was a leak at 9:21, when the tech on that truck got the address and none of what the dispatcher knew about the building.

Visible friction is a signal to inspect, not a verdict, and manual doesn’t automatically mean irrational. So the first question isn’t how to automate it. Ask what friction the workaround creates, then what problem it already solves. A replacement has to survive both answers. The graphic leaves four honest outcomes, keep it, improve it, test something else, or not know yet. Builders skip the last one.

Thursday Is the Test

Process maps have excellent manners. Intake, schedule, service, closeout. On a quiet Tuesday that’s the whole business, and anyone watching would think the office could be run from anywhere.

Then a client cancels, a tech finishes early, an emergency lands. Nina’s Thursday had all three before 9:30, and each one dropped the day onto a path with different questions. What changed? Who decides? What can be overridden? What has to be updated before the work returns to the routine?

The practitioner advice on the graphic comes from a thread about crew scheduling. “The group chat should be for exceptions, not the whiteboard for tomorrow’s plan.”[4] It sits in quiet tension with the crew chat in the second graphic, which carries tomorrow’s plan, and both can be true in one shop. A chat holding tomorrow’s plan is doing a job the spreadsheet can’t. A chat holding every exception is also holding context nobody wrote down.

Nobody in the evidence behind this episode measured how often small service firms leave the routine path, and I won’t borrow a number from hospital alert systems to fill the gap. Human-factors research has long described how people rely on, ignore, and override automated systems, which is one reason a standard path still needs someone watching its edges.[5] That supports the mechanism. It doesn’t size it.

What Nina can do is count. She scrolls back through a week of Dave’s group chat and marks every override. Most trace to two recurring causes. One shop, one week, no rate. Still, repeated overrides should teach the workflow something, and at Dave’s shop the answer to who owns the decision was Dave’s phone. If nobody owns the exception, you haven’t seen the whole workflow yet.

Sent Is Not the Same as Usable

Go back to 9:20. The dispatcher sends the tech the restaurant’s address. He arrives to a locked service entrance, a manager he has never met, and a panel in a room nobody mentioned. He calls the dispatcher. She calls the restaurant. The main stays tripped another twenty minutes. The file arrived. The orientation didn’t.

Data is visible, so we celebrate the send. Context isn’t, so we notice it only when the receiver calls back. The repaired handoff doesn’t pretend everything crosses. Context travels on a dashed line because some of it only transfers through practice and review, and the receiver’s standard is ready to act, or knows when to ask. Nobody receives a handoff that anticipates everything. A usable one tells you when to call. And the loop has to close. The receiver confirms what they understood, and the sender clarifies what happens next, because a handoff is confirmation, not transmission.

The best-matched evidence here is clinical. A 2024 systematic review and meta-analysis found structured handoffs associated with fewer defined handoff errors in the acute-care studies it examined.[6] That is a hospital finding about a mechanism. It says nothing about how often dispatch handoffs fail in electrical shops or what those failures cost.

I have watched a requirements package arrive exactly when the plan said it would and still fail as a handoff. The receiving team had the requirement, the acceptance language, and the due date, but not the decision history behind them. They could read what had been decided. They could not tell which assumptions were fixed, which ones were negotiable, or why several obvious alternatives were already off the table. The people who had supposedly finished the work ended up rebuilding the conversation for the people who had inherited it.

Of the six receiver questions, the last is the one a builder can observe from outside. Can they say back what they’re about to do? If the receiver has to rebuild the context before acting, inspect the handoff.

The Task Moved. The Questions Came Back.

The fourth graphic is one handoff. The fifth is what happens when the task itself changes hands and keeps coming back, and Nina’s notebook offer puts her on its right side, as the new performer of Dave’s Thursdays.

The word task makes work sound more detachable than it sometimes is. “Reassign the 9:20 call” sounds portable. “Reassign the 9:20 call, knowing the restaurant’s service door stays locked until eleven, the panel is in dry storage, and the manager on shift has never met your tech” is a different object. The first is a task. The second is a task plus context.

Share

The practitioner rooms describe what happens when only the first one moves. In a self-employed forum, a commenter described an HVAC friend who spent more time correcting his offshore assistant’s estimates than it would have taken to write them himself. The thread’s answer wasn’t a cheaper assistant. It was someone who knows a shutoff from a gate valve.[7] An electricians’ forum put it more bluntly. Hire into the office too early and you’re just paying someone to be confused with you.[8] An old HVAC thread keeps getting quoted because it compresses the problem into one exchange, in which a tech explains that the person on the phone answers the phone for a living.[9] Three rooms, one mechanism. When the receiver lacks the trade context, the task comes back as correction.

Customer history sits on the dashed line for the same reason. Plenty of it lives in records. Plenty more lives in sentences like before you call her, there’s something you should know. Where each item lands depends on the shop, and the graphic says so.

The research cuts both ways. Knowledge-transfer studies find that how well knowledge moves depends on the knowledge itself and on who receives it,[10] and a 2024 systematic review found that documentation can carry information without carrying the experience that makes it usable.[11] Yet context-heavy work does cross organizational boundaries in some settings, when there is usable history, real specialist capability, clear escalation, and retained oversight.[10] Those are conditions to test, not a score. Externalization can also create monitoring, coordination, and quality-control work of its own, and no direct trade-office correction rate is established.

Nina tests them on herself. She knows scheduling across three industries. She doesn’t know which tripped main means a bad breaker and which means an electrician has to be on site before anyone promises a time. In her offer, she would own that exception, on Dave’s customer’s call, without Dave’s authority. That is the hard cost, and it would be hers.

The Tool Isn’t the Transition

We price the destination. We underprice the journey.

Nina’s second draft dropped the idea of running Dave’s Thursdays. She would set up field-service software for shops like his, get the crews onto it, and leave. No exceptions to own. Just the tool.

Read more

Plain-English premise: An approval record proves someone was at the gate. It doesn’t show what they checked, or whether they could have stopped the work.

Judgment Seat: The approver whose name is on the record, whether or not the gate gave them enough to judge.

Episode 1:

👋 Last week’s closing promised an episode called The Button That Used to Be a Person. The research changed my mind. The person didn’t disappear. They clicked. Episode One budgeted the reviewer’s attention. Episode Two asked what the sender owes before review begins. This week is about the moment in between: the approval itself, and what it can and can’t tell anyone who reads it later.

Episode 2:

Research Binder: the receipts (citations + source notes) are compiled in a PDF at the bottom of this post.

⚡ Pressure

AI can expand what arrives for approval, and the click that clears it records presence, not judgment.

🎯 Payoff

This issue gives you the Gate Honesty Check: five questions that test whether an approval gate can support the judgment its record implies.

Pressure Moment

In mid-September, a software engineer posted a short after-action note. He had approved a pull request an AI agent wrote for a system he designed. Tests passed. The diff looked reasonable. He merged it without fully tracing one change. Two days later, a background job quietly started double-processing records. It’s one person’s public account, from software, and it says nothing about how often this happens. What it shows is narrower and more useful. He did check things. The merge record would look identical whether he traced that change or not.

The log recorded a human. It didn’t record the check.

I’ve inherited plenty of “approved” decisions in enterprise environments where the approval was real, but the judgment behind it wasn’t recoverable. We could prove who signed and what moved forward. Reconstructing what they had actually checked sometimes required emails, meeting notes, and conversations with people who were still around. The problem wasn’t necessarily the review. It was that the record couldn’t carry the review forward.

None of this means the approval was empty. Sometimes a green checkmark is the compact end of an excellent review. Sometimes the review was thinner. Sometimes we simply can’t tell, and not knowing is different from knowing that no judgment occurred. An incomplete record is not proof of an empty mind. The useful question is narrower: what evidence justifies treating this particular approval as meaningful oversight?

Was a human present? Did meaningful judgment occur? Did it improve the outcome? The log can answer the first: who clicked, when, and what state the system recorded. The other two need more evidence: what the person inspected, what evidence they used, and what would have made them refuse.

The one number on that graphic shows why the design around a decision matters. In a 199-participant experiment (Buçinca, Malaya & Gajos, 2021), people using simple explainable AI accepted incorrect advice on one wrong-advice subdecision about 64 percent of the time, versus 48 percent under cognitive-forcing designs. That’s a conditional result, not a 16-point cut in overall failure. It still shows that how a decision is set up changes what people do with bad advice. A human’s presence alone doesn’t guarantee that.

What the Record Can Prove

Organizations like signatures because one field compresses hours of work into “Approved by Taylor Kim.” That compression is useful. It also loses a lot.

A meta-analysis of 106 experiments and 370 effect sizes (Vaccaro, Almaatouq & Malone, 2024) found that human-AI combinations beat humans working alone on average, yet underperformed the better of the two working alone, with results varying widely by task. That finding supports neither “humans make AI better” nor “get humans out of the way.” Performance depends on the task, the system, and how the collaboration is designed. So read the record for what it can support: a sign-off shows presence, a review trail offers observability, and only outcome data speaks to effect.

In one hospital audit (Salgado, Barber & Danic, Journal of Patient Safety), independent audio review of 100 surgical cases found lower checklist fidelity than the paper completion records indicated. That doesn’t mean the clinicians were lying, or that no safety work occurred. The audit compared spoken checklist items with paper records, not anyone’s private judgment. Missing detail doesn’t prove missing judgment. It proves the record can’t settle the question on its own.

The answer isn’t more paperwork or surveillance. It’s deciding what each record is for. Approval status tells you who acted. Review evidence tells you what was inspected. Outcome data tells you what happened next. One checkmark can’t do all three jobs.

Watching Isn’t Stopping

Now imagine the reviewer spots a problem. What happens next? Darren Kimura, a technology executive quoted in CIO this month, described many companies that claim a human in the loop as actually having “a human watching the loop”: someone who can flag a concern but can’t stop, change, reject, or escalate it. He’s a vendor voice, so treat the line as a sharp distinction, not a measurement.

Share

The NIST AI Risk Management Framework tells organizations to assign mechanisms to supersede, disengage, or deactivate AI systems that perform inconsistently with their intended use. That’s a governance principle, not proof of outcome benefit. The closest outcome evidence comes from hospitals. A Cochrane review of escalation systems (McGaughey et al.) covered 4 randomized trials with 455,226 participants plus 7 nonrandomized studies. It couldn’t isolate authority as the active ingredient, because pathway design, recognition, communication, response, and implementation were bundled together.

That doesn’t mean authority is irrelevant. It means we should be precise about the pathway. Can the reviewer detect the problem? Can they act on it? Will the system honor the action? An approver without a usable pathway may be an observer with a better title.

Who Decided the Depth?

Episode One dealt with review capacity. This section asks a different question: when more work arrives, who decides how deep each review goes?

The forty concepts and the dozens of daily pull requests are illustrations, not prevalence estimates. We don’t have evidence that faster AI output broadly causes worse review. We do have a systematic review of 40 studies across 17 task types (Lyell & Coiera) that found automation-related omission and commission errors across different settings. Its methods were too varied to produce a product-team error rate. If the arrival rate changes while the review design stays fixed, something still decides which work gets scrutiny. It might be consequence, reversibility, or uncertainty. Or it might be whatever arrived while the reviewer still had patience. Only the first three are decisions. Don’t let arrival volume silently decide review depth.

Easy to Read Isn’t Easy to Verify

AI is very good at producing something humans are trained to reward: a coherent explanation.

Across controlled studies with 1,626 online participants (Bansal et al., 2021), AI explanations increased acceptance of both correct and incorrect advice, without significantly improving team accuracy over showing confidence alone. In maze experiments reporting 731 participants (Vasconcelos et al., 2023), making the proposed answer easier to check helped people reject wrong advice in some tested conditions, but not all. And the experiment behind the 64-to-48 figure showed no overall performance advantage for the forcing designs.

The goal isn’t more friction. It’s better verification. Show the claim, expose something the reviewer can inspect independently, and make the failure point checkable. That’s the same reason Episode Two’s Handoff Brief asked for an inspectable verification anchor.

Not Every Output Deserves the Same Gate

Uniformity feels safe. If one review is good, the same review for everything must be safer. That doesn’t follow either.

In the MASAI randomized trial (105,915 women analyzed), AI-supported selective single or double reading used 44.2 percent fewer screen readings than double reading everything. Sensitivity was 80.5 percent in the AI-supported arm and 73.8 percent with standard double reading, with specificity at 98.5 percent in both. Every examination still retained a human reader, and total labor savings weren’t established. That’s interesting, but it isn’t permission to copy a mammography workflow into Product Operations.

Routing can also miss. In a different paired mammography study (Elías-Cabot et al., 2026), 11 low-score cancers were auto-labeled normal by the AI strategy. Parallel standard reading found them, so they were policy-level routing misses rather than patients the trial left untreated. Selective review needs visible bypass risk. The Light, Standard, and Deep tiers are Episode One’s Review Budget applied at the gate: a design inference, not a clinical result.

🧭 The Judgment Move

Orienting sentence: The Gate Honesty Check asks five questions about the gate, not the person standing at it. It’s a diagnostic, not a certificate, and it isn’t a lie detector for employees.

Question 1: Can the reviewer act? Is there a direct or delegated path to stop, change, defer, or escalate before the action moves forward? A routed escalation that reliably holds the work counts. Basis: institutional support (NIST); no study shows one form of authority outperforms another.

Question 2: Do they have enough to judge? Do they have the task-relevant time, context, and evidence to inspect the work? Basis: design inference, informed by evidence that verification difficulty is implicated in automation-related errors.

Question 3: What did they personally check? Name the specific artifact, constraint, evidence, or failure point inspected. Not what the AI summarized, and not what the workflow claims was reviewed. “I was there” isn’t an answer. Basis: design inference; a named check makes a gap visible but doesn’t prove the check happened.

Question 4: What would trigger refusal? Define the failure, change, or uncertainty that would require a stop, revision, deferral, or escalation. Basis: design inference.

Question 5: Will the intervention be enforced? What evidence shows that an appropriate stop, change, deferral, or escalation would actually be honored? Basis: design inference; escalation research shows pathways are bundled, not that any one element works.

There is deliberately no question asking whether the gate has ever said no. A refusal quota would turn the check into performance theater. A reviewer can approve ten excellent artifacts in a row and still be exercising excellent judgment, and can reject something for terrible reasons. The action is not the judgment.

Done state: the gate’s weakest answer is named and routed to someone who can fix it. Five completed fields is not the goal.

Constraint: the questions test the design of the gate, not the character of the reviewer. Once they’re used to score people, they stop producing honest answers.

I’ve seen gates where the reviewer did exactly what the process asked: they approved the work with explicit conditions attached. The judgment wasn’t the weak point. The weakness was downstream. Nothing in the workflow actually held the next step until those conditions were satisfied, so “approved with conditions” slowly became indistinguishable from “approved.” That taught me to ask a different question: not whether someone can raise a concern, but whether the system is built to honor it.

Limitation: the check doesn’t prove judgment. It helps show where judgment could break. It can’t verify that a stated check happened, and no question has been validated as reducing errors.

Subscribe now

Run it on one real gate this week.

🧪 What Holds Up

Primary claim label: Evidence for the separable pieces; Design inference for the Gate Honesty Check as a bundle.

Basis: Records and observed execution can diverge (Salgado et al.). Automation-related errors appear across settings (Lyell & Coiera). Interaction design can change how people handle wrong advice (Buçinca et al.). Explanations alone don’t reliably improve accuracy (Bansal et al.).

Counterpoint, labeled Evidence: An approval can be the honest output of a good review. Human-AI teams outperformed humans alone on average (Vaccaro et al.), and selective review with a retained human reader performed well in MASAI. Enforcement has costs too: in one hospital trial, a hard-stop ordering restriction was stopped early after treatment delays.

Scope and transfer limit: this evidence comes from clinical screening, lab experiments, hospital escalation systems, and software practitioners. No study here directly tested AI approval gates in Product, Program, Design, or Research teams.

What remains unknown: whether the Gate Honesty Check changes gate behavior, catches more errors, or improves outcomes. It hasn’t been tested as a bundle.

The green checkmark isn’t the villain. Pretending it answers questions it was never designed to answer is.

🛑 Do Not Use When

You want to score, rank, or monitor individual reviewers. The questions test gates, not people.

The work is low-consequence and reversible, and a reliable automated check already covers it. Route it light instead of adding ceremony.

You need a compliance certificate. Neither NIST nor the EU AI Act validates this method.

Judgment Seat

I’ve been in roles where my name was attached to an approval even though I didn’t control every condition that made the work safe to move forward. In the moment, everyone understood where the dependencies and decision rights actually lived. Months later, the record remembered only the name and the word “approved.” That taught me that accountability and authority can travel separately, even when the system makes them look like the same thing.

Who generated the work: the submitter and whatever tool produced something ready to approve.

Who must understand it: the approver, to the depth the gate actually allows.

Who can stop it: whoever holds a usable pathway. If nobody does, the gate is observation.

Who pays if it is wrong: whoever carries the consequence. The approval record names someone, but it doesn’t decide that.

🎁 Your Gate Honesty Check

Gate Honesty Check PDF

The Gate Honesty Check helps us test whether an approval gate can support the judgment we expect from it. Five simple questions look at whether the reviewer can act, has enough information to judge, can show what was checked, knows what would trigger a refusal, and has a system that will honor a stop or escalation. There are no scores, rankings, or certificates. The goal is to find the weakest part of the gate, give it an owner, and decide what needs to change next.

Gate Honesty Check PDF

462KB ∙ PDF file

Download

This is more than a five-question checklist. It is a practical kit for making approval work easier to see and improve. The package includes a working workbook, a one-page Field Sheet for meetings, an Approval Evidence Card that lets a reviewer’s check travel with the decision, and a guide with worked examples showing what strong answers look like. It can be used for a five-minute check, a 15-minute review, or a deeper redesign of the gate. The tool is evidence-informed, but it is not a safety or compliance certificate and does not promise a better outcome.

Download

Gate Honesty Check Workbook

The Gate Honesty Check Workbook turns the five questions into a working tool you can use on a real approval gate. It helps you name the gate, record what the approval is supposed to mean, test whether the reviewer can act, and see what the record does and does not prove. At the end, you name the weakest answer, choose who owns the fix, set the next action, and pick a date to check it again.

Gate Honesty Check Workbook

156KB ∙ XLSX file

Download

This workbook helps you move from “something feels wrong” to a clear action you can take. It includes a fast five-question check, a deeper stress test, a one-page Gate Record, an action library, worked examples, and a register for tracking gates over time. Nothing is scored or graded. The goal is to make weak points easier to see, easier to explain, and easier to fix without turning the tool into a people-ranking system.

Download

This Week

Three actions for the judgment owner: Pick one gate you approve through and answer Questions 1 and 5 in writing. On your next approval, write one line naming what you personally checked. Before your next click, name one thing that would have made you refuse.

One action for the system owner: Confirm in writing whether a flag at your highest-stakes gate can actually stop or defer the work, and who is responsible for honoring it.

One action for learning and provenance: For one gate, compare what the approval record captures with what it doesn’t show. Add one field that makes the check visible to whoever reads it next.

📎 Forward This

A green checkmark can record participation without proving scrutiny. Before trusting it, ask what the person inspected, what would have made them refuse, and whether they could actually stop the work.

🔚 Closing

The approval moved at the speed of a click.

Whether anyone judged the work still has to be shown, not assumed.

Next week, Episode Four looks at what leaders reach for when oversight is hard to see. The Scoreboard Is Lying asks what AI usage metrics can and can’t tell you about the work.

Because when a gate is hard to trust, the temptation is to count something instead.

This question is part of the larger challenge behind my upcoming book, Collaborate Better: From Silos to Synergy – How to Build Unstoppable Teams. Better collaboration begins when context, ownership, and judgment travel with the work.

Learn more at CollaborateBetter.us.

Regards,

Mark 👋

Citations And Research Binder

336KB ∙ PDF file

Download

This Citations and Research Binder supports Judgment Layer Episode 002: *If You Send It, You Own It*. It gathers the research behind the article’s claims about review readiness, context-building, explanation framing, inspectable evidence, uncertainty, meaningful human judgment, and the role-specific duties that remain when AI assists the work. The binder also supports the Explainability Standard’s core principles: naming what changed, explaining why, showing what was verified and where the evidence can be inspected, and making uncertainty visible before another person is asked to judge the work. It distinguishes experimental findings, observational studies, qualitative research, governance frameworks, professional standards, counterevidence, and Empathy Engine synthesis while defining the limits of each. The complete Four-Part Handoff Brief remains an evidence-informed design inference rather than a validated intervention, and the research does not establish that AI always increases downstream review work, that explanation guarantees understanding, or that completing a handoff brief proves correctness, approval, or legal compliance.

Download

Previous:

More Content to Discover:

A barrier can be real without being an opportunity.

👋 Welcome to my paid subscriber-only edition of Empathy Engine (🔒 Leader’s Dispatch). Each week I build evidence-forward tools for product leads who need to say no, defend tradeoffs, and lock in decisions before they get rewritten later.

In April 2023, a bookkeeper posted a short message to a bookkeeping forum. The books were done. The client wanted the filing done too. She wrote that she is a bookkeeper, not a CPA, and that the asking had to stop.¹

One post from one forum proves nothing about how often this happens. It does show a person who found the edge of her work and stood on the right side of it in the busiest month of the year. She did not describe an advantage. She described a line.

Nina reads the thread on a Tuesday night, in a tab beside the offer she has been building for a week. Documentation support for small medical practices, built around policies and records in an audit-ready folder. Twelve years of operations work across healthcare, home services, and logistics sit behind it, and the offer feels earned. Her notes hold one sentence she is proud of. Most people are scared of this, and that is the moat.

Nina and the professionals she consults in this piece are a fictional composite built from recurring patterns in the research, not real people.

Then she asks the question that ruins the pitch. What, exactly, is the friction doing?

Her sentence ran three leaps together. We see difficulty and start imagining defensibility. We see fewer entrants and start imagining margin. We see paperwork and start imagining a market for removing it. Some of those leaps survive inspection. None survives without it.

I’ve seen requirements treated as though they created protection simply by existing. Sometimes they created useful discipline. Sometimes they created recurring work. Sometimes they only told us where another question began. The requirement was real. The protection we imagined around it wasn’t.

This episode swaps the castle for a filter. Friction comes in three kinds, and the economics is a fourth question that sits outside them. By the end you will have the Regulatory Friction & Boundary Filter, six questions that organize what you do not yet know about a regulated market. It doesn’t answer the legal ones.

Episode 4:

Research Binder: the receipts, methodology notes, and source boundaries are compiled at the bottom of this post.

The Word Everyone Reaches For

Moat is a fine word for a castle and a weak word for a market. It arrives with a conclusion attached. The wall keeps rivals out, so the owner inside keeps the margin. When people call compliance, licensing, or paperwork a moat, they usually borrow that conclusion without earning it.

The research behind this episode supports a narrower sentence. A requirement can change who enters, what work keeps arriving, and which activity a person may perform. Whether any of that produces a margin is a separate question, and the research does not answer it for solo builders in any market.

I use regulatory friction as a working label for the license requirements, approvals, compliance rules, and standards that sit between a builder and a market. It is a practitioner label with no legal standing, and it tells you nothing about which rules apply to you. The first graphic makes the swap.

Friction goes in the top and three conditions come out, each with its own limit. A deterrent can reduce entry or mobility, and it doesn’t prove better margins. A burden can consume time, staff, and operating attention, and it doesn’t prove buyer demand. A boundary can constrain activity, role, entity, or scope, and it doesn’t mean “find a partner and proceed.” Economics sits below all three, and the graphic marks the path to it with two words. Not automatically.

The number in the lower left is one of the best-supported in this episode. Peter Blair and Bobby Chung estimated that occupational licensing reduced equilibrium labor supply by 17 to 27 percent in the occupations and research designs they studied.² It counts labor supply in studied occupations. It says nothing about the competitors in your town or what any of them earn.

Same Friction. Different Job.

Nina spends the night sorting her one proud sentence into parts. Most people are scared of this holds three separate claims, and only one is about entry. It is hard to get in. Some work never stops once you are in. Some acts belong to someone with a credential. The tail of her sentence, and that is the moat, assumes an economic result none of the three delivers.

The second graphic opens with the question she has been avoiding. Which problem am I actually looking at?

One question per condition. What does it take to get in? What must someone keep doing? Where could a real boundary apply? The third is worded on purpose. It asks where a boundary could apply, and it leaves where one does apply to a qualified professional who knows your facts and your jurisdiction.

The band across the middle is the warning to attach to every regulated-market pitch. Don’t collapse all three into “moat.” A deterrent is not good economics. A burden is not buyer demand. A boundary is not “just find a partner.” Effects vary by occupation, sector, role, and jurisdiction.

Classify first. Evaluate second. Nina now has three columns where she used to have a castle.

Some Friction Keeps People Out

Start with the deterrent, since it has the most direct peer-reviewed estimates of the three. Requirements can reduce access, and the third graphic asks who is on the outside of the gate.

Share

Barrier to whom? A deterrent sorts people into inside and outside, and you are the person outside. Johnson and Kleiner found that state-specific licensing exams were associated with lower interstate movement of workers in the occupations they studied, though they caution that their design doesn’t fully establish causality.³ Exams, credentials, and state variation can slow movement.

The widest banner carries the point of the section. Fewer entrants ≠ better economics. A smaller labor pool does not prove better margins for a new entrant. Entry friction is real. The economic upside is not automatic.

For the person outside the gate, the deterrent is also a bill. Where a credential is required, that means exam preparation and experience hours before your first invoice, at a size this research cannot tell you. Nothing in it shows that a margin follows.

Measure the barrier. Don’t romanticize it. Ask what the requirement changes: access, time, mobility, cost, or scope. Then inspect demand and economics separately.

Some Friction Follows You Home Every Night

Entry may be a moment. The workload keeps returning. The door opened, and the work didn’t end. The woman in the graphic asks the question every operator eventually asks. Who owns this next week?

I’ve watched teams clear the gate and then spend every week servicing what came after it. Records had to be refreshed, evidence had to be maintained, questions had to be answered, and exceptions had to be chased. The approval was static. The operating environment was not. The real operating question became: who owns this next week?

Two numbers anchor the graphic, and each carries a boundary. Francesco Trebbi and Miao Ben Zhang estimated that labor devoted to regulation-related tasks equals 1.3 to 3.3 percent of the average U.S. firm’s wage bill, using establishment-level data on roughly 400,000 firms a year from 2002 to 2014.⁴ That describes an average firm’s payroll. It says nothing about profit or about your evenings.

The second number is narrower. In the American Medical Association’s December 2025 survey of 1,000 U.S. physicians, respondents reported 13 hours a week of physician and staff time on prior authorizations and 40 authorizations per physician per week. Ninety-five percent of surveyed physicians reported care delays.⁵ That is a health-care example only, and it doesn’t travel to bookkeeping, real estate, or trades.

The graphic’s third box asks who imposed the work. A regulator, a payer, an insurer, a platform, a customer, and your own internal process each create recurring work, and each responds differently when you try to remove it. Prior authorization is a requirement health plans impose. That makes it a strong example of recurring burden and a weak example of regulation in the narrow sense.

Price the work that repeats. Before entering, identify what must happen every week, month, quarter, and year. Ask who owns it, how long it takes, and whether it can actually be simplified. That is the Workflow Pain Index applied to obligation.

Some Friction Is a Boundary, Not a Chore

A chore can be delegated, outsourced, or automated. A boundary can decide who is permitted to do the work at all.

Nina’s first instinct sounds reasonable. Great, she thinks, I’ll package the annoying part. Then she takes the four service lines in her draft and rewrites each as a verb. Review a practice’s records. Tell a practice what to change. Handle information a patient would consider private. Supervise a junior contractor doing the same. She circles two and cannot tell, from an outline, whether either puts her inside a regulated role. Nobody can, without her facts. Name the act, not just the offer. A new label doesn’t move the line.

Read more

Plain-English premise: AI can write the explanation almost as readily as it wrote the change. Someone still has to know whether either one is true.

Judgment Seat: The submitter who mistakes a fluent summary for proof that they understand what they built.

Episode 1:

👋 Last week, The Judgment Layer named the Review Budget: a team’s finite attention before it quietly runs out. That issue ended on a question: if review is scarce, who earns the right to ask for it? This week answers from the other side: before a reviewer spends that budget, the sender owes them something specific.

Research Binder: the receipts (citations + source notes) are compiled in a PDF at the bottom of this post.

⚡ Pressure

AI made fluent explanation cheap to generate, which means explanation can now arrive looking suspiciously like understanding.

🎯 Payoff

This issue gives you the Four-Part Handoff Brief: four questions a submitter answers in their own words before anyone spends review attention on the work.

Pressure Moment

The failure mode looks something like this. Earlier in this series’ research, I had a set of precise, polished statistics in front of me: +180%, +140%, +115%, +95%, each wrapped in an explanation confident enough that nothing about it asked to be double-checked. If I had treated that polish as proof, it would have crossed straight into a published piece. We checked it instead. The numbers did not hold up, and we quarantined them before they went anywhere near a reader. The presentation had already arrived finished. The verification had not, and looking at it would never have told you that.

That same gap gets more dangerous once it crosses to someone who wasn’t in the room for the checking, a colleague reading a deck, a partner reading a brief, anyone whose only signal is how finished the work looks.

🧭 The Judgment Move

Orienting sentence: The Four-Part Handoff Brief separates orientation from evaluation. Before a reviewer judges the work, the submitter supplies what changed, why, what was checked, and what still is not settled, in their own words.

Step 1: Name the meaningful change, the actual decision that moved, not a restated file list. A generated cover letter listing every diff isn’t this field. If a second AI wrote the explanation of the first AI’s work, human understanding still hasn’t been demonstrated.

Reviewers in that study usually built context before evaluating the change. The Handoff Brief makes more of that orientation visible at send time, instead of assuming the artifact explains itself.

Step 2: State why this shape: the reason, tradeoff, or material alternative that matters. “The model suggested this structure” describes where the idea came from. It doesn’t defend it.

Step 3: Point to what was verified, and where a reviewer can inspect it themselves: a test, a source, a run, a citation. A general assertion of confidence is not this field.

Step 4: Name what remains uncertain, unverified, or disputed. For a consequential handoff, don’t bury this in fine print. Uncertainty can change how the rest of the handoff should be read.

Then evaluate. Once enough orientation is visible, the reviewer can move to approval, revision, or escalation. Orientation and evaluation are two different jobs, and collapsing them is how a reviewer ends up reconstructing the submitter’s reasoning before they can begin their own.

Done state: a ready consequential handoff carries a named change, a stated reason, an inspectable verification anchor, and a named uncertainty, visible before evaluation starts.

Constraint: the four fields scale with consequence. A low-risk change may need one sentence per field. A high-stakes handoff needs real inspectable anchors, not a checklist filled in from memory.

Limitation: this move cannot prove the submitter’s stated verification happened, or that their explanation reflects real understanding rather than a second generated summary dressed as one. It only makes the absence of that information expensive to hide.

What that looks like assembled is worth seeing once the roles around it are clear.

🧪 What Holds Up

Primary claim label: Evidence, for the separable pieces; Design inference for the bundle.

Basis: Generation speed and total task completion are not the same measurement. A field study of over five thousand customer-support agents found AI access improved resolutions per hour by roughly fifteen percent in a bounded task. A separate randomized study of sixteen experienced open-source developers on two hundred forty-six tasks found AI access made completion nineteen percent slower, despite those developers expecting a gain. Same tool category, opposite result: the two studies measured different workflows.

Share

A related caution: a meta-analysis of one hundred six experiments and three hundred seventy effect sizes found human-AI combinations did not automatically outperform the better standalone human or AI. A human checkpoint is not, by itself, evidence that judgment was exercised. One controlled study did find a targeted friction step reduced inappropriate reliance on wrong AI advice, from sixty-four percent to forty-eight percent, so the direction isn’t hopeless. It just isn’t automatic.

Counterpoint, labeled Evidence: in a randomized, 199-participant hotel-review classification task, adding a feature-importance explanation to AI advice raised participants’ relative reliance on it, from roughly thirty percent to thirty-nine percent, while accuracy showed no significant difference. That gives the Handoff Brief a design risk worth guarding against: a polished Field 2 or Field 3 may influence how confidently the handoff is received before its claims are independently checked.

Scope and transfer limit: no study in this evidence base directly measured this article’s specific mechanism, an underprepared AI-assisted submitter’s work creating additional downstream orientation or checking work for a distinct reviewer. That transfer is a recognized practitioner pattern, not a demonstrated cause. Treat it as the undermeasured problem it is.

What remains unknown: whether the Four-Part Handoff Brief, used as a bundle, reduces defects, review time, or reviewer burden. It hasn’t been tested that way. Treat it as a design inference assembled from adjacent, separately-supported pieces, not a validated intervention.

🛑 Do Not Use When

• The task is bounded, low-risk, and a reliable automated check already covers it. Route it to light review instead.

• Nobody involved has real authority to return, narrow, or escalate the work. A Brief without stopping power just documents who absorbed the risk.

• It’s being used to police whether AI was used, rather than whether the work can be explained. That’s a different question, and this isn’t a detection tool.

Judgment Seat

While working on this issue, I sent the infographics for an evidence audit. Claude challenged two of the graphics because the studies behind them weren’t in the evidence inventory I had shared. The reviewer was right to push back: it couldn’t verify my explanation without seeing the actual final evidence packet. I knew those sources had entered during later gap-closure research, so my first instinct was to explain why the graphics were supported instead of just sending the packet. Once I did, the reviewer independently checked the sources and cleared the graphics.

What stayed with me was that the graphics weren’t actually wrong. My handoff was incomplete. I possessed context the reviewer needed and had assumed my explanation could substitute for making that context inspectable. My confidence wasn’t inspectable. The evidence packet was.

I didn’t own the reviewer’s judgment. I owned giving it enough to exercise it.

Who generated the work: the submitter, working with whatever tool produced a result that looked finished.

Who must understand it: the submitter, before they ask anyone else to spend attention on it.

Who can stop it: whoever has been given real authority to return, narrow, or escalate the work, not merely to sign off on it.

Who pays if it is wrong: consequences can land across the submitter, reviewer, system owner, customer, or organization. Who should carry it depends on role, authority, competence, and control, not on who happened to hit send.

Generation, review, and system design are three different jobs, and a single AI-assisted handoff can move through all three. NIST’s AI Risk Management Framework, the EU AI Act, and the ACM Code of Ethics all support the general principle that human duties persist and stay role-specific when AI assists the work. None of the three prescribes this exact three-way split. That mapping is this series’ own operating model, built on what those sources establish, not a restatement of them.

Assembled, the four fields from earlier in this issue look like this:

Subscribe now

That is the whole proposal: proportional, challengeable, and honest about what it can’t prove. Use more when consequence is high and hard to verify. Use less when the risk is already bounded elsewhere.

The Explainability Standard

A one-page guide for making work easier to review. It shows the four questions to answer before sending important work, how much detail to use, and when a reviewer should send the work back because key context is missing. Keep it near your desk when you need the Standard fast.

Explainability Standard

345KB ∙ PDF file

Download

A one-page cheat sheet for fast choices before work moves forward.

Download

The Explainability Brief

A two-page working template for a real handoff. It helps you explain what changed, why the decision was made, what was checked, and what is still uncertain, while giving the reviewer space to ask questions or request more context. Use it when another person needs to understand the work before they can judge it.

Explainability Brief

179KB ∙ PDF file

Download

A simple form that helps make key facts clear before review begins.

Download

The Explainability Standard Workbook

A reusable Excel workbook for teams that want to use the Standard again and again. It includes a live Handoff Brief, an evidence log, an uncertainty log, four worked examples, and a handoff tracker, plus guidance on when to skip the process, keep it light, use the full version, or escalate. It turns the idea from the article into a system a team can actually use.

Explainability Standard Workbook

275KB ∙ XLSX file

Download

A repeat-use team tool for tracking evidence, open questions, and handoffs over time.

Download

Bundle description

Together, these three tools take the Explainability Standard from idea to practice. The Quick Reference helps us remember it, the Brief helps us use it once, and the Workbook helps us build it into the way work moves across a team.

This Week

Three actions for the judgment owner: before your next handoff, write one sentence naming what you’re still uncertain about, and put it first, not last. Pick one artifact in your queue and explain the meaningful change out loud, without reopening the tool that generated it. Next time a summary turns out to be AI-generated restating a diff without reasoning, send it back before spending review time reconstructing what the sender could have made visible.

One action for the system owner: decide out loud who has standing to return an incomplete Brief unread, and tell the team.

One action for learning and provenance: on one review this week, note whether the uncertainty the submitter named in Field 4 turned out to be the uncertainty that actually mattered. Keep the answer visible to whoever reads it next.

📎 Forward This

A generated cover letter that lists every changed line is not an explanation. If you cannot say what changed, why, what you checked, and what you still don’t know, it is not ready to cost someone else’s week.

🔚 Closing

The explanation arrived exactly as fast as the work did.

Understanding still has to be built by hand.

Next week, Episode Three turns to the checkpoint itself. The Button That Used to Be a Person asks what happens when human-in-the-loop becomes an approval queue instead of a meaningful control.

Because even explainable work can overwhelm the person approving it.

This question is part of the larger challenge behind my upcoming book, Collaborate Better: From Silos to Synergy – How to Build Unstoppable Teams. Better collaboration begins when context, ownership, and judgment travel with the work.

Learn more at CollaborateBetter.us.

Regards,

Mark 👋

Citations And Research Binder

232KB ∙ PDF file

Download

This Citations and Research Binder supports Judgment Layer Episode 002: *If You Send It, You Own It*. It gathers the research behind the article’s claims about review readiness, context-building, explanation framing, inspectable evidence, uncertainty, meaningful human judgment, and the role-specific duties that remain when AI assists the work. The binder also supports the Explainability Standard’s core principles: naming what changed, explaining why, showing what was verified and where the evidence can be inspected, and making uncertainty visible before another person is asked to judge the work. It distinguishes experimental findings, observational studies, qualitative research, governance frameworks, professional standards, counterevidence, and Empathy Engine synthesis while defining the limits of each. The complete Four-Part Handoff Brief remains an evidence-informed design inference rather than a validated intervention, and the research does not establish that AI always increases downstream review work, that explanation guarantees understanding, or that completing a handoff brief proves correctness, approval, or legal compliance.

Download

Previous:

More Content to Discover:

What looks like a “trust moat” may be a pile of verification, transition, and continuity risks the newcomer has not removed.

👋 Welcome to my paid subscriber-only edition of Empathy Engine (🔒 Leader’s Dispatch). Each week I build evidence-forward tools for product leads who need to say no, defend tradeoffs, and lock in decisions before they get rewritten later.

There are two stories about the same kind of moment, and they refuse to agree with each other.

In the first, a company runs an expensive diagnostic, can’t fix the problem, and quotes a full replacement. The homeowner calls the one-person plumber she already uses. He’s there in twenty minutes. He fixes it in ten.

That is the story trust is supposed to explain. The familiar provider wins. The bigger operation loses.

In the second, a general contractor’s regular guy goes quiet. No call, no text, no kept window. The GC calls a larger outfit he’s never used before. They show up when they say they will. He pays a documented 322% premium in that one case, and does not regret it.

Same category of decision. Opposite ending.

If trust explains the first story, it has to explain why it vanished in the second. If reliability explains the second, it has to explain why the plumber’s homeowner didn’t go looking for a bigger, unfamiliar brand instead.

It is tempting to call one of these irrational. It is even more tempting to reach for a single word, trust, and let it explain both. The incumbent has the relationship, the familiar name, the years of history. The challenger has the better system. Surely trust is the variable that decides it.

That explanation is satisfying because it makes the problem feel emotional. It also hides most of the work.

The buyer may not be protecting a relationship. The buyer may be protecting a filing deadline, a set of credentials, a history of exceptions, a fragile cutover, or a process no one has documented well enough to move. The buyer may not know whether your cleaner method will produce a better result. They may not even be able to tell whether the work you propose is necessary.

The better offer may still leave the buyer with the harder problem.

That is the uncomfortable idea in this episode. In service-heavy markets, the product is rarely just the promised output. The purchase can also include the buyer’s ability to verify the work, transfer accumulated context, survive the transition, and recover when something goes wrong.

Call that trust if you want. Just do not stop there. What the newcomer sees and what the buyer still has to solve are two different lists, and confusing them is the whole mistake.

Research Binder: the receipts, methodology notes, and source boundaries are compiled at the bottom of this post.

None of that newcomer list buys a shorter transition. A cleaner workflow doesn’t verify itself, a sharper report doesn’t move a credential, and a faster response time doesn’t schedule a safe cutover. The switch has its own price, whether you put it on the invoice or the buyer discovers it later.

“Trust” is a crowded suitcase

When a buyer says, “I trust my accountant,” the sentence can hold several different meanings.

It might mean:

• She knows the history behind the books.

• He remembers which customer always pays late.

• They have the portal access, payroll calendar, prior filings, and supporting records.

• Someone answers when a deadline gets close.

• The buyer believes the provider is competent.

• The buyer cannot easily judge whether the provider is competent.

• The current arrangement is familiar.

• Changing it would be disruptive.

Those are not interchangeable.

Trust is a willingness to rely on someone when you are vulnerable. Reputation is information borrowed from other people. Familiarity is recognition. Satisfaction is an evaluation of past experience. Loyalty is continued preference or behavior. Switching cost is what makes leaving expensive. Lock-in is what constrains the exit. Responsiveness is communication behavior. Availability is capacity.

Keep them separate and the market starts to become inspectable. That is the whole method behind this episode: six separate questions instead of one flattering word. The first one is whether the buyer can actually judge the work at all.

I’ve seen this in coaching work more times than I can count. A leader will tell me, “We trust her,” about the person everyone depends on. Then we start pulling the sentence apart. She knows where everything is. She remembers why the exceptions exist. People know she’ll answer. And if she disappeared for two weeks, nobody is entirely sure who could reconstruct the history. Some of that is trust. Some of it is competence, context, responsiveness, and a dependency the organization hasn’t documented yet.

The buyer may not be able to check the work

Some services contain what economists call credence attributes. The buyer may be able to see that something improved while still being unable to determine whether the prescribed intervention was necessary, whether a less expensive intervention would have worked, or whether the expert chose the right level of service.

That distinction matters. A repair can work without proving that it was the only repair the buyer needed. A tax return can be filed without making every underlying judgment legible to the client. An IT environment can become more stable without the owner being able to evaluate every configuration decision.

Controlled experiments in stylized credence markets show that the rules around expert responsibility and independent checking can materially change provider behavior.¹ In one laboratory study, giving buyers access to a second opinion reduced overtreatment, but the value of that option weakened when searching became more costly.²

Those findings do not prove widespread overtreatment in bookkeeping, accounting, or managed IT. They do not make every expert service a credence market. They do establish a narrower point: when the buyer cannot fully evaluate what should have been done, a technically superior claim is not self-authenticating.

“We do this better” asks the buyer to evaluate the very thing the buyer may not be equipped to evaluate. A working fix and a necessary fix are not the same claim.

The buyer can usually confirm the screen went from broken to fixed. They may still be unable to tell whether the diagnosis, the replacement, the reconfiguration, and the test were all necessary, or whether a smaller intervention would have worked. That gap is exactly what a “we do it better” pitch has to close, and a case study or a credential doesn’t close it by itself. It only works when the buyer has some independent way to recognize better, not just a claim of it.

The switch has its own product requirements

Imagine a small business replacing a long-time bookkeeper.

The new provider is not inheriting a clean list of monthly tasks. They may be inheriting years of categorization habits, undocumented exceptions, account mappings, spreadsheet patches, owner preferences, payroll rhythms, filing dependencies, and unresolved questions.

Or imagine replacing a managed IT provider. The new firm may need administrative access, device inventories, licensing records, network documentation, backup information, vendor contacts, security configurations, and a workable sequence for taking control without interrupting the business.

Research on business-to-business switching costs separates procedural, financial, and relational burdens rather than treating switching as one price tag. In one qualitative study of recent staffing-agency switchers, researchers identified eight distinct switching-cost facets: uncertainty, search, learning, setup, sunk investment, lost performance, brand relationship loss, and personal relationship loss.³ In a companion study of one supplier’s business customers, relational switching costs predicted lower actual switching even after satisfaction was included, while satisfaction was the strongest predictor of share of wallet.³ That combination is important. It means staying can reflect real satisfaction, real relationship value, real switching burden, or some mixture of all three.

This is where lazy disruption stories fail. They look at an incumbent’s weak interface or slow workflow and assume the buyer is tolerating obvious mediocrity. But a buyer can dislike part of the service and still make a rational decision to stay, because the task in front of them is visible and the context behind it is not.

That changes the economics of entry. The newcomer isn’t only selling next month’s bookkeeping. Someone has to pay for recovering last year’s context first, and that cost may not be in the proposal yet. Some of what’s on that stack may be documented and easy to move. Some of it lives only in a person’s memory, and can become expensive to transfer the moment that person stops answering.

The relevant comparison is not:

Old provider versus better provider.

It is:

Old provider, with known performance and embedded context, versus a new provider plus the cost and uncertainty of becoming operational.

That does not prove the incumbent has a moat. It means the entrant has two jobs: improve the service and make the transition survivable.

Accumulated context is useful until it becomes invisible

Relationship history can create genuine value.

The provider may know why the business does something that looks inefficient. They may remember the customer exception that never made it into the procedure. They may understand which recurring error is harmless and which one signals a serious problem. They may know who actually makes a decision despite what the org chart says.

That knowledge is not identical to trust. It is operational context.

The danger comes when context exists only inside a person, an inbox, a spreadsheet, or an undocumented sequence of workarounds. At that point, the relationship may be valuable and fragile at the same time.

But context living somewhere inconvenient is not automatically the same thing as a provider holding a buyer hostage. Some of what looks like lock-in is just ordinary transition work, done badly or not at all.

Share

For tax practitioners, federal rules impose duties to return client records needed for compliance, subject to defined limits and exceptions.⁴ That is an authoritative statement about professional obligations. It is not evidence that records are routinely withheld, nor does it measure how much portability changes switching behavior. It defines a narrow professional duty, not the prevalence of lock-in.

The practical inspection question is simpler:

Can the buyer identify what must move, who controls it, and what a safe handoff requires?

If the answer is no, the entrant has found transition exposure. They have not yet found misconduct, demand, or a profitable opportunity.

Timing can turn friction on or off

The same buyer may be highly switchable in June and nearly immovable in March.

Accounting has filing cycles. Payroll has recurring deadlines. Managed IT has maintenance windows, renewals, migrations, and periods when interruption is unusually costly. A planned boundary can lower transition risk. An active crisis can either accelerate switching or make any transition intolerable.

Read more

Plain-English premise: AI can draft the deck in minutes. Someone still has to understand it before it moves.

Judgment Seat: The reviewer who inherits the reading the moment someone says “quick review.”

👋 Welcome to the opening of The Judgment Layer. For seven issues, this series will examine what happens after AI makes output easier to produce: who reads it, verifies it, challenges it, and ultimately stands behind it. We begin with the last human in the chain and the Review Budget that turns finite attention from an invisible burden into an explicit decision.

Research Binder: the receipts (citations + source notes) are compiled in a PDF at the bottom of this post.

⚡ Pressure

Generation got faster. In many workflows, the remaining verification capacity is still unnamed.

🎯 Payoff

This issue gives you the Review Budget: a way to name your team’s finite review capacity and route the work that does not fit it.

Pressure Moment

A teammate used AI to draft a slide deck. It reached a manager, who asked for a “quick review.” The manager ran the deck back through AI, then sent it forward for confirmation. The deck kept moving. What actually got checked stayed unclear. That is one practitioner’s public account, not a documented record of every hand the deck passed through or a measured time cost. Read it as a recognizable pattern, not a verified case file. The pattern it points to is ordinary: an artifact can look finished several times over before anyone has actually finished the judgment underneath it.

I’ve had AI-drafted decks land in my inbox after two or three people had already given them a “quick review.” The slides looked finished, the language sounded confident, and the status updates were neatly summarized. Yet once I started checking the story against the actual work, I found the missing dependency, the unresolved decision, and the risk that had somehow softened into a “watch item.” Nobody had been careless. Each person had reviewed a different piece of the deck, and by the time it reached me, “Can you take a quick look?” really meant, “Can you figure out whether any of us should trust this?”

🧭 The Judgment Move

Orienting sentence: A Review Budget names how much qualified review capacity your team actually has this week, matches depth to consequence, and routes anything that does not fit instead of quietly shrinking the check.

The reviewer’s job was never “read it fast.” It was: recover the intent, corroborate the claims, correct what is wrong, and decide whether the work can stand behind it. “Quick” was never a time estimate. It named the handoff, not the four jobs sitting inside it.

Step 1: Name this week’s review capacity in three tiers, light, standard, and deep, at the team or role level. Do not track it against individuals.

Step 2: For each artifact ready for approval, ask what breaks if it is wrong, how reversible the decision is, what remains uncertain, and whether a valid check can catch failure.

Step 3: Assign review depth from those answers, not from how the request was worded. Urgent is not a depth.

Step 4: When incoming demand exceeds available capacity, route the overflow. Return it, narrow it, defer it, automate the parts with valid deterministic checks, reassign it, or escalate it.

Step 5: If none of those routes are available and the work must move anyway, record that the risk was consciously accepted, and by whom.

Done state: every artifact that reaches approval carries a named review depth and a named owner. Any missing time, context, expertise, or authority is visible before approval, not after.

Constraint: the Review Budget only works if leadership treats a depleted budget as a routing decision, not a performance problem.

I remember sitting with a leadership team after several deliverables had backed up waiting for approval. Their first question wasn’t, “Which work should move first?” It was, “Who owns this?” A name went onto the screen, everyone relaxed slightly, and the meeting moved on. Nothing had been deprioritized, no additional reviewer had been assigned, and the same person still had more work than anyone could responsibly check. We hadn’t solved the capacity problem. We had simply given it someone’s name.

Limitation: this move cannot manufacture reviewer expertise, add hours to a day, or prove that a lighter review was safe. It only makes the shortage visible enough to govern.

🧪 What Holds Up

Primary claim label: Evidence.

Basis: A randomized study of sixteen experienced developers across two hundred forty-six real repository tasks found that AI access made completion nineteen percent slower, even though the same developers had expected a twenty-four percent gain. Generation speed, checking effort, and total completion time are three different measurements, and the first does not predict the other two.

Share

A separate meta-analysis of one hundred six experiments found that human-AI combinations underperformed the better standalone actor on average, even while outperforming humans working alone. A human’s presence in the loop is not, by itself, evidence of meaningful oversight. Oversight requires attention, context, expertise, and the authority to actually stop the work, not just a name on the approval.

Counterpoint, labeled Evidence: this does not mean AI always adds review work. A workplace study of over five thousand customer support agents found a real throughput gain, concentrated among newer and less experienced workers, in a bounded, correctable task. A separate study of professional video annotators found faster completion and higher accuracy under a high-recall AI condition. When a task is bounded, success criteria are stable, the output is inspectable, and a reliable correction path exists, AI can reduce total effort without moving work downstream at all.

Scope and transfer limit: the studies behind this issue concentrate in software engineering, fact-checking, and bounded annotation work. Comparable field evidence for Product, Program, Design, and Research artifacts does not yet exist. Treat the mechanism as real and the cross-functional prevalence as unproven.

What remains unknown: how often this transfer happens outside software, and whether the Review Budget itself reduces defects, delay, or burnout. It has not been tested as a bundled intervention. Present it as a design inference, not a validated fix.

🛑 Do Not Use When

1. The task is bounded, the success criteria are stable, and a reliable automated or deterministic check already exists. Route it to light review or automation instead.

2. You cannot name who has the authority to stop the work. A Review Budget without stopping power just documents who absorbed the risk.

3. The goal is to justify a compliance claim. Neither NIST guidance nor the EU AI Act validates this method.

Judgment Seat

I’ve generated the work, inherited work I then had to understand, and paid the price when something moved forward incorrectly. The seat I’ve deliberately stayed out of is pretending I could stop something when the decision rights belonged to an executive, product owner, or government leader. I could surface the risk, make the tradeoff visible, and recommend a pause. Calling that authority, however, would’ve let the actual decision-maker disappear from the room.

Who generated the work: whoever used the fastest available tool to produce something that looked done.

Who must understand it: the person whose name goes on the approval, whether or not they built it.

Who can stop it: only whoever has been given real authority to return, narrow, or escalate it, not just to sign it.

Who pays if it is wrong: the person asked to approve or defend the artifact can inherit consequences they did not create, and uncertainty they were not given time to resolve.

🎁 Your Review Budget Statement

I’m including the Review Budget Statement in two complementary formats because naming limited attention is only useful if a team can act on that constraint. The one-page PDF helps create the decision together; the Excel workbook helps carry that decision through the week. Both versions turn “Can someone give this a quick review?” into a more honest conversation about available capacity, required review depth, and where work should go when the two don’t match.

Review Budget Statement — One-Page PDF

Use the printable PDF to establish a team’s review budget during planning, intake, or a live leadership conversation. In one page, the team can name its available Light, Standard, and Deep review capacity; match review depth to consequence, reversibility, uncertainty, and valid automated controls; test whether meaningful human oversight is actually possible; and choose an explicit route for work that doesn’t fit.

Review Budget Statement

382KB ∙ PDF file

Download

The PDF is intentionally simple enough to complete together on a screen, whiteboard, or printed page. Its value lies in making the tradeoff visible before overloaded reviewers are left to negotiate it privately. Rather than quietly weakening the review, the team must return, narrow, defer, automate valid checks, reassign, or escalate the work.

Download

Review Budget Statement — Operational Excel Workbook

Use the Excel workbook when the Review Budget needs to become a working operating practice rather than a one-time conversation. The workbook provides a weekly capacity view, an incoming Artifact Log, automated tracking of committed and remaining review slots, and a Meaningful-Oversight Gate that checks for attention, context, expertise, time, and authority.

When an artifact exceeds available capacity, the workbook makes the routing decision explicit instead of allowing “urgent” to masquerade as a review depth. It also includes a Review Gaps & Residual Risk Log so teams can record what wasn’t checked, what remains unverified, who consciously accepted the risk, and what should trigger reconsideration. No invisible shortcuts. No individual productivity leaderboard. No pretending qualified human attention is an unlimited organizational resource.

Review Budget Statement Workbook

91.7KB ∙ XLSX file

Download

The Review Budget Statement is an evidence-informed design inference, not an independently validated bundled intervention. I’m gifting both versions so teams can begin testing the underlying practice immediately: fund the review, match it to the consequence, and route the work that doesn’t fit.

Download

This Week

Three actions for the judgment owner: name your own light, standard, and deep review slots for this week. Pick one artifact currently in your queue and assign it a depth before you open it. Return one thing that does not fit your available depth instead of rushing it through.

One action for the system owner: decide, out loud, who has the authority to escalate an overloaded review queue, and tell your team.

One action for learning and provenance: for one reviewed artifact this week, write down what you checked and what you didn’t, so the gap is visible to whoever reads it next.

📎 Forward This

A “quick review” request does not come with its own time estimate. Name what must be checked before you accept the depth it is asking for.

🔚 Closing

Subscribe now

The output moved fast. The understanding did not move with it.

Someone still has to decide whether it deserves to.

Next week, Episode Two turns the responsibility upstream. If You Send It, You Own It asks what every submitter should be able to explain about AI-assisted work: what changed, why it changed, what was checked, and what remains uncertain.

Because authorship responsibility doesn’t transfer to the reviewer.

This question is part of the larger challenge behind my upcoming book, Collaborate Better: From Silos to Synergy – How to Build Unstoppable Teams. Better collaboration begins when context, ownership, and judgment travel with the work.

Learn more at CollaborateBetter.us.

Regards,

Mark 👋

Citations And Research Binder

204KB ∙ PDF file

Download

This Citations and Research Binder supports Judgment Layer Episode 001: *The Last Human Who Read It*. It gathers the research behind the article’s claims about AI-assisted generation, downstream verification work, finite reviewer attention, meaningful human oversight, and the conditions under which AI can increase or reduce total effort. The binder also supports the Review Budget Statement’s core principles: naming available capacity, matching review depth to consequence, testing whether meaningful oversight is possible, and routing work that exceeds the budget. It distinguishes peer-reviewed findings, field studies, institutional guidance, qualitative research, counterevidence, and Empathy Engine synthesis while defining the limits of each. The complete Review Budget Statement remains an evidence-informed design inference rather than a validated intervention, and the research does not establish that AI always creates more review work, universally reduces productivity, or that assigning a human reviewer guarantees reliable verification or safe approval.

Download

Previous:

More Content to Discover:

The spreadsheet can be real. The delay can be measurable. The business can still be imaginary.

👋 Welcome to my paid subscriber-only edition of Empathy Engine (🔒 Leader’s Dispatch). Each week I build evidence-forward tools for product leads who need to say no, defend tradeoffs, and lock in decisions before they get rewritten later.

The spreadsheet was ugly. That wasn’t the signal btw

The signal was that the same work kept traveling through it, people had built behavior around it, and a mistake could escape the workflow and reach a customer.

Twelve tabs. Estimates in one place. Crew assignments somewhere else. Warranty notes copied because the official system doesn’t quite carry them where they need to go. Customer follow-up divided among software, memory, text messages, and whatever somebody remembered to write down before leaving for the day.

Episode 1 Part A:

Episode 1 Part B:

A few weeks ago, Nina would have loved this. She had already paid for the expensive lesson of starting with a fashionable category and working backward toward a problem. Episode 1 taught her that a category can feel validated before a single buyer has validated anything. Episode 2 taught her that a business can hand off nearly every task and still keep every hard decision in one head.

Episode 2 Part A:

Episode 2 Part B:

Now she has a harder problem. Too many ugly workflows, and no defensible way to know which ones deserve another question.

That distinction, between a workflow that has earned inspection and a workflow that has earned a business, is the reason this third episode became more complicated than expected.

The series premise stays intact. Boring itself creates nothing. We’re looking for repeated paid pain, not beige industries.

Episode 3 spends its time on the word we’re most likely to rush past: repeated. Before anyone can ask whether someone will pay to remove a pain, we need to know whether we’re looking at a recurring workflow at all, what consequence follows it, who actually carries the work, whether the same ritual shows up somewhere else, what the workaround already accomplishes, and whether the design that looks broken might already be rational.

That’s what I’m calling the Workflow Pain Index.

The name stays. The score doesn’t.

This isn’t a 0 to 100 instrument. Nothing here deserves weighting, traffic lights, a high opportunity label, or a threshold that quietly converts annoyance into demand. The useful version is a qualitative inspection filter. It helps decide whether a workflow has earned another question, nothing more.

Research Binder: the receipts, methodology notes, and source boundaries are compiled at the bottom of this post.

Ugly Isn’t the Opportunity

MEASURABLE PAIN ≠ PROVEN DEMAND

Consider prior authorization, a workflow with nothing to do with HVAC. A 2026 JAMA Health Forum analysis examined 205,896 U.S. retail-pharmacy transactions initially rejected for prior authorization.1 Thirty five percent were processed within one day. The other 65 percent took multiple days, with a median of six days and an interquartile range of three to twelve.

That is a measurable consequence. Delay happened. It can be counted and described. It still cannot be leapt from to therefore somebody will buy our solution.

1. A Bad Day Isn’t a Pattern

The first question is recurrence, and it starts smaller than most builders want it to. A missed handoff, a late invoice, a bad Friday: those are incidents. Memorable is not the same thing as recurring.

Three questions separate a story from a pattern. What exactly repeats. How often. Over what period. If we can’t answer those three, we don’t yet have recurrence evidence, no matter how vivid the story is.

Medicare Advantage gives us a cleaner example of what countable recurrence looks like. KFF’s analysis of CMS administrative data found 52.8 million Medicare Advantage prior-authorization determinations in 2024, equal to 1.7 determinations per enrollee across 33 million people.2

Those figures don’t say whether prior authorization is commercially attractive to an outside builder, how consequential any single determination was, or whether anyone has permission to intervene. They say something narrower: the workflow has a countable recurring unit. That is enough for this pillar, no more.

One bad Friday is memorable. Friday after Friday is a ritual. A messy office is a tour. A weekly ritual with a name is a clue.

Nina tries this on Dave’s business first. He tells her exceptions land on his phone every Friday, and he has a hunch it gets worse in busy season. When she actually tries to name the unit and the period, she can’t, not yet.

That gap is not a failure. It is the next thing to go find out.

2. Frequent Doesn’t Mean Consequential

Frequency asks how often something happens. Consequence asks what happens when it does. Those two questions have to stay separate. Adding them together in your head, instead of asking them one at a time, is how an ugly-looking workflow gets promoted into a false opportunity.

An analysis of operational-loss events across 121 banks in 17 countries found the single most frequent event type accounted for 30.6 percent of all events but only 24.9 percent of total loss. A lower-frequency event type accounted for just 18.2 percent of events and 52.4 percent of loss.3 The thing that happened more often was not the thing that produced the larger share of loss.

Consequence can also be measured cleanly, per event, when the transaction is well defined. The 2024 CAQH Index put the provider cost of a manual medical prior authorization at roughly 24 minutes and $12.88, versus 10 minutes and $5.38 fully electronic.4 That’s a real, bounded, per-event number. It still measures burden, not market demand.

Share

3. The Person Doing the Work May Not Control the Fix

Three chairs, not one. The doer enters the record, handles the request, reconciles the exception. The workflow manager owns the queue, catches exceptions, maintains the workaround. The decision authority approves a process change, allocates resources, decides whether the current design stays. In a microfirm, one person can hold all three chairs. As firms grow, the work spreads, and the three chairs stop being the same person.

A 2017 NFIB survey of 750 U.S. small employers found that owners personally performed financial paperwork and recordkeeping 58.1 percent of the time in firms with 1 to 9 employees. That share fell to 36.7 percent at 10 to 19 employees, and to 22.5 percent at 20 to 249 employees, while employee and combined execution rose to fill the gap.5

That measures who does the work. It says nothing about who controls the budget, whether approval is slow, or whether a solution will ever get purchased. Don’t ask only who hates this. Ask who does it, who manages it, and who can change it, and expect three different names, not one.

Nina has been asking Dave’s dispatcher how she feels about the Friday scramble. That was the wrong first question. The dispatcher is the doer. Dave is the decision authority. Nobody has told Nina yet who the workflow manager even is.

4. One Company Is a Story. Repeated Rituals Become Evidence.

Read more

Plain-English premise: A consequential roadmap decision is not finished when everyone nods. It is finished when the trigger was tested, the trade-off was named, the state was declared, the reasoning was captured, the interpretation was confirmed, and one current record was published where execution can find it.

👋 Welcome to the capstone of Executive Override. For six issues, this series treated one pressure moment at a time: the override with no owner, the urgent ask with no tradeoff, the stakeholder room that agrees and then fragments, the rationale that evaporates, the record that cannot close, the commercial escalation that impersonates a decision. This week puts those moments on one operating sequence, stage by stage.

Research Binder: the receipts (citations + source notes) are compiled in a PDF at the bottom of this post.

Episode 6:

⚠ Tension

The room believes the fight is over. By Friday, three functions are executing three different versions of what just got decided.

🎯 Payoff

Inside: the Logged Decision Workflow, a six-stage sequence for turning agreement into one decision execution can actually inspect and use.

Trigger Moment

I’ve watched one meeting produce three honest plans: the plan leadership believed it authorized, the plan Product believed it negotiated, and the plan Engineering believed it could deliver. Agreement wasn’t the problem. The decision never became transferable.

A cross-functional roadmap review had just landed. Product, Engineering, Operations, and a commercial lead had spent the hour on the same collision: keep the original release window, absorb a late enterprise request, or slip a commitment customers could already see. People were tired. Someone said the sentence that ends too many of these rooms: “I think we’re aligned.” Laptops closed.

By Friday, the delivery lead was working the original window. Sales was briefing the customer on the new request. The sponsor remembered authorizing a risk exception nobody had written down. Three honest readings of one meeting, later becoming three Slack threads and three different builds. No one was lying. The decision agreed to in the room never made it into the systems where execution actually lives.

That is the failure this series has been circling from six different angles. Not the disagreement. The missing hand-off from a room that felt closed into a record that could travel.

The meeting ended. The decision didn’t travel.

The room can agree while execution inherits different calls.

Notice what the diagram actually traces. Not who was wrong, but where the decision stopped carrying information. Authority, trade-off, rationale, and current record all survived the meeting. None of them survived the hand-off. That is the whole argument in one picture, and it is the reason a single “are we aligned?” question in the room is never enough. The fix is not a better recap. It is asking each executing function to restate the decision, the authority behind it, and the trade-off it accepted, before anyone leaves.

🎒 The Move: The Logged Decision Workflow

Orienting sentence: When a consequential decision looks resolved in the room, run six stages before anyone leaves, so the trigger, the tradeoff, the state, the reasoning, the interpretation, and one current record travel together into execution.

This sequence is something I’ve assembled from practices that already have their own track record elsewhere. It hasn’t been tested as one integrated method. Treat it as a structured way of working, not a finding you can cite.

Stage 1: Recognize the trigger. Not every message is a decision event. The channel a request arrives through doesn’t decide its weight. A Slack DM can move a commitment as much as a formal review can. One of the most consequential decisions I’ve had to catch wasn’t made in a decision meeting. It arrived disguised as a quick update, then revealed itself when the roadmap no longer matched what teams had been told to deliver.

CMMI’s own decision-analysis guidance exists for exactly this filter. Organizations need pre-set criteria for which issues warrant a formal process, on the stated grounds that not every decision is significant enough to earn one. NASA runs the same logic through four conditions: complexity, uncertainty, competing attributes, diverse stakeholders. Borrow the discipline, not the bureaucracy. If cost, schedule, or scope hasn’t moved past a threshold you set in advance, if the risk is low, if nothing external has been promised, keep it lightweight and move on. The test is materiality, not formality.

Stage 2: Expose the trade-off. A new priority does not create new capacity on its own. Name which move makes it real: remove an existing commitment, delay one to a later date, resize the new or existing scope, fund real capacity, or accept the risk and name who owns it.

New work doesn’t create new capacity.

A new priority requires a named portfolio move.

“We’ll make it work” is not one of those five moves. It’s two live plans wearing one name. The practical discipline is a short commitment ledger: what moved, who decided it, who owns the consequence, which plan now governs. Cooperation from the team is valuable. Cooperation is not capacity. I once told a leader, ‘We can absolutely make this the priority, but we can’t make it additional capacity.’ Naming [existing commitment] as the work that would move turned an urgent preference into an actual portfolio decision.

Stage 3: Name the decision state. Teams routinely compress three different conditions into the word “aligned”: people understood what was said, people supported the direction, and someone with actual authority committed the organization. Those three can coexist. They are not interchangeable, and a room can clear one while quietly failing the other two.

Declare one of five states instead: proposed, decided, deferred, superseded, reopened. This isn’t just tidiness. Decision rights are genuinely contested in practice. At one automaker, 83% of marketers and 64% of product developers each separately claimed the right to decide standard features, and the new models were delayed while that fight played out (Rogers and Blenko, HBR, 2006). Unresolved role ambiguity carries a real, if modest, drag on individual performance across a meta-analysis of 169 samples (Gilboa et al.). Neither finding proves that naming a state fixes the underlying authority conflict. What it does is make the conflict visible enough to argue about directly, instead of discovering it three weeks later as a memory fight.

Stage 4: Capture the decision logic. A final answer is not the same as a usable record. “Delay the launch” can survive for months while the reasoning underneath it quietly goes stale.

The answer survived. The why didn’t.

Preserve enough that someone who wasn’t in the room could later explain why the option won: the decision, the rationale, the alternatives considered, the assumptions and constraints, the consequences accepted, who had the authority to decide, and what would justify reopening it. That’s more than most teams keep and less than most templates ask for. In two controlled experiments, a value-based subset, six of thirteen possible information categories, cut documentation effort 40 to 70 percent depending on the task, without any claim about whether the underlying decisions were better (Falessi et al.). Keep it small on purpose. I once had to reopen a decision months after the answer had been recorded but the reasoning had scattered. We spent more time reconstructing why the choice had made sense than evaluating whether it still did.

Stage 5: Confirm the interpretation. A reply confirms that communication happened. It does not confirm the decision arrived intact.

Share

“Sounds good,” a thumbs-up, “got it,” all three close the message. None of them tell you whether Product, Engineering, and Marketing now believe the same thing changed. Ask each executing function to restate what starts, stops, or moves, and have the person who made the call confirm or correct that restatement before treating it as settled. The evidence for why this matters comes from outside product work, and it’s worth being honest about that gap. In simulated operating-room emergencies, directed closed-loop requests completed 18 of 18 times, against 81 percent otherwise (Lee et al., BJA). But among paramedics actually trained in the technique, only 2.7 percent of 676 real communication events closed the loop the textbook way (Marzuki et al.). That’s the caution worth sitting with. The method works when people actually run it. Most people, most of the time, don’t.

Stage 6: Publish the current record. “One source of truth” gets misheard as “one enormous file,” which is how good records turn into abandoned ones.

One current record doesn’t have to mean one file.

One current answer does not require everything in one document.

One place declares what’s current. Links carry the decision out to the backlog, the risk register, the customer commitments, wherever those artifacts already live. Superseded versions stay visible as history instead of quietly disappearing. In a controlled study, 71 participants using traceability links completed maintenance tasks 24 percent faster and produced 50 percent more correct solutions than participants working without them (Rempel and Mäder). That’s evidence for finding and changing the right thing later. It is not evidence that the original decision was good. The record should make change visible. It shouldn’t make a bad decision permanent.

Constraint: Run the sequence in the meeting or the same day. Six stages, one declared record. No seventh ceremony, even though the evidence would probably support one (verifying implementation, confirming supersession). Six is what fits on a page without losing legibility.

Done state: One current record exists, the named authority is on it, affected functions restated the consequence and had it verified, and a revisit condition is dated or conditional.

Limitation: The workflow cannot invent decision rights, repair bad faith, or make an unsafe record culture safe. It makes a decision inspectable. It does not make every inspectable decision wise.

🖊 What Holds Up Under Pressure

Primary claim label: Evidence-informed components; the full sequence is untested as a package.

Basis: Every stage above draws on something with its own track record: decision-rights conflict, documentation-effort research, closed-loop communication, traceability. What none of those sources tested is this exact six-stage sequence run together, in order, on a live roadmap decision.

Counterpoint: Clearer roles help coordination. Captured rationale saves reconstruction work. Confirmed interpretation catches misalignment before it ships. None of that proves that running all six stages, in this order, prevents relitigation or produces compliance. The confirmation stage carries the largest transfer risk in the set. It’s a design analogy borrowed from clinical and high-reliability work, not a finding about product teams.

Limitation: Use the evidence to justify the components, not the finished workflow. The six-stage sequence is how I’ve stitched the series together. That’s a design choice, and it’s labeled as one here and on the tool itself.

🛑 Do Not Use When

1. The change is low-stakes. Button copy, standup swaps, and harmless backlog shuffles don’t clear the materiality test in Stage 1, and running six stages on them is documentation theater.

2. Decision rights are missing, or the record will be used as a weapon. EO-005 already named those failure modes. A better-labeled sequence won’t repair either one.

3. You’re still in a real emergency. Stabilize first. Run the workflow on the planning conversation that follows the fire, not during it.

Decision Logged: The Decision Examination Board

A decision doesn’t become durable because someone announced it, everyone nodded, or work already started. It becomes durable when another person, someone who wasn’t in the room, can inspect it.

Logged Decision Workflow: Fillable Edition

Complete the six-stage workflow directly from a computer or tablet while the decision is being made or immediately afterward. The interactive fields help capture the trigger, trade-off, decision state, authority, reasoning, shared interpretation, and authoritative current record in one portable document that can be saved, circulated, and linked to execution.

Logged Decision Workflow Fillable

378KB ∙ PDF file

Download

Choose this edition when the decision will be documented remotely, shared electronically, or maintained as part of a digital audit trail.

Download

Logged Decision Workflow: Print Edition

Print this one-page edition and bring it into the room when handwriting will keep the conversation moving faster than another open application. The spacious writing areas and six-stage layout help the facilitator capture the trigger, trade-off, decision state, authority, reasoning, shared interpretation, and record location as the discussion unfolds.

Logged Decision Workflow Printable

349KB ∙ PDF file

Download

Choose this edition for in-person workshops, planning sessions, executive reviews, or any meeting where the completed sheet can later be transferred into the authoritative digital record.

Download

Subscribe now

Questionable does not mean undecided. It means inspectable. A decision that can absorb “why did this option win?” three months later without anyone treating the question as an attack is a decision that actually held. One that can’t is a decision that only looked finished on the day it was made.

This Week

As a coach, I listen for the moment a room says, ‘We’re aligned.’ That’s usually when I ask each function to describe what starts, stops, or moves next.

One action for the team: Before the next material roadmap review, write the six stage names at the top of your notes. Don’t let the meeting close until the trade-off, the state, and the record location each have a stated answer.

One action for the leader in the room: Before anyone stands up, say the decision state out loud, proposed, decided, deferred, superseded, or reopened. Not “we’re aligned.”

One definition of done: The decision has traveled when its state and authority are declared, its trade-off and reasoning are recorded, affected functions have restated the consequence, one current record is findable, and the reopening condition is named.

The series, in working order

EO-001 taught the receipt. EO-002 taught the pushback. EO-003 taught the alignment agenda. EO-004 taught the durable record. EO-005 taught where templates break. EO-006 taught the commercial escalation. EO-007 is the order those tools run in when they all have to fire in one room, on one clock.

📎 Forward This

A decision is not finished when the room agrees. It is finished when the trigger was tested, the trade-off was named, the state was declared, the reasoning was captured, the interpretation was confirmed, and one current record was published where execution can find it.

That same operating belief sits at the heart of my upcoming book, Collaborate Better, because better collaboration is not louder agreement. It is the discipline of making tradeoffs, ownership, and next steps visible enough for people to move together. Learn more at CollaborateBetter.us.

⬅ Closing

The meeting can close. The decision still has to travel.

If execution cannot inspect one current record, the fight is not over. It only changed rooms.

Next week, Empathy Engine launches a brand-new series: The Judgment Layer. AI made generating work cheap, but someone still has to decide what’s true, useful, safe, and worth acting on. This series follows the professionals quietly inheriting that burden, then gives them practical ways to reclaim judgment before speed outruns understanding.

Regards,

Mark 👋

Research notes

Rogers, P., and Blenko, M. (2006). Who Has the D? Harvard Business Review. Gilboa, S., et al. Role ambiguity meta-analysis. CMMI DAR SP 1.1; NASA Systems Engineering Handbook 6.5, 6.8. Falessi, D., Briand, L., Cantone, G., Capilla, R., and Kruchten, P. Simula Research Laboratory. Lee, et al. British Journal of Anaesthesia 135(5). Marzuki, et al. Paramedic closed-loop communication observation. Rempel, P., and Mäder, P. IEEE Transactions on Software Engineering. Full citations live in the Research Binder attached to this issue.

Citations And Research Binder

260KB ∙ PDF file

Download

This Citations and Research Binder supports Executive Override Episode 007: *How to Finish a Roadmap Decision Before Execution Inherits Three Different Answers*. It gathers the research behind the article’s claims about material decision triggers, displaced capacity, decision rights, decision states, preserved rationale, closed-loop communication, traceability, authoritative records, supersession, and appropriate reopening conditions. The binder also supports the six-stage Logged Decision Workflow of Recognize, Expose, Name, Capture, Confirm, and Publish, along with the fillable and print editions used to carry a consequential decision from discussion into execution. It documents the evidence behind each component while distinguishing established findings, adjacent-domain evidence, practitioner guidance, and Empathy Engine synthesis. The binder also defines the limits of those claims. The complete workflow is treated as an evidence-informed design choice rather than a validated intervention, and the research does not establish how frequently roadmap decisions fragment, that read-back guarantees execution, that documentation makes a decision wise, or that one authoritative record guarantees compliance.

Download

Previous:

More Content to Discover:

👋 Welcome to my paid subscriber-only edition of Empathy Engine (🔒 Leader’s Dispatch). Each week I build evidence-forward tools for product leads who need to say no, defend tradeoffs, and lock in decisions before they get rewritten later.

Part A watched Dave’s team absorb nearly everything. Scheduling moved to a dispatcher. Routine service moved to the crews. Standard pricing moved to an estimator. Customer support moved to an office coordinator. Then a pricing exception, a technical edge case, a customer repair, and an unusual approval all arrived at Dave’s phone in a single Friday.

Episode 2 Part A:

Part A asked where the judgment lives. Part B asks whether it needs to live there.

That question needs more than one Friday to answer honestly, so Nina builds a structured way to inspect it. Five dimensions, watched separately: Judgment, Relationships, Knowledge, Exception Routing, and Observed Continuity. One checkpoint that never gets scored: Authority and Credentials. No total. Concentrated does not automatically mean broken, and Unknown is allowed.

Research Binder: the receipts (citations + source notes) are compiled in a PDF at the bottom of this post.

Signing Off Is Not the Same as Deciding

Nina watches two approvals move through Dave’s business in the same week.

The first is small. A dispatcher already knows a customer complaint deserves a modest fee waived. She walks it to Dave anyway. He glances up, says yes, and the conversation is over in under a minute. Nothing about his answer required anything Dave alone could supply.

The second is the pricing exception from Friday, a retrofit condition the estimator hadn’t priced before. Dave asks two questions the estimator hadn’t considered, changes the number, and takes on a liability call nobody else on the team was positioned to make.

One escalation reached Dave because the answer needed him. The other reached him because the org chart still said it had to. Micro-level evidence from 761 decision tasks inside one Fortune 50 process found that knowledge specialization and information demands, not habit alone, shape where decision authority actually lands, though that finding describes one large-firm process, not a prevalence rate for owner-led service businesses.₁ Formal sign-off and substantive decision contribution are analytically separable events, and the separating question isn’t who approved it. It’s what changed because they touched it.

I’ve been in rooms where everyone knew the answer before the executive walked in. We weren’t waiting for judgment. We were waiting for permission. I’ve also been in rooms where one well-placed question from the person with the broader view changed the plan completely. That’s the difference I care about.

Can the Trust Travel?

The commercial customer who texted Dave’s cell in Part A hasn’t gone away. Nina still has to ask what kind of relationship that actually is.

A sole-link relationship runs entirely through Dave. A specialized-team relationship runs through Dave, the estimator, and the service lead, each contributing something the customer has separately come to trust. Those aren’t the same exposure. Research tracking roughly 1,800 professional moves and nearly 18,000 client decisions found that strong individual ties increased how often a client followed an employee to a new firm, while firm history and specialized team relationships reduced that exposure.₂ That’s professional-services evidence, not a general owner-handoff retention rate, but the shape of the question transfers cleanly. Owner access may be the value here. Sole-link dependence is a separate question, and the honest way to ask it is whether that trust has already shown evidence of traveling to someone besides Dave.

The Process Can Be Written Down. The Reasoning May Not Be.

Dave’s team has a documented rate sheet. What isn’t written anywhere is why the after-hours emergency rate sometimes bends, which customers have earned a longer grace window, and which edge case changes the standard answer. The process tells the team what to do. It doesn’t tell them why the rule bends.

Researchers call the pattern of who knows what, and whether the team trusts that expertise enough to rely on it, transactive memory. One validated measure of it has been tested across 124 laboratory teams, 64 MBA consulting teams, and 27 technology-company teams.₃ None of those populations is an owner-led service business, and the scorecard doesn’t administer or adapt that instrument. It borrows the observational question. Specialization by itself isn’t a problem. Reasoning nobody besides Dave can locate is worth inspecting.

Share

One Escalation Is a Clue. Recurrence Is the Question.

Friday’s exception traveled from the customer, to an employee, to a team lead, to Dave, and waited there before it resolved. That trace tells Nina what happened once. It doesn’t tell her whether that route is how this business works or how this one case happened to work.

Process-log research tracking roughly 70,000 cases across five organizations found that exception paths took measurably longer to resolve than standard paths, tracked by frequency, handoffs, destination, and latency.₄ Longer doesn’t automatically mean worse, and no validated threshold separates a healthy escalation rate from a congested one. What separates expert leverage from possible expert congestion is whether the case was genuinely rare and consequential with scarce expertise added, or whether it’s the same class of request, repeatedly treated as if it were rare, with the team already knowing the likely answer before Dave ever says yes.

One escalation is a clue. Nina still needs more than a single Friday before she calls it a pattern.

My instinct kicks in when the room stops debating the problem and starts predicting the approval. If everyone already knows what the answer will be but the work still has to climb the same ladder, that’s usually the moment I stop calling it an exception.

Read more

What the Audit Actually Changed

When Pinkie (aka ) first offered to audit Empathy Engine, I expected a website project. I thought we would clean up the homepage, sharpen the About page, and make the publication easier for a new reader to understand. We did that. I didn’t expect the audit to end up asking harder questions than the ones about layout.

BEFORE / AFTER

The first changes are easy to spot. The navigation bar is cleaner and more intentional. The generic publication name has been replaced with a real Empathy Engine wordmark. And the footer now gives readers a clearer path to find me elsewhere, through LinkedIn, Instagram, and YouTube.

Those may sound like small design decisions. They weren’t. Each one forced a question about what deserved to stay, what needed to change, and what kind of publication I wanted Empathy Engine to become. The old version reflected years of accumulated ideas, experiments, professional identities, and assumptions about what I was supposed to be building. The new version reflects more deliberate choices.

What should a new reader see first? What should be easy to find, and what should disappear? Those are website questions. But answering them honestly meant admitting the old site wasn’t cluttered by accident. It was cluttered because I hadn’t decided what I actually wanted it to say.

At some point, the website stopped being the only thing under review. So was I.

The new Empathy Engine is cleaner, but that’s almost beside the point. It’s more honest. I’ve spent much of my career helping organizations adapt through change, from the dot-com era to today’s AI disruption. This project reminded me that adaptation isn’t only something organizations have to do. People do, too. Sometimes that means putting away old assumptions, old ambitions, and old versions of ourselves that no longer fit. Then we get to ask a better question: what do I actually want to build now?

For me, Empathy Engine is becoming part of that answer, a place to develop useful ideas, create tools that give people a real competitive advantage, explore what human judgment means in an AI-shaped world, and build the body of work I want to be known for.

Pinkie audited my Substack. I didn’t expect my to audit me back.

BEFORE / AFTER: The About Page

This change was less about appearance and more about clarity. The old About page had a lot to say, but it asked a new reader to work too hard to understand what Empathy Engine was really for. The new version makes the value proposition much more direct: what problems we tackle, what practical tools subscribers receive, what paid members get, and who is behind the publication.

It also gave me room to explain something the old version barely touched: who I am beyond a list of credentials, what Agile actually means in plain English, and how my experience connects to the work I publish now. The result feels less like a résumé pasted onto a newsletter and more like an introduction with a point of view that matters for readers, and that distinction matters. A good About page shouldn’t merely prove that the author has experience. It should help the reader understand why that experience is useful to them.

Share

And the before-and-after image only shows the surface. Pinkie pushed me to rethink far more than layout and copy. She inspired me to dig into the real value proposition of Empathy Engine and ask what I’m actually giving subscribers that is distinctive, useful, and worth returning for. That work led to a much more substantial About page and a completely reimagined START HERE page, which previously existed only as a vague idea in my head.

So if the screenshots make the transformation look mostly visual, they are only showing the tip of the iceberg. The real change lives in the substance.

Visit the About and START HERE pages if you want to see the full transformation.

BEFORE / AFTER: The Paid Tier

The paid offer changed just as much. Before, the benefits were there, but they read more like a schedule of what subscribers received. The new version makes the value much more explicit: paid members now see the three core parts of the offer at a glance, The Leader’s Dispatch, 48-hour early access, and the Digital Arsenal of premium tools designed for immediate use.

The biggest shift is the Digital Arsenal. I don’t want paid subscribers to feel like they’re simply buying more articles. I want them to understand that they’re receiving thoroughly researched, enterprise-grade frameworks, canvases, scorecards, diagnostics, and decision tools that can give them a real competitive advantage in their jobs and businesses. That’s a much higher standard than “bonus content,” and it’s the standard I want Empathy Engine to be known for.

START HERE: Built From Scratch

There is no BEFORE image for this one. The START HERE page didn't exist. Pinkie's audit pushed me to create it from scratch, and that turned into one of the most valuable changes in the entire reinvention. Rather than dumping a new reader into an archive and hoping they figure out where to begin, the page now gives them a deliberate path into Empathy Engine.

Subscribe now

The greatest hits

Five cornerstone articles. I selected five of the strongest and most popular pieces in my Substack, each chosen because it gives readers a practical advantage in a different part of the work: AI adoption, judgment bottlenecks, leadership, role design, and governance.

A field briefing instead of a reading list. Each article is introduced by what the reader will actually learn, diagnose, or be able to do differently after reading it, not just a summary of what it's about.

A clearer promise. The page explains what Empathy Engine is trying to help readers become better at: using AI intelligently, leading people, making stronger decisions, and operating under pressure.

A real starting point. New readers no longer have to guess which article best represents the publication or where they should begin.

Facing the Changes

“So I turned myself to face me
But I’ve never caught a glimpse”

David Bowie, Changes

That lyric has been rattling around in my head throughout this project. I started by looking at a Substack homepage and asking what needed fixing. Somewhere between the navigation bar, the About page, the START HERE collection, and the uncomfortable business of deciding what Empathy Engine actually promises people, the question became much more personal. I had to look at some old ideas about who I was, what counted as success, and what I thought I was still supposed to be chasing.

There is something mildly ridiculous about needing a website audit to provoke that kind of reckoning, which feels appropriate. Reinvention rarely arrives with a brass band and a tasteful PowerPoint deck explaining the transformation roadmap. Sometimes somebody named Pinkie points out that your Substack could use some work, and a few weeks later you are reconsidering the architecture of your professional life. I can laugh at that now, because the exercise left me clearer about what I want Empathy Engine to become, and more importantly, about the person I want to become while building it. That’s the thing nobody tells you about a good audit: it never stays confined to the thing you asked someone to look at.

The publication on the other side is more focused, more useful, and more recognizably mine. I am, too. I still can’t claim to have figured out the destination, and perhaps that would defeat the point anyway. For now, I’m comfortable facing the direction I’ve chosen, making the next decision deliberately, and accepting that time may change me.

Thank you, Pinkie of , who set out to fix a homepage and ended up helping me confront my shadow self instead. Empathy Engine, and the people who read it, are better for it.

Regards,

Mark 👋