When an AI feature looks ready, someone still has to say what good enough means, and own it
A researcher on an AI product team kept asking a question nobody wanted to answer. The team had a groundedness score for its large language model feature, roughly a measure of whether the model’s answers stay anchored to their sources. So the researcher asked the obvious next thing: “If we get a groundedness score of 0.5, what do we do?” The developers didn’t want to answer. An AI engineer kept pushing for a plan, and the best that came back was “we’ll monitor for trends.” Then came the question underneath all the others. “When I ask if we’d pull the product for bad scores, no one answers.”
That account comes from a study published at CHI 2026, in which researchers interviewed 19 people who evaluate LLM products in the wild. The participant is identified only as P15, someone working in research. I’m not going to dress it up as a tense meeting with a whiteboard and a long silence, because the paper doesn’t describe one. It describes something quieter, which P15 summed up this way: “We’re collecting data without a plan for what we’re going to do with it.” The researchers found that 17 of their 19 participants were affected by the same gap between gathering evaluation results and turning them into action.
👋 Welcome to this week’s 2nd edition of Empathy Engine. Every Wednesday, I publish a new article for paid subscribers first, then unlock the full piece for everyone late Thursday morning. Each week, I turn product leadership friction into practical tools, sharper language, and more defensible decisions.
Research Binder: the receipts (citations + source notes) are compiled in a PDF at the bottom of this post.
Notice what isn’t broken in that story. The feature works, at least well enough to show people. The team is measuring something. The engineer is asking exactly the right question. What’s missing is a person who says, out loud and on the record, what good enough means for this feature, what happens when the bar isn’t met, and who gets to decide. Nobody has measured how often this happens across teams or what it costs them, and I’m not going to pretend otherwise. What the research can show is why a working demo can’t answer the question P15 kept asking.
As an Agile coach, I’ve encountered situations where delivery dashboards showed encouraging numbers but offered little guidance about what leadership should do next. Velocity was stable, work was being completed, and the reports looked reassuring. Yet those measurements didn’t necessarily establish whether the team was delivering the outcomes stakeholders needed. The harder question was always what decision the evidence justified.
This is the seventh and final episode of The Judgment Layer. For six weeks we’ve been circling the same human moment from different angles: the person who reads the work, hands it off, clicks approve, reads the scoreboard, trains the next reviewer, and owns the voice on the slide. This last one is about the question that sits above all of them. When an AI feature appears to work, who decides it’s good enough, how is that judgment written down, and what can the next reviewer actually rely on?
A working example isn’t a release decision
A demo is one convincing result. It’s a real result, and it matters, because it tells you the feature can do the thing. But a release decision has to answer questions a single good run never touches. What counts as good for this use? What must not happen? Would another reviewer judge it the same way? And who is actually allowed to say yes?
Those four boxes on the right side of the board are the rest of this episode. Acceptable asks what counts as good. Failures asks what must not happen. Agreement asks whether other reviewers would see it the same way, and, as we’ll get to, whether it matters if they do. Authority asks who can decide. The demo on the left answers none of them, which is why the line across the middle is the one I’d want every product team to tape to a monitor: a working example isn’t a release decision.
That’s not a knock on demos. A demo starts the conversation. It just doesn’t settle whether the output is good enough, and the trouble starts when a team treats the first as if it were the second. The question in the bottom corner of the right panel is the one worth carrying into every review: what happens beyond that result?
The tests we choose change what we find
Start with the most basic problem. Whatever you put in front of a feature decides what you’ll learn about it. A demo is a test session too. Someone picked the inputs, someone sat at the keyboard, and the conditions were whatever happened to be on hand that afternoon. Change the conditions and you change what shows up.
One small study makes that visible. In 2020, Marco Tulio Ribeiro and colleagues gave 18 people with NLP experience the same model and the same task: a system that judges whether two questions mean the same thing. They had two hours to find its problems. Six people tested it unaided. Six were given a list of capabilities to check, things like whether the model handles negation or changes in names. Six got that list plus templates for writing tests quickly. Afterward, each participant rated the problems they had found on a five-point scale, where 3 meant a bug worth investigating and fixing.
The unaided group averaged 2.2 tests flagged as bugs rated 3 or higher. The capability-list group averaged 5.5. The list plus templates averaged 6.2. Look at where the jump happens. Most of it came from the list alone, from simply being told what to look at. The templates added a little more on top.
Now the honest part, which is printed right on the board. Participants rated their own bugs, so there was no ground truth about how serious each one really was, and the paper doesn’t report a significance test for these averages. It’s six people per condition, one model, one task, and a study that predates today’s LLM products. So I’m not telling you structured testing prevents failures. Finding more problems doesn’t prove fewer future failures. What the study shows is narrower and, for our purposes, more useful: the shape of a test session shapes what surfaces in it.
Which brings us to the takeaway at the bottom of the board. Before you trust what a test told you, ask what the test conditions didn’t examine. A demo examined one path through the feature, chosen by someone who wanted it to work. That’s worth knowing. It just isn’t the whole picture.
One prompt, more than one answer
Here’s the part that makes AI features different from the software most of us grew up approving. With ordinary code, if a test passes on Tuesday, it passes on Wednesday, unless someone changed something. With a generative model, you can send the exact same request twice and get two different answers back. That means a demo isn’t just one path chosen by a friendly operator. It’s one draw.
Shuyin Ouyang and colleagues measured this directly in a study published in ACM Transactions on Software Engineering and Methodology in 2025. They took 829 Python programming problems from three well-known benchmarks and asked ChatGPT to write code for each one, five separate times, with the same prompt. Then they ran the tests and compared the results.
At the default setting, the share of tasks where none of the five requests produced matching test outputs was 75.76% for the CodeContests problems, 51.00% for APPS, and 47.56% for HumanEval. The researchers also tried setting the temperature to 0, the dial that’s supposed to make a model’s answers more predictable. It reduced the variation. It didn’t eliminate it.
Two cautions keep this honest. First, different outputs aren’t automatically wrong outputs. Two pieces of code can both be acceptable while behaving differently on a test. Second, this is code generation with one model family, so I’m not claiming every AI feature varies this much. The narrower point holds anyway: if the same request can produce different results, one successful run is a sample of one.
The variation doesn’t stop at repeated requests, either. In a benchmark of AI agents completing customer service tasks with simulated users, the τ-bench study (ICLR 2025) found agents that succeeded once often failed when the same task was repeated, with consistency across eight attempts falling below 25% in the retail domain. And the model you approved may not stay the model you’re running. Lingjiao Chen, Matei Zaharia and James Zou compared the March and June 2023 versions of GPT-4 and found its accuracy at identifying prime numbers fell from 84% to 51% on the same questions. That’s one study of one hosted service, and it shows that a change like that can happen, not how often it does. But it’s enough to make the takeaway on this board stick: one successful run doesn’t establish repeatability, and a result from last quarter describes last quarter’s model.
Good can change while we’re checking it
The obvious fix is to write the standard down first, then test against it. Decide what good looks like, then go look. It’s sensible advice, and it runs into a problem that anyone who has reviewed real AI output will recognize. You often don’t know what you care about until you see what the system actually produces.
Shreya Shankar and colleagues watched this happen. In a study presented at UIST 2024, nine industry practitioners used a tool called EvalGen to define criteria for judging an LLM’s outputs and then grade those outputs. The researchers put the loop in one sentence: “users need criteria to grade outputs, but grading outputs helps users define criteria.”
As participants worked through real outputs, their criteria evolved. They added new criteria when they ran into kinds of bad output they hadn’t anticipated, and they reinterpreted the ones they already had. Define, review, revise, and around again. The authors went as far as saying it is “impossible to completely determine evaluation criteria prior to human judging of LLM outputs.”
Keep the scale in view. That’s nine people, one tool, one task, and short sessions, so it can’t tell you how common this is. But it matches what showed up in the CHI 2026 interviews, where 12 of the 19 participants described beginning their evaluation with informal “vibe checks.” My read, and it is a read rather than a finding, is that those vibe checks aren’t laziness. They’re often where the real criteria first show up.
So the goal isn’t a standard that never moves. A standard written before anyone has looked at real outputs is a draft, and that’s fine. The problem is a standard that changes silently, so the next reviewer is grading against rules nobody remembers revising. That’s why the takeaway on this board is a habit rather than a rule: date each standard, and record why it changes.
Agreement is not correctness
If one person’s judgment shifts, the natural move is to bring in more people. Get the PM, the engineer, and the designer in a room, look at the outputs together, and see where everyone lands. When they all agree, it feels like the question has been answered. Agreement feels like proof. It isn’t, and there’s a study that shows the gap unusually cleanly.
In 2021, Elizabeth Clark and colleagues asked crowd workers in the United States to read short passages, stories, news articles, and recipes, and decide whether each was written by a person or by GPT-3. Some evaluators got training first, including annotated examples. Because this task has a known right answer, the researchers could measure two separate things: whether people agreed with each other, and whether they were actually correct.
Without example training, evaluators landed around 50%, which is chance for a two-choice task. With example training, accuracy rose to 55%. That gain was significant overall but not within the individual domains. Meanwhile, agreement between evaluators stayed low across every domain and training setting, with Krippendorff’s alpha at or below 0.11, where 0 means chance-level agreement and 1 means perfect agreement. And the sentence I keep coming back to: in the researchers’ words, “higher agreement did not correspond to higher accuracy.”
Be careful with what this does and doesn’t cover. Clark’s task was spotting AI-written text, not judging whether an output was good enough to ship, and the evaluators were crowd workers looking at GPT-3-era writing. But the distinction it isolates travels: whether reviewers judge alike and whether the judgment holds up are two different questions. A second study sharpens the point. When Sashank Santhanam and colleagues showed crowd workers a reference example before they rated chatbot responses (CHI 2020), ratings became more consistent in some conditions and also drifted toward the example’s score. The authors suggested the added consistency “may be a result of anchoring bias.” Examples can make reviewers more alike without making them more right.
That changes what disagreement means. Sometimes it’s a sign that nobody wrote the criteria down. Sometimes it’s a sign that people want legitimately different things. Back in the CHI 2026 interviews, P15 described the moment of realizing “we were all talking about completely different things actually,” with designers and product managers focused on user experience while the AI engineers meant something else entirely. Another participant, P7, recalled technical people calling an output a bug while the person using it was delighted. Neither side was wrong. They were answering different questions. That’s why the takeaway on this board doesn’t say “get to consensus.” It says preserve disagreement, and examine the reasons.
Someone still has to own the decision
If agreement doesn’t settle it, what does? Not a better rubric, and not a bigger spreadsheet. At some point a person with the authority to accept the risk has to say yes, no, or not yet, and put their name next to it. That’s the step P15 couldn’t get anyone to take, and it turns out to be a familiar gap. In a 2026 study of GenAI adoption at two software companies, Yu and colleagues reported that “ownership is fragmented across phases.” A separate set of interviews with responsible AI practitioners, published at CSCW 2021, captured the same frustration in four words: “Whose job is this?” That study was about ethics and risk concerns rather than quality sign-off, but the sound of the question is the same.
Guidance is clearer than practice here. The NIST AI Risk Management Framework, a voluntary U.S. framework, recommends that roles and responsibilities for managing AI risk be documented and clear (GOVERN 2.1), and that someone determine whether a system achieves its intended purpose and whether deployment should proceed (MANAGE 1.1). One practitioner put the human version of that plainly. Writing about AI features shipped into inspection apps, Lee Nguyen of Dev Station explained why the team pulled automated pass and fail decisions: “Nobody will sign a certificate that a model decided.”
The board splits the work into four jobs that often get blurred together. Someone specifies what counts as acceptable. Someone evaluates what was checked and, just as important, what wasn’t. Someone records what remains disputed. And someone authorizes, meaning they accept the remaining risk. Around those four sit three roles that are easy to conflate: the evidence owner who maintains the record, the decision owner who has authority, and the revisit trigger that says what would reopen the call.
I want to be straight about the status of that structure. It’s an Empathy Engine proposal, and it hasn’t been tested. NIST recommends separating these responsibilities, and interview studies keep finding that the separation is fuzzy in real teams, but no study I found shows that naming an owner makes AI quality decisions better. What naming an owner does is make the decision traceable to someone. That’s also why the orange line on this board matters so much: a completed record doesn’t grant approval authority. A signature records who decided. It doesn’t certify that the evidence was enough, and it doesn’t hand authority to someone who didn’t already have it.
The takeaway is two small acts. Name the authorized owner. Record what could change the decision.
About that number ten
A confession. The working title for this episode was “Ten Examples Before the Argument,” and the idea had real appeal. Before your team argues about whether a feature is ready, write down ten concrete examples of good and bad output, and argue about those instead. Starting small is sensible advice, and it’s common advice too. One evaluation vendor’s guide for product managers suggests five to ten examples to begin with, and on the same page says ten to two hundred is usually enough to identify and fix patterns. Starting is the easy part. The trouble is what ten clean results can and can’t tell you once you’ve got them.
So I ran the numbers. Imagine a feature with a problem that shows up on 1% of requests. You test it ten times, the tests are independent of each other, and every one passes. How likely was that?
With a 1% failure rate, ten independent trials find at least one failure only 9.6% of the time. The other 90.4% of the time, you see ten clean passes and nothing to worry about. Raise the failure rate to 5% and ten trials catch it 40.1% of the time. At 10%, they catch it 65.1% of the time. Put plainly, a problem that shows up once in every twenty requests would be missed by ten random checks more often than not. To have a 95% chance of catching a one-in-twenty problem, you’d need roughly 60 randomly chosen cases. That’s a different exercise from a starter set.
There’s an even bigger catch in the small print, and it’s the one I’d most like you to remember. Those numbers assume random, independent cases drawn from real use. Most teams don’t pick their examples at random. They pick the ones they know, the ones that matter, or the ones that were handy. That’s a perfectly good way to start a conversation, but it changes what the results can mean. Ten random passes can’t rule out an infrequent failure. Ten handpicked passes can’t tell you how often failures happen at all. They tell you about those ten cases. Passing ten examples tells us about ten examples.
Illustration only. Assumes cases drawn at random from real use, judged the same way, and independent of each other.
So here’s where I’ve landed on the number. Ten is a practical starting number, and nothing more. No study I found shows that ten, or any fixed number, is enough, and the right size depends on how rare a failure you need to catch, how precise you need to be, and what happens if you’re wrong. That’s why this episode isn’t called “Ten Examples” anymore. Ten passes aren’t proof of reliability.
Make the judgment inspectable
Put the pieces together and you can see what P15 was missing. Not a better score, and not more data. P15 already had data. What was missing was a record that said what the system was for, what had been checked, where people disagreed, what nobody had looked at, who could decide, and what would bring the question back. That’s what the next reviewer needs to know, and it’s what a demo never hands them.
Each section of the record answers one of the questions this episode has been working through. For this use names the task and context, because good enough for an internal draft isn’t good enough for a customer email. Evidence checked records what passed, what failed, and who chose those cases and how, so nobody mistakes a handpicked set for a random one. Disagreement keeps the split views on the page instead of averaging them away. Not covered is the box I’d fight hardest for, because it’s where honesty lives: what remains unknown or untested. Decision owner names who authorizes use and accepts the remaining risk. Revisit trigger says what would reopen the call, whether that’s a new model version, a changed prompt, a revised standard, or an incident. And the versions and date line at the bottom keeps the whole thing anchored to the system that was actually checked.
That “not covered” box deserves a warning of its own. When Weixin Liang and colleagues analyzed 32,111 AI model cards for a 2024 paper in Nature Machine Intelligence, the Limitations section was among the least completed, filled out in only 17.4% of cards, with Evaluation close behind at 15.4%. That was public model documentation, not internal review records, but the pattern is worth borrowing as a caution. The fields people skip are often the fields that matter most. If your “not covered” box is empty, treat that as a question, not an answer.
One more line belongs in any record for a feature that takes actions rather than just producing text. An output that reviewers judged acceptable is not evidence that an agent actually did what it reported. An anonymous developer on the OpenAI Developer Community described their team’s biggest reliability problem as agents reporting work as done when nothing had happened, including a payment marked as sent when the underlying call had failed. For agents, the record needs evidence of what the system did, such as logs or state changes, kept separate from what it said. And even that evidence, as the same poster acknowledged, “doesn’t prove the answer was right.” Keep sensitive or regulated data out of the record unless you’re authorized to store it there.
Now the limits, which are printed on the board for a reason. This is a proposed, untested aid. Documentation isn’t validation. A record doesn’t prove coverage, settle disagreement, or authorize use, and high-stakes or regulated uses still require whatever testing, review, and approval apply to them. A documented decision can still be wrong. What the record does is make the judgment something the next person can read, question, and change. The takeaway is the question I’d want at the bottom of every one of these: what evidence would make us reconsider?
Your one bounded next decision
You don’t need a new platform or a governance committee to start. Pick one AI-assisted feature your team is about to approve, or already has. This week, write down six things for it: what it’s for, the real outputs you looked at (including the bad and borderline ones) and who picked them, where reviewers disagreed, what you didn’t check, who has the authority to accept the result, and what would make you look again. Date it. Put the model and prompt version next to the date. Then hand it to someone who wasn’t in the room and ask them what they’d rely on.
🎁 Complete Bundle Description
A great AI demo doesn’t mean the work is ready. Who decides what happens next?
The Quality Judgment Record gives teams a simple way to check AI work and write down what they find. It helps us see what worked, what failed, what wasn’t tested, and where people disagree. It also helps us name who can make the final decision and what might cause us to review it again.
This bundle includes a step-by-step guide, an Excel workbook, a two-page decision record, and a meeting field sheet. Each tool has a different job, but they work together to make our decisions easier to examine and explain.
The goal is simple: help the next person understand what we checked, what we don’t know, and why a decision was made.
The kit doesn’t test or approve AI systems for us. A completed record doesn’t prove that an AI system is safe, reliable, or ready to use.
Learn how to review AI work before making a decision.
This 12-page guide walks us through the questions behind a good review. It explains how to choose examples, spot missing tests, handle disagreements, and decide when another review is needed. It also includes research findings, a sample review meeting, and a filled-out example to show how the process works.
Keep the evidence organized so nothing important gets lost
This Excel workbook gives us one place to track AI test examples, expected results, known failures, and reviewer disagreements. We can also record changes to our standards, save model and prompt versions, and note when a decision needs another look. It includes four made-up examples to help us learn and a chart showing why a small set of passing tests can miss problems.
Put the evidence, open questions, and decision in one clear record
This two-page form captures what the AI feature is meant to do, what was checked, what failed, and what remains unknown. It also records where reviewers disagree, who has the right to decide, and what would cause another review. The result is a document the next person can read without having to start from scratch.
Bring the right questions into the room before anyone says yes
This printable sheet helps guide an AI review meeting. The front has space to write down what was tested, where people disagree, what wasn’t covered, and who can decide. The back has eight questions that help the team challenge its first impressions and think about what evidence might change its mind.
The last decision in the series
This series started with a simple question: who was the last human who actually read the thing? Every episode since has followed that person a little further. Into the handoff, where finished work still has to be owned by someone. To the approval button, where a click can happen without a judgment behind it. Past the scoreboard, where usage stands in for outcomes. Into the bench of reviewers we have to train, and onto the slide whose voice someone has to claim.
This last one lands where all of them were pointing. AI makes producing work cheap. It doesn’t make deciding cheap, and it doesn’t make the decision anyone’s job by default. If there’s one thing I’d want you to carry out of these seven weeks, it’s three sentences you can say in your next review. Agreement shows we share a view, not that the view is right. Passing ten examples tells us about ten examples. Who decides is a separate question from what the tests show.
The demo is not the decision. The decision is a person, a date, and a reason the next reviewer can read, question, and change.
As we close The Judgment Layer, I hope one idea stays with us: AI can help produce the work, but people still have to understand it, question it, and own the decisions that follow. That responsibility is also at the heart of my new book, Collaborate Better, which is finally available for purchase! The book explores how we can build stronger teams, make better decisions, and turn shared understanding into meaningful action. If this seven-part series has challenged how we think about leadership, teamwork, and accountability, I invite you to continue the journey at CollaborateBetter.us.
Regards,
Mark 👋
Previous:
More Content to Discover:
Half Your Team Is Using AI. You Don't Know Which Half.
Intro If you read the piece we published last week on Leadership in Change, you saw the structural diagnosis: mandate pressure, decision collapse, intake collapse. Three failures that sit underneath almost every AI pilot that goes sideways. That piece mapped the architecture. What it did not cover is what happens on the human side of the same problem, wh…






















The most dangerous AI demo might be the one everyone agrees was a success. Ten passing tests can still miss important failures, and agreement doesn’t prove we’re right.
Before calling an AI system ready, I’d ask: What’s the one unanswered question that could change our decision? I’m curious which questions others would put on that list.