When an AI feature looks ready, someone still has to say what good enough means, and own it
A researcher on an AI product team kept asking a question nobody wanted to answer. The team had a groundedness score for its large language model feature, roughly a measure of whether the model’s answers stay anchored to their sources. So the researcher asked the obvious next thing: “If we get a groundedness score of 0.5, what do we do?” The developers didn’t want to answer. An AI engineer kept pushing for a plan, and the best that came back was “we’ll monitor for trends.” Then came the question underneath all the others. “When I ask if we’d pull the product for bad scores, no one answers.”
That account comes from a study published at CHI 2026, in which researchers interviewed 19 people who evaluate LLM products in the wild. The participant is identified only as P15, someone working in research. I’m not going to dress it up as a tense meeting with a whiteboard and a long silence, because the paper doesn’t describe one. It describes something quieter, which P15 summed up this way: “We’re collecting data without a plan for what we’re going to do with it.” The researchers found that 17 of their 19 participants were affected by the same gap between gathering evaluation results and turning them into action.
👋 Welcome to this week’s 2nd edition of Empathy Engine. Every Wednesday, I publish a new article for paid subscribers first, then unlock the full piece for everyone late Thursday morning. Each week, I turn product leadership friction into practical tools, sharper language, and more defensible decisions.
Research Binder: the receipts (citations + source notes) are compiled in a PDF at the bottom of this post.
Notice what isn’t broken in that story. The feature works, at least well enough to show people. The team is measuring something. The engineer is asking exactly the right question. What’s missing is a person who says, out loud and on the record, what good enough means for this feature, what happens when the bar isn’t met, and who gets to decide. Nobody has measured how often this happens across teams or what it costs them, and I’m not going to pretend otherwise. What the research can show is why a working demo can’t answer the question P15 kept asking.
As an Agile coach, I’ve encountered situations where delivery dashboards showed encouraging numbers but offered little guidance about what leadership should do next. Velocity was stable, work was being completed, and the reports looked reassuring. Yet those measurements didn’t necessarily establish whether the team was delivering the outcomes stakeholders needed. The harder question was always what decision the evidence justified.
This is the seventh and final episode of The Judgment Layer. For six weeks we’ve been circling the same human moment from different angles: the person who reads the work, hands it off, clicks approve, reads the scoreboard, trains the next reviewer, and owns the voice on the slide. This last one is about the question that sits above all of them. When an AI feature appears to work, who decides it’s good enough, how is that judgment written down, and what can the next reviewer actually rely on?
A working example isn’t a release decision
A demo is one convincing result. It’s a real result, and it matters, because it tells you the feature can do the thing. But a release decision has to answer questions a single good run never touches. What counts as good for this use? What must not happen? Would another reviewer judge it the same way? And who is actually allowed to say yes?
Those four boxes on the right side of the board are the rest of this episode. Acceptable asks what counts as good. Failures asks what must not happen. Agreement asks whether other reviewers would see it the same way, and, as we’ll get to, whether it matters if they do. Authority asks who can decide. The demo on the left answers none of them, which is why the line across the middle is the one I’d want every product team to tape to a monitor: a working example isn’t a release decision.
That’s not a knock on demos. A demo starts the conversation. It just doesn’t settle whether the output is good enough, and the trouble starts when a team treats the first as if it were the second. The question in the bottom corner of the right panel is the one worth carrying into every review: what happens beyond that result?
The tests we choose change what we find
Start with the most basic problem. Whatever you put in front of a feature decides what you’ll learn about it. A demo is a test session too. Someone picked the inputs, someone sat at the keyboard, and the conditions were whatever happened to be on hand that afternoon. Change the conditions and you change what shows up.
One small study makes that visible. In 2020, Marco Tulio Ribeiro and colleagues gave 18 people with NLP experience the same model and the same task: a system that judges whether two questions mean the same thing. They had two hours to find its problems. Six people tested it unaided. Six were given a list of capabilities to check, things like whether the model handles negation or changes in names. Six got that list plus templates for writing tests quickly. Afterward, each participant rated the problems they had found on a five-point scale, where 3 meant a bug worth investigating and fixing.
The unaided group averaged 2.2 tests flagged as bugs rated 3 or higher. The capability-list group averaged 5.5. The list plus templates averaged 6.2. Look at where the jump happens. Most of it came from the list alone, from simply being told what to look at. The templates added a little more on top.
Now the honest part, which is printed right on the board. Participants rated their own bugs, so there was no ground truth about how serious each one really was, and the paper doesn’t report a significance test for these averages. It’s six people per condition, one model, one task, and a study that predates today’s LLM products. So I’m not telling you structured testing prevents failures. Finding more problems doesn’t prove fewer future failures. What the study shows is narrower and, for our purposes, more useful: the shape of a test session shapes what surfaces in it.
Which brings us to the takeaway at the bottom of the board. Before you trust what a test told you, ask what the test conditions didn’t examine. A demo examined one path through the feature, chosen by someone who wanted it to work. That’s worth knowing. It just isn’t the whole picture.
One prompt, more than one answer
Here’s the part that makes AI features different from the software most of us grew up approving. With ordinary code, if a test passes on Tuesday, it passes on Wednesday, unless someone changed something. With a generative model, you can send the exact same request twice and get two different answers back. That means a demo isn’t just one path chosen by a friendly operator. It’s one draw.
Shuyin Ouyang and colleagues measured this directly in a study published in ACM Transactions on Software Engineering and Methodology in 2025. They took 829 Python programming problems from three well-known benchmarks and asked ChatGPT to write code for each one, five separate times, with the same prompt. Then they ran the tests and compared the results.
At the default setting, the share of tasks where none of the five requests produced matching test outputs was 75.76% for the CodeContests problems, 51.00% for APPS, and 47.56% for HumanEval. The researchers also tried setting the temperature to 0, the dial that’s supposed to make a model’s answers more predictable. It reduced the variation. It didn’t eliminate it.
Two cautions keep this honest. First, different outputs aren’t automatically wrong outputs. Two pieces of code can both be acceptable while behaving differently on a test. Second, this is code generation with one model family, so I’m not claiming every AI feature varies this much. The narrower point holds anyway: if the same request can produce different results, one successful run is a sample of one.
The variation doesn’t stop at repeated requests, either. In a benchmark of AI agents completing customer service tasks with simulated users, the τ-bench study (ICLR 2025) found agents that succeeded once often failed when the same task was repeated, with consistency across eight attempts falling below 25% in the retail domain. And the model you approved may not stay the model you’re running. Lingjiao Chen, Matei Zaharia and James Zou compared the March and June 2023 versions of GPT-4 and found its accuracy at identifying prime numbers fell from 84% to 51% on the same questions. That’s one study of one hosted service, and it shows that a change like that can happen, not how often it does. But it’s enough to make the takeaway on this board stick: one successful run doesn’t establish repeatability, and a result from last quarter describes last quarter’s model.
Good can change while we’re checking it
The obvious fix is to write the standard down first, then test against it. Decide what good looks like, then go look. It’s sensible advice, and it runs into a problem that anyone who has reviewed real AI output will recognize. You often don’t know what you care about until you see what the system actually produces.
Shreya Shankar and colleagues watched this happen. In a study presented at UIST 2024, nine industry practitioners used a tool called EvalGen to define criteria for judging an LLM’s outputs and then grade those outputs. The researchers put the loop in one sentence: “users need criteria to grade outputs, but grading outputs helps users define criteria.”
As participants worked through real outputs, their criteria evolved. They added new criteria when they ran into kinds of bad output they hadn’t anticipated, and they reinterpreted the ones they already had. Define, review, revise, and around again. The authors went as far as saying it is “impossible to completely determine evaluation criteria prior to human judging of LLM outputs.”
Keep the scale in view. That’s nine people, one tool, one task, and short sessions, so it can’t tell you how common this is. But it matches what showed up in the CHI 2026 interviews, where 12 of the 19 participants described beginning their evaluation with informal “vibe checks.” My read, and it is a read rather than a finding, is that those vibe checks aren’t laziness. They’re often where the real criteria first show up.
So the goal isn’t a standard that never moves. A standard written before anyone has looked at real outputs is a draft, and that’s fine. The problem is a standard that changes silently, so the next reviewer is grading against rules nobody remembers revising. That’s why the takeaway on this board is a habit rather than a rule: date each standard, and record why it changes.
Agreement is not correctness
If one person’s judgment shifts, the natural move is to bring in more people. Get the PM, the engineer, and the designer in a room, look at the outputs together, and see where everyone lands. When they all agree, it feels like the question has been answered. Agreement feels like proof. It isn’t, and there’s a study that shows the gap unusually cleanly.
In 2021, Elizabeth Clark and colleagues asked crowd workers in the United States to read short passages, stories, news articles, and recipes, and decide whether each was written by a person or by GPT-3. Some evaluators got training first, including annotated examples. Because this task has a known right answer, the researchers could measure two separate things: whether people agreed with each other, and whether they were actually correct.
