A Count Isn’t a Verdict
Plain-English premise: An AI usage dashboard can accurately show that people used the tool. By itself, it can’t show that the work got better.
Judgment Seat: The manager about to attach a usage number to a person, a team, a review, or a claim that the rollout worked.
Episode 1:
Episode 2:
👋 Last week’s closing promised an episode called The Scoreboard Is Lying. The research changed my mind again. The dashboards in this research weren’t lying. The ones whose definitions I could inspect were counting exactly what they were built to count. The trouble starts when we ask a count to testify about something it never observed. Episode One budgeted the reviewer’s attention. Episode Two asked what the sender owes before review begins. Episode Three asked what an approval can prove. This week moves up a level, to the number leaders reach for when they want to know whether any of it is working.
Episode 3:
Research Binder: the receipts (citations + source notes) are compiled in a PDF at the bottom of this post.
⚡ Pressure
AI usage is easy to count, and the count can quietly become the verdict on the work.
🎯 Payoff
This issue gives you the AI Proxy Audit: four questions to ask before a usage number becomes a verdict about people or progress.
Pressure Moment
In an April 2025 memo, Duolingo announced that AI use would be part of what it evaluated in performance reviews. In an interview published April 10, 2026, CEO Luis von Ahn said the company had backtracked. He said he didn’t know whether employees were using AI just to be seen using it. What he recounted was their question: “Do you just want us to use AI for AI’s sake?” His answer, in effect, was that the company had been pushing something that sometimes didn’t fit instead of holding people accountable for the outcome.
That’s one company, in its CEO’s account. No scored review field or effective date is in the public record. The employees weren’t asking whether AI mattered. They were asking what their use of it was supposed to prove.
I’ve had the experience of looking at completed work in Azure DevOps and then watching the same work return as follow-on tickets. The original story was still closed. The velocity was still real. The points had still been delivered. But the dashboard had stopped observing before the consequences did. That changed how I read activity metrics. The number can be perfectly accurate about what happened inside its measurement boundary and still be insufficient evidence that the work itself improved.
The Number Gets Promoted
An AI dashboard might accurately report how many people opened the tool this month, how many prompts were recorded, and how many suggestions were accepted. Those can be perfectly legitimate observations. Then someone carries the number into another room: the boardroom, the performance review, the transformation update, the compensation conversation. Somewhere along the way, an activity count acquires a promotion. It stops being evidence that the tool was used and starts standing in for evidence that the work improved.
That headline is deliberately provocative, so it’s worth being precise about it. An unsupported verdict can still turn out to be correct. What an accurate activity count can’t do is establish the stronger conclusion on its own. The gap between the two isn’t a flaw in the data. It’s an inference, and an inference needs its own evidence.
So the question this episode keeps returning to is easy to ask and harder to answer: What is this number allowed to prove? It’s the question underneath the whole series, applied one level up. Episodes One through Three asked who could actually defend a deck, a handoff, or an approval. This week, what needs defending is the conclusion someone draws from a number.
Four Kinds of Evidence on One Screen
Not every number on an AI dashboard belongs to the same evidentiary category. Some observe activity, some are estimates built on assumptions, some are perceptions users report, and some measure downstream work. Arranged in matching tiles, they look alike. The activity indicators, at least, are defined in public. Microsoft’s Copilot Dashboard documentation counts active users from recorded Copilot activity, and GitHub’s documentation calculates acceptance rate from counts of suggestions generated and accepted. Neither definition assesses whether the work that followed was any good. That isn’t a hidden flaw. It’s what those measures are for.
Estimates are where a number gets most persuasive. Microsoft’s dashboard also shows an estimated assisted value: estimated Copilot-assisted hours multiplied by a configurable hourly rate, with a documented default of $72 an hour. Change the rate and the dollar figure moves while nothing about the work does. That makes it a modeling input, not observed savings, and on its own it doesn’t establish realized cash savings, revenue, improved quality, or net ROI (Microsoft Copilot Dashboard documentation, as inspected September 2026).
Once the claim changes, the evidence question changes with it. “Did people use the tool?” is one question. “Did avoidable rework decline for comparable work?” is another, and “Did AI cause that decline?” is harder still. Moving between them isn’t a climb from bad data to good data. It’s a move between different claims, which is why the useful question about an AI metric isn’t whether it’s good. It’s good for what?
Sometimes Usage Is Exactly the Right Thing to Measure
There’s a lazy version of this argument that calls every usage metric a vanity metric. I don’t buy it. Organizations spend real money on AI tools, and leaders have a legitimate reason to know who has access, who started, who stopped, and where people are getting stuck. For those questions, usage data isn’t a weak proxy. It’s the right evidence.
Disney offers a documented example. Business Insider inspected a manager’s message to an engineer whose recorded AI use was low; the note asked about tool access, barriers, and what support would help. Separately, an unnamed Disney engineer said colleagues saw their name near the top of an internal usage dashboard and reached out to learn their methods. Other Disney employees described pressure around the same dashboard, so this is a documented check-in and a reported learning opportunity, not a verdict on the dashboard (Business Insider, April 30 and May 8, 2026).
JPMorganChase shows the same boundary drawn from the top. In the bank’s official February 23, 2026 Q&A, CEO Jamie Dimon discussed LLM usage and the time savings users perceive, while declining to count some of those savings as financial return: “That’s not in an NPV.” That’s a stated valuation boundary, not proof of bankwide practice. Business Insider’s April 2026 reporting on internal dashboards that track and rank engineers’ AI use belongs in the same picture, along with the bank’s statement that the data isn’t used in performance management and employee accounts of feeling pressure anyway.
None of this shows that monitoring improved anyone’s work, and a useful purpose doesn’t by itself settle privacy, fairness, or legal acceptability. What it shows is usage data doing the job usage data can do. A metric doesn’t become foolish when we change the question. It becomes insufficient when we ask it to answer a question beyond the support we have for that use.
When the Number Enters a Review
Usage information doesn’t become dangerous because someone places it near a review. The question changes. An adoption question asks whether people are trying the tool. A performance question asks what their usage says about whether they’re doing the job well, and that second question carries a much larger interpretive burden.
Duolingo is the clearest public example of that shift, and its limits matter as much as its story. The memo announced AI use as an evaluation factor; the CEO’s later account describes backing away from it at a point he didn’t date. The record doesn’t establish a numerical usage score, exact workforce coverage, or any measured effect on performance, pay, or promotion.
Amazon supplies a narrower example. By May 29, 2026, Amazon had confirmed that KiroRank, an informal, employee-built AI-usage leaderboard, had been deprecated, describing it as a beta tool used by some employees rather than a formal or approved one. Reporting credited to the Financial Times quoted senior vice president Dave Treadwell telling staff, “Please don’t use AI just for the sake of using AI.” The explanations for the shutdown differ: 404 Media reported an internal announcement saying the goal had been accomplished, alongside employee accounts pointing to waste and gaming. The reporting doesn’t settle which was the real reason, and I won’t pretend it does. Amazon did tell Fortune that it discouraged measuring developer productivity through token utilization, while treating token monitoring for cost and efficiency as a separate matter.
KiroRank wasn’t a performance review, and that’s the point. Metrics don’t sit on a dashboard waiting to be read; organizations decide what they’re allowed to influence. A number can prompt curiosity, inform coaching, enter an evaluation, or move money. Those are different decisions, and they don’t automatically inherit the same evidentiary permission. A new decision use needs its own justification.
A Million Prompts
Shoosmiths is the best-documented case in this research, and it isn’t a failure story. That’s what makes it a fair test of the argument: here, adoption was the explicit goal.
In April 2025, the UK law firm announced a firm-wide target of 1,000,000 Microsoft Copilot prompts, linked to a £1,000,000 conditional staff bonus pool subject to other eligibility and financial conditions. City AM reported in December 2025 that the target had been reached more than four months early. Financial Times reporting from July 2026, republished by The Irish Times, referred to 1,200 staff who had “claimed their bonus,” with about 200 partners on a separate reward arrangement. A second £1,000,000 was reportedly set aside for AI-fluency accreditation, which the firm’s chief executive described as the plan all along: “We always knew that in year two it was going to be about AI fluency.” The exact cash distributed hasn’t been verified.
The detail I find most instructive isn’t the headline number. City AM reported higher usage among trainees and associates and lower usage among partners, which the chief executive regarded as consistent with their supervisory role. That’s usage read in the context of the role instead of ranked as a single score, and it still isn’t evidence that either group was doing better work.
So what did a million prompts establish? That the firm hit its prompt target, which answers a real adoption question. The sources examined don’t link the incentive to productivity, quality, or client value. That isn’t evidence those improvements didn’t happen. Unestablished isn’t the same as false. It means the stronger conclusion hasn’t been earned from the evidence in hand, and the practical fix is to define the intended outcome before setting the target, then assess what changed afterward.
When the Reward Changes What We Notice
There’s a famous shortcut available here: invoke Goodhart’s Law, declare that people game whatever gets measured, and move on. It would make a punchier article, and it would outrun the evidence. The AI-specific research I examined doesn’t establish that usage incentives generally cause gaming, that visible AI metrics reduce quality, or that leaderboards necessarily distort performance.
What we do have is adjacent laboratory research. In experiments published in The Accounting Review (Choi, Hecht & Tayler, 2012), participants in a strategy task were compensated on a narrow measure, and their later choices shifted in ways consistent with treating the measure as the goal it stood for. Researchers call this strategy surrogation. Paying on three measures instead of one reduced the overall surrogation score, though the evidence for reducing the most strategically costly choices was less consistent. These were graduate business students in a lab, not employees using AI. What transfers is a question worth testing: does a rewarded measure shape later judgment? What doesn’t transfer is any AI-specific effect size, prevalence, gaming rate, or harm.
That leaves four layers worth keeping separate: what’s visible, what gets attached to it, how people respond, and what happens to the work. Salience isn’t gaming. A measure can become more noticeable or more consequential without anyone manipulating it, and gaming is a behavioral claim that needs behavioral evidence. Incentives aren’t automatically bad, either. Shoosmiths used one to encourage adoption, and whether an incentive supports learning or distorts attention is an empirical question in each setting.
This is also where the promotion from the start of this issue gets a name. I use “the Proxy Transfer” as an editorial label for granting a metric authority beyond what its evidence supports. It’s related to strategy-surrogation research, and I’m not claiming a new or validated AI-specific mechanism. Before we reward a measure, the better question isn’t whether people will game it. It’s what response the reward might encourage, and how we’ll check.
Authority Belongs to the Use
A metric doesn’t carry one universal level of authority. It has an intended use, and the same AI-usage information can serve several: observing activity, informing where to ask questions, contributing to an evaluation of work, or feeding a consequential decision.
Those aren’t validated stages, and there’s no automatic climb. Interpretive reach and decision stakes are different dimensions. Deciding where to spend on training can be consequential while asking little of the number beyond what it observes. Claiming that a team’s work improved asks a great deal of the same number, even before anything rides on the claim. The more consequential the use, the more carefully its interpretation has to be justified, but the justification follows the inference, not just the size of the decision.
The principle is older than enterprise AI. The Society for Industrial and Organizational Psychology’s Principles for the Validation and Use of Personnel Selection Procedures (5th ed., 2018) hold that a procedure should be used only for purposes its validity evidence supports. NIST’s AI Risk Management Framework 1.0 (2023) ties validation to a specific intended use and documented limits. Neither prescribes this ladder or validates my four questions; applying their principle to AI-usage metrics is my adaptation.
That’s also why “more consequential” shouldn’t be translated into “collect more data.” More screenshots won’t rescue a weak inference, and neither will another tile on the same dashboard. Sometimes existing evidence fits the intended use. Sometimes the claim needs an outcome measure or a comparison. Sometimes the honest move is to narrow the claim, and sometimes the evidence stays insufficient. That’s allowed. Uncertainty is a better management outcome than false precision with a cyan progress bar.
Visibility Isn’t Authority
AI is giving organizations extraordinary visibility into activity, and that’s useful. It also creates a peculiar management hazard: the easiest things to count can become the easiest things to believe. The Judgment Layer isn’t asking anyone to distrust the number. It’s asking us to stop confusing visibility with authority, which takes a few deliberate questions before the number gets used.
🧭 The Judgment Move
Orienting sentence: The AI Proxy Audit is four questions to ask before a usage number becomes a verdict: what it observes, what it leaves unobserved, what we’re trying to establish, and what authority we’re about to give it.
Question 1: What does this number directly observe? Stay literal. Name the activity the measure records: use, a defined activation event, prompts, tokens, recorded AI-suggestion acceptance. Stay inside its definitions and coverage. Basis: Evidence; vendor documentation defines these indicators from activity records.
Question 2: What does it leave unobserved? Name the parts of the claim the number doesn’t directly measure: quality, rework, cycle time, customer outcome, financial return. Unobserved doesn’t mean unchanged or unimportant; it means the number didn’t measure it. Basis: Evidence for what the definitions include; the gaps depend on your claim.
Question 3: What are we actually trying to establish? Name the claim before choosing the evidence. “Are people using the tool?” is an activity question, “Did the work improve?” is an outcome question, and “Did AI contribute to the change?” is a contribution question. A dashboard can answer the first beautifully and say very little about the other two. Basis: Design inference, informed by SIOP’s intended-use principle.
Question 4: What authority are we about to give this number? Name the decision use: an adoption check-in, a support conversation, a performance review, compensation, staffing, a transformation claim. Then ask what use, if any, the evidence justifies. Existing evidence may suffice if it fits. Basis: Design inference, informed by SIOP and NIST.
The answers point to one of three responses: use the measure for the bounded question it supports, seek outcome-relevant evidence for the stronger claim, or narrow the claim. If support stays insufficient, the stronger conclusion stays unestablished. Organizations hate leaving a box blank, but a blank box is often more honest than a number doing a job it was never qualified for.
Done state: the number moves forward with a named decision use and visible support for that use, or the claim attached to it is narrowed to what the number can carry.
Constraint: this only works if “we don’t have evidence for that yet” is an acceptable answer in the room where the number gets used.
I’ve been in leadership conversations where the dashboard looked healthy while some of the supposedly completed work was quietly returning as rework. The numbers weren’t wrong. Stories had closed. Velocity had been recorded. The mistake would’ve been treating those facts as proof that delivery had improved. Over time, I started asking a different question: what stayed closed? That small shift changed the conversation from whether the metric was accurate to whether we were asking it to prove something it had never measured.
Limitation: these questions can’t prove a metric is accurate, prove AI caused an outcome, prove a person performed well, or prove a decision is fair or legal. Nor can they prove that asking them improves decisions. The AI Proxy Audit is an unvalidated, principles-informed discussion aid. It can expose an inference gap; it can’t close one.
🧪 What Holds Up
Primary claim label: Evidence for the measurement distinction; Design inference for the AI Proxy Audit as a bundle.
Basis: Microsoft and GitHub documentation define adoption and usage indicators from activity records, without independent assessment of the associated work. SIOP’s Principles and NIST’s AI RMF both tie valid use to a specific intended purpose, supported by evidence and documented limits. Shoosmiths shows a named employer using a defined prompt-volume target tied to a bonus pool, with target achievement reported by independent outlets.
Counterpoint, labeled Evidence: usage-related information isn’t meaningless. You’ve seen “Generative AI at Work” (Brynjolfsson, Li & Raymond, Quarterly Journal of Economics, 2025) in Episodes One and Two; it’s here for a different reason. In a staggered rollout covering 5,172 customer-support agents, the study found larger estimated productivity gains among workers with higher initial adherence to the AI’s suggestions. Adherence isn’t raw prompt volume, and this isn’t a test of whether usage predicts work quality, but it blocks any claim that usage tells you nothing.
Scope and transfer limit: SIOP’s Principles were written for personnel selection; NIST’s framework is voluntary AI-system guidance; the surrogation evidence comes from a lab. The named cases come mostly through newsroom reporting and executive accounts, and none establishes how common usage-based evaluation is in Product, Program, Design, or Research teams.
What remains unknown: whether raw usage improves prediction of work quality once role and task are accounted for (no qualifying test was found); whether evaluative stakes on AI usage change behavior or outcomes in real workplaces; whether any case paired a usage-driven personnel decision with outcome evidence pointing the other way (none was found, which isn’t proof it never happens); and whether the AI Proxy Audit improves decisions. It hasn’t been tested.
🛑 Do Not Use When
The question really is access, adoption, or usage cost. Usage data may be the right evidence; answer it directly instead of adding ceremony.
You want to rate, rank, or monitor individual employees. These questions examine an inference, not a person.
You need a compliance, fairness, or legal answer. Neither SIOP nor NIST prescribes this aid, and it can’t settle privacy or employment-policy questions.
Judgment Seat
I’ve spent much of my career sitting between the people doing the work and the people reading the dashboard. That seat changes how a number looks. At team level, the context is obvious: dependencies, rework, tradeoffs, and exceptions are part of the story. By the time the number travels upward, much of that context can disappear. I learned that interpreting metrics isn’t just a reporting task. It’s also a responsibility to preserve the boundary between what the number observed and what someone wants it to mean.
Who generated the number: the tool’s telemetry, recording what it was built to record.
Who must understand it: the manager about to use it, to the depth of the decision it will inform.
Who can stop it: whoever owns the decision the number is about to enter: the review process, the staffing call, or the transformation narrative.
Who pays if it is wrong: the person, team, or rollout judged by a number that couldn’t see the work, and the organization that believes it learned something it didn’t.
🎁 Your AI Proxy Audit
The AI Proxy Audit turns the four questions into a working page for the moment before a usage number enters a review, a staffing decision, or a board slide. It asks you to name what the number observes, what it leaves unobserved, what claim you’re trying to support, and what decision authority you’re about to give it. There are no scores, thresholds, or automatic verdicts. It’s an unvalidated, principles-informed judgment aid, offered so teams can start testing the practice: name the claim, match the evidence, and narrow what the number can’t carry.
This Week
Three actions for the judgment owner: Pick one AI usage number you report or receive, and write one sentence naming what it directly observes. Write down which decision it currently informs. Before your next review cycle, check whether any AI-use expectation names the outcome it’s supposed to support.
One action for the system owner: Decide in writing which decisions AI usage data may inform on its own, such as access, support, and cost, and which require outcome-relevant evidence first.
One action for learning and provenance: The next time a usage number appears in a deck, add one line beneath it: what it observes, and what it doesn’t.
📎 Forward This
An AI usage number can be accurate and still be asked to prove too much. Before it judges a person or a program, ask what it’s allowed to prove.
🔚 Closing
The dashboard counted the activity correctly.
Deciding what that count is allowed to prove is still human work.
Next week, Episode Five asks the season’s longest-horizon question. Who Trains the Next Reviewer? looks at the tasks that once formed future judgment, and what happens when they’re the first ones automated. Because a usage number can show who’s using the tool. It can’t show who’s learning to judge the work.
This question is part of the larger challenge behind my upcoming book, Collaborate Better: From Silos to Synergy – How to Build Unstoppable Teams. Better collaboration begins when context, ownership, and judgment travel with the work.
Learn more at CollaborateBetter.us.
Regards,
Mark 👋
Previous:
More Content to Discover:
Half Your Team Is Using AI. You Don't Know Which Half.
Intro If you read the piece we published last week on Leadership in Change, you saw the structural diagnosis: mandate pressure, decision collapse, intake collapse. Three failures that sit underneath almost every AI pilot that goes sideways. That piece mapped the architecture. What it did not cover is what happens on the human side of the same problem, wh…
The most dangerous AI metric may be the one that’s completely accurate. The trouble starts when a number that measures usage gets promoted into evidence of performance, value, or progress.
What’s one metric in your organization that might be carrying more authority than it has actually earned?