AI Use Is Not AI Value: Why Token Counts, Adoption Dashboards, and Agent Activity Metrics are Poor Proxies for Business Impact
Microsoft, Meta, Shopify, Amazon and the rest of the AI-forward enterprise set have discovered a management problem hiding inside their own enthusiasm. Once a new technology becomes strategic, the organization wants a number. AI makes numbers easy: tokens consumed, prompts submitted, active users, agent runs, Copilot retention, code suggested, code accepted. The numbers look precise. They are not the same as evidence. Organizations that confuse usage with value will invest heavily in AI while leaving their operating model largely unchanged.
Knowledge managers committed the same error in the 90s and 2000s, counting questions, responses or the number of knowledge bases as proof that knowledge management was working. They should have been looking at the impact, reuse and capability enhancement from the articles rather than participation.
The problem is never measurement. Measurement is a given. The problem starts when the easiest measurement becomes the management system. A usage metric that begins as telemetry quickly becomes a target. A target becomes an incentive. An incentive becomes behavior. Soon, people are no longer trying to improve work; they are trying to make the dashboard move — or just gaming it so they appear on it.
Microsoft’s Copilot Dashboard in Viva Insights is a useful example because it exposes both the value and the risk of enterprise AI telemetry. Microsoft says the dashboard helps organizations assess readiness, adoption, impact, and sentiment. It also includes benchmarks, agent-related insights, group-level data and other tools that make adoption visible to leaders. Microsoft’s separate announcement of Copilot adoption benchmarks positions the feature as a way to compare usage internally and externally. The Register captured the management implication more bluntly: the dashboard can help bosses spot teams that are not using Copilot as much as others.
In a “count activity as value” world, measurement transforms from diagnostic into dashboard kabuki. If a leader asks why one business unit has fewer active Copilot users than a peer group, the next meeting is unlikely to focus first on work design, data readiness, process quality or customer outcomes. It will focus on increasing its numbers so it is reflected in the measurement rather than the measurement of value.
Table of Contents
Adoption Is Not Implementation or Impact
An organization can deploy Microsoft 365 Copilot broadly and still have no AI capabilities implemented. It can also have modest adoption numbers and very strong implementation in a few high-value workflows. The dashboard does not know the difference unless the organization links AI telemetry to real-world outcomes.
Adoption asks whether people touched the tool. Implementation asks whether the tool was accepted. Impact asks whether the changed work improved cost, quality, speed, resilience, risk, customer experience, employee experience or strategic optionality. These are different questions. Confusing them produces a familiar management mistake: mistaking motion for progress.
The Microsoft case is not unique. It is simply better instrumented. And that is a problem with modern technology. If nothing else, it makes counting almost any activity easy. Any platform that reports active users, weekly use, retention, AI-assisted meetings, prompts per employee, generated documents, or summarized threads risks creating the same distortion. The more beautiful the dashboard, the more tempting the misuse. What modern technology cannot do is determine whether the business is better after all this activity takes place. That’s why my The Serendipity Economy explicitly recognizes that realizing or discovering value derives from measurement activities independent of the process that purports to create value, and that those activities must be as well-designed as the initial automation.
AI vendors need adoption data. IT needs adoption data. Finance needs consumption data. Security needs anomaly data. None of those needs justify turning raw usage into a proxy for value. The metric should remain backstage unless it is joined to the purpose of the workflow.
Tokenmaxxing: The New Lines‑of‑Code Fallacy
Diginomica argues the basic point cleanly: token consumption, time in AI tools and the percentage of code written by AI are activity metrics, not productivity metrics. They may reveal where usage occurs. They do not reveal whether the work improved.
The newer and more combustible version of this error is tokenmaxxing. The term describes the practice of maximizing LLM token use to signal AI adoption, engagement or productivity. Business Insider has reported on the debate over AI token leaderboards, including Meta’s internal leaderboard and the broader discussion over whether token consumption should count as evidence of effective AI work. The Information reported that Meta employees competed for token status on an internal leaderboard that tracked tens of trillions of tokens. The particulars may shift as companies rename dashboards, shut down informal tools or restrict access, but the underlying behavior is the same: people adapt to what management demands be made visible.
As with knowledge-base metrics, developers will recognize this as lines-of-code metric reborn with a transformer accent. (Why managers and leaders do not confounds me to no end). In times past, organizations rewarded developers for writing more lines of code. So they wrote more lines. Reward teams for burning more tokens and they will burn more tokens. Neither metric answers whether the code is easier to maintain, whether the feature solved a customer problem, whether the defect rate dropped or whether the organization learned anything of lasting value.
The Pragmatic Engineer pointed to Shopify as a more useful case because Shopify reportedly moved away from the leaderboard framing toward a usage dashboard with circuit breakers and spend-spike detection. That was a good move. Token data can expose runaway agents, expensive loops, model-selection mistakes and infrastructure bugs. Monitoring activity belongs in cost management, observability and anomaly detection. It does not belong on a scoreboard that encourages people to confuse consumption with competence.
Amazon reached a similar conclusion when it shut down an employee-created AI leaderboard and told employees not to use AI just to use AI, according to Business Insider reporting on Amazon. Indeed’s CIO made the same management argument from the opposite direction: Indeed monitors AI spend but avoids tokenmaxxing-style leaderboards because incentives around raw usage can push the wrong behavior, as described in Business Insider’s interview with Indeed.
The healthiest version of token measurement is boring. It belongs in a finance, platform engineering and AI operations context. It should help teams detect cost anomalies, choose models, manage context windows, reduce waste and understand which workflows consume compute. It should not define who is “good at AI”d or act as a proof point that AI adoption is going well.
Development Metrics Need Output, Quality and Consequence
Software development makes the measurement problem clearer because AI coding tools produce visible artifacts quickly. Code appears. Pull requests arrive. Tests run. Tokens disappear. The temptation is to draw a straight line from AI use to engineering productivity.
That line needs several bends. A recent study of Microsoft’s early 2026 rollout of command-line AI coding agents, including Claude Code and GitHub Copilot CLI, found that adopters merged roughly 24 percent more pull requests than they otherwise would have, while the authors also acknowledge that merged pull requests are a proxy for output rather than value. That distinction is important. A merged pull request can be useful, unnecessary, risky, redundant or expensive to maintain. The study is valuable because it does not pretend that usage alone is the answer; it connects adoption to an output measure and names the limits of the proxy.
The better measurement approach should combine several signals: flow efficiency, cycle time, defect escape rate, rework, security findings, architectural debt, maintainability, developer cognitive load, customer-visible impact and operational incidents. AI may improve some of these while degrading others. Usage metrics alone cannot help discern that trade-off. For instance, a code assistant that slightly increases cycle time but dramatically reduces defect escape and architectural debt may create more value than one that only optimizes raw output.
The percentage of code generated by AI is particularly weak. It says nothing about whether the code fits the architecture, whether it passes meaningful tests, whether it survives review, whether it introduces hidden dependencies or whether it leaves the next developer with a mess. A small AI-assisted refactoring that removes complexity may create more value than thousands of generated lines of code.
Agentic AI Makes Activity Metrics More Dangerous
Chat interfaces can waste time and money. Agentic AI can also create operational consequences. Agents do not just answer. They observe, plan, retrieve, call tools, modify records, open tickets, write code, schedule work, trigger workflows and sometimes act across systems without a person watching every step. That autonomy changes the measurement obligation. Agents create a blast radius driven by the consequences they create when they are wrong or misaligned.
Gartner warns that applying uniform governance across AI agents will fail, predicting that by 2027, 40 percent of enterprises will demote or decommission autonomous agents because governance gaps surface only after production incidents. Gartner’s model distinguishes agents by autonomy level and scope, from observe-only agents to agents that act with approval or act autonomously across trust boundaries. That is a good axis for measurement.
A read-only knowledge retrieval agent should be measured on answer accuracy, source grounding, retrieval freshness, coverage and failed-query patterns. An advisory agent should be measured on recommendation quality, expert override rate and decision improvement. An agent that acts with approval should be measured by approval rates, rollback rates, exception routing, and the time it takes to return to the process. A fully autonomous agent should be measured like an operational actor: incidents caused, incidents prevented, changes made, auditability, escalation effectiveness, termination reliability and contribution to business outcomes.
Research on runtime governance reaches the same basic conclusion from a technical perspective. The MI9 Agent Intelligence Protocol paper argues that agentic systems create runtime risks that conventional pre-deployment governance cannot fully anticipate, requiring telemetry, continuous authorization monitoring, conformance checks and containment strategies. In business language: do not measure an agent only by how often it runs. Measure whether it stays inside its purpose, whether it can be stopped, whether its decisions can be reconstructed and whether its actions improve the work it touches.
A Serious Insights Lens: Use Cases, ROI, Design, Capability and Maturity
The better way to measure AI starts where Serious Insights has argued AI programs should start: with the work. In A Framework for Enterprise AI Success, I argued for defining clear objectives, aligning AI with business goals, understanding stakeholder pain points, establishing success criteria and evaluating feasibility before treating AI as an implementation project. That still applies, and it applies more strongly now that agentic systems can act.
AI should be managed as a portfolio of use cases, not as a monolithic platform rollout. In a Serious Insights interview on the future of AI, I framed the more useful view as a portfolio of use cases with different levels of evidence. Some use cases have proven ROI. Some are learning bets. Some remain experiments. Some should be rejected because the data, workflow or technology maturity does not support them.
ROI also needs a broader frame than revenue alone. In Enterprise AI Insights from the Field, I noted that much of the leverage may sit in back-office and mid-office work such as HR, claims processing, customer support, procurement and internal operations, where redundancy and fragmentation create measurable opportunity. AI value may appear as hours saved, errors avoided, faster cycle times, lower support burden, better compliance, improved resilience or clearer decisions. Those are outcomes. Tokens are residue.
Design matters because AI does not drop into work in a neutral way. The workflow, user role, handoff, human review point, exception path and escalation rule determine what the AI system can reasonably improve. A drafting assistant can shorten report preparation only if the organization knows what “good enough” means and where review belongs. A support agent can improve service only if it has trusted knowledge, clear authority and a mechanism to learn from unresolved cases. A procurement agent can reduce cycle time only if it understands policy, supplier risk, negotiation constraints and approval boundaries.
Capability analysis matters because AI systems are uneven. A large language model may summarize a policy well and still fail to reason about conflicting obligations. A retrieval system may work well when source authority is clear and fail when documents are stale, duplicated or semantically inconsistent. In Is Your Data AI-Ready?, I argued that AI use cases are often selected before data constraints are understood and that intake should include data feasibility assessments based on source authority, completeness, sensitivity and remediation cost. That recommendation should be part of every AI measurement framework.
Maturity matters because the technology, vendors, regulations, costs and failure modes keep moving. In The End of Future-Proofing, I argued for scenario thinking over prediction because AI forces ongoing triage rather than stable future-proofing. Measurement should reflect that reality. Organizations need to know which workflows depend on which models, which substitutes exist, how quickly they can shift providers, what happens when a model is restricted, and how gracefully work degrades when AI is unavailable or wrong.
The April 2026 State of AI update made the same point from an adoption perspective: organizations should be cautious about using survey data to drive competitive investments, and the better path remains strategic capability alignment, AI competency development and safety readiness. A leaderboard answers none of those questions.
What Better AI Measurement Looks Like: A Layered Model for AI Measurement
A serious AI measurement model has layers. The first layer is telemetry: tokens, latency, cost, active users, prompt counts, agent runs, model calls, tool calls and failure rates. This layer keeps the system observable. It should not define success.
The second layer is workflow performance: cycle time, throughput, quality, rework, escalations, handoff delay, error rate, customer satisfaction, employee effort, compliance exceptions and operational resilience. This is where AI either changes the work or fails to justify itself.
The third layer is capability maturity: data readiness, model fit, integration stability, governance proportionality, test coverage, auditability, human review quality, agent autonomy level and resilience to model or vendor substitution. This layer explains whether the measured improvement can survive beyond a demo.
The fourth layer is strategic contribution: differentiation, option value, learning, new services, reduced complexity, improved knowledge flow, better decisions and the ability to absorb future shocks. Some of this will resist quarterly measurement. That does not make it soft. It means leadership needs a richer conversation about measurement than “how many tokens did we burn?”
The final question should not be “Are people using AI?” It should be “Where has AI earned a role in the operating model, where is it still experimental, where is it neutral, and where is it making the organization look more modern while making the work worse?”
Practical Actions for Leaders
- Separate telemetry from performance management. Collect token counts, active users, prompt volumes and agent runs, but keep those measures in operations, cost management and observability. Do not use them as employee rankings, team rankings or executive evidence of AI success. A metric used for cost control behaves very differently from a metric used to judge people.
- Require every AI deployment to declare its workflow, owner and outcome. Before approving a tool or agent, document the work it enters, the decision it influences, the user role it supports, the business outcome it should improve and the person accountable for its behavior. If the use case cannot name an outcome beyond “more AI use,” it is not ready for scale.
- Create use-case scorecards instead of adoption scoreboards. For each use case, define baseline performance, expected improvement, cost-to-serve, quality measures, risk measures, and review cadence. A customer support assistant might track first-contact resolution, handle time, escalation quality and customer satisfaction. A coding assistant might track cycle time, defect escape rate, review burden and maintainability. A procurement agent might track cycle time, compliance exceptions and negotiated value.
- Classify agents by autonomy and blast radius. Use a tiered model: observe, advise, act with approval, act autonomously. Match controls and metrics to each tier. Observe agents require source-quality and accuracy measurement. Advisory agents require recommendation-quality and override tracking. Action agents require approval, rollback, exception and incident metrics. Autonomous agents require audit trails, containment, kill switches and periodic recertification.
- Instrument agentic systems for purpose limits, not just activity. Track whether agents stay within authorized tools, data sources, workflows and decision boundaries. Measure unauthorized tool calls, policy conflicts, escalation failures, repeated retries, loops, abandoned tasks and termination reliability. The best agent dashboard will look more like operational risk management than a consumer-app engagement chart.
- Tie AI cost to units of work. Move beyond vendor invoices and token totals. Calculate cost per resolved ticket, cost per drafted contract, cost per reconciled invoice, cost per accepted pull request, cost per research brief or cost per decision package. Then compare those costs with baseline labor, quality, cycle time and risk. Some AI will look cheap until rework is counted. Some expensive AI will prove valuable because it prevents downstream failure.
- Measure rework as aggressively as output. AI-generated output can create invisible debt. Track edits, rejected drafts, failed tests, hallucinated sources, legal review failures, security findings, customer corrections and post-release defects. Productivity gains that vanish into cleanup are not gains.
- Build data feasibility into the AI intake process. Before a use case is approved, score the data and knowledge sources it requires: authority, freshness, completeness, sensitivity, access rights, semantic consistency and remediation cost. If the organization cannot explain what the AI system is supposed to know, it cannot honestly measure whether the system works.
- Review AI impact in existing business forums. Do not isolate AI measurement in an AI committee that reports adoption success to itself. Bring AI scorecards into product reviews, operational reviews, risk reviews, audit reviews and strategy sessions. AI should be evaluated as a change to the operating model, not as a promotional campaign for a platform.
- Run scenario exercises around AI failure and metric gaming. Stress-test what happens when a model becomes unavailable, costs spike, a vendor changes terms, an agent acts outside purpose, a leaderboard distorts behavior or adoption rises while outcomes fall. Use the results to revise governance, architecture, procurement language, escalation plans and communications.
- Retire AI systems that cannot prove their role. Treat retirement as a normal part of the portfolio. Some tools will remain experiments. Some agents will be demoted to advisory roles. Some use cases will fail because the data is not ready or the technology is not mature enough. A disciplined organization will remove AI from work where it does not help, even when the usage chart looks good.
The organizations that mature fastest will not be the ones with the biggest token charts. They will be the ones that can explain where AI improves work, where it increases risk, where it has earned autonomy and where it should remain a draft assistant, a retrieval tool or a rejected idea. AI use is a signal. AI value is an argument that must be proved in the work.
For more serious insights on AI, click here.
All images via AI from a prompt by the author, unless otherwise noted.
Did you find AI Use Is Not AI Value useful? If so, please like, share, or comment. Thank you.
Leave a Reply