The Ben Schein interview

So Ben, where do enterprise leaders first lose visibility?

The honest answer is upstream of all five options: they lose visibility the moment AI spend stops mapping to a workload. Model selection, prompt volume, agent loops, and embedded SaaS AI are all just places where that one failure shows up. Without a shared cost model, every team meters AI differently, some through licenses, some through tokens, some not at all, and the CFO is left adding apples to exhaust fumes.

The first practical casualty is usually embedded SaaS AI, because it hides inference inside someone else’s subscription line.

The second is agentic workflows, because one request can quietly become forty model calls. But the root cause is the same: no common unit that ties an AI action to a workload, an owner, and an outcome.

---

Top takeaways from the interview:

• Attribution Is the Real Bottleneck. AI costs become invisible the moment spend stops mapping to a specific workload, whether it’s hidden in embedded SaaS AI or buried inside agentic workflows that quietly multiply into dozens of model calls. Schein argues a mature cost dashboard must start with full attribution — who triggered an action, which workload it served, which model ran, and what it cost — because without that row of data, “you do not have a cost problem yet; you have a visibility problem, and it is worse”.
• Measure Path Efficiency, Not Just Completion. Schein reframes agent monitoring around the idea that finishing a task isn’t enough if the route was needlessly expensive; the key metric is path efficiency — steps, retries, and tool calls relative to what a task actually required. He recommends structural safeguards like budget envelopes, step ceilings, and full action logging, warning that “autonomy is not the risk. Unobserved autonomy is”.
• Treat Model Selection as Financial Discipline. Because model pricing spans a roughly 30x spread and keeps falling, Schein frames model portfolio management as procurement strategy rather than pure technical architecture. He advises defaulting high-volume work to small models in batch mode, tracking the share of tokens on premium tiers monthly, and codifying settled use cases into deterministic code — since “a prompt you re-run bills you every time,” while code runs free.

---

What are you seeing inside Domo about the difference between AI projects that show measurable value and AI projects that create activity without operating leverage?

The projects that show value start with a question someone is already paid to answer. Inside Domo, our best internal AI work looks boring on purpose: turning engineering tickets into customer-ready release notes, routing and summarizing support cases, analyzing our own consumption data to guide the business. Each one has an owner, a defined output, and a moment where we locked it down: once the exploration found the right approach, we hardened it into governed pipelines so it runs the same way every time. The AI stopped being a conversation and became infrastructure.

The projects that create activity without leverage have the opposite shape. Open-ended assistance with no owner, no defined output, and no point where anything gets codified. The work feels productive because tokens are flowing, but nothing compounds. My shorthand: value compounds when AI output becomes an asset you keep; activity evaporates when it becomes a message someone has to read. If a project has been in pilot for six months and nothing has been locked down yet, that is not a pilot; that is a subscription.

What does a mature AI cost dashboard need to show beyond total spend?

Attribution first. Every AI interaction should carry who triggered it, what workload it belongs to, which datasets it touched, which model ran, and what it cost. If you cannot produce that row, you do not have a cost problem yet; you have a visibility problem, and it is worse.

From there, a mature dashboard shows unit economics rather than totals: cost per workflow run, cost per solution, trend per solution over time. It shows the mix: what share of tokens runs on premium tiers versus small models, what share runs batch versus interactive, and what share of runs is automatic versus human-initiated. Those three ratios predict next quarter’s bill better than any forecast. It should also surface zombies: agents and scheduled jobs still consuming resources whose output nobody has opened in weeks.

One more view matters, and it is about people rather than machines: adoption as a threshold, not a leaderboard. I do not need a ranking of who used the most tokens. Publishing one just invites people to game the meter.

What I need to know is who is working with AI at some meaningful threshold and who has not started, because the second group is quietly being left behind while the first group compounds. The dashboard should make the gap visible so leaders can close it with enablement, not shame. And it should pair every material cost line with an outcome line at the same grain, so the review conversation is never about whether spend went up, it is about whether yield went up with it.

How should companies monitor agent behavior so they can distinguish productive autonomy from expensive motion?

Measure agents the way you would measure a new employee: not by how busy they are, but by what they finish, and by how direct the route was. Completion alone is not enough, because an agent can reach the right answer through an absurdly expensive path. So beyond tokens per completed outcome, the metric I want is path efficiency: how many steps, retries, and tool calls the run actually took against what the task plausibly required. Think of it as the difference between the odometer and the map. Two agents can both complete the trip, but if one drove three times the distance, that is a design problem you want to see before it scales.

An agent in expensive motion shows the signatures: retries per success climbing, tool-call chains getting longer for the same task, fan-out where one request spawns dozens of sub-calls, and runs that end without an action anyone used. A healthy agent shows the opposite: its path per outcome shortens over time as the design matures.

Structurally, three controls matter. Give each agent a budget envelope and a step ceiling so runaway loops fail cheap and loud instead of quietly expensive. Log every action with full attribution: which agent, which trigger, which data, which model, what cost, what result. And place human checkpoints only where judgment actually changes the outcome. Routing everything through approval trains reviewers to click yes, which is how you end up paying for both the agent and the person pretending to watch it. Autonomy is not the risk. Unobserved autonomy is.

How should leaders think about model portfolio management as a financial discipline, not just a technical architecture?

Treat it like procurement for a commodity input with a 30x price spread, because that is what it is. The published rate difference between the cheapest and most expensive models is larger than the spread most companies tolerate in any other input they buy, and unlike most inputs, the price curve is falling fast. Portfolio management means capturing that curve instead of being captured by it.

In practice: set a default tier policy where high-volume, repetitive work runs on small models in batch mode and frontier models are reserved for the problems that genuinely need them. Watch one number monthly: the share of total tokens running on premium tiers, and make someone defend it. Preserve the ability to swap, including running your own models on your own infrastructure, because switching ability is what converts vendor pricing power into your negotiating power. And remember the cheapest model call is the one you stop making: once a use case is settled, codify it into deterministic logic. Code you have written runs free forever, a prompt you re-run bills you every time.

What are the early warning signs that AI costs are outrunning value?

The clearest one is divergence: token consumption doubling while completed outcomes stay flat. Usage growth is only good news when yield grows with it.

A warning sign leaders create for themselves is turning usage into a scoreboard. The moment token counts become a proxy for being AI-forward, people start maxing the meter: longer prompts than the task needs, redundant runs, frontier models on routine questions, AI inserted into work that did not ask for it.

You get the appearance of adoption and the invoice of adoption without the value of adoption. It is not a contest or a race. The question that actually matters is a threshold question: who is genuinely working with AI and who is not, because the people not using it at all are the real risk to the business, and the people performing usage are the real risk to the budget.

Beyond that, watch for these patterns. Interactive spend on repetitive questions, meaning people are re-asking what should have been codified once. Automatic triggers multiplying, agents and scheduled jobs generating output on a calendar whether or not anyone needed it. Solutions with no owner, which is how spend survives long after value stops. Rising verification load downstream, where recipients spend more time checking AI output than the sender saved producing it.

Researchers at Stanford and BetterUp Labs found workers spend nearly two hours cleaning up each piece of low-quality AI work they receive, which means the cost shows up in someone else’s calendar before it shows up in your invoice. And frontier models doing routine work, the single most common silent overspend. None of these require waiting for finance to flag them. They are all visible in usage data weeks earlier, if the usage data carries attribution.

Can you define the new AI operating metric stack, and how can a cost model and a benefit model coexist at this pace of change?

I think of it as four layers. The meter layer is raw consumption: tokens, calls, agent actions. Necessary, but on its own it is just a utility bill. The unit layer converts that into economics: cost per workflow run, cost per decision supported, cost per governed agent action. This layer is also where efficiency lives, and it needs to be more sophisticated than a completion count.

The maturing version measures the path, not just the destination: tokens and steps per completed outcome, retries per success, actual route against a reasonable route for the task. Usage tells you the meter is spinning. Completion tells you something came out. Path efficiency tells you whether the machine is well designed, and it is the number that should improve quarter over quarter as solutions mature.

The yield layer connects units to business value: cost per avoided labor hour, cost per improved forecast, cost per case resolved. And the governance layer measures trust in the other three: what share of AI actions are fully attributed, what share of spend has a named owner, what share of outputs are grounded in governed data.

The way cost and benefit coexist is grain matching. Pair every cost line with a benefit line at the same level: this workflow costs this per run and saves this per run. Where the benefit is hard to quantify, say a better forecast, borrow the method from the research: measure the error with and without the AI in the loop and price the difference. Then hold the whole stack loosely.

Model prices are falling, and capabilities are shifting quarterly, so a benefit case that was marginal in January may clear easily by June, and a premium workload may deserve demotion to a small model even sooner. The discipline is not building the perfect model once. It is reviewing a living one on a cadence, the way you would any other operating budget.

About Ben Schein

Ben Schein is the Chief AI and Analytics Officer at Domo. Ben has over two decades of experience leading user adoption and implementing large-scale BI and analytics initiatives that deliver quantifiable business value. As a Domo user and content creator for over a decade, Ben brings empathy, intellectual humility, and transparency to his role as Chief Analytics Officer and SVP of Product, in which he oversees Domo’s Analytics strategy, Product Management and UX teams, as well as guides the overall product roadmap for Domo. Ben also leads Domo’s Strategic Architecture Group (SAG), which advises on architectural patterns for complex implementations. He is a passionate advocate of sparking the fire of data curiosity and innovation for Domo customers across the globe.

Prior to Domo, Ben worked at Target Corporation, where he led merchandising analytics and enterprise BI capabilities within the Enterprise Data Analytics and BI (EDABI) Center of Excellence.

Ben holds a bachelor’s degree in Philosophy, Politics and Economics from the University of Pennsylvania and an MBA in Strategy and Finance from the Carlson School of Management at the University of Minnesota.

For more serious insights on AI, click here.

For more serious insights on learning, click here.

Did you enjoy the Ben Schein interview? If so, like, share or comment. Thank you!