Avani Desai on AI Safety, Certification, and Accountability: A Serious Insights Interview

An AI agent recommends a payment that should never be made. Is the failure in the model’s reasoning—or in the organization that gave it permission to move the money? That distinction sits at the heart of this conversation with Avani Desai, Schellman’s CEO. Rather than promise that AI can be certified as universally safe, Desai asks a harder question: can an organization show it understands the risks and has controls in place to keep a model’s mistakes from becoming real-world harm?

In this Serious Insights interview, we press Desai on what AI assurance can establish—and what it cannot. Why don’t thousands of adversarial tests prove a system is safe? Who tests the testers? What happens to an assessment when an agent gains new tools or permissions? And when an independently assessed system causes harm, who answers for it? Her responses make the case for moving beyond trust labels to evidence, while confronting the limits of testing, the responsibilities certification cannot transfer, and the governance required when technology begins acting on our behalf.

Top takeaways from the interview:

• Certification is evidence, not a guarantee. An assessment applies to defined criteria, a specified system or deployment, and a period of validity. Its credibility rests on the evaluator’s independence, the quality of the evidence, and transparency about scope and limitations—not the logo on the certificate.
• The critical boundary is between a model’s mistake and an agent’s action. AI risk depends on the entire deployment: models, data, tools, permissions, and workflows. Authorization, transaction limits, segregation of duties, and human oversight help prevent imperfect outputs from producing unacceptable outcomes.
• Assurance must change as the system changes. Material updates to prompts, retrieval sources, tools, or permissions should trigger risk evaluation and, when warranted, immediate retesting. Periodic assessments need continuous monitoring, event-driven testing, and clearly assigned accountability before an incident occurs.

The Avani Desai Interview:

Let’s start with the word “safe.” Who gets to define it? Is it the model developer, the deploying company, the auditor, the regulator, or the people who could be harmed? What happens when those definitions conflict?

I actually think we’re asking the wrong question when we ask whether AI is safe. Safe compared to what? A human making the same decision? An existing technology? Or some standard of perfection that we’ve never applied to anything else? We don’t expect airplanes, financial systems, or medical devices to be completely risk-free. We expect organizations to understand the risks, establish appropriate controls, and demonstrate that those controls work. AI should not be any different.

The challenge is that we have multiple parties defining safety based on their own perspectives. The developer understands the technology, the enterprise understands how it’s being used, regulators establish expectations, and the people affected experience the consequences. All of those perspectives matter, but none should have exclusive authority. This is exactly why independent assurance exists. We need a way to move beyond what a company says about its technology to what can actually be demonstrated through evidence. We need to be comfortable acknowledging that safety isn’t static. Just remember, a system that is appropriate today may not be appropriate tomorrow as its capabilities, permissions, or intended use change!

You argue that independent evaluators need deeper access to AI systems. What access is actually necessary: model weights, training data, system prompts, evaluation tools, deployment logs, permissions, and incident histories? If the company being audited controls that access, how does an evaluator know it has seen enough?

I don’t believe independent assurance means auditors need unrestricted access to everything. In fact, I think that would create an entirely different set of security and intellectual property risks. What matters is whether we have sufficient evidence to independently validate the claims being made. There is an important distinction between technical evaluation and independent assurance. Organizations like METR, for example, focus on evaluating frontier model capabilities. An assurance provider like Schellman can assess whether an organization has established appropriate thresholds, whether evaluations occurred as required, and whether the organization actually followed its commitments when those thresholds were reached. Those are complementary roles, and both are necessary.

For deployed agents, we may need to examine system prompts, permissions, tool calls, logs, and adversarial testing results. For frontier model governance, we may need evidence of capability evaluations, escalation decisions, release gates, and management responses. But the key is that we cannot simply accept management’s representation that something works. We have to independently validate it. And if we don’t have sufficient access or evidence, we should be prepared to say we cannot provide the assurance being requested.

You cite an AIUC-1 audit involving more than 250 pieces of evidence and 2,000 adversarial scenarios. How were those scenarios selected, what universe of possible failures do they represent, and how can an auditor know it isn’t just testing the risks people already imagined?

I think this question gets at one of the biggest challenges with AI assurance. How do you test a technology when you don’t necessarily know all the ways it could fail? We’ve been assessing technology for decades, but AI introduces a different level of unpredictability. Traditional software generally follows defined instructions. Generative AI systems can produce different outputs under similar conditions, and agents can combine those outputs with real-world actions. The 2,000 adversarial scenarios are meaningful because they challenge the system across a broad range of risks, including prompt injection, unauthorized tool use, data leakage, and manipulation of the agent’s intended behavior.

But I wouldn’t suggest that 2,000 scenarios, or even 20,000, prove a system is safe. What matters is how those tests were designed, whether they reflect the actual deployment environment, what happened when the system failed, and whether the organization has controls to identify new risks. This is why I believe technical testing and traditional assurance have to come together. One evaluates how the system behaves. The other evaluates whether the organization has the governance, accountability, monitoring, and response mechanisms necessary to manage that behavior. I will say, neither is sufficient on its own.

What constitutes a passing safety result for a probabilistic system? Is there an acceptable failure rate, and how should an evaluator treat a failure that is extremely unlikely but potentially catastrophic?

I think we need to stop looking for one universal definition of a passing AI system. If an AI system produces an inaccurate marketing recommendation one percent of the time, that’s one conversation. If an autonomous agent has a one percent chance of executing an unauthorized financial transaction, that’s an entirely different conversation. And what about a failure that occurs only once in a million interactions but could cause catastrophic harm? A low probability doesn’t automatically make that risk acceptable. The industry needs to become much more sophisticated about risk thresholds, severity, and the consequences of failure.

We also need to separate the performance of the underlying model from the effectiveness of the controls around it. A model can make mistakes, but the system should be designed so that those mistakes don’t automatically result in harmful actions. For example, an agent might incorrectly determine that a payment should be made. But that doesn’t mean it should have unrestricted authority to execute the payment. This is where traditional control concepts like segregation of duties, authorization, transaction limits, and human oversight become incredibly relevant. We don’t necessarily need perfect models. It might be a pipe dream, but the end result should be, we need systems designed to prevent model failures from becoming unacceptable real-world outcomes.

AI risk often emerges from the combination of a model, system prompt, retrieval source, interface, tools, permissions, data, and organizational workflow. Does an assurance result apply to the underlying model, one configured agent, or one specific deployment? Where does the evaluated system begin and end?

This is where I think the industry is going to have to rethink how we approach assurance. We’re moving from AI systems that answer questions to AI agents that actually do things. They can connect to applications, access sensitive data, make decisions, and execute workflows. The risk is no longer limited to the model itself. It’s the combination of the model, the data, the permissions, the tools, and the environment in which it operates.

An underlying model could be independently evaluated and perform extremely well. But if an enterprise connects that model to its financial systems with excessive permissions, the risk profile changes completely. So when we talk about assurance, we have to be very specific about what we’re assuring. Is it the foundation model? Is it a particular agent? Is it the deployment environment? Or is it the business process in which that agent operates?

I think we’ll increasingly see different layers of assurance, much like we do today with cloud infrastructure, platforms, and applications. But those layers need to connect, and organizations need to understand which risks are inherited and which remain their responsibility. Otherwise, we are just creating a lot of certifications without actually addressing the risks that matter.

AIUC-1 calls for technical testing at least quarterly. Models, prompts, retrieval data, and tool permissions can change much more frequently than that. What kinds of changes should automatically invalidate an assessment and trigger immediate retesting?

This is one of the biggest opportunities for our profession, but also one of the biggest challenges. Our traditional assurance models were built around systems that generally changed through established release cycles. AI systems can change far more frequently, and sometimes the most significant changes aren’t even to the underlying model. A new tool connection, a change in permissions, a different retrieval source, or an update to a system prompt can materially alter what an agent is capable of doing.

I wouldn’t say every change automatically invalidates an assessment. But every material change should trigger a risk evaluation, and some should require immediate retesting. For example, moving an agent from read-only access to the ability to execute transactions is a fundamentally different risk profile.

We’ll see assurance evolve toward a combination of periodic independent assessments, continuous monitoring, and event-driven testing. The interesting question is how we make that scalable. If AI adoption grows as quickly as many expect, we cannot rely exclusively on manual testing every time something changes. So, we’re going to need more automation in assurance itself, while preserving the independence and professional judgment that make assurance valuable.

A system can pass general safety tests and still be unsuitable for a particular use—a benefits decision, a hiring recommendation, medical triage, or a relationship with a vulnerable child. Is an assurance provider prepared to say, “This system may work as designed, but it should not be used here”?

Absolutely! An important distinction between demonstrating that a system meets defined requirements and deciding whether it should be deployed in a particular environment. Just because we can use AI for something doesn’t mean we should. We have to recognize that risk tolerance changes based on the people affected and the consequences of getting something wrong. A system used to recommend a restaurant doesn’t require the same level of oversight as one influencing healthcare decisions or determining access to financial services.

But I would be careful about suggesting that auditors should become the ultimate decision-makers about where AI can be used. Our responsibility is to independently evaluate the system against established criteria, identify material risks and control weaknesses, and provide credible evidence to the people responsible for making those decisions. Management, boards, regulators, and other stakeholders still have roles to play. Where I think assurance adds tremendous value is in making sure those decisions are based on evidence rather than marketing claims or assumptions about what the technology can do.

Who tests the testers? What qualifications should an AI evaluator possess, who accredits them, and how do we know that two competent evaluators examining the same system would reach comparable conclusions?

Funny you ask, I was just thinking about this and this is actually one of the questions I care most about. We’re seeing an explosion of AI certifications, frameworks, trust labels, and companies claiming they can validate whether AI is safe. But who is validating the organizations making those claims? Independent assurance is a profession built on standards, accreditation, independence, competency, and quality management. Those principles exist for a reason. Here at the firm, we’ve spent years building our business around those requirements. We became the first accredited AIUC-1 auditor, and we’ve gone through accreditation for ISO 42001. That process isn’t easy, nor should it be.

I also think AI assurance requires a new combination of expertise. You need people who understand audit methodology and internal controls, but you also need specialists who understand model behavior, adversarial testing, AI architecture, and emerging technical risks. And perhaps most importantly, we need consistency. If two independent evaluators assess the same system against the same requirements, their conclusions should be reasonably consistent. Otherwise, what value does the certification actually provide? I worry that without rigor, we’ll end up with a fragmented market where organizations shop for the certification that gives them the answer they want, and you aren’t building trust with anyone that way.

There is a danger that certification becomes a trust symbol stripped of its scope and limitations. What should customers and the public be allowed to see about what was tested, what was excluded, what failed, and how long the result remains valid? How does certification build trust?

I’ve always believed that trust is built in drops and lost in buckets. And the fastest way to undermine trust in AI assurance is to create the impression that a certification means a system is universally safe. It doesn’t! We’ve seen this challenge in cybersecurity for years. Organizations obtain a SOC 2 report or an ISO certification, and sometimes the market interprets that as a guarantee that the organization cannot experience a security incident.

That’s never been the purpose of independent assurance. AI introduces additional complexity because the system can evolve, its behavior can vary, and the risks can change depending on how it’s deployed. Customers should have visibility into the scope of the assessment, the criteria used, the system or deployment covered, the limitations, and the period of validity. We also need to be thoughtful about how we communicate material findings without exposing sensitive information or creating additional security risks.

Ultimately, the credibility of a certification comes from the quality and independence of the underlying assessment, not the logo on the certificate. I don’t think this should be any different; if we’re going to build public trust in AI, we have to be disciplined about what assurance actually means.

When an independently assured system causes harm, what happens next (shutdown a production system supporting a mission-critical function)? Who is responsible then: the developer, deployer, systems integrator, auditor, or all of them—and what evidence would persuade you that the assurance process itself had failed?

 I think this question illustrates why accountability has to be established before AI systems are deployed, not after something goes wrong. As AI becomes more autonomous, we’re going to see situations where responsibility is distributed across multiple parties. The model developer built the underlying technology, another company integrated it, an enterprise deployed it, and an independent evaluator assessed certain aspects of it.

But by no means does that mean accountability disappears. Each party has responsibilities based on its role, and those responsibilities need to be clearly defined. I also don’t believe the answer to every AI incident is to immediately shut down the system. If an AI system supports critical infrastructure, healthcare, or financial services, shutting it down could create consequences of its own.

Organizations need predefined response plans, containment mechanisms, escalation procedures, and the ability to limit or revoke an agent’s permissions when necessary. From an assurance perspective, I would want to know whether the failure resulted from a risk that was within the scope of the assessment. Were the controls properly evaluated? Was sufficient evidence obtained? Did the evaluator identify weaknesses but fail to appropriately reflect them in the conclusion? Or did something materially change after the assessment?  These are all these small distinctions that matter.

I also think we need to be very clear about the difference between assurance and insurance. An independent assessment is not a guarantee that something will never fail. What assurance should provide is credible evidence that defined requirements were met, relevant controls were evaluated, and the organization had mechanisms to manage the risks. As we move toward increasingly autonomous AI systems, that distinction is only going to become more important. This is the larger conversation we need to have. We’re moving from technology that supports human decisions to technology that can act on our behalf. Again, the end result needs to be, our governance, accountability, and assurance models need to evolve just as quickly as the technology itself.

About Avani Desai, Chief Executive Officer of Schellman

Avani Desai is the Chief Executive Officer of Schellman, one of the world’s leading independent cybersecurity and technology assurance firms. Under her leadership, Schellman has expanded its global presence and emerged as a leader in assurance for emerging technologies, helping some of the world’s most innovative companies build trust in an increasingly complex digital economy. With more than two decades of experience spanning cybersecurity, risk management, operations, and business strategy, Avani brings a unique perspective to the intersection of technology, business, and trust. She is particularly focused on how artificial intelligence, quantum computing, digital sovereignty, and evolving regulatory expectations are reshaping the way organizations operate and demonstrate accountability.

Avani has helped position Schellman at the forefront of AI assurance, including becoming the first accredited AIUC-1 auditor and advancing independent assessments of AI systems. In 2026, Schellman performed more than 100 AI assurance engagements, supporting organizations ranging from frontier AI developers to enterprises deploying AI in critical business processes. Her work focuses on moving the conversation beyond AI principles and governance to independent testing, measurable outcomes, and evidence that technology operates as intended. A recognized industry voice, Avani has been featured in Forbes, The Wall Street Journal, and CIO.com. She speaks frequently about the future of technology assurance, AI governance, cybersecurity, global compliance, and the business implications of emerging technologies.

Beyond her professional responsibilities, Avani is committed to philanthropy, education, and expanding opportunities for women and underserved communities. Her board and community involvement have included Arnold Palmer Medical Center, Philanos, the Central Florida Foundation, and 100 Women Strong, where she has supported data-driven philanthropy focused on women and children. The daughter of immigrants, Avani believes deeply in the power of education, opportunity, and giving back. She is an avid marathon runner and a proud mother of three. She believes that leadership, much like running, is about consistency, resilience, and the willingness to keep moving forward.

For more serious insights on AI, click here.

For more serious insights on learning, click here.

Did you enjoy the Avani Desai interview? If so, like, share, or comment. Thank you!