Photo By: charlesdeluvio
As artificial intelligence systems become increasingly capable and autonomous, organizations face a new kind of risk: not simply whether an AI system will make a mistake, but whether its behavior may change in ways that are difficult for humans to anticipate.
For Melissa Cohoe, Global Strategist for Security, Risk & Resilience at NewRocket, the concern extends beyond the growing discussion around AI deception, manipulation and attempts to circumvent rules. Focusing too narrowly on those behaviors, she argues, risks overlooking the broader challenge.
“The root of the issue is that though AI was built by humans and is being used to replace or augment humans, it does not make decisions in a way that is entirely predictable by humans,” Cohoe says.
That unpredictability is becoming increasingly important as organizations move AI from experimental deployments into systems capable of making decisions and taking actions with greater autonomy.
Humans naturally tend to interpret AI behavior through a human lens, assigning motivations such as deception or manipulation to actions that may have emerged simply because the system was pursuing a particular objective. But AI does not possess the same context or lived experience as a person, making human assumptions about how it will behave potentially unreliable.
For organizations, that means AI oversight needs to evolve alongside the technology.
Cohoe advocates monitoring decisions, escalating those that appear to fall outside defined norms, auditing decisions and providing continuous assurance of an AI agent’s behavior. In other words, organizations need to treat AI as something that sits somewhere between traditional technology and human behavior when it comes to risk management.
That becomes particularly important when considering the traditional security principle of “trust, but verify.”
The approach assumes that once a technology has been tested and verified, its behavior can generally be trusted until the next review cycle. AI complicates that model because its behavior can change as data, models and operating environments change. Systems may also behave differently when they recognize they are being evaluated.
“Point-in-time verification can only be one benchmark,” Cohoe says.
Instead, organizations need to ask a broader question: not simply whether an AI system produced the desired result during testing, but how likely it is to continue producing that result across different operating scenarios.
That means looking for signals that could indicate covert or manipulative behavior, evaluation awareness, benchmark gaming, attempts to appear aligned with testers or what Cohoe describes as “strategic compliance.” At the same time, organizations still need to account for more familiar AI problems, including hallucinations and sycophancy.
The implication is a shift from periodic testing toward continuous behavioral monitoring.
Organizations need to establish whether an AI system continues to behave as it did when it was initially tested—and whether that behavior remains appropriate as the surrounding environment changes.
“Trust, but verify, is still true,” Cohoe says, “but with AI, trust must be continuously earned through ongoing observation, evidence, assurance, and intervention.”
That shift also changes how organizations should think about AI deployment itself. Cohoe argues that technology decisions have always been risk decisions, even when organizations have not explicitly described them that way. Choosing a particular system, for example, inherently involves weighing the likelihood and potential impact of undesirable outcomes.
AI, however, makes the case for making those risk decisions more explicit.
Rather than waiting for sophisticated risk-management frameworks to emerge, Cohoe recommends organizations begin by making implicit decisions visible. One starting point is to define risk in terms of its potential impact—whether financial, regulatory, safety-related or related to customers.
The next step is ownership.
Organizations need to establish who owns a particular risk, who is accountable for the resulting outcomes and who determines what level of risk is acceptable. As organizations become more mature, those responsibilities may be distributed among operational owners, risk teams, committees and executive leadership.
Finally, organizations need to define expectations: what each role is responsible for and what is expected at different levels of risk.
It may sound like a significant undertaking, particularly for organizations still developing their AI governance capabilities. But Cohoe argues that the alternative is more problematic: leaving existing risks undefined does not make them disappear.
“Something is better than nothing,” she says.
That may ultimately be the central challenge for businesses entering the next phase of AI adoption. The question is no longer simply whether an AI system can perform a task successfully. It is whether an organization can continuously understand, monitor and manage the risks created by a system whose behavior may evolve beyond the conditions under which it was originally tested.
As AI becomes more autonomous, trust is unlikely to be something organizations establish once and then retain. Instead, it may become an ongoing process—one that depends on evidence, observation and the willingness to intervene when an AI system behaves outside the boundaries its human operators consider acceptable.