BIP America

collapse
Home / Daily News Analysis / Marinela Profi, SAS: On governing autonomous AI agents

Marinela Profi, SAS: On governing autonomous AI agents

Oct 11, 2026  Twila Rosenbaum  73 views
Marinela Profi, SAS: On governing autonomous AI agents

Why autonomous AI agents demand a new governance playbook

Autonomous AI agents are software systems that can plan, reason, and take actions to achieve goals with minimal human intervention. Unlike traditional AI models that produce a single output, agents can chain together tools, query databases, call APIs, and interact with other systems. This shift from prediction to action is what makes them powerful and perilous. In a recent discussion, Marinela Profi of SAS said that governing these agents requires a different approach from governing static models. The reason is simple: an agent's behavior emerges from its interactions with its environment, not just from its training data. That makes pre-deployment testing necessary but insufficient. Organizations need continuous oversight, clear escalation paths, and the ability to intervene when an agent goes off track.

Profi notes that the enterprise conversation about AI governance has matured rapidly. A few years ago, many companies were focused on model accuracy and bias. Today, they are asking harder questions about autonomy. Who is responsible when an agent negotiates a contract, approves a loan, or orders inventory? How do you audit a decision that was made through a chain of tool calls? What happens when an agent learns from its own actions and drifts from its original objective? These questions do not have easy answers, but they must be addressed before agents are given real authority.

The building blocks of agent governance

According to Profi, effective governance for autonomous AI agents rests on several pillars. The first is a clear policy framework that defines what agents are allowed to do, what they must never do, and who can authorize exceptions. Policies should be written in plain language and translated into machine-readable rules where possible. They should cover data access, financial limits, customer interactions, and safety-critical operations. Without such guardrails, agents may optimize for the wrong objective or take shortcuts that violate organizational values.

The second pillar is risk assessment. Not all agents carry the same risk. A customer service agent that drafts email responses is different from an agent that can execute trades or control industrial equipment. Profi suggests classifying agents by autonomy level, potential impact, and reversibility of actions. High-risk agents should face stricter controls, including mandatory human approval for certain actions, real-time monitoring, and rollback capabilities. Lower-risk agents can operate with lighter oversight, but they still need logging and periodic review.

The third pillar is human oversight. Profi emphasizes that human-in-the-loop is not a binary choice. It is a spectrum. At one end, a human approves every action. At the other, a human monitors aggregate performance and intervenes only when anomalies appear. The right point on the spectrum depends on context. For example, an agent that schedules meetings can operate with minimal oversight. An agent that summarizes medical records should have a clinician review its output. An agent that moves money should have transaction limits and dual authorization. The key is to design oversight that is meaningful, not performative.

Transparency, auditability, and explainability

Autonomous agents are often opaque because they combine multiple models, tools, and memory systems. Profi argues that transparency must be engineered into the agent architecture. Every action should be logged with a timestamp, the inputs that triggered it, the reasoning step if available, and the outcome. Audit trails should be immutable and searchable. This is not just for compliance. It is for learning and improvement. When an agent fails, teams need to reconstruct what happened. Without detailed logs, root cause analysis becomes guesswork.

Explainability is another challenge. Traditional explainable AI techniques work for single model predictions. They do not easily extend to multi-step agent behavior. Profi suggests that organizations adopt a layered approach. At the model level, use interpretable models where possible and post-hoc explanations where necessary. At the agent level, require agents to produce a rationale for each action. At the system level, build dashboards that show agent activity, success rates, and exceptions. This combination gives human overseers the context they need to trust or challenge the agent.

Security and adversarial risks

Agents that can call APIs and execute code introduce new attack surfaces. A malicious actor could manipulate an agent through prompt injection, data poisoning, or tool exploitation. Profi warns that security teams must be involved from the start. Agents should follow the principle of least privilege. They should only have access to the tools and data they need for their assigned tasks. Credentials should be rotated frequently. Tool calls should be validated against a whitelist. Anomaly detection should flag unusual patterns, such as an agent accessing a database at an odd hour or attempting to escalate its permissions.

Red teaming is essential. Organizations should simulate attacks where adversaries try to trick agents into revealing sensitive information, performing unauthorized actions, or disrupting operations. These exercises reveal weaknesses that standard testing misses. Profi notes that red teaming should be continuous, not a one-time event, because agents and their environments change over time.

Regulatory and standards landscape

The regulatory environment for AI is evolving quickly. The European Union's AI Act introduces risk-based obligations for AI systems, including transparency and human oversight requirements. The NIST AI Risk Management Framework provides a voluntary structure for managing AI risks. ISO standards are emerging for AI governance and management systems. Profi advises organizations to map their agent governance program to these frameworks. Doing so not only supports compliance but also builds confidence with customers and partners.

However, Profi cautions against treating compliance as the finish line. Regulations often lag behind technology. Organizations that wait for perfect rules will either move too slowly or take unnecessary risks. Instead, they should adopt a principles-based approach that emphasizes accountability, fairness, safety, and contestability. These principles can guide decisions even when specific rules are unclear.

Operationalizing governance in the enterprise

Governance is not just a set of documents. It is a set of practices embedded in the development and operation of agents. Profi recommends establishing an AI governance council with representatives from legal, compliance, security, data science, engineering, and business units. This council should review high-risk agents before deployment, set policy, and adjudicate disputes. It should also have the authority to pause or shut down an agent that behaves unexpectedly.

Development teams need tools to implement governance. These include model registries, policy engines, monitoring platforms, and audit log systems. SAS, where Profi works, has been developing analytics and AI governance capabilities that help organizations manage the lifecycle of models and agents. The goal is to make governance a seamless part of the MLOps and LLMOps pipelines, not a separate bureaucratic layer.

Training and culture are equally important. Employees who build, deploy, or oversee agents need to understand the risks. They should know how to report concerns, how to interpret agent logs, and how to intervene safely. Profi suggests creating clear playbooks for common failure scenarios. For example, if an agent starts making unauthorized purchases, the playbook might require immediately revoking its credentials, notifying the security team, and conducting a forensic review.

Balancing innovation and control

The tension between innovation and control is real. If governance is too heavy, teams cannot experiment. If it is too light, agents can cause harm. Profi argues for a staged approach. Start with low-risk pilots in sandboxed environments. Measure performance, document lessons, and gradually expand autonomy as trust is earned. Use feature flags to enable or disable agent capabilities without redeploying the entire system. Build feedback loops that allow human overseers to correct the agent in real time.

This staged approach also helps with change management. Stakeholders are more likely to accept autonomous agents when they see that safeguards are working. Customers and regulators are more likely to trust organizations that can demonstrate control. Profi points out that trust is not a one-time achievement. It is maintained through consistent transparency, responsiveness, and accountability.

The road ahead

Autonomous AI agents will become more capable and more pervasive. They will handle more complex tasks, interact with more systems, and make more decisions. This will create enormous value, but it will also create new governance challenges. Profi believes that the organizations that succeed will be those that treat governance as a competitive advantage. They will be able to deploy agents faster because they have the controls in place to manage risk. They will be able to scale because they have the oversight to maintain quality. They will be able to innovate because they have the confidence to experiment within safe boundaries.

The governance of autonomous AI agents is not a problem to be solved once. It is a capability to be developed continuously. It requires technical tools, organizational processes, and a culture of responsibility. As Profi puts it, the question is not whether to govern agents, but how to govern them well enough to unlock their potential without losing control. The next phase of enterprise AI will be defined by how well organizations can answer that question.


Source: AI News News


Share:

Your experience on this site will be improved by allowing cookies Cookie Policy