AI agents are no longer prototypes. They query databases, draft contracts, route transactions, and trigger downstream systems — often without a human reviewing each step. That operational reality creates a governance problem that most compliance programs were not designed to handle.
This page is the reference point for Directors and VPs of AI Risk, Responsible AI, and Information Governance who are building or evaluating a governance program, and for CTOs at regulated organizations who are deploying agents and need controls that hold up under audit.
It covers what AI agent governance actually means, which regulations are forcing the issue, what a defensible program looks like component by component, and where most organizations fail before they get there.
What is AI agent governance
AI agent governance is the set of policies, technical controls, and accountability structures that determine how AI agents are authorized to act, what they are permitted to do, how their actions are recorded, and who is responsible when something goes wrong.
It is distinct from AI model governance, which focuses on how a model was trained, evaluated, and deployed. Agent governance focuses on runtime behavior: what an agent does once it is running in your environment, against your data, on behalf of your organization.
Why agents require different controls than models
A model produces an output. An agent takes an action. That distinction matters enormously for governance.
When a model generates a response, a human typically reviews it before anything happens. When an agent executes a workflow — querying a database, calling an API, sending a message, updating a record — the action is already complete by the time anyone looks at a log. The blast radius of a misconfigured or misbehaving agent is therefore much larger than the blast radius of a misconfigured model.
Agents also chain actions together. A single agent invocation can trigger dozens of downstream steps across multiple systems. Each step is a potential compliance event. Each step needs to be attributable, bounded, and auditable.
Traditional model governance frameworks — bias testing, accuracy benchmarks, model cards — do not address any of this. You need a separate control layer.
The governance gap most organizations hit
The pattern is consistent across regulated mid-market organizations: a team deploys one to five AI agents to automate real workflows, the agents work, adoption spreads, and then someone in legal, compliance, or risk asks a question that nobody can answer.
Which agents have access to customer data? What did agent X do last Tuesday? Who approved that workflow? How much did it cost and which model ran it?
That moment — the governance wall — is where most organizations realize they need a structured program rather than ad hoc controls bolted onto individual deployments.
The regulatory landscape driving AI agent governance
AI agent governance is not a best-practice exercise. For organizations in regulated verticals, it is increasingly a compliance requirement with specific attestation and documentation obligations.
CMMC L2 and DFARS 252.204-7021
Defense Industrial Base contractors subject to CMMC Level 2 must demonstrate that systems processing Controlled Unclassified Information (CUI) have documented access controls, audit logging, and configuration management. AI agents that touch CUI — and many do, because they are deployed precisely to process and route sensitive information — fall inside that perimeter. DFARS 252.204-7021 requires contractors to have a current CMMC certification as a condition of contract award. An agent that processes CUI without documented controls is a certification gap.
SEC 2026 AI exam priorities
The SEC has signaled that AI governance will be an examination priority for registered investment advisers and broker-dealers. Examiners are expected to look at whether firms have documented the AI systems they use, how those systems are supervised, and whether disclosures to clients are accurate. Agents that execute or influence investment-related workflows are directly in scope.
NYDFS AI guidance
The New York Department of Financial Services has issued guidance requiring covered entities to address AI risk within their existing risk management frameworks. That includes third-party AI systems and automated decision-making tools. Agents that make or influence credit, underwriting, or fraud decisions require documented controls and governance oversight.
NDAA §1513 and Defense Industrial Base requirements
Section 1513 of the National Defense Authorization Act directs the Department of Defense to develop guidance on AI use in defense contexts. For DIB contractors, this creates downstream pressure to demonstrate that AI systems — including agents — meet documentation and accountability standards consistent with DoD AI ethics principles.
10-K AI risk disclosure and cyber insurance AI security riders
Public companies are now disclosing AI risk in 10-K filings. The quality of those disclosures depends on whether the organization actually knows what AI systems it operates and what controls govern them. Without an agent inventory and a documented governance program, the disclosure is either incomplete or inaccurate — both of which create liability.
Cyber insurance carriers are adding AI Security Riders to policies that require attestation of specific controls around AI system access, logging, and incident response. Organizations that cannot attest to those controls face coverage gaps or premium increases.
Core components of an AI agent governance program
A defensible AI agent governance program has six components. Each one addresses a specific failure mode.
Agent inventory and classification
You cannot govern what you have not catalogued. The first requirement is a complete, current inventory of every AI agent running in your environment: what it does, what data it accesses, what systems it can modify, who deployed it, and what model or models it uses.
Inventory without classification is incomplete. Each agent should be assigned a risk tier based on the sensitivity of the data it touches, the reversibility of the actions it takes, and the regulatory perimeter it operates within. A tier-one agent that processes CUI or customer financial data requires more stringent controls than a tier-three agent that summarizes internal meeting notes.
Pre-dispatch controls and spend caps
Pre-dispatch controls are rules that evaluate an agent’s intended action before it executes. They are the governance equivalent of a four-eyes check — except they operate at machine speed.
Spend caps are a specific and critical form of pre-dispatch control. They define the maximum cost an agent is permitted to incur in a single step or session. A cap that holds a step before it overspends prevents runaway inference costs and creates a natural circuit breaker for misbehaving agents. Without spend caps, a misconfigured agent can generate thousands of dollars in model costs before anyone notices.
Pre-dispatch controls can also enforce permission boundaries — blocking an agent from accessing a data source it was not explicitly authorized to use — and flag actions that require human review before proceeding.
Signed audit trails
Every action an agent takes should produce a signed, tamper-evident log entry that records what the agent did, which model it used, what the input and output were, what it cost, and when it happened. Signed means cryptographically attested — not just a database row that an administrator could edit.
Audit trails serve two purposes. First, they enable incident investigation: when something goes wrong, you can reconstruct exactly what happened. Second, they satisfy regulatory documentation requirements. An examiner asking for evidence of AI system supervision needs to see a log, not a policy document.
Access and permission scoping
Agents should operate under the principle of least privilege. An agent authorized to read customer records should not have write access. An agent authorized to query one database should not have credentials to query another. Permission scoping should be defined at deployment time and enforced at runtime — not assumed based on the credentials of the human who deployed the agent.
This is an area where many organizations have significant gaps. Agents are often deployed with broad credentials because it is easier, and the permissions are never reviewed after initial deployment.
Human-in-the-loop checkpoints
Not every agent action should be fully automated. A governance program defines which action types require human review before execution, which require human review after execution, and which can proceed without review. Those thresholds should be documented, enforced technically, and reviewed periodically.
Human-in-the-loop checkpoints are particularly important for actions that are irreversible (sending an external communication, executing a financial transaction), high-stakes (decisions that affect a customer’s account status), or novel (action types the agent has not previously taken in production).
Cost attribution and model accountability
Every agent action has a cost: the inference cost of the model call, the compute cost of the workflow execution, and potentially the cost of downstream API calls. A governance program attributes those costs to the specific agent, workflow, and business unit that incurred them.
Cost attribution serves compliance purposes — demonstrating that AI spending is authorized and controlled — and operational purposes — identifying agents that are consuming disproportionate resources. Model accountability means knowing which model version ran a given step, so that if a model is updated or deprecated, you can assess the impact on governed workflows.
Building your governance architecture: a step-by-step approach
Step 1: Map every agent in your environment
Start with a discovery exercise. Interview engineering teams, review cloud provider logs, and audit API keys and service accounts. The goal is a complete list of every agent or automated AI workflow running in production, staging, or pilot. Include agents built on third-party platforms, not just those built in-house.
Document for each agent: the business owner, the deployment date, the data sources it accesses, the systems it can modify, and the model or models it uses.
Step 2: Classify by risk tier
Apply your classification rubric to each agent in the inventory. A simple three-tier system works for most organizations:
- Tier 1: Accesses regulated data (CUI, PII, financial records), takes irreversible actions, or operates in a regulated workflow. Requires full pre-dispatch controls, signed audit trail, and periodic human review.
- Tier 2: Accesses internal data, takes reversible actions, operates in internal workflows. Requires audit trail and defined permission scope.
- Tier 3: Accesses only public or anonymized data, takes no external actions. Requires basic logging.
Step 3: Define pre-dispatch rules
For each tier, define the rules that govern what an agent is permitted to do before it does it. At minimum, Tier 1 agents should have spend caps, permission boundary checks, and human-in-the-loop triggers for defined action types. Document the rules, implement them technically, and test them before the agent goes to production.
Step 4: Instrument the audit trail
Implement logging at the agent execution layer, not just at the application layer. Each log entry should capture the agent ID, the workflow step, the model used, the input hash, the output hash, the cost, the timestamp, and the authorization context. Sign each entry. Store logs in a system that is separate from the agent’s operational environment and that restricts modification.
Step 5: Assign ownership and review cadence
Every agent in the inventory should have a named business owner and a named technical owner. Define a review cadence — quarterly for Tier 1, semi-annually for Tier 2 — at which the owner confirms that the agent’s permissions, rules, and documentation are current. Build the review into your existing risk management calendar.
Common governance failures and how to prevent them
Failure: the shadow agent. An engineer deploys an agent using a personal API key. It never appears in the inventory. It processes customer data for six months before anyone knows it exists. Prevention: require all AI API keys to be provisioned through a central system that logs the associated agent and workflow.
Failure: the permission creep. An agent is deployed with read access to one database. Over time, its permissions are expanded to accommodate new use cases. Nobody reviews the cumulative permission set. Prevention: enforce permission reviews at each change and at each periodic review cycle.
Failure: the unsigned log. Audit logs exist but are stored in a mutable database. During an incident investigation, it is impossible to confirm that the logs have not been altered. Prevention: implement cryptographic signing at the point of log creation.
Failure: the uncapped agent. An agent is deployed without a spend cap. A prompt injection attack causes it to run thousands of model calls in an hour. The bill arrives before anyone notices the behavior. Prevention: require spend caps as a deployment prerequisite for all Tier 1 and Tier 2 agents.
Failure: the undocumented model update. A third-party model provider updates the model version an agent uses. The agent’s behavior changes. Nobody knows which model version ran the steps that are now under review. Prevention: log the specific model version for every agent invocation, not just the model name.
AI agent governance for the Champion: running the internal evaluation
If you are a Director or VP of AI Risk, Responsible AI, or Information Governance, you are likely the person who identified the governance gap and now needs to build the case internally for a structured program — and potentially for tooling to support it.
The internal evaluation has three phases.
Phase 1: Establish the risk surface. Use the inventory and classification process described above to produce a documented picture of every agent in your environment and its risk tier. This document is the foundation of your internal case. It makes the risk concrete and attributable.
Phase 2: Map to regulatory obligations. For each Tier 1 agent, identify the specific regulatory requirements it implicates — CMMC, SEC, NYDFS, or others relevant to your organization. Document the gap between current controls and required controls. This converts a governance conversation into a compliance conversation, which is easier to resource.
Phase 3: Define the target state. Specify what a compliant program looks like: the controls required, the tooling needed, the ownership model, and the review cadence. Present the target state alongside the gap analysis and a proposed timeline. The ask to leadership is not abstract — it is a specific program with a specific cost and a specific risk reduction.
The economic buyer — your CCO, CRO, or CISO — needs to see regulatory exposure quantified and a credible path to closing it. The technical buyer — your CTO or engineering lead — needs to see that the controls are implementable without blocking the teams that are building with agents.
AI agent governance for the CTO: moving fast without losing control
If you are a CTO at a regulated organization under $50M in revenue, you are probably the person who approved the first agent deployment and the person who will be asked to attest to AI controls in a compliance review. You need governance that does not slow down your engineering team.
The practical approach is to build governance into the deployment pipeline rather than layering it on afterward.
Make the inventory automatic. Every agent deployment should register itself in a central catalog as part of the deployment process. If registration is a manual step, it will be skipped.
Make spend caps a deployment default. Set a conservative default cap for all new agents. Teams can request a higher cap through a documented approval process. This creates a forcing function for cost review without blocking deployment.
Make the audit trail infrastructure, not application code. Logging should happen at the infrastructure layer — the agent runtime or the API gateway — not inside each agent’s application code. Application-layer logging is inconsistent and easy to omit.
Make permission scoping a template. Provide engineering teams with permission templates for common agent types. A template for a data-read agent, a template for a data-write agent, a template for an external-communication agent. Teams use the template; the template enforces least privilege.
The goal is a governance architecture that a team of two or three engineers can maintain and that produces the documentation a compliance review requires without a separate compliance engineering effort.
Frequently asked questions
What is the difference between AI governance and AI agent governance?
AI governance is the broader discipline covering how an organization develops, deploys, and oversees AI systems. AI agent governance is a specific subset focused on the runtime behavior of agents — systems that take actions autonomously. Agent governance requires controls that traditional AI governance frameworks do not address, particularly around pre-dispatch authorization, action logging, and cost attribution.
Which regulations explicitly require AI agent governance?
No regulation currently uses the term “AI agent governance” explicitly. However, CMMC L2, SEC examination guidance, NYDFS AI guidance, and cyber insurance AI Security Riders all impose requirements — around access controls, audit logging, and risk documentation — that apply directly to AI agents operating in regulated environments. The absence of agent-specific language does not create an absence of obligation.
How do I prioritize governance controls when I have limited resources?
Start with the agents that touch regulated data or take irreversible actions. Apply Tier 1 controls to those agents first. Tier 2 and Tier 3 agents can follow on a longer timeline. The risk-tiered approach lets you demonstrate a credible governance program to regulators and auditors while managing implementation cost.
What does a signed audit trail actually require technically?
At minimum: a log entry for each agent action that includes a cryptographic hash of the entry content, signed with a key that the agent runtime does not control. The signing key should be managed by a separate system — a key management service — and the logs should be stored in a write-once or append-only store. This makes it possible to verify that a log entry has not been altered after the fact.
How often should agent governance controls be reviewed?
Tier 1 agents: quarterly. Tier 2 agents: semi-annually. Any agent: immediately after a significant change to the agent’s permissions, data sources, or workflow logic, and immediately after a model version change by the underlying provider.
AI agent governance is not a one-time project. It is an ongoing program that needs to keep pace with the rate at which your organization deploys agents and the rate at which regulators clarify their expectations. The organizations that build the program now — inventory, controls, audit trail, ownership — will be in a materially better position when the next examination cycle or insurance renewal arrives.