Clarion Street

Auditing the Unpredictable: The Compliance Gap in Agentic Infrastructure

·6 min read

Compliance assumes a fixed target. The way we certify software is to define a set of controls, check them on a schedule, and attest that the system was in a known state on a known date. CMMC, the Department of War’s framework for verifying contractor cybersecurity, is the cleanest example: a point-in-time assessment of NIST SP 800-171 controls, run annually at Level 1 and every three years at Level 2. The model is deterministic by design: a static control set, checked on a fixed cadence.

Agentic software is the opposite. OpenAI defines agentic AI systems as systems that can pursue complex goals with limited direct supervision. Limited supervision is not incidental to that definition. It is the part that breaks the audit model. A system that decides its own tool calls at runtime does not have a static control set you can check in advance, because there is no fixed “supposed to do” to check against.

The thing you would certify keeps moving

The reference architecture a compliance team would map controls onto is the one the vendor retired last quarter. LangChain, past 145,000 GitHub stars, has repositioned itself as “the agent engineering platform,” with LangSmith as its observability and evaluation layer. Microsoft’s AutoGen, past 60,000 stars, is now explicitly in maintenance mode, superseded by the Microsoft Agent Framework 1.0. CrewAI, past 57,000 stars, sells a commercial control plane, CrewAI AMP, that bundles observability, governance, security, and real-time tracing.

The supersession is not chaos for its own sake. MAF 1.0 promises stable APIs and long-term support, which is exactly what auditable infrastructure needs. But the churn still matters for compliance, because there is no stable artifact to assess.

What already governs AI was written for different software

None of the existing AI governance contemplates an agent. It is real, and it is substantial, but it was written for systems that sit still.

NIST’s AI Risk Management Framework, released in January 2023 and now under revision, is voluntary and organized around four continuous functions: Govern, Map, Measure, and Manage. Its companion Generative AI Profile, published July 2024, names four considerations: governance, content provenance, pre-deployment testing, and incident disclosure. The EU AI Act, in force since August 2024, is a binding product-safety model with risk tiers and fines up to 7% of worldwide turnover, but it is an ex-ante conformity regime. You prove compliance before deployment, not continuously at runtime. ISO/IEC 42001, the first AI management system standard, is a Plan-Do-Check-Act cycle that gets closest to continuous but operates at the policy level. OWASP’s GenAI LLM Top 10 shipped a 2026 edition in August.

What none of these answers is the question an auditor would actually ask about an agent: what did it do at 2:14 AM, and what was it allowed to do? NIST RMF is continuous but voluntary and policy-level. The EU AI Act is binding but ex-ante. ISO 42001 is a management system, not a runtime log. There is a gap between “we have an AI governance program” and “we can prove what this agent did.”

The stack forming to close the gap

The stack is being assembled from parts that were built for other problems. It has three pieces.

The first is observation. If you cannot predict what an agent will do, you can record what it did. LangSmith sells “agent evals, observability, and debugging.” CrewAI AMP bundles tracing and observability. Open Policy Agent, a CNCF-graduated policy engine, writes an audit trail for every policy decision.

The second is constraint, and this is the one I find more interesting. NVIDIA’s NeMo Guardrails offers “programmable guardrails” for LLM systems through a domain-specific language called Colang. Open Policy Agent now ships a worked example called “AI Tool Calling”: Rego policies that decide which tools an agent can call, what parameters are permitted, and how the tools may be used. The example denies Bash, Write, and Edit, and requires HTTPS-only WebFetch. That is guardrails as policy-as-code, and it is the first time I have seen tool access, rather than model output, treated as the thing to govern.

The third is a chokepoint. Anthropic open-sourced the Model Context Protocol in November 2024 as “a new standard for connecting AI assistants to the systems where data lives,” aimed at secure, two-way connections between agents and their tools. If every tool call has to pass through one standardized boundary, that boundary is where you enforce the policy and record what happened. MCP’s role here is narrow. It is the boundary, not the policy.

Taken together, observation, constraint, and a chokepoint form something I will call Agentic Compliance. The name is mine, not an industry term, and it names a convergence rather than a standard. Nobody has turned these three pieces into a framework an assessor would accept.

The honest gap

That last sentence is the honest core of the argument, and I want to be careful not to oversell it. Open Policy Agent, NeMo Guardrails, and LangSmith are developer tools. None of them is certified, mapped to a control framework, or accepted by any auditor as evidence that a control exists. There is real distance between “you can observe and constrain an agent” and “an assessor will accept that observation as proof.”

The gap is not confined to tooling. It is in the regulations themselves. CMMC, NIST RMF, the EU AI Act, and ISO 42001 are all pre-agentic in their framing. A reader could reasonably conclude the compliance gap is a regulatory gap, and no amount of Rego policy closes a regulation that does not contemplate the software you are running.

The timing sharpens the point. Even the deterministic regime is wobbling. In July 2026 the Department of War suspended third-party CMMC Phase 2 certification mid-rollout, and I wrote separately about what that means for contractors. With the point-in-time model itself under review, there is no clear path to certifying agentic software.

Bottom line

The compliance gap for agentic systems is structural, not cosmetic. Point-in-time assessment of a static control set cannot certify software that decides its own tool calls at runtime. What replaces it is not a better checklist. It is observation of what the agent did, constraint on what it could do, and a standardized boundary where both can be applied. Those pieces exist today as engineering tools. They have not become a certifiable framework, and until they do, the honest answer to “is your agent compliant” is that nobody can tell you yet, because the people who write the standards have not caught up with the software they are being asked to certify.

Sources

  1. Cybersecurity Maturity Model Certification — Wikipedia
  2. langchain-ai/langchain — GitHub
  3. microsoft/autogen — GitHub
  4. crewAIInc/crewAI — GitHub
  5. Practices for Governing Agentic AI Systems — OpenAI, December 14, 2023
  6. AI Risk Management Framework — NIST
  7. NIST AI 600-1: AI RMF Generative AI Profile — NIST, July 2024
  8. OWASP GenAI LLM Top 10 — GitHub
  9. NVIDIA NeMo Guardrails — GitHub
  10. Open Policy Agent
  11. Introducing the Model Context Protocol — Anthropic, November 25, 2024
  12. Artificial Intelligence Act — Wikipedia
  13. ISO/IEC 42001:2023 — AI management systems — ISO