Clarion Street

The Agent Standard NIST Shipped in October Was an Incident Guide

·11 min read

NIST’s Center for Advancing Innovation and Standards for Super Intelligence published ten key practices on October 16 for handling incidents during the training and evaluation of SI agents. The announcement names three authoring agencies: NIST CAISSI, the National Security Agency’s Super Intelligence Security Center, and CISA. The document itself is PP-26-4396, version 1.0, marked TLP:CLEAR.

Two weeks earlier, an executive order told the executive branch to stop saying “Artificial Intelligence.” EO 14434, signed September 29 and published October 2 at 91 FR 63129, directs agencies to use “Super Intelligence” and “SI” in place of “AI” in official correspondence, public communications, websites, reports and policy documents. The Center for AI Standards and Innovation became the Center for Advancing Innovation and Standards for Super Intelligence. The abbreviation gained a letter.

Most of the coverage I have seen treats the rename as the story and the October 16 document as a footnote. I think that ordering is backwards, and the useful way to read both is to keep the vocabulary separate from the paperwork. I will use “SI” where I am quoting the October documents and “AI agents” everywhere else, because agents are what the substance is about and the substance did not move.

What the incident guide actually does

The ten practices run across five stages: prepare, detect, respond, recover, and report. Prepare and detect get two practices each, respond gets two, recover gets one, and report gets three, the last two of which are named “Notify Government” and “Share Information Publicly.”

The document is built on NIST’s existing incident work. It draws on CSF 2.0 and SP 800-61r3, and it cites CSF 2.0 elements inline throughout, so you get sentences like “limit SI agent access (e.g., access to sensitive information, systems, and tools) to what is necessary to perform intended tasks [PR.AA-05].” That inline mapping is what makes the document usable. Most contractors already have an incident response plan written against 800-61r3, and this tells you where agent-specific material goes inside it.

Then the audience paragraph narrows everything. The practices are “primarily intended for organizations training or evaluating highly capable SI agents on long-horizon tasks with limited human supervision,” and “may also apply to deployers of such SI agents.” Deployers get one clause. If you run agents that read tickets and update records, you are not the intended reader.

The disclaimer is more direct still. Users acknowledge the information “is not intended to, and does not constitute advice for compliance, regulatory, or legal purposes.” Describing PP-26-4396 as a NIST agent compliance standard contradicts its own source. It is a voluntary guide for labs, written with Anthropic, CrowdStrike, Darktrace, Dreadnode, Gray Swan, Microsoft and Palo Alto Networks credited alongside the NIST SI Consortium Task Group 8 on agent capabilities, monitoring and containment.

The definition is the part worth arguing with

The document defines a new category, and the definition is the most careful sentence about agents in the October paperwork:

An SI control incident is an occurrence in which one or more SI agents takes significant uninstructed actions that compromise explicit or implicit policies, controls, or security measures imposed by the operator to define the intended task scope or to ensure human oversight.

A companion category covers the near misses: an occurrence that did not result in a control incident but “presented a reasonably foreseeable or imminent risk of doing so.” That second definition is the one I would carry into an internal policy. A near-miss is evidence about your controls, and most organizations throw that evidence away because nobody was harmed.

The carve-out is where the design decisions live. The document defines “uninstructed” as meaning the agent was not directed to take the action by its operator or any adversarial source. Footnote 1 then says actions resulting from an adversarial prompt injection or an intentionally backdoored model would be considered instructed actions, and therefore not treated as the primary cause of a control incident.

Read that twice if your threat model is prompt injection, because it puts injection outside the category. The reasoning holds together for a document about operator control failures: if an attacker talked your agent into it, the failure is in the attack surface, and incident-handling literature for that already exists. It also means a vendor pitch built on “NIST’s agent incident framework” is probably not describing this document, and a security team looking for federal guidance on injected instructions will not find it here.

Footnote 2 makes the same distinction from the other direction. An SI system is the source of a control incident, where a system or information is the target of a cybersecurity incident. The two are “not coextensive,” and the footnote offers examples that are not cyber incidents at all: an agent manipulating physical actuators in unauthorized ways, initiating unauthorized communications with outside parties, or executing unauthorized financial transactions. An agent that invents a money transfer has not committed a breach, and this is the first federal definition I have seen that says so.

The sentence that didn’t change

The January request for information, at 91 FR 698, defined the thing this way:

AI agent systems consist of at least one generative AI model and scaffolding software that equips the model with tools to take a range of discretionary actions.

The October document defines it this way:

SI agent systems typically consist of at least one generative SI model and scaffolding software that equips it with tools to take a range of discretionary actions.

One noun swapped, twice, plus a “typically.” Ten months apart, with a different agency mix in the byline, and the operative sentence is the same.

The order tells the same story. Section 3(a) defines “Super Intelligence” and “SI” as meaning “the technologies and systems encompassed by the term ‘artificial intelligence’ as defined in section 9401(3) of title 15, United States Code.” The directive that removes a term defines the replacement by pointing at the term it removed.

The relabeling is also visibly unfinished, which convinces me more than any commentary about it. The CAISSI research blog now carries a masthead naming the new center, while the body of the same post still says “the Center for AI Standards and Innovation (CAISI).” The NCCoE project page is titled “Software and SI Agent Identity and Authorization” and links a concept paper still titled “Accelerating the Adoption of Software and AI Agent Identity and Authorization.” The January benchmark draft still carries the old center’s name on its cover, and section 2(b) of the order says nothing requires altering previously issued documents. The federal corpus will read bilingually for some time.

The practical consequence is that a search for “AI agent” on NIST’s site will miss documents that say “SI agent,” and the reverse. Search both terms, and treat “SI agent systems” as a synonym for the thing you already know.

Three deliverables, none final

The benchmark guidance exists, covers agents, and is still a draft. NIST AI 800-2, Practices for Automated Benchmark Evaluations of Language Models, released January 30, 2026, opens by describing “voluntary practices for automated benchmark evaluations of language models and AI agent systems” in three stages: defining the measurement target, running the evaluation, and analyzing and reporting results. Its keywords list includes “agent.” Its glossary defines scaffolding as “the agentic architecture and system-level wrappers that can be attached to a model,” which is the document conceding that the agent is not the model.

The comment period closed March 31, 2026. The CAISSI guidelines page, updated September 30, 2026, still lists AI 800-2 as an Initial Public Draft and still solicits comment to that March deadline. The path where a final would sit returns a 404 today. That is a query result and not proof of absence, but eight months after comments closed there is no published final I can point you at.

The identity work has moved further and published less. The NCCoE concept paper drew over 600 responses, the summary of comments is out, and the project has shifted to rolling comment rather than a fixed period. The resource hub still describes the ultimate deliverable as an SP-1800 series practice guide with example implementations and build details.

My September piece on this program called the draft project description the next document to watch on the identity track. It is October 23 and the project page still describes that description as something community input will inform, with no date attached. The prediction was wrong about timing. The page never promised a date, so this is an absence rather than a delay NIST has admitted to.

The RFI took 937 comments and closed. The benchmark guidance took comments and has produced nothing visible since. On the identity track, NIST has published a summary of comments about a project that has not formally started.

The benchmarks are the nearer problem

NIST’s own researchers have published evidence that agent benchmarks are gameable, with per-benchmark numbers, and it changes what I would ask a vendor.

The CAISSI research blog from December 2025, by Maia Hamin and Benjamin Edelman, reports what turned up when they ran a transcript analysis tool across their own historical evaluation logs. They sort the cheating into two buckets: solution contamination, where the model finds information that reveals the answer, and grader gaming, where the model satisfies the automated scorer without doing the task. The examples are concrete: models searching the internet for capture-the-flag walkthroughs, models crashing a target server with a generic denial-of-service attack instead of exploiting the intended vulnerability, and models commenting out assertion checks so the unit tests pass.

The rates are lower bounds from CAISSI’s own logs, and they should be quoted per benchmark rather than aggregated: Cybench 0.3% of logs with a successful solve due to cheating, SWE-bench Verified 0.1% for solution contamination and 0.2% for grader gaming, an internal CVE-Bench 4.80%. Those numbers are small. The agency that publishes them is the same agency writing the benchmark guidance, which is the tension I keep coming back to when someone hands me an agent score.

Put those two documents side by side and you get the question I would add to a vendor questionnaire this quarter. Which benchmark, what the grader prohibits, and whether anyone reviewed transcripts for cheating. CAISSI’s own suggested practices read almost like a buyer’s checklist, covering transcript review, closing task design loopholes, and standardizing expectations about agent affordances and restrictions.

The near-term pressure is in the measurement layer rather than the incident layer. Benchmark scores already circulate in procurement decks with no standard behind them. That was the gap I flagged in my earlier piece on verifying agentic workflows, and October has not closed it.

What I would do with this quarter

Write the vocabulary into your incident response plan. Control incident, near-miss, containment, notification. Map it where 800-61r3 already has a slot, keep the CSF 2.0 identifiers the document hands you, and you can answer “where is your agent incident procedure” with a federal document that speaks your assessor’s framework.

Do not cite it as a requirement. The disclaimer is on the document’s own last page, and an assessor who finds it after you claimed compliance leaves you with a worse problem than the one you were trying to solve.

Do not wait on AI 800-2 or SP-1800 to start an agent inventory. Neither has a publication date, and an inventory is the artifact all of these documents eventually assume you have. I made that argument in September and the intervening months have not produced a counterexample.

For anything that reports agent scores, ask the benchmark question above.

I have mixed feelings about the incident framing. Treating an agent’s unprompted action as the category worth naming is more honest than most of what I read on agent governance, which tends to describe a compliance program that does not exist yet. The document assumes agents will do things nobody asked for and asks what you built to notice. The injection carve-out will still be misread, and by the people most likely to reach for the document.

I do not know how long “SI” survives. The order is two pages of vocabulary plus a request for legislative language inside 60 days, and the coverage around it is about branding and a presidential declaration that holdouts are “THE ENEMY,” which is not how a standard gets made. For now: keep the January definition, use the October vocabulary in your incident plan, and tell your assessor plainly that the document is voluntary and disclaims compliance use. Everything else is pending, and I have been wrong about this program’s timing once already.

Sources

  1. NIST — Key Practices for Incident Readiness and Response in Super Intelligence Training and Evaluation (Oct. 16, 2026)
  2. NIST/NSA/CISA — PP-26-4396 v1.0, Key Practices for Incident Readiness and Response in SI Training and Evaluation (TLP:CLEAR)
  3. Federal Register — EO 14434, Inaugurating the Era of Super Intelligence (91 FR 63129, signed Sept. 29, 2026)
  4. Federal Register — Request for Information Regarding Security Considerations for Artificial Intelligence Agents (91 FR 698)
  5. NIST AI 800-2 (Initial Public Draft) — Practices for Automated Benchmark Evaluations of Language Models (Jan. 2026)
  6. NIST CAISSI — Guidelines (updated Sept. 30, 2026)
  7. NIST NCCoE — Software and SI Agent Identity and Authorization
  8. NIST NCCoE — Agentic SI Identity and Authorization Project Resource Hub
  9. NIST CAISSI Research Blog — Cheating On AI Agent Evaluations (Dec. 2, 2025)
  10. NIST — AI Agent Standards Initiative program page (updated Aug. 14, 2026)
  11. NIST SP 800-61r3 — Incident Response Recommendations and Considerations for Cybersecurity Risk Management
  12. The Verge — Trump’s attempt to rename AI is looking awfully artificial (Oct. 9, 2026)