The Observability Layer Just Became an Agent

In September I wrote that the agent stack now has a trust layer, the observability and evaluation tooling that decides whether anyone can see and score what an agent did. That piece was about consolidation. ClickHouse bought Langfuse, Gartner named a new category, and the useful number was the spread between tracing, which nearly everyone does, and evaluation, which almost nobody does. My read at the time was that the trust layer is a dashboard you read after something goes wrong. You looked at traces, scored outputs, and shipped the fix yourself.

That read went stale on September 24, and I can name the date because it is when LangChain shipped LangSmith Engine v2. The change is not a better chart. It is that the observability layer now acts.

I want to be careful about what I am claiming before I make it. This is one vendor’s product, much of it in private beta, and every fact below comes from LangChain’s own blog and docs. The features are not independently benchmarked. But the shape of the thing is worth sitting with even with those caveats attached. The operator’s job is now done by software, and this is the cleanest example of it I have seen.

What Engine v2 does

LangSmith Engine is an agent. LangChain says so directly: it is “an in-platform agent that automates work at each step of the agent development lifecycle.” Since it launched in May it has analyzed more than 60 million traces and diagnosed tens of thousands of issues, by LangChain’s count. The first version scanned production traces for errors and unmet requests, grouped related traces into a single issue, and attached a root cause and a proposed fix.

Version two adds three behaviors. It red-teams your agent to find failures that have not happened yet. It flags trajectories that are wasteful rather than wrong, and tracks trends in error rate, latency, and cost. And it validates its own proposed fixes before showing them to you, so the fix you approve has already been tested against an eval set.

Red-teaming, but not the kind you are thinking of

The word is going to cause confusion, so I want to pull it apart early. When the rest of this site talks about red-teaming, it usually means adversarial security testing, the work I covered around kernel-level sandboxing and who investigates the machine. That is people trying to break a system on purpose. Engine v2’s red-teaming is narrower and closer to home. It is prompt-level bug and behavior hunting against your own agent, run automatically before the failures reach production.

As the docs describe it, Engine reviews recent production traces and, if you have connected one, your code repository, to learn what the agent is for and how it behaves. From that it forms hypotheses, each naming one specific way the agent could fail. It then sends synthetic requests to a deployment and judges how the agent responds. A confirmed finding comes from a probe sent during the run, not from an error in an old trace.

It tests ten issue classes, and the list is worth skimming because it reads like the checklist nobody has had time to write. Indirect injection, where retrieved content or tool output tries to steer the agent. Data exposure, where hidden context leaks into a response. Instruction hierarchy, where the agent’s own instructions get overridden. Tool safety, auth isolation, session integrity. The boring-but-real ones are there too: input robustness and reliability around retries and loops.

The detail I keep coming back to is the warning LangChain puts in the docs. Probes are real requests. Each one creates a run on your deployment, and your agent can call its tools while it responds. The docs tell you to point red-teaming at a staging deployment, where synthetic traffic and tool side effects are acceptable. That warning matters because the thing testing your agent can itself do things, and you have to decide where its side effects are allowed to land.

The operational half

The second new behavior is where the word “operate” shows up in its literal sense. Engine v2 does not just look for wrong answers. It looks for waste. It flags “inefficient agent trajectories,” which means repetitive tool calls and unnecessarily long paths, and it tracks trends in error rate, latency, and cost.

This matters because it is the work that used to require a human watching a dashboard over time. An agent that answers correctly but calls the same tool three times when it needs one is not failing an eval. It is quietly burning tokens and latency, and nobody notices until the bill or the p95 latency graph moves. Engine v2 watches for that drift and turns it into a queued issue with a proposed alternative.

The view that makes this possible is Trajectories, which shipped the same week. A trajectory flattens a long agent session, every human, AI, and tool message across the main agent and any subagents, into a single chronological read. The point is that a reviewer can find where behavior diverged without descending into the full nested trace. And Trajectories do double duty. They can be scored by online evaluators, and they can be exported as supervised fine-tuning data through LangChain’s smithtune tooling. Observability output flows back into the model that produced it. That is a second loop, separate from the fix loop, and it is the clearest sign that observability has stopped being read-only.

The fix loop

The third behavior is the one that closes the loop to a pull request. For agents running on LangSmith Deployment, Engine v2 does not just hand you a proposed fix and wish you luck. It reproduces the failure by running the offending inputs against your agent. It proposes a change, tests it against an eval set, evaluates the result, and adjusts. Only when it confirms the fix resolves the issue does it surface it, with an option to open a PR with a click.

LangChain frames this as the answer to a real bind. Shipping a fix untested risks a bad customer experience. Testing it manually slows development down. The previous version made you pick one. Engine v2 does the offline test for you.

I have mixed feelings about this one, and I think the honest response is both. On one hand, a fix that has been run against a regression eval before it reaches you is genuinely useful, and it is the first time I have seen the full detect, diagnose, propose, test, adjust loop automated by the vendor’s own tooling. On the other hand, the tool proposing the fix is the same vendor’s product that defined the eval set and the benchmark. LangChain says Engine v2 is “more than 2x better” at detecting issues, measured on its own IssueBench, and produces fixes that are “25% more effective,” measured on Terminal-Bench. Those are LangChain’s numbers about LangChain’s product, and the loop that validates your fix is also the loop that sells you the platform.

What this means

Back in September, the trust layer was something you read. The gap the industry was trying to close was between tracing, which everyone does, and evaluation, which almost nobody does. Engine v2 does not close that gap so much as move it. The dashboard did not get smarter. It got replaced by a second agent whose whole job is to watch the first one, find what it would fail at, and hand back a fix that has already been tested.

There is a version of this that is pure marketing, and I want to avoid it. “Observability became an operator” is a clean line and it is also a vendor’s framing, and the features are private beta with no independent measurement. The version I actually believe is narrower. A specific piece of operator work, finding failures before they happen, spotting waste, and pre-testing fixes, is now a product you can turn on. Whether it holds up outside LangChain’s own benchmarks is an open question, and the fact that the testing agent can itself make tool calls is a new failure mode to think about, not just a new feature to adopt.

Still, the direction is not in doubt. When I wrote about the accountability gap after OpenAI’s rogue swarms, the recurring problem was that no one has the power or the tooling to look at what an agent did after the fact. Engine v2 is a different answer to a related question. It does not investigate after the fact. It runs before the fact, on your own agent, on your own terms, and hands you a validated change. That is the operator’s job, done by software, and it arrived faster than most of the governance machinery that was supposed to do the same work with humans in the loop.

Caveats

Everything in this post comes from LangChain’s own blog and docs, and the two headline features, red-teaming and automated fix validation, are both in private beta for LangSmith Deployment users. I have not seen independent benchmarks of either, and the “2x better detection” and “25% more effective fixes” figures are measured on LangChain’s own IssueBench and Terminal-Bench.

There is a small discrepancy I should flag rather than smooth over. The Engine v2 announcement says Engine has analyzed “more than 70M traces,” while the same week’s roundup says “more than 60M traces.” I have used the lower figure. It is a rounding-level disagreement, but it is the kind of thing that tells you the number is a marketing figure, not a measurement.

The red-teaming here is not the security red-teaming the rest of the site discusses, and conflating the two would be a mistake. It hunts prompt-level bugs and behavior drift on a staging deployment. It is not an adversarial security audit, and LangChain’s own docs warn that its probes have real tool side effects, which is exactly why you point it at staging.

Sources

1. LangSmith Engine v2: Red teaming and automated testing — LangChain, September 24, 2026
2. New in LangSmith: Engine v2, Managed Deep Agents, Fine-Tuning, and More — LangChain, September 25, 2026
3. Trajectories now in LangSmith: A readable view of every agent session — LangChain, September 24, 2026
4. Find and fix your agent’s issues with LangSmith Engine — LangChain Docs