Clarion Street

Observability Was Table Stakes. The Eval Layer Is the Next Open-Source Battle.

·8 min read

For a while the question about AI agents was whether they could do the thing. Sometime in early 2026 that stopped being the question. The question now is whether you can see what the agent did and measure whether it did it right. The first half is mostly solved. The second is not, and the gap between the two is where the next open-source fight is going to happen.

I can put a date on when this stopped being an opinion and became a ledger entry. On January 16, 2026, ClickHouse announced it had acquired Langfuse, which it called “the leading open-source platform for LLM observability, evaluations, and prompt management.” In the same announcement it closed a $400 million Series D at a $15 billion valuation led by Dragoneer, and Langfuse confirmed the deal in its own post. A database company spent part of a $400 million raise on an LLM observability company. That is a bet on the layer that decides whether anyone can trust an agent.

Two weeks later, on February 2, 2026, Gartner published a Market Guide for AI Evaluation and Observability Platforms. A new Gartner market guide is a lagging indicator. The market has to exist before an analyst firm names a category for it. The part worth noticing is that the category groups evaluation and observability into a single bucket, and its stated rationale is that nondeterminism in generative and agentic systems makes reliability hard to measure and user trust hard to earn. That grouping is the consolidation signal.

The deal, in numbers

Langfuse ended 2025 with more than 20,000 GitHub stars, more than 26 million SDK installs a month, more than 6 million Docker pulls, and adoption by 19 of the Fortune 50 and 63 of the Fortune 500, according to ClickHouse’s announcement and an Orrick note on the transaction. Langfuse already ran on ClickHouse underneath, which is why the two companies can describe the fit the way they do. Langfuse’s CEO, Marc Klingen, put it directly: “We built Langfuse on ClickHouse because LLM observability and evaluation is fundamentally a data problem.”

That framing is doing real work. ClickHouse is not a monitoring company. It is a database company. Its read of the deal is that watching an agent is not a dashboards problem but a data problem. Every trace, every tool call, every scored output is a row you need to store, query, and join against your own data. The acquisition announcement frames the shift as moving from systems observability to “AI quality monitoring,” and it names the gap plainly. Traditional observability tells you whether a service is up and fast. It “reveals nothing about whether your model’s outputs are correct, helpful, safe, or aligned with user intent.”

What the spread says

The sharpest number in this story does not come from either company. It comes from LangChain’s State of Agent Engineering survey, published in June 2026 with 1,340 respondents. Eighty-nine percent of organizations say they have implemented observability for their agents. Only 52.4 percent run offline evaluations, and 37.3 percent run online evaluations in production.

Nearly nine in ten can trace what an agent did. Barely a third can score whether what it did was good. Quality is the single biggest barrier to production, cited by about a third of respondents.

The gap between “we can trace it” and “we can score it” is the under-built half. Observability became table stakes quickly because a trace is legible. It is a data structure an SDK can ship in a weekend. An evaluation is a judgment, and judgments are harder to automate. They need datasets, rubrics, baselines, and someone to decide what “right” means for a given task. By my read, nobody has productized evaluation the way tracing got productized, and the 89-versus-37 spread is the evidence for it.

From vibe checks to gates

The industry has a name for how agents got shipped before evaluation existed: vibe-checking. You chat with the agent, it feels right, you ship it. ClickHouse’s announcement describes it as “essentially placing a bet on output quality without any way to measure it.” The corrective Google Cloud and Red Hat both describe is continuous, dataset-driven evaluation wired into the build.

Google Cloud’s writeup splits the work into two modes. Discovery mode raises the ceiling, using a human reviewer to find out what a model can do. Defense mode protects the floor, using dataset-driven, gated, automated evaluation so a change does not silently make things worse. Red Hat documents an eight-stage evaluation journey that ends in continuous monitoring. Both describe the same migration: evaluation stops being something a researcher does once and becomes something the CI pipeline does every time.

Langfuse’s stated goal after the acquisition is “closing the loop from production data to better prompts, evaluations, and product decisions,” with investment in “production monitoring and analytics for real agent systems (not just offline evals).” An eval that runs only before deploy is a snapshot. What closes the loop is production data flowing back into the datasets you score against.

The open-source question

For open-source developers the acquisition cuts two ways, and I want to hold both.

The comfortable reading is stability. ClickHouse says Langfuse stays MIT-licensed for its core features, stays self-hostable, and that Langfuse Cloud keeps running unchanged. Those are the right things to say, and there is reason to believe them, because ClickHouse has run this exact playbook before with PeerDB and HyperDX. Category-leading open-source projects built on ClickHouse, acquired, kept open.

The uncomfortable reading is stewardship. When the leading open-source tool in a category gets bought by a database company, the tool’s roadmap now serves that company’s database strategy. An MIT license protects you from the code being taken away. It does not protect you from the tool’s incentives being rewritten. Forrester’s mid-2026 read names the pressure: three-quarters of enterprise leaders say they are adopting agentic AI, few run it in meaningful production, and a “trust tax” applies because every autonomous action has to be logged and defensible to an auditor. That tax gets paid in tooling, and whoever owns the tooling collects it.

There is also a structural signal worth noticing. In Eric Roby’s 2026 agent stack diagram, evaluation and observability appear as a first-class vertical rail that cuts across every layer. Roby notes that Letta’s original map of the stack left eval out entirely. That is the shape of the consolidation. It is not one more box inside a layer. It is a column running through all of them.

The caveats

A few limits on what I am claiming.

The “stays open source, self-hostable, cloud unchanged” commitments come from ClickHouse’s and Langfuse’s own announcements. Treat them as vendor promises, not independent guarantees. The Orrick note confirms the transaction happened; it does not tell you what the licensing looks like two years out.

The survey number is the best public data I found, but it has an interest attached. The State of Agent Engineering survey is produced by LangChain, the maker of LangSmith, a competing observability and evaluation platform. Its “evals lag observability” framing lines up with a vendor selling eval tooling. I am using the numbers anyway, because they are the best public survey on the topic, but the source has skin in the game.

The Gartner “first” claim is softer than it sounds. That characterization comes from vendor press releases. Gartner’s public landing page for the guide exposes the summary I quoted above, but the full research behind it is paywalled, so I could verify only the title, the date, and that public summary.

There is a live community concern I want to flag without presenting it as settled. In the Hacker News discussion of the deal, commenters pointed out that ClickHouse is US-headquartered while Langfuse is a German GmbH, which raises GDPR and data-residency questions for Langfuse Cloud customers. That is community commentary, not an adjudicated outcome.

The numbers also wobble a little across ClickHouse’s two announcements, which give SDK install figures of 23.1 million and 26 million a month. I have used the more consistent figure, 26 million, and the more conservative star count of 20,000-plus.

Bottom line

The agent stack has a trust layer now, and it is consolidating around two halves that are not equally mature. Observability won. Evaluation has not, and the gap between them, 89 percent tracing against 37 percent scoring, is the most useful number I have seen on this in 2026.

For the open-source crowd, that gap is the opportunity and the risk at the same time. Whoever makes production evaluation as easy as tracing became will win the next round, and will be the next acquisition. For people already on Langfuse, the consolidation is a stability bet, and it holds only as long as the vendor commitments from January do. For everyone building the half that is still missing, it is a clear signal that the field is not settled.

Sources

  1. ClickHouse welcomes Langfuse — ClickHouse, January 16, 2026
  2. Langfuse joins ClickHouse — Langfuse, January 16, 2026
  3. ClickHouse raises $400M Series D, acquires Langfuse — ClickHouse, January 16, 2026
  4. Open-source LLM Observability: Langfuse Acquired by ClickHouse — Orrick
  5. State of Agent Engineering — LangChain, June 2026
  6. Market Guide for AI Evaluation and Observability Platforms — Gartner, February 2, 2026
  7. From Vibe Checks to Continuous Evaluation — Google Cloud, February 2026
  8. Eval-driven development: Build and evaluate reliable AI agents — Red Hat, March 2026
  9. The State of Agentic AI in 2026 — Forrester, June 2026
  10. The 2026 AI Agent Stack, Drawn from Scratch — Eric Roby, Brain Bytes
  11. Openlayer Recognized in 2026 Gartner Market Guide — PR Newswire
  12. ClickHouse acquires Langfuse — Hacker News