The Serving Layer Is Learning What a Session Is

Inference serving has spent most of its life optimizing one shape: a request comes in, tokens come out, the next request arrives. The router spreads requests across replicas to balance load. It does not remember which replica handled a request before, because it assumes requests have nothing to do with each other.

That assumption was always an approximation. Even a plain multi-turn chatbot carries session context, and that context sits in the KV cache on the replica that processed the earlier turns. For a chatbot the approximation was cheap, because turns were short and a cold replica could rebuild the context without much pain.

Agents make the approximation expensive. An agent session runs for minutes or hours and makes dozens of requests, each depending on the context the earlier turns built. Send turn thirty to a replica that never saw the first twenty-nine, and that replica has to recompute them before it can answer. The cost is the difference between keeping a session warm and paying to rebuild it on every turn.

AIBrix v0.8.0, the Kubernetes-native inference stack in the vLLM project, shipped on October 12 with a headline feature that names the problem: stateful routing for agentic workloads.

Pinning a session to its cache

The mechanics are easy to describe and hard to get right. An application attaches a session identifier to a request. The gateway looks up that session and sends the request to the backend replica it is bound to, the one that already holds the session’s KV-cache and prefix-cache state. The binding is stored in Redis, so it holds even when later turns arrive through a different gateway replica.

This is not a new idea. Load balancers have done sticky sessions for decades, keeping a browser pinned to the server that holds its session. What is new is what the pin preserves and what the router weighs when it picks a replica.

Losing locality now costs more, because agent contexts are long and recomputing one means replaying the prefill over the whole conversation. Pinning a session to one replica can also overload that replica, if too many sessions land on the same warm cache. Those two forces pull against each other.

The release states the answer as three terms: session affinity, prefix-cache locality, and replica load. Affinity and locality pull a request toward the replica that already has its state. Load pulls it away when that replica is full. The router scores replicas on output-token throughput, committed work, and available KV capacity rather than raw request count. A new least-request-top-k strategy stops several gateways from piling onto one freshly ready backend, and per-replica request and in-flight limits stop traffic from reaching a replica that has hit its ceiling.

The serving layer can no longer treat every request as an independent unit of work. It has to weigh where the state lives against where the capacity is.

The model stops being the pod

The same shift shows up one layer down, in how a model’s lifecycle is managed. Kubernetes has always managed inference at the pod level, one deployment per model. The release argues that a model’s lifecycle does not need to match its pod’s, and it introduces ModelClaim to split them.

ModelClaim, still experimental, treats a GPU pod as a long-lived runtime pool that can host several models. A claim says which models should be available and how much GPU memory each needs. When a model goes idle, the controller can put it to sleep and reclaim its memory for other models. When traffic returns, the model is activated again on the same runtime if there is room, or moved elsewhere. KV-cache capacity is redistributed as workloads shift.

The point is the same one the routing change makes, just about memory instead of requests. A model and a replica used to be the same thing. Now they are decoupled, so GPU capacity can be shared across models instead of bound statically to one deployment.

What I do not trust yet

I want to be clear about what this release proves and what it does not. It proves the serving layer is starting to think of agents as sessions. It does not prove that the approach works at production scale, because most of the interesting parts are labeled experimental.

ModelClaim, ModelWarmup, and the external-routing hook are all experimental. Session pinning itself is the headline feature and is not marked experimental, but the release also spends real effort hardening the pinning protocol: safe initial claims, refresh behavior, and repinning when a pinned backend has to be bypassed. That list says the feature is still learning its failure modes, which is honest and also a reason to watch it rather than adopt it blindly.

Everything here is the project’s own release notes. There is no independent benchmark behind the throughput and capacity claims, and no production case study of a real agent workload running on top of it. The direction is clear, and the magnitudes are not.

I also want to flag what this is not, because the site has used the word “stateful” twice already in different senses. The Reef post is about stateful evolution, agents that learn while they work. The durable-execution post is about the state a workflow keeps so a half-finished job survives a crash. AIBrix is one layer below both: state the serving layer keeps, so a session stays on the replica that already holds its cache. Three meanings at three layers. The word keeps getting reused because the interesting problems spread from the model into the layers around it.

Sources

1. AIBrix v0.8.0 Release: Building a More Dynamic Inference Platform — AIBrix project blog, October 12, 2026