🚀 AIBrix v0.8.0 Release
We are excited to announce AIBrix v0.8.0, the latest release of the open-source, Kubernetes-native infrastructure stack for scalable GenAI inference. This release brings 443 commits from 104 contributors since v0.7.0, bringing major advances in model lifecycle management, inference-aware routing, agentic and asynchronous workloads, and production reliability.
AIBrix v0.7.0 focused on making the platform more composable and self-service, with the introduction of the AIBrix Console, a production OpenAI-compatible Batch API, multi-engine support, KV-cache-centric P/D disaggregation, and a highly available gateway with composable routing.
With v0.8.0, we are taking the next step: making inference infrastructure more dynamic, workload-aware, and adaptable.
This release introduces a new model lifecycle abstraction that decouples models from the Pods hosting them, significantly expands inference-aware routing, adds distributed session affinity for agentic workloads, brings asynchronous video generation into the gateway, improves interoperability with inference engines and external routers, and continues to harden batch inference and multi-node orchestration.
Models as First-Class Resources with ModelClaim
Kubernetes traditionally manages inference workloads at the Pod level. In practice, however, the lifecycle of a model does not always need to match the lifecycle of its Pod.
A GPU runtime may host multiple models, each with its own workload and lifecycle. Models can become hot or cold independently, and an idle model can release GPU memory while keeping the underlying runtime alive, allowing that capacity to be used by other models.
AIBrix v0.8.0 introduces ModelClaim as an experimental feature, a Kubernetes-native abstraction that manages model lifecycle independently from the GPU Pods hosting them. Instead of treating every model as a dedicated Pod, GPU Pods can operate as long-lived runtime pools while ModelClaims describe which models should be available and how much GPU capacity they require.
The ModelClaim controller manages this lifecycle end to end. It places models on runtimes with sufficient GPU memory, manages per-model memory allocations, and dynamically redistributes KV-cache capacity as workloads change. Once a model becomes ready, it can be exposed to the Gateway for serving without requiring a new Pod to be created.
When a model becomes idle, AIBrix can put it to sleep and reclaim its GPU memory for other models. If traffic returns, the model can be activated again on the same runtime when capacity permits, or relocated to another runtime when necessary. ModelClaim also handles recovery and rescheduling after terminal engine failures and improves placement fairness when multiple models compete for available GPU capacity.
Together, these capabilities allow model availability to be reconciled independently from Pod lifecycle, laying the foundation for higher-density inference serving where GPU capacity can be shared dynamically across models instead of statically binding one model to one deployment.
For the design and implementation details, see AIBrix ModelClaim: Engine-Aware GPU Sharing for Long-Tail Models.
Model Warmup and Operational Improvements
AIBrix v0.8.0 introduces ModelWarmup as an experimental feature for preparing Kubernetes nodes before inference workloads begin serving traffic.
ModelWarmup allows operators to target selected nodes and preload container images and run custom preparation actions ahead of workload placement. By moving these operations off the critical startup path, nodes can be prepared in advance for anticipated inference workloads, reducing initialization overhead when new serving capacity is brought online.
As an experimental feature in this release, ModelWarmup establishes the foundation for more advanced model preparation and faster workload startup, while its API and capabilities continue to evolve.
Stateful Routing for Agentic Workloads
Agentic applications increasingly generate sequences of related inference requests rather than isolated prompts.
Routing each turn independently can lose useful locality. A replica that processed earlier turns may already contain KV-cache state or other session-local information that makes it preferable for subsequent requests.
AIBrix v0.8.0 introduces Redis-backed session-key pinning for distributed Gateway deployments. (#2742)
Applications can provide a session identifier, and AIBrix can associate that session with a backend replica. Because the mapping is stored in shared state, the affinity continues to work even when subsequent requests arrive through different AIBrix Gateway replicas.
Additional changes harden the distributed pinning protocol, including safe initial claims, refresh behavior, and repinning when a selected backend must be bypassed. (#2765, #2768, #2773)
This gives AIBrix a foundation for routing multi-turn chat and agentic workloads using a combination of:
session affinity + prefix-cache locality + replica load
rather than treating every request independently.
Asynchronous Multimodal Inference with vLLM-Omni
Not every generative workload fits the traditional synchronous request-response model.
Video and other long-running multimodal generation workloads can take substantially longer than text generation and often need asynchronous execution, job tracking, and result retrieval.
AIBrix v0.8.0 expands multimodal serving with vLLM-Omni video generation support. (#2686)
The Gateway now also includes an asynchronous job registry and video routing path, enabling long-running generation requests to remain associated with the appropriate backend while their execution progresses. (#2740)
This extends the AIBrix serving model from:
request → inference → response
toward:
submit → route → execute asynchronously → track → retrieve result
and provides infrastructure for serving long-running multimodal workloads through the same Gateway and control plane used for traditional online inference.
More Inference-Aware Gateway Routing
AIBrix v0.8.0 continues evolving the Gateway from request-level load balancing toward inference-aware traffic management.
Per-replica capacity limits. Routing profiles can define per-replica RPS and in-flight request capacity at the deployment level. Every replica inherits these limits, so aggregate capacity follows the fleet as it scales, and the Gateway stops sending traffic to replicas that have reached their threshold. (#2726)
Load balancing using inference work. The Gateway can now score replicas using richer signals such as output-token throughput, committed work, and available KV capacity, and P/D routing gains token-load-aware decode scoring. This makes routing better suited to heterogeneous deployments and highly variable sequence lengths. (#2771, #2842)
Safer least-request routing. The new least-request-top-k strategy selects from a small group of lightly loaded replicas instead of always picking the single minimum, so multiple Gateway replicas no longer pile onto the same backend (for example, a newly ready replica). It is also the load component of AIBrix’s automatic blended routing. (#2918)
External Routing
AIBrix is also expanding support for external routing systems, currently as an experimental feature, allowing the Gateway to participate in deployments where routing decisions are produced outside AIBrix while AIBrix continues to provide service discovery, traffic management, and the surrounding inference infrastructure.
Together, these capabilities move AIBrix toward a model where operators can combine:
AIBrix infrastructure + their preferred inference engine + their preferred routing strategy
without requiring the entire serving stack to come from a single implementation.
Additional Platform Improvements
Beyond the major features above, AIBrix v0.8.0 includes a broad set of improvements across orchestration, autoscaling, LoRA serving, batch inference, reliability, and developer experience.
More elastic and resilient batch inference. The Batch platform continues to mature with adaptive AIMD concurrency control (#2530), capacity-aware concurrency, alignment between jobs and compute provisioning windows (#2575), and improvements to retry handling, output persistence, dispatch errors, cancellation, and job loading. Together, these changes allow batch workloads to make better use of elastic compute while reacting more gracefully to changing backend capacity.
Multi-node inference and StormService. StormService continues to mature as AIBrix’s abstraction for complex distributed inference workloads. v0.8.0 adds drain-aware RoleSet rollout and scale-in (#2651), progress deadlines (#2604), historical node replacement scheduling (#2628), improved deployment-mode handling (#2617), and tighter integration with Volcano gang scheduling (#2672, #2733). Additional fixes improve RoleSet validation and status handling, ownership scoping, scaling behavior, and controller correctness for long-running multi-node workloads.
LoRA and ModelAdapter. LoRA serving becomes more integrated with the AIBrix management experience. The Console now provides Kubernetes-backed ModelAdapter management (#2514), while the serving stack adds multi-LoRA examples, per-adapter Gateway metrics, improved ModelAdapter status reporting (#2624), and expanded controller integration coverage. Additional fixes strengthen adapter loading, pod selection, reconciliation, and validation (#2710, #2871, #2875, #2899), making multi-adapter deployments easier to operate reliably.
Expanded inference API and routing support. The Gateway broadens its inference-engine API coverage, including SGLang /v1/decisions, /v1/systemone, and /v1/score, as well as native vLLM /classify routing (#2911, #2919, #2999, #3015). Support for Messages API paths is also hardened, including /v1/messages/count_tokens, while TensorRT-LLM integration expands with generation-first parallel P/D routing (#2794). Together, these changes allow a wider range of engine-specific and OpenAI-compatible workloads to operate through a common Gateway.
Routing and P/D reliability. A number of changes harden distributed routing under failure and concurrency. Token-load state can be shared across Gateway replicas (#2868), model discovery can be made optional while continuing to expose ready backends (#2913, #2915), and P/D routing now handles cases where the selected decode backend fails to respond or a prefill failure leaves a silent decode stream (#2859, #2860, #2933). Additional fixes improve prefix-cache scoring and KV-event handling, request accounting, routing of tool calls and array prompts, and validation of unsupported P/D endpoints.
Installation, testing, and developer experience. v0.8.0 substantially expands integration and end-to-end testing across the project, including an installation smoke pipeline (#2730), StormService + Volcano gang-scheduling coverage (#2733), ModelRouter lifecycle tests (#2743), ModelAdapter tests, and additional P/D protocol coverage. Helm and installation fixes also keep CRDs, RBAC, and webhooks aligned with the rapidly expanding AIBrix API surface.
Together, these improvements may be less visible than the headline features, but they represent an important part of v0.8.0: making AIBrix safer to operate, easier to validate, and more predictable under production workloads.
Looking Ahead
AIBrix started as a collection of Kubernetes-native building blocks for scalable LLM inference. Each release has expanded the boundary of what the infrastructure can manage.
v0.7.0 made those building blocks more composable and accessible through a common operational platform. With v0.8.0, AIBrix takes another step toward a more dynamic, workload-aware inference platform.
Looking forward, we plan to continue extending these ideas toward more dynamic and heterogeneous inference systems: higher-density model serving, routing across increasingly large fleets, richer agentic workloads, and deeper coordination between scheduling and inference-aware traffic management. We are also working on omni-modal and multi-stage serving (#1966, #2962), where stages such as Encode, Prefill, and Decode can be independently placed, scaled, and routed based on their resource and latency characteristics, along with faster model preparation and warmup.
AIBrix is an open-source project, and none of this would be possible without the community. Thank you to everyone who contributed code, reviews, bug reports, documentation, design discussions, and feedback throughout the v0.8.0 development cycle.
We welcome feedback, production experiences, and contributions as we continue building the next generation of AIBrix.
Get started with AIBrix v0.8.0 and let us know what you build on GitHub.