NVIDIA Details AICR's Validated Kubernetes Runtime for Dynamo, with Infrastructure Checks Before Model Deployment

NVIDIA's October 30, 2026, walkthrough describes how AI Cluster Runtime (AICR) builds a validated Kubernetes foundation for NVIDIA Dynamo inference. A DynamoGraphDeployment describes how a model-serving graph should run, but it does not by itself ensure GPU drivers, resource allocation, scheduling, network ingress and monitoring are ready. AICR addresses that infrastructure layer with versioned recipes that bundle Dynamo and supporting components, then check whether a chosen cluster meets the recipe. This is a runtime deployment and validation procedure, not a claim that a new model was trained or that a particular application achieved a measured performance gain.

The article distinguishes two request-routing paths. In the simpler path, a Dynamo Frontend sits behind a Kubernetes Service. With KV routing enabled it tokenizes a request, compares prefix overlap and current load, selects a suitable worker and dispatches the request. It is suited to a model-specific endpoint or an existing proxy that merely forwards to the Frontend. The second path uses Kubernetes Gateway API and its Inference Extension. An HTTPRoute resolves to an InferencePool; the Dynamo operator creates the pool and its Endpoint Picker reference from the DynamoGraphDeployment. A Gateway calls the Dynamo EPP, which selects a worker based on the same KV-aware information and returns routing metadata. The Gateway itself then proxies the HTTP request to the selected worker's Frontend sidecar. Selection by the EPP and actual forwarding by the Gateway are different responsibilities.

Both paths can sit behind an HTTPRoute, so their distinction is not simply “with a gateway” versus “without a gateway.” It is which component owns request transport to the worker, including timeouts, retries and fallback: Dynamo Frontend in the first path, the Gateway in the second. The second path adds a compatible Gateway, InferencePool, EPP and worker sidecars. NVIDIA calls the first path its smallest and more production-proven serving topology. The post limits the second path to the operator-managed DynamoGraphDeployment integration, not an experimental standalone vanilla-vLLM on-ramp. For Dynamo 1.4, that EPP uses a Go framework with a Rust KV router embedded through CGO; a native Rust EPP is presented as a future Dynamo 1.5 direction, not a feature already in the 1.4.2 recipe.

KV-cache awareness matters when repeated agent interactions share long prefixes such as instructions, tool definitions and previous messages. Sending a request to a worker that already holds relevant KV blocks can avoid repeating prompt processing. Workers publish cache events through a local ZMQ source; Dynamo normalizes and republishes them to its event plane. The request path uses TCP by default, while the cache-event plane uses ZMQ. Both routing modes use worker-published live cache and load state. A predicted cache map can be configured instead, but NVIDIA warns it can miss evictions and drift. The article also discusses separating prefill from decode; actual cache-miss rates depend on workload and cannot be inferred from this architecture diagram alone.

The AICR workflow has ordered steps. First, aicr snapshot captures cluster facts. Second, aicr recipe selects a Dynamo inference recipe from the snapshot, recording pinned versions, platform constraints and checks. In its stated AICR v1.0.0 Amazon EKS example for GB200 NVL72 and GB300 NVL72, the Dynamo Platform chart is pinned to 1.4.2 among other named components; these are example-specific tested versions, not universal latest versions. An optional offline validation can compare a recipe to a different captured snapshot, but behavioral checks are skipped without a live cluster. Third, aicr bundle renders Helm values and manifests, with deployment alternatives such as Helmfile and GitOps bundles. Fourth, the runtime is deployed. Fifth, aicr validate performs deployment and conformance checks against the live cluster before the application's DynamoGraphDeployment is installed.

Typical checks cover GPU capacity and nvidia-smi, GPU Operator and Dynamo health, DRA access, gang scheduling and autoscaling, Prometheus metrics, and Gateway/API readiness when using Kubernetes-native ingress. NVIDIA says performance validation is a separate phase because it launches a temporary serving workload and consumes GPUs. Thus a successful deployment or conformance check does not demonstrate application throughput. The recipe's checks vary by platform, and the article does not show that our own cluster has passed them. The useful design point is a reproducible order from captured cluster state to chosen versions, rendered runtime and live infrastructure checks, with the application deployed only afterward.