NVIDIA Describes Its DGX SuperPOD AI Factory Operations Agent: Six Domain Agents on NemoClaw and OpenShell, With Read-Only Credentials for Kubernetes Checks; No Accuracy Figure and NGC Early-Access Images

On November 4, 2026, NVIDIA published "How NVIDIA Uses AI Agents to Streamline AI Factory Operations" on its developer blog, written by Kavya G Y and Rohan Adwankar. The post describes the AI Factory Operations Agent, which NVIDIA says it built to streamline the operation of its own internal DGX SuperPOD clusters and to reduce manual work for its site reliability engineers (SREs). Instead of checking compute, network, hardware and scheduler dashboards one by one, an SRE asks questions in a chat interface, and specialized sub-agents gather and correlate the evidence. The sub-agents talk through the Model Context Protocol (MCP). The page also carries an automatic summary generated by an NVIDIA Nemotron model; we did not use it and relied on the post's own text and the GitHub repository.

The architecture is built around NVIDIA's NemoClaw blueprint. A user-facing agent runtime receives the SRE's question and routes it to a domain agent. Agents reach the infrastructure through OpenClaw plugins and MCP servers, and each MCP server exposes a small set of approved operational tools. The framework is built on NVIDIA OpenShell and, according to the post, can run fully on premises, with inference served locally by NVIDIA Nemotron open models. NVIDIA says the framework runs across its internal DGX H100, DGX B200 and DGX GB300 production clusters. That describes NVIDIA's own deployment, and it is not a hardware requirement stated for other users.

Deployment is a Kubernetes control-plane application. Helm installs the web interface, the OpenClaw gateway, the MCP tool servers, the inference endpoint and the OpenShell runtime as services. For shell commands, OpenShell creates a separate Kubernetes sandbox per session instead of running commands in the main gateway pod. Kubernetes inspection goes through a server-side run_kubectl tool that uses a pinned kubectl binary and registered read-only credentials, and OpenShell receives neither kubectl nor Kubernetes credentials. The post shows the result of kubectl auth can-i checks from the agent's side: it can get pods and configmaps, but cannot get secrets or create or update pods. This is a useful picture of the permission boundary, taken from one configuration.

Six domain agents are listed. The Job Recovery agent detects training failures and goodput degradation and performs root cause analysis across scheduler, application, hardware and network layers. The Hardware agent triages PCIe, ECC, GPU, NVMe and thermal faults down to a component. The Observability agent queries Prometheus, telemetry and Grafana Loki logs and builds dashboards on demand. The Research agent searches preloaded documentation such as NVIDIA Mission Control manuals and DGX SuperPOD reference architectures, and teams can add their own runbooks. The Kubernetes agent gives read-only awareness of pods, services, deployments, GPU resources and node state. The Cluster Management agent works with NVIDIA Base Command Manager APIs through diagnostic tools for advisory work. Cross-cutting MCP tools include web search, code execution and Grafana dashboards, and teams can connect their own MCP servers.

NVIDIA offers five lessons from production use: keep agent boundaries clear, start read-only so that accurate investigation builds trust before any automation, correlate across layers to find causes that isolated tools miss, define MCP tools precisely so routing improves, and show tool calls, evidence and conclusions so that SREs can check them before acting. The post says NVIDIA keeps refining the framework by measuring root cause analysis accuracy and tracking SRE agreement. It publishes no accuracy figure, no agreement rate, no incident counts and no time saved. The demo video is described as an incident injected into a DGX cluster for internal demonstration.

Availability needs a careful reading, because the two sources stress different things. The post calls the orchestration framework experimental open source software whose APIs, configuration and deployment requirements may change between releases, and points to the GitHub repository. The repository, under the Apache 2.0 license, also calls it an experimental release and adds that in the early access stage the published Helm chart and container images require an invitation to a private NGC registry, and that a login to nvcr.io alone does not grant pull access. It lists Kubernetes, kubectl and Helm 3 as requirements, an OpenAI-compatible endpoint or a chart-managed vLLM for the model, and a GPU only when you deploy that vLLM. Separately, the post says NVIDIA customers can ask their account manager for access to NVIDIA-developed cluster operations agents. The README also lists workflows that the post does not name, such as Slurm job analysis and VictoriaMetrics queries, and four interfaces: a browser UI, an HTTP API, a command-line client and an MCP bridge for coding agents.

For multi-tenant clusters the post describes scoped agents, tools, sandboxes and Kubernetes permissions, so that tenant SREs see only their own namespaces, jobs and nodes while central operations keep a broader view, with OpenShell enforcing kernel-level restrictions. It recommends a dedicated GPU-accelerated control plane that is isolated from the production cluster, so that agent reasoning does not use training GPUs, and mentions an RTX PRO Workstation or DGX Spark for smaller deployments.

Our reading: this is an architecture and lessons post from a vendor describing its own internal tool. The design choices are concrete and reusable, namely read-only credentials, per-session sandboxes and narrow MCP tools, but there is no measured result to weigh against the effort. Teams considering it should treat access to the container images as a gating step and test it on a non-production cluster first.