GPU Operator

A k3s worker that already runs on the second Proxmox compute node can use that host’s NVIDIA GPU. The host binds the card to vfio-pci, the worker VM receives it as a PCI device, and NVIDIA GPU Operator — reconciled by Argo CD Application gpu-operator — installs the guest driver, container toolkit, and device plugin. Pods then request nvidia.com/gpu. Do not helm install the operator from a workstation, and do not terraform apply the five-node stack to attach a GPU on a cluster whose VMs have drifted from Terraform.

The pin is Helm chart gpu-operator v26.7.0 from https://helm.ngc.nvidia.com/nvidia. Patch bumps within that series follow the same path; a different chart version is a deliberate re-pin.

Role in the stack

#

Proxmox keeps the GPU unbound from nvidia and nouveau so the guest can own it. GPU Operator then loads the NVIDIA driver inside the worker, not on the hypervisor. Node Feature Discovery labels the worker when PCI vendor 10de is present; driver, toolkit, and device-plugin DaemonSets follow that label. Control-plane nodes stay CPU-only.

The toolkit must talk to k3s containerd, which does not live at /etc/containerd. Public values set CONTAINERD_CONFIG to /var/lib/rancher/k3s/agent/etc/containerd/config.toml, CONTAINERD_SOCKET to /run/k3s/containerd/containerd.sock, RUNTIME_CONFIG_SOURCE to file (k3s does not put containerd on PATH), and the NVIDIA drop-in under config.toml.d. CDI stays on; the NRI plugin stays off because worker containerd is older than 1.7.30.

Namespace gpu-operator is privileged Pod Security. Kyverno excludes that namespace from the Fail-closed webhook so a Kyverno outage cannot stall driver pods. DCGM exporter is on; the existing Prometheus stack scrapes its ServiceMonitor. There is no Longhorn claim and no Cloudflare hostname.

KubeVirt on the workers does not receive this GPU. Nested VM passthrough is a later change.

Host: IOMMU, vfio, passthrough

#

Do this on <pve-2-node> only, against the existing worker VM <vmid> that already lives there (<k3s-gpu-worker> in Kubernetes). Confirm the host still has a console without that GPU (iGPU, serial, or IPMI). Do not enable pcie_acs_override. If the NVIDIA audio function shares the IOMMU group, pass that function too.

Enable IOMMU in firmware, then in the host kernel. Use amd_iommu=on or intel_iommu=on to match the CPU, plus iommu=pt. Blacklist nouveau (and any host nvidia module) for that GPU. Bind the PCI IDs from lspci -nn | grep -i nvidia to vfio-pci, update initramfs, and reboot the host if the kernel args or vfio bind are new.

Attach the device to the existing VM. Machine type q35 and CPU host are the usual pair. Keep a virtual VGA on the guest so the Proxmox console does not need the passed GPU as primary display:

qm set <vmid> -hostpci0 <gpu-pci-id>,pcie=1

Add hostpci1 for the audio function when it is a separate BDF. Stop and start the VM (a live add is unreliable). In the guest, lspci must list NVIDIA. Public Terraform may include optional gpu_passthrough_pci_ids for a greenfield cluster; leave those empty here and do not apply Terraform to attach the GPU on this cluster.

Site facts (Proxmox node name, VMID, PCI BDFs, Kubernetes node name) belong in homelabs-private/clusters/<cluster>/config/cluster.yaml under gpu: or in your local notes. Public git keeps placeholders.

GitOps layout

#

The Application is multi-source: Helm repo https://helm.ngc.nvidia.com/nvidia, public values, private overlay, and the public namespace path. Destination namespace is gpu-operator. Sync-wave is -1 (with Kyverno and Longhorn). CreateNamespace=true and ServerSideApply=true. Automated prune is off; selfHeal is on. The overlay is comments only — do not put toolkit env or a node name there, because Helm replaces lists.

AppProject

#

Allow Helm repo https://helm.ngc.nvidia.com/nvidia, destination namespace gpu-operator, and cluster-scoped ClusterPolicy (nvidia.com). The homelab AppProject is applied outside app-of-apps. Apply the live project before you expect Argo to accept the Application:

kubectl apply -f homelabs-private/argocd/projects/homelab.yaml

First sync

#

Passthrough in the guest must already show NVIDIA. Merge public homelabs (values, namespace, Kyverno exclude lists) first, apply AppProject, then commit the private Application.

kubectl -n argocd get application gpu-operator
kubectl -n gpu-operator get pods
kubectl get clusterpolicy

ClusterPolicy should become ready. The passthrough worker should show allocatable nvidia.com/gpu. Control-plane nodes should not. Driver, toolkit, and device-plugin DaemonSets run on that worker; NFD runs cluster-wide.

A one-shot CUDA Job is the smoke test. Request nvidia.com/gpu: 1, confirm it schedules on <k3s-gpu-worker>, then delete the Job. Do not leave that Job in GitOps.

Rollback

#

Delete Application gpu-operator and wait for the namespace to drain. Stop the worker VM, remove hostpci, and start it again. Rebind the GPU to a host driver only if that host needs a display. Revert the Kyverno exclude lists in a follow-up commit after the namespace is gone.