Longhorn

Longhorn provides replicated block volumes for chatty RWO workloads (databases, app config, game worlds) on dedicated node SSDs. Leave shared media and libraries on NFS (nfs-csi). The cluster default StorageClass is longhorn-ssd; set storageClassName: local-path only when you want non-replicated node-local disks or a rollback target during a cutover.

Role in the stack

#

GitOps installs Longhorn from the upstream Helm chart (https://charts.longhorn.io, pin e.g. 1.12.0 on the Argo Application) plus public values/manifests and a private overlay. Sync wave places Longhorn after Vault/ESO and before application waves that will consume the StorageClass.

Hypervisor disks

#

k3s guest disks live on Proxmox local-lvm (host SSD), not on NAS-backed datastores and not on Ceph for this generation. The same extendable virt disk may host the guest OS and the Longhorn data mount; prefer a separate logical volume or partition for Longhorn so replica growth does not silently fill /.

Do not expect Proxmox cross-node live migrate or HA restart for these VMs. Durability comes from spreading guests across compute hosts, Longhorn replicas inside the cluster, and backups, not from shared hypervisor storage. Node provisioning patterns live in K3s (Terraform / Proxmox); site IPs and hostnames stay in homelabs-private/clusters/<cluster>/config/cluster.yaml.

Node disk prep

#

On each node that should store replicas, create a stable mount for Longhorn data (prefer a dedicated LV on the local-lvm virt disk), then register it with Longhorn.

sudo mkdir -p /var/lib/longhorn-ssd
# If using a dedicated LV/partition: mount it and persist in fstab, then:
df -h /var/lib/longhorn-ssd   # confirm local disk, leave comfortable free space (e.g. ≥20%)

Register the dedicated path, not the OS root filesystem. After Longhorn is running, label and annotate only prepared nodes so the manager creates a disk with tag ssd (matching diskSelector: ssd on StorageClass longhorn-ssd):

kubectl label node <node> node.longhorn.io/create-default-disk=config --overwrite
kubectl annotate node <node> node.longhorn.io/default-disks-config='[{"path":"/var/lib/longhorn-ssd","allowScheduling":true,"tags":["ssd"]}]' --overwrite

Nodes without a prepared mount must not get the create-default-disk label. Confirm disks appear on nodes.longhorn.io with the ssd tag before relying on the StorageClass.

When you enlarge a virt disk in Proxmox, grow the guest filesystem afterward (typical GPT + ext4 path):

sudo growpart /dev/<disk> <partition>
sudo resize2fs /dev/<partition-or-lv>
df -h /var/lib/longhorn-ssd

Grow the LV between growpart and resize2fs when Longhorn uses a separate logical volume. Leave headroom before large migrations; expand again when replica capacity runs low.

Install

#

1. Merge public homelabs Longhorn values and longhorn/manifests (StorageClass longhorn-ssd).
2. Update AppProject allow-lists: chart repo https://charts.longhorn.io, destination namespace longhorn-system, and cluster-scoped kinds needed for CSI (StorageClass, CSIDriver, PriorityClass, mutating webhooks). Apply the AppProject with kubectl; it is not self-healed by app-of-apps.
3. Commit the Argo Application and private overlay; sync Application longhorn.
4. Verify managers/CSI Ready, StorageClass longhorn-ssd present (and marked default), and SSD-tagged disks only. After sync, unset the k3s local-path default annotation so only one default remains:

kubectl annotate storageclass local-path storageclass.kubernetes.io/is-default-class=false --overwrite
kubectl get storageclass

local-path is owned by the k3s local-storage Addon; re-check the default after k3s upgrades if it flips back.

Public values keep the UI as ClusterIP, disable Ingress, enable chart NetworkPolicies for k3s, disable the chart default StorageClass, and set preUpgradeChecker.jobEnabled: false so Argo CD does not run the Helm pre-upgrade Job before the ServiceAccount exists (required for GitOps installs). They also set metrics.serviceMonitor.enabled: true so kube-prometheus-stack scrapes Longhorn disk and volume metrics.

Control plane taints and engine image

#

k3s control-plane nodes typically carry node-role.kubernetes.io/master and/or control-plane NoSchedule taints. Chart values must tolerate those taints in two places: pod tolerations (so Longhorn Manager can run on every node that hosts SSD disks) and defaultSettings.taintToleration (so the engine-image DaemonSet schedules on the same nodes).

If taintToleration does not match the live taints, engine-image pods stay Pending on control-plane nodes. Volumes that need a replica there then fail with symptoms such as disks are unavailable or insufficient storage even when df shows free space. Confirm engine-image DaemonSet pods are Ready on every SSD-tagged node before blaming capacity alone.

Capacity gate

#

Before each migration wave (and after disk expands), confirm every ssd-tagged Longhorn disk has roughly ≥20% free space. Plan for replica multiplication: a claim of size N with numberOfReplicas: 3 needs about 3N across the pool, plus rebuild headroom. Migrate one PVC at a time; wait until the new volume is Healthy before starting the next. Expand Proxmox virt disks and grow the guest filesystem when free space would drop below that gate.

UI access

#

The Longhorn UI has no public Ingress or tunnel in the baseline layout. Use kubectl port-forward to the UI Service in longhorn-system from a trusted admin context. Treat UI access as full storage administration.