My Kubernetes cluster still runs on Proxmox, with each Kubernetes node running as a VM that boots from local SSD storage on the Proxmox host and uses a dedicated Longhorn mount on that same virtual disk.
That SSD-backed pool is where I put the busy workloads that should not live on slow shared media, while Longhorn keeps three replicas spread across the three physical machines.
The design is deliberately simple: Proxmox stays responsible for flexible compute, local SSD backs the VM disks, Longhorn handles Kubernetes volume replication, and there is no Ceph layer underneath.
Why I kept Proxmox without Ceph#
I do not want these machines to become only “the Kubernetes cluster,” because in my setup I still need them to stay useful as general hypervisors.
Sometimes I need another Kubernetes cluster or a small test environment that has nothing to do with the main one, and Proxmox gives me that without buying more hardware or turning every experiment into a change on the production-ish Kubernetes nodes.
The trade-off is that the Kubernetes VMs use local disks, so I am not using Proxmox HA for them and they do not live-migrate; if a Proxmox host dies, the VMs on that host are down until I restore or rebuild them, which is acceptable because I care more about keeping application data available inside Kubernetes than making every VM a highly available Proxmox guest.
The usual answer would be Ceph, with VM disks on shared replicated storage and Proxmox HA restarting guests elsewhere, but that did not fit this setup. The cluster has three Proxmox nodes, M-01, M-02, and M-03, and because I also run extra lab VMs, change things often, and want room to experiment, I do not want to operate a second distributed storage system there.
The bigger problem is nested replication, because I already need Longhorn inside Kubernetes for volumes, and if I put Ceph under the VM disks while Longhorn runs inside the VMs, both layers end up copying the same data. Ceph replicates VM disk blocks, Longhorn replicates Kubernetes volume blocks, rebuilds hit the network twice, and storage debugging turns into checking two distributed systems instead of one.
With three Ceph copies underneath and three Longhorn replicas above, the write path gets expensive quickly, and I do not need that in this cluster. Application data HA belongs to Longhorn, hypervisor and VM HA are not goals here, and VM disk recovery is handled through backups and rebuilds rather than another live storage layer.
Having three nodes did not automatically make Ceph the right answer for this lab.
Where the HA boundary is#
Longhorn protects data at the Kubernetes layer, so a volume with numberOfReplicas=3 can lose a node and keep serving data while Longhorn rebuilds the missing replica onto another eligible disk.
The important detail is failure domains, because Longhorn sees Kubernetes nodes rather than Proxmox hosts, and since my Kubernetes nodes are VMs, Longhorn does not automatically know which physical machine a VM is running on. If I am not careful enough, two replicas could land on two VMs on the same Proxmox host, which looks redundant from Kubernetes but is not redundant when the physical machine disappears.
That is why I make placement explicit: for each Kubernetes cluster, I run three control-plane VMs and three nodes, every VM has an SSD-backed Longhorn disk, and every physical machine carries exactly one control-plane VM and one node.
| Physical machine | Proxmox role | k3s VMs |
|---|---|---|
| M-01 | Compute | 1 control-plane + 1 node |
| M-02 | Compute | 1 control-plane + 1 node |
| M-03 | Compute | 1 control-plane + 1 node |
That gives me control plane 1+1+1 and nodes 1+1+1, so if one physical machine fails, etcd still has quorum, Kubernetes still has two nodes for workloads, and Longhorn still has two physical copies of each properly placed three-replica volume.
Three replicas fit the failures I actually expect, such as a k3s VM rebooting, a guest disk filling up, a VM being rebuilt, or one Proxmox host going offline. After a VM-level failure, two copies remain and Longhorn can rebuild the third onto another eligible VM if the host is still healthy; after a full host loss, the volume is degraded rather than gone, and I can restore the missing VMs before letting Longhorn rebuild the third replica.
Two replicas could survive one host loss if placement is perfect, but that leaves less room for the VM-level failures that happen more often, while four or five replicas would mostly waste SSD. If I ever put these VM disks on replicated Proxmox storage, I would revisit the replica count instead of keeping three as a leftover default.
The key is to make physical placement part of the storage design, because Longhorn replica anti-affinity can keep replicas away from the same Kubernetes node, but that is not enough when the Kubernetes nodes are VMs. If two VMs sit on the same Proxmox host, they share the same real failure domain.
Capacity and latency still matter#
Replica count is capacity planning, so a 10 GiB PVC with three replicas consumes roughly 30 GiB across the Longhorn pool before filesystem and snapshot overhead.
I try to keep about 20% free on every Longhorn disk so rebuilds have somewhere to land, and when a disk gets tight, I grow the VM disk in Proxmox and expand the guest filesystem.
The VM layout is intentionally boring: one virtual disk per Kubernetes VM, OS and Longhorn data on the same extendable SSD-backed disk, the same local-lvm pattern on each Proxmox host, and the same Longhorn mount path on each VM.
That separate Longhorn mount matters because replica growth should not fill the root path and take the node with it.
Longhorn on VMs is still storage on top of storage, so latency matters more than a nice sequential throughput number, and slow backing disks or a bad virtual disk setup can show up as replica disconnects and rebuilds. For this setup, Longhorn belongs on SSD-backed VM disks rather than virtualized spinning disks.
What this setup optimizes for#
This setup optimizes for flexibility and failure modes I can reason about, while avoiding a stack where Ceph, Proxmox HA, and Longhorn are all copying the same bytes.
I can still create another VM when I need one, application disks can survive a Kubernetes node failure, and a whole Proxmox host can go down without deleting the only copy of a volume.
Local SSD under the guests, three Longhorn replicas, one control-plane VM and one node on each compute host, and no second distributed storage layer underneath it: that is my lab’s storage design.