↓ Skip to main content
mi-homes.org

HashiCorp Vault

Table of Contents

HashiCorp Vault holds API keys, database passwords, tokens, and other sensitive values so they never appear in Git or Helm values. The Kubernetes pattern is Vault (KV v2) → External Secrets Operator → Kubernetes Secret → pods (env or volumes). Any workload that reads from a Secret can use this flow; you only align Vault paths and ExternalSecret fields with the app’s expectations.

Role in the stack
#

flowchart LR
  subgraph git["Git"]
    VY[Helm values / manifests]
    NS[Namespace and wiring]
  end
  subgraph cluster["Cluster"]
    V[Vault KV v2]
    ESO[External Secrets Operator]
    KS[Kubernetes Secret]
  end
  VY --> V
  NS --> V
  ESO -->|Vault API + token| V
  ESO --> KS

Vault must be running and unsealed before ClusterSecretStore and ExternalSecret resources can sync. In GitOps setups, install Vault first, then External Secrets Operator, then applications that declare ExternalSecret objects (or use equivalent sync waves).

What you store in Vault
#

Use the KV secrets engine (this cluster uses v2 under a mount such as secret). Each secret is a set of key–value fields at a logical path. Prefix by scope (team/product, environment/app, or project/service) so paths stay unique and policies can match prefixes. Keep field names stable, the same property names you will reference in ExternalSecret remoteRef.property (for example DATABASE_URL, API_KEY). Use one path per logical secret (per app, per environment, or per credential bundle), depending on how you want to rotate and audit.

Example (paths and keys are illustrative):

vault kv put secret/prod/checkout-service DATABASE_URL='…' API_KEY='…'

Concrete path used by Dex OIDC (GitHub OAuth credentials):

vault kv put secret/homelab/dex \
  GITHUB_CLIENT_ID=<oauth-client-id> \
  GITHUB_CLIENT_SECRET=<oauth-client-secret> \
  GOOGLE_CLIENT_ID=<google-oauth-client-id> \
  GOOGLE_CLIENT_SECRET=<google-oauth-client-secret>

Grafana adds GRAFANA_CLIENT_SECRET on this Dex path; see Prometheus. Argo CD adds ARGOCD_CLIENT_SECRET; see Argo CD. Patch those fields rather than replacing GitHub and Google keys. Do not put Grafana break-glass admin credentials or an Argo CD UI password here.

Immich OAuth client secrets are not stored under homelab/dex. Each Immich instance uses Vault KV secret/homelab/immich/immich-<instance> property OAUTH_CLIENT_SECRET, synced by ESO into the Immich config Secret. Non-secret clientId lives in the private ExternalSecret template. See Immich OAuth.

See Dex OIDC for how ESO syncs this into the dex namespace.

Anything that needs to read these values through ESO must be allowed by Vault policy on that path.

Deploying Vault on Kubernetes
#

Install Vault with Helm or your GitOps tool. Standalone with a persistent volume is the usual choice for production-like data retention. Internal-only clusters often use plain HTTP to the Vault Service; use TLS if the API is exposed or policy requires it. The sidecar injector is optional; disable it if you only sync via External Secrets Operator. The UI is optional, for operators. Pin chart and Vault versions to match your Kubernetes version (kubeVersion in the Helm chart).

If you deploy Vault via Argo CD app-of-apps, register it as a child application:

kubectl apply -k argocd
kubectl apply -f clusters/<cluster>/argocd/app-of-apps.yaml
kubectl apply -f clusters/<cluster>/argocd/applications/hashicorp-vault.yaml

Prerequisites: push homelabs to the branch used by targetRevision, push homelabs-private, and ensure Argo CD has private repo credentials.

Check status:

kubectl -n argocd get applications
kubectl -n argocd get application hashicorp-vault -o yaml

Bootstrap after install
#

  1. Initialize and unseal Vault (HashiCorp quick start). Protect unseal keys and root token.
  2. Enable KV v2 on the mount your ClusterSecretStore will use. A fresh Vault has no secret/ engine until you create it; without this, vault kv put secret/... returns 403 and ESO reports missing secrets:
vault secrets enable -path=secret kv-v2
vault secrets list   # expect secret/  kv

The mount path must match what External Secrets Operator expects (path: secret, version: v2 in the ClusterSecretStore).

  1. Write secrets at paths your applications will reference from ExternalSecret resources:
vault kv put secret/<env>/<app-name> <KEY>=<value> [<KEY2>=<value2> …]
  1. Define a policy that grants read (and list if needed) on exactly the KV paths ESO should read. On this layout that is policy eso-read under secret/homelab/. Mint an orphan token with a TTL and store it as Kubernetes Secret vault-eso-token, never the root token. Commands and expiry notes: External Secrets Operator, mint a read-only token.
  2. Prefer Kubernetes auth for ESO when you want tokens to rotate by themselves. A static tokenSecretRef token expires and ESO will not renew it. See token expiry and rotation.

Operations
#

kubectl -n <vault-namespace> get pods
# Confirm unsealed; spot-check: vault kv get secret/<path>

If Vault is sealed, External Secrets Operator cannot sync; ClusterSecretStore shows InvalidProviderConfig (HTTP 503 / “Vault is sealed”) until you unseal. The full day-2 recovery path after a node or cluster restart is below; ESO-side force-sync and stuck-pod remount steps live on the External Secrets Operator page.

After a node or cluster restart
#

Standalone Vault with Shamir seal and file storage on a PVC comes back Initialized=true, Sealed=true after a node reboot, pod eviction, or cluster restart. The secret data on the PVC is not lost; only the seal state is. Until you unseal, every ExternalSecret that uses the Vault ClusterSecretStore fails with SecretSyncedError, and Argo CD apps that depend on those resources can show Degraded even when the Kubernetes Secret objects still exist (deletionPolicy: Retain). The Vault KV values themselves were not deleted.

Confirm the seal state:

kubectl -n vault get pods
kubectl -n vault exec hashicorp-vault-0 -- vault status
# expect Initialized: true, Sealed: true
# Unseal Progress 0/<threshold> (example: threshold 3 of 5 Shamir shares)

Unseal with the threshold number of different unseal key shares (never commit keys or the root token). Repeat the command until Sealed is false:

kubectl -n vault exec -it hashicorp-vault-0 -- vault operator unseal
# repeat until Sealed: false

Confirm Vault and the cluster-wide store are healthy again:

kubectl -n vault exec hashicorp-vault-0 -- vault status
# Sealed: false
kubectl -n vault get pods
# hashicorp-vault-0 Ready 1/1
kubectl get clustersecretstore vault
# READY=True

Then force External Secrets Operator to reconcile (refresh intervals can be long after an outage) and remount any pods that started before secrets were ready; see External Secrets Operator, after a node or cluster restart.

If the store is Ready but an ExternalSecret still errors with Secret does not exist, the KV path is missing (or the field names do not match). Write the path once, then wait for ESO to refresh (or annotate the ExternalSecret to force reconcile). Example for Dex (values from your password manager; do not commit them):

vault kv put secret/homelab/dex \
  GITHUB_CLIENT_ID=… \
  GITHUB_CLIENT_SECRET=… \
  GOOGLE_CLIENT_ID=… \
  GOOGLE_CLIENT_SECRET=…

After unseal, if ClusterSecretStore stays False for several minutes, restart the ESO controller so it leaves error backoff:

kubectl -n external-secrets-system rollout restart deployment/external-secrets-operator

Optional later improvement: transit / cloud KMS auto-unseal so day-2 does not require manual shares after every restart.

Adding a new application
#

Choose a Vault path and field names (see naming above). Write the secret in Vault and extend the ESO token’s policy if the path was not covered before. In the app’s namespace, add an ExternalSecret that maps those paths and properties to the Kubernetes Secret keys your Deployment or Helm chart expects (External Secrets Operator).

No change to Vault’s Helm values is required for each new app unless you add new mounts or auth methods.

Related#

References
#