↓ Skip to main content
mi-homes.org

HashiCorp Vault

Table of Contents

HashiCorp Vault holds API keys, database passwords, tokens, and other sensitive values so they never appear in Git or Helm values. The Kubernetes pattern is Vault (KV v2) → External Secrets Operator → Kubernetes Secret → pods (env or volumes). Any workload that reads from a Secret can use this flow; you only align Vault paths and ExternalSecret fields with the app’s expectations.

Role in the stack
#

flowchart LR
  subgraph git["Git"]
    VY[Helm values / manifests]
    NS[Namespace and wiring]
  end
  subgraph cluster["Cluster"]
    V[Vault KV v2]
    ESO[External Secrets Operator]
    KS[Kubernetes Secret]
  end
  VY --> V
  NS --> V
  ESO -->|Vault API + token| V
  ESO --> KS

Vault must be running and unsealed before ClusterSecretStore and ExternalSecret resources can sync. In GitOps setups, install Vault first, then External Secrets Operator, then applications that declare ExternalSecret objects (or use equivalent sync waves).

What you store in Vault
#

Use the KV secrets engine (this cluster uses v2 under a mount such as secret). Each secret is a set of key–value fields at a logical path. Prefix by scope (team/product, environment/app, or project/service) so paths stay unique and policies can match prefixes. Keep field names stable, the same property names you will reference in ExternalSecret remoteRef.property (for example DATABASE_URL, API_KEY). Use one path per logical secret (per app, per environment, or per credential bundle), depending on how you want to rotate and audit.

Example (paths and keys are illustrative):

vault kv put secret/prod/checkout-service DATABASE_URL='…' API_KEY='…'

Concrete path used by Dex OIDC (GitHub OAuth credentials):

vault kv put secret/homelab/dex \
  GITHUB_CLIENT_ID=<oauth-client-id> \
  GITHUB_CLIENT_SECRET=<oauth-client-secret> \
  GOOGLE_CLIENT_ID=<google-oauth-client-id> \
  GOOGLE_CLIENT_SECRET=<google-oauth-client-secret>

Grafana adds GRAFANA_CLIENT_SECRET on this Dex path; see Prometheus. Do not put Grafana break-glass admin credentials here. Argo CD is not a Dex client.

Immich OAuth client secrets are not stored under homelab/dex. Each Immich instance uses Vault KV secret/homelab/immich/immich-<instance> property OAUTH_CLIENT_SECRET, synced by ESO into the Immich config Secret. Non-secret clientId lives in the private ExternalSecret template. See Immich OAuth.

See Dex OIDC for how ESO syncs this into the dex namespace.

Anything that needs to read these values through ESO must be allowed by Vault policy on that path.

Deploying Vault on Kubernetes
#

Install Vault with Helm or your GitOps tool. Standalone with a persistent volume is the usual choice for production-like data retention. Internal-only clusters often use plain HTTP to the Vault Service; use TLS if the API is exposed or policy requires it. The sidecar injector is optional; disable it if you only sync via External Secrets Operator. The UI is optional, for operators. Pin chart and Vault versions to match your Kubernetes version (kubeVersion in the Helm chart).

If you deploy Vault via Argo CD app-of-apps, register it as a child application:

kubectl apply -k argocd
kubectl apply -f clusters/<cluster>/argocd/app-of-apps.yaml
kubectl apply -f clusters/<cluster>/argocd/applications/hashicorp-vault.yaml

Prerequisites: push homelabs to the branch used by targetRevision, push homelabs-private, and ensure Argo CD has private repo credentials.

Check status:

kubectl -n argocd get applications
kubectl -n argocd get application hashicorp-vault -o yaml

Bootstrap after install
#

  1. Initialize and unseal Vault (HashiCorp quick start). Protect unseal keys and root token.
  2. Enable KV v2 on the mount your ClusterSecretStore will use. A fresh Vault has no secret/ engine until you create it; without this, vault kv put secret/... returns 403 and ESO reports missing secrets:
vault secrets enable -path=secret kv-v2
vault secrets list   # expect secret/  kv

The mount path must match what External Secrets Operator expects (path: secret, version: v2 in the ClusterSecretStore).

  1. Write secrets at paths your applications will reference from ExternalSecret resources:
vault kv put secret/<env>/<app-name> <KEY>=<value> [<KEY2>=<value2> …]
  1. Define a policy that grants read (and list if needed) on exactly the KV paths ESO should read. Attach that policy to a token or identity used by ESO (see the External Secrets Operator page for storing the token in Kubernetes).
  2. Prefer narrow policies per environment or service family; avoid a single token that can read the entire KV tree unless operational needs justify it.

Operations
#

kubectl -n <vault-namespace> get pods
# Confirm unsealed; spot-check: vault kv get secret/<path>

If Vault is sealed, External Secrets Operator cannot sync; ClusterSecretStore shows InvalidProviderConfig (HTTP 503 / “Vault is sealed”) until you unseal. The full day-2 recovery path after a node or cluster restart is below; ESO-side force-sync and stuck-pod remount steps live on the External Secrets Operator page.

After a node or cluster restart
#

Standalone Vault with Shamir seal and file storage on a PVC comes back Initialized=true, Sealed=true after a node reboot, pod eviction, or cluster restart. The secret data on the PVC is not lost; only the seal state is. Until you unseal, every ExternalSecret that uses the Vault ClusterSecretStore fails with SecretSyncedError, and Argo CD apps that depend on those resources can show Degraded even when the Kubernetes Secret objects still exist (deletionPolicy: Retain). The Vault KV values themselves were not deleted.

Confirm the seal state:

kubectl -n vault get pods
kubectl -n vault exec hashicorp-vault-0 -- vault status
# expect Initialized: true, Sealed: true
# Unseal Progress 0/<threshold> (example: threshold 3 of 5 Shamir shares)

Unseal with the threshold number of different unseal key shares (never commit keys or the root token). Repeat the command until Sealed is false:

kubectl -n vault exec -it hashicorp-vault-0 -- vault operator unseal
# repeat until Sealed: false

Confirm Vault and the cluster-wide store are healthy again:

kubectl -n vault exec hashicorp-vault-0 -- vault status
# Sealed: false
kubectl -n vault get pods
# hashicorp-vault-0 Ready 1/1
kubectl get clustersecretstore vault
# READY=True

Then force External Secrets Operator to reconcile (refresh intervals can be long after an outage) and remount any pods that started before secrets were ready; see External Secrets Operator, after a node or cluster restart.

If the store is Ready but an ExternalSecret still errors with Secret does not exist, the KV path is missing (or the field names do not match). Write the path once, then wait for ESO to refresh (or annotate the ExternalSecret to force reconcile). Example for Dex (values from your password manager; do not commit them):

vault kv put secret/homelab/dex \
  GITHUB_CLIENT_ID=… \
  GITHUB_CLIENT_SECRET=… \
  GOOGLE_CLIENT_ID=… \
  GOOGLE_CLIENT_SECRET=…
vault kv put secret/homelab/plex PLEX_CLAIM=…

After unseal, if ClusterSecretStore stays False for several minutes, restart the ESO controller so it leaves error backoff:

kubectl -n external-secrets-system rollout restart deployment/external-secrets-operator

Optional later improvement: transit / cloud KMS auto-unseal so day-2 does not require manual shares after every restart.

Adding a new application
#

Choose a Vault path and field names (see naming above). Write the secret in Vault and extend the ESO token’s policy if the path was not covered before. In the app’s namespace, add an ExternalSecret that maps those paths and properties to the Kubernetes Secret keys your Deployment or Helm chart expects (External Secrets Operator).

No change to Vault’s Helm values is required for each new app unless you add new mounts or auth methods.

Related#

References
#