External Secrets Operator

External Secrets Operator watches ExternalSecret resources and materializes ordinary Kubernetes Secret objects from an upstream backend. With HashiCorp Vault as that backend, the same pattern works for any application: store values in Vault, declare mappings in Git, and mount or env-inject the resulting Secret as usual.

Role in the stack

#

flowchart LR
  subgraph git["Git"]
    CSS[ClusterSecretStore]
    X[ExternalSecret per app]
  end
  subgraph cluster["Cluster"]
    ESO[External Secrets Operator]
    V[Vault KV v2]
    KS[Kubernetes Secret]
    P[Pods]
  end
  CSS --> ESO
  X --> ESO
  ESO -->|token auth| V
  ESO -->|reconcile| KS
  KS --> P

A cluster-scoped ClusterSecretStore is the one connection from the cluster to Vault (URL, mount, auth); every namespace can reference it. A namespace-scoped ExternalSecret maps Vault paths and properties onto a target Secret, one per app or per secret bundle. ESO reconciles on an interval and when specs change.

Install ESO after Vault is available and unsealable, and before workloads that depend on synced secrets, or make sure those workloads tolerate a short delay until the Secret exists.

Deploying the operator

#

Install from the upstream Helm chart (or your GitOps pipeline). Set installCRDs: true so the ExternalSecret and ClusterSecretStore APIs exist. One replica is enough for a small cluster; add more if you want availability. Version-pin the chart against your Kubernetes version.

If you deploy ESO via Argo CD app-of-apps, register it as a child application:

kubectl apply -k argocd
kubectl apply -f clusters/<cluster>/argocd/app-of-apps.yaml
kubectl apply -f clusters/<cluster>/argocd/applications/external-secrets-operator.yaml

ClusterSecretStore: point ESO at Vault

#

Define a single cluster-wide store (the name is arbitrary but must match ExternalSecret references). It must match how Vault is deployed: server is the Vault API URL reachable from the ESO pods (often in-cluster Service DNS); path and version are the KV mount and v2 if you use KV v2; auth is often a token read from a Kubernetes Secret (a long-lived token with a tight Vault policy). Other auth methods (Kubernetes auth, AppRole) are supported; token auth is a common starting point.

Illustrative structure:

spec:
  provider:
    vault:
      server: http://<vault-service>.<vault-namespace>.svc.cluster.local:8200
      path: secret
      version: v2
      auth:
        tokenSecretRef:
          name: <token-secret-name>
          namespace: <eso-namespace>
          key: token

Adjust server, mount path, and secret reference to your environment. This layout uses Secret vault-eso-token in namespace external-secrets-system, key token. Do not put the Vault root token in that Secret. Root is only for init, unseal, and minting the ESO token.

Mint a read-only Vault token for ESO

#

Token auth is a static credential. It expires. The ClusterSecretStore goes InvalidProviderConfig when the token is gone or past TTL, and every ExternalSecret stops syncing. Keep the token in a password manager; Git never stores it.

Log into Vault with the root token (or another admin token), then write a policy that can only read KV v2 under secret/homelab/ (the prefix every ExternalSecret in this layout uses):

kubectl -n vault exec -it statefulset/hashicorp-vault -- vault login

kubectl -n vault exec -i statefulset/hashicorp-vault -- vault policy write eso-read - <<'EOF'
path "secret/data/homelab/*" {
  capabilities = ["read", "list"]
}
path "secret/metadata/homelab/*" {
  capabilities = ["read", "list"]
}
path "sys/mounts" {
  capabilities = ["read"]
}
EOF

kubectl -n vault exec -it statefulset/hashicorp-vault -- \
  vault token create -policy=eso-read -orphan -ttl=8760h -display-name=eso-read

-ttl=8760h is one year from issue time. Calendar that date. Create the Kubernetes Secret (or replace it) with the printed token value, not root:

kubectl create secret generic vault-eso-token \
  -n external-secrets-system \
  --from-literal=token=YOUR_ESO_READ_TOKEN

If the Secret already exists:

kubectl create secret generic vault-eso-token \
  -n external-secrets-system \
  --from-literal=token=YOUR_ESO_READ_TOKEN \
  --dry-run=client -o yaml | kubectl apply -f -

Confirm the store:

kubectl get clustersecretstore vault
# READY=True, STATUS Valid

If it stays Invalid for a few minutes, restart the controller so it leaves backoff:

kubectl -n external-secrets-system rollout restart deployment/external-secrets-operator

If you add a Vault path outside secret/homelab/…, extend eso-read and mint a new token (policy changes do not apply to tokens already issued).

Token expiry and rotation

#

ESO does not renew tokenSecretRef. A one-year orphan token dies at TTL; there is no in-tree auto-refresh for this auth mode.

Manual (what you run today): before TTL, mint another eso-read token, kubectl apply it onto vault-eso-token, optionally restart ESO. Same commands as mint a read-only token.

Periodic token plus CronJob (vault token create -period=… and vault token renew on a schedule) only works if something renews before the period elapses. ESO will not do that for you. A CronJob that shells into Vault with root to renew is extra secret surface; skip it.

Preferred auto-rotation: Vault Kubernetes auth so ESO never stores a long-lived Vault token. Enable the auth method, create a role bound to ServiceAccount external-secrets-operator in namespace external-secrets-system with policy eso-read and a short ttl (for example one hour). Point ClusterSecretStore at auth.kubernetes and that role instead of tokenSecretRef. Vault issues a new client token whenever ESO logs in; Kubernetes rotates the projected SA JWT. After that, you can delete vault-eso-token. That store change is a GitOps edit to homelabs/external-secrets-operator/manifests/cluster-secret-store.yaml, not a yearly Secret rewrite.

Until Kubernetes auth is wired, treat the static token as a dated credential: password manager + a reminder before 8760h.

ExternalSecret: per-application pattern

#

For each application (or each distinct secret bundle), add an ExternalSecret in the same namespace as the workload:

remoteRef.key is the KV path (for KV v2, the path segment after the mount, e.g. prod/myapp). remoteRef.property is the field name inside that secret. They must match what you wrote with vault kv put.

Dex reads GitHub OAuth credentials from Vault path homelab/dex via homelabs/dex/manifests/externalsecret.yaml. See Dex OIDC.

Reference the generated Secret in your Deployment, StatefulSet, or Helm chart (envFrom, env, volumes) the same way you would for any hand-created secret.

Multiple applications

#

One ClusterSecretStore can back many ExternalSecret resources in different namespaces. Each app team only needs RBAC to create ExternalSecret in their namespace; they do not need direct Vault access if operations manage Vault paths and ESO token policy. Tighten Vault policy so the ESO token can only read paths that are actually referenced in Git (or split stores/tokens per environment).

Operations

#

kubectl get clustersecretstore
kubectl get externalsecret -A
kubectl describe externalsecret -n <namespace> <name>

After a node or cluster restart

#

When Vault reseals after a restart (Shamir seal, file storage on a PVC), the ClusterSecretStore for Vault goes Ready=False with InvalidProviderConfig. Dependent ExternalSecret resources fail with messages such as ClusterSecretStore "vault" is not ready and land in SecretSyncedError. Argo CD applications that surface those resources can show Degraded even when the Kubernetes Secret objects still exist, especially when deletionPolicy: Retain kept them through the outage. Vault KV data on the PVC was not deleted; only the seal blocked the API.

Unseal Vault first and wait until the store is Ready again; see HashiCorp Vault, after a node or cluster restart. Then bring ESO back in sync:

1. Confirm the store recovered:

kubectl get clustersecretstore vault
# READY=True

2. Force External Secrets to reconcile. Refresh intervals may be as long as one hour; after a long outage, annotate every ExternalSecret so the controller does not wait:

kubectl annotate externalsecret -A --all force-sync="$(date +%s)" --overwrite

3. Confirm resources flip to synced and ready:

kubectl get externalsecret -A
# expect SecretSynced / Ready=True

4. If workloads are stuck Pending or ContainerCreating waiting on missing Secrets (common in namespaces created while Vault was sealed), delete those pods after the Secrets appear so they remount the new volumes or env:

kubectl -n <namespace> delete pod <pod-name>
# or delete the owning Deployment/StatefulSet pods so the controller recreates them

If the store stays False for several minutes after Vault is unsealed, restart the ESO controller so it leaves error backoff:

kubectl -n external-secrets-system rollout restart deployment/external-secrets-operator

Troubleshooting

#

Rotate the Vault token used by ESO

#

Follow token expiry and rotation. Short version: mint a new eso-read token, replace vault-eso-token, restart ESO if the store stays Invalid.