External Secrets Operator

External Secrets Operator watches ExternalSecret resources and materializes ordinary Kubernetes Secret objects from an upstream backend. With HashiCorp Vault as that backend, the same pattern works for any application: store values in Vault, declare mappings in Git, and mount or env-inject the resulting Secret as usual.

Role in the stack

#

flowchart LR
  subgraph git["Git"]
    CSS[ClusterSecretStore]
    X[ExternalSecret per app]
  end
  subgraph cluster["Cluster"]
    ESO[External Secrets Operator]
    V[Vault KV v2]
    KS[Kubernetes Secret]
    P[Pods]
  end
  CSS --> ESO
  X --> ESO
  ESO -->|token auth| V
  ESO -->|reconcile| KS
  KS --> P

A cluster-scoped ClusterSecretStore is the one connection from the cluster to Vault (URL, mount, auth); every namespace can reference it. A namespace-scoped ExternalSecret maps Vault paths and properties onto a target Secret, one per app or per secret bundle. ESO reconciles on an interval and when specs change.

Install ESO after Vault is available and unsealable, and before workloads that depend on synced secrets, or make sure those workloads tolerate a short delay until the Secret exists.

Deploying the operator

#

Install from the upstream Helm chart (or your GitOps pipeline). Set installCRDs: true so the ExternalSecret and ClusterSecretStore APIs exist. One replica is enough for a small cluster; add more if you want availability. Version-pin the chart against your Kubernetes version.

If you deploy ESO via Argo CD app-of-apps, register it as a child application:

kubectl apply -k argocd
kubectl apply -f clusters/<cluster>/argocd/app-of-apps.yaml
kubectl apply -f clusters/<cluster>/argocd/applications/external-secrets-operator.yaml

ClusterSecretStore: point ESO at Vault

#

Define a single cluster-wide store (the name is arbitrary but must match ExternalSecret references). It must match how Vault is deployed: server is the Vault API URL reachable from the ESO pods (often in-cluster Service DNS); path and version are the KV mount and v2 if you use KV v2; auth is often a token read from a Kubernetes Secret (a long-lived token with a tight Vault policy). Other auth methods (Kubernetes auth, AppRole) are supported; token auth is a common starting point.

Illustrative structure:

spec:
  provider:
    vault:
      server: http://<vault-service>.<vault-namespace>.svc.cluster.local:8200
      path: secret
      version: v2
      auth:
        tokenSecretRef:
          name: <token-secret-name>
          namespace: <eso-namespace>
          key: token

Adjust server, mount path, and secret reference to your environment.

Bootstrap: Vault token for ESO

#

Create a Kubernetes Secret in the ESO namespace holding the Vault token ESO should use:

kubectl create secret generic <token-secret-name> \
  -n <eso-namespace> \
  --from-literal=token=YOUR_VAULT_TOKEN

The token must be allowed by Vault policy to read every KV path that any ExternalSecret in the cluster will use (or use multiple stores / auth methods per environment if you split access). After the token exists and Vault is unsealed, the ClusterSecretStore should become Ready.

ExternalSecret: per-application pattern

#

For each application (or each distinct secret bundle), add an ExternalSecret in the same namespace as the workload:

remoteRef.key is the KV path (for KV v2, the path segment after the mount, e.g. prod/myapp). remoteRef.property is the field name inside that secret. They must match what you wrote with vault kv put.

Dex reads GitHub OAuth credentials from Vault path homelab/dex via homelabs/dex/manifests/externalsecret.yaml. See Dex OIDC.

Reference the generated Secret in your Deployment, StatefulSet, or Helm chart (envFrom, env, volumes) the same way you would for any hand-created secret.

Multiple applications

#

One ClusterSecretStore can back many ExternalSecret resources in different namespaces. Each app team only needs RBAC to create ExternalSecret in their namespace; they do not need direct Vault access if operations manage Vault paths and ESO token policy. Tighten Vault policy so the ESO token can only read paths that are actually referenced in Git (or split stores/tokens per environment).

Operations

#

kubectl get clustersecretstore
kubectl get externalsecret -A
kubectl describe externalsecret -n <namespace> <name>

After a node or cluster restart

#

When Vault reseals after a restart (Shamir seal, file storage on a PVC), the ClusterSecretStore for Vault goes Ready=False with InvalidProviderConfig. Dependent ExternalSecret resources fail with messages such as ClusterSecretStore "vault" is not ready and land in SecretSyncedError. Argo CD applications that surface those resources can show Degraded even when the Kubernetes Secret objects still exist, especially when deletionPolicy: Retain kept them through the outage. Vault KV data on the PVC was not deleted; only the seal blocked the API.

Unseal Vault first and wait until the store is Ready again; see HashiCorp Vault, after a node or cluster restart. Then bring ESO back in sync:

1. Confirm the store recovered:

kubectl get clustersecretstore vault
# READY=True

2. Force External Secrets to reconcile. Refresh intervals may be as long as one hour; after a long outage, annotate every ExternalSecret so the controller does not wait:

kubectl annotate externalsecret -A --all force-sync="$(date +%s)" --overwrite

3. Confirm resources flip to synced and ready:

kubectl get externalsecret -A
# expect SecretSynced / Ready=True

4. If workloads are stuck Pending or ContainerCreating waiting on missing Secrets (common in namespaces created while Vault was sealed), delete those pods after the Secrets appear so they remount the new volumes or env:

kubectl -n <namespace> delete pod <pod-name>
# or delete the owning Deployment/StatefulSet pods so the controller recreates them

If the store stays False for several minutes after Vault is unsealed, restart the ESO controller so it leaves error backoff:

kubectl -n external-secrets-system rollout restart deployment/external-secrets-operator

Troubleshooting

#

Rotate the Vault token used by ESO

#

1. Issue a new token (or update policy + token) in Vault.
2. Update the Kubernetes token Secret ESO uses (same name and key as in tokenSecretRef).
3. ESO reconciles automatically; restart ESO pods if you need an immediate reconnect.