Dex OIDC

Dex is an OpenID Connect (OIDC) identity broker. It federates a GitHub login into standard OIDC ID tokens that the Kubernetes API server trusts. Combined with the existing Cloudflare Tunnel and the kubelogin (kubectl oidc-login) plugin, running kubectl triggers a GitHub browser login and only verified identities receive a valid bearer token.

The API server itself stays private on https://<api-server-host>:6443 (see homelabs-private/clusters/<cluster>/config/cluster.yaml). Only the Dex OIDC endpoint is published through the tunnel. No router ports are opened.

Two channels: identity and network

#

Remote access uses two separate paths:

Identity is verified before the API server accepts a request: kubelogin must obtain an ID token from Dex (via GitHub) on every fresh login. Network access alone is not enough: the API server rejects unsigned or expired tokens.

How the pieces fit

#

flowchart TB
  subgraph local["Workstation"]
    K[kubectl]
    KL[kubelogin]
    B[Browser]
    VPN[Tailscale / WireGuard / LAN]
  end
  subgraph cf["Cloudflare edge"]
    T[(Tunnel: dex.your-domain)]
  end
  subgraph cluster["Home cluster (private)"]
    CFD[cloudflared]
    D[Dex]
    API[k3s API server
private :6443]
  end
  GH[(GitHub OAuth)]

  K -->|exec credential plugin| KL
  KL -->|opens| B
  B -->|OIDC authorize| T --> CFD --> D
  D -->|federate| GH
  GH -->|code| D
  D -->|ID token| KL
  KL -->|bearer token| K
  K -->|HTTPS API call| VPN --> API
  API -->|fetch JWKS / discovery| T

1. kubectl invokes kubelogin as a client-go credential plugin.
2. kubelogin opens the browser to Dex at https://dex.<your-domain> (reached through the tunnel).
3. Dex federates the login to GitHub; GitHub returns the user’s identity (and optionally org/team membership).
4. Dex mints an OIDC ID token (with email and optionally groups claims) and hands it back to kubelogin.
5. kubectl sends the ID token to the private API server over VPN/LAN. The API server validates iss and the signature against Dex’s discovery/JWKS endpoints (fetched via egress to the same public Dex URL), then applies RBAC.

Cloudflare Tunnel as the stealth bridge

#

The cluster already runs cloudflared (namespace cloudflare-tunnel) as a token-managed tunnel. See Cloudflare Tunnel for deployment and hostname setup.

Add a Public Hostname for Dex in the Cloudflare Zero Trust dashboard:

Cloudflare terminates TLS at the edge with a valid public certificate, so the API server needs no oidc-ca-file. Do not put Cloudflare Access in front of the Dex hostname; the OIDC endpoints must be reachable directly for discovery, token, and JWKS calls.

Verify discovery through the tunnel:

curl -s https://dex.<your-domain>/.well-known/openid-configuration | jq .issuer

The issuer value must match the API server oidc-issuer-url exactly.

GitHub OAuth App

#

Create a GitHub OAuth App under the account or org that owns the cluster login:

• Personal account: GitHub user → Settings → Developer settings → OAuth Apps
• Organization: org → Settings → Developer settings → OAuth Apps

URLs (replace <your-domain>):

• Homepage URL: https://dex.<your-domain>
• Authorization callback URL: https://dex.<your-domain>/callback

Store the credentials in Vault so External Secrets Operator can sync them into the dex namespace:

vault kv put secret/homelab/dex \
  GITHUB_CLIENT_ID=<oauth-client-id> \
  GITHUB_CLIENT_SECRET=<oauth-client-secret> \
  GOOGLE_CLIENT_ID=<google-oauth-client-id> \
  GOOGLE_CLIENT_SECRET=<google-oauth-client-secret>

Grafana is a confidential Dex client (GRAFANA_CLIENT_SECRET on this path); see Prometheus. Patch that field onto this KV rather than replacing GitHub and Google keys. Grafana’s break-glass admin user is a different Vault path on the same Prometheus page. Argo CD is not a Dex client.

The ExternalSecret in homelabs/dex/manifests/externalsecret.yaml materializes GitHub and Google keys into a Kubernetes Secret named dex-github (owned by the ExternalSecret). The Dex Helm release reads that Secret via envVars / secretKeyRef. Connectors expand $GITHUB_CLIENT_ID / $GOOGLE_CLIENT_SECRET style placeholders. Day-2 source of truth is Vault; do not maintain a parallel hand-applied dex-github Secret.

Immich does not use Dex as its OIDC issuer; see Immich OAuth (Google). Do not put Immich Google client secrets in homelab/dex.

Break-glass (Vault/ESO down only): copy homelabs/dex/manifests/secret-dex-github.yaml.template to a local file, fill values, and kubectl apply once. Do not commit the filled file. When Vault is healthy again, ensure vault kv put secret/homelab/dex … matches, wait until kubectl -n dex get externalsecret dex-github is Ready=True, then let ESO own the Secret (no separate manual copy).

Google OAuth

#

Dex can still federate Google for apps that use Dex as issuer (optional connector). Immich uses Google directly and needs its own Google Cloud OAuth Web client with Immich redirect URIs, not the Dex callback.

For a Dex-brokered Google connector:

1. Google Cloud Console → APIs & Services → Credentials → Create OAuth client ID → Web application.
2. Authorized redirect URI: https://dex.<your-domain>/callback.
3. Store GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET in Vault path homelab/dex (see command above).

Optional: restrict which Google accounts can log in with hostedDomains on the Google connector in overlays/dex/values.yaml. Publish the OAuth consent screen when leaving Testing mode.

Repo layout (public vs private)

#

General rules follow the standard public/private repo split. Dex-specific split:

Dex deployment (GitOps)

#

Dex is an Argo CD application (clusters/<cluster>/argocd/applications/dex.yaml) with multiple sources:

1. Upstream Dex Helm chart + private overlays/dex/values.yaml
2. Public homelabs/dex/manifests (namespace, ExternalSecret)
3. Private manifests/oidc (RBAC only)

Ensure the Argo CD AppProject allow-list includes the dex namespace and https://charts.dexidp.io; otherwise the Application will fail with InvalidSpecError.

Production values (overlays/dex/values.yaml) set issuer https://dex.<your-domain> (must match the API server oidc-issuer-url exactly), kubernetes storage so refresh tokens survive restarts, and idTokens: "8h" plus refreshTokens (reuseInterval: "5m", idle 168h, absolute 720h) so kubelogin can refresh silently. A longer ID token and reuse window cut down browser re-logins from concurrent kubectl / IDE clients. The GitHub connector may include an org filter depending on whether the OAuth app owner is a personal account or a GitHub Organization (see RBAC mapping). An optional Google connector is for apps that federate Google through Dex (Immich does not; see Immich OAuth). The kubernetes static client is declared public so kubelogin uses PKCE (no client secret on the workstation).

Additional confidential static clients live under staticClients in overlays/dex/values.yaml (production hostnames). Public homelabs/dex/values.yaml only shows the kubernetes PKCE client. Grafana adds a client; see Prometheus. Argo CD does not; its UI is ClusterIP plus port-forward. See Argo CD.

For confidential clients, set secretEnv to the environment variable name (no $) and wire that name in Dex envVars. Dex expands $VAR on GitHub and Google connectors; it does not expand secret: $VAR on staticClients. That form makes Dex treat the dollar-sign string as the secret (invalid_client_secret).

For bootstrap before Argo CD owns Dex, or when Vault/ESO is unavailable:

cd homelabs-private/clusters/<cluster>
./scripts/apply-oidc.sh   # pre-Argo CD only; day-2 use argocd app sync dex
./scripts/patch-k3s-apiserver-oidc.sh

Interim Google OAuth keys on dex-github (break-glass only): copy manifests/bootstrap/dex-github-oauth.yaml.template, fill values, kubectl apply once. Prefer Vault + ESO.

Kubernetes API server (k3s)

#

OIDC flags on the API server (replace <your-domain>):

kube-apiserver-arg:
  - "oidc-issuer-url=https://dex.<your-domain>"
  - "oidc-client-id=kubernetes"
  - "oidc-username-claim=email"
  - "oidc-username-prefix=oidc:"
  - "oidc-groups-claim=groups"
  - "oidc-groups-prefix=oidc:"

These belong in /etc/rancher/k3s/config.yaml on each control-plane node. Do not confuse this with /etc/rancher/k3s/k3s.yaml; that file is the admin kubeconfig (break-glass client certificates), not the server configuration.

The flags must be a YAML list under kube-apiserver-arg:, not bare --kube-apiserver-arg=... lines.

On a fresh cluster (Ansible), generic placeholders live in homelabs/kubernetes/k3s-ansible/inventory/my-cluster/group_vars/all.yml (oidc_issuer_url: https://dex.example.com). Override per cluster in homelabs-private/clusters/<cluster>/ansible/group_vars/all.yml. On a live cluster (day-2), ./scripts/patch-k3s-apiserver-oidc.sh writes /etc/rancher/k3s/config.yaml on each control-plane node and restarts k3s.

The oidc: prefixes namespace OIDC identities so they cannot collide with built-in users or groups.

Keep the legacy client-cert kubeconfig as a break-glass path until the OIDC flow is confirmed.

RBAC mapping (personal account vs organization)

#

Personal account (no org filter)

#

When the OAuth app owner is a personal GitHub account (not a GitHub Organization), Dex typically runs without an orgs filter. Any user authorized by the OAuth app can authenticate. RBAC binds cluster-admin to the admin user’s GitHub primary email (must match exactly):

File: homelabs-private/clusters/<cluster>/manifests/oidc/rbac.yaml.

After login, confirm:

kubectl auth whoami
# Username: oidc:<admin-email>

A personal GitHub username and a GitHub Organization can share the same slug (e.g. user example vs org example). If Dex is configured with an orgs filter but the OAuth app is owned by the personal account, authentication fails. Remove the org filter for personal-account setups.

GitHub Organization + teams

#

When you use a GitHub Organization and teams (k8s-admins, k8s-readonly):

1. Re-enable the org filter in overlays/dex/values.yaml (see homelabs/dex/values.yaml for the template).
2. Replace the email User binding with group bindings from homelabs/dex/manifests/rbac.yaml.template:

Example org filter in overlays/dex/values.yaml:

orgs:
  - name: your-github-org
    teams:
      - k8s-admins
      - k8s-readonly
teamNameField: slug
loadAllGroups: false

Workstation setup

#

1. Private network path

#

Join the home network overlay (Tailscale or WireGuard) or work from LAN so https://<api-server-host>:6443 is reachable. The Cloudflare Tunnel does not expose the Kubernetes API.

2. Install kubectl and kubelogin

#

# kubectl (example for Linux amd64)
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl && sudo mv kubectl /usr/local/bin/

# kubelogin via krew
kubectl krew install oidc-login

3. Configure kubeconfig

#

Start from homelabs-private/clusters/<cluster>/kubeconfig-oidc.yaml.template. Replace <certificate-authority-data> with the cluster CA:

# From a machine that already has LAN access to the cluster
kubectl config view --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}'

Copy the template to ~/.kube/config (or merge contexts). Prefer a single ~/.kube/config with one OIDC context as the default; remove stray copies elsewhere.

The exec plugin block:

users:
  - name: oidc-dex
    user:
      exec:
        apiVersion: client.authentication.k8s.io/v1
        command: kubectl
        args:
          - oidc-login
          - get-token
          - --oidc-issuer-url=https://dex.<your-domain>
          - --oidc-client-id=kubernetes
          - --oidc-pkce-method=S256
          - --oidc-extra-scope=email
          - --oidc-extra-scope=groups
          - --oidc-extra-scope=offline_access
contexts:
  - name: <cluster-context>
    context:
      cluster: <cluster-name>
      user: oidc-dex

kubelogin listens on http://localhost:8000 for the OAuth callback, which is why that URL is registered as a redirect URI on the kubernetes static client.

Keep any existing cert-based context (for example legacy-cert-admin) as break-glass.

The login flow

#

kubectl config use-context <cluster-context>
kubectl oidc-login clean   # clear any cached token
kubectl get nodes

Running kubectl get nodes:

1. kubelogin opens the browser to Dex (via the tunnel).
2. Dex redirects to GitHub; you approve access.
3. Dex returns an ID token; kubelogin caches it.
4. kubectl sends the token to the private API server over VPN/LAN.

Subsequent commands reuse the cached token until it expires, then refresh silently (no browser) when a refresh token is available.

Confirm the claims and your effective access:

kubectl oidc-login setup \
  --oidc-issuer-url=https://dex.<your-domain> \
  --oidc-client-id=kubernetes \
  --oidc-pkce-method=S256
kubectl auth whoami
kubectl auth can-i '*' '*' --all-namespaces

Post-migration cutover

#

After the first successful OIDC login, run a short confidence window before retiring cert-based daily use:

1. Use the OIDC context exclusively for normal work; keep the cert kubeconfig as break-glass elsewhere.
2. Confirm kubectl auth whoami shows oidc:<admin-email> (personal account) or the expected group (org mode).
3. Confirm kubectl auth can-i '*' '*' --all-namespaces for admin users.
4. Optionally wait 10+ minutes idle, then run kubectl get nodes; token refresh should succeed without opening the browser.
5. Merge OIDC into ~/.kube/config and set it as the default context when satisfied.

Health checks

#

kubectl config current-context
kubectl auth whoami
kubectl get nodes
kubectl -n dex get pods
kubectl get application dex -n argocd
curl -s https://dex.<your-domain>/.well-known/openid-configuration | jq .issuer

Expected outcomes:

Verify and troubleshoot

#

# Discovery reachable through the tunnel
curl -s https://dex.<your-domain>/.well-known/openid-configuration | jq .issuer

# Dex pod and synced secret
kubectl -n dex get pods
kubectl -n dex get externalsecret dex-github
kubectl -n dex get secret dex-github

# API server reachable on private path
curl -k https://<api-server-host>:6443/readyz