Prometheus

Prometheus stores cluster and application time series. The supported install is prometheus-community kube-prometheus-stack (Prometheus Operator, Prometheus, Alertmanager, Grafana, node-exporter, kube-state-metrics, and the default Kubernetes rules) as an Argo CD Application, not a workstation helm install.

Git in homelabs and homelabs-private is the desired state; Argo CD applies it. Before the first sync, seed Vault, apply the AppProject if it lives outside app-of-apps, and publish Grafana through the existing Cloudflare Tunnel. Merging a PR is not enough on its own: without those live pieces the Application will not become Healthy.

This layout pins chart 88.3.0 on the Application targetRevision as a worked example. Patch bumps within 88.x are the same install path; a new major needs a deliberate re-pin.

What the stack is for

#

Grafana is the operator UI. Publish it at grafana.<your-domain> through the existing Cloudflare Tunnel and log in with Dex (GitHub). Prometheus and Alertmanager stay ClusterIP only: no public hostname, no Ingress, no tunnel origin.

Immich does not get its own Prometheus. One stack in namespace monitoring scrapes every instance’s chart ServiceMonitor. Longhorn’s chart ServiceMonitor is on for the same reason, and Navidrome enables native /metrics the same way. See Immich metrics for the Immich side. Do not publish a dedicated metrics hostname for Navidrome.

GitOps layout

#

The Application is multi-source, the same pattern as Longhorn: upstream chart plus public values plus a private overlay, then extra git paths for dashboards, rules, and ExternalSecrets.

Public values set Grafana to ClusterIP, Dex OIDC skeleton hostnames (grafana.example.com / dex.example.com), the password login form off, enableAdminAPI: false, 15-day Prometheus retention, cluster-wide ServiceMonitor selectors, k3s in-process control-plane scrapes off, kubelet Service in monitoring, CoreDNS chart Service disabled, and the Grafana sidecar looking for ConfigMaps labeled grafana_dashboard=1. The private overlay replaces example.com, wires Grafana OIDC to the ESO secret, maps Grafana Admin to the cluster admin email, and may shrink the Prometheus PVC.

Do not put live hostnames, admin emails, or secrets in homelabs/prometheus/. Site values live in homelabs-private/clusters/<cluster>/config/cluster.yaml (grafana.hostname, github.admin_email) and the overlay.

Sync wave is 0 (after Longhorn -1 so longhorn-ssd exists, before Immich so ServiceMonitor CRDs exist). CreateNamespace=true and ServerSideApply=true; the Operator install creates CRDs.

AppProject

#

Allow the Helm repo https://prometheus-community.github.io/helm-charts and destination namespace monitoring. Do not add kube-system as a destination for this app. The kubelet Service is created in monitoring; the CoreDNS Service from the chart is disabled because that template hardcodes kube-system.

The homelab AppProject is applied outside app-of-apps. Committing the YAML is not enough. Apply the live project before you expect Argo to accept the Application:

kubectl apply -f homelabs-private/argocd/projects/homelab.yaml

Without the live destination, sync fails because namespace monitoring is not an allowed destination in project homelab.

First sync

#

Do this after the public homelabs tree for Prometheus, Immich metrics, and Longhorn ServiceMonitor is on main. Order matters: secrets and the AppProject before a successful Application sync.

1. Seed Vault. Grafana admin credentials are a break-glass local login, not the day-to-day path. The Dex confidential client secret is a new field on the existing Dex KV; add it without dropping GitHub or Google keys:

vault kv put secret/homelab/monitoring/grafana \
  ADMIN_USER='admin' \
  ADMIN_PASSWORD='<grafana-admin-password>'

vault kv patch secret/homelab/dex \
  GRAFANA_CLIENT_SECRET='<grafana-dex-client-secret>'

2. Apply the AppProject (command above) so monitoring and the prometheus-community repo are allowed.
3. Commit the private Application, overlay, ExternalSecrets, Dex grafana static client, and Immich scrape NetworkPolicies. App-of-apps discovers the Application; sync prometheus in the Argo UI or with argocd app sync prometheus.
4. Add a tunnel Public Hostname for grafana.<your-domain> pointing at the Grafana ClusterIP Service. Leave Cloudflare Access off so Dex can complete login. Do not publish Prometheus or Alertmanager.
5. After the Application is Healthy, confirm CRDs and pods. Immich targets appear after those Applications sync with immich.metrics.enabled: true and the scrape NetworkPolicy in each instance tree.

kubectl -n argocd get application prometheus
kubectl -n monitoring get pods
kubectl get crd servicemonitors.monitoring.coreos.com prometheusrules.monitoring.coreos.com

Vault must be unsealed and the Grafana and Dex ExternalSecrets SecretSynced before Grafana OIDC can work. See Vault and External Secrets Operator.

Grafana login (Dex)

#

Day-to-day Grafana login is Dex only. The username/password form is off (auth.disable_login_form: true, auth.basic.enabled: false). Anonymous auth is off. Sign-up through Dex is on so the first successful OIDC login can create the Grafana user.

Dex already federates GitHub for kubectl and Argo CD. Grafana is another confidential static client (id: grafana, public: false) with redirect https://grafana.<your-domain>/login/generic_oauth. Store the shared client secret in Vault; ESO syncs it into Dex and Grafana.

On the Dex static client, set secretEnv to the environment variable name, with no $. Dex expands $VAR on GitHub and Google connectors, but it does not expand secret: $VAR on staticClients. That form makes Dex treat the dollar-sign string as the secret. Grafana then reports “Failed to get token from provider” and Dex logs invalid_client_secret.

The private overlay grants Grafana Admin only when the Dex email claim equals github.admin_email from clusters/<cluster>/config/cluster.yaml. This layout also sets role_attribute_strict: true, so unmatched Dex users are denied rather than logged in as Viewer. Public values keep a placeholder expression and must not encode the live admin email.

The Grafana admin existingSecret stays in Vault as break-glass if OIDC is misconfigured. There is no default admin/admin in git, and that user cannot use the login page while the form is disabled. To use it, temporarily set auth.disable_login_form: false in the overlay, sync, then turn the form off again.

Storage and retention

#

Prometheus, Grafana, and Alertmanager persistent volumes use StorageClass longhorn-ssd. Prometheus retention is 15 days, one replica, WAL compression on. Public values default the Prometheus claim to 50Gi, Grafana 10Gi, Alertmanager 5Gi.

A private overlay may shrink the Prometheus PVC when Longhorn SSD free space would not absorb a 50Gi volume at three replicas. Keep the 15-day retention; set prometheusSpec.retentionSize so the TSDB cannot fill the smaller claim. Do not cut retention to sneak the stack onto a tight pool; expand SSDs instead. Grafana and Alertmanager sizes stay unless they themselves trip the free-space gate. See Longhorn for the capacity habit (leave headroom on every ssd disk; one new claim at a time).

What Prometheus scrapes

#

Selectors are nil-on-Helm-values (serviceMonitorSelectorNilUsesHelmValues: false and the PodMonitor, Probe, ScrapeConfig, and rule equivalents). Prometheus therefore picks up ServiceMonitor and PrometheusRule objects in application namespaces, not only the release. Immich, Longhorn, and Navidrome monitors do not need scrape YAML in the Prometheus chart.

On k3s, kube-controller-manager, kube-scheduler, etcd, and kube-proxy run in-process. Public values disable those scrapes and the matching defaultRules so Alertmanager is not flooded with TargetDown. kube-apiserver, kubelet, CoreDNS (ServiceMonitor in monitoring only), node-exporter, and kube-state-metrics stay on.

Any GitOps namespace can emit a ServiceMonitor this Prometheus will scrape. Do not grant application ServiceAccounts permission to create servicemonitors. Homelab is single-tenant GitOps.

Immich and Longhorn

#

Shared Immich values set immich.metrics.enabled: true. That is the chart’s native OpenTelemetry → Prometheus path plus a ServiceMonitor. Do not add a custom OTLP sidecar. Each instance NetworkPolicy must allow the metrics ports from Prometheus in the monitoring namespace only. Details and the crash-loop warning for NODE_OPTIONS live on the Immich page.

Longhorn public values set metrics.serviceMonitor.enabled: true so disk and volume metrics land in the same Prometheus.

Dashboards ship as ConfigMaps in homelabs/prometheus/manifests/ (Immich overview, adapted from the public Kubernetes dashboard, plus a smaller golden-signal board). The Grafana sidecar loads them because they are labeled grafana_dashboard=1. Immich PrometheusRules (scrape down, HTTP 5xx ratio, API p95 latency, server restart rate) live next to those ConfigMaps. Alerts are warning severity in Alertmanager and Grafana. The stack does not configure email or SMS receivers. Treat the first scrape window as a soak for PromQL names; empty panels are possible until Immich has emitted samples.

Rollback

#

Revert git rather than helm uninstall. Disable or delete Application prometheus, set immich.metrics.enabled: false if you must stop exporters, and drop the scrape NetworkPolicies if you are tearing the path down. Leave the Prometheus/Grafana/Alertmanager PVCs until you no longer need the TSDB; prune them only after the Application is gone so a re-sync does not recreate empty volumes you meant to keep. The Dex grafana client can stay unused; remove the static client and Vault keys after Grafana is gone.

Immich user HTTP does not depend on metrics ports. Turning metrics off does not change library traffic.

See also

#

• Dex OIDC: Grafana confidential client and GitHub login
• Cloudflare Tunnel: Grafana hostname
• Longhorn: longhorn-ssd and ServiceMonitor
• Immich: native metrics, scrape policy, no OTLP sidecar
• Navidrome: native /metrics in-cluster, no public metrics hostname