Cookies & Containers

Or how I learned to love the cloud.

Bring Your Own Prometheus to Istio

Bring Your Own Prometheus to Istio

June 12, 2020 — How to integrate your standalone Prometheus with Istio-enabled Pods as mentioned briefly in the Istio docs here.

2020-06-25-adventures-telco-xos.html · 2020-06-12-byo-prometheus.html · 2020-05-17-co2-sensor-raspi.html

Istio 1.5 comes with a new control plane architecture, but still allows for the easy enabling of Kiali, Prometheus, Grafana and so on. Those built-in components are not production-ready and usually only used for monitoring istio-system components.

If you want to set up your own Prometheus in a separate namespace, you need a way to collect metrics from Istio-enabled Pods. The easy version is to set your global mTLS to PERMISSIVE, allowing both encrypted and unencrypted traffic between your application and system Pods. Prometheus will then simply get Pod metrics without encrypting the requests.

This approach isn’t ideal for a few reasons, so this blog shows how to encrypt the scrape jobs with certificates requested from Istio itself. The idea is simple, add a sidecar to Prometheus that requests certificated from Istio and then doesn't catch and encrypt all traffic for you, but instead let's the Prometheus Pod use the certificates to encrypt the scrape requests itself.

Custom Sidecar

If your Prometheus is in an istio-injection:enabled namespace, turn off the automatic sidecar injection by adding

template: metadata: annotations: "false"

to the prometheus Pod deployment spec.

Manually add a an istio-proxy sidecar to your Prometheus Pod.

- name: istio-proxy image: imagePullPolicy: IfNotPresent args: - proxy - sidecar - --domain - $(POD_NAMESPACE).svc.cluster.local - --configPath - /etc/istio/proxy - --binaryPath - /usr/local/bin/envoy - --serviceCluster - istio-proxy-prometheus - --drainDuration - 45s - --parentShutdownDuration - 1m0s - --discoveryAddress - istio-pilot.istio-system.svc:15012 - --proxyLogLevel=warning - --proxyComponentLogLevel=misc:error - --connectTimeout - 10s - --proxyAdminPort - "15000" - --controlPlaneAuthPolicy - NONE - --dnsRefreshRate - 300s - --statusPort - "15020" - --trust-domain=cluster.local - --controlPlaneBootstrap=false env: - name: OUTPUT_CERTS value: /etc/istio-certs - name: JWT_POLICY value: third-party-jwt - name: PILOT_CERT_PROVIDER value: istiod - name: CA_ADDR value: istio-pilot.istio-system.svc:15012 - name: POD_NAME valueFrom: fieldRef: fieldPath: - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: INSTANCE_IP valueFrom: fieldRef: fieldPath: status.podIP - name: SERVICE_ACCOUNT valueFrom: fieldRef: fieldPath: spec.serviceAccountName - name: HOST_IP valueFrom: fieldRef: fieldPath: status.hostIP - name: ISTIO_META_POD_NAME valueFrom: fieldRef: fieldPath: - name: ISTIO_META_CONFIG_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: ISTIO_META_MESH_ID value: cluster.local - name: ISTIO_META_CLUSTER_ID value: Kubernetes ports: - containerPort: 15090 name: http-envoy-prom protocol: TCP readinessProbe: failureThreshold: 30 httpGet: path: /healthz/ready port: 15020 scheme: HTTP initialDelaySeconds: 1 periodSeconds: 2 successThreshold: 1 timeoutSeconds: 1 volumeMounts: - mountPath: /var/run/secrets/istio name: istiod-ca-cert - mountPath: /etc/istio/proxy name: istio-envoy - mountPath: /var/run/secrets/tokens name: istio-token - mountPath: /etc/istio-certs/ name: istio-certs

Add the following volumes to your Pod

- name: istio-cert emptyDir: medium: Memory - name: istio-envoy emptyDir: medium: Memory - name: istio-token projected: defaultMode: 420 sources: - serviceAccountToken: audience: istio-ca expirationSeconds: 43200 path: istio-token - name: istiod-ca-cert configMap: defaultMode: 420 name: istio-ca-root-cert

Add scrape rule for Istio Pods (usually in the Prometheus configmap):

- job_name: 'kubernetes-pods-istio-secure' scheme: https tls_config: ca_file: /etc/istio-certs/root-cert.pem cert_file: /etc/istio-certs/cert-chain.pem key_file: /etc/istio-certs/key.pem insecure_skip_verify: true # Prometheus does not support secure naming. kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true # sidecar status annotation is added by sidecar injector and # istio_workload_mtls_ability can be specifically placed on a Pod to indicate its ability to receive mtls traffic. - source_labels: [__meta_kubernetes_pod_annotation_sidecar_istio_io_status, __meta_kubernetes_pod_annotation_istio_mtls] action: keep regex: (([^;]+);([^;]*))|(([^;]*);(true)) - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__] # Only keep address that is host:port action: keep # otherwise an extra target with ':443' is added for https scheme regex: ([^:]+):(\d+) - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 target_label: __address__ - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: namespace

The manually injected sidecar will request a certificate and key from istiod and save those into /etc/istio-certs/ and this part

tls_config: ca_file: /etc/istio-certs/root-cert.pem cert_file: /etc/istio-certs/cert-chain.pem key_file: /etc/istio-certs/key.pem insecure_skip_verify: true

from step 3 will then be used to configure TLS for the scrape requests.

Now you should have your own Prometheus scraping metrics with encrypted traffic.


If the Prometheus or Istio version change, check out the Prometheus that comes with Istio and how the sidecar of that instance requests certificates and uses them.

View source

Gmail icon Twitter icon GitHub icon
Built with Scroll v31.4.1