How to integrate your standalone Prometheus with Istio-enabled pods as mentioned briefly in the
Istio docs here.
Istio 1.5 comes with a new control plane architecture, but still allows for the easy enabling of Kiali, Prometheus, Grafana and so on. Those built-in components are not production-ready and usually only used for monitoring istio-system components.
If you want to set up your own Prometheus in a separate namespace, you need a way to collect metrics from Istio-enabled pods. The easy version is to set your global mTLS to PERMISSIVE, allowing both encrypted and unencrypted traffic between your application and system pods. Prometheus will then simply get pod metrics without encrypting the requests.
This approach isn’t ideal for a few reasons, so this blog shows how to encrypt the scrape jobs with certificates requested from Istio itself.
The idea is simple, add a sidecar to Prometheus that requests certificated from Istio and then doesn't catch and encrypt all traffic for you, but instead let's the Prometheus pod use the certificates to encrypt the scrape requests itself.
1. If your Prometheus is in an istio-injection:enabled namespace, turn of the automatic sidecar injection by adding
to the prometheus pod deployment spec.
2. Manually add a an istio-proxy sidecar to your Prometheus pod.
Add the following volumes to your pod
3. Add scrape rule for Istio pods (usually in the Prometheus configmap):
The manually injected sidecar will request a certificate and key from istiod and save those into /etc/istio-certs/ and this part
from step 3 will then be used to configure TLS for the scrape requests.
Now you should have your own Prometheus scraping metrics with encrypted traffic.
If the Prometheus or Istio version change, check out the Prometheus that comes with Istio and how the sidecar of that instance requests certificates and uses them.