Hello, I hope you are well and safe, I don't know if you scrape Metrics from your Kubernetes workloads dynamically using Consul Service Catalog as Service Discovery Method.

I would like to mention that all the workloads are running on Kubernetes, so in order to monitor them I am using Prometheus Operator, as a result, I have configured it to identify the pods that match specific criteria in the metadata fields.

Finally, in the following example, I will try to show you how you can define a Prometheus scraper that uses Consul Service Catalog as a Service Registry.

Let's Dive in...

- job_name: "test-consul-sd-backend"
  consul_sd_configs:
    - server: "consul-server.consul-system.svc.cluster.local:8500"
  relabel_configs:
    - source_labels: [__meta_consul_service]
      regex: .*backend.*
      action: keep
    - source_labels: [__meta_consul_service]
      target_label: job
  scrape_interval: 60s
  metrics_path: "/api/metrics"

of course, you can any of the fields found in the Prometheus documentation: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config

I hope you like the tutorial, if you do give it a thumbs up! and follow me on Twitter, also you can subscribe to my Newsletter to avoid missing any of the upcoming tutorials.

Media Attribution

I would like to thank Clark Tibbs for designing the awesome photo I am using in my posts.

Thank you, Cheers!!!