Lately, we are working a lot with rancher and therefore we are also using the integrated rancher haproxy service, which makes it so easy to startup stacks and connect them to the outside world. But there are more complex use cases, which are very common and cannot be achieved with the rancher ui. Some of these use cases include enabling haproxy stats, securing routes with basic auth or scraping stats information with Prometheus (or any other metrics tool). In this post, we will describe how to enable haproxy stats and scrape them with Prometheus.
This post assumes that you are familiar with Grafana and Prometheus. The easiest way to get started with these tools is to spin up the Prometheus stack provided in the rancher community catalog. Let’s go through the steps.
Enable stats
Hidden in the comments of Issue 2179 in the rancher repo user teamon describes how to enable stats:
- Create a new rule as described in the picture
- Open the tab custom haproxy.cfg and the highlighted config part as shown in the picture
- Save your changes
Now you can access the stats on port 9000 with path haproxy_stats. Assuming your loadbalancer is named lb and lives in the stack mystack, the url will be
http://lb.mystack:9000/haproxy_stats
Scrape the data
There is a lot information on current traffic that is routed through your load balancer. But the UI is not the nicest and there is no aggregation. So, we wanted to scrape the data and add it to our monitoring solution. In our case this is Grafana for visualization and Prometheus as the scraping backend. The Prometheus project provides an easy to use Docker container to transform the HAProxy data to a format that Prometheus understands.
All you have to do is to create a new service in rancher with the following config:
Name: lb-stats-exporter
Image: quay.io/prometheus/haproxy-exporter
Command: -haproxy.scrape-uri="http://lb.mystack:9000/haproxy_stats;csv"
The Container will start and with these settings the transformed data will be reachable via
http://lb-stats-exporter.mystack:9101/metrics
Quite easy, isn’t it?
The last part is to add our shiny new stats to the Prometheus config in order to get them scraped. Just add this rule to the prometheus.yml (usually lives in /etc/prom-conf):
- job_name: 'haproxy'
dns_sd_configs:
- names:
- 'lb-stats-exporter.mystack'
refresh_interval: 15s
type: A
port: 9101
metric_relabel_configs:
- source_labels: [__name__]
regex: .*
action: replace
target_label: alias
replacement: my-fancy-lb
Looks like a lot of configuration for a simple task. Indeed, we add a config for relabeling. Labels are a powerful feature, which enables every service, that exports metric information in the Prometheus format to add multi-dimensional metadata to identify and group services in tools like Grafana. Prometheus adds a way to add label information when scraping the metrics from a target service. What we do is to add the label ‘alias’ with the static value ‘my-fancy-lb’ to every scraped metric from this loadbalancer. In general, this is not the preferred way to add labels, since it will add cpu load in Prometheus itself during scraping. We do this, because we need the alias label as described later and didn’t find a way to configure custom labels in the HAProxy-exporter. Do you know of one?
Restart Prometheus and you will see the new haproxy in the targets section of the Prometheus UI. But now we want to see some nice graphs, right?
Visualize the data
The easiest way to get started is to import existing Grafana dashboards from Grafana.net. We found this dashboard quite appealing. Just go to the dashboard section in your Grafana UI, open the dashboard list and click import. You just need to enter 367 as the id of the dashboard and Grafana will do the rest.
Et voilà, you see some nice stats. Just select my-fancy-lb from the load balancer dropdown list.