Prometheus is an excellent timeseries-based monitoring platform. As well, it’s easily pluggable with the equally excellent dashboard software, Grafana.

One of my colleagues thought it would be really cool if we got links to the Grafana dashboard for the given server when an alert is received in Slack. That looked possible, because the Grafana dashboard links use the node name and port as URL parameters, see below:

http://grafana:3000/dashboard/db/node-exporter-full?refresh=1m&orgId=1&var-node=web-01&var-port=9100

var-node and var-port parameters just need to be filled with this info. However, this is received from Prometheus with a single label, such as instance=web-01:9100.

We can use the power of Prometheus templating functions to generate a URL using the reReplaceAll regular expression function.

For example, the following template will capture the left part of the instance label (the host name), and output it:

{{ reReplaceAll "(.*):(.*)" "${1}" .Labels.instance }}

Here goes our final template to generate a slack notification including the URL, to be added to the alertmanager configuration:

receivers:
- name: ops-slack
  slack_configs:
  - username: prometheus
    send_resolved: true
    channel: alerts
    title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] Monitoring Event Notification'
    text: >-
        {{ range .Alerts }}
          *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
          *Description:* {{ .Annotations.description }}
          *Graph:* <http://grafana.com:3000/dashboard/db/node-exporter-full?refresh=1m&orgId=1&var-node={{ reReplaceAll "(.*):(.*)" "${1}" .Labels.instance }}&var-port={{ reReplaceAll "(.*):(.*)" "${2}" .Labels.instance }} |:chart_with_upwards_trend:>
          {{ end }}
        {{ end }}