Checkmk alert integration

​

Configure Checkmk

#!/usr/bin/env python3
# Flashduty
# Sends Checkmk host and service notifications to a Flashduty Checkmk integration.
# Parameter 1: the integration push URL.
import json
import os
import sys
import urllib.error
import urllib.request

url = os.environ.get("NOTIFY_PARAMETER_1", "")
if not url:
    sys.stderr.write("missing parameter 1: Flashduty push URL\n")
    sys.exit(2)

context = {
    key[len("NOTIFY_"):]: value
    for key, value in os.environ.items()
    if key.startswith("NOTIFY_") and not key.startswith("NOTIFY_PARAMETER")
}
request = urllib.request.Request(
    url,
    data=json.dumps(context).encode("utf-8"),
    headers={"Content-Type": "application/json"},
    method="POST",
)
try:
    with urllib.request.urlopen(request, timeout=10) as response:
        sys.stdout.write("Flashduty accepted the notification: HTTP %d\n" % response.status)
except urllib.error.HTTPError as e:
    sys.stderr.write("Flashduty rejected the notification: HTTP %d %s\n" % (e.code, e.read()[:500]))
    sys.exit(1 if e.code >= 500 or e.code == 429 else 2)
except (urllib.error.URLError, OSError) as e:
    sys.stderr.write("Flashduty is unreachable: %s\n" % e)
    sys.exit(1)

chmod +x ~/local/share/check_mk/notifications/flashduty

​

Alert Key

​

Alert lifecycle

​

Severity

​

Alert content

• Title: <service name> on <host name> for service notifications, Host <host name> for host notifications
• Description: the plugin output SERVICEOUTPUT or HOSTOUTPUT, followed by the long output when present
• Labels: host, resource (host name), service, check (service name, service notifications only), what (HOST or SERVICE), notification_type, state, site, host_alias, host_address, host_groups, service_groups (service notifications only), and Checkmk host and service labels: HOSTLABEL_env becomes host_label_env and SERVICELABEL_app becomes service_label_app, with characters such as / and . in the key replaced by _

​

Troubleshooting

• Flashduty is not listed as a method in Checkmk: make sure the script is in the site’s local/share/check_mk/notifications/ directory, is named flashduty, is executable, and has # Flashduty on its second line
• No notification is sent: check the site’s var/log/notify.log to confirm the rule matched the event and that the user under Recipient has not disabled notifications
• The script reports HTTP 4xx: make sure parameter 1 is the full push URL including integration_key
• The same notification is posted several times: Recipient selects several contacts; select only one user
• The alert does not recover: make sure the object has really returned to OK or UP. If the problem happened during flapping or downtime, make sure the rule includes Start or end of flapping state and Start or end of downtime