Icinga 2 alert integration

​

Configure Icinga 2

object User "flashduty" {
  display_name = "Flashduty"
}

object NotificationCommand "flashduty-notification" {
  command = [
    "/usr/bin/curl", "--silent", "--show-error", "--fail",
    "--max-time", "10",
    "--header", "Content-Type: application/json"
  ]

  arguments = {
    "--data-binary" = {
      order = 1
      value = {{
        var resolve = macro
        return Json.encode({
          notification_type = resolve("$notification.type$")
          host_name = resolve("$host.name$")
          host_display_name = resolve("$host.display_name$")
          host_address = resolve("$host.address$")
          host_state = resolve("$host.state$")
          host_last_state = resolve("$host.last_state$")
          host_output = resolve("$host.output$")
          host_groups = resolve("$host.groups$")
          service_name = resolve("$service.name$")
          service_display_name = resolve("$service.display_name$")
          service_state = resolve("$service.state$")
          service_last_state = resolve("$service.last_state$")
          service_output = resolve("$service.output$")
          service_groups = resolve("$service.groups$")
        })
      }}
    }
    "--url" = {
      order = 2
      value = "$flashduty_push_url$"
    }
  }

  vars.flashduty_push_url = "<push URL>"
}

apply Notification "flashduty" to Host {
  command = "flashduty-notification"
  users = [ "flashduty" ]
  assign where true
}

apply Notification "flashduty" to Service {
  command = "flashduty-notification"
  users = [ "flashduty" ]
  assign where true
}

icinga2 daemon -C
systemctl reload icinga2

​

Alert Key

​

Alert lifecycle

​

Severity

​

Alert content

• Title: <service display name> on <host display name> for service notifications, Host <host display name> for host notifications. Object names are used when no display name is set
• Description: the plugin output service_output or host_output
• Labels: host, resource (host name), service, check (service name, service notifications only), notification_type, state, host_display_name, host_address, host_groups, service_display_name and service_groups (service notifications only). Multiple groups are joined with commas

​

Troubleshooting

• No notification is sent: check the object’s notification history in Icinga Web, confirm the flashduty notification objects exist (run icinga2 daemon -C --dump-objects to write the object cache, then icinga2 object list --type Notification --name '*flashduty'), and confirm notifications are not disabled on the host or service
• The Icinga 2 log shows exit code 22: curl got an HTTP 4xx/5xx response. Confirm that vars.flashduty_push_url is the full push URL and includes integration_key
• The Icinga 2 log shows exit code 6, 7, or 28: the node that runs notifications cannot resolve, connect to, or reach the push URL within 10 seconds. Check DNS, the firewall, and proxies
• The same notification is pushed several times: the apply Notification rules have several users. Keep only flashduty
• A service turned CRITICAL but nothing was pushed: the object is still in a SOFT state. Notifications are sent only after max_check_attempts results, when it enters a HARD state
• The alert does not recover: confirm that the object has really returned to OK or UP, and that no types filter on the Notification or on the flashduty user excludes Recovery