HashiCorp Consul alert integration

Use the watch mechanism built into Consul to send the state of Consul health checks to Flashduty On-call. Each health check on each node maps to one Flashduty alert: the alert is raised when the check turns warning or critical, and recovers automatically when the check returns to passing. This integration works with both Consul Community Edition and Consul Enterprise. It needs only a watch definition on one Consul agent, with no extra scripts or plugins.

How it works

Whenever any watched health check changes (its status or its output), a Consul watch of type checks POSTs the current state of all watched checks to the push URL as one JSON array. Flashduty turns each check in the array into one event:

A warning or critical check raises an alert, or merges into the alert it already has
A passing check recovers its alert; if there is no such alert, nothing happens
The _node_maintenance and _service_maintenance:<service ID> checks created by maintenance mode (consul maint) are planned changes and never create alerts

The watch sends the current state once as soon as it is loaded, so checks that are already warning or critical raise alerts right away.

Prerequisites

Network: the Consul agent that runs the watch must be able to reach the Flashduty push URL (for example https://api.flashcat.cloud).
Permissions: you need to be able to edit that agent’s configuration directory (for example /etc/consul.d) and run consul reload.
ACL: when ACLs are enabled, the token the watch uses needs read access to all nodes and services, as described below.

In Consul

Choose the agent that runs the watch

Where: The host of a Consul agentThe watch queries the health checks of the whole datacenter, so configure it on one agent: a server, or a dedicated client. Configuring the same watch on several agents does not create duplicate alerts (events for the same check merge), but it multiplies the event count of every alert.

Add the watch definition

Where: The agent’s configuration directory (for example /etc/consul.d) › flashduty-watch.jsonCreate a file named flashduty-watch.json with the content below:

Parameter	Value
`type` (Required)	`checks`
`handler_type` (Required)	`http`
`http_handler_config.path` (Required)	`{integration_url}`
`http_handler_config.method` (Required)	`POST`
`http_handler_config.timeout` (Optional)	For example `10s`
`filter` / `service` (Optional)	Narrow the watch to some checks, see the notes below
`token` (Required when ACLs are enabled)	The ACL token

{
  "watches": [
    {
      "type": "checks",
      "handler_type": "http",
      "http_handler_config": {
        "path": "{integration_url}",
        "method": "POST",
        "timeout": "10s"
      }
    }
  ]
}

Do not set the state parameter. With "state": "critical", a check drops out of the array when it recovers, Flashduty never receives passing, and the alert cannot recover automatically.
To watch only some checks, narrow the watch with the filter parameter, for example "filter": "ServiceName == \"web\" or CheckID == \"serfHealth\"", or watch the checks of one service with "service": "web" (this leaves out node-level checks such as serfHealth).
When ACLs are enabled, add "token": "<ACL token>" to the watch. The token’s policy needs at least:
node_prefix "" { policy = "read" } service_prefix "" { policy = "read" }

Reload the configuration

Where: A terminal on the host of the Consul agent

consul reload

For a Docker deployment, put the file in the directory mounted at /consul/config in the container, then run:

docker exec <container-name> consul reload

Once loaded, the agent immediately sends the current state of all checks to Flashduty. If the agent log shows http watch handler failed with output, Flashduty rejected the request; the log line carries the HTTP status and the reason.

Where: A terminal on the host of the Consul agent (agent HTTP API)Consul has no test notification button. Register a TTL check to verify the setup instead. A TTL check starts as critical, so it raises an alert right away:

curl -X PUT --data '{"ID": "flashduty-test", "Name": "Flashduty test", "TTL": "30m"}' \
  http://127.0.0.1:8500/v1/agent/check/register

Set the check to passing, and the alert recovers:

curl -X PUT http://127.0.0.1:8500/v1/agent/check/pass/flashduty-test

Deregister the check when you are done:

curl -X PUT http://127.0.0.1:8500/v1/agent/check/deregister/flashduty-test

When ACLs are enabled, add -H "X-Consul-Token: <ACL token>" to each command.After a few seconds, go back to Flashduty. If the integration shows a Latest event time, the setup works.

Alert key

Flashduty builds the Alert Key from Partition, Node, and CheckID.

Node is the name of the node the check runs on.
CheckID is the ID of the check. Consul requires CheckID to be unique on a node; a service check’s default ID looks like service:<service ID>, and the node liveness check is serfHealth.
Partition is the admin partition in Consul Enterprise and is empty in Community Edition.

Changes to a check’s status, name, output, or notes do not change its Alert Key. When the agent restarts and sends the same check again, the Alert Key stays the same. A watch only returns the checks of its own datacenter, and the payload does not include the datacenter name. If several datacenters share one integration and have nodes with the same name, checks with the same ID on those nodes merge into one alert. We recommend one integration per datacenter.

Status and severity

Consul `Status`	Flashduty status	Flashduty severity
`critical`	Triggered	Critical
`warning`	Triggered	Warning
`passing`	Recovered	The alert keeps its severity from before recovery (the recovery event itself is recorded as Info and does not change the alert severity)

When a check moves between warning and critical, Flashduty keeps one alert per severity: warning turning critical creates a new Critical alert, and passing recovers both.

Labels

Label	Source
`host`, `resource`	Node name `Node`
`check`	Check name `Name`
`check_id`	Check ID `CheckID`
`check_type`	Check type `Type`, for example `http`, `tcp`, `ttl`
`service`, `service_id`	`ServiceName` and `ServiceID` of the service (empty for node-level checks)
`service_tags`	Service tags, comma-separated
`status`	Consul status
`namespace`, `partition`	Namespace and admin partition in Consul Enterprise
`source`	Always `consul`

The alert title is <check name> on <node name>, and the alert description is the check’s Output and Notes.

FAQ

A check or node was removed and its alert did not recover. What should I do?

When a warning or critical check is deregistered, or its node leaves the cluster, it no longer appears in the array the watch sends, so Flashduty never receives a recovery. Close the alert by hand. In environments where nodes are removed often, turn on the auto-resolve timeout in the channel.

Does Consul retry a failed delivery?

No. Consul’s HTTP handler does not retry a failed request. The next time any watched check changes, the watch sends the current state of every check again, which fills in a missed trigger or recovery.

Why does the event count of an alert keep growing?

Every change to any watched check, including a change in output, makes the watch send all checks, and each failing check merges one more event into its alert. Use the filter or service parameter to watch only the checks your on-call team acts on.

The agent log shows request body exceeds 1 MiB limit. What should I do?

One delivery must not exceed 1 MiB. With many checks, split them across several watches with the filter or service parameter; all watches can use the same push URL.

Do I need consul-alerts, as in the PagerDuty integration?

No. consul-alerts is a third-party daemon; this integration uses Consul’s own watch mechanism and needs no extra component.

For more parameters, see the Consul documentation on Watches and the Check API.

Quick Start

Product Comparison

Channels

Incident Management

Post-Mortem

Status Page

Analytics

Integration Settings

Configuration

Advanced

Integrations

Best Practices