curl. Each entity and check pair maps to one Flashduty alert: it triggers when the check status is non-zero and closes automatically when the status returns to 0.
In Flashduty On-call
You can get the integration push URL in either of the following ways.
Use a dedicated integration
- In the Flashduty console, select Channel and open a channel
- Select Configuration → Integrations → Private integration, then click Add an integration
- Select Sensu Go and click Save
- Open the new integration card and copy the Push URL
Use a shared integration
- In the Flashduty console, select Integration Center → Alert Events
- Select Sensu Go and enter an integration name
- Configure the default route and select a channel. You can add more rules under Routes after creation
- Click Save and copy the generated Push URL
Configure Sensu Go
The steps below are based on Sensu Go 6.x and use
sensuctl. You need permission to create handlers and pipelines and to edit checks in the target namespace.
1
Make sure the backend can run curl
Pipe handlers run on the machine that hosts the Sensu backend (
sensu-backend), not on agents. Make sure curl is installed on the backend machine and that it can reach the Flashduty push URL.The official Docker image
sensu/sensu does not include curl. You can build your own image from it and run apk add --no-cache curl.2
Create the handler and pipeline
Save the following as Then create them in the target namespace:Handlers and pipelines belong to a namespace. If you use several namespaces, run the command once in each (
flashduty.yml and replace <push URL> with the full push URL of the Flashduty integration (keep the quotes around it):sensuctl create --file flashduty.yml --namespace <namespace>).is_incident is a built-in Sensu filter. It passes only events with a non-zero status and resolution events (the first event whose status goes from non-zero to 0). Without it, every OK check result is also posted to Flashduty, and the volume grows with every check interval.3
Add the pipeline to your checks
For each check that should send alerts, export its definition:In Then create the check again:The change takes effect from the next check execution.To be alerted when an agent stops reporting, add the following to the agent configuration file
check.yml, replace pipelines: [] under spec with the following. If the check already has pipelines, add this entry to the list instead:On Sensu Go 6.14, adding a pipeline with
sensuctl edit check fails with cannot have both pipelines and fallback_pipeline defined at the same time. The export-and-create steps above avoid this./etc/sensu/agent.yml and restart the agent. Keepalive events are then posted to Flashduty as well:4
Verify the lifecycle
Sensu Go has no “send test notification” button. Put a check into WARNING or CRITICAL for real (for example by lowering a threshold) and confirm that Flashduty receives an active alert. Then let the check recover, or run
sensuctl event resolve <entity name> <check name>, and confirm that the alert closes.You can also create events by hand for a proxy entity through the Sensu events API: first send an event with status 2, then one with status 0, using the same entity name and check name in both requests.Alert Key
Flashduty computes the Alert Key from the entity’s namespace
entity.metadata.namespace, the entity name entity.metadata.name, and the check name check.metadata.name. Trigger, repeat, and resolution events for the same check on the same entity carry the same three fields, so they land on the same alert. Events from proxy checks (proxy_entity_name) use the proxy entity as their entity, so each proxy entity gets its own alert.
Changes to the status, check output, occurrences, time, event ID, or silenced state do not change the Alert Key. When the status rises to a higher severity (for example from WARNING to CRITICAL), Flashduty opens a new alert at the higher severity and keeps the earlier alert open; the resolution event closes both. Renaming an entity or check produces a new alert; close any alert left open under the old name by hand.
Flashduty rejects a request that lacks entity or check (for example a metrics-only event), or lacks the namespace, entity name, check name, or check.status. curl then exits with a non-zero code, and the Sensu backend log records the handler failure.
Alert lifecycle
With the
is_incident filter, every execution of a check that stays non-zero is posted once, and these events merge into the same alert.
Severity
A resolution event takes the severity of the most recent non-zero status in
check.history.
Alert content
- Title:
<entity name>: <check name> - Description: the check output
check.output, followed by the namespace, status code,check.state, andoccurrences - Labels:
resource(entity name),check(check name),namespace,host(the entity’s host nameentity.system.hostname),entity_class,proxy_entity_name,status,state(passing,failing, orflapping),occurrences, andis_silencedwhen the event is silenced (Sensu Go before 6.14 only, because later versions do not send silenced events)
redact field are not written to labels.
Troubleshooting
- Flashduty receives no alert: search the backend log for runs of the
flashdutyhandler (journalctl -u sensu-backend), make sure the check’spipelinesincludeflashduty, and make sure the event status is non-zero - The handler reports
curl: not found:curlis not installed on the backend machine or container - The handler reports HTTP 4xx: make sure the push URL in the command is complete and includes
integration_key, and that the request carries theContent-Type: application/jsonheader - The alert does not recover: make sure the pipeline uses the
is_incidentfilter and notnot_silenced, that the check was not silenced in Sensu when it recovered, and that the entity name and check name did not change before the recovery - Keepalive alerts are not posted: make sure
keepalive-pipelinesis set tocore/v2.Pipeline.flashdutyand that the pipeline exists in the agent’s namespace