Skip to main content
Use a Splunk Observability Cloud Webhook integration to send detector alerts to Flashduty On-call. Each Splunk alert (incident) maps to one Flashduty alert: it is created when the detector triggers and recovers automatically when the alert clears.
This integration receives detector alerts from Splunk Observability Cloud (formerly SignalFx), such as metric, APM, RUM, and Synthetics detectors. For search alerts from Splunk Enterprise or Splunk Cloud Platform, use the Splunk Alert Events integration.

​
In Flashduty On-call

You can get the integration push URL in either of the following ways.

​
Use a dedicated integration

  1. In the Flashduty console, select Channel and open a channel
  2. Select Configuration → Integrations → Private integration, then click Add an integration
  3. Select Splunk Observability Cloud and click Save
  4. Open the new integration card and copy the Push URL

​
Use a shared integration

  1. In the Flashduty console, select Integration Center → Alert Events
  2. Select Splunk Observability Cloud and enter an integration name
  3. Configure the default route and select a channel. You can add more rules under Routes after creation
  4. Click Save and copy the generated Push URL

​
Configure Splunk Observability Cloud

1

Create a Webhook integration

Creating a Webhook integration requires the Splunk Observability Cloud administrator role.
  1. Sign in to Splunk Observability Cloud and select Data Management in the navigation menu
  2. Go to the Available integrations tab (or click Add Integration on the Deployed integrations tab), set the filter to All, then search for and select Webhook
  3. On the Summary page of the creation wizard, click Next. If you see the list of existing integrations first, click New Integration
  4. On the Configure connection page, fill in:
    • Name: for example Flashduty. You select the integration by this name in detectors
    • URL: paste the complete Flashduty integration push URL
    • Method: select POST
    • Shared secret: optional, and can be left empty. Flashduty authenticates the request by the integration_key in the push URL and does not verify the X-SFX-Signature header
    • Headers: leave empty
  5. Click Next and keep the default template on the Customize message page
  6. Click Next, review the integration, and click Save
After saving, you can click Run test on the integration’s page and choose a log level to send a test notification (eventType is WEBHOOK_TEST). Flashduty answers with success and creates no alert.
2

Add the webhook as a detector alert recipient

  1. Create or edit the detector whose alerts should reach Flashduty
  2. In the Alert recipients step, click Add Recipient, select Webhook, then select the integration you created (for example Flashduty)
  3. Activate and save the detector
Each detector rule has its own recipients, so add the webhook to every rule that should notify Flashduty. Splunk Observability Cloud notifies Flashduty both when the detector triggers an alert and when the alert clears.
3

Verify the lifecycle

Give a test detector a threshold that is already met and confirm that Flashduty receives an active alert. Then restore the threshold, or wait until the condition is no longer met, and confirm that the same alert recovers.
If you change the template on the Customize message page, keep the incidentId, statusExtended (or status), and severity fields under the same names. A request without incidentId or a status field is rejected.

​
Alert Key

Flashduty uses incidentId as the Alert Key. Splunk documents that the trigger and clear notifications of one alert carry the same incidentId, so both land on the same Flashduty alert. Changes to the rule name, severity, status, timestamps, message text, or dimension values do not change the Alert Key.

​
Status and severity

Flashduty reads statusExtended, and status when statusExtended is empty: An empty or any other status is rejected. While a muting rule covers an alert, Splunk Observability Cloud sends no notification. If the alert is still active when the muting ends, Splunk sends an anomalous notification. Severity comes from the rule’s severity: A recovery keeps the alert’s original severity.

​
Alert content

  • Title: the rule name rule, or the detector name detector when rule is empty
  • Description: messageBody, or description when messageBody is empty
  • Labels:
    • incident_id, detector_id, detector, detector_url, rule, severity, originating_metric, detect_on_condition, runbook_url, tip
    • check (same as the title), resource (the alert source src, such as a host or pod name), and source (always splunk-observability)
    • every dimension in dimensions. Characters such as . and - in a key become _, so k8s.pod.name becomes k8s_pod_name. Keys that start with a digit are dropped. An alert has at most 50 labels; when there are more dimensions, the first ones in key order are kept
orgId, imageUrl, and inputs are not written to labels.

​
Troubleshooting

  • Splunk reports a failed notification: confirm that the URL is the complete push URL and includes integration_key. Splunk retries a notification that does not get a 200 response for up to 24 hours
  • Flashduty returns a parameter error: confirm that the template still contains incidentId and statusExtended
  • The alert does not recover: confirm that the detector rule still has the webhook as a recipient. If you changed the template, confirm that the clear notification carries the same incidentId as the trigger
  • Alerts during muting: while a Splunk muting rule is active, trigger notifications are not sent, but alerts that were active before muting started still send their clear notifications. When the muting period ends, Splunk resends notifications for alerts that are still active
For field details, see Integrate a webhook in the Splunk Observability Cloud developer guide.