> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flashduty.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Splunk Observability Cloud alert integration

> Send Splunk Observability Cloud (formerly SignalFx) detector alerts to Flashduty On-call through a webhook.

Use a Splunk Observability Cloud Webhook integration to send detector alerts to Flashduty On-call. Each Splunk alert (incident) maps to one Flashduty alert: it is created when the detector triggers and recovers automatically when the alert clears.

<Note>
  This integration receives detector alerts from Splunk Observability Cloud (formerly SignalFx), such as metric, APM, RUM, and Synthetics detectors. For search alerts from Splunk Enterprise or Splunk Cloud Platform, use the [Splunk Alert Events](/en/on-call/integration/alert-integration/alert-sources/splunk) integration.
</Note>

<div className="hide">
  ## In Flashduty On-call

  ***

  You can get the integration push URL in either of the following ways.

  ### Use a dedicated integration

  1. In the Flashduty console, select **Channel** and open a channel
  2. Select **Configuration** → **Integrations** → **Private integration**, then click **Add an integration**
  3. Select **Splunk Observability Cloud** and click **Save**
  4. Open the new integration card and copy the **Push URL**

  ### Use a shared integration

  1. In the Flashduty console, select **Integration Center → Alert Events**
  2. Select **Splunk Observability Cloud** and enter an integration name
  3. Configure the default route and select a channel. You can add more rules under **Routes** after creation
  4. Click **Save** and copy the generated **Push URL**
</div>

## Configure Splunk Observability Cloud

***

<Steps>
  <Step title="Create a Webhook integration">
    Creating a Webhook integration requires the Splunk Observability Cloud administrator role.

    1. Sign in to Splunk Observability Cloud and select **Data Management** in the navigation menu
    2. Go to the **Available integrations** tab (or click **Add Integration** on the **Deployed integrations** tab), set the filter to **All**, then search for and select **Webhook**
    3. On the **Summary** page of the creation wizard, click **Next**. If you see the list of existing integrations first, click **New Integration**
    4. On the **Configure connection** page, fill in:
       * **Name**: for example `Flashduty`. You select the integration by this name in detectors
       * **URL**: paste the complete Flashduty integration push URL
       * **Method**: select `POST`
       * **Shared secret**: optional, and can be left empty. Flashduty authenticates the request by the `integration_key` in the push URL and does not verify the `X-SFX-Signature` header
       * **Headers**: leave empty
    5. Click **Next** and keep the default template on the **Customize message** page
    6. Click **Next**, review the integration, and click **Save**

    After saving, you can click **Run test** on the integration's page and choose a log level to send a test notification (`eventType` is `WEBHOOK_TEST`). Flashduty answers with success and creates no alert.
  </Step>

  <Step title="Add the webhook as a detector alert recipient">
    1. Create or edit the detector whose alerts should reach Flashduty
    2. In the **Alert recipients** step, click **Add Recipient**, select **Webhook**, then select the integration you created (for example `Flashduty`)
    3. Activate and save the detector

    Each detector rule has its own recipients, so add the webhook to every rule that should notify Flashduty. Splunk Observability Cloud notifies Flashduty both when the detector triggers an alert and when the alert clears.
  </Step>

  <Step title="Verify the lifecycle">
    Give a test detector a threshold that is already met and confirm that Flashduty receives an active alert. Then restore the threshold, or wait until the condition is no longer met, and confirm that the same alert recovers.
  </Step>
</Steps>

<Warning>
  If you change the template on the **Customize message** page, keep the `incidentId`, `statusExtended` (or `status`), and `severity` fields under the same names. A request without `incidentId` or a status field is rejected.
</Warning>

## Alert Key

***

Flashduty uses `incidentId` as the Alert Key. Splunk documents that the trigger and clear notifications of one alert carry the same `incidentId`, so both land on the same Flashduty alert.

Changes to the rule name, severity, status, timestamps, message text, or dimension values do not change the Alert Key.

## Status and severity

***

Flashduty reads `statusExtended`, and `status` when `statusExtended` is empty:

| Splunk `statusExtended` | Meaning                                                              | Flashduty status |
| :---------------------- | :------------------------------------------------------------------- | :--------------- |
| `anomalous`             | The detect condition is met and the alert fires                      | Triggered        |
| `ok`                    | The detect condition is no longer met, or the clear condition is met | Recovered        |
| `manually resolved`     | A user resolved the alert in the UI or through the API               | Recovered        |
| `stopped`               | The detector that triggered the alert was edited or deleted          | Recovered        |

An empty or any other status is rejected.

While a muting rule covers an alert, Splunk Observability Cloud sends no notification. If the alert is still active when the muting ends, Splunk sends an `anomalous` notification.

Severity comes from the rule's `severity`:

| Splunk `severity`        | Flashduty severity |
| :----------------------- | :----------------- |
| `Critical`               | Critical           |
| `Major`                  | Critical           |
| `Minor`                  | Warning            |
| `Warning`                | Warning            |
| `Info`                   | Info               |
| Empty or any other value | Warning            |

A recovery keeps the alert's original severity.

## Alert content

***

* **Title**: the rule name `rule`, or the detector name `detector` when `rule` is empty
* **Description**: `messageBody`, or `description` when `messageBody` is empty
* **Labels**:
  * `incident_id`, `detector_id`, `detector`, `detector_url`, `rule`, `severity`, `originating_metric`, `detect_on_condition`, `runbook_url`, `tip`
  * `check` (same as the title), `resource` (the alert source `src`, such as a host or pod name), and `source` (always `splunk-observability`)
  * every dimension in `dimensions`. Characters such as `.` and `-` in a key become `_`, so `k8s.pod.name` becomes `k8s_pod_name`. Keys that start with a digit are dropped. An alert has at most 50 labels; when there are more dimensions, the first ones in key order are kept

`orgId`, `imageUrl`, and `inputs` are not written to labels.

## Troubleshooting

***

* **Splunk reports a failed notification**: confirm that the URL is the complete push URL and includes `integration_key`. Splunk retries a notification that does not get a `200` response for up to 24 hours
* **Flashduty returns a parameter error**: confirm that the template still contains `incidentId` and `statusExtended`
* **The alert does not recover**: confirm that the detector rule still has the webhook as a recipient. If you changed the template, confirm that the clear notification carries the same `incidentId` as the trigger
* **Alerts during muting**: while a Splunk muting rule is active, trigger notifications are not sent, but alerts that were active before muting started still send their clear notifications. When the muting period ends, Splunk resends notifications for alerts that are still active

For field details, see [Integrate a webhook](https://dev.splunk.com/observability/docs/integrations/webhook_integration_overview/) in the Splunk Observability Cloud developer guide.