Skip to main content
Use a Cisco ThousandEyes classic webhook (the webhook you configure in an alert rule) to send alert events to Flashduty On-call. Each alert that ThousandEyes generates maps to one Flashduty alert: ThousandEyes sends ALERT_NOTIFICATION_TRIGGER when the alert triggers, and sends ALERT_NOTIFICATION_CLEAR when it clears, which recovers the alert. Both test alerts (Cloud Agent and Enterprise Agent tests) and agent notifications (agent offline, clock offset, and so on) are supported.

In Flashduty On-call

You can get the integration push URL in either of the following ways.

Use a dedicated integration

  1. In the Flashduty console, select Channel and open a channel
  2. Select Configuration → Integrations → Private integration, then click Add integration
  3. Select ThousandEyes, then click Save
  4. Open the generated integration card and copy the Push URL

Use a shared integration

  1. In the Flashduty console, go to Integration Center → Alert Events
  2. Select ThousandEyes and enter an integration name
  3. Configure the default route and select a channel. You can add more rules under Routing after creation
  4. Click Save and copy the generated Push URL

Configure ThousandEyes

Editing alert rules and webhooks requires permission to manage alert rules in ThousandEyes. Webhook alert notifications are not available in the ThousandEyes for Government instance.
1

Add a webhook in ThousandEyes

Where: ThousandEyes › Manage › Alert Rules › Notifications › Webhooks › Configure Webhooks › Add New WebhookEditing alert rules and webhooks requires permission to manage alert rules in ThousandEyes. Webhook alert notifications are not available in the ThousandEyes for Government instance. Expand an alert rule, open the Notifications tab, and in the Webhooks section click Configure Webhooks (Edit webhooks if webhooks already exist), then click Add New Webhook.Click Test. Webhook test completed successfully means the URL is reachable. Then save the webhook.
Use this alert-rule webhook, which ThousandEyes calls a classic webhook. Do not use a custom webhook operation under Manage → Integrations → Integrations 2.0. Flashduty parses the fixed classic webhook format.
2

Select the webhook in ThousandEyes alert rules

Where: ThousandEyes › Manage › Alert Rules › Notifications › WebhooksSave the alert rule. Select the webhook in every alert rule that should send alerts, including the rules for agent notifications.
3

Trigger an alert and verify in ThousandEyes

Where: ThousandEyes › Manage › Alert Rules › Notifications › Webhooks › Configure Webhooks › Test
  1. Clicking Test sends a WEBHOOK_TEST event. Flashduty returns success but does not create an alert, so Test only proves connectivity
  2. Make an alert rule that uses the webhook trigger (for example, lower its threshold temporarily), and confirm that Flashduty receives an active alert
  3. Wait for the alert to clear (or restore the threshold), and confirm that the alert is recovered

Alert key

Flashduty uses the ThousandEyes alert ID (alert.alertId) as the Alert Key. ThousandEyes assigns a unique alertId to each generated alert, and the trigger and clear notifications of one alert carry the same alertId, so the clear notification recovers the matching Flashduty alert.
  • Triggering again: when an alert triggers again after it clears, ThousandEyes generates a new alert with a new alertId, and Flashduty creates a new alert
  • Multiple agents: an alert that involves several agents still has one alertId, so it is one Flashduty alert; each agent’s metrics are listed in the alert description
  • The eventId differs on every delivery and is not part of the Alert Key. Changes to the rule name, severity, metric values, or time do not change the Alert Key either. Requests without alert.alertId are rejected

Status and severity

The status is determined by eventType: The severity is determined by the alert rule severity (alert.severity): Alert rules created with Add New Alert Rule in ThousandEyes default to the Info severity, and such alerts are Info in Flashduty; the built-in Default Alert Rules default to Minor. For a higher severity, change the alert rule’s Severity in ThousandEyes. A recovered alert keeps the severity it had when it triggered.

Labels

The alert title is “alert rule name: test name”, or “alert rule name: agent name” for agent notifications. The alert description contains the alert condition and each agent’s metrics when the alert triggered.

Troubleshooting

  • Flashduty returns a parameter error: make sure the URL is complete, not encoded, and includes integration_key, and that you use the classic webhook in the alert rule rather than a custom webhook
  • No alert after clicking Test: this is expected. The test event does not create an alert; verify with a real alert
  • The alert does not recover: make sure the alert has cleared in ThousandEyes. A global alert clears only after all of its agents no longer meet the condition; while any agent is still alerting, no clear notification is sent
  • An alert rule sends nothing: webhooks are selected per alert rule. Make sure the Flashduty webhook is selected in that rule’s Notifications
For more on the fields, see ThousandEyes Classic Webhooks for Alert Notifications.