Skip to main content

Incidents, alerts, and events

When Flashduty On-call receives an alert event (such as a Zabbix notification), the system triggers an alert, which in turn triggers an incident. Multiple similar active alerts may be grouped into the same incident for unified assignment, notification, and handling. An incident is a group of similar alerts. Without noise reduction, an incident equals an alert. Conversely, with noise reduction, an incident equals its associated multiple alerts. For more about alert noise reduction, read Understanding Noise Reduction.

Incident severity, status, and progress

Severity

Incidents, alerts, and events all use these three severity levels. Severity is capitalized, which is important when using APIs. The severity generation rules are:
  • Event Severity: Alert events from different integration sources (like Zabbix and Nightingale) have different severity enumerations. Flashduty On-call maps them to these three standard severities according to specific rules. For mapping details, refer to the specific integration documentation. To customize severity, see Alert Processing.
  • Alert Severity: Equals the highest severity among associated events. As new events keep merging in, the alert severity only escalates to the worst level and never drops because of a lower-severity event.
  • Incident Severity: Equals the highest severity among associated alerts.

Processing progress

API fields represent progress with the capitalized enum values Triggered / Processing / Closed. Use these raw values when calling APIs or configuring filters.

Incident status

Incident status follows the state of its associated alerts in the original monitoring system: “Recovered” or “Not Recovered”.
Incident auto-recovery leads to automatic closure (of processing progress); but manually closing an incident has no effect on incident status.

Incident labels

Labels are a fundamental concept in Flashduty On-call. Different labels describe alert and incident information across various dimensions, and are extensively used in filtering, searching, and grouping scenarios.

Label generation rules

Flashduty extracts alert labels from the event messages that the monitoring system reports. Different sources have different extraction methods, but generally we follow the capture everything relevant principle. For example, for Prometheus-sourced alert events, Flashduty On-call extracts Labels and Annotations information from the Payload.
  • An auto-triggered incident’s labels equal the labels of its first associated alert
  • An incident you create by hand in the console has no labels: the create form has no label input, and labels can’t be edited or added after the incident is created
  • When you create an incident through the Open API (POST /incident/create), you can pass a labels object in the request body. It is stored with the incident and shows up in the label panel and in label search. The rules match alert labels: at most 50 labels, labels with blank values are dropped, label names are normalized, and values are trimmed and cut to the length limit
Flashduty On-call provides label enhancement for automatic label generation. Go to Configure Label Enhancement to learn more.

System labels

Platform-internal labels (names starting with __, such as __alert_rule_id) are collected in a separate System labels section, collapsed by default.
  • The Show system labels / Hide system labels button next to the label panel heading expands or collapses the section. It appears only when system labels exist. The choice is saved as a per-user preference shared by the web console and the mobile app
  • While collapsed, the JSON view and the JSON you copy exclude system labels. System labels are not reordered by drag-and-drop
  • The count next to the heading is the total number of labels, including system labels
  • Hovering a system label on a Monit alert opens a hover card that explains it in plain words (alert rule, data source, values and queries at trigger time) and offers a jump

Label value rendering

Link labels stay in the label list. Some label values are rendered specially. For incidents aggregated from RUM alerts, hovering the issue_id value opens a RUM Issue card; click the link in the card to open the issue. For Monitors entity-tree alerts, View Rule appears next to __entity_alert_rule_id and opens the matching alert rule in the entity tree. If a label value is invalid or the needed context is missing, Flashduty shows it as plain text. Special label value rendering: Values longer than two lines are collapsed behind a View all / Collapse toggle; images and query expressions are not collapsed.

Incident lifecycle

1

Trigger New Incident

You can trigger incidents in the following ways:
  • Auto-trigger: Flashduty On-call receives an alert event from an integration (like Zabbix notification), event auto-triggers an alert, alert auto-triggers an incident
  • Manual trigger: Click Create Incident button in Flashduty On-call console, fill in title, description, severity, and so on. to trigger a new incident
  • External reporting: After enabling external reporting, external personnel can submit incident tickets through a dedicated link without logging into a Flashduty account. See External Incident Submission for details
2

Assignment and Notification

After a new incident triggers, Flashduty On-call sequentially matches escalation rules under the channel. After matching an escalation rule, the system assigns the incident to individuals, team members, or on-call personnel and sends notifications.
If no escalation rule matches, Flashduty assigns the incident to no one and sends no notifications.
You can set different escalation rules for different time periods or incident types to achieve flexible assignment. The system allows you to set multiple levels within an escalation rule. If current level responders don’t acknowledge and resolve the incident within the specified time, the system automatically escalates to the next level.You can flexibly arrange notification methods in escalation rules. Flashduty On-call supports many group chat and direct message notification channels. Direct messages are one-to-one push channels (like voice, SMS, email), group chats push messages to messaging groups (like Feishu/Lark, Dingtalk, Slack) with additional mentions for assignees.
If you assign an incident to a schedule with no one on-call (empty schedule), Flashduty notifies no individuals, but it still posts to any group chat channels you configured.
3

Acknowledge and Resolve

On-call personnel can acknowledge immediately upon receiving notification. You can acknowledge incidents by voice calls or instant messages. After acknowledgment, incident progress changes to Processing.
Flashduty On-call currently doesn’t restrict incidents to only be acknowledged by “assigned responders”. Anyone who sees the incident can acknowledge it.
Closing an incident changes progress to Closed. If all associated alerts recover and Close with alert recovery is enabled in the channel, the incident closes automatically. Conversely, if you manually close an incident, all associated alerts are automatically closed. This means these alerts will no longer merge new events.

Incident timeline

Every incident has a timeline for tracing changes and actions at different historical moments. For example, at what time, through what channel, who was notified, and notification results. Each timeline record shows the full date and time (YYYY-MM-DD HH:mm:ss) by default; hovering switches to a relative time (for example, “3 minutes ago”).

Triggering incidents

Trigger from an integration

Flashduty On-call supports most common monitoring systems, including Prometheus, Zabbix, Nightingale, and cloud monitoring. Go to Alert Integration for specific steps.
Flashduty On-call supports dedicated and shared integration modes:
  • Dedicated Integration: Deliver alerts to a channel’s dedicated integration, incidents trigger within that channel
  • Shared Integration: Deliver alerts to Integration Center’s shared integration, then configure routing to deliver alerts to different channels by rules

Trigger with the API

Flashduty On-call provides a custom event standard, allowing you to report alerts using the standard protocol, suitable for any non-integrated monitoring system. For details, read Custom Alert Events.
To ensure system stability, Flashduty On-call enforces rate limits on alert push APIs (8000 requests/second per API, 2000 requests/minute per account, 2000 requests/minute per integration). Exceeding these limits returns a 429 status code — wait and retry. See Integrate Data - Rate Limits for details.
Ensure you actively close alerts, or set incident auto-close timeout in the channel. Too many incidents will severely degrade console search performance. The system may close historical incidents without notification.

Trigger by email

Flashduty On-call provides email integration, allowing you to report alerts by sending emails, suitable for all monitoring systems supporting email alerts. For details, read Email Integration Guide.
You can set specific email prefixes for each integration. You can also contact us to set a memorable dedicated domain for your account. For example, order-service@alerts.example.com.

Trigger from the console

You can click the Create button on the incident list page to manually create an incident. Follow these steps:
1

Enter incident title

Enter the incident title with a brief description of what happened. This field is required.
2

Select severity

Choose Critical, Warning, or Info.
3

Select channel

Select the channel the incident belongs to. If you create the incident inside a channel, Flashduty fills in this field and you can’t change it.
4

Select assignment method

After selecting a channel, the system displays assignment method options:
  • By Policy: Select an escalation rule from the current channel. If no escalation rules are available in the channel, this option is disabled
  • Direct Assignment: Select responders (individuals, schedules or teams). When you pick a team, Flashduty expands it into its members at the moment of assignment and notifies them; the expansion rules and the size limits are in Escalate and reassign incidents. In Notification, choose Follow personal preferences or Use unified settings. Unified settings require at least one notification channel. You can use the system default notification template or select an existing template. Direct assignment notifies individuals only; it does not send group chat notifications
5

Enter incident description (optional)

Enter a detailed description of the incident. The editor supports rich text and Markdown syntax.

Understanding Noise Reduction

Understand the relationship between events, alerts, incidents and noise reduction principles

Search and View Incidents

Master incident list and details page usage

Handle and Update Incidents

Learn how to acknowledge, snooze, close incidents

Escalate and Reassign Incidents

Learn how to reassign and escalate