Incidents, alerts, and events
When Flashduty On-call receives an alert event (such as a Zabbix notification), the system triggers an alert, which in turn triggers an incident. Multiple similar active alerts may be grouped into the same incident for unified assignment, notification, and handling. An incident is a group of similar alerts. Without noise reduction, an incident equals an alert. Conversely, with noise reduction, an incident equals its associated multiple alerts. For more about alert noise reduction, read Understanding Noise Reduction.Incident severity, status, and progress
Severity
Incidents, alerts, and events all use these three severity levels. Severity is capitalized, which is important when using APIs. The severity generation rules are:
- Event Severity: Alert events from different integration sources (like Zabbix and Nightingale) have different severity enumerations. Flashduty On-call maps them to these three standard severities according to specific rules. For mapping details, refer to the specific integration documentation. To customize severity, see Alert Processing.
- Alert Severity: Equals the highest severity among associated events. As new events keep merging in, the alert severity only escalates to the worst level and never drops because of a lower-severity event.
- Incident Severity: Equals the highest severity among associated alerts.
Processing progress
API fields represent progress with the capitalized enum values
Triggered / Processing / Closed. Use these raw values when calling APIs or configuring filters.
Incident status
Alert status represents the incident’s state in the original monitoring system, that is, “Recovered” or “Not Recovered”. Incident status depends entirely on its associated alerts.Incident auto-recovery leads to automatic closure (of processing progress); but manually closing an incident has no effect on incident status.
Incident labels
Labels are a fundamental concept in Flashduty On-call. Different labels describe alert and incident information across various dimensions, and are extensively used in filtering, searching, and grouping scenarios.Label generation rules
Flashduty extracts alert labels from the event messages that the monitoring system reports. Different sources have different extraction methods, but generally we follow the capture everything relevant principle. For example, for Prometheus-sourced alert events, Flashduty On-call extracts Labels and Annotations information from the Payload. Flashduty On-call provides label enhancement for automatic label generation. Go to Configure Label Enhancement to learn more.System labels
The label panel on the incident/alert details page collects platform-internal labels into a separate System labels section, with a Hide system labels / Show system labels toggle (with an eye icon) next to the heading. Which labels count as system labels depends on the alert source:- The toggle appears only when the labels actually contain system labels. Its tooltip states the rule used for the current alert source: Monit alerts show “Monit system labels start with
__alert_rule_or__entity_alert_”, other sources show “System labels are platform labels starting with__, such as__alert_rule_idand__name__” - System labels are collapsed by default. While collapsed, the bottom of the list holds a dashed button reading “N system labels collapsed (first few keys…)”; clicking it expands a separate System labels section with a divider, and the toggle next to the heading collapses it again
- Flashduty saves the toggle as a per-user preference that persists across sessions and devices; the mobile app and the web console share the same preference
- The count badge next to the heading is the total number of labels (system labels and related links included), not the number of rows currently visible
- While system labels are collapsed, the JSON view and the JSON you copy exclude them too; drag-to-reorder applies to regular labels only, and system labels are never reordered
- System labels written by Monit come with a hover card that translates the machine-readable label into a plain-language alias and offers a jump:
__alert_rule_id→ the alert rule,__alert_rule_folder_*→ the rule’s folder,__alert_rule_data_source_*→ the data source,__alert_rule_event_kind→ firing / repeating / resolved,__alert_rule_valuesand__alert_rule_queries→ each main query’s value and expression at trigger time,__alert_rule_relate_*→ related queries and their result at trigger time,__alert_rule_investigation_context*→ investigation entries
Related links and label value rendering
The label panel shows a Links button bar above the list and renders some label values specially. Related links: labels whose value is anhttp(s):// link and that are not images stop acting as regular, sortable labels — they are lifted above the list, one button per link. The button name comes from the label key:
Buttons follow the order of the table above; keys not listed come after them in alphabetical order. Hovering a button shows the full key and URL. Image links (a key containing
image, or a link ending in an image extension) are not related links and stay in the label list. This extraction applies to regular labels only — link labels already in the System labels section stay where they are.
Special label value rendering: label values are recognized by format and rendered as follows. Recognition looks only at the key name and the value’s format; it does not depend on per-integration metadata.
Forms that already have a fixed display size — images and query expressions — are not truncated to two lines; other values longer than two lines are collapsed behind a View all / Collapse toggle.
Incident lifecycle
1
Trigger New Incident
You can trigger incidents in the following ways:
- Auto-trigger: Flashduty On-call receives an alert event from an integration (like Zabbix notification), event auto-triggers an alert, alert auto-triggers an incident
- Manual trigger: Click Create Incident button in Flashduty On-call console, fill in title, description, severity, and so on. to trigger a new incident
- External reporting: After enabling external reporting, external personnel can submit incident tickets through a dedicated link without logging into a Flashduty account. See External Incident Submission for details
2
Assignment and Notification
After a new incident triggers, Flashduty On-call sequentially matches escalation rules under the channel. After matching an escalation rule, the system assigns the incident to individuals, team members, or on-call personnel and sends notifications.You can set different escalation rules for different time periods or incident types to achieve flexible assignment. The system allows you to set multiple levels within an escalation rule. If current level responders don’t acknowledge and resolve the incident within the specified time, the system automatically escalates to the next level.You can flexibly arrange notification methods in escalation rules. Flashduty On-call supports many group chat and direct message notification channels. Direct messages are one-to-one push channels (like voice, SMS, email), group chats push messages to messaging groups (like Feishu/Lark, Dingtalk, Slack) with additional mentions for assignees.
If you assign an incident to a schedule with no one on-call (empty schedule), Flashduty notifies no individuals, but it still posts to any group chat channels you configured.
3
Acknowledge and Resolve
On-call personnel can acknowledge immediately upon receiving notification. You can acknowledge incidents by voice calls or instant messages. After acknowledgment, incident progress changes to Processing.Closing an incident changes progress to Closed. If alerts associated with the incident auto-recover, the incident also auto-closes. Conversely, if you manually close an incident, all associated alerts are automatically closed. This means these alerts will no longer merge new events.
Incident timeline
Every incident has a timeline for tracing changes and actions at different historical moments. For example, at what time, through what channel, who was notified, and notification results. Each timeline record shows the full date and time (YYYY-MM-DD HH:mm:ss) by default; hovering switches to a relative time (for example, “3 minutes ago”).
Triggering incidents
Trigger from an integration
Flashduty On-call supports most common monitoring systems, including Prometheus, Zabbix, Nightingale, and cloud monitoring. Go to Alert Integration for specific steps.Trigger with the API
Flashduty On-call provides a custom event standard, allowing you to report alerts using the standard protocol, suitable for any non-integrated monitoring system. For details, read Custom Alert Events.Trigger by email
Flashduty On-call provides email integration, allowing you to report alerts by sending emails, suitable for all monitoring systems supporting email alerts. For details, read Email Integration Guide.Trigger from the console
You can click the Create button on the incident list page to manually create an incident. Follow these steps:1
Enter incident title
Enter the incident title with a brief description of what happened. This field is required.
2
Select severity
Choose Critical, Warning, or Info.
3
Select channel
Select the channel the incident belongs to. If you create the incident inside a channel, Flashduty fills in this field and you can’t change it.
4
Select assignment method
After selecting a channel, the system displays assignment method options:
- By Policy: Select an escalation rule from the current channel. If no escalation rules are available in the channel, this option is disabled
- Direct Assignment: Manually select responders (individuals or schedules) for direct assignment
5
Enter incident description (optional)
Enter a detailed description of the incident. The editor supports rich text and Markdown syntax.
Related topics
Understanding Noise Reduction
Understand the relationship between events, alerts, incidents and noise reduction principles
Search and View Incidents
Master incident list and details page usage
Handle and Update Incidents
Learn how to acknowledge, snooze, close incidents
Escalate and Reassign Incidents
Learn how to reassign and escalate