Incidents, alerts, and events
When Flashduty On-call receives an alert event (such as a Zabbix notification), the system triggers an alert, which in turn triggers an incident. Multiple similar active alerts may be grouped into the same incident for unified assignment, notification, and handling. An incident is a group of similar alerts. Without noise reduction, an incident equals an alert. Conversely, with noise reduction, an incident equals its associated multiple alerts. For more about alert noise reduction, read Understanding Noise Reduction.Incident severity, status, and progress
Severity
Incidents, alerts, and events all use these three severity levels. Severity is capitalized, which is important when using APIs. The severity generation rules are:
- Event Severity: Alert events from different integration sources (like Zabbix and Nightingale) have different severity enumerations. Flashduty On-call maps them to these three standard severities according to specific rules. For mapping details, refer to the specific integration documentation. To customize severity, see Alert Processing.
- Alert Severity: Equals the highest severity among associated events. As new events keep merging in, the alert severity only escalates to the worst level and never drops because of a lower-severity event.
- Incident Severity: Equals the highest severity among associated alerts.
Processing progress
API fields represent progress with the capitalized enum values
Triggered / Processing / Closed. Use these raw values when calling APIs or configuring filters.
Incident status
Incident status follows the state of its associated alerts in the original monitoring system: “Recovered” or “Not Recovered”.Incident auto-recovery leads to automatic closure (of processing progress); but manually closing an incident has no effect on incident status.
Incident labels
Labels are a fundamental concept in Flashduty On-call. Different labels describe alert and incident information across various dimensions, and are extensively used in filtering, searching, and grouping scenarios.Label generation rules
Flashduty extracts alert labels from the event messages that the monitoring system reports. Different sources have different extraction methods, but generally we follow the capture everything relevant principle. For example, for Prometheus-sourced alert events, Flashduty On-call extracts Labels and Annotations information from the Payload. Flashduty On-call provides label enhancement for automatic label generation. Go to Configure Label Enhancement to learn more.System labels
Platform-internal labels (names starting with__, such as __alert_rule_id) are collected in a separate System labels section, collapsed by default.
- The Show system labels / Hide system labels button next to the label panel heading expands or collapses the section. It appears only when system labels exist. The choice is saved as a per-user preference shared by the web console and the mobile app
- While collapsed, the JSON view and the JSON you copy exclude system labels. System labels are not reordered by drag-and-drop
- The count next to the heading is the total number of labels, including system labels
- Hovering a system label on a Monit alert opens a hover card that explains it in plain words (alert rule, data source, values and queries at trigger time) and offers a jump
Label value rendering
Link labels stay in the label list. Some label values are rendered specially. For incidents aggregated from RUM alerts, hovering theissue_id value opens a RUM Issue card; click the link in the card to open the issue. For Monitors entity-tree alerts, View Rule appears next to __entity_alert_rule_id and opens the matching alert rule in the entity tree. If a label value is invalid or the needed context is missing, Flashduty shows it as plain text.
Special label value rendering:
Values longer than two lines are collapsed behind a View all / Collapse toggle; images and query expressions are not collapsed.
Incident lifecycle
1
Trigger New Incident
You can trigger incidents in the following ways:
- Auto-trigger: Flashduty On-call receives an alert event from an integration (like Zabbix notification), event auto-triggers an alert, alert auto-triggers an incident
- Manual trigger: Click Create Incident button in Flashduty On-call console, fill in title, description, severity, and so on. to trigger a new incident
- External reporting: After enabling external reporting, external personnel can submit incident tickets through a dedicated link without logging into a Flashduty account. See External Incident Submission for details
2
Assignment and Notification
After a new incident triggers, Flashduty On-call sequentially matches escalation rules under the channel. After matching an escalation rule, the system assigns the incident to individuals, team members, or on-call personnel and sends notifications.You can set different escalation rules for different time periods or incident types to achieve flexible assignment. The system allows you to set multiple levels within an escalation rule. If current level responders don’t acknowledge and resolve the incident within the specified time, the system automatically escalates to the next level.You can flexibly arrange notification methods in escalation rules. Flashduty On-call supports many group chat and direct message notification channels. Direct messages are one-to-one push channels (like voice, SMS, email), group chats push messages to messaging groups (like Feishu/Lark, Dingtalk, Slack) with additional mentions for assignees.
If you assign an incident to a schedule with no one on-call (empty schedule), Flashduty notifies no individuals, but it still posts to any group chat channels you configured.
3
Acknowledge and Resolve
On-call personnel can acknowledge immediately upon receiving notification. You can acknowledge incidents by voice calls or instant messages. After acknowledgment, incident progress changes to Processing.Closing an incident changes progress to Closed. If all associated alerts recover and Close with alert recovery is enabled in the channel, the incident closes automatically. Conversely, if you manually close an incident, all associated alerts are automatically closed. This means these alerts will no longer merge new events.
Incident timeline
Every incident has a timeline for tracing changes and actions at different historical moments. For example, at what time, through what channel, who was notified, and notification results. Each timeline record shows the full date and time (YYYY-MM-DD HH:mm:ss) by default; hovering switches to a relative time (for example, “3 minutes ago”).
Triggering incidents
Trigger from an integration
Flashduty On-call supports most common monitoring systems, including Prometheus, Zabbix, Nightingale, and cloud monitoring. Go to Alert Integration for specific steps.Trigger with the API
Flashduty On-call provides a custom event standard, allowing you to report alerts using the standard protocol, suitable for any non-integrated monitoring system. For details, read Custom Alert Events.Trigger by email
Flashduty On-call provides email integration, allowing you to report alerts by sending emails, suitable for all monitoring systems supporting email alerts. For details, read Email Integration Guide.Trigger from the console
You can click the Create button on the incident list page to manually create an incident. Follow these steps:1
Enter incident title
Enter the incident title with a brief description of what happened. This field is required.
2
Select severity
Choose Critical, Warning, or Info.
3
Select channel
Select the channel the incident belongs to. If you create the incident inside a channel, Flashduty fills in this field and you can’t change it.
4
Select assignment method
After selecting a channel, the system displays assignment method options:
- By Policy: Select an escalation rule from the current channel. If no escalation rules are available in the channel, this option is disabled
- Direct Assignment: Select responders (individuals, schedules or teams). When you pick a team, Flashduty expands it into its members at the moment of assignment and notifies them; the expansion rules and the size limits are in Escalate and reassign incidents. In Notification, choose Follow personal preferences or Use unified settings. Unified settings require at least one notification channel. You can use the system default notification template or select an existing template. Direct assignment notifies individuals only; it does not send group chat notifications
5
Enter incident description (optional)
Enter a detailed description of the incident. The editor supports rich text and Markdown syntax.
Related topics
Understanding Noise Reduction
Understand the relationship between events, alerts, incidents and noise reduction principles
Search and View Incidents
Master incident list and details page usage
Handle and Update Incidents
Learn how to acknowledge, snooze, close incidents
Escalate and Reassign Incidents
Learn how to reassign and escalate