Core concepts
Threshold evaluation mode
This mode is suitable for scenarios requiring threshold comparison on aggregated values, such as monitoring “error log count in the last 5 minutes”.Configuration
- Query Statement: Write SQL aggregate query, returning value columns and (optional) grouping columns.
- Example: Count error log quantity by service in the last 5 minutes.
- Example: Count error log quantity by service in the last 5 minutes.
- Field Mapping:
- Label fields: Select
service_nameto identify the alert object. After you select label fields, other non-value fields are carried with the alert as additional information. - Value fields: Select
error_cntfor threshold evaluation. - If Label fields is empty, every returned field except the value fields becomes a label. See Query result field mapping for the complete behavior.
- Label fields: Select
- Threshold Conditions:
- Use
$A.field_nameto reference values. - Example:
Critical: $A.error_cnt > 50,Warning: $A.error_cnt > 10. - Shorthand: If only one value field is configured, you can directly use
$A, like$A > 50.
- Use
How it works
The engine executes SQL query and gets two-dimensional table data. It groups data by “label fields”, then extracts “value fields” values to compare against threshold expressions.Recovery logic
See Alert recovery for the recovery options. A recovery query is a separate SQL statement; it supports${label_name} variables and Go templates such as {{ $labels.<label_name> }} (alert engine v0.75.4 or later).
Recovery Query Example
Recovery Query Example
If the alert SQL found that network card with The engine will replace variables with actual values before executing the query. If data is found, recovery is determined.
network_host="a", interface="b" is down, the recovery SQL can be written as:Data exists mode
This mode is suitable for scenarios where filter logic is written directly in SQL, or when you only care about “whether data is returned”.Configuration
- Query Statement: Use
HAVINGclause in SQL to directly filter out anomalous data.- Example: Directly query services with error count exceeding 50.
- Example: Directly query services with error count exceeding 50.
- Field Mapping:
- Label fields and value fields are optional in this mode. If both are empty, Monitors treats every returned field as a label.
- To identify alerts with only stable fields and keep other columns as troubleshooting context, explicitly select Label fields. See Query result field mapping.
Recovery logic
See Alert recovery for the recovery options. “No data found” does not always mean recovery (log collection may be down). When you need a stricter condition, configure a recovery query: a separate SQL statement that, once it returns data, marks the alert as recovered. It can reference the alert event’s labels with${label_name}.
Data exists mode cannot distinguish alert levels; use threshold evaluation when you need levels.
No data mode
This mode is used to monitor scenarios where “data is expected but actually missing”, commonly used to monitor log collection pipeline interruption or periodic task non-execution.Configuration
- Query Statement: Write a SQL query that is expected to continuously return data.
- Example: Query log reporting heartbeat from all hosts.
- Example: Query log reporting heartbeat from all hosts.
- Evaluation Rules:
- The engine periodically executes this SQL.
- If a
host_nameappeared in previous cycles but no longer appears in current cycle (and N consecutive cycles) query results, triggers “No Data” alert. - Note: This is the opposite of data exists mode: data exists alerts when data is found, missing data alerts when data is not found.
Recovery logic
See Alert recovery for the recovery options.Use case
A common log alerting need: count ERROR logs in the last 5 minutes and alert above a threshold. The alert message then shows the most recent ERROR log as a sample. Configuration approach:- Main Alert Condition: Use threshold evaluation mode, SQL statement counts ERROR logs in the last 5 minutes, configure threshold conditions.
- Related Query: Configure a related query, SQL statement queries the most recent ERROR log, using
${service_name}and other variables to limit to specific service. - Rule Notes Description: Reference related query results in alert rule’s notes description, using
$relatesvariable to render the original log.