Skip to main content
Monitors implements log and metric data monitoring through ElasticSearch SQL feature, supporting flexible aggregate queries and alert evaluation.

Core concepts

Due to SQL feature dependency, only ElasticSearch 6.3 and above versions are supported.
Use explicit, stable column aliases in SQL and match their casing exactly in field mappings. This prevents mappings from breaking when query fields change.

Threshold evaluation mode

This mode is suitable for scenarios requiring threshold comparison on aggregated values, such as monitoring “error log count in the last 5 minutes”.

Configuration

  1. Query Statement: Write SQL aggregate query, returning value columns and (optional) grouping columns.
    • Example: Count error log quantity by service in the last 5 minutes.
  2. Field Mapping:
    • Label fields: Select service_name to identify the alert object. After you select label fields, other non-value fields are carried with the alert as additional information.
    • Value fields: Select error_cnt for threshold evaluation.
    • If Label fields is empty, every returned field except the value fields becomes a label. See Query result field mapping for the complete behavior.
  3. Threshold Conditions:
    • Use $A.field_name to reference values.
    • Example: Critical: $A.error_cnt > 50, Warning: $A.error_cnt > 10.
    • Shorthand: If only one value field is configured, you can directly use $A, like $A > 50.

How it works

The engine executes SQL query and gets two-dimensional table data. It groups data by “label fields”, then extracts “value fields” values to compare against threshold expressions.
Label field combination uniquely identifies an alert object. Query results cannot have multiple rows with the same label field value combination.

Recovery logic

See Alert recovery for the recovery options. A recovery query is a separate SQL statement; it supports ${label_name} variables and Go templates such as {{ $labels.<label_name> }} (alert engine v0.75.4 or later).
If the alert SQL found that network card with network_host="a", interface="b" is down, the recovery SQL can be written as:
The engine will replace variables with actual values before executing the query. If data is found, recovery is determined.

Data exists mode

This mode is suitable for scenarios where filter logic is written directly in SQL, or when you only care about “whether data is returned”.

Configuration

  1. Query Statement: Use HAVING clause in SQL to directly filter out anomalous data.
    • Example: Directly query services with error count exceeding 50.
  2. Field Mapping:
    • Label fields and value fields are optional in this mode. If both are empty, Monitors treats every returned field as a label.
    • To identify alerts with only stable fields and keep other columns as troubleshooting context, explicitly select Label fields. See Query result field mapping.

Recovery logic

See Alert recovery for the recovery options. “No data found” does not always mean recovery (log collection may be down). When you need a stricter condition, configure a recovery query: a separate SQL statement that, once it returns data, marks the alert as recovered. It can reference the alert event’s labels with ${label_name}. Data exists mode cannot distinguish alert levels; use threshold evaluation when you need levels.

No data mode

This mode is used to monitor scenarios where “data is expected but actually missing”, commonly used to monitor log collection pipeline interruption or periodic task non-execution.

Configuration

  1. Query Statement: Write a SQL query that is expected to continuously return data.
    • Example: Query log reporting heartbeat from all hosts.
  2. Evaluation Rules:
    • The engine periodically executes this SQL.
    • If a host_name appeared in previous cycles but no longer appears in current cycle (and N consecutive cycles) query results, triggers “No Data” alert.
    • Note: This is the opposite of data exists mode: data exists alerts when data is found, missing data alerts when data is not found.

Recovery logic

See Alert recovery for the recovery options.

Use case

A common log alerting need: count ERROR logs in the last 5 minutes and alert above a threshold. The alert message then shows the most recent ERROR log as a sample. Configuration approach:
  • Main Alert Condition: Use threshold evaluation mode, SQL statement counts ERROR logs in the last 5 minutes, configure threshold conditions.
  • Related Query: Configure a related query, SQL statement queries the most recent ERROR log, using ${service_name} and other variables to limit to specific service.
  • Rule Notes Description: Reference related query results in alert rule’s notes description, using $relates variable to render the original log.