ElasticSearch

​

Core Concepts

​

1. Threshold Evaluation Mode

​

Configuration

1. Query Statement: Write SQL aggregate query, returning value columns and (optional) grouping columns.

• Example: Count error log quantity by service in the last 5 minutes.
  SELECT service_name, count(*) AS error_cnt 
  FROM "app-logs-*" 
  WHERE "@timestamp" > now() - INTERVAL 5 MINUTES AND log_level = 'ERROR'
  GROUP BY service_name
  

2. Field Mapping:

• Label fields: Select service_name to identify the alert object. After you select label fields, other non-value fields are carried with the alert as additional information.
• Value fields: Select error_cnt for threshold evaluation.
• If Label fields is empty, every returned field except the value fields becomes a label. See Query Result Field Mapping for the complete behavior.

3. Threshold Conditions:

• Use $A.field_name to reference values.
• Example: Critical: $A.error_cnt > 50, Warning: $A.error_cnt > 10.
• Shorthand: If only one value field is configured, you can directly use $A, like $A > 50.

​

How It Works

​

Recovery Logic

SELECT network_host, interface, status FROM "network-status-*" 
WHERE "@timestamp" > now() - INTERVAL 5 MINUTES 
  AND network_host = '${network_host}' 
  AND interface = '${interface}' 
  AND status = 'UP'

​

2. Data Exists Mode

​

Configuration

1. Query Statement: Use HAVING clause in SQL to directly filter out anomalous data.

• Example: Directly query services with error count exceeding 50.
  SELECT service_name, count(*) AS error_cnt 
  FROM "app-logs-*" 
  WHERE "@timestamp" > now() - INTERVAL 5 MINUTES AND log_level = 'ERROR'
  GROUP BY service_name
  HAVING count(*) > 50
  

2. Field Mapping:

• Label fields and value fields are optional in this mode. If both are empty, Monitors treats every returned field as a label.
• To identify alerts with only stable fields and keep other columns as troubleshooting context, explicitly select Label fields. See Query Result Field Mapping.

​

Recovery Logic

• Recovery When Data Disappears: When SQL query result is empty (i.e., no longer satisfies HAVING condition), the engine determines incident recovery. This is the most commonly used recovery method.
• Recovery Query:
  • Scenario: Sometimes “no data found” doesn’t mean recovery (might be log collection down), or need stricter recovery conditions (like no errors for N consecutive minutes).
  • Configuration: Write an independent SQL statement for recovery evaluation. As long as that query can find data, the incident is considered recovered.
  • Variable Support: Supports using ${label_name} in recovery SQL to reference alert event label values for precise recovery detection.
• Manual Close: Keep the alert active until it is closed manually.

​

Pros and Cons Analysis

​

3. No Data Mode

​

Configuration

1. Query Statement: Write a SQL query that is expected to continuously return data.

• Example: Query log reporting heartbeat from all hosts.
  SELECT host_name 
  FROM "heartbeat-logs-*" 
  WHERE "@timestamp" > now() - INTERVAL 5 MINUTES 
  GROUP BY host_name
  

2. Evaluation Rules:

• The engine periodically executes this SQL.
• If a host_name appeared in previous cycles but no longer appears in current cycle (and N consecutive cycles) query results, triggers “No Data” alert.
• Note: This is opposite to Data Exists mode. Data Exists means “alert when data found”, No Data means “alert when data not found”.

​

Recovery Logic

​

4. Use Case

• Main Alert Condition: Use Threshold mode, SQL statement counts ERROR logs in the last 5 minutes, configure threshold conditions.
• Related Query: Configure a related query, SQL statement queries the most recent ERROR log, using ${service_name} and other variables to limit to specific service.
• Rule Notes Description: Reference related query results in alert rule’s notes description, using $relates variable to render the original log.