ClickHouse

Monitors supports using standard SQL syntax to query ClickHouse and trigger alerts based on query results.

​

Prerequisites

​

1. Threshold Evaluation Mode

This mode is suitable for scenarios requiring threshold comparison on aggregated values.

​

Configuration

1. Query Statement: Write SQL aggregate query, returning value columns and (optional) label columns.

• Example: Count error log quantity by service in the last 5 minutes.
  SELECT 
      service_name, 
      count(*) AS error_cnt 
  FROM app_log 
  WHERE timestamp > now() - INTERVAL 5 MINUTE AND level = 'error'
  GROUP BY service_name
  

2. Field Mapping:

• Value fields: Select error_cnt for threshold evaluation.
• Label fields: Select service_name to identify the alert object. After you select label fields, other non-value fields are carried with the alert as additional information.
• See Query Result Field Mapping for the complete behavior.

3. Threshold Conditions:

• Use $A.field_name to reference values.
• Example: Critical: $A.error_cnt > 50, Warning: $A.error_cnt > 10.

​

How It Works

Monitors distinguishes alert objects by their label fields and evaluates thresholds with their value fields. If Label fields is empty, every returned field except the value fields becomes a label.

​

Recovery Logic

​

2. Data Exists Mode

This mode is suitable for scenarios where filter logic is written directly in SQL.

​

Configuration

1. Query Statement: Use HAVING clause in SQL to directly filter out anomalous data.

• Example: Directly query services with error count exceeding 50.
  SELECT 
      service_name, 
      count(*) AS error_cnt 
  FROM app_log 
  WHERE timestamp > now() - INTERVAL 5 MINUTE AND level = 'error'
  GROUP BY service_name
  HAVING count(*) > 50
  

2. Evaluation Rules: As long as SQL query returns data, triggers alert.

​

Pros and Cons Analysis

​

Recovery Logic

• Recovery When Data Disappears: When SQL query result is empty, determines recovery
• Recovery Query: Supports configuring additional query statements to assist in determining recovery status
• Manual Close: Keep the alert active until it is closed manually

​

3. No Data Mode

This mode is used to monitor scenarios where “data is expected but actually missing”.

​

Configuration

1. Query Statement: Write a SQL query that is expected to continuously return data.

• Example: Query heartbeat reports from all probes.
  SELECT probe_id, max(timestamp) as last_seen
  FROM probe_heartbeat
  WHERE timestamp > now() - INTERVAL 5 MINUTE
  GROUP BY probe_id
  

2. Evaluation Rules: If a probe_id appeared in previous cycles but cannot be found in current and N consecutive cycles, triggers “No Data” alert.

​

Recovery Logic

No-data alerts support configuring the alert ending mode, which decides how the alert ends:

​

4. Best Practices