Core concepts
Related queries and recovery queries support the
${label_name} variable and the Go template {{ $labels.<label_name> }}. The main query doesn’t support variables; a main query containing ${...} is rejected at save time with variables are only supported in relate and recovery queries.
The SQL after variable substitution is only known at runtime, so it can’t be pre-checked at save time. The alert engine renders the variables per alert instance and then checks that the rendered text is a read-only query; if it isn’t a read-only query, that evaluation run fails.
Query restrictions
A MySQL rule’s SQL must be a single read-only statement. The main query, relate queries, and recovery queries are checked by the same rules, and a rule that breaks them is rejected at save time with an error that starts with the field name, for examplerule config invalid: query(A) expr: ….
- The statement must start with
SELECT,WITH,SHOW,DESC,DESCRIBE, orEXPLAIN; anything else is refused withMySQL statement must start with SELECT, WITH, SHOW, DESC, DESCRIBE, or EXPLAIN. INTO,INSERT,UPDATE, andDELETEmay not appear anywhere in the statement, which is what stopsSELECT … INTO OUTFILE,WITH … DELETE, andINSERT … SELECT. The message isMySQL keyword INTO is not allowed, with the keyword upper-cased, and theINSERT(...)string function is not affected.- Only one statement is allowed; more than one returns
multiple MySQL statements are not allowed. - Executable comments are refused with
executable SQL comments are not allowed. This covers MySQL/*! … */, MariaDB/*M! … */, and TiDB/*T! … */; ordinary/* … */comments are fine. EXPLAIN ANALYZEis refused withMySQL EXPLAIN ANALYZE is not allowed; plainEXPLAINandDESCwork.
Threshold evaluation mode
This mode is suitable for scenarios requiring threshold comparison on aggregated values.Configuration
- Query Statement: Write SQL aggregate query, returning value columns and (optional) label columns.
- Example: Count error log quantity by service in the last 5 minutes (assuming there’s a log table).
- Example: Count error log quantity by service in the last 5 minutes (assuming there’s a log table).
- Field Mapping:
- Value fields: Select
error_cntfor threshold evaluation. - Label fields: Select
service_nameto identify the alert object. After you select label fields, other non-value fields are carried with the alert as additional information. - See Query result field mapping for the complete behavior.
- Value fields: Select
- Threshold Conditions:
- Use
$A.field_nameto reference values. - Example:
Critical: $A.error_cnt > 50,Warning: $A.error_cnt > 10.
- Use
How it works
Monitors distinguishes alert objects by their label fields and evaluates thresholds with their value fields. If Label fields is empty, every returned field except the value fields becomes a label.Recovery logic
See Alert recovery for the recovery options. A recovery query is a separate SQL statement that supports the${label_name} variable and the Go template {{ $labels.<label_name> }}. These rules are subject to the same version requirement as above: if any instance in the cluster is older than v0.76.0, saving or enabling the rule is rejected.
Data exists mode
This mode is suitable for scenarios where filter logic is written directly in SQL.Configuration
- Query Statement: Use
HAVINGclause in SQL to directly filter out anomalous data.- Example: Directly query services with error count exceeding 50.
- Example: Directly query services with error count exceeding 50.
- Evaluation Rules: As long as SQL query returns data (Result Set is not empty), triggers alert.
Recovery logic
See Alert recovery for the recovery options.No data mode
This mode is used to monitor scenarios where “data is expected but actually missing”.Configuration
- Query Statement: Write a SQL query that is expected to continuously return data.
- Example: Query heartbeat reports from all probes.
- Example: Query heartbeat reports from all probes.
- Evaluation Rules: If a
probe_idappeared in previous cycles but cannot be found in current and N consecutive cycles, triggers “No Data” alert.
Recovery logic
See Alert recovery for the recovery options.Best practices
Index Optimization
Index Optimization
Always include time range filtering in
WHERE clause and ensure the time field has an index, otherwise it may cause full table scan.Recommended syntax: log_time > now() - INTERVAL 5 MINUTEField Case
Field Case
Monitors converts column names returned by MySQL to lowercase. Use lowercase names for both label fields and value fields.